Skip to content

Commit f4f51a2

Browse files
chandrasekharan-zipstackhari-kuriakose
chandrasekharan-zipstack
and
hari-kuriakose
authored
Script to run v2 migration commands (Zipstack#791)
* Script to run migration commands, some docs added for migration to v2, minor fix to run on OSS * Apply suggestions from code review Co-authored-by: Hari John Kuriakose <hari@zipstack.com> Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> * Script to support printing release warnings, addressed review comments * Addressed review comments, restructured migrating README.md, updated run-platform.sh to support showing release notes * Apply suggestions from code review Co-authored-by: Hari John Kuriakose <hari@zipstack.com> Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> * Minor formatting changes --------- Signed-off-by: Chandrasekharan M <117059509+chandrasekharan-zipstack@users.noreply.github.com> Co-authored-by: Hari John Kuriakose <hari@zipstack.com>
1 parent 826fc71 commit f4f51a2

7 files changed

Lines changed: 323 additions & 19 deletions

File tree

‎backend/account_v2/templates/login.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@
104104
</style>
105105
</head>
106106
<body>
107-
<form method="post" action="{% url 'login' %}">
107+
<form method="post" action="{% url 'public:login' %}">
108108
<div class="spinner">
109109
<!-- Spinner animation -->
110110
<div class="lds-dual-ring"></div>

‎backend/migrating/v2/README.md‎

Lines changed: 114 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,119 @@
1-
# V2 Migrations and Commands
1+
# V2 Migration Guide
22

33
This folder contains scripts and management commands for performing data migrations from v1 to v2 for the Unstract Multitenancy system.
4+
From version `v0.93.0` onwards, there have been signifcant changes in how data is stored. This warrants a data migration and the below steps can come in handy.
45

5-
## Overview
6+
> [!WARNING]
7+
> If you're updating from a very old version of Unstract and you face issues, its recommended to update to `v0.92.1` with
8+
`./run-platform.sh -u -v v0.92.1` first and then perform this data migration.
69

7-
The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.
10+
## Performing the migration
11+
12+
> [!TIP]
13+
> If the existing data is important to you, consider [backing up and restoring to a new database](/backend/migrating/v2/README.md#preparing-for-the-migration-safest-approach) beforehand.
14+
15+
Run the `migrate-to-v2.sh` script to perform schema and data migrations in one shot.
16+
17+
```shell
18+
./migrate-to-v2.sh
19+
```
20+
21+
If you prefer explicitly executing the migration commands, refer [these steps below](/backend/migrating/v2/README.md#manually-performing-V2-migration)
22+
23+
## Preparing for the migration (safest approach)
24+
25+
Follow below steps to backup existing data in case of any issues during the migration
26+
27+
### Backup existing data
28+
29+
- Make sure `pg_dump` is installed. This would usually be bundled with postgres and can be installed with
30+
31+
```shell
32+
sudo apt-get install postgresql-client
33+
```
34+
35+
- Take a backup of the existing data with the below command (provide the appropriate options)
36+
37+
```shell
38+
pg_dump -h localhost -p 5432 -U unstract_dev -d unstract_db -f unstract-backup-$(date +"%Y-%m-%d_%H-%M-%S").sql
39+
```
40+
41+
### Create a new database
42+
43+
- Create a new database, for example `unstract_db_v2`
44+
45+
### Restore backup to the new database
46+
47+
- Exec into the DB if required (from [/docker](/docker/))
48+
49+
```shell
50+
docker exec -it unstract-db /bin/bash
51+
```
52+
53+
- Execute the below command to restore the backup in the new DB
54+
55+
```shell
56+
psql -h localhost -p 5432 -U unstract_dev -d unstract_db_v2 -f unstract-backup-<DateTime>.sql
57+
```
58+
59+
## Manually performing V2 migration
60+
61+
### Configuring the env
62+
63+
- Make sure to update all the necessary `.env` files
64+
- For example, in case of `v2` ensure `backend`, `prompt-service`, and `platform-service` `.env` contain `DB_SCHEMA=unstract_v2`
65+
66+
### Create a new schema
67+
68+
- Create a new schema within the above DB, for example `unstract_v2`
69+
- This can be done directly in the DB or
70+
- Run the command supported from [/backend/manage.py](/backend/manage.py), for example in case of `v2`
871

72+
```shell
73+
python manage.py create_v2_schema
74+
```
75+
76+
### Run schema migrations
77+
78+
- Schema migrations are run by django, this gets invoked when running the backend service
79+
- Or run it explicitly with
80+
81+
```shell
82+
python manage.py migrate
83+
```
84+
85+
- This ensures that the required tables are created or altered
86+
87+
### Run data migrations
88+
89+
- This helps move the existing data from your current schema to the new schema
90+
- It can either be run locally or within the backend container
91+
92+
```shell
93+
docker exec -it unstract-backend /bin/bash
94+
```
95+
96+
- Make sure to activate the virtual environment
97+
98+
```shell
99+
source .venv/bin/activate
100+
```
101+
102+
- Perform the data migration
103+
104+
```shell
105+
python manage.py migrate_to_v2
106+
```
107+
108+
- Test to see if the applied migrations work as expected.
109+
110+
## Django Commands Reference
111+
112+
The commands provided here help you manage the migration process, including creating and dropping schemas and applying the necessary migrations to transition from v1 to v2.
9113

10114
**Pre-requisites:**
11115
Ensure that `migrating.v2` is added to the `SHARED_APPS` in your Django settings.
12116

13-
## Available Commands
14-
15117
### 1. List Commands
16118

17119
To view all available management commands, use:
@@ -27,6 +129,7 @@ To set up the v2 schema in your database, execute:
27129
```bash
28130
python manage.py create_v2_schema
29131
```
132+
30133
This command initializes the schema required for v2 migrations.
31134

32135
### 3. Drop V2 Schema
@@ -36,9 +139,11 @@ To remove the v2 schema from your database, run:
36139
```bash
37140
python manage.py drop_v2_schema
38141
```
142+
39143
Note: Use this command with caution as it will permanently delete the v2 schema and its associated data.
40144

41145
### 4. Migrate to V2 Schema
146+
42147
To apply migrations and transition data from v1 to v2, use:
43148

44149
```bash
@@ -47,13 +152,14 @@ python manage.py migrate_to_v2
47152

48153
This command performs the migration, updating your database to the v2 schema and transforming the data accordingly.
49154

50-
51155
## Notes
52-
- Ensure you have backups of your data before running migration commands.
156+
157+
- Ensure you have backed up your data before attempting any migration.
53158
- Verify your database configuration and environment variables are correctly set.
54159
- Review the migration logs for any issues or errors during the migration process.
55160

56-
For more details on each command and its options, refer to the command help output using
161+
For more details on each command and its options, refer to the command help output using
162+
57163
```bash
58164
python manage.py help <command_name>
59165
```

‎backend/migrating/v2/migrate-to-v2.sh‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/usr/bin/env bash
2+
3+
# Usage: ./migrate-to-v2.sh
4+
5+
set -o nounset # exit if a variable is not set
6+
set -o errexit # exit for any command failure"
7+
8+
# text color escape codes (\033 == \e but OSX doesn't respect the \e)
9+
blue_text='\033[94m'
10+
green_text='\033[32m'
11+
red_text='\033[31m'
12+
default_text='\033[39m'
13+
yellow_text='\033[33m'
14+
15+
# set -x/xtrace uses PS4 for more info
16+
PS4="$blue_text""${0}:${LINENO}: ""$default_text"
17+
18+
# Function to check and create virtual environment
19+
activate_venv() {
20+
if [[ ! -d ".venv" ]]; then
21+
echo "No .venv found, please create one and install dependencies. Refer 'run-platform.sh' or 'dev-env-cli.sh";
22+
exit 1;
23+
else
24+
echo "Virtual environment found. Activating..."
25+
source .venv/bin/activate || { echo "Failed to activate virtual environment"; exit 1; }
26+
fi
27+
}
28+
29+
script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")
30+
manage_script_dir=$(readlink -f "$script_dir/../../")
31+
root_dir=$(readlink -f "$manage_script_dir/..")
32+
33+
# Configure environment variables for all services
34+
echo -e "${blue_text}Configuring environment variables for the migration${default_text}"
35+
"$root_dir/run-platform.sh" -e || { echo "Failed to configure environment"; exit 1; }
36+
37+
cd $manage_script_dir
38+
activate_venv
39+
40+
echo -e "${blue_text}Running schema creation command...${default_text}"
41+
python manage.py create_v2_schema || { echo "Schema creation failed"; exit 1; }
42+
43+
echo -e "${blue_text}Running schema migration command...${default_text}"
44+
python manage.py migrate || { echo "Schema migration failed"; exit 1; }
45+
46+
echo -e "${blue_text}Running data migration command...${default_text}"
47+
SCHEMAS_TO_MIGRATE=_ALL_ python manage.py migrate_to_v2 || { echo "Data migration failed"; exit 1; }

‎run-platform.sh‎

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ default_text='\033[39m'
1111
yellow_text='\033[33m'
1212

1313
# set -x/xtrace uses PS4 for more info
14-
PS4="$blue_text""${0}:${LINENO}: ""$default_text"
14+
PS4="$blue_text"'${0}:${LINENO}: '"$default_text"
15+
1516

1617
debug() {
1718
if [ "$opt_verbose" = true ]; then
@@ -136,30 +137,33 @@ do_git_pull() {
136137
return
137138
fi
138139

140+
current_version=$(git describe --tags --abbrev=0)
139141
echo "Fetching release tags."
140142
git fetch --quiet --tags
141143

142144
if [[ "$opt_version" == "latest" ]]; then
143-
branch=`git describe --tags --abbrev=0`
145+
target_branch=`git ls-remote --tags origin | awk -F/ '{print $3}' | sort -V | tail -n1`
144146
elif [[ "$opt_version" == "main" ]]; then
145-
branch="main"
147+
target_branch="main"
146148
opt_build_local=true
147149
echo -e "Choosing ""$blue_text""local build""$default_text"" of Docker images from ""$blue_text""main""$default_text"" branch."
148150
elif [ -z $(git tag -l "$opt_version") ]; then
149151
echo -e "$red_text""Version not found.""$default_text"
150-
if [[ ! $opt_version == v* ]]; then
151-
echo -e "$red_text""Version must be provided with a 'v' prefix (e.g. v0.47.0).""$default_text"
152+
153+
version_regex="^v([0-9]+)\.([0-9]+)\.([0-9]+)(-[a-zA-Z0-9]+(\.[0-9]+)?)?$"
154+
if [[ ! $opt_version =~ $version_regex ]]; then
155+
echo -e "$red_text""Version must be provided with a 'v' prefix and follow SemVer (e.g. v0.47.0).""$default_text"
152156
fi
153157
exit 1
154158
else
155-
branch="$opt_version"
159+
target_branch="$opt_version"
156160
fi
157161

158-
echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$branch""$default_text""."
159-
git checkout --quiet $branch
162+
echo -e "Performing ""$blue_text""git checkout""$default_text"" to ""$blue_text""$target_branch""$default_text""."
163+
git checkout --quiet $target_branch
160164

161-
echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$branch""$default_text""."
162-
git pull --quiet $(git remote) $branch
165+
echo -e "Performing ""$blue_text""git pull""$default_text"" on ""$blue_text""$target_branch""$default_text""."
166+
git pull --quiet $(git remote) $target_branch
163167
}
164168

165169
copy_or_merge_envs() {
@@ -277,6 +281,9 @@ run_services() {
277281
else
278282
echo -e "$green_text""Updated platform to $opt_version version.""$default_text"
279283
fi
284+
285+
# Show release notes on version update if applicable
286+
python3 "$script_dir/scripts/release-notes/print_release_notes.py" "$current_version" "$target_branch"
280287
fi
281288
echo -e "\nOnce the services are up, visit ""$blue_text""http://frontend.unstract.localhost""$default_text"" in your browser."
282289
echo -e "\nSee logs with:"
@@ -314,6 +321,8 @@ script_dir=$(dirname "$(readlink -f "$BASH_SOURCE")")
314321
first_setup=false
315322
# Extract service names from docker compose file.
316323
services=($(VERSION=$opt_version $docker_compose_cmd -f $script_dir/docker/docker-compose.build.yaml config --services))
324+
current_version=""
325+
target_branch=""
317326

318327
display_banner
319328
parse_args $*

‎scripts/release-notes/README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Print Release Notes
2+
3+
Accepts `current_version` and `target_version` as inputs to print the necessary release notes / warnings to be shown while updating a release
4+
5+
Run below to get more information on the usage
6+
7+
```shell
8+
python print_release_notes.py -h
9+
```
10+
11+
Make sure to create / activate a virtual environment with the below commands
12+
13+
```shell
14+
pdm venv create -w virtualenv --with-pip
15+
eval $(pdm venv activate in-project)
16+
```

0 commit comments

Comments
 (0)