Upgrade ​

From v0.4, we provide a built-in utility to help upgrade your previous GreptimeDB deployment to the latest version if there are some breaking changes. It's recommended to use this method to upgrade your GreptimeDB with different versions.

The lowest version that can be upgraded by this tool is v0.3.0.

CLI ​

This tool is in the greptime binary. You need to prepare the corresponding binary of target version before start.

greptime cli export --help

greptime cli export --help

And the help text is as follows:

greptime-cli-export 

USAGE:
    greptime cli export [OPTIONS] --addr <ADDR> --output-dir <OUTPUT_DIR> --target <TARGET>

OPTIONS:
        --addr <ADDR>                  Server address to connect
        --database <DATABASE>          The name of the catalog to export. Default to "greptime-*""
                                       [default: ]
    -h, --help                         Print help information
    -j, --export-jobs <EXPORT_JOBS>    Parallelism of the export [default: 1]
        --max-retry <MAX_RETRY>        Max retry times for each job [default: 3]
        --output-dir <OUTPUT_DIR>      Directory to put the exported data. E.g.:
                                       /tmp/greptimedb-export
    -t, --target <TARGET>              Things to export [possible values: create-table, table-data]

greptime-cli-export 

USAGE:
    greptime cli export [OPTIONS] --addr <ADDR> --output-dir <OUTPUT_DIR> --target <TARGET>

OPTIONS:
        --addr <ADDR>                  Server address to connect
        --database <DATABASE>          The name of the catalog to export. Default to "greptime-*""
                                       [default: ]
    -h, --help                         Print help information
    -j, --export-jobs <EXPORT_JOBS>    Parallelism of the export [default: 1]
        --max-retry <MAX_RETRY>        Max retry times for each job [default: 3]
        --output-dir <OUTPUT_DIR>      Directory to put the exported data. E.g.:
                                       /tmp/greptimedb-export
    -t, --target <TARGET>              Things to export [possible values: create-table, table-data]

Here explains the meaning of some important options

• --addr: The gRPC address of the Frontend node or Standalone process.
• --output-dir: The directory to put the exported data. Give a path at your current machine. The exported SQL files will be put in that directory.
• --target: The things to export. create-table can export the CREATE TABLE clause for each table. table-data can export the data of each table alone with COPY FROM clause.

For a complete upgrade, you will need to execute this tools twice with each target options.

Example ​

Here is a complete example for upgrading from v0.3.0 to v0.4.0.

In the following text, we assume that you have a Frontend's gRPC endpoint is available at 127.0.0.1:4001. The output dir is /tmp/greptimedb-export.

Export CREATE TABLE ​

greptime cli export --addr '127.0.0.1:4001' --output-dir /tmp/greptimedb-export --target create-table

greptime cli export --addr '127.0.0.1:4001' --output-dir /tmp/greptimedb-export --target create-table

If success, you will see something like this

2023-10-20T09:41:06.500390Z  INFO cmd::cli::export: finished exporting greptime.public with 434 tables
2023-10-20T09:41:06.500482Z  INFO cmd::cli::export: success 1/1 jobs

2023-10-20T09:41:06.500390Z  INFO cmd::cli::export: finished exporting greptime.public with 434 tables
2023-10-20T09:41:06.500482Z  INFO cmd::cli::export: success 1/1 jobs

And now the output directory structure is

/tmp/greptimedb-export
└── greptime-public.sql

/tmp/greptimedb-export
└── greptime-public.sql

Export table data ​

greptime cli export --addr '127.0.0.1:4001' --database greptime-public --output-dir /tmp/greptimedb-export --target table-data

greptime cli export --addr '127.0.0.1:4001' --database greptime-public --output-dir /tmp/greptimedb-export --target table-data

The log output is similar to the previous one. And the output directory structure is

/tmp/greptimedb-export
├── greptime-public
│   ├── up.parquet
│   └── other-tables.parquet
├── greptime-public_copy_from.sql
└── greptime-public.sql

/tmp/greptimedb-export
├── greptime-public
│   ├── up.parquet
│   └── other-tables.parquet
├── greptime-public_copy_from.sql
└── greptime-public.sql

New files are greptime-public_copy_from.sql and greptime-public. The former one contains the COPY FROM clause for each table. The latter one contains the data of each table.

Import table schema and data ​

Then you need to execute SQL files generated by the previous step. First is greptime-public.sql. SQL generated in previous step is in PostgreSQL dialect, we will use PG protocol in the following steps. This document assumes the client is psql.

Before executing the following command, you need first to create the corresponding database in your new deployment (but in this example, the database greptime-public is the default one).

This command will create all the tables in the new version of GreptimeDB.

psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptime-public.sql

psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptime-public.sql

And then import the data

psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptime-public_copy_from.sql

psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptime-public_copy_from.sql

Clean up ​

At this step all the data is migrated. You can check the data in the new cluster.

After confirming that the data is correct, you can clean up the old cluster and the temporary --output-dir. /tmp/greptimedb-export at this example.

Recommended overall process ​

This section gives a recommended overall process for upgrading GreptimeDB smoothly. You can skip this section if your environment can go offline on the upgrade progress.

1. Create a brand new v0.4 cluster
2. Export and import create-table
3. Switch workload to the new cluster
4. Export and import table data

Caveats

• Changes to table structure between step 2 and 3 will be lost
• Old data is invisible in new cluster until step 4 is done