# GreptimeDB Documentation
> GreptimeDB is an open-source observability database for metrics, logs, traces, and wide events. Drop-in replacement for Prometheus, Loki & Elasticsearch, or the single backend for OpenTelemetry.
This file contains all documentation content in a single document following the llmstxt.org standard.
## GreptimeDB
# GreptimeDB Documentation
**GreptimeDB** is an open-source observability database that stores metrics, logs, and traces in one engine. Run it as the single OpenTelemetry backend in place of Prometheus, Loki, and Elasticsearch, keep the data on object storage, and query all of it with [SQL](/user-guide/query-data/sql.md) and [PromQL](/user-guide/query-data/promql.md).
- **[Getting Started](/getting-started/overview.md)**
Install GreptimeDB, write your first rows, and run your first query.
- **[User Guide](/user-guide/overview.md)**
Ingestion protocols, queries, pipelines, flows, and running GreptimeDB in production.
- **[For AI Agents](/faq-and-others/vibecoding.md)**
Run GreptimeDB from a coding agent, with no custom integration.
- **[Reference](/reference/sql/overview.md)**
SQL syntax, functions, and data types.
## Ask about GreptimeDB
Installation, ingestion, queries, configuration — answered from this documentation.
## Why GreptimeDB
**Replace three systems with one.** Metrics, logs, and traces go into one columnar engine with native OpenTelemetry support, instead of a separate store, query language, and operational overhead for each signal.
**Pay object-storage prices.** S3, Azure Blob, and GCS are the primary store, and compute scales independently of them. Columnar storage and observability-tuned compression cut cost by up to 50x — [OceanBase Cloud](/user-guide/concepts/why-greptimedb.md#what-production-users-report) keeps 300 TB of logs and audit data on GreptimeDB and reports 60%+ lower storage cost after moving from Loki.
**Keep your collectors and dashboards.** [Prometheus remote write](/user-guide/ingest-data/for-observability/prometheus.md) and [OTLP](/user-guide/ingest-data/for-observability/opentelemetry.md) on the write path, [PromQL](/user-guide/query-data/promql.md) and the [Jaeger API](/user-guide/query-data/jaeger.md) on the read path, each on its own HTTP endpoint. SQL arrives over the [MySQL](/user-guide/protocols/mysql.md) and [PostgreSQL](/user-guide/protocols/postgresql.md) wire protocols. [Grafana](/user-guide/integrations/grafana.md) dashboards keep working. Each [protocol page](/user-guide/protocols/overview.md) states what it covers and what it does not.
**Give agents one query interface.** In [Agent RCA Bench](https://rca-bench.greptime.com/), six models investigated the same 14 incidents on GreptimeDB and on Prometheus, Loki, and Tempo: 40% fewer wrong diagnoses, 48% fewer input tokens read, about 45% lower cost to run.
More in [Why GreptimeDB](/user-guide/concepts/why-greptimedb.md) and [Observability 2.0 and wide events](/user-guide/concepts/observability-2.md).
## Build with AI agents
The quickstart guide is hosted as a fetchable Markdown file. A coding agent reads it at runtime, with no installation required.
> **AGENT ONBOARDING:** Read https://docs.greptime.com/SKILL.md and follow the instructions to use GreptimeDB with your AI agent — deploy, configure, ingest, and query.
- **[MCP Server](/user-guide/integrations/mcp.md)** — query tools an agent can call: SQL, TQL, and range queries, read-only by default. It also manages pipelines and dashboards, which write.
- **[Skills](/faq-and-others/vibecoding.md)** — pipelines, flows, triggers, table design, and performance diagnosis, in the [Agent Skills](https://agentskills.io/) format.
- **[Semantic layer](/user-guide/concepts/semantic-layer.md)** — `greptime.semantic.*` table metadata recording the signal, source, and unit behind each table.
- **[llms.txt](https://docs.greptime.com/llms.txt)** — a structured index of the whole site; append `.md` to any page URL for its raw Markdown.
## Keep exploring
- [Tutorials](/tutorials/k8s-metrics-monitor.md): end-to-end walkthroughs, such as monitoring a Kubernetes cluster.
- [GreptimeDB Enterprise](/enterprise/overview.md): read replicas, workload isolation, RBAC, audit logging, and disaster recovery.
- [Contributor Guide](/contributor-guide/overview.md): internals, for anyone working on GreptimeDB itself.
- [Roadmap](https://greptime.com/blogs/2026-02-11-greptimedb-roadmap-2026): the project's direction and planned work.
- [Release Notes](/release-notes): every released version.
- [FAQ](/faq-and-others/faq.md): common questions about deployment, ingestion, and queries.
---
## GreptimeDB Cluster
The GreptimeDB cluster can run in cluster mode to scale horizontally.
## Deploy the GreptimeDB cluster in Kubernetes
For production environments, we recommend deploying the GreptimeDB cluster in Kubernetes. Please refer to [Deploy on Kubernetes](/user-guide/deployments-administration/deploy-on-kubernetes/overview.md).
## Use Docker Compose
:::tip NOTE
Although Docker Compose is a convenient way to run the GreptimeDB cluster, it is only for development and testing purposes.
For production environments or benchmarking, we recommend using Kubernetes.
:::
### Prerequisites
Using Docker Compose is the easiest way to run the GreptimeDB cluster. Before starting, make sure you have already installed the Docker.
### Step1: Download the YAML file for Docker Compose
```
wget https://raw.githubusercontent.com/GreptimeTeam/greptimedb/v1.2.1/docker/docker-compose/cluster-with-etcd.yaml
```
### Step2: Start the cluster
```
GREPTIMEDB_VERSION=v1.2.1 \
docker compose -f ./cluster-with-etcd.yaml up
```
If the cluster starts successfully, it will listen on 4000-4003. , you can access the cluster by referencing the [Quick Start](../quick-start.md).
### Clean up
You can use the following command to stop the cluster:
```
docker compose -f ./cluster-with-etcd.yaml down
```
By default, the data will be stored in `./greptimedb-cluster-docker-compose`. You also can remove the data directory if you want to clean up the data.
## Next Steps
Learn how to write data to GreptimeDB in [Quick Start](../quick-start.md).
---
## GreptimeDB Dashboard
GreptimeDB stores metrics, logs, and traces in a single database. The [Dashboard](https://github.com/GreptimeTeam/dashboard) is a built-in web UI for exploring that data after installation — no extra components required. For the motivation behind unified observability storage, see [Why GreptimeDB](/user-guide/concepts/why-greptimedb.md).
## Access the Dashboard
The Dashboard is embedded in the GreptimeDB binary since v0.2.0. After starting [GreptimeDB Standalone](greptimedb-standalone.md) or [GreptimeDB Cluster](greptimedb-cluster.md), open:
```
http://localhost:4000/dashboard
```
To disable the Dashboard HTTP service, set `--disable-dashboard` when starting the frontend. See [frontend command-line options](/reference/command-lines/frontend.md).
## Explore your data
The Dashboard provides a general query entry and signal-specific views:
| View | Data scope | Description |
| --- | --- | --- |
| [Table Query](#table-query) | Any data | General query entry. Switch between SQL and PromQL in the editor. |
| [Metrics Query](#metrics-query) | Metrics | Browse and query metrics with a PromQL-oriented UI. |
| [Logs Query](#logs-query) | Logs | Filter and search logs with a builder or code editor. |
| [Traces Query](#traces-query) | Traces | Search traces and inspect spans with a builder or code editor. |
## Table Query
Table Query is the Dashboard's general query entry. You can query any data in GreptimeDB and switch between SQL and PromQL in the editor. Browse databases and tables on the left, run queries with **Run Query**, **Explain Query**, or **Run All**, and view results below.
- Browse databases and tables in the left panel.
- Write multiple queries in a single editor.
- Inspect results in a table with pagination.
- Use **Explain Query** to visualize [`EXPLAIN ANALYZE`](/reference/sql/explain.md) execution plans.
See [SQL](/user-guide/query-data/sql.md) and [PromQL](/user-guide/query-data/promql.md) for query syntax.
## Metrics Query
Metrics Query is a metrics-specific view. Browse available metrics, enter a metric name or PromQL expression, and switch between list and chart views.
- Search and browse metrics in the left panel.
- Run queries from the editor and view label sets with values.
- Toggle between list and chart display.
See [PromQL](/user-guide/query-data/promql.md) for query syntax. You can also run PromQL from [Table Query](#table-query).
## Logs Query
Logs Query is a logs-specific view. Select a database and log table, add filters, and inspect log lines. Use **Builder** for point-and-click queries or **Code** to write queries directly.
- Set time range, filters, order, and limit in the builder.
- View a row-count chart over time above the results.
- Enable **Live** for streaming results or export results as CSV.
See [Log Query](/user-guide/query-data/log-query.md) and [Logs](/user-guide/logs/overview.md) for more details.
## Traces Query
Traces Query is a traces-specific view. Select a trace table, filter by trace ID or span attributes, and inspect trace records.
- Use **Builder** or **Code** to compose queries.
- Filter root spans (for example, `parent_span_id` not exist) or search by `trace_id`.
See [Traces](/user-guide/traces/overview.md) and [Jaeger](/user-guide/query-data/jaeger.md) for more details.
## Visualization
The Dashboard embeds [Perses](https://perses.dev/) for building observability dashboards. Click **Visualization** in the sidebar to create, edit, and browse dashboards — no separate Perses installation required.
GreptimeDB pre-configures two datasource plugins that connect to your local instance:
| Query type | Datasource plugin | Panel types | Use for |
| --- | --- | --- | --- |
| **PromQL** | `PrometheusDatasource` | `TimeSeriesChart`, `GaugeChart`, `StatChart` | Prometheus metrics, `node_exporter`, migrated Grafana dashboards |
| **SQL time series** | `GreptimeDBDatasource` | `TimeSeriesChart`, `StatChart`, `Table` | GreptimeDB tables with `RANGE`, `ALIGN`, and `FILL` |
| **Logs** | `GreptimeDBDatasource` | `LogsTable` | Log tables with SQL filters and dashboard variables |
| **Traces** | `GreptimeDBDatasource` | `TraceTable`, `TracingGanttChart` | Trace tables — trace list and per-trace Gantt view |
For example, the built-in **Node Exporter** dashboard uses PromQL to show CPU, memory, disk, and network panels with variables such as `job` and `host`:
See [Perses integration](/user-guide/integrations/perses.md) for datasource connection, panel configuration, and Grafana migration.
## More capabilities
The Dashboard also includes:
### Logs Pipelines
Create, edit, and test log pipelines in the UI. See [Manage Pipelines](/user-guide/logs/manage-pipelines.md).
### Ingest
Write data manually in [InfluxDB Line Protocol](/user-guide/ingest-data/for-iot/influxdb-line-protocol.md) format.
### Flow
View and manage Flow tasks for continuous computation. See [Manage Flow](/user-guide/flow-computation/manage-flow.md).
---
## GreptimeDB Standalone
We use the simplest configuration for you to get started. For a comprehensive list of configurations available in GreptimeDB, see the [configuration documentation](/user-guide/deployments-administration/configuration.md).
## Deploy the GreptimeDB standalone in Kubernetes
For production environments, we recommend deploying the GreptimeDB standalone in Kubernetes. Please refer to [Deploy on Kubernetes](/user-guide/deployments-administration/deploy-on-kubernetes/overview.md).
## Binary
### Download from website
You can try out GreptimeDB by downloading the latest stable build releases from the [Download page](https://greptime.com/download).
### Linux and macOS
For Linux and macOS users, you can download the latest build of the `greptime` binary by using the following commands:
```shell
curl -fsSL \
https://raw.githubusercontent.com/greptimeteam/greptimedb/main/scripts/install.sh | sh -s v1.2.1
```
Once the download is completed, the binary file `greptime` will be stored in your current directory.
You can run GreptimeDB in the standalone mode:
```shell
./greptime standalone start
```
### Windows
If you have WSL([Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/about)) enabled, you can launch a latest Ubuntu and run GreptimeDB like above!
Otherwise please download the GreptimeDB binary for Windows at our [official site](https://greptime.com/resources), and unzip the downloaded artifact.
To run GreptimeDB in standalone mode, open a terminal (like Powershell) at the directory where the GreptimeDB binary locates, and execute:
```shell
.\greptime standalone start
```
## Docker
Make sure the [Docker](https://www.docker.com/) is already installed. If not, you can follow the official [documents](https://www.docker.com/get-started/) to install Docker.
```shell
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:v1.2.1 standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
```
:::tip NOTE
To avoid accidentally exit the Docker container, you may want to run it in the "detached" mode: add the `-d` flag to
the `docker run` command.
:::
The data will be stored in the `greptimedb_data/` directory in your current directory.
If you want to use another version of GreptimeDB's image, you can download it from our [GreptimeDB Dockerhub](https://hub.docker.com/r/greptime/greptimedb). In particular, we support GreptimeDB based on CentOS, and you can try image `greptime/greptimedb-centos`.
:::tip NOTE
If you are using a Docker version lower than [v23.0](https://docs.docker.com/engine/release-notes/23.0/), you may experience problems with insufficient permissions when trying to run the command above, due to a [bug](https://github.com/moby/moby/pull/42681) in the older version of Docker Engine.
You can:
1. Set `--security-opt seccomp=unconfined`, for example:
```shell
docker run --security-opt seccomp=unconfined -p 4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:v1.2.1 standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
```
2. Upgrade the Docker version to v23.0.0 or higher;
:::
## Binding address
GreptimeDB binds to `127.0.0.1` by default. If you need to accept connections from other addresses, you can start with the following parameters.
> :::danger Warning
> If the computer running GreptimeDB is directly exposed to the internet, binding to `0.0.0.0` is dangerous and will expose the instance to everybody on the internet.
```shell
./greptime standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
```
```shell
docker run -p 0.0.0.0:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:v1.2.1 standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
```
You can also refer to the [Configuration](/user-guide/deployments-administration/configuration.md) document to modify the bind address in the configuration file.
## Next Steps
Learn how to write data to GreptimeDB in the [Quick Start](../quick-start.md).
---
## GreptimeDB Installation Overview
# Installation
Follow these instructions to install GreptimeDB:
- [GreptimeDB Standalone](greptimedb-standalone.md) runs as a standalone system in a single process.
- [GreptimeDB Cluster](greptimedb-cluster.md) runs as a distributed cluster.
## Check database health
After starting GreptimeDB, you can check its status to ensure it is running.
```shell
curl http://localhost:4000/health
```
If the GreptimeDB instance is running healthily, you will see the following response:
```json
{}
```
## Next steps
- [Quick Start](/getting-started/quick-start.md): Ingest and query data in GreptimeDB using MySQL or PostgreSQL clients.
---
## GreptimeDB Getting Started Overview
# Getting Started
> **AGENT ONBOARDING:** Read https://docs.greptime.com/SKILL.md and follow the instructions to use GreptimeDB with your AI agent — deploy, configure, ingest, and query.
Get started with GreptimeDB quickly by following these steps:
- [Installation](./installation/overview.md): Learn how to install GreptimeDB as a standalone or cluster.
- [Quick Start](./quick-start.md): Get started with GreptimeDB using your preferred protocols or languages.
- [For AI Agents](../faq-and-others/vibecoding.md): Use GreptimeDB with AI agents via the MCP Server, Skills, and machine-readable docs.
---
## Quick Start
Before proceeding, please ensure you have [installed GreptimeDB](./installation/overview.md).
This guide uses SQL to walk you through GreptimeDB's core capabilities — from ingestion to cross-signal correlation across metrics, logs, and traces. SQL is also GreptimeDB's management interface for creating tables, setting TTL policies, and configuring indexes.
:::tip Already running Prometheus, OpenTelemetry, Loki or ES?
You can start ingesting data immediately using your existing tools — no schema creation needed (GreptimeDB [creates tables automatically](/user-guide/ingest-data/overview.md#automatic-schema-generation)):
- [Prometheus Remote Write](/user-guide/ingest-data/for-observability/prometheus.md)
- [OpenTelemetry (OTLP)](/user-guide/ingest-data/for-observability/opentelemetry.md)
- [Loki Protocol](/user-guide/ingest-data/for-observability/loki.md)
- [Elasticsearch](/user-guide/ingest-data/for-observability/elasticsearch/)
Continue with this guide to see what you can do with the data once it's in.
:::
**You'll learn (10–15 minutes):**
- Connect to GreptimeDB and create metrics, logs, and traces tables
- Query and aggregate data with SQL
- Search logs by keyword with full-text index
- Compute p95 latency in time windows using range queries
- **Correlate metrics, logs, and traces in a single query**
- Combine SQL and PromQL
## Connect to GreptimeDB
GreptimeDB supports [multiple protocols](/user-guide/protocols/overview.md) for interacting with the database. In this guide, we use SQL for simplicity.
If your GreptimeDB instance is running on `127.0.0.1` with the MySQL client default port `4002` or the PostgreSQL client default port `4003`, connect using:
```shell
mysql -h 127.0.0.1 -P 4002
```
Or
```shell
psql -h 127.0.0.1 -p 4003 -d public
```
You can also use the built-in Dashboard at `http://127.0.0.1:4000/dashboard` to run all the SQL queries in this guide.
By default, GreptimeDB does not have [authentication](/user-guide/deployments-administration/authentication/overview.md) enabled. You can connect without providing a username and password.
## Create tables
We'll create three tables to simulate a real scenario: gRPC latency metrics, application logs, and request traces. Two application servers, `host1` and `host2`, are recording data. Starting from `2024-07-11 20:00:10`, `host1` begins experiencing issues.
### Metrics table
```sql
-- Metrics: gRPC call latency in milliseconds
CREATE TABLE grpc_latencies (
ts TIMESTAMP TIME INDEX,
host STRING,
method_name STRING,
latency DOUBLE,
PRIMARY KEY (host, method_name)
);
```
- `ts`: Timestamp when the metric was collected (the [time index](/user-guide/concepts/data-model.md)).
- `host` and `method_name`: [Tag](/user-guide/concepts/data-model.md) columns identifying the time series.
- `latency`: [Field](/user-guide/concepts/data-model.md) column containing the actual measurement.
### Logs table
```sql
-- Logs: application error logs
CREATE TABLE app_logs (
ts TIMESTAMP TIME INDEX,
host STRING,
api_path STRING,
log_level STRING,
log_msg STRING FULLTEXT INDEX WITH('case_sensitive' = 'false'),
PRIMARY KEY (host, log_level)
) WITH ('append_mode'='true');
```
- `log_msg` enables [full-text index](/user-guide/manage-data/data-index.md#full-text-index) for keyword search.
- [`append_mode`](/user-guide/deployments-administration/performance-tuning/design-table.md#when-to-use-append-only-tables) optimizes for log workloads (no deduplication overhead).
### Traces table
```sql
-- Traces: request spans
CREATE TABLE traces (
ts TIMESTAMP TIME INDEX,
trace_id STRING SKIPPING INDEX,
span_id STRING,
parent_span_id STRING,
service_name STRING,
operation STRING,
duration DOUBLE,
status_code INT,
PRIMARY KEY (service_name)
) WITH ('append_mode'='true');
```
For high-cardinality `trace_id`s, we have enabled the [skip index](/user-guide/manage-data/data-index.md#skipping-index).
:::tip
We use SQL to ingest data below, so we create the tables manually. However, GreptimeDB is [schemaless](/user-guide/ingest-data/overview.md#automatic-schema-generation) — when using protocols like OpenTelemetry, Prometheus Remote Write, or InfluxDB Line Protocol, tables are created automatically.
:::
## Write data
Let's insert sample data simulating the scenario. Before `20:00:10`, both hosts are normal. After `20:00:10`, `host1` starts experiencing latency spikes.
### Normal period (before 20:00:10)
```sql
INSERT INTO grpc_latencies (ts, host, method_name, latency) VALUES
('2024-07-11 20:00:06', 'host1', 'GetUser', 103.0),
('2024-07-11 20:00:06', 'host2', 'GetUser', 113.0),
('2024-07-11 20:00:07', 'host1', 'GetUser', 103.5),
('2024-07-11 20:00:07', 'host2', 'GetUser', 107.0),
('2024-07-11 20:00:08', 'host1', 'GetUser', 104.0),
('2024-07-11 20:00:08', 'host2', 'GetUser', 96.0),
('2024-07-11 20:00:09', 'host1', 'GetUser', 104.5),
('2024-07-11 20:00:09', 'host2', 'GetUser', 114.0);
```
### Anomalous period (after 20:00:10)
`host1`'s latency becomes unstable with spikes up to several thousand milliseconds:
Click to expand INSERT statements
```sql
INSERT INTO grpc_latencies (ts, host, method_name, latency) VALUES
('2024-07-11 20:00:10', 'host1', 'GetUser', 150.0),
('2024-07-11 20:00:10', 'host2', 'GetUser', 110.0),
('2024-07-11 20:00:11', 'host1', 'GetUser', 200.0),
('2024-07-11 20:00:11', 'host2', 'GetUser', 102.0),
('2024-07-11 20:00:12', 'host1', 'GetUser', 1000.0),
('2024-07-11 20:00:12', 'host2', 'GetUser', 108.0),
('2024-07-11 20:00:13', 'host1', 'GetUser', 80.0),
('2024-07-11 20:00:13', 'host2', 'GetUser', 111.0),
('2024-07-11 20:00:14', 'host1', 'GetUser', 4200.0),
('2024-07-11 20:00:14', 'host2', 'GetUser', 95.0),
('2024-07-11 20:00:15', 'host1', 'GetUser', 90.0),
('2024-07-11 20:00:15', 'host2', 'GetUser', 115.0),
('2024-07-11 20:00:16', 'host1', 'GetUser', 3000.0),
('2024-07-11 20:00:16', 'host2', 'GetUser', 95.0),
('2024-07-11 20:00:17', 'host1', 'GetUser', 320.0),
('2024-07-11 20:00:17', 'host2', 'GetUser', 115.0),
('2024-07-11 20:00:18', 'host1', 'GetUser', 3500.0),
('2024-07-11 20:00:18', 'host2', 'GetUser', 95.0),
('2024-07-11 20:00:19', 'host1', 'GetUser', 100.0),
('2024-07-11 20:00:19', 'host2', 'GetUser', 115.0),
('2024-07-11 20:00:20', 'host1', 'GetUser', 2500.0),
('2024-07-11 20:00:20', 'host2', 'GetUser', 95.0);
```
### Error logs during the anomaly
```sql
INSERT INTO app_logs (ts, host, api_path, log_level, log_msg) VALUES
('2024-07-11 20:00:10', 'host1', '/api/v1/resource', 'ERROR', 'Connection timeout'),
('2024-07-11 20:00:10', 'host1', '/api/v1/billings', 'ERROR', 'Connection timeout'),
('2024-07-11 20:00:11', 'host1', '/api/v1/resource', 'ERROR', 'Database unavailable'),
('2024-07-11 20:00:11', 'host1', '/api/v1/billings', 'ERROR', 'Database unavailable'),
('2024-07-11 20:00:12', 'host1', '/api/v1/resource', 'ERROR', 'Service overload'),
('2024-07-11 20:00:12', 'host1', '/api/v1/billings', 'ERROR', 'Service overload'),
('2024-07-11 20:00:13', 'host1', '/api/v1/resource', 'ERROR', 'Connection reset'),
('2024-07-11 20:00:13', 'host1', '/api/v1/billings', 'ERROR', 'Connection reset'),
('2024-07-11 20:00:14', 'host1', '/api/v1/resource', 'ERROR', 'Timeout'),
('2024-07-11 20:00:14', 'host1', '/api/v1/billings', 'ERROR', 'Timeout'),
('2024-07-11 20:00:15', 'host1', '/api/v1/resource', 'ERROR', 'Disk full'),
('2024-07-11 20:00:15', 'host1', '/api/v1/billings', 'ERROR', 'Disk full'),
('2024-07-11 20:00:16', 'host1', '/api/v1/resource', 'ERROR', 'Network issue'),
('2024-07-11 20:00:16', 'host1', '/api/v1/billings', 'ERROR', 'Network issue');
```
### Trace spans during the anomaly
```sql
INSERT INTO traces (ts, trace_id, span_id, parent_span_id, service_name, operation, duration, status_code) VALUES
('2024-07-11 20:00:12', 'abc123', 'span1', '', 'host1', 'POST /api/v1/resource', 1050.0, 2),
('2024-07-11 20:00:12', 'abc123', 'span2', 'span1', 'host1', 'GetUser', 1000.0, 2),
('2024-07-11 20:00:14', 'def456', 'span3', '', 'host1', 'POST /api/v1/billings', 4250.0, 2),
('2024-07-11 20:00:14', 'def456', 'span4', 'span3', 'host1', 'CreateBilling', 4200.0, 2),
('2024-07-11 20:00:16', 'ghi789', 'span5', '', 'host1', 'POST /api/v1/resource', 3100.0, 2),
('2024-07-11 20:00:16', 'ghi789', 'span6', 'span5', 'host1', 'GetUser', 3000.0, 2),
('2024-07-11 20:00:12', 'jkl012', 'span7', '', 'host2', 'POST /api/v1/resource', 115.0, 0),
('2024-07-11 20:00:12', 'jkl012', 'span8', 'span7', 'host2', 'GetUser', 108.0, 0);
```
## Query data
### Filter by tags and time index
Query the latency of `host1` after `2024-07-11 20:00:15`:
```sql
SELECT *
FROM grpc_latencies
WHERE host = 'host1' AND ts > '2024-07-11 20:00:15';
```
```sql
+---------------------+-------+-------------+---------+
| ts | host | method_name | latency |
+---------------------+-------+-------------+---------+
| 2024-07-11 20:00:16 | host1 | GetUser | 3000 |
| 2024-07-11 20:00:17 | host1 | GetUser | 320 |
| 2024-07-11 20:00:18 | host1 | GetUser | 3500 |
| 2024-07-11 20:00:19 | host1 | GetUser | 100 |
| 2024-07-11 20:00:20 | host1 | GetUser | 2500 |
+---------------------+-------+-------------+---------+
5 rows in set (0.14 sec)
```
Calculate the 95th percentile latency grouped by host:
```sql
SELECT
host,
approx_percentile_cont(0.95) WITHIN GROUP (ORDER BY latency) AS p95_latency
FROM grpc_latencies
WHERE ts >= '2024-07-11 20:00:10'
GROUP BY host;
```
```sql
+-------+-------------------+
| host | p95_latency |
+-------+-------------------+
| host1 | 4164.999999999999 |
| host2 | 115 |
+-------+-------------------+
2 rows in set (0.11 sec)
```
### Search logs by keyword
The `@@` operator performs [full-text search](/user-guide/logs/fulltext-search.md) on indexed columns:
```sql
SELECT *
FROM app_logs
WHERE lower(log_msg) @@ 'timeout'
AND ts > '2024-07-11 20:00:00'
ORDER BY ts;
```
```sql
+---------------------+-------+------------------+-----------+--------------------+
| ts | host | api_path | log_level | log_msg |
+---------------------+-------+------------------+-----------+--------------------+
| 2024-07-11 20:00:10 | host1 | /api/v1/billings | ERROR | Connection timeout |
| 2024-07-11 20:00:10 | host1 | /api/v1/resource | ERROR | Connection timeout |
| 2024-07-11 20:00:14 | host1 | /api/v1/billings | ERROR | Timeout |
| 2024-07-11 20:00:14 | host1 | /api/v1/resource | ERROR | Timeout |
+---------------------+-------+------------------+-----------+--------------------+
```
### Range query
Use [range queries](/reference/sql/range.md) to calculate the p95 latency in 5-second windows:
```sql
SELECT
ts,
host,
approx_percentile_cont(0.95) WITHIN GROUP (ORDER BY latency)
RANGE '5s' AS p95_latency
FROM grpc_latencies
ALIGN '5s' FILL PREV
ORDER BY host, ts;
```
```sql
+---------------------+-------+-------------+
| ts | host | p95_latency |
+---------------------+-------+-------------+
| 2024-07-11 20:00:05 | host1 | 104.5 |
| 2024-07-11 20:00:10 | host1 | 4200 |
| 2024-07-11 20:00:15 | host1 | 3500 |
| 2024-07-11 20:00:20 | host1 | 2500 |
| 2024-07-11 20:00:05 | host2 | 114 |
| 2024-07-11 20:00:10 | host2 | 111 |
| 2024-07-11 20:00:15 | host2 | 115 |
| 2024-07-11 20:00:20 | host2 | 95 |
+---------------------+-------+-------------+
8 rows in set (0.06 sec)
```
Range queries are powerful for time-window aggregation. Read the [manual](/reference/sql/range.md) to learn more.
### Correlate metrics, logs, and traces
This is where a unified database shines. One query to correlate p95 latency, error counts, and slow trace spans — across all three signal types:
```sql
WITH
-- Metrics: per-host p95 latency in 5s windows
metrics AS (
SELECT
ts, host,
approx_percentile_cont(0.95) WITHIN GROUP (ORDER BY latency)
RANGE '5s' AS p95_latency
FROM grpc_latencies
ALIGN '5s' FILL PREV
),
-- Logs: per-host ERROR count in 5s windows
logs AS (
SELECT
ts, host,
count(log_msg) RANGE '5s' AS num_errors
FROM app_logs
WHERE log_level = 'ERROR'
ALIGN '5s'
),
-- Traces: per-host slow span count in 5s windows
slow_traces AS (
SELECT
date_bin(INTERVAL '5' seconds, ts) AS ts,
service_name AS host,
COUNT(*) AS slow_spans,
MAX(duration) AS max_span_duration
FROM traces
WHERE duration > 500
GROUP BY date_bin(INTERVAL '5' seconds, ts), service_name
)
SELECT
m.ts,
m.host,
m.p95_latency,
COALESCE(l.num_errors, 0) AS num_errors,
COALESCE(t.slow_spans, 0) AS slow_spans,
t.max_span_duration
FROM metrics m
LEFT JOIN logs l ON m.host = l.host AND m.ts = l.ts
LEFT JOIN slow_traces t ON m.host = t.host AND m.ts = t.ts
ORDER BY m.ts, m.host;
```
```sql
+---------------------+-------+-------------+------------+------------+-------------------+
| ts | host | p95_latency | num_errors | slow_spans | max_span_duration |
+---------------------+-------+-------------+------------+------------+-------------------+
| 2024-07-11 20:00:05 | host1 | 104.5 | 0 | 0 | NULL |
| 2024-07-11 20:00:05 | host2 | 114 | 0 | 0 | NULL |
| 2024-07-11 20:00:10 | host1 | 4200 | 10 | 4 | 4250 |
| 2024-07-11 20:00:10 | host2 | 111 | 0 | 0 | NULL |
| 2024-07-11 20:00:15 | host1 | 3500 | 4 | 2 | 3100 |
| 2024-07-11 20:00:15 | host2 | 115 | 0 | 0 | NULL |
| 2024-07-11 20:00:20 | host1 | 2500 | 0 | 0 | NULL |
| 2024-07-11 20:00:20 | host2 | 95 | 0 | 0 | NULL |
+---------------------+-------+-------------+------------+------------+-------------------+
8 rows in set (0.02 sec)
```
The picture is clear: **during the `20:00:10` – `20:00:15` window, `host1` experienced p95 latency spikes (up to 4200ms), 10 error logs, and 4 slow trace spans (max 4250ms). `host2` was unaffected**. In a traditional three-pillar stack, this correlation requires switching between Prometheus, Loki, and Jaeger. With GreptimeDB, it's one query.
### Query with PromQL
GreptimeDB natively supports [PromQL](/user-guide/query-data/promql.md). In the Dashboard, switch to the PromQL tab and run:
```promql
quantile_over_time(0.95, grpc_latencies{host!=""}[5s])
```
You can also query via the Prometheus-compatible HTTP API:
```bash
curl -X POST \
-H 'Authorization: Basic ' \
--data-urlencode 'query=quantile_over_time(0.95, grpc_latencies{host!=""}[5s])' \
--data-urlencode 'start=2024-07-11 20:00:00Z' \
--data-urlencode 'end=2024-07-11 20:00:20Z' \
--data-urlencode 'step=15s' \
'http://localhost:4000/v1/prometheus/api/v1/query_range'
```
Output
```json
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "grpc_latencies",
"host": "host1",
"method_name": "GetUser"
},
"values": [
[
1720728015.0,
"3560"
]
]
},
{
"metric": {
"__name__": "grpc_latencies",
"host": "host2",
"method_name": "GetUser"
},
"values": [
[
1720728015.0,
"114.2"
]
]
}
]
}
}
```
### Mix SQL and PromQL
Use [TQL](/reference/sql/tql.md) to embed PromQL inside SQL — combining the power of both:
```sql
TQL EVAL ('2024-07-11 20:00:00Z', '2024-07-11 20:00:20Z', '15s')
quantile_over_time(0.95, grpc_latencies{host!=""}[5s]);
```
```sql
+---------------------+---------------------------------------------------------+-------+-------------+
| ts | prom_quantile_over_time(ts_range,latency,Float64(0.95)) | host | method_name |
+---------------------+---------------------------------------------------------+-------+-------------+
| 2024-07-11 20:00:15 | 3560 | host1 | GetUser |
| 2024-07-11 20:00:15 | 114.2 | host2 | GetUser |
+---------------------+---------------------------------------------------------+-------+-------------+
```
You can even use PromQL as a CTE in a correlation query:
```sql
WITH
metrics AS (
TQL EVAL ('2024-07-11 20:00:00Z', '2024-07-11 20:00:20Z', '5s')
quantile_over_time(0.95, grpc_latencies{host!=""}[5s])
),
logs AS (
SELECT
ts, host,
COUNT(log_msg) RANGE '5s' AS num_errors
FROM app_logs
WHERE log_level = 'ERROR'
ALIGN '5s'
)
SELECT
m.*,
COALESCE(l.num_errors, 0) AS num_errors
FROM metrics AS m
LEFT JOIN logs AS l ON m.host = l.host AND m.ts = l.ts
ORDER BY m.ts, m.host;
```
```sql
+---------------------+---------------------------------------------------------+-------+-------------+------------+
| ts | prom_quantile_over_time(ts_range,latency,Float64(0.95)) | host | method_name | num_errors |
+---------------------+---------------------------------------------------------+-------+-------------+------------+
| 2024-07-11 20:00:10 | 140.89999999999998 | host1 | GetUser | 10 |
| 2024-07-11 20:00:10 | 113.8 | host2 | GetUser | 0 |
| 2024-07-11 20:00:15 | 3560 | host1 | GetUser | 4 |
| 2024-07-11 20:00:15 | 114.2 | host2 | GetUser | 0 |
| 2024-07-11 20:00:20 | 3400 | host1 | GetUser | 0 |
| 2024-07-11 20:00:20 | 115 | host2 | GetUser | 0 |
+---------------------+---------------------------------------------------------+-------+-------------+------------+
```
## GreptimeDB Dashboard
GreptimeDB includes a built-in [Dashboard](./installation/greptimedb-dashboard.md) at `http://localhost:4000/dashboard`. Use **Table Query** as the general query entry (SQL and PromQL), or switch to **Metrics Query**, **Logs Query**, and **Traces Query** for signal-specific views.
```sql
SELECT * FROM grpc_latencies;
```
## Next steps
**Connect your existing stack:**
- [Prometheus Remote Write](/user-guide/ingest-data/for-observability/prometheus.md) — point your Prometheus at GreptimeDB
- [OpenTelemetry](/user-guide/ingest-data/for-observability/opentelemetry.md) — configure OTel Collector to send metrics, logs, and traces
- [Jaeger](/user-guide/query-data/jaeger.md) — use GreptimeDB as Jaeger's storage backend
- [Loki](/user-guide/ingest-data/for-observability/loki.md) — send logs using Loki protocol
- [Elasticsearch](/user-guide/ingest-data/for-observability/elasticsearch/) — send logs, traces and events using Elasticsearch `_bulk` API
- Find [all ingestion methods](/user-guide/ingest-data/overview/#recommended-data-ingestion-methods).
**Visualize and monitor:**
- [Grafana integration](/user-guide/integrations/grafana.md) — connect Grafana with SQL or PromQL datasource
- [Official Dashboard](/getting-started/installation/greptimedb-dashboard.md) — the embedded dashboard at `http://localhost:4000/dashboard`
**Go deeper:**
- [Why GreptimeDB](/user-guide/concepts/why-greptimedb.md) — architecture, cost comparison, and how GreptimeDB compares
- [Observability 2.0 and wide events](/user-guide/concepts/observability-2.md) — where wide events fit, and what they cost
- [Demo scene](https://github.com/GreptimeTeam/demo-scene/) — more hands-on examples
- [User Guide](/user-guide/overview.md) — complete reference
---
## Architecture
GreptimeDB can run as one standalone process or as a distributed cluster. Standalone deployments use the configured local or object-storage backend. Distributed deployments can use shared object storage to separate persistent data files from compute, allowing compute and storage capacity to be adjusted independently.
Object storage is not the only state in the system. WAL records accepted writes, Metasrv manages cluster metadata and Region routes, and local disks can cache remote data. Availability and failover depend on the selected WAL mode, Metasrv deployment, Region placement, shared storage, and available Datanodes.
For why real-time monitoring and historical analysis can share this storage and query foundation, see [Why GreptimeDB](./why-greptimedb.md#one-system-for-real-time-monitoring-and-historical-analysis).
## High-level Architecture
## Components
Distributed mode has three core components and an optional Flow runtime:
- [**Metasrv**](/contributor-guide/metasrv/overview.md): Manages catalogs, schemas, tables, Region routes, procedures, and scheduling metadata.
- [**Frontend**](/contributor-guide/frontend/overview.md): Accepts client protocols, authenticates requests, plans distributed queries, and routes reads and writes using Metasrv metadata.
- [**Datanode**](/contributor-guide/datanode/overview.md): Hosts Regions, executes reads and writes, records WAL entries, performs compaction, and persists data files to the configured local or object-storage provider.
- [**Flownode (optional)**](/contributor-guide/flownode/overview.md): Runs [Flow](/user-guide/flow-computation/overview.md) tasks that continuously compute and materialize derived data in sink tables.
In standalone mode, one GreptimeDB process provides these database functions without separate service deployment.
## How It Works
The following paths describe distributed mode. Standalone mode runs the same database functions in one process.
### Write Path
1. A client sends data through a supported ingestion protocol.
2. Frontend resolves table and Region routes from Metasrv metadata.
3. Frontend splits the request and forwards rows to the Datanodes that host the target Regions.
4. A Datanode writes to memory and the configured [WAL](/user-guide/deployments-administration/wal/overview.md), then eventually flushes immutable data files to the table's configured [storage provider](./storage-location.md).
### Query Path
1. A client submits SQL, PromQL, or an explicitly supported query API such as the Jaeger-compatible API.
2. Frontend plans the query and dispatches work to Datanodes that host the relevant Regions.
3. Datanodes read in-memory data and persistent files, use local caches when applicable, apply pruning and indexes, and return partial results.
4. Frontend combines the results and returns the response.
### Flow Path (Optional)
When Flow is enabled, Frontend mediates its data path. In streaming mode, Frontend mirrors writes to Flownode. In batching mode, Flownode queries source tables and writes materialized results to sink tables through Frontend. Source and sink tables keep their own schema, TTL, indexes, and storage settings. See [Flow Computation](/user-guide/flow-computation/overview.md).
For storage responsibilities, see [Storage Location](./storage-location.md). For implementation details, see the [Contributor Guide](/contributor-guide/overview.md).
---
## Data Model
## Model
GreptimeDB uses a unified observability data model based on relational tables with a time index and `Tag`, `Timestamp`, and `Field` column semantics. Metrics, logs, traces, and events share this model while remaining in separate tables with workload-specific schemas.
Every GreptimeDB table has a name and exactly one time index. Columns have the following roles:
- `Tag` columns participate in the primary key and group related rows. For metrics, they usually represent labels that identify a time series. Tags are optional: append-only log and event tables can have no primary-key columns.
- The `Timestamp` column is declared as the table's time index. It records event or sample time, helps GreptimeDB organize data by time, and enables efficient time-range queries.
- `Field` columns store measurements, log content, trace attributes, or other values. They can use numeric, string, JSON, timestamp, and other supported data types.
For tables with primary-key columns, persisted rows are ordered by `(primary key, timestamp)`. Tables can merge rows with the same primary key and timestamp according to their [merge mode](/reference/sql/create.md#create-a-table-with-merge-mode). Append-only tables disable deduplication and, when they have no primary key, order persisted rows by timestamp. GreptimeDB stores table data in immutable Parquet SST files; see [Data Layout in SST Files](/contributor-guide/datanode/storage-engine.md#data-layout-in-sst-files) for details.
Schema design affects write amplification, compression, index size, and query pruning. See the [Schema Design Guide](/user-guide/deployments-administration/performance-tuning/design-table.md) before choosing primary-key columns and indexes.
### Metrics
The following table stores host resource metrics:
```sql
CREATE TABLE IF NOT EXISTS system_metrics (
host STRING,
idc STRING,
cpu_util DOUBLE,
memory_util DOUBLE,
disk_util DOUBLE,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY(host, idc),
TIME INDEX(ts)
);
```
- `host` and `idc` are Tag columns declared by `PRIMARY KEY`.
- `ts` is the Timestamp column declared by `TIME INDEX`.
- `cpu_util`, `memory_util`, and `disk_util` are Field columns.
- With the default [`last_row` merge mode](/reference/sql/create.md#create-a-table-with-merge-mode), queries keep the latest row for each `host`, `idc`, and `ts` combination.
See [Prometheus Data Model](/user-guide/ingest-data/for-observability/prometheus.md#data-model) for the mapping between Prometheus metrics and GreptimeDB tables.
### Logs
An append-only table is often a better fit for access logs:
```sql
CREATE TABLE access_logs (
access_time TIMESTAMP TIME INDEX,
remote_addr STRING,
http_status STRING,
http_method STRING,
http_refer STRING,
user_agent STRING,
request STRING
) WITH ('append_mode' = 'true');
```
- `access_time` is the Timestamp column.
- The table has no Tag columns or primary key.
- The remaining columns are Fields.
- [`append_mode`](/reference/sql/create.md#create-an-append-only-table) disables deduplication and deletion. It fits immutable log records, but not workloads that need row updates or deletes.
- Without a primary key, persisted rows are ordered by `access_time`.
See [Create a Table](/user-guide/deployments-administration/manage-data/basic-table-operations.md#create-a-table) and the [CREATE TABLE reference](/reference/sql/create.md) for column-role syntax and table options.
### Traces
GreptimeDB accepts OpenTelemetry traces through OTLP/HTTP and maps spans to tables with a time index, trace identifiers, span identifiers, attributes, and duration fields. See the [OTLP Trace Data Model](/user-guide/ingest-data/for-observability/opentelemetry.md#data-model-2).
Trace ingestion, storage, and SQL queries are first-class capabilities. GreptimeDB also provides a Jaeger-compatible query API.
## Design Considerations
The table model provides several practical properties:
- schemas expose types and column roles to the storage and query engines;
- SQL can filter, aggregate, and join data across tables;
- a row can contain multiple Field columns, avoiding the extra rows required by single-value models;
- tables can choose primary keys, merge behavior, append-only mode, indexes, TTL, and storage providers independently;
- automatic schema generation can create tables and columns for supported ingestion protocols;
- the same Tag, Timestamp, and Field concepts apply to metrics, logs, traces, and wide events without requiring them to share a table.
GreptimeDB manages table schemas with SQL. See [Table Management](/user-guide/deployments-administration/manage-data/basic-table-operations.md) and [Automatic Schema Generation](/user-guide/ingest-data/overview.md#automatic-schema-generation).
Tables can also carry an optional [table semantic layer](./semantic-layer.md) describing signal identity and ingestion metadata for machine consumers. Read [Observability 2.0 and wide events](./observability-2.md) for one optional way to retain more context in this model.
---
## Common Questions
## How does GreptimeDB handle metrics, logs, and traces?
GreptimeDB uses the same columnar engine and the same Tag, Timestamp, and Field column semantics for metrics, logs, and traces. The signals can use separate physical tables with different schemas, indexes, TTL settings, and ingestion paths.
All three signal types can be queried with SQL. Metrics also support PromQL, while traces have a separate Jaeger-compatible query API. Flow can continuously derive and materialize aggregates from source tables. See the [Data Model](./data-model.md), [Logs](/user-guide/logs/overview.md), and [Traces](/user-guide/traces/overview.md).
## Does GreptimeDB support updates?
Partially. For a table that uses deduplication, writing another row with the same primary key and time index updates Field values according to the table's `merge_mode`. The primary key and time index identify the row and are not updated in place.
For tables created with SQL, the default `last_row` mode keeps the latest row. The `last_non_null` mode keeps the latest non-null value for each Field. Auto-created tables can use protocol-specific defaults. Deduplication applies within a Region. GreptimeDB accepts any table column as a partition column, but for a deduplicating table, partition columns should be chosen from the primary key so rows with the same primary key remain in one Region and can be merged correctly. See [Table Sharding](/user-guide/deployments-administration/manage-data/table-sharding.md#partition).
Append-only tables disable deduplication, so repeated keys and timestamps create additional rows rather than updates. See [Update Data](/user-guide/manage-data/overview.md#update-data) and the [`merge_mode` reference](/reference/sql/create.md#create-a-table-with-merge-mode).
## Does GreptimeDB support deletion?
Yes, for tables that allow deletion. You can delete matching rows with SQL, truncate a table, or expire data with TTL. Append-only tables do not support row deletion. See [Delete Data](/user-guide/manage-data/overview.md#delete-data).
## Can I set TTL or retention policy for different tables or measurements?
Yes. TTL can be set at database or table level, and a table-level setting takes precedence. See [Manage Data Retention with TTL Policies](/user-guide/manage-data/overview.md#manage-data-retention-with-ttl-policies).
## What are the compression rates of GreptimeDB?
There is no single compression ratio. Results depend on schema width, value distribution, primary-key cardinality, repetition, SST format, indexes, and workload. Indexes can improve filtering while increasing storage and write cost. Measure with representative data and retention settings rather than applying a general ratio.
One published edge benchmark provides a concrete but workload-specific result: after writing 10 million TSBS rows on a Qualcomm SA8155P, GreptimeDB Edge used 87 MB and SQLite used 1,686 MB. This result applies to the tested Edge edition, schema, and configuration; it is not a general compression ratio for GreptimeDB deployments. See the [GreptimeDB Edge and SQLite report](https://greptime.com/blogs/2024-08-30-sqlite).
For schema and index trade-offs, see the [Schema Design Guide](/user-guide/deployments-administration/performance-tuning/design-table.md) and [Data Indexes](/user-guide/manage-data/data-index.md).
## How does GreptimeDB address the high cardinality issue?
High cardinality still has a cost: more distinct primary-key values can increase metadata, index, memory, and query work. GreptimeDB provides several controls:
- The `flat` SST format reduces per-series overhead and is the default for new tables. It is recommended for high-cardinality primary keys. The [Flat Format engineering article](https://greptime.com/blogs/2025-12-22-flat-format) explains its memtable and merge-path design and includes benchmark conditions.
- Inverted, full-text, and skipping indexes can be added where their selectivity and storage cost fit the query workload. Indexing every column is usually unnecessary.
- Append-only tables skip deduplication work when records are immutable and do not need updates or deletes.
- Table partitioning distributes Regions across Datanodes in a cluster, but partition design and load distribution still matter.
- Schema design should keep identifiers in the primary key only when they are used for grouping, deduplication, or filtering.
There is no universal cardinality limit that applies to every schema and deployment. Test expected series counts, write rates, query predicates, and retention together. See [`sst_format`](/reference/sql/create.md#create-a-table-with-sst-format), [Data Indexes](/user-guide/manage-data/data-index.md), and the [Schema Design Guide](/user-guide/deployments-administration/performance-tuning/design-table.md).
## Does GreptimeDB support continuous aggregate or downsampling?
Yes. [Flow](/user-guide/flow-computation/overview.md) continuously computes over source-table data as new rows arrive and materializes the results in a sink table. It can implement fixed-window aggregation and downsampling while retaining the source data under its own TTL.
## Can I store data in object storage in the cloud?
Yes. Persistent data files can be stored in Amazon S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and supported S3-compatible services. Object storage is separate from WAL, metadata, and local cache; each has its own durability and recovery role. See [Storage Location](./storage-location.md) and [Storage Options](/user-guide/deployments-administration/configuration.md#storage-options).
## How is GreptimeDB's performance compared to other solutions?
Performance depends on workload, schema, indexes, retention, hardware, object storage, cache, concurrency, and query shape. Use benchmark results only with their test conditions and compare them with your own expected workload.
Published reports include:
- [GreptimeDB vs. InfluxDB](https://greptime.com/blogs/2024-08-07-performance-benchmark)
- [GreptimeDB vs. TimescaleDB](https://greptime.com/blogs/2025-12-09-greptimedb-vs-timescaledb-benchmark)
- [GreptimeDB vs. Grafana Mimir](https://greptime.com/blogs/2024-08-02-datanode-benchmark)
- [JSONBench: one billion JSON documents](https://greptime.com/blogs/2025-03-18-jsonbench-greptimedb-performance) — records the third-party benchmark result and reproduction steps
- [Log workload: GreptimeDB, ClickHouse, and Elasticsearch](https://greptime.com/blogs/2025-03-10-log-benchmark-greptimedb)
- [GreptimeDB vs. Loki](https://greptime.com/blogs/2025-08-07-beyond-loki-greptimedb-log-scenario-performance-report)
- [GreptimeDB Edge vs. SQLite on Qualcomm SA8155P](https://greptime.com/blogs/2024-08-30-sqlite)
## Does GreptimeDB have disaster recovery solutions?
GreptimeDB provides the components needed to build a recovery plan, but object storage alone is not a complete solution. Persistent data files, WAL, Metasrv metadata, Region placement and failover, and deployment configuration each affect recovery point and recovery time objectives.
Cluster failover also depends on a correctly configured topology and healthy replacement Datanodes. Back up and test metadata and WAL recovery alongside object-store policies. See [Disaster Recovery](/user-guide/deployments-administration/disaster-recovery/overview.md), [WAL Overview](/user-guide/deployments-administration/wal/overview.md), and [Storage Location](./storage-location.md).
## Does GreptimeDB support geospatial computation?
Yes. GreptimeDB provides SQL functions for WKT values, Geohash, H3, S2, spatial relationships, distance, and area. These functions do not create or imply a separate database spatial index. See [Geospatial Functions](/reference/sql/functions/geo.md).
## Any JSON support?
GreptimeDB provides two JSON column types:
- The experimental [`JSON`](/reference/sql/data-types.md#json-type-experimental) type stores general JSON values and works with [JSON functions](/reference/sql/functions/json.md).
- [`JSON2`](/user-guide/logs/json2.md), available since v1.2 and currently Beta, stores subpaths in a structured columnar form for logs and other semi-structured data. It supports dot-path access, `json_get`, and optional type hints. JSON2 currently requires an append-only table.
Neither type currently provides index options for JSON subpaths. Put stable, frequently filtered attributes in typed columns when they need inverted, full-text, or skipping indexes. See [Data Indexes](/user-guide/manage-data/data-index.md).
## More Questions?
For deployment, migration, operations, and schema questions, see the full [FAQ](/faq-and-others/faq.md).
---
## Key Concepts
## Database
A database is a namespace for tables and other objects. It groups data for management and access, but is not by itself a complete tenant-isolation boundary.
## Time-Series Table
A GreptimeDB table follows a relational schema and has exactly one time index. Columns use Tag, Timestamp, and Field semantics. Time-series tables are a natural fit for metrics and IoT workloads, while logs, traces, and event data use the same table model with schemas and options suited to those signals.
Tables can be created with SQL or automatically from supported ingestion protocols. In distributed deployments, a table can be partitioned into Regions placed on different Datanodes. See [Data Model](./data-model.md).
## Time Index and Primary Key
Every GreptimeDB table has exactly one time index. It records sample or event time and lets the storage engine organize and prune data by time.
Primary-key columns use Tag semantics and identify a series or record group. In tables that use deduplication, the primary key and time index identify rows that are merged according to the table's [`merge_mode`](/reference/sql/create.md#create-a-table-with-merge-mode). This is not a general relational uniqueness constraint: append-only tables can omit the primary key, and append-only mode keeps repeated keys and timestamps as separate rows.
## Table Engine
A table engine controls how table data is written, organized, compacted, and read. The main engines are:
- **Mito Engine**: The general-purpose engine for time-indexed tables, including logs, traces, and event data.
- **Metric Engine**: Built on Mito Engine and optimized for large numbers of Prometheus-style metric tables. User-facing metric tables remain logical tables and can share physical tables and metadata internally.
Table engines are separate from storage providers such as local files, Amazon S3, or Google Cloud Storage. See [Table Engines](/reference/about-greptimedb-engines.md) and [Storage Location](./storage-location.md).
## Region
A Region is a physical partition of a table and the basic unit of storage, scheduling, and migration. A Region is hosted by a Datanode, while Metasrv maintains its route and placement metadata. Clients normally access tables through Frontend without addressing Regions directly.
## Data Types
GreptimeDB columns are strongly typed. Automatic schema generation can create tables and add compatible columns, but values written to an existing column must match or be cast to its data type. See [Data Types](/reference/sql/data-types.md).
## Index
An index is an optional data structure used to accelerate selected query patterns. GreptimeDB provides inverted, full-text, and skipping indexes with different selectivity, storage, and write-cost trade-offs. See [Data Indexes](/user-guide/manage-data/data-index.md).
## View
A view is a named SQL query presented as a virtual table. It stores the query definition, not a materialized copy of its result; querying the view reads the underlying tables.
## Pipeline
Pipeline processes data during ingestion. It can parse, transform, enrich, and route incoming records before storing them in tables. See [Pipeline](/user-guide/logs/use-custom-pipelines.md).
## Flow
A Flow is a continuous computation over incoming rows after they reach source tables. It updates and materializes its result in a sink table, which can be queried and managed like other tables. See [Flow Computation](/user-guide/flow-computation/overview.md).
---
## Observability 2.0
Observability 2.0 is an industry term for an approach to telemetry, not a product category. It usually refers to retaining wide events and analyzing them without deciding every question in advance.
GreptimeDB's [data model](./data-model.md) supports native metrics, logs, and traces alongside wide events. This page focuses on the wide-event approach associated with Observability 2.0. GreptimeDB supports this practice but does not require it; metrics, logs, and traces remain first-class capabilities.
## The Limits of Three Pillars
Metrics, logs, and traces remain useful abstractions. The problem is not the signals themselves, but the operational boundary that often forms around each one:
1. **Separate context**: When signals are stored and queried in separate systems, correlating an alert with the relevant logs and traces takes extra work.
2. **Questions fixed at collection time**: Pre-aggregated metrics answer known questions efficiently, but cannot recover dimensions that were not recorded.
3. **Lost structure**: Plain-text logs often contain useful fields that are expensive to parse and index later.
Shared schema concepts, storage infrastructure, and query tools reduce these boundaries. Wide events are one way to retain more context, not a replacement for every metric, log, or trace.
## Wide Events
A wide event is a structured record with many fields describing one operation or business event. It can include high-cardinality values such as user IDs, session IDs, trace IDs, and request attributes.
### What is a Wide Event?
For example, an event for a POST request might include user and subscription data, database and cache operations, HTTP attributes, outcome, and duration:
```json
{
"timestamp": "2026-08-12T08:15:30Z",
"method": "POST",
"path": "/articles",
"service": "articles",
"outcome": "ok",
"status_code": 201,
"duration": 268,
"user": {
"id": "fdc4ddd4-8b30-4ee9-83aa-abd2e59e9603",
"subscription": { "plan": "free", "trial": true }
},
"db": {
"query": "INSERT INTO articles (...)"
},
"cache": { "operation": "write" },
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736"
}
```
Capture only the context that is useful and safe to retain. Credentials, personal data, query parameters, prompts, and request bodies may require filtering or redaction before ingestion.
### Views Derived from Wide Events
In this approach, a wide event can produce several views:
- a metric aggregated by status and time window;
- a searchable log record containing the event details;
- a trace or span view linked by trace and span IDs.
This is a useful mental model, not a requirement that every signal must be reconstructed from raw events. Native metrics are often the better representation for fixed aggregations, and standard trace data remains useful for call graphs and latency analysis.
## AI and the Need for Fine-Grained Observability
AI applications often need to relate model requests, responses, tool calls, latency, token use, evaluations, and application state. A structured event can keep this context queryable when the instrumentation captures it.
The same trade-offs apply as in other workloads: prompts and responses can be large or sensitive, session identifiers create high cardinality, and incomplete instrumentation produces incomplete context. Teams should choose fields, retention periods, and redaction rules based on the questions they need to answer.
The [table semantic layer](./semantic-layer.md) can describe what each table represents so that agents and tools do not have to infer signal type, source, or metric type from column names.
## How GreptimeDB Supports Wide Events
GreptimeDB stores wide events in regular time-indexed tables. Pipeline, SQL, and Flow support different stages of the approach:
| Stage | GreptimeDB capability | Role |
| --- | --- | --- |
| Ingest, when needed | [Pipeline](/user-guide/logs/use-custom-pipelines.md) | Parse, transform, and enrich incoming logs before storage. |
| Store and analyze | Time-indexed tables and SQL | Retain wide events for detailed and retrospective analysis. |
| Derive | [Flow](/user-guide/flow-computation/overview.md) | Continuously aggregate incoming rows into separate metrics tables. |
Wide events do not require native metrics, logs, traces, and raw events to share one table. Each can use separate tables with different schemas, retention policies, and indexes.
Pipeline and Flow handle different stages. Pipeline parses, transforms, and enriches logs during ingestion. Its output is structured, multi-column data; when those fields retain the context of an operation or business event, each row can serve as a wide event. Flow can continuously aggregate the same event rows as they arrive into metrics tables for dashboards and alerts.
For example, Flow can derive a status metric from an event table:
```sql
CREATE FLOW http_status_count
SINK TO status_metrics
AS
SELECT
status_code,
COUNT(*) AS count,
date_bin('1 minute'::INTERVAL, timestamp) AS time_window
FROM access_logs
GROUP BY status_code, time_window;
```
The stored events remain available for detailed SQL queries, while the sink table serves fixed dashboards and alerts efficiently.
## Trade-offs
The wide-event approach changes where you pay for flexibility:
- **Wider events increase data volume.** More fields and repeated context consume ingestion bandwidth and storage, even with columnar compression.
- **High cardinality and long retention increase cost.** Keep only useful dimensions, and set retention independently for raw and derived data.
- **Complete context depends on instrumentation quality.** Missing identifiers, inconsistent schemas, or poor propagation cannot be repaired by the database.
- **Native metrics still fit fixed aggregations.** Counters, gauges, histograms, recording rules, dashboards, and alerts usually do not need a raw event behind every sample.
Schema governance, sampling, redaction, and retention are part of the design. A wide event should be as wide as the investigation requires, not as wide as the application can produce.
## Adoption Paths
### Keep Native Signals and Unify Storage and Query
Continue using existing ingestion protocols. Query metrics with PromQL, query all signal types with SQL, and use the Jaeger-compatible API for traces. Store each signal in separate GreptimeDB tables, and use shared identifiers and SQL when cross-signal analysis is needed. This path minimizes instrumentation and dashboard changes.
Start with [Prometheus](/user-guide/ingest-data/for-observability/prometheus.md), [logs](/user-guide/logs/overview.md), [OpenTelemetry](/user-guide/ingest-data/for-observability/opentelemetry.md), or [traces](/user-guide/traces/overview.md).
### Add Raw Events Where Full Context Matters
Instrument selected business operations or AI workflows as structured events. If the context starts in logs, use [Pipeline](/user-guide/logs/use-custom-pipelines.md) to extract and transform it into structured columns during ingestion. Keep the event table for retrospective analysis, and use [Flow](/user-guide/flow-computation/overview.md) to derive metrics for known dashboards and alerts. This path provides more context at the cost of higher data volume and stricter schema and retention management.
The two paths can coexist. Adopt raw events only where the additional context justifies their cost.
## Further Reading
- [Observability 2.0 and the Database for It](https://greptime.com/blogs/2025-04-25-greptimedb-observability2-new-database) — An earlier description of the wide-event approach
- [Unified Storage for Observability - GreptimeDB's Approach](https://greptime.com/blogs/2024-12-24-observability) — GreptimeDB's unified storage model
- [Agent Observability: Can the Old Playbook Handle the New Game?](https://greptime.com/blogs/2025-12-11-agent-observability) — Context needed for AI application debugging
- [Scaling Observability at Poizon](https://greptime.com/blogs/2025-05-06-poizon-observability-greptimedb-monitoring-use-case) — Raw events and continuous aggregation in production
---
## GreptimeDB Concepts Overview
# Concepts
GreptimeDB is an open-source observability database. It uses one columnar engine to process metrics, logs, and traces, and supports scalable deployments backed by object storage. The signals share Tag, Timestamp, and Field column semantics, but can remain in separate tables with different schemas and retention policies.
**Start here:**
- [Why GreptimeDB](./why-greptimedb.md) — Product scope, unified processing, scaling, protocol boundaries, and deployment options
- [Data Model](./data-model.md) — Shared Tag, Timestamp, and Field semantics for metrics, logs, traces, and event data
- [Architecture](./architecture.md) — Standalone and distributed deployments, compute-storage separation, and component responsibilities
**Deep dives:**
- [Observability 2.0 and wide events](./observability-2.md) — The role and trade-offs of wide events alongside native signals
- [Table Semantic Layer](./semantic-layer.md) — Optional metadata that tells machine consumers what observability concept each table represents
- [Storage Location](./storage-location.md) — Local storage, object storage, and per-table storage providers
- [Key Concepts](./key-concepts.md) — Tables, Regions, time index, data types, views, and Flow
- [Common Questions](./features-that-you-concern.md) — FAQ on updates, deletion, TTL, compression, high cardinality, and other technical boundaries
## Further Reading
- [Observability 2.0 and the Database for It](https://greptime.com/blogs/2025-04-25-greptimedb-observability2-new-database) — An earlier description of the wide-event approach
- [Unifying Logs and Metrics](https://greptime.com/blogs/2024-06-25-logs-and-metrics)
- [GreptimeDB Storage Engine Design](https://greptime.com/blogs/2022-12-21-storage-engine-design)
---
## Table Semantic Layer (Experimental)
:::warning
The semantic layer is experimental and may change in future releases. Tables without semantic metadata keep working unchanged; the layer is optional and additive.
:::
The semantic layer stores metadata on a table so machine consumers such as LLM agents, alert and dashboard builders, [MCP servers](/user-guide/integrations/mcp.md), and ETL pipelines can identify what the table represents without inferring it from column names.
## Why it exists
GreptimeDB ingests OTLP metrics, traces, and logs, plus Prometheus remote write, InfluxDB Line Protocol, OpenTSDB, Loki Push API, and Elasticsearch Bulk API data. Some protocols, especially OpenTelemetry, carry semantic metadata that is not represented by ordinary table columns. The amount and quality of metadata varies by protocol:
- An OTLP traces table looks like any other wide table; signal type and source must be guessed from naming.
- An OTLP metric's unit (`s`, `By`) is discarded by the row encoders and is unrecoverable from the data.
- OTLP aggregation temporality (`cumulative` vs `delta`) is invisible in the metric name.
- A Prometheus `counter` inferred from a `_total` suffix is not a protocol declaration. Without semantic metadata, the table does not record that distinction.
This metadata is available during ingestion but is not represented in ordinary data rows. Keeping it allows an alert generator to distinguish a rate from an absolute value, a dashboard builder to select a visualization by signal type, and an agent to inspect a structured catalog instead of inferring meaning from column names.
## How it works
The layer uses existing SQL interfaces. It does not add a protocol or DDL keyword. It has three parts:
1. **`greptime.semantic.*` table options** store table identity and ingestion metadata alongside options such as `ttl` and `table_data_model`.
2. **Column `COMMENT`** stores additional column-level information through standard SQL.
3. **[`information_schema.table_semantics`](/reference/sql/information-schema/table-semantics.md)** is the discovery view. It returns one row for each table with at least one `greptime.semantic.*` option.
## Vocabulary
All keys are flat strings under the `greptime.semantic.` prefix, and all values are strings. The vocabulary contains only metadata that a consumer cannot readily recover from the schema, columns, or metric naming conventions. It omits constants, values already encoded in a metric name such as the Prometheus `_total` suffix, and information that only repeats a column.
The whitelist is closed: an unrecognized key under the prefix (such as `greptime.semantic.future.key`) or an out-of-domain value is rejected.
### Common keys (all signals)
| Key | Description | Example values |
| --- | --- | --- |
| `greptime.semantic.signal_type` | The telemetry signal the table represents. | `metric` / `trace` / `log` / `event` / `unknown` |
| `greptime.semantic.source` | The ingestion ecosystem that wrote the data. | `opentelemetry` / `prometheus` / `influxdb` / `opentsdb` / `loki` / `elasticsearch` / `custom` / `mixed` / `unknown` |
| `greptime.semantic.pipeline` | The internal ingestion data model. The signal-agnostic successor to `table_data_model`. | `greptime_trace_v1` |
### Trace keys
| Key | Description | Example values |
| --- | --- | --- |
| `greptime.semantic.trace.conventions` | The semantic-conventions version the rows conform to, typically an OTel schema URL. | `https://opentelemetry.io/schemas/1.27.0` / `mixed` / `unknown` |
### Metric keys
| Key | Description | Example values |
| --- | --- | --- |
| `greptime.semantic.metric.type` | The instrument kind. | `counter` / `gauge` / `histogram` / `summary` / `updown_counter` / `gauge_histogram` / `info` / `stateset` / `mixed` / `unknown` |
| `greptime.semantic.metric.unit` | The unit in [UCUM](https://ucum.org/) notation. Discarded by the row encoders, so unrecoverable once ingested. | `s` / `By` / `{request}` |
| `greptime.semantic.metric.temporality` | Aggregation temporality (OTLP only). Invisible in the metric name. | `cumulative` / `delta` / `mixed` / `unknown` |
| `greptime.semantic.metric.metadata_quality` | How the metric type was obtained and how reliable `metric.type` is. | `declared` (the protocol stated it) / `inferred` (derived from a name suffix) / `unknown` |
| `greptime.semantic.metric.original_name` | The original OpenTelemetry name, recorded when the table name is converted to Prometheus naming. | `http.server.duration` |
Consumers should verify an `inferred` counter before applying `rate()` semantics because the type was derived from its name rather than declared by the protocol.
`unknown` and `mixed` are shared sentinel values. `unknown` means the value could not be determined when the option was set. `mixed` means a single-valued key received conflicting values over the table's lifetime, for example when a table received rows from more than one source. Treat a single-valued semantic key as descriptive metadata, not an enforced constraint.
## Automatic stamping on ingestion
Supported auto-create paths stamp identity (`signal_type` + `source`). OTLP metrics also carry metric type, unit, and temporality because the OTLP wire format declares them; OTLP traces carry pipeline and convention metadata. Other protocols generally provide only identity metadata, as shown below.
| Ingestion path | `signal_type` | `source` | Additional keys |
| --- | --- | --- | --- |
| OTLP metrics | `metric` | `opentelemetry` | `metric.type`, `metric.unit`, `metric.temporality`, `metric.metadata_quality` = `declared`, `metric.original_name` |
| OTLP traces | `trace` | `opentelemetry` | `pipeline` = `greptime_trace_v1`, `trace.conventions` |
| OTLP logs | `log` | `opentelemetry` | — |
| Prometheus remote write | `metric` | `prometheus` | identity only |
| InfluxDB line protocol | `metric` | `influxdb` | identity only |
| OpenTSDB | `metric` | `opentsdb` | identity only |
| Loki | `log` | `loki` | identity only |
| Elasticsearch Bulk API | `log` | `elasticsearch` | identity only |
Semantic options are set when the table is created and are not updated by later writes. For example, later writes do not change `metadata_quality` from `inferred` to `declared` or revise `trace.conventions`.
## Manual tagging with DDL
You can set the same options yourself in `CREATE TABLE ... WITH (...)`. Only whitelisted keys with a valid value are accepted:
```sql
CREATE TABLE my_metrics (
ts TIMESTAMP TIME INDEX,
val DOUBLE
) WITH (
'greptime.semantic.signal_type' = 'metric',
'greptime.semantic.source' = 'custom',
'greptime.semantic.metric.type' = 'counter',
'greptime.semantic.metric.unit' = 'By'
);
```
The options appear in `SHOW CREATE TABLE` output and in the `table_semantics` view.
## Discovering semantic metadata
A consumer can list all tables with semantic metadata:
```sql
SELECT table_schema, table_name, signal_type, source, pipeline, metadata_quality, semantic_options
FROM information_schema.table_semantics
ORDER BY table_name;
```
`signal_type`, `source`, `pipeline`, and `metadata_quality` are promoted to dedicated columns; the remaining signal-specific keys are folded into the `semantic_options` JSON string (with the `greptime.semantic.` prefix stripped). See the [`TABLE_SEMANTICS`](/reference/sql/information-schema/table-semantics.md) reference for the full schema and more examples.
The [GreptimeDB MCP Server](/user-guide/integrations/mcp.md) reads this view so AI assistants can identify tables from metadata instead of manual descriptions.
---
## Storage Location
GreptimeDB can keep persistent data files on a local file system or in object storage. Local storage is the default fit for standalone deployments. Shared object storage is commonly used by distributed deployments that need compute-storage separation.
Storage location is separate from table engine selection. [Mito Engine and Metric Engine](/reference/about-greptimedb-engines.md) define how table data is organized and processed; local file systems, Amazon S3, Google Cloud Storage, and Azure Blob Storage are storage providers used to persist the resulting data files. Metric Engine is built on Mito Engine and uses its storage capabilities.
## Local and Object Storage
With local storage, persistent data files are placed under the configured `data_home`. This is simple for standalone deployments, but the files are tied to that host and must be included in its backup and recovery plan.
With object storage, persistent data files are placed in a configured bucket or container. Datanodes can use local disks as cache while object storage remains the shared persistent location. Object storage makes it possible to scale compute separately from storage, but query latency and cost still depend on cache behavior, network performance, request volume, and provider pricing.
## Storage Responsibilities
A GreptimeDB deployment contains several kinds of state:
| State | Responsibility | Typical location | Durability and recovery |
| --- | --- | --- | --- |
| Persistent data files | Table data, SST files, and persistent indexes | Local file system or object storage | Protect local files with backups. For object storage, configure retention, versioning, and replication to match recovery requirements. |
| WAL | Records accepted writes before they are represented in persistent data files | Local WAL, remote WAL, or Noop WAL, depending on configuration | Protects accepted but unflushed writes. Recovery depends on the selected WAL mode; Noop WAL does not retain these writes. |
| Metadata | Catalogs, schemas, table definitions, Region routes, and procedure state | Local metadata in standalone mode; Metasrv state in cluster mode | Required to reconstruct database state. Persist, replicate, or back it up according to the deployment mode. |
| Local cache | Cached object-storage data and temporary index data | Datanode or standalone local disk | Disposable and rebuildable from persistent data. It should not be part of the durability boundary. |
| Process logs | Operational logs from GreptimeDB components | Local logging destination or configured log collector | Not required for database recovery. Retain them according to operational and compliance requirements. |
These requirements are deployment-dependent. See the recovery documentation below before assigning an RPO or RTO.
## Object Storage and Disaster Recovery
Putting persistent data files in object storage does not by itself provide a complete disaster-recovery plan. Recovery also depends on the selected WAL mode, metadata backup or replication, Region state, object-store versioning and retention, and deployment configuration.
For recovery planning, see [Disaster Recovery](/user-guide/deployments-administration/disaster-recovery/overview.md) and the [WAL Overview](/user-guide/deployments-administration/wal/overview.md). Disabling WAL with Noop WAL changes the durability boundary and should be evaluated separately.
## Supported Storage Providers
GreptimeDB supports:
- local file storage;
- Amazon S3 and S3-compatible services, including MinIO, DigitalOcean Spaces, Tencent Cloud Object Storage, and Baidu Object Storage;
- Google Cloud Storage;
- Azure Blob Storage;
- Alibaba Cloud OSS.
See [Storage Options](/user-guide/deployments-administration/configuration.md#storage-options) for the complete configuration keys and current provider list.
## Multiple Storage Providers
An administrator can configure multiple storage providers under `[[storage.providers]]`. A table can then select one configured provider through its `storage` table option:
```sql
CREATE TABLE archive_events (
ts TIMESTAMP TIME INDEX,
payload STRING
) WITH (storage = 'archive_s3');
```
This setting chooses where the table engine stores persistent files. It does not select a different table engine. Provider names and credentials must be configured before the table is created. See [Create a Table with Custom Storage](/reference/sql/create.md#create-a-table-with-custom-storage).
---
## Why GreptimeDB
Teams usually evaluate a new observability backend when separate signal stores, storage expansion, or long-term analysis have become too expensive to operate. GreptimeDB addresses these problems with one columnar engine for metrics, logs, and traces, while keeping the data model and deployment choices explicit.
## What GreptimeDB Is
GreptimeDB is an open-source observability database. It stores and queries metrics, logs, and traces with one columnar engine and a shared SQL layer. It can run as a standalone process on local storage or as a distributed cluster backed by object storage.
GreptimeDB is not a general-purpose transactional database. Its storage, indexing, retention, and query paths are designed for append-heavy, time-indexed workloads such as observability and IoT data.
## Why One Engine for Three Signals
When metrics, logs, and traces live in separate databases, each store brings its own capacity model, lifecycle policy, query path, and failure modes. Correlation also depends on moving data between systems or coordinating several queries during an incident.
GreptimeDB uses the same columnar engine and [Tag, Timestamp, and Field semantics](./data-model.md) across the three signal types. This gives teams:
- one storage and lifecycle-management system for metrics, logs, and traces;
- SQL across all signal tables and PromQL for metrics;
- [Flow](/user-guide/flow-computation/overview.md) for continuous aggregation and materialized derived data;
- SQL correlation through common identifiers when instrumentation records them.
Unification happens at the engine, storage, and query layers. It does not require metrics, logs, and traces to use one table, one schema, or the [wide-event model](./observability-2.md).
## One Columnar Engine, Separate Tables
Reducing the number of systems should not force unlike signals into the same schema. Metric samples, log records, spans, and wide events have different access patterns and retention needs.
GreptimeDB lets each workload use tables suited to its data:
- metrics commonly use primary-key columns for labels and PromQL for queries;
- logs often use append-only tables and text or inverted indexes selected for the workload;
- traces retain trace and span identifiers and can be queried with SQL or the Jaeger-compatible API;
- wide events can retain more context where retrospective analysis justifies the added storage cost.
These tables can have different schemas, indexes, TTL settings, compaction options, and storage providers. See [Data Model](./data-model.md) and [Storage Location](./storage-location.md) for the underlying mechanisms.
## One System for Real-Time Monitoring and Historical Analysis
Incident response needs fast queries over recent data. Trend analysis, capacity work, and investigations may scan weeks or months. Splitting these workloads between a monitoring backend and an analytics database adds another ingestion path, another copy of the data, and another system to operate.
In distributed deployments backed by object storage, GreptimeDB handles both workloads with the same engine, storage system, and query layer. Recent data can be served from memory and local caches, while persistent data remains in object storage for longer-range queries. Historical analysis does not require a separate analytics database and data-copy pipeline.
## Object Storage and Independent Scaling
When persistent files are tied to compute nodes, adding capacity can require moving data, rebalancing local disks, or provisioning storage and compute together. That makes a storage-heavy observability cluster harder to change as retention grows.
In distributed GreptimeDB deployments configured with shared object storage, persistent data files are kept in services such as Amazon S3, Google Cloud Storage, or Azure Blob Storage. Datanodes perform writes, compaction, and queries, while local disks can cache remote data. Compute and object-storage capacity can be adjusted separately without copying every persistent data file between Datanodes. See [Architecture](./architecture.md) and [Storage Location](./storage-location.md).
## Protocols and Query Boundaries
Replacing collection agents, dashboards, and client code can cost more than deploying a new database. GreptimeDB therefore accepts data through several established protocols:
- Prometheus Remote Write for metrics;
- OpenTelemetry OTLP/HTTP for metrics, logs, and traces;
- Loki Push API for logs;
- Elasticsearch Bulk API for document ingestion;
- InfluxDB Line Protocol, MySQL, PostgreSQL, and the GreptimeDB gRPC and HTTP APIs.
These integrations cover specific ingestion or client interfaces, not every feature of the source system. Loki ingestion does not provide LogQL, and the open-source Elasticsearch integration covers the Bulk API rather than the full Query DSL. Use [SQL](/user-guide/query-data/sql.md) across signal types, [PromQL](/user-guide/query-data/promql.md) for metrics, and the Jaeger-compatible query API for traces. Review each protocol page before planning a migration.
## Open-Source and Enterprise Scope
Edition boundaries matter when evaluating scale, availability, and operating effort. The open-source project includes standalone and cluster deployment, object-storage support, SQL and PromQL, Flow, indexing, and the ingestion interfaces listed above.
[GreptimeDB Enterprise](/enterprise/overview.md) adds separately documented capabilities such as read replicas, workload isolation through Datanode groups, automatic Region balancing and repartitioning, RBAC, LDAP integration, audit logging, and enterprise disaster-recovery options. These capabilities are not implied when this guide describes the open-source cluster.
## Configuration Choices
Using one system changes the controls you operate; it does not make the workloads identical. You can tune each workload through these controls:
- [Table design](/user-guide/deployments-administration/performance-tuning/design-table.md): choose schemas, primary keys, indexes, append-only behavior, and partitioning. Set retention with [TTL policies](/user-guide/manage-data/overview.md#manage-data-retention-with-ttl-policies) and tune [compaction](/user-guide/deployments-administration/manage-data/compaction.md) when the workload needs it.
- [Capacity planning](/user-guide/deployments-administration/capacity-plan.md): size compute, memory, and local cache for the expected ingestion rate and mix of recent and historical queries. Use the [performance tuning guide](/user-guide/deployments-administration/performance-tuning/performance-tuning-tips.md) to adjust cache and query behavior from runtime evidence.
- [Durability and recovery](/user-guide/deployments-administration/disaster-recovery/overview.md): select a [WAL mode](/user-guide/deployments-administration/wal/overview.md), metadata storage, object-storage policy, and backup and restore process that meet the deployment's RPO and RTO.
- [Region operations](/user-guide/deployments-administration/manage-data/overview.md): plan table sharding, manual Region migration, and failover for distributed deployments. Enterprise deployments can also use [Datanode groups](/enterprise/deployments-administration/deploy-on-kubernetes/configure-datanode-groups.md) for workload isolation and [Region Balancer](/enterprise/autopilot/region-balancer.md) for automatic balancing.
The right settings depend on the workload and deployment mode; they are not a single preset shared by every signal.
## What Production Users Report
Published case studies give results for specific workloads and configurations:
- [OceanBase Cloud](https://greptime.com/blogs/2025-07-22-user-case-obcloud-log-management-greptimedb) runs 80+ GreptimeDB clusters with 300 TB of log and SQL audit data under a seven-day retention period. The case study reports around 1 GB/s of sustained writes and 60%+ lower overall log storage cost after moving from Loki.
- [Poizon](https://greptime.com/blogs/2025-05-06-poizon-observability-greptimedb-monitoring-use-case) uses Flow to maintain 10-second, 1-minute, and 10-minute rollups from detailed events. Its case study reports that pre-aggregation reduced P99 query latency from seconds to milliseconds.
Results depend on schema, indexes, retention, hardware, object-storage pricing, cache configuration, and query workload. Use published [benchmark reports](https://greptime.com/blogs/2024-09-09-report-summary) with their test conditions when sizing a deployment.
## Agent-Driven Investigation
[Agent RCA Bench](https://rca-bench.greptime.com/) had six models investigate the same 14 incidents through the native Prometheus, Loki, and Tempo query APIs, and through read-only SQL and PromQL over GreptimeDB tables.
Measured against the three separate backends, the GreptimeDB interface produced:
- **40% fewer wrong diagnoses** — 38 wrong answers in 168 investigations against 63, or 130 correct root causes against 105.
- **48% fewer input tokens read** — 199.8 million against 381.8 million. All six models read fewer tokens, and five of six were more accurate.
- **About 45% lower cost to run** — USD 98–101 against USD 184–187 for the full set of investigations.
The comparison changes the store, the query languages, and the tool surface together; it measures interface bundles, not the storage engine alone. The [report](https://github.com/GreptimeTeam/agent-rca-bench/blob/main/REPORT.md) publishes the protocol, the per-run artifacts, and the scope of each claim.
## Compare Against Your Current Stack
The useful comparison is usually against the systems already carrying the workload: Prometheus, Mimir, or Thanos for metrics; Loki or Tempo for logs and traces; Elasticsearch; VictoriaMetrics; or ClickHouse and ClickStack.
The [GreptimeDB comparison hub](https://greptime.com/compare/) links to product-specific pages covering architecture, protocol and query differences, migration paths, and benchmark conditions. Use the page for the system you run today rather than a context-free feature matrix.
## Choose a Next Step
- Run the [Quick Start](/getting-started/quick-start.md) to test ingestion and queries locally.
- Review the [product comparisons](https://greptime.com/compare/) against your current stack.
- Use the [migration guides](/user-guide/migrate-to-greptimedb/overview.md) to check protocol, query, dashboard, and historical-data changes before a production cutover.
---
## Authentication
Authentication occurs when a user attempts to connect to the database. In GreptimeDB, users are authenticated by "user
provider"s. There are various implementations of user providers in GreptimeDB:
- [Static User Provider](./static.md): A simple built-in user provider implementation that finds users from a static
file.
- [LDAP User Provider](/enterprise/deployments-administration/authentication.md): **Enterprise feature.** A user provider implementation that authenticates users against an external LDAP
server.
---
## Static User Provider
GreptimeDB supports username/password authentication with `static_user_provider`, which loads credentials from a file or a command-line argument at startup. `watch_file_user_provider` uses the same file format and reloads credentials when the file changes.
## Standalone Mode
GreptimeDB reads the user configuration from a file where each line defines a user with their password and optional permission mode.
### Basic Configuration
The basic format uses `=` as a separator between username and password:
```
greptime_user=greptime_pwd
alice=aaa
bob=bbb
```
Users configured this way have read-write access by default. File parsing follows these rules:
- Blank lines and lines starting with `#` are ignored after trimming leading and trailing whitespace from each line.
- Each credential must contain exactly one `=`. Plaintext passwords containing `=` are not supported, including with the `plain:` prefix. Store a supported hashed verifier for such passwords.
- Whitespace around `=` is not removed from the username or password. Do not add spaces around the separator.
- If a username appears more than once, the last valid entry takes effect.
- Malformed entries are skipped. The file must exist and contain at least one valid credential; otherwise, provider initialization fails.
- A read error, including invalid UTF-8, stops parsing. Valid credentials read before the error can still be loaded.
### Permission Modes
An optional permission mode controls read and write access. The format is:
```
username:permission_mode=password
```
Permission modes are case-insensitive:
- `rw`, `readwrite`, or `read_write` - Read and write access (default when omitted)
- `ro`, `readonly`, or `read_only` - Read-only access
- `wo`, `writeonly`, or `write_only` - Write-only access
An unrecognized permission mode makes the entry invalid: file providers skip the entry, while `static_user_provider:cmd` fails to initialize.
These modes are not scoped to individual databases or tables.
Example configuration with mixed permission modes:
```
admin=admin_pwd
alice:readonly=aaa
bob:writeonly=bbb
viewer:ro=viewer_pwd
editor:rw=editor_pwd
```
In this configuration:
- `admin` has read-write access (default)
- `alice` has read-only access
- `bob` has write-only access
- `viewer` has read-only access
- `editor` has explicitly set read-write access
### Password Formats
Since v1.1, passwords can be stored as plaintext or hashed verifiers. The supported formats are:
- `plain:` — plaintext. This is the default when no prefix is given.
- `pbkdf2_sha256:::` — a PBKDF2-SHA256 hash stored at rest.
- `mysql_native_password:` — a hashed verifier for MySQL `mysql_native_password` authentication.
- `pg_scram_sha256::::` — a SCRAM-SHA-256 verifier for PostgreSQL SASL authentication. Available since v1.2.
The hashed verifier examples below use the password `password` and, where required, the salt `salt`:
```
admin=plain:admin_pwd
alice=pbkdf2_sha256:4096:73616c74:c5e478d59288c841aa530db6845c4c8d962893a001ce4e11a4963873aa98134a
bob=mysql_native_password:2470c0c06dee42fd1618bb99005adca2ec9d1e19
carol=pg_scram_sha256:4096:73616c74:945e1c466fc9932efadc23781edc5d1e78d5e10f005933652af1a6105154f084:b9bf0e811b1fb6793671c0cc3adedf7c75cd72291191092ad65878c5a02aad2c
```
Permission modes combine with verifier formats. The verifier goes after the `=`:
```
alice:readonly=pbkdf2_sha256:4096:73616c74:c5e478d59288c841aa530db6845c4c8d962893a001ce4e11a4963873aa98134a
```
#### Protocol Compatibility
Protocol support depends on the verifier format and the authentication method selected by the provider:
| Verifier | HTTP/gRPC username/password | PostgreSQL SCRAM-SHA-256 | PostgreSQL cleartext | MySQL `mysql_native_password` |
| --- | --- | --- | --- | --- |
| `plain:` (or legacy `user=password`) | yes | yes | yes | yes |
| `pbkdf2_sha256:...` | yes | no | yes | no |
| `mysql_native_password:...` | no | no | no | yes |
| `pg_scram_sha256:...` | yes | yes | yes | no |
`static_user_provider` and `watch_file_user_provider` negotiate `mysql_native_password`, not `mysql_clear_password`. Users configured with `pbkdf2_sha256` or `pg_scram_sha256` cannot authenticate over MySQL through these providers. Enabling TLS or a client's cleartext authentication plugin does not change the server's selected method.
Hashed verifiers protect stored credentials; they do not encrypt network traffic. Enable TLS for production connections, particularly when using HTTP/gRPC username/password authentication or PostgreSQL cleartext authentication.
:::warning Breaking change
Passwords are prefix-parsed. A legacy plaintext password that literally starts with `plain:`, `pbkdf2_sha256:`, `mysql_native_password:`, or `pg_scram_sha256:` changes meaning. Use the `plain:` prefix to keep the literal value. For example, to keep the literal password `plain:secret`, configure it as `user=plain:plain:secret`.
:::
#### PostgreSQL SCRAM-SHA-256
SCRAM-SHA-256 lets PostgreSQL clients authenticate without sending the password in cleartext.
GreptimeDB selects SCRAM-SHA-256 only when every configured user has a plaintext password (with or without `plain:`) or a `pg_scram_sha256:` verifier. These two formats can be mixed.
With SCRAM-SHA-256, the server also sends an authentication challenge for an unknown username. After the client submits its proof, the server returns the same authentication failure result as for an incorrect password.
:::warning
A single `pbkdf2_sha256:` or `mysql_native_password:` entry makes PostgreSQL authentication fall back to cleartext for all users of that provider. A user with a `mysql_native_password:` verifier cannot authenticate through that fallback either, because the verifier cannot validate a cleartext password.
:::
Channel binding (`SCRAM-SHA-256-PLUS`) is not supported.
After configuring all users with SCRAM-compatible credentials, verify the method with libpq or `psql` 16 or newer:
```shell
psql "host=127.0.0.1 port=4003 user=carol dbname=public require_auth=scram-sha-256"
```
When the instance has fallen back to cleartext, that command fails with `server requested a cleartext password`.
### Generating Password Verifiers
The `greptime user hash-password` command generates password verifiers without starting the server. It is available since v1.1:
```shell
./greptime user hash-password --password-stdin
```
The command reads one line from stdin, removes trailing carriage returns and newlines, and prints the verifier to stdout. Empty input is rejected. `--password-stdin` does not disable terminal echo. In Bash, read the password without echo before piping it to the command:
```bash
read -r -s password && printf '%s' "$password" | ./greptime user hash-password --password-stdin
```
Use the output as the password value in the user configuration file:
```
admin=pbkdf2_sha256:4096::
```
Options:
- `--format ` — verifier format, `pbkdf2_sha256` (default), `mysql_native_password`, or `pg_scram_sha256`.
- `--password ` — plaintext password. Mutually exclusive with `--password-stdin`; exactly one is required. Prefer `--password-stdin` in scripts, since `--password` can leak through shell history or process listings.
- `--password-stdin` — read one line containing the plaintext password from stdin.
- `--iterations ` — PBKDF2-SHA256 / SCRAM-SHA-256 iteration count (default `4096`, range `1..=1000000`).
- `--salt-len ` — random salt length in bytes (default `16`, range `1..=1024`).
- `--salt-hex ` — fixed salt as hex instead of a random one, overriding `--salt-len`. The decoded salt must contain `1..=1024` bytes.
`--iterations`, `--salt-len`, and `--salt-hex` apply only to salted formats and are ignored for `mysql_native_password`.
To generate a `mysql_native_password` verifier:
```shell
./greptime user hash-password --password-stdin --format mysql_native_password
```
To generate a PostgreSQL SCRAM-SHA-256 verifier:
```shell
./greptime user hash-password --password-stdin --format pg_scram_sha256
```
### Starting the Server
Set `--user-provider` to `static_user_provider:file:`, replacing `` with the user configuration file path:
```shell
./greptime standalone start --user-provider='static_user_provider:file:'
```
The provider loads valid users and their permission modes into memory at startup. File changes take effect only after a restart.
Credentials can also be passed inline with `static_user_provider:cmd`. Separate entries with commas:
```shell
./greptime standalone start --user-provider='static_user_provider:cmd:admin=admin_pwd,alice:ro=alice_pwd'
```
The entries use the same credential syntax as the file. Inline plaintext passwords cannot contain `,` or `=`. Invalid entries fail provider initialization. Command-line credentials can appear in shell history and process listings; use a credential file for deployment.
### Dynamic File Reloading
`watch_file_user_provider` monitors a credential file and reloads users and permission modes without restarting the server:
```shell
./greptime standalone start --user-provider='watch_file_user_provider:'
```
The file must exist and contain at least one valid credential at startup. On reload:
- If the file cannot be opened or contains no valid credentials, the provider retains the previous configuration.
- Otherwise, the loaded credentials replace the previous configuration. Malformed entries are skipped; they do not reject the entire file. Users omitted from the loaded result are removed, including users whose entries became invalid.
Reloading does not disconnect existing MySQL or PostgreSQL sessions or update the user information already attached to them. Changed credentials and permission modes apply to subsequent authentication.
## Kubernetes Cluster
Configure users in `values.yaml`. See the [Helm Chart Configuration](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#authentication-configuration).
---
## Capacity Plan
This guide provides general advice on the CPU, memory, and storage requirements for GreptimeDB.
:::note
The ratios and sizes on this page come from Greptime's testing and deployments.
Use them as a starting point for sizing and validate them against your own
workload.
:::
GreptimeDB is designed to be lightweight upon startup,
which allows for the database to be initiated with minimal server resources.
However, when configuring your server capacity for a production environment,
there are several key considerations:
- Data points processed per second
- Query requests per second
- Data volume
- Data retention policy
- Hardware costs
To monitor the various metrics of GreptimeDB, please refer to [Monitoring](/user-guide/deployments-administration/monitoring/overview.md).
## CPU
Generally, applications that handle many concurrent queries, process large amounts of data,
or perform other compute-intensive operations will require more CPU cores.
Here are some recommendations for CPU resource usage,
but the actual number of CPUs you should use depends on your workload.
Consider allocating 30% of your CPU resources for data ingestion,
with the remaining 70% to querying and analytics.
A recommended guideline for resource allocation is to maintain a CPU to memory ratio of 1:4 (for instance, 8 core to 32 GB).
However, if your workload consists primarily of data ingestion with few queries,
a ratio of 1:2 (8 core to 16 GB) can also be acceptable.
## Memory
In general, the more memory you have, the faster your queries will run.
For basic workloads, it's recommended to have at least 8 GB of memory, and 32 GB for more advanced ones.
## Storage
GreptimeDB compresses data on disk, and stored data typically comes out at about 1/8 to 1/10 of the raw input size.
The actual ratio depends on your schema, column types, and how repetitive the values are.
Data can be stored either in a local file system or in cloud storage, such as AWS S3.
For more information on storage options,
please refer to the [storage configuration](/user-guide/deployments-administration/configuration.md#storage-options) documentation.
Cloud storage is highly recommended for data storage due to its simplicity in managing storage.
With cloud storage, local disk is used for query-related caches and the Write-Ahead Log (WAL), and about 200GB is usually enough.
Size it from your own cache configuration and WAL retention rather than treating 200GB as a fixed requirement.
In order to manage the storage costs effectively,
it is recommended setting a [retention policy](/user-guide/concepts/features-that-you-concern.md#can-i-set-ttl-or-retention-policy-for-different-tables-or-measurements).
## Example
Consider a scenario where your database handles a query rate of about 200 simple queries per second (QPS) and an ingestion rate of approximately 300k data points per second, using cloud storage for data.
Given the high ingestion and query rates,
here's an example of how you might allocate resources:
- CPU: 8 cores
- Memory: 32 GB
- Storage: 200 GB
Whether this allocation holds depends on your schema, average row width, index and TTL settings, and the time range and concurrency of the queries behind the 200 QPS figure.
Watch ingestion lag, query latency, and memory when you run your own workload against it.
---
## GreptimeDB Deployment Configuration
# Configuration
GreptimeDB supports **layered configuration** with the following precedence order (where each item overrides the one below it):
- Greptime command line options
- Configuration file options
- Environment variables
- Default values
You only need to set up the configurations you require.
GreptimeDB will assign default values for any settings not configured.
## How to set up configurations
### Greptime command line options
You can specify several configurations using command line arguments.
For example, to start GreptimeDB in standalone mode with a configured HTTP address:
```shell
greptime standalone start --http-addr 127.0.0.1:4000
```
For all the options supported by the Greptime command line,
refer to the [GreptimeDB Command Line Interface](/reference/command-lines/overview.md).
### Configuration file options
You can specify configurations in a TOML file.
For example, create a configuration file `standalone.example.toml` as shown below:
```toml
[storage]
data_home = "./greptimedb_data"
type = "File"
```
Then, specify the configuration file using the command line argument `-c [file_path]`.
```sh
greptime [standalone | frontend | datanode | flownode | metasrv] start -c config/standalone.example.toml
```
For example, to start in standalone mode:
```bash
greptime standalone start -c standalone.example.toml
```
#### Example files
Below are example configuration files for each GreptimeDB component,
including all available configurations.
In actual scenarios,
you only need to configure the required options and do not need to configure all options as in the sample file.
- [standalone](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/standalone.example.toml)
- [frontend](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/frontend.example.toml)
- [datanode](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/datanode.example.toml)
- [flownode](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/flownode.example.toml)
- [metasrv](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/metasrv.example.toml)
### Helm Configurations
When deploying GreptimeDB on Kubernetes using Helm charts,
you can configure certain settings directly in the Helm `values.yaml` file.
Please refer to the [Helm configuration documentation](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) for all Helm-supported configurations.
For configurations that are available only in the [options](#options) section of this document,
you can [inject complete TOML configuration files](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#injecting-configuration-files) into your deployment.
### Environment variable
Every item in the configuration file can be mapped to environment variables.
For example, to set the `data_home` configuration item for the datanode using an environment variable:
```toml
# ...
[storage]
data_home = "/data/greptimedb"
# ...
```
Use the following shell command to set the environment variable in the following format:
```
export GREPTIMEDB_DATANODE__STORAGE__DATA_HOME=/data/greptimedb
```
#### Environment Variable Rules
- Each environment variable should have the component prefix, for example:
- `GREPTIMEDB_FRONTEND`
- `GREPTIMEDB_METASRV`
- `GREPTIMEDB_DATANODE`
- `GREPTIMEDB_FLOWNODE`
- `GREPTIMEDB_STANDALONE`
- Use **double underscore `__`** separators. For example, the data structure `storage.data_home` is transformed to `STORAGE__DATA_HOME`.
The environment variable also accepts lists that are separated by commas `,`, for example:
```
GREPTIMEDB_METASRV__META_CLIENT__METASRV_ADDRS=127.0.0.1:3001,127.0.0.1:3002,127.0.0.1:3003
```
## Options
In this section, we will introduce some main configuration options.
For all options, refer to the [Configuration Reference](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/config.md) on Github.
### Runtime options
GreptimeDB executes background work on several dedicated Tokio runtimes.
These options are valid in all subcommands (`standalone`, `datanode`, `frontend`, and `metasrv`).
```toml
[runtime]
# The number of threads to execute the runtime for global read operations.
# Defaults to the number of CPU cores.
global_rt_size = 8
# The number of threads to execute compaction operations.
# Defaults to max(num_cpus / 2, 1).
compact_rt_size = 4
# The maximum number of blocking threads for compaction operations.
# Compaction picker CPU work runs on the compact runtime's blocking thread pool,
# so this limit prevents bursts of compaction planning from consuming all available CPUs.
# Defaults to max(num_cpus / 2, 1). An explicit 0 is clamped to 1.
compact_rt_max_blocking_threads = 4
```
| Option | Type | Default | Description |
| ------------------------------------ | ------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `runtime.global_rt_size` | Integer | The number of CPU cores | The number of threads to execute the runtime for global read operations. |
| `runtime.compact_rt_size` | Integer | `max(num_cpus / 2, 1)` | The number of threads to execute compaction operations. |
| `runtime.compact_rt_max_blocking_threads` | Integer | `max(num_cpus / 2, 1)` | The maximum number of blocking threads for compaction operations. Compaction picker CPU work runs on this blocking thread pool. An explicit `0` is clamped to `1`. |
### Write memory limiter options
Memory limiter options control the total memory used by concurrent write requests across all protocols (HTTP, gRPC, and Arrow Flight).
These options are valid in `frontend` and `standalone` subcommands.
```toml
# Maximum total memory for all concurrent write request bodies and messages
# Set to 0 to disable the limit (unlimited by default)
max_in_flight_write_bytes = "1GB"
# Policy when write bytes quota is exhausted
# Options: "wait" (default, 10s timeout), "wait()" (e.g., "wait(30s)"), "fail"
write_bytes_exhausted_policy = "wait"
```
| Option | Type | Default | Description |
| ------------------------------- | ------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `max_in_flight_write_bytes` | String | `0` | Maximum total memory for all concurrent write request bodies and messages (HTTP, gRPC, Flight). Set to `0` to disable the limit (unlimited). Supports units: `B`, `KB`, `MB`, `GB`, etc. Example: `1GB` limits total concurrent writes to 1GB. |
| `write_bytes_exhausted_policy` | String | `wait` | Policy when write bytes quota is exhausted. Options: `wait` (default, waits up to 10 seconds), `wait()` (custom timeout, e.g., `wait(30s)`), `fail` (immediately reject the request). |
### Datanode query concurrency limit
These options limit how many read queries can run at the same time on a datanode.
A query counts toward the limit until it finishes or the client closes the result stream.
If the limit is reached, a new query waits up to `concurrent_query_limiter_timeout` for an available slot before it fails.
These options are valid in the `datanode` subcommand.
```toml
# Maximum number of read queries that can run at the same time on the datanode.
# Set to 0 to disable the limit (unlimited by default).
max_concurrent_queries = 0
# Maximum time a query waits for an available slot when max_concurrent_queries is reached.
concurrent_query_limiter_timeout = "100ms"
```
| Option | Type | Default | Description |
| ---------------------------------- | ------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `max_concurrent_queries` | Integer | `0` | Maximum number of read queries that can run at the same time on the datanode. Set to `0` to disable the limit. |
| `concurrent_query_limiter_timeout` | String | `100ms` | Maximum time a query waits for an available slot after `max_concurrent_queries` is reached. The query fails if no slot is available in time. |
### Protocol options
Protocol options are valid in `frontend` and `standalone` subcommands,
specifying protocol server addresses and other protocol-related options.
In particular, `http.enable_api_server` lets you start a dedicated public HTTP API server that serves only the `/v1` APIs and `/dashboard`, while keeping the main HTTP server for internal and operational endpoints.
This separation is useful when you want to expose GreptimeDB to end users or applications without also exposing admin paths such as health, metrics, config, or debug endpoints.
To use it, set `http.enable_api_server = true` and configure `http.api_server_addr` to the address you want to expose externally, while keeping `http.addr` private for operator access.
:::tip NOTE
The HTTP protocol configuration is available for all GreptimeDB components: `frontend`, `datanode`, `flownode`, and `metasrv`.
:::
Below is an example configuration with default values.
You can change the values or disable certain protocols in your configuration file.
For example, to disable OpenTSDB protocol support, set the `enable` parameter to `false`.
Note that HTTP and gRPC protocols cannot be disabled for the database to function correctly.
```toml
[http]
addr = "127.0.0.1:4000"
timeout = "0s"
body_limit = "64MB"
enable_cors = true
# cors_allowed_origins = ["https://example.com"] # Optional: customize allowed origins
prom_validation_mode = "strict"
experimental_enable_prometheus_native_histogram = false
experimental_enable_explain_analyze_stream = true
# Enable the dedicated public HTTP API server (serves /v1 and /dashboard only)
enable_api_server = false
api_server_addr = "127.0.0.1:4006"
[grpc]
bind_addr = "127.0.0.1:4001"
runtime_size = 8
[grpc.tls]
mode = "disable"
cert_path = ""
key_path = ""
watch = false
[mysql]
enable = true
addr = "127.0.0.1:4002"
runtime_size = 2
keep_alive = "0s"
prepared_stmt_cache_size = 10000
[mysql.tls]
mode = "disable"
cert_path = ""
key_path = ""
watch = false
[postgres]
enable = true
addr = "127.0.0.1:4003"
runtime_size = 2
keep_alive = "0s"
[postgres.tls]
mode = "disable"
cert_path = ""
key_path = ""
watch = false
[opentsdb]
enable = true
[influxdb]
enable = true
# Default merge mode for tables automatically created by InfluxDB protocol.
# Available values: "last_non_null", "last_row".
default_merge_mode = "last_non_null"
[jaeger]
enable = true
[otlp]
enable = true
trace_ingest_chunk_size = 512
[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "0s"
max_batch_rows = 100000
max_concurrent_flushes = 256
worker_channel_capacity = 65526
max_inflight_requests = 3000
flow_notification_queue_capacity = 1024
```
The following table describes the options in detail:
| Option | Key | Type | Description |
| ---------- | -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| http | | | HTTP server options |
| | addr | String | Server address, "127.0.0.1:4000" by default |
| | timeout | String | HTTP request timeout. Set to `0s` to disable timeout (default: "0s"). When Prometheus Remote Write [batching mode](/user-guide/ingest-data/for-observability/prometheus.md#batching-mode) is enabled, a non-zero timeout less than or equal to `prom_store.pending_rows_flush_interval` plus 1 second is adjusted to that value. |
| | body_limit | String | HTTP max body size, "64MB" by default |
| | enable_cors | Boolean | Whether to enable HTTP CORS support, true by default. |
| | cors_allowed_origins | Array | Customized allowed origins for HTTP CORS. |
| | prom_validation_mode | String | Whether to check if strings are valid UTF-8 strings in Prometheus remote write requests. Available options: `strict`(reject any request with invalid UTF-8 strings), `lossy`(replace invalid characters with [UTF-8 REPLACEMENT CHARACTER U+FFFD, which looks like �](https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G24272)), `unchecked`(do not validate strings). |
| | experimental_enable_prometheus_native_histogram | Boolean | Experimental: enable Prometheus remote write v2 native histogram ingestion, false by default. |
| | experimental_enable_explain_analyze_stream | Boolean | Experimental: enable `POST /v1/sql/analyze/stream` for streaming `EXPLAIN ANALYZE VERBOSE` metrics, true by default. |
| | enable_api_server | Boolean | Whether to start the dedicated public HTTP API server. This server serves only the `/v1` APIs and `/dashboard`, making it safe to expose to end users. The main HTTP server (`addr`) is intended for internal use. Disabled by default; set to `true` to enable. |
| | api_server_addr | String | The address to bind the dedicated public HTTP API server, `"127.0.0.1:4006"` by default. Only takes effect when `enable_api_server` is `true`. |
| grpc | | | gRPC server options |
| | bind_addr | String | The address to bind the gRPC server, "127.0.0.1:4001" by default |
| | runtime_size | Integer | The number of server worker threads, 8 by default |
| | max_connection_age | String | Maximum lifetime of a gRPC connection that the server keeps it. Refer to ["MAX_CONNECTION_AGE"](https://grpc.io/docs/guides/keepalive/) for details. Defaults to not set. Example: "1h" for 1 hour, "30m" for 30 minutes |
| | flight_compression | String | Compression mode for frontend side Arrow IPC service. Available options: `none`: disable all compression, `transport`: only enable gRPC transport compression (zstd), `arrow_ipc`: only enable Arrow IPC compression (lz4), `all`: enable all compression. Default value is `none`. |
| mysql | | | MySQL server options |
| | enable | Boolean | Whether to enable MySQL protocol, true by default |
| | addr | String | Server address, "127.0.0.1:4002" by default |
| | runtime_size | Integer | The number of server worker threads, 2 by default |
| | keep_alive | String | Server-side keep-alive time. Set to `0s` to disable. |
| | prepared_stmt_cache_size | Integer | Maximum entries in the MySQL prepared statement cache, 10000 by default. |
| influxdb | | | InfluxDB Protocol options |
| | enable | Boolean | Whether to enable InfluxDB protocol in HTTP API, true by default |
| | default_merge_mode | String | Default merge mode for tables automatically created by InfluxDB protocol. Available values: `last_non_null`, `last_row`. Default: `last_non_null` |
| opentsdb | | | OpenTSDB Protocol options |
| | enable | Boolean | Whether to enable OpenTSDB protocol in HTTP API, true by default |
| jaeger | | | Jaeger protocol options |
| | enable | Boolean | Whether to enable Jaeger protocol in HTTP API, true by default. |
| otlp | | | OpenTelemetry protocol options |
| | enable | Boolean | Whether to enable OpenTelemetry protocol in HTTP API, true by default. |
| | trace_ingest_chunk_size | Integer | Maximum spans per trace ingest chunk. Set to `0` to disable splitting. |
| prom_store | | | Prometheus remote storage options |
| | enable | Boolean | Whether to enable Prometheus Remote Write and read in HTTP API, true by default |
| | with_metric_engine | Boolean | Whether to use the metric engine on Prometheus Remote Write, true by default |
| | pending_rows_flush_interval | String | Interval between batch flushes for Prometheus Remote Write. Set to a non-zero duration (e.g. `500ms`) to enable [batching mode](/user-guide/ingest-data/for-observability/prometheus.md#batching-mode). `0s` by default (disabled) |
| | max_batch_rows | Integer | Maximum number of rows per batch before a flush is triggered, 100000 by default |
| | max_concurrent_flushes | Integer | Maximum number of concurrent flush operations, 256 by default |
| | worker_channel_capacity | Integer | Capacity of the internal worker channel for receiving rows, 65526 by default |
| | max_inflight_requests | Integer | Maximum number of in-flight write requests waiting for batch completion, 3000 by default |
| | flow_notification_queue_capacity | Integer | Maximum number of pending logical-table flow notifications in the shared queue, 1024 by default. Only used in [batching mode](/user-guide/ingest-data/for-observability/prometheus.md#batching-mode). The value must be greater than 0. When the queue is full, notifications are dropped and counted by the `greptime_prom_store_flow_notification_dropped_total` metric. |
| postgres | | | PostgresSQL server options |
| | enable | Boolean | Whether to enable PostgresSQL protocol, true by default |
| | addr | String | Server address, "127.0.0.1:4003" by default |
| | runtime_size | Integer | The number of server worker threads, 2 by default |
| | keep_alive | String | Server-side keep-alive time. Set to `0s` to disable. |
For MySQL, Postgres and gRPC interface, TLS can be configured to enable transport
layer security.
| Option | Key | Type | Description |
| ----------------------------------------- | ----------- | ------- | ------------------------------------------------------------- |
| `mysql.tls`, `postgres.tls` or `grpc.tls` | | | TLS configuration for MySQL and Postgres |
| | `mode` | String | TLS mode, options are `disable`, `prefer` and `require` |
| | `cert_path` | String | File path for TLS certificate |
| | `key_path` | String | File path for TLS private key |
| | `watch` | Boolean | Watch file system changes and reload certificate and key file. Auto reload is not supported by `grpc.tls`; keep `grpc.tls.watch` set to `false`. |
### Query options
The `query` options are valid in standalone, datanode and frontend modes, which controls the query engine's behavior.
The following table describes the options in detail:
| Option | Key | Type | Description |
| ----------- | ------- | ---- | ----------------------------------------------------------------------------------- |
| parallelism | Integer | `0` | Parallelism of the query engine. Default to 0, which means the number of CPU cores. |
A sample configuration:
```toml
[query]
parallelism = 0
```
### Storage options
The `storage` options are valid in datanode and standalone mode, which specify the database data directory and other storage-related options.
GreptimeDB supports storing data in local file system, AWS S3 and compatible services (including MinIO, digitalocean space, Tencent Cloud Object Storage(COS), Baidu Object Storage(BOS) and so on), Azure Blob Storage and Aliyun OSS.
| Option | Key | Type | Description |
| ------- | ------------------------- | ------- | -------------------------------------------------------------------------------- |
| storage | | | Storage options |
| | type | String | Storage type, supports "File", "S3" and "Oss" etc. |
| File | | | Local file storage options, valid when type="File" |
| | data_home | String | Database storage root directory, "./greptimedb_data" by default |
| | copy_root | String | Root directory for standalone SQL access to local files (default `/copy`). Relative paths in `COPY` and external tables are resolved below this directory; absolute paths are accepted only when they fall inside it. Distributed deployments always reject local-file SQL access. See [Migrate Local SQL File Access](/user-guide/deployments-administration/migrate-local-sql-file-access.md) for upgrade guidance. |
| S3 | | | AWS S3 storage options, valid when type="S3" |
| | name | String | The storage provider name, default is `S3` |
| | bucket | String | The S3 bucket name |
| | root | String | The root path in S3 bucket |
| | endpoint | String | The API endpoint of S3 |
| | region | String | The S3 region |
| | access_key_id | String | The S3 access key id |
| | secret_access_key | String | The S3 secret access key |
| | enable_virtual_host_style | Boolean | Send API requests in virtual host style instead of path style. Default is false. |
| Oss | | | Aliyun OSS storage options, valid when type="Oss" |
| | name | String | The storage provider name, default is `Oss` |
| | bucket | String | The OSS bucket name |
| | root | String | The root path in OSS bucket |
| | endpoint | String | The API endpoint of OSS |
| | access_key_id | String | The OSS AccessKey ID |
| | access_key_secret | String | The OSS AccessKey Secret |
| Azblob | | | Azure Blob Storage options, valid when type="Azblob" |
| | name | String | The storage provider name, default is `Azblob` |
| | container | String | The container name |
| | root | String | The root path in container |
| | endpoint | String | The API endpoint of Azure Blob Storage |
| | account_name | String | The account name of Azure Blob Storage |
| | account_key | String | The access key |
| | sas_token | String | The shared access signature |
| Gcs | | | Google Cloud Storage options, valid when type="Gcs" |
| | name | String | The storage provider name, default is `Gcs` |
| | root | String | The root path in the GCS bucket |
| | bucket | String | The GCS bucket name |
| | scope | String | The GCS service scope |
| | credential_path | String | The GCS credentials path |
| | endpoint | String | The API endpoint of GCS |
A file storage sample configuration:
```toml
[storage]
data_home = "./greptimedb_data"
type = "File"
```
A S3 storage sample configuration:
```toml
[storage]
type = "S3"
bucket = "test_greptimedb"
root = "/greptimedb"
access_key_id = ""
secret_access_key = ""
```
#### Alibaba Cloud OSS authentication with RRSA/OIDC
When using ACK RAM Roles for Service Accounts (RRSA) to access OSS, keep the
following environment variables injected by RRSA available to the GreptimeDB
process:
- `ALIBABA_CLOUD_ROLE_ARN`
- `ALIBABA_CLOUD_OIDC_PROVIDER_ARN`
- `ALIBABA_CLOUD_OIDC_TOKEN_FILE`
GreptimeDB v1.1.0 changed the OSS credential lookup order to: configured access
keys, environment access keys, ECS RAM Role credentials, then
AssumeRoleWithOIDC. If the Pod can obtain ECS node credentials from metadata,
those credentials take precedence over RRSA credentials. An OSS permission
error does not cause the credential chain to retry with OIDC.
For deployments that use RRSA/OIDC, set
`ALIBABA_CLOUD_ECS_METADATA_DISABLED=true` in the GreptimeDB process environment
to skip ECS metadata credentials. For Kubernetes, add the following to the
GreptimeDB container's `env` list (this is a container fragment, not Helm values):
```yaml
env:
- name: ALIBABA_CLOUD_ECS_METADATA_DISABLED
value: "true"
```
Apply this setting to the standalone instance or all datanodes accessing OSS,
and restart the affected instances. Keep the RRSA variables and token mount in
place. Configured access keys and environment access keys still take precedence;
this setting does not force OIDC when those keys are present.
Do not disable ECS metadata if the deployment relies on ECS RAM Role credentials.
See the [OSS upgrade guidance](/user-guide/deployments-administration/upgrade.md#oss-rrsa-credential-precedence)
for upgrade symptoms and verification steps.
### Storage http client
`[storage.http_client]` sets the options for the http client that is used to send requests to the storage service.
Only applied for storage types "S3", "Oss", "Azblob" and "Gcs".
| Key | Type | Default | Description |
| ------------------------ | ------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pool_max_idle_per_host` | Integer | `1024` | The maximum idle connection per host allowed in the pool. |
| `connect_timeout` | String | `30s` | The timeout for only the connect phase of a http client. |
| `timeout` | String | `30s` | The total request timeout, applied from when the request starts connecting until the response body has finished. Also considered a total deadline. |
| `pool_idle_timeout` | String | `90s` | The timeout for idle sockets being kept-alive. |
### Storage engine provider
`[[storage.providers]]` setups the table storage engine providers. Based on these providers, you can create a table with a specified storage, see [create table](/reference/sql/create.md#create-table):
```toml
# Allows using multiple storages
[[storage.providers]]
name = "S3"
type = "S3"
bucket = "test_greptimedb"
root = "/greptimedb"
access_key_id = ""
secret_access_key = ""
[[storage.providers]]
name = "Gcs"
type = "Gcs"
bucket = "test_greptimedb"
root = "/greptimedb"
credential_path = ""
```
All configured providers' names can be used as the `storage` option when creating tables.
For storage from the same provider, if you want to use different S3 buckets as storage engines for different tables, you can set different `name` values and specify the `storage` option when creating the table.
### Object storage cache
When using remote storage services like AWS S3, Alibaba Cloud OSS, or Azure Blob Storage, fetching data during queries can be time-consuming. To address this, GreptimeDB provides a write cache mechanism to speed up repeated data access.
You can configure the cache size and behavior in the mito config if you don't want to use the default values.
```toml
[[region_engine]]
[region_engine.mito]
write_cache_size = "10GiB"
# Download files from object storage to fill the cache on write cache miss
enable_refill_cache_on_read = true
```
By default, GreptimeDB automatically downloads files from object storage to fill the write cache when there's a cache miss during queries (`enable_refill_cache_on_read = true`). This improves subsequent query performance by keeping frequently accessed data in the write cache. You can disable this behavior by setting `enable_refill_cache_on_read = false` if you want to minimize network traffic or storage costs.
Read [Performance Tuning Tips](/user-guide/deployments-administration/performance-tuning/performance-tuning-tips.md) for more detailed info.
### WAL options
GreptimeDB supports three WAL storage options—Local WAL, Remote WAL, and Noop WAL. See the [WAL Overview](/user-guide/deployments-administration/wal/overview.md) for a comparison of the options. For detailed configurations, refer to the [Local WAL](/user-guide/deployments-administration/wal/local-wal.md), [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md), and [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) documentation.
### Logging options
`frontend`, `metasrv`, `datanode` and `standalone` can all configure log and tracing related parameters in the `[logging]` section:
```toml
[logging]
dir = "./greptimedb_data/logs"
level = "info"
enable_otlp_tracing = false
enable_per_region_metrics = false
otlp_endpoint = "http://localhost:4318/v1/traces"
otlp_export_protocol = "http"
append_stdout = true
[logging.tracing_sample_ratio]
default_ratio = 1.0
```
- `dir`: log output directory.
- `level`: output log level, available log level are `info`, `debug`, `error`, `warn`, the default level is `info`.
- `enable_otlp_tracing`: whether to turn on tracing, not turned on by default.
- `enable_per_region_metrics`: whether to expose Prometheus per-region query load metrics, including `greptime_mito_region_query_cpu_time` and `greptime_mito_region_query_scanned_bytes`. This option is disabled by default because it creates one time series per region. Query stats reported through heartbeats and exposed in `INFORMATION_SCHEMA.REGION_STATISTICS` are not controlled by this option.
- `otlp_endpoint`: The OTLP endpoint for exporting traces. Defaults to `http://localhost:4318/v1/traces` when `otlp_export_protocol` is `http`, and `http://localhost:4317` when it is `grpc`. An endpoint without a scheme is prefixed with `http://`.
- `otlp_export_protocol`: The OTLP export protocol for traces. Available values: `http` (HTTP with binary protobuf) and `grpc`. Defaults to `http`.
- `otlp_headers`: Additional HTTP headers sent with OTLP export requests, configured as key-value pairs in the `[logging.otlp_headers]` table. Only takes effect when `otlp_export_protocol` is `http`.
- `append_stdout`: Whether to append logs to stdout. Defaults to `true`.
- `tracing_sample_ratio`: This field can configure the sampling rate of tracing. How to use `tracing_sample_ratio`, please refer to [How to configure tracing sampling rate](/user-guide/deployments-administration/monitoring/tracing.md#guide-how-to-configure-tracing-sampling-rate).
How to use distributed tracing, please reference [Tracing](/user-guide/deployments-administration/monitoring/tracing.md#tutorial-use-jaeger-to-trace-greptimedb)
### Pipeline options
`frontend` and `standalone` cache pipeline definitions in memory. The cache is configured in the `[pipeline]` section:
```toml
[pipeline]
cache_ttl = "10s"
```
- `cache_ttl`: Time to live of the local pipeline cache, `10s` by default. In a cluster, a pipeline created or deleted on one Frontend takes effect on the other Frontends after at most this duration. A longer TTL reduces reads of the `greptime_private.pipelines` table and increases that delay.
### Event recording
Recorded events are stored in the `greptime_private.events` system table.
```toml
[event_recorder]
# TTL of the events table. Defaults to 90 days.
ttl = "90d"
# Omit this option to record all current and future event types.
# Use an empty array to disable event recording.
event_types = ["create_table", "drop_table"]
```
- `ttl`: The TTL of the events table. The default is `90d`.
- `event_types`: The event types to record. When omitted, all current and future event types are recorded. Set it to `[]` to disable event recording.
Standalone supports the following event types:
```text
create_database, alter_database, drop_database,
create_flow, drop_flow,
create_table, create_logical_tables, alter_table, alter_logical_tables,
drop_table, undrop_table, purge_dropped_table, truncate_table,
create_view, drop_view, admin_function
```
`undrop_table` and `purge_dropped_table` require GreptimeDB Enterprise.
In distributed deployments, the Frontend supports the following event type:
```text
admin_function
```
Metasrv supports the following event types:
```text
region_migration,
create_database, alter_database, drop_database,
create_flow, drop_flow,
create_table, create_logical_tables, alter_table, alter_logical_tables,
drop_table, undrop_table, purge_dropped_table, truncate_table,
create_view, drop_view,
repartition, repartition_group,
batch_gc, wal_prune
```
`undrop_table` and `purge_dropped_table` require GreptimeDB Enterprise.
### Region engine options
The parameters corresponding to different storage engines can be configured for `datanode` and `standalone` in the `[region_engine]` section. Currently, options for `mito` and `metric` region engines are available.
Frequently used options:
```toml
[[region_engine]]
[region_engine.mito]
num_workers = 8
manifest_checkpoint_distance = 10
max_background_flushes = 4
max_background_compactions = 2
max_background_purges = 4
auto_flush_interval = "1h"
global_write_buffer_size = "1GB"
global_write_buffer_reject_size = "2GB"
default_region_write_buffer_size = "0"
sst_meta_cache_size = "512MB"
vector_cache_size = "512MB"
page_cache_size = "512MB"
write_cache_size = "5GB"
write_cache_ttl = "8h"
scan_memory_limit = "unlimited"
scan_memory_on_exhausted = "fail"
min_compaction_interval = "0m"
schedule_compaction_after_edit = true
default_flat_format = true
sst_write_buffer_size = "8MB"
max_concurrent_scan_files = 384
[region_engine.mito.index]
aux_path = ""
staging_size = "2GB"
staging_ttl = "7d"
metadata_cache_size = "64MiB"
content_cache_size = "128MiB"
content_cache_page_size = "64KiB"
result_cache_size = "128MiB"
[region_engine.mito.inverted_index]
create_on_flush = "auto"
create_on_compaction = "auto"
apply_on_query = "auto"
mem_threshold_on_create = "auto"
intermediate_path = ""
```
Mito selects the memtable implementation for each Region according to its table options and SST format. When `default_flat_format` is `true`, Regions without an explicit `sst_format` use flat SSTs and the bulk memtable. Configure `memtable.type` as a database or table option; `[region_engine.mito.memtable]` is not an engine setting. See [table options](/reference/sql/create.md#table-options).
Available options:
| Key | Type | Default | Descriptions |
| ---------------------------------------- | ------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `num_workers` | Integer | `8` | Number of region workers. |
| `manifest_checkpoint_distance` | Integer | `10` | Number of meta action updated to trigger a new checkpoint for the manifest. |
| `compress_manifest` | Bool | `false` | Whether to compress manifest and checkpoint file by gzip. |
| `max_background_flushes` | Integer | `Auto` | Max number of running background flush jobs (default: 1/2 of cpu cores). |
| `max_background_compactions` | Integer | `Auto` | Max number of running background compaction jobs (default: 1/4 of cpu cores). |
| `max_background_purges` | Integer | `Auto` | Max number of running background purge jobs (default: cpu cores). |
| `auto_flush_interval` | String | `1h` | Interval to auto flush a region if it has not flushed yet. Can be overridden per table with the [`auto_flush_interval` table option](/reference/sql/create.md#table-options). |
| `global_write_buffer_size` | String | `1GB` | Global write buffer size for all regions. If not set, it's default to 1/8 of OS memory with a max limitation of 1GB. |
| `global_write_buffer_reject_size` | String | `2GB` | Global write buffer size threshold to reject write requests. If not set, it's default to 2 times of `global_write_buffer_size` |
| `default_region_write_buffer_size` | String | `0` | Default per-region write buffer stall threshold. For a positive value, GreptimeDB schedules a flush when mutable memtable usage reaches half the value, stalls writes at the value, and rejects writes at twice the value. Setting it to `0` disables the default per-region limit. A table's `write_buffer_size` overrides this value, including an explicit `0` that disables the limit for that table. |
| `sst_meta_cache_size` | String | Auto | Cache size for SST metadata. Setting it to 0 to disable the cache.If not set, it's default to 1/8 of OS memory with a max limitation of 512MB. |
| `vector_cache_size` | String | `512MB` | Cache size for vectors and arrow arrays. Setting it to 0 to disable the cache.If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `page_cache_size` | String | `512MB` | Cache size for pages of SST row groups. Setting it to 0 to disable the cache.If not set, it's default to 1/8 of OS memory. |
| `write_cache_size` | String | `5GiB` | Capacity for write cache. If your disk space is sufficient, it is recommended to set it larger. |
| `write_cache_ttl` | String | `8h` | TTL for write cache. Defaults to 8 hours. |
| `preload_index_cache` | Bool | `true` | Preload index (puffin) files into cache on region open (default: true).When enabled, index files are loaded into the write cache during region initialization,which can improve query performance at the cost of longer startup times. |
| `index_cache_percent` | Integer | `20` | Percentage of write cache capacity allocated for index (puffin) files (default: 20).The remaining capacity is used for data (parquet) files.Must be between 0 and 100 (exclusive). For example, with a 5GiB write cache and 20% allocation,1GiB is reserved for index files and 4GiB for data files. |
| `enable_refill_cache_on_read` | Bool | `true` | Enable refilling cache on read operations (default: true).When disabled, cache refilling on read won't happen. |
| `manifest_cache_size` | String | `256MB` | Capacity for manifest cache (default: 256MB). |
| `selector_result_cache_size` | String | `512MB` | Cache size for time series selector (e.g. `last_value()`). Setting it to 0 to disable the cache.If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `range_result_cache_size` | String | `512MB` | Cache size for flat range scan results. Setting it to 0 to disable the cache.If not set, it's default to 1/16 of OS memory with a max limitation of 512MB. |
| `prefilter_result_cache_size` | String | `128MB` | Cache size for prefilter results. Setting it to 0 to disable the cache.If not set, it's default to 1/32 of OS memory with a max limitation of 128MB. |
| `sst_write_buffer_size` | String | `8MB` | Buffer size for SST writing. |
| `max_concurrent_scan_files` | Integer | `384` | Maximum number of SST files to scan concurrently. |
| `allow_stale_entries` | Bool | `false` | Whether to allow stale WAL entries during replay. |
| `scan_memory_limit` | String | `unlimited` | Memory limit for table scans across all queries. Supports absolute size (e.g., "2GB") or percentage of system memory (e.g., "20%"). Setting it to 0 or "unlimited" disables the limit. |
| `scan_memory_on_exhausted` | String | `fail` | Behavior when scan memory is exhausted. Options: `fail` (fail fast), `wait` or `wait()` (wait for memory). |
| `min_compaction_interval` | String | `0m` | Minimum time interval between two compactions. Set to "0m" (default) to allow compactions to run immediately without restriction. |
| `schedule_compaction_after_edit` | Bool | `true` | Whether to allow scheduling a compaction after a successful region edit.Setting this to `true` is a necessary but not sufficient condition for scheduling compaction after a region edit. Other constraints, such as `min_compaction_interval`, may still prevent compaction from being scheduled.Setting this to `false` guarantees that compaction will not be scheduled after a region edit. |
| `default_flat_format` | Bool | `true` | Whether Regions without an explicit `sst_format` use flat SSTs. Flat SSTs use the bulk memtable. |
| `scan_parallelism` | Integer | `0` | (Deprecated, use `max_concurrent_scan_files` instead) Legacy option for scan parallelism. |
| `index` | -- | -- | The options for index in Mito engine. |
| `index.aux_path` | String | `""` | Auxiliary directory path for the index in the filesystem. This path is used to store intermediate files for creating the index and staging files for searching the index. It defaults to `{data_home}/index_intermediate`. The default name for this directory is `index_intermediate` for backward compatibility. This path contains two subdirectories: `__intm` for storing intermediate files used during index creation, and `staging` for storing staging files used during index searching. |
| `index.staging_size` | String | `2GB` | The maximum capacity of the staging directory. |
| `index.staging_ttl` | String | `7d` | TTL for staging directory. Defaults to 7 days. Setting to "0s" disables TTL. |
| `index.metadata_cache_size` | String | `64MiB` | Cache size for index metadata.If not set, it's default to 1/32 of OS memory with a max limitation of 64MiB. |
| `index.content_cache_size` | String | `128MiB` | Cache size for index content.If not set, it's default to 1/16 of OS memory with a max limitation of 128MiB. |
| `index.content_cache_page_size` | String | `64KiB` | Page size for index content cache. |
| `index.result_cache_size` | String | `128MiB` | Cache size for index query results.If not set, it's default to 1/16 of OS memory with a max limitation of 128MiB. |
| `inverted_index` | -- | -- | The options for inverted index in Mito engine. |
| `inverted_index.create_on_flush` | String | `auto` | Whether to create the index on flush.- `auto`: automatically- `disable`: never |
| `inverted_index.create_on_compaction` | String | `auto` | Whether to create the index on compaction.- `auto`: automatically- `disable`: never |
| `inverted_index.apply_on_query` | String | `auto` | Whether to apply the index on query- `auto`: automatically- `disable`: never |
| `inverted_index.mem_threshold_on_create` | String | `auto` | Memory threshold for performing an external sort during index creation.- `auto`: 1/16 of the system memory- `unlimited`: disable external sorting, forcing all sorting operations to happen in memory- A fixed size, such as `64MB` |
| `inverted_index.intermediate_path` | String | `""` | File system path to store intermediate files for external sorting (default `{data_home}/index_intermediate`). |
The `metric` engine is optimized for handling metrics data with a large number of small tables.
:::note
Starting from v1.2, sparse primary key encoding is always enabled for the metric engine and cannot be disabled. It encodes only non-null primary key columns, improving write and query performance. Any `sparse_primary_key_encoding` setting in your configuration file is accepted but has no effect.
:::
### Specify meta client
The `meta_client` options are valid in `datanode` and `frontend` mode, which specify the Metasrv client information.
```toml
metasrv_addrs = ["127.0.0.1:3002"]
timeout = "3s"
connect_timeout = "1s"
ddl_timeout = "10s"
tcp_nodelay = true
```
The `meta_client` configures the Metasrv client, including:
- `metasrv_addrs`: The Metasrv address list.
- `timeout`: operation timeout, `3s` by default.
- `connect_timeout`, connect server timeout, `1s` by default.
- `ddl_timeout`, DDL execution timeout, `10s` by default.
- `tcp_nodelay`, `TCP_NODELAY` option for accepted connections, true by default.
### Heartbeat configuration
In distributed mode, heartbeat intervals are controlled by Metasrv using the `heartbeat_interval` option.
```toml
# Metasrv-only option
heartbeat_interval = "3s"
# Top-level option for reporting selected env vars in heartbeat extensions
heartbeat_env_vars = ["AZ", "REGION"]
```
| Key | Type | Default | Description |
| ----------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `heartbeat_interval` | String | `3s` | Metasrv base heartbeat interval. The frontend heartbeat interval is 6 times this value, and the datanode/flownode heartbeat interval is equal to this value. Heartbeat intervals are negotiated from Metasrv during handshake; local node configs do not override this. |
| `heartbeat_env_vars` | Array | `[]` | A top-level option. Lists the environment variable keys to read at startup and report to Metasrv through heartbeat messages. The values are sent as heartbeat extensions and stored in Metasrv's node information. For example, `heartbeat_env_vars = ["AZ", "REGION"]` collects the `AZ` and `REGION` environment variables. **Do not include sensitive variables** (e.g., secrets or credentials), as their values will be transmitted to and stored in Metasrv. |
### Default time zone configuration
The `default_timezone` option is applicable in both `frontend` and `standalone` modes, with a default value of `UTC`.
It specifies the default client timezone for interactions with GreptimeDB.
If the time zone is [specified in the clients](/user-guide/timezone.md#specify-time-zone-in-clients), this option will be overridden for that client session.
```toml
default_timezone = "UTC"
```
The `default_timezone` value can be any named time zone, such as `Europe/Berlin` or `Asia/Shanghai`.
For information on how the client time zone affects data ingestion and querying,
refer to the [Time Zone](/user-guide/timezone.md#impact-of-time-zone-on-sql-statements) guide.
### Metasrv-only configuration
The `datanode.client` options configure outbound gRPC connections from Metasrv to Datanodes.
The client message size limits are independent of the `[grpc]` server message size limits.
```toml
# The working home directory.
data_home = "./greptimedb_data"
# Store server address default to etcd store.
# For postgres store, the format is:
# "password=password dbname=postgres user=postgres host=localhost port=5432"
# For mysql store, the format is:
# "mysql://user:password@ip:port/dbname"
# For etcd store, the format is:
# "127.0.0.1:2379"
store_addrs = ["127.0.0.1:2379"]
# If it's not empty, the metasrv will store all data with this key prefix.
store_key_prefix = ""
# The datastore for meta server.
# Available values:
# - `etcd_store` (default value)
# - `memory_store`
# - `postgres_store`
# - `mysql_store`
backend = "etcd_store"
# Table name in RDS to store metadata. Effect when using a RDS kvbackend.
# **Only used when backend is RDS kvbackend.**
meta_table_name = "greptime_metakv"
# Optional PostgreSQL schema for metadata table and election table name qualification.
# When PostgreSQL public schema is not writable (e.g., PostgreSQL 15+ with restricted public),
# set this to a writable schema. GreptimeDB will use `meta_schema_name`.`meta_table_name`.
# **Only used when backend is `postgres_store`.**
meta_schema_name = "greptime_schema"
# Automatically create PostgreSQL schema if it doesn't exist.
# **Only used when backend is `postgres_store`.**
auto_create_schema = true
# Advisory lock id in PostgreSQL for election. Effect when using PostgreSQL as kvbackend
# Only used when backend is `postgres_store`.
meta_election_lock_id = 1
# Datanode selector type.
# - `round_robin` (default value)
# - `lease_based`
# - `load_based`
# For details, please see "https://docs.greptime.com/contributor-guide/metasrv/selector".
selector = "round_robin"
# Whether to enable region failover.
# This feature is only available on GreptimeDB running on cluster mode and:
# - Using Remote WAL, or Local WAL with `allow_region_failover_on_local_wal` set to `true`
# - Using shared storage (e.g., S3)
enable_region_failover = false
## The delay before starting region failure detection.
## This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started.
## Especially useful when the cluster is not deployed with GreptimeDB Operator and maintenance mode is not enabled.
region_failure_detector_initialization_delay = "10m"
# Whether to allow region failover on local WAL.
# **This option is not recommended to be set to true,
# because it may lead to data loss during failover.**
allow_region_failover_on_local_wal = false
## Max allowed idle time before removing node info from metasrv memory.
node_max_idle_time = "24hours"
# Base heartbeat interval for calculating distributed time constants.
# Heartbeat intervals are negotiated from metasrv during handshake; local node configs do not override this.
# heartbeat_interval = "3s"
# Whether to enable GreptimeDB telemetry. Enabled by default.
# enable_telemetry = true
# TLS configuration for kv store backend (applicable for etcd, PostgreSQL, and MySQL backends).
# If TLS is configured here and in `store_addrs`, this section overrides `store_addrs` TLS settings.
[backend_tls]
mode = "prefer"
cert_path = ""
key_path = ""
ca_cert_path = ""
## The backend client options.
## Currently, only applicable when using etcd as the metadata store.
[backend_client]
## The keep alive timeout for backend client.
keep_alive_timeout = "3s"
## The keep alive interval for backend client.
keep_alive_interval = "10s"
## The connect timeout for backend client.
connect_timeout = "3s"
## The gRPC server options.
[grpc]
bind_addr = "127.0.0.1:3002"
server_addr = "127.0.0.1:3002"
runtime_size = 8
## The maximum receive message size for gRPC server.
max_recv_message_size = "512MB"
## The maximum send message size for gRPC server.
max_send_message_size = "512MB"
## The server side HTTP/2 keep-alive interval
http2_keep_alive_interval = "10s"
## The server side HTTP/2 keep-alive timeout.
http2_keep_alive_timeout = "3s"
## The HTTP server options.
[http]
addr = "127.0.0.1:4000"
timeout = "0s"
body_limit = "64MB"
## Enable the dedicated public HTTP API server (serves /v1 and /dashboard only).
enable_api_server = false
api_server_addr = "127.0.0.1:4006"
## Procedure storage options.
[procedure]
## Procedure max retry time.
max_retry_times = 12
## Initial retry delay of procedures, increases exponentially
retry_delay = "500ms"
## Auto split large values.
max_metadata_value_size = "1500KiB"
## Max running procedures.
## The maximum number of procedures that can be running at the same time.
## If the number of running procedures exceeds this limit, the procedure will be rejected.
max_running_procedures = 128
# Failure detectors options.
# GreptimeDB uses the Phi Accrual Failure Detector algorithm to detect datanode failures.
[failure_detector]
## Maximum acceptable φ before the peer is treated as failed.
## Lower values react faster but yield more false positives.
threshold = 8.0
## The minimum standard deviation of the heartbeat intervals.
## So tiny variations don't make φ explode. Prevents hypersensitivity when heartbeat intervals barely vary.
min_std_deviation = "100ms"
## The acceptable pause duration between heartbeats.
## Additional extra grace period to the learned mean interval before φ rises, absorbing temporary network hiccups or GC pauses.
acceptable_heartbeat_pause = "10000ms"
## Datanode options.
[datanode]
## Datanode client options.
[datanode.client]
## Operation timeout.
timeout = "10s"
## Connect server timeout.
connect_timeout = "10s"
## `TCP_NODELAY` option for accepted connections.
tcp_nodelay = true
## The maximum receive message size for the gRPC client.
max_recv_message_size = "512MB"
## The maximum send message size for the gRPC client.
max_send_message_size = "512MB"
[wal]
# Available wal providers:
# - `raft_engine` (default): there're none raft-engine wal config since metasrv only involves in remote wal currently.
# - `kafka`: metasrv **have to be** configured with kafka wal config when using kafka wal provider in datanode.
provider = "raft_engine"
# Kafka wal config.
## The broker endpoints of the Kafka cluster.
broker_endpoints = ["127.0.0.1:9092"]
## Automatically create topics for WAL.
## Set to `true` to automatically create topics for WAL.
## Otherwise, use topics named `topic_name_prefix_[0..num_topics)`
auto_create_topics = true
## Number of topics.
num_topics = 64
## Topic selector type.
## Available selector types:
## - `round_robin` (default)
selector_type = "round_robin"
## A Kafka topic is constructed by concatenating `topic_name_prefix` and `topic_id`.
topic_name_prefix = "greptimedb_wal_topic"
## Expected number of replicas of each partition.
replication_factor = 1
## Above which a topic creation operation will be cancelled.
create_topic_timeout = "30s"
## The connect timeout for kafka client.
## **It's only used when the provider is `kafka`**.
connect_timeout = "3s"
## The timeout for kafka client.
## **It's only used when the provider is `kafka`**.
timeout = "3s"
# The Kafka SASL configuration.
# **It's only used when the provider is `kafka`**.
# Available SASL mechanisms:
# - `PLAIN`
# - `SCRAM-SHA-256`
# - `SCRAM-SHA-512`
# [wal.sasl]
# type = "SCRAM-SHA-512"
# username = "user_kafka"
# password = "secret"
# The Kafka TLS configuration.
# **It's only used when the provider is `kafka`**.
# [wal.tls]
# server_ca_cert_path = "/path/to/server_cert"
# client_cert_path = "/path/to/client_cert"
# client_key_path = "/path/to/key"
```
| Key | Type | Default | Descriptions |
| --------------------------------------------- | ------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data_home` | String | `./greptimedb_data/metasrv/` | The working home directory. |
| `bind_addr` | String | `127.0.0.1:3002` | The bind address of metasrv. |
| `server_addr` | String | `127.0.0.1:3002` | The communication server address for frontend and datanode to connect to metasrv, "127.0.0.1:3002" by default for localhost. |
| `store_addrs` | Array | `["127.0.0.1:2379"]` | Store server address. Configure the address based on your backend type, for example:- Use `127.0.0.1:2379` to connect to etcd- Use `"password=password dbname=postgres user=postgres host=localhost port=5432"` to connect to postgres- Use `"mysql://user:password@ip:port/dbname"` to connect to mysql |
| `selector` | String | `round_robin` | Datanode selector type.- `round_robin` (default value)- `lease_based`- `load_based`For details, see [Selector](/contributor-guide/metasrv/selector.md) |
| `enable_region_failover` | Bool | `false` | Whether to enable region failover.This feature is only available on GreptimeDB running on cluster mode and:- Using Remote WAL, or Local WAL with `allow_region_failover_on_local_wal` set to `true`- Using shared storage (e.g., S3). |
| `region_failure_detector_initialization_delay` | String | `10m` | The delay before starting region failure detection. This delay helps prevent Metasrv from triggering unnecessary region failovers before all Datanodes are fully started. Especially useful when the cluster is not deployed with GreptimeDB Operator and maintenance mode is not enabled. |
| `allow_region_failover_on_local_wal` | Bool | `false` | Whether to allow region failover on local WAL.**This option is not recommended to be set to true, because it may lead to data loss during failover.** |
| `node_max_idle_time` | String | `24hours` | Max allowed idle time before removing node info from metasrv memory. Nodes that haven't sent heartbeats for this duration will be considered inactive and removed. |
| `backend_tls` | -- | -- | TLS configuration for kv store backend. Applicable for etcd, PostgreSQL, and MySQL backends. If TLS is configured here and in `store_addrs`, this section overrides `store_addrs` TLS settings. |
| `backend_tls.mode` | String | `prefer` | TLS mode for the kv store backend. Options are `disable`, `prefer`, `require`, `verify_ca`, and `verify_full`. |
| `backend_tls.cert_path` | String | -- | File path for the client TLS certificate. |
| `backend_tls.key_path` | String | -- | File path for the client TLS private key. |
| `backend_tls.ca_cert_path` | String | -- | File path for the trusted CA certificate. |
| `backend_client` | -- | -- | The backend client options.Currently, only applicable when using etcd as the metadata store. |
| `backend_client.keep_alive_timeout` | String | `3s` | The keep alive timeout for backend client. |
| `backend_client.keep_alive_interval` | String | `10s` | The keep alive interval for backend client. |
| `backend_client.connect_timeout` | String | `3s` | The connect timeout for backend client. |
| `grpc` | -- | -- | The gRPC server options. |
| `grpc.bind_addr` | String | `127.0.0.1:3002` | The address to bind the gRPC server. |
| `grpc.server_addr` | String | `127.0.0.1:3002` | The communication server address for frontend and datanode to connect to metasrv. |
| `grpc.runtime_size` | Integer | `8` | The number of server worker threads. |
| `grpc.max_recv_message_size` | String | `512MB` | The maximum receive message size for the gRPC server. |
| `grpc.max_send_message_size` | String | `512MB` | The maximum send message size for the gRPC server. |
| `grpc.http2_keep_alive_interval` | String | `10s` | The server side HTTP/2 keep-alive interval. |
| `grpc.http2_keep_alive_timeout` | String | `3s` | The server side HTTP/2 keep-alive timeout. |
| `http` | -- | -- | HTTP server options. |
| `http.addr` | String | `127.0.0.1:4000` | HTTP server address. |
| `http.timeout` | String | `0s` | HTTP request timeout. Set to `0s` to disable timeout. |
| `http.body_limit` | String | `64MB` | HTTP max body size. |
| `http.enable_api_server` | Bool | `false` | Whether to start the dedicated public HTTP API server. When enabled, a separate server is started at `api_server_addr` that serves only `/v1` APIs and `/dashboard`. The main HTTP server is kept for internal use. |
| `http.api_server_addr` | String | `127.0.0.1:4006` | The address to bind the dedicated public HTTP API server. Only takes effect when `enable_api_server` is `true`. |
| `backend` | String | `etcd_store` | The datastore for metasrv.- `etcd_store` (default)- `memory_store` (In memory metadata storage - only used for testing.)- `postgres_store`- `mysql_store` |
| `meta_table_name` | String | `greptime_metakv` | Table name in RDS to store metadata. Effect when using a RDS kvbackend.**Only used when backend is RDS kvbackend.** |
| `meta_schema_name` | String | -- | Optional PostgreSQL schema for metadata table and election table name qualification. When PostgreSQL public schema is not writable (e.g., PostgreSQL 15+ with restricted public), set this to a writable schema. GreptimeDB will use `meta_schema_name.meta_table_name`.**Only used when backend is `postgres_store`.** |
| `auto_create_schema` | Bool | `true` | Automatically create PostgreSQL schema if it doesn't exist. When enabled, the system will execute `CREATE SCHEMA IF NOT EXISTS ` before creating metadata tables. This is useful in production environments where manual schema creation may be restricted. Note: The PostgreSQL user must have CREATE SCHEMA permission for this to work.**Only used when backend is `postgres_store`.** |
| `meta_election_lock_id` | Integer | `1` | Advisory lock id in PostgreSQL for election. Effect when using PostgreSQL as kvbackend**Only used when backend is `postgres_store`.** |
| `procedure` | -- | -- | Procedure storage options. |
| `procedure.max_retry_times` | Integer | `12` | Procedure max retry time. |
| `procedure.retry_delay` | String | `500ms` | Initial retry delay of procedures, increases exponentially. |
| `procedure.max_metadata_value_size` | String | `1500KiB` | Maximum size of a procedure metadata value before it is split into multiple backend entries. |
| `procedure.max_running_procedures` | Integer | `128` | The maximum number of procedures that can be running at the same time. If the number of running procedures exceeds this limit, the procedure will be rejected. |
| `failure_detector` | -- | -- | -- |
| `failure_detector.threshold` | Float | `8.0` | Maximum acceptable φ before the peer is treated as failed.Lower values react faster but yield more false positives. |
| `failure_detector.min_std_deviation` | String | `100ms` | The minimum standard deviation of the heartbeat intervals.So tiny variations don't make φ explode. Prevents hypersensitivity when heartbeat intervals barely vary. |
| `failure_detector.acceptable_heartbeat_pause` | String | `10000ms` | The acceptable pause duration between heartbeats.Additional extra grace period to the learned mean interval before φ rises, absorbing temporary network hiccups or GC pauses. |
| `datanode` | -- | -- | Datanode options. |
| `datanode.client` | -- | -- | Datanode client options. |
| `datanode.client.timeout` | String | `10s` | Operation timeout. |
| `datanode.client.connect_timeout` | String | `10s` | Connect server timeout. |
| `datanode.client.tcp_nodelay` | Bool | `true` | `TCP_NODELAY` option for accepted connections. |
| `datanode.client.max_recv_message_size` | String | `512MB` | The maximum receive message size for the gRPC client. |
| `datanode.client.max_send_message_size` | String | `512MB` | The maximum send message size for the gRPC client. |
| `wal` | -- | -- | -- |
| `wal.provider` | String | `raft_engine` | -- |
| `wal.broker_endpoints` | Array | -- | The broker endpoints of the Kafka cluster. |
| `wal.auto_prune_interval` | String | `0s` | Interval of automatically WAL pruning.Set to `0s` to disable automatically WAL pruning which delete unused remote WAL entries periodically. |
| `wal.auto_prune_logical_delete` | Bool | `false` | Whether automatic WAL pruning only updates metadata and skips Kafka `DeleteRecords`. See [Remote WAL configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md#metasrv-configuration). |
| `wal.flush_trigger_size` | String | `512MB` | Estimated WAL size of a region above which Metasrv triggers a flush of the region. See [Remote WAL configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md#metasrv-configuration). |
| `wal.checkpoint_trigger_size` | String | `128MB` | Estimated WAL size of a region above which Metasrv triggers a checkpoint of the region. See [Remote WAL configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md#metasrv-configuration). |
| `wal.auto_prune_parallelism` | Integer | `10` | Concurrent task limit for automatically WAL pruning. Each task is responsible for WAL pruning for a kafka topic. |
| `wal.num_topics` | Integer | `64` | Number of topics. |
| `wal.selector_type` | String | `round_robin` | Topic selector type.Available selector types:- `round_robin` (default) |
| `wal.topic_name_prefix` | String | `greptimedb_wal_topic` | A Kafka topic is constructed by concatenating `topic_name_prefix` and `topic_id`. |
| `wal.replication_factor` | Integer | `1` | Expected number of replicas of each partition. |
| `wal.create_topic_timeout` | String | `30s` | Above which a topic creation operation will be cancelled. |
| `wal.connect_timeout` | String | `3s` | The connect timeout for kafka client.**It's only used when the provider is `kafka`**. |
| `wal.timeout` | String | `3s` | The timeout for kafka client.**It's only used when the provider is `kafka`**. |
| `wal.sasl` | String | -- | The Kafka SASL configuration. |
| `wal.sasl.type` | String | -- | The SASL mechanisms, available values: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`. |
| `wal.sasl.username` | String | -- | The SASL username. |
| `wal.sasl.password` | String | -- | The SASL password. |
| `wal.tls` | String | -- | The Kafka TLS configuration. |
| `wal.tls.server_ca_cert_path` | String | -- | The path of trusted server ca certs. |
| `wal.tls.client_cert_path` | String | -- | The path of client cert (Used for enable mTLS). |
| `wal.tls.client_key_path` | String | -- | The path of client key (Used for enable mTLS). |
### Datanode-only configuration
```toml
node_id = 42
[grpc]
bind_addr = "127.0.0.1:3001"
server_addr = "127.0.0.1:3001"
runtime_size = 8
[runtime]
query_rt_size = 7
ingest_rt_size = 8
```
| Key | Type | Description |
| ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| node_id | Integer | The datanode identifier, should be unique. |
| grpc.bind_addr | String | The address to bind the gRPC server, `127.0.0.1:3001` by default. |
| grpc.server_addr | String | The address advertised to the metasrv, and used for connections from outside the host. If left empty or unset, the server will automatically use the IP address of the first network interface on the host, with the same port number as the one specified in `grpc.bind_addr`. |
| grpc.runtime_size | Integer | The number of gRPC server worker threads, 8 by default. |
| runtime.query_rt_size | Integer | The number of threads to execute datanode query operations. Defaults to `max(num_cpus - 1, 1)`. |
| runtime.ingest_rt_size | Integer | The number of threads to execute datanode ingestion operations. Defaults to the number of CPU cores. |
For the common runtime options shared by all components, see [Runtime options](#runtime-options).
### Frontend-only configuration
The `datanode.client` options configure outbound gRPC connections from Frontend to Datanodes.
The client message size limits are independent of the `[grpc]` server message size limits.
```toml
[datanode]
[datanode.client]
connect_timeout = "1s"
tcp_nodelay = true
max_recv_message_size = "512MB"
max_send_message_size = "512MB"
```
| Key | Type | Default | Description |
| ----------------------------------------------- | ------ | ------- | ------------------------------------------------------ |
| `datanode` | -- | -- | Datanode options. |
| `datanode.client` | -- | -- | Datanode client options. |
| `datanode.client.connect_timeout` | String | `1s` | Connect server timeout. |
| `datanode.client.tcp_nodelay` | Bool | `true` | `TCP_NODELAY` option for accepted connections. |
| `datanode.client.max_recv_message_size` | String | `512MB` | The maximum receive message size for the gRPC client. |
| `datanode.client.max_send_message_size` | String | `512MB` | The maximum send message size for the gRPC client. |
---
## Common Helm Chart Configurations
For each Helm Chart, you can create a `values.yaml` file for configuration. When you need to apply configurations, you can use the `helm upgrade` command:
```
helm upgrade --install ${release-name} ${chart-name} --namespace ${namespace} -f values.yaml
```
## GreptimeDB Cluster Chart
For complete configuration options, please refer to [GreptimeDB Cluster Chart](https://github.com/GreptimeTeam/helm-charts/tree/main/charts/greptimedb-cluster/README.md).
### GreptimeDB Image Configuration
The top-level variable `image` is used to configure the global GreptimeDB image for the cluster, as shown below:
```yaml
image:
# -- The image registry
registry: docker.io
# -- The image repository
repository: greptime/greptimedb
# -- The image tag
tag: "v1.2.1"
# -- The image pull secrets
pullSecrets: []
```
If you want to configure different images for each role in the cluster, you can use the `${role}.podTemplate.main.image` field (where `role` can be `meta`, `frontend`, `datanode` and `flownode`). This field will **override** the top-level `image` configuration, as shown below:
```yaml
image:
# -- The image registry
registry: docker.io
# -- The image repository
repository: greptime/greptimedb
# -- The image tag
tag: "v1.2.1"
# -- The image pull secrets
pullSecrets: []
frontend:
podTemplate:
main:
image: "greptime/greptimedb:latest"
```
In this case, the `frontend` image will be set to `greptime/greptimedb:latest`, while other components will use the top-level `image` configuration.
### Service Ports Configuration
You can configure service ports using the following fields:
- `httpServicePort`: Configures the HTTP service port, default `4000`
- `grpcServicePort`: Configures the gRPC service port, default `4001`
- `mysqlServicePort`: Configures the MySQL service port, default `4002`
- `postgresServicePort`: Configures the PostgreSQL service port, default `4003`
- `frontend.internalPort`: Configures the internal service port, default `4010`. This port is used for internal communication between GreptimeDB components.
### Datanode Storage Configuration
You can configure Datanode storage through the `datanode.storage` field, as shown below:
```yaml
datanode:
storage:
# -- Storage class for datanode persistent volume
storageClassName: null
# -- Storage size for datanode persistent volume
storageSize: 20Gi
# -- Storage retain policy for datanode persistent volume
storageRetainPolicy: Retain
# -- The dataHome directory, default is "/data/greptimedb/"
dataHome: "/data/greptimedb"
```
- `storageClassName`: Configures the StorageClass, defaults to Kubernetes current default StorageClass
- `storageSize`: Configures the storage size, default `20Gi`. You can use common capacity units, such as `50Gi`, etc.
- `storageRetainPolicy`: Configures the storage retention policy, default `Retain`. If set to `Delete`, the storage will be deleted when the cluster is deleted
- `dataHome`: Configures the data directory, default `/data/greptimedb/`
### Resource Configuration
The top-level variable `base.podTemplate.main.resources` is used to globally configure resources for each role, as shown below:
```yaml
base:
podTemplate:
main:
resources:
requests:
memory: "1Gi"
cpu: "1"
limits:
memory: "2Gi"
cpu: "2"
```
If you want to configure different resources for each role in the cluster, you can use the `${role}.podTemplate.main.resources` field (where `role` can be `meta`, `frontend`, `datanode`, `flownode`, etc.). This field will **override** the top-level `base.podTemplate.main.resources` configuration, as shown below:
```yaml
base:
podTemplate:
main:
resources:
requests:
memory: "1Gi"
cpu: "1"
limits:
memory: "2Gi"
cpu: "2"
frontend:
podTemplate:
main:
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "4"
memory: "8Gi"
```
### Role Replicas Configuration
For different roles, the number of replicas can be configured through the `${role}.replicas` field, as shown below:
```yaml
frontend:
replicas: 3
datanode:
replicas: 3
```
You can achieve horizontal scaling by configuring the number of replicas.
### Environment Variable Configuration
You can configure global environment variables through the `base.podTemplate.main.env` field, and configure different environment variables for each Role through the `${role}.podTemplate.main.env` field, as shown below:
```yaml
base:
podTemplate:
main:
env:
- name: GLOBAL_ENV
value: "global_value"
frontend:
podTemplate:
main:
env:
- name: FRONTEND_ENV
value: "frontend_value"
```
### Injecting Configuration Files
For different Role services, youcan inject custom TOML configuration files through the `${role}.configData` field, as shown below:
```yaml
datanode:
configData: |
[[region_engine]]
[region_engine.mito]
# Number of region workers
num_workers = 8
```
You can learn about GreptimeDB configuration options through [config.md](https://github.com/GreptimeTeam/greptimedb/blob/main/config/config.md).
In addition to using the `${role}.configData` field to inject configuration files, you can also specify corresponding files through `${role}.configFile`, as shown below:
```yaml
frontend:
configFile: "configs/frontend.toml"
```
In this case, ensure that the configuration file path matches the directory where the `helm upgrade` command is executed.
:::note
User-injected configuration files have lower priority by default than configuration items managed by GreptimeDB Operator. Some configuration items can only be configured through GreptimeDB Operator, and these items are exposed by default in `values.yaml`.
The following default configurations are managed by GreptimeDB Operator:
- Logging configuration;
- Datanode's Node ID;
:::
### Authentication Configuration
The Helm Chart does not enable User Provider mode authentication by default. You can enable User Provider mode authentication and configure user information through the `auth.enabled` field, as shown below:
```yaml
auth:
enabled: true
users:
- username: "admin"
password: "admin"
permission: "readwrite"
- username: "grafana"
password: "grafana_pwd"
permission: "readonly"
- username: "telegraf"
password: "telegraf_pwd"
permission: "writeonly"
```
GreptimeDB supports three permission modes:
| Mode | Shorthand | Allowed Operations | Typical Use Cases |
|------|-----------|-------------------|-------------------|
| `readwrite` | `rw` | Read + Write | Administrators, development environments |
| `readonly` | `ro` | Read only | Dashboards, reporting systems, read replicas |
| `writeonly` | `wo` | Write only | Data collectors, IoT devices, log aggregators |
:::note
Users without an explicit permission mode default to `readwrite`, maintaining backward compatibility.
### Logging Configuration
The top-level variable `logging` is used to configure global logging levels, as shown below:
```yaml
# -- Global logging configuration
logging:
# -- The log level for greptimedb, only support "debug", "info", "warn", "debug"
level: "info"
# -- The log format for greptimedb, only support "json" and "text"
format: "text"
# -- The logs directory for greptimedb
logsDir: "/data/greptimedb/logs"
# -- Whether to log to stdout only
onlyLogToStdout: false
# -- indicates whether to persist the log with the datanode data storage. It **ONLY** works for the datanode component.
persistentWithData: false
# -- The log filters, use the syntax of `target[span\{field=value\}]=level` to filter the logs.
filters: []
```
Where:
- `logging.level`: Configures the global log level, supports `debug`, `info`, `warn`, `error`.
- `logging.format`: Configures the global log format, supports `json` and `text`.
- `logging.logsDir`: Configures the global log directory, default `/data/greptimedb/logs`.
- `logging.onlyLogToStdout`: Configures whether to output only to stdout, disabled by default.
- `logging.persistentWithData`: Configures whether to persist logs with data storage, only applies to the `datanode` component, disabled by default.
- `logging.filters`: Configures global log filters, supports the syntax `target[span\{field=value\}]=level`. For example, if you want to enable `debug` level logging for certain components:
```yaml
logging:
level: "info"
format: "json"
filters:
- mito2=debug
```
Each role's logging configuration can be configured through the `${role}.logging` field, with fields consistent with the top-level `logging` and will **override** the top-level `logging` configuration, for example:
```yaml
frontend:
logging:
level: "debug"
```
### Enabling Flownode
The Helm Chart does not enable Flownode by default. You can enable Flownode through the `flownode.enabled` field, as shown below:
```yaml
flownode:
enabled: true
```
Other fields of `flownode` are configured similarly to other Roles, for example:
```yaml
flownode:
enabled: true
replicas: 1
podTemplate:
main:
resources:
requests:
memory: "1Gi"
cpu: "1"
limits:
memory: "2Gi"
cpu: "2"
```
### Object Storage Configuration
The `objectStorage` field is used to configure cloud object storage (such as AWS S3 and Azure Blob Storage, etc.) as the GreptimeDB storage layer.
#### AWS S3
```yaml
objectStorage:
credentials:
# AWS access key ID
accessKeyId: ""
# AWS secret access key
secretAccessKey: ""
s3:
# AWS S3 bucket name
bucket: ""
# AWS S3 region
region: ""
# The root path in bucket is 's3:////data/...'
root: ""
# The AWS S3 endpoint, see more detail: https://docs.aws.amazon.com/general/latest/gr/s3.html
endpoint: ""
```
#### Using IAM Roles for Service Accounts (IRSA) for S3
Instead of providing static access keys, you can use [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) (IRSA) to grant S3 access to GreptimeDB. This approach is more secure as it eliminates the need to manage long-lived credentials.
IRSA and [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) are two different mechanisms. The configuration below is IRSA: it annotates the service account with the role ARN. Pod Identity instead requires a pod identity association and no such annotation.
First, configure the datanode service account with the IAM role annotation. Only the datanode reads from and writes to S3:
```yaml
datanode:
podTemplate:
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: ${YOUR_IAM_ROLE_ARN}
```
Make sure the IAM role has permissions to read and write to the target S3 bucket:
- `s3:PutObject`
- `s3:ListBucket`
- `s3:GetObject`
- `s3:DeleteObject`
Then, configure the object storage without credentials:
```yaml
objectStorage:
s3:
bucket: "${YOUR_S3_BUCKET}"
region: "${YOUR_S3_REGION}"
root: "greptimedb"
```
:::note
When using IRSA, omit the `objectStorage.credentials` section entirely. The datanode pods will automatically obtain temporary credentials through the IAM role associated with the service account.
:::
#### Google Cloud Storage
```yaml
objectStorage:
credentials:
# GCP serviceAccountKey JSON-formatted base64 value
serviceAccountKey: ""
gcs:
# Google Cloud Storage bucket name
bucket: ""
# Google Cloud OAuth 2.0 Scopes, example: "https://www.googleapis.com/auth/devstorage.read_write"
scope: ""
# The root path in bucket is 'gcs:////data/...'
root: ""
# Google Cloud Storage endpoint, example: "https://storage.googleapis.com"
endpoint: ""
```
#### Azure Blob
```yaml
objectStorage:
credentials:
# Azure account name
accountName: ""
# Azure account key
accountKey: ""
azblob:
# Azure Blob container name
container: ""
# The root path in container is 'blob:////data/...'
root: ""
# Azure Blob endpoint, see: "https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-query-endpoint-srp?tabs=dotnet#query-for-the-blob-storage-endpoint"
endpoint: ""
```
#### AliCloud OSS
```yaml
objectStorage:
credentials:
# AliCloud access key ID
accessKeyId: ""
# AliCloud access key secret
accessKeySecret: ""
oss:
# AliCloud OSS bucket name
bucket: ""
# AliCloud OSS region
region: ""
# The root path in bucket is 'oss:////data/...'
root: ""
# The AliCloud OSS endpoint
endpoint: ""
```
#### Volcengine TOS
TOS ([Torch Object Storage](https://www.volcengine.com/docs/6349)) is a massive, secure, cost-effective, user-friendly, highly reliable, and highly available object storage service provided by [Volcengine](https://www.volcengine.com).
```yaml
objectStorage:
credentials:
# Volcengine access key ID
accessKeyId: ""
# Volcengine secret access key
secretAccessKey: ""
s3:
# Volcengine TOS bucket name
bucket: ""
# Volcengine TOS region
region: ""
# The root path in bucket is 'tos:////data/...'
root: ""
# The Volcengine TOS endpoint, see more detail: https://www.volcengine.com/docs/6349/107356
endpoint: ""
# Enable virtual host style so that OpenDAL will send API requests in virtual host style instead of path style.
enableVirtualHostStyle: true
```
### Prometheus Monitor Configuration
If you have [prometheus-operator](https://github.com/prometheus-operator/prometheus-operator) installed, you can create Prometheus PodMonitor to monitor GreptimeDB through the `prometheusMonitor.enabled` field as follows:
```yaml
prometheusMonitor:
# -- Create PodMonitor resource for scraping metrics using PrometheusOperator
enabled: false
# -- Interval at which metrics should be scraped
interval: "30s"
# -- Add labels to the PodMonitor
labels:
release: prometheus
```
### Debug Pod Configuration
The debug pod has various tools installed (such as mysql-client, psql-client, etc.). You can exec into the debug pod for debugging. Create it with the `debugPod.enabled` field as follows:
```yaml
debugPod:
# -- Enable debug pod
enabled: false
# -- The debug pod image
image:
registry: docker.io
repository: greptime/greptime-tool
tag: "20250606-04e3c7d"
# -- The debug pod resource
resources:
requests:
memory: 64Mi
cpu: 50m
limits:
memory: 256Mi
cpu: 200m
```
### Configuring Metasrv Backend Storage
#### Using MySQL and PostgreSQL as Backend Storage
You can configure the backend storage for the metasrv through the `meta.backendStorage` field.
Let's take MySQL as an example.
```yaml
meta:
backendStorage:
mysql:
# -- MySQL host
host: "mysql.default.svc.cluster.local"
# -- MySQL port
port: 3306
# -- MySQL database
database: "metasrv"
# -- MySQL table
table: "greptime_metakv"
# -- MySQL credentials
credentials:
# -- MySQL credentials secret name
secretName: "meta-mysql-credentials"
# -- MySQL credentials existing secret name
existingSecretName: ""
# -- MySQL credentials username
username: "root"
# -- MySQL credentials password
password: "test"
```
- `mysql.host`: The MySQL host.
- `mysql.port`: The MySQL port.
- `mysql.database`: The MySQL database.
- `mysql.table`: The MySQL table.
- `mysql.credentials.secretName`: The MySQL credentials secret name.
- `mysql.credentials.existingSecretName`: The MySQL credentials existing secret name. If you want to use an existing secret, you should make sure the secret contains the following keys: `username` and `password`.
- `mysql.credentials.username`: The MySQL credentials username. It will be ignored if `mysql.credentials.existingSecretName` is set. The `username` will be stored in the `username` key of the secret with `mysql.credentials.secretName`.
- `mysql.credentials.password`: The MySQL credentials password. It will be ignored if `mysql.credentials.existingSecretName` is set. The `password` will be stored in the `password` key of the secret with `mysql.credentials.secretName`.
Most of the fields of `meta.backendStorage.postgresql` are the same as the fields of `meta.backendStorage.mysql`. For example:
```yaml
meta:
backendStorage:
postgresql:
# -- PostgreSQL host
host: "postgres.default.svc.cluster.local"
# -- PostgreSQL port
port: 5432
# -- PostgreSQL database
database: "metasrv"
# -- PostgreSQL table
table: "greptime_metakv"
# -- PostgreSQL Advisory lock id used for election, shouldn't be used in other clusters or applications.
electionLockID: 1
# -- PostgreSQL credentials
credentials:
# -- PostgreSQL credentials secret name
secretName: "meta-postgresql-credentials"
# -- PostgreSQL credentials existing secret name
existingSecretName: ""
# -- PostgreSQL credentials username
username: "root"
# -- PostgreSQL credentials password
password: "root"
```
- `postgresql.host`: The PostgreSQL host.
- `postgresql.port`: The PostgreSQL port.
- `postgresql.database`: The PostgreSQL database.
- `postgresql.table`: The PostgreSQL table.
- `postgresql.electionLockID`: The Advisory lock id in PostgreSQL for election.
- `postgresql.credentials.secretName`: The PostgreSQL credentials secret name.
- `postgresql.credentials.existingSecretName`: The PostgreSQL credentials existing secret name. If you want to use an existing secret, you should make sure the secret contains the following keys: `username` and `password`.
- `postgresql.credentials.username`: The PostgreSQL credentials username. It will be ignored if `postgresql.credentials.existingSecretName` is set. The `username` will be stored in the `username` key of the secret with `postgresql.credentials.secretName`.
- `postgresql.credentials.password`: The PostgreSQL credentials password. It will be ignored if `postgresql.credentials.existingSecretName` is set. The `password` will be stored in the `password` key of the secret with `postgresql.credentials.secretName`.
#### Using etcd as Backend Storage
:::tip NOTE
The configuration structure has changed between chart versions:
- In older version: `meta.etcdEndpoints`
- In newer version: `meta.backendStorage.etcd.endpoints`
Always refer to the latest [values.yaml](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-cluster/values.yaml) in the Helm chart repository for the most up-to-date configuration structure.
:::
The etcd backend storage can be configured through the `meta.backendStorage.etcd` field.
```yaml
meta:
backendStorage:
etcd:
# -- Etcd endpoints
endpoints: ["etcd.etcd-cluster.svc.cluster.local:2379"]
# -- Etcd store key prefix
storeKeyPrefix: ""
```
- `etcd.endpoints`: The etcd endpoints.
- `etcd.storeKeyPrefix`: The etcd store key prefix. All keys will be stored with this prefix. If you want to use one etcd cluster for multiple GreptimeDB clusters, you can configure different store key prefixes for each GreptimeDB cluster. It's only for testing and debugging purposes.
### Enable Region Failover
You can enable Region Failover through the `meta.enableRegionFailover` field. Before enabling Region Failover, ensure your deployment meets the prerequisites outlined in the [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) documentation. If your configuration does not meet the prerequisites, the **Operator will fail to deploy the cluster components**.
```yaml
meta:
enableRegionFailover: true
```
### Enable IPv6
The `enableIPv6` field controls IPv6 support. Set it to true to enable IPv6 address allocation and communication. Defaults to false. Ensure that the underlying network infrastructure supports IPv6 before enabling this option.
```yaml
enableIPv6: true
```
### Enable GC
Repartitioning depends on shared object storage and GC. You can enable GC on both metasrv and datanode with the following example:
```yaml
meta:
configData: |
[gc]
enable = true
gc_cooldown_period = "5m"
datanode:
configData: |
[[region_engine]]
[region_engine.mito]
[region_engine.mito.gc]
enable = true
lingering_time = "10m"
unknown_file_lingering_time = "1d"
```
Make sure the datanode `lingering_time` is longer than the metasrv `gc_cooldown_period` to avoid deleting files that may still be in use.
#### Enable Region Failover on Local WAL
To enable Region Failover on local WAL, you need to set both `meta.enableRegionFailover: true` and add `allow_region_failover_on_local_wal = true` in the `meta.configData` field.
:::warning WARNING
Enabling Region Failover on local WAL may lead to data loss during failover. Ensure your operator version is greater than or equal to v0.2.2.
:::
```yaml
meta:
enableRegionFailover: true
configData: |
allow_region_failover_on_local_wal = true
```
### Dedicated WAL Volume
Configuring a dedicated WAL volume allows you to use a separate disk with a custom `StorageClass` for the WAL directory when deploying a GreptimeDB Datanode.
```yaml
dedicatedWAL:
enabled: true
raftEngine:
fs:
storageClassName: io2 # Use aws ebs io2 storage class for WAL for better performance.
name: wal
storageSize: 20Gi
mountPath: /wal
```
### Enable Remote WAL
To enable Remote WAL, both Metasrv and Datanode must be properly configured. Before proceeding, make sure to read the [Remote WAL Configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md) documentation for a complete overview of configuration options and important considerations.
```yaml
remoteWal:
enabled: true
kafka:
brokerEndpoints: ["kafka.kafka-cluster.svc:9092"]
meta:
configData: |
[wal]
provider = "kafka"
replication_factor = 1
auto_prune_interval = "30m"
datanode:
configData: |
[wal]
provider = "kafka"
overwrite_entry_start_id = true
```
---
## Deploying a GreptimeDB Cluster with Frontend Groups
In this guide, you will learn how to deploy a GreptimeDB cluster on Kubernetes with a frontend group consisting of multiple frontend instances.
## Prerequisites
- [Docker](https://docs.docker.com/get-started/get-docker/) >= v23.0.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
- [GreptimeDB Operator](https://github.com/GrepTimeTeam/greptimedb-operator) >= v0.3.0
- [ETCD](https://github.com/bitnami/charts/tree/main/bitnami/etcd)
## Upgrade operator
Install the GreptimeDB Operator, setting the image version to be greater than or equal to `v0.3.0`.
For detailed instructions on upgrading the operator, please refer to the [GreptimeDB Operator Management](/user-guide/deployments-administration/deploy-on-kubernetes/greptimedb-operator-management.md#upgrade) guide.
## Frontend Groups Configuration
:::tip NOTE
The configuration structure has changed between chart versions:
- In older version: `meta.etcdEndpoints`
- In newer version: `meta.backendStorage.etcd.endpoints`
Always refer to the latest [values.yaml](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-cluster/values.yaml) in the Helm chart repository for the most up-to-date configuration structure.
:::
When configuring frontend groups, ensure that each group includes a `name` field. The following `values.yaml` example demonstrates how to define separate frontend groups for read and write operations:
```yaml
frontend:
enabled: false # Disable default frontend group
frontendGroups:
- name: read
replicas: 1
config: |
default_timezone = "UTC"
[http]
timeout = "60s"
template:
main:
resources:
limits:
cpu: 2000m
memory: 2048Mi
- name: write
replicas: 1
meta:
replicas: 1
backendStorage:
etcd:
endpoints:
- "etcd.etcd-cluster.svc.cluster.local:2379"
datanode:
replicas: 1
```
You can use the following command to apply the configuration:
```
helm upgrade --install greptimedb greptime/greptimedb-cluster --namespace default -f values.yaml
```
## Validity
When setting the frontend groups, the name must be set.
```yaml
frontendGroups:
# - name: read #<=========The name must be set=============>
- replicas: 1
```
## Verify the Installation
Check the status of the pods:
```bash
kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
greptimedb-datanode-0 1/1 Running 0 32s
greptimedb-flownode-0 1/1 Running 0 17s
greptimedb-frontend-read-6d45bc9b89-hftqz 1/1 Running 0 23s
greptimedb-frontend-write-557b6585c6-jq874 1/1 Running 0 23s
greptimedb-meta-58cd4cff6c-zp7s9 1/1 Running 0 37s
```
To check the services:
```bash
kubectl get service -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
greptimedb-datanode ClusterIP None 4001/TCP,4000/TCP 102s
greptimedb-flownode ClusterIP None 4001/TCP 2m5s
greptimedb-frontend-read ClusterIP 10.96.174.200 4001/TCP,4000/TCP,4002/TCP,4003/TCP 42s
greptimedb-frontend-write ClusterIP 10.96.223.1 4001/TCP,4000/TCP,4002/TCP,4003/TCP 42s
greptimedb-meta ClusterIP 10.96.195.163 3002/TCP,4000/TCP 3m4s
```
## Conclusion
You have successfully deployed a GreptimeDB cluster with a frontend group consisting of read and write instances. You can now proceed to explore the functionality of your GreptimeDB cluster or integrate it with additional tools as needed.
---
## Deploying GreptimeDB Cluster with Remote WAL
In this guide, you will learn how to deploy GreptimeDB with Remote WAL on Kubernetes. Before you start, it's recommended to read the [Deploy GreptimeDB Cluster](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md) guide first.
## Prerequisites
- [Docker](https://docs.docker.com/get-started/get-docker/) >= v23.0.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
## Dependencies
Before deploying a GreptimeDB cluster with Remote WAL, ensure that the metadata storage and Kafka cluster are properly set up or that existing instances are available.
- Metadata storage: you can refer to [Manage Metadata Overview](/user-guide/deployments-administration/manage-metadata/overview.md) for more details. In this example, we use etcd as the metadata storage.
- Kafka Cluster: you can refer to [Manage Kafka](/user-guide/deployments-administration/wal/remote-wal/manage-kafka.md) for more details.
## Remote WAL Configuration
:::tip NOTE
The configuration structure has changed between chart versions:
- In older version: `meta.etcdEndpoints`
- In newer version: `meta.backendStorage.etcd.endpoints`
Always refer to the latest [values.yaml](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-cluster/values.yaml) in the Helm chart repository for the most up-to-date configuration structure.
:::
This example assumes you have a Kafka cluster running in the `kafka-cluster` namespace, and an etcd cluster running in the `etcd-cluster` namespace. The `values.yaml` file is as follows:
```yaml
meta:
backendStorage:
etcd:
endpoints: ["etcd.etcd-cluster.svc.cluster.local:2379"]
configData: |
[wal]
provider = "kafka"
# Testing value: no redundant copy. Set this from your durability policy in production.
replication_factor = 1
topic_name_prefix = "gtp_greptimedb_wal_topic"
auto_prune_interval = "30m"
datanode:
configData: |
[wal]
provider = "kafka"
overwrite_entry_start_id = true
remoteWal:
enabled: true
kafka:
brokerEndpoints: ["kafka.kafka-cluster.svc.cluster.local:9092"]
```
## Deploy GreptimeDB Cluster
You can deploy the GreptimeDB cluster with the following command:
```bash
helm upgrade --install mycluster \
--values values.yaml \
greptime/greptimedb-cluster \
-n default
```
## Best Practices
- **Avoid switching WAL storage options in an existing cluster**. If you need to change the WAL storage backend (e.g., from local to remote), you must **tear down the entire cluster** and perform a clean redeployment. This includes deleting:
- All PersistentVolumeClaims (PVCs) used by the GreptimeDB cluster.
- The object storage directory used by the cluster.
- The metadata storage associated with the cluster.
- Use a **minimal viable setup (MVP) to verify the cluster is functioning correctly**. This includes basic operations such as creating tables and inserting data to ensure the database works as expected.
## Next Steps
- Follow the [Deploy GreptimeDB Cluster](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md) guide to access your GreptimeDB cluster.
- Follow the [Quick Start](/getting-started/quick-start.md) guide to create tables and insert data.
- For more information about Remote WAL configuration, see [Remote WAL Configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md).
---
## Deploy GreptimeDB Cluster
In this guide, you will learn how to deploy a GreptimeDB cluster on Kubernetes using the GreptimeDB Operator.
:::note
The following output may have minor differences depending on the versions of the Helm charts and environment.
:::
## Prerequisites
- [Docker](https://docs.docker.com/get-started/get-docker/) >= v23.0.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) >= v0.20.0
## Create a test Kubernetes cluster
:::warning
Using `kind` is not recommended for production environments or performance testing. For such use cases, we recommend using cloud-managed Kubernetes services such as [Amazon EKS](https://aws.amazon.com/eks/), [Google GKE](https://cloud.google.com/kubernetes-engine/), or [Azure AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/), or deploying your own production-grade Kubernetes cluster.
:::
There are many ways to create a Kubernetes cluster for testing purposes. In this guide, we will use [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) to create a local Kubernetes cluster. You can skip this step if you want to use the existing Kubernetes cluster.
Here is an example using `kind` v0.20.0:
```bash
kind create cluster
```
Expected Output
```bash
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Thanks for using kind! 😊
```
Check the status of the cluster:
```bash
kubectl cluster-info
```
Expected Output
```bash
Kubernetes control plane is running at https://127.0.0.1:60495
CoreDNS is running at https://127.0.0.1:60495/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```
## Add the Greptime Helm repository
We provide the [official Helm repository](https://github.com/GreptimeTeam/helm-charts) for the GreptimeDB Operator and GreptimeDB cluster. You can add the repository by running the following command:
```bash
helm repo add greptime https://greptimeteam.github.io/helm-charts/
helm repo update
```
Check the charts in the Greptime Helm repository:
```
helm search repo greptime
```
Expected Output
```bash
NAME CHART VERSION APP VERSION DESCRIPTION
greptime/greptimedb-cluster 0.8.3 1.0.1 A Helm chart for deploying GreptimeDB cluster i...
greptime/greptimedb-enterprise-dashboard 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB Enterpr...
greptime/greptimedb-infra-test 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB infra t...
greptime/greptimedb-operator 0.5.9 0.5.5 The greptimedb-operator Helm chart for Kubernetes.
greptime/greptimedb-remote-compaction 0.1.1 0.1.0 Remote compaction components (scheduler, compac...
greptime/greptimedb-standalone 0.4.2 1.0.1 A Helm chart for deploying standalone greptimedb
```
## Install and verify the GreptimeDB Operator
It's ready to use Helm to install the GreptimeDB Operator on the Kubernetes cluster.
### Install the GreptimeDB Operator
The [GreptimeDB Operator](https://github.com/GrepTimeTeam/greptimedb-operator) is a Kubernetes operator that manages the lifecycle of GreptimeDB cluster.
Let's install the latest version of the GreptimeDB Operator in the `greptimedb-admin` namespace:
```bash
helm install greptimedb-operator greptime/greptimedb-operator -n greptimedb-admin --create-namespace
```
Expected Output
```bash
NAME: greptimedb-operator
LAST DEPLOYED: Sun Apr 26 20:43:58 2026
NAMESPACE: greptimedb-admin
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
Welcome to use greptimedb-operator
Chart version: 0.5.9
GreptimeDB Operator version: 0.5.5
***********************************************************************
Installed components:
* greptimedb-operator
The greptimedb-operator is starting, use `kubectl get deployment greptimedb-operator -n greptimedb-admin` to check its status.
```
:::note
There is another way to install the GreptimeDB Operator by using `kubectl` and `bundle.yaml` from the latest release:
```bash
kubectl apply -f \
https://github.com/GreptimeTeam/greptimedb-operator/releases/latest/download/bundle.yaml \
--server-side
```
This method is only suitable for quickly deploying GreptimeDB Operator in the test environments and is not recommended for production use.
:::
### Verify the GreptimeDB Operator installation
Check the status of the GreptimeDB Operator:
```bash
kubectl get pods -n greptimedb-admin -l app.kubernetes.io/instance=greptimedb-operator
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
greptimedb-operator-68d684c6cf-qr4q4 1/1 Running 0 4m8s
```
You also can check the CRD installation:
```bash
kubectl get crds | grep greptimedb
```
Expected Output
```bash
greptimedbclusters.greptime.io 2026-04-26T12:43:58Z
greptimedbstandalones.greptime.io 2026-04-26T12:43:58Z
```
The GreptimeDB Operator will use `greptimedbclusters.greptime.io` and `greptimedbstandalones.greptime.io` CRDs to manage GreptimeDB cluster and standalone resources.
## Install the etcd cluster
The GreptimeDB cluster requires an etcd cluster for metadata storage. Let's install an etcd cluster using Bitnami's etcd Helm [chart](https://github.com/bitnami/charts/tree/main/bitnami/etcd).
```bash
helm install etcd \
oci://registry-1.docker.io/bitnamicharts/etcd \
--version 12.0.8 \
--set replicaCount=3 \
--set auth.rbac.create=false \
--set auth.rbac.token.enabled=false \
--create-namespace \
--set global.security.allowInsecureImages=true \
--set image.registry=docker.io \
--set image.repository=greptime/etcd \
--set image.tag=3.6.1-debian-12-r3 \
-n etcd-cluster
```
Expected Output
```bash
NAME: etcd
LAST DEPLOYED: Sun Apr 26 20:49:03 2026
NAMESPACE: etcd-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: etcd
CHART VERSION: 12.0.8
APP VERSION: 3.6.1
** Please be patient while the chart is being deployed **
etcd can be accessed via port 2379 on the following DNS name from within your cluster:
etcd.etcd-cluster.svc.cluster.local
To create a pod that you can use as a etcd client run the following command:
kubectl run etcd-client --restart='Never' --image docker.io/greptime/etcd:3.6.1-debian-12-r3 --env ETCDCTL_ENDPOINTS="etcd.etcd-cluster.svc.cluster.local:2379" --namespace etcd-cluster --command -- sleep infinity
Then, you can set/get a key using the commands below:
kubectl exec --namespace etcd-cluster -it etcd-client -- bash
etcdctl put /message Hello
etcdctl get /message
To connect to your etcd server from outside the cluster execute the following commands:
kubectl port-forward --namespace etcd-cluster svc/etcd 2379:2379 &
echo "etcd URL: http://127.0.0.1:2379"
WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
- resources
- preUpgradeJob.resources
- disasterRecovery.cronjob.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
Substituted images detected:
- docker.io/greptime/etcd:3.6.1-debian-12-r3
```
Wait for the etcd cluster to be ready:
```bash
kubectl get pods -n etcd-cluster -l app.kubernetes.io/instance=etcd
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 2m8s
etcd-1 1/1 Running 0 2m8s
etcd-2 1/1 Running 0 2m8s
```
You can test the etcd cluster by running the following command:
```bash
kubectl -n etcd-cluster \
exec etcd-0 -- etcdctl endpoint health \
--endpoints=http://etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379,http://etcd-1.etcd-headless.etcd-cluster.svc.cluster.local:2379,http://etcd-2.etcd-headless.etcd-cluster.svc.cluster.local:2379
```
Expected Output
```bash
http://etcd-1.etcd-headless.etcd-cluster.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 3.008575ms
http://etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 3.136576ms
http://etcd-2.etcd-headless.etcd-cluster.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 3.147702ms
```
## Setup `values.yaml`
The `values.yaml` file contains parameters and configurations for GreptimeDB and is the key to defining the Helm chart.
For example, a minimal GreptimeDB cluster configuration is as follows:
```yaml
image:
# Image registry:
# Use `docker.io` for OSS GreptimeDB,
# consult staff for Enterprise GreptimeDB
registry:
# Image repository:
# Use `greptime/greptimedb` for OSS GreptimeDB,
# consult staff for Enterprise GreptimeDB
repository:
# Image tag:
# use database version for OSS GreptimeDB, for example, `v1.2.1`
# consult staff for Enterprise GreptimeDB
tag:
pullSecrets: []
initializer:
registry: docker.io
repository: greptime/greptimedb-initializer
tag: "v0.6.1"
frontend:
replicas: 1
meta:
replicas: 1
backendStorage:
etcd:
endpoints: ["etcd.etcd-cluster.svc.cluster.local:2379"]
datanode:
replicas: 1
flownode:
replicas: 1
```
The configuration above for the GreptimeDB cluster is not recommended for production use.
You should adjust the configuration according to your requirements.
You can refer to the [configuration documentation](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) for the complete `values.yaml` configuration options.
## Install the GreptimeDB cluster
Now that the GreptimeDB Operator and etcd cluster are installed,
and `values.yaml` is configured,
you can deploy a minimal GreptimeDB cluster:
```bash
helm upgrade --install mycluster \
greptime/greptimedb-cluster \
--values /path/to/values.yaml \
-n default
```
Expected Output
```bash
Release "mycluster" does not exist. Installing it now.
NAME: mycluster
LAST DEPLOYED: Sun Apr 26 21:00:40 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
Welcome to use greptimedb-cluster
Chart version: 0.8.3
GreptimeDB Cluster version: 1.0.1
***********************************************************************
Installed components:
* greptimedb-meta
* greptimedb-datanode
* greptimedb-frontend
* greptimedb-flownode
The greptimedb-cluster is starting, use `kubectl get pods -n default` to check its status.
```
When starting the cluster installation, we can check the status of the GreptimeDB cluster with the following command. If you use a different cluster name and namespace, you can replace `mycluster` and `default` with your configuration:
```bash
kubectl -n default get greptimedbclusters.greptime.io mycluster
```
Expected Output
```bash
NAME FRONTEND DATANODE META FLOWNODE PHASE VERSION AGE
mycluster 1 1 1 1 Running v1.0.1 111s
```
The above command will show the status of the GreptimeDB cluster. When the `PHASE` is `Running`, it means the GreptimeDB cluster has been successfully started.
You also can check the Pods status of the GreptimeDB cluster:
```bash
kubectl -n default get pods
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
mycluster-datanode-0 1/1 Running 0 2m51s
mycluster-flownode-0 1/1 Running 0 2m26s
mycluster-frontend-5894994974-w2cls 1/1 Running 0 2m32s
mycluster-meta-58cd4cff6c-ddbxq 1/1 Running 0 2m58s
```
As you can see, we have created a minimal GreptimeDB cluster consisting of 1 frontend, 1 datanode, 1 flownode, and 1 metasrv by default. For information about the components of a complete GreptimeDB cluster, you can refer to [architecture](/user-guide/concepts/architecture.md).
## Explore the GreptimeDB cluster
:::warning
For production use, you should access the GreptimeDB cluster or Grafana inside the Kubernetes cluster or using the LoadBalancer type service.
:::
### Access the GreptimeDB cluster
You can access the GreptimeDB cluster by using `kubectl port-forward` the frontend service:
```bash
kubectl -n default port-forward svc/mycluster-frontend 4000:4000 4001:4001 4002:4002 4003:4003
```
Expected Output
```bash
Forwarding from 127.0.0.1:4000 -> 4000
Forwarding from [::1]:4000 -> 4000
Forwarding from 127.0.0.1:4001 -> 4001
Forwarding from [::1]:4001 -> 4001
Forwarding from 127.0.0.1:4002 -> 4002
Forwarding from [::1]:4002 -> 4002
Forwarding from 127.0.0.1:4003 -> 4003
Forwarding from [::1]:4003 -> 4003
```
Please note that when you use a different cluster name and namespace, you can use the following command, and replace `${cluster}` and `${namespace}` with your configuration:
```bash
kubectl -n ${namespace} port-forward svc/${cluster}-frontend 4000:4000 4001:4001 4002:4002 4003:4003
```
:::warning
If you want to expose the service to the public, you can use the `kubectl port-forward` command with the `--address` option:
```bash
kubectl -n default port-forward --address 0.0.0.0 svc/mycluster-frontend 4000:4000 4001:4001 4002:4002 4003:4003
```
Please make sure you have the proper security settings in place before exposing the service to the public.
:::
Open the browser and navigate to `http://localhost:4000/dashboard` to access by the [GreptimeDB Dashboard](https://github.com/GrepTimeTeam/dashboard).
If you want to use other tools like `mysql` or `psql` to connect to the GreptimeDB cluster, you can refer to the [Quick Start](/getting-started/quick-start.md).
## Cleanup
:::danger
The cleanup operation will remove the metadata and data of the GreptimeDB cluster. Please make sure you have backed up the data before proceeding.
:::
### Stop the port-forwarding
Stop the port-forwarding for the GreptimeDB cluster:
```bash
pkill -f kubectl port-forward
```
### Uninstall the GreptimeDB cluster
To uninstall the GreptimeDB cluster, you can use the following command:
```bash
helm -n default uninstall mycluster
```
### Delete the PVCs
The PVCs wouldn't be deleted by default for safety reasons. If you want to delete the PV data, you can use the following command:
```bash
kubectl -n default delete pvc -l app.greptime.io/component=mycluster-datanode
```
### Cleanup the etcd cluster
You can use the following command to clean up the etcd cluster:
```bash
kubectl -n etcd-cluster exec etcd-0 -- etcdctl del "" --from-key=true
```
### Destroy the Kubernetes cluster
If you are using `kind` to create the Kubernetes cluster, you can use the following command to destroy the cluster:
```bash
kind delete cluster
```
## Next Steps
If you want to deploy a GreptimeDB cluster with Remote WAL, you can refer to [Configure Remote WAL](/user-guide/deployments-administration/deploy-on-kubernetes/configure-remote-wal.md) for more details.
---
## Deploy GreptimeDB Infrastructure Tests
In this guide, you will learn how to deploy GreptimeDB infrastructure testing tools on Kubernetes.
## Prerequisites
- [Docker](https://docs.docker.com/get-started/get-docker/) >= v23.0.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) >= v0.20.0
## Create a test Kubernetes cluster
:::warning
Using `kind` is not recommended for production environments or performance testing. For such use cases, we recommend using cloud-managed Kubernetes services such as [Amazon EKS](https://aws.amazon.com/eks/), [Google GKE](https://cloud.google.com/kubernetes-engine/), or [Azure AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/), or deploying your own production-grade Kubernetes cluster.
:::
There are many ways to create a Kubernetes cluster for testing purposes. In this guide, we will use [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) to create a local Kubernetes cluster. You can skip this step if you want to use the existing Kubernetes cluster.
Here is an example using `kind` v0.20.0:
```bash
kind create cluster
```
Expected Output
```bash
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Thanks for using kind! 😊
```
Check the status of the cluster:
```bash
kubectl cluster-info
```
Expected Output
```bash
Kubernetes control plane is running at https://127.0.0.1:60495
CoreDNS is running at https://127.0.0.1:60495/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```
## Add the Greptime Helm repository
We provide the [official Helm repository](https://github.com/GreptimeTeam/helm-charts) for the GreptimeDB Operator and GreptimeDB cluster. You can add the repository by running the following command:
```bash
helm repo add greptime https://greptimeteam.github.io/helm-charts/
helm repo update
```
Check the charts in the Greptime Helm repository:
```
helm search repo greptime
```
Expected Output
```bash
NAME CHART VERSION APP VERSION DESCRIPTION
greptime/greptimedb-cluster 0.8.3 1.0.1 A Helm chart for deploying GreptimeDB cluster i...
greptime/greptimedb-enterprise-dashboard 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB Enterpr...
greptime/greptimedb-infra-test 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB infra t...
greptime/greptimedb-operator 0.5.9 0.5.5 The greptimedb-operator Helm chart for Kubernetes.
greptime/greptimedb-remote-compaction 0.1.1 0.1.0 Remote compaction components (scheduler, compac...
greptime/greptimedb-standalone 0.4.2 1.0.1 A Helm chart for deploying standalone greptimedb
```
## Configuration
Create a custom configuration file `infra-test-values.yaml`:
```yaml
image:
# -- The image registry
registry: docker.io
# -- The image repository
repository: greptime/greptime-tool
# -- The image tag
tag: "20260521-c19a702"
# -- The image pull secrets
pullSecrets: []
# -- Configure to the tests
case:
disk:
enabled: true
storageClass: null
size: 20Gi
cpu:
enabled: true
rds:
enabled: true
host: "your-rds-host"
port: 3306
database: "test"
username: "your-rds-username"
password: "your-rds-password"
s3:
enabled: true
bucket: "bucket-name"
region: "s3-region"
accessKeyID: "your-access-key-id"
secretAccessKey: "your-secret-access-key"
kafka:
enabled: true
endpoint: "your-kafka-endpoint"
```
## Installation
Install with custom configuration:
```bash
helm upgrade --install greptimedb-infra-test greptime/greptimedb-infra-test \
--values infra-test-values.yaml \
-n default
```
## View Test Results
The testing tool runs as a Kubernetes job, with results output in the pod logs:
```bash
kubectl get pod -n default
```
Expected output
```bash
NAMESPACE NAME READY STATUS RESTARTS AGE
default greptimedb-infra-test-n7z74 0/1 Completed 0 10m
```
```bash
kubectl logs greptimedb-infra-test-n7z74 -n default
```
Expected output
```bash
================== Starting testing... ==================
================== Fetching UUID for kube-system namespace ==================
Successfully got UUID: 419a9914-5f9c-4d47-af69-3d09789db7a3
================== Disk tests ==================
================== Running full I/O test ==================
seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
seq-write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
rand-iops: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.28
Starting 6 processes
seq-read: Laying out IO file (1 file / 1024MiB)
seq-read: (groupid=0, jobs=6): err= 0: pid=14: Thu May 21 19:01:15 2026
read: IOPS=48.2k, BW=188MiB/s (198MB/s)(11.1GiB/60436msec)
slat (nsec): min=625, max=615290k, avg=53392.68, stdev=2634423.19
clat (nsec): min=875, max=636646k, avg=7912987.95, stdev=30341131.85
lat (usec): min=52, max=636649, avg=7966.49, stdev=30463.52
clat percentiles (usec):
| 1.00th=[ 223], 5.00th=[ 375], 10.00th=[ 562], 20.00th=[ 979],
| 30.00th=[ 1303], 40.00th=[ 1680], 50.00th=[ 2311], 60.00th=[ 3490],
| 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 13042], 95.00th=[ 23725],
| 99.00th=[103285], 99.50th=[270533], 99.90th=[421528], 99.95th=[438305],
| 99.99th=[488637]
bw ( KiB/s): min= 9969, max=450583, per=99.87%, avg=192706.79, stdev=24021.38, samples=596
iops : min= 2490, max=112644, avg=48175.32, stdev=6005.33, samples=596
write: IOPS=34.0k, BW=133MiB/s (139MB/s)(8021MiB/60436msec); 0 zone resets
slat (nsec): min=708, max=615509k, avg=77566.35, stdev=3387406.46
clat (usec): min=241, max=640859, avg=22321.94, stdev=54921.92
lat (usec): min=281, max=640862, avg=22399.63, stdev=55019.71
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 8],
| 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12],
| 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 26], 95.00th=[ 54],
| 99.00th=[ 359], 99.50th=[ 414], 99.90th=[ 485], 99.95th=[ 502],
| 99.99th=[ 634]
bw ( KiB/s): min=15032, max=284142, per=99.98%, avg=135879.94, stdev=12430.31, samples=596
iops : min= 3757, max=71033, avg=33968.77, stdev=3107.56, samples=596
lat (nsec) : 1000=0.01%
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.95%, 500=3.99%
lat (usec) : 750=3.41%, 1000=3.77%
lat (msec) : 2=15.07%, 4=11.77%, 10=31.21%, 20=20.84%, 50=5.71%
lat (msec) : 100=1.14%, 250=0.99%, 500=1.12%, 750=0.02%
cpu : usr=1.88%, sys=8.37%, ctx=786665, majf=0, minf=263
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.1%
issued rwts: total=2915521,2053386,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=188MiB/s (198MB/s), 188MiB/s-188MiB/s (198MB/s-198MB/s), io=11.1GiB (11.9GB), run=60436-60436msec
WRITE: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=8021MiB (8411MB), run=60436-60436msec
Disk stats (read/write):
vda: ios=2915375/2054793, merge=4/1130, ticks=8259465/6477837, in_queue=14772291, util=82.28%
================== Running mixed read/write test ==================
fiotest: (g=0): rw=rw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16
...
fio-3.28
Starting 8 processes
fiotest: Laying out IO file (1 file / 1024MiB)
fiotest: (groupid=0, jobs=8): err= 0: pid=22: Thu May 21 19:02:19 2026
read: IOPS=42.5k, BW=2657MiB/s (2786MB/s)(4092MiB/1540msec)
slat (nsec): min=1541, max=3755.0k, avg=6912.45, stdev=28881.60
clat (usec): min=30, max=52156, avg=1045.12, stdev=2441.43
lat (usec): min=45, max=52191, avg=1052.13, stdev=2441.68
clat percentiles (usec):
| 1.00th=[ 118], 5.00th=[ 188], 10.00th=[ 265], 20.00th=[ 392],
| 30.00th=[ 506], 40.00th=[ 635], 50.00th=[ 783], 60.00th=[ 922],
| 70.00th=[ 1074], 80.00th=[ 1254], 90.00th=[ 1680], 95.00th=[ 2311],
| 99.00th=[ 3949], 99.50th=[ 4948], 99.90th=[47973], 99.95th=[49546],
| 99.99th=[50070]
bw ( MiB/s): min= 2578, max= 2871, per=100.00%, avg=2697.33, stdev=16.88, samples=24
iops : min=41246, max=45946, avg=43154.67, stdev=270.28, samples=24
write: IOPS=42.6k, BW=2662MiB/s (2792MB/s)(4100MiB/1540msec); 0 zone resets
slat (usec): min=2, max=3271, avg= 8.33, stdev=25.30
clat (usec): min=95, max=52303, avg=1920.68, stdev=2554.97
lat (usec): min=138, max=52312, avg=1929.12, stdev=2555.52
clat percentiles (usec):
| 1.00th=[ 310], 5.00th=[ 498], 10.00th=[ 693], 20.00th=[ 988],
| 30.00th=[ 1254], 40.00th=[ 1516], 50.00th=[ 1762], 60.00th=[ 1975],
| 70.00th=[ 2212], 80.00th=[ 2474], 90.00th=[ 2835], 95.00th=[ 3294],
| 99.00th=[ 4817], 99.50th=[ 5800], 99.90th=[50594], 99.95th=[50594],
| 99.99th=[50594]
bw ( MiB/s): min= 2560, max= 2861, per=100.00%, avg=2701.42, stdev=15.97, samples=24
iops : min=40960, max=45780, avg=43219.67, stdev=255.59, samples=24
lat (usec) : 50=0.01%, 100=0.26%, 250=4.39%, 500=12.56%, 750=12.72%
lat (usec) : 1000=12.92%
lat (msec) : 2=34.10%, 4=21.45%, 10=1.28%, 20=0.01%, 50=0.23%
lat (msec) : 100=0.06%
cpu : usr=2.91%, sys=9.45%, ctx=35300, majf=0, minf=295
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=99.9%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=65475,65597,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=2657MiB/s (2786MB/s), 2657MiB/s-2657MiB/s (2786MB/s-2786MB/s), io=4092MiB (4291MB), run=1540-1540msec
WRITE: bw=2662MiB/s (2792MB/s), 2662MiB/s-2662MiB/s (2792MB/s-2792MB/s), io=4100MiB (4299MB), run=1540-1540msec
Disk stats (read/write):
vda: ios=58387/58501, merge=35/82, ticks=56442/57649, in_queue=114196, util=49.76%
================== CPU tests ==================
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 6
On-line CPU(s) list: 0-5
Vendor ID: Apple
Model: 0
Thread(s) per core: 1
Core(s) per cluster: 6
Socket(s): -
Cluster(s): 1
Stepping: 0x0
BogoMIPS: 48.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
================== PostgreSQL Tests ==================
================== Running connection test ==================
✓ Connection successful
PostgreSQL version: 17.5
================== Running latency test ==================
Query 1: 62.14ms
Query 2: 32.75ms
Query 3: 26.37ms
Query 4: 27.06ms
Query 5: 27.67ms
================== Running write performance test ==================
DROP TABLE
CREATE TABLE
Testing single row INSERT performance (1000 rows)...
Inserted 1000 rows in 50.45ms
Average: .05ms per insert
Testing bulk INSERT performance (10000 rows)...
k Bulk inserted 10000 rows in 292.07ms
Throughput: 34238 rows/sec
================== Running read performance test ==================
Total rows: 10001
Count query took: 29.80ms
SELECT with condition (id < 1000): 30.32ms
Aggregate query (GROUP BY): 32.23ms
================== Running concurrent connection test ==================
Spawned 20 concurrent connections (all completed)
================== Running database info query ==================
Database size: 8531091 bytes
Active connections: 6
PostgreSQL uptime: 2026-05-21 17:37:54.672924+00
================== Running cleanup ==================
DROP TABLE
================== PostgreSQL tests completed ==================
================== Generating 10MB test file... ==================
1+0 records in
1+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0255081 s, 411 MB/s
================== Running S3 transfer test... ==================
================== Upload s3 testfile... ==================
100.00% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.49 MB / 10.49 MB (491.69 kB/s) 22s (1/1)
Script started on 2026-05-21 19:02:22+00:00 []
100.00% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.49 MB / 10.49 MB (491.69 kB/s) 22s (1/1)
Script done on 2026-05-21 19:02:44+00:00 [COMMAND_EXIT_CODE="0"]
================== Upload time: 22 seconds ==================
================== Download s3 testfile... ==================
100.00% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.49 MB / 10.49 MB (7.48 MB/s) 1.6s (1/1)
Script started on 2026-05-21 19:02:44+00:00 []
100.00% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.49 MB / 10.49 MB (7.48 MB/s) 1.6s (1/1)
Script done on 2026-05-21 19:02:45+00:00 [COMMAND_EXIT_CODE="0"]
================== Download time: 1 seconds ==================
================== Verifying file integrity... ==================
================== S3 test passed ==================
File size: 10485760 bytes
MD5 checksum: 29022c552b0f81a9c89f6ff676a5c102
================== Kafka test ==================
================== Creating Kafka Topic ==================
================== Starting Consumer ==================
================== Producing Messages ==================
Produce time: 1 seconds
================== Consumed Messages ==================
Successfully consumed 23 messages
================== All tests completed! ==================
```
---
## Deploy GreptimeDB Standalone
In this guide, you will learn how to deploy a GreptimeDB standalone on Kubernetes.
:::note
The following output may have minor differences depending on the versions of the Helm charts and environment.
:::
## Prerequisites
- [Docker](https://docs.docker.com/get-started/get-docker/) >= v23.0.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
- [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) >= v0.20.0
## Create a test Kubernetes cluster
:::warning
Using `kind` is not recommended for production environments or performance testing. For such use cases, we recommend using cloud-managed Kubernetes services such as [Amazon EKS](https://aws.amazon.com/eks/), [Google GKE](https://cloud.google.com/kubernetes-engine/), or [Azure AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/), or deploying your own production-grade Kubernetes cluster.
:::
There are many ways to create a Kubernetes cluster for testing purposes. In this guide, we will use [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) to create a local Kubernetes cluster. You can skip this step if you want to use the existing Kubernetes cluster.
Here is an example using `kind` v0.20.0:
```bash
kind create cluster
```
Expected Output
```bash
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.27.3) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Thanks for using kind! 😊
```
Check the status of the cluster:
```bash
kubectl cluster-info
```
Expected Output
```bash
Kubernetes control plane is running at https://127.0.0.1:60495
CoreDNS is running at https://127.0.0.1:60495/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
```
## Add the Greptime Helm repository
We provide the [official Helm repository](https://github.com/GreptimeTeam/helm-charts) for the GreptimeDB Operator and GreptimeDB cluster. You can add the repository by running the following command:
```bash
helm repo add greptime https://greptimeteam.github.io/helm-charts/
helm repo update
```
Check the charts in the Greptime Helm repository:
```
helm search repo greptime
```
Expected Output
```bash
NAME CHART VERSION APP VERSION DESCRIPTION
greptime/greptimedb-cluster 0.8.3 1.0.1 A Helm chart for deploying GreptimeDB cluster i...
greptime/greptimedb-enterprise-dashboard 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB Enterpr...
greptime/greptimedb-infra-test 0.1.0 0.1.0 The Helm chart for deploying GreptimeDB infra t...
greptime/greptimedb-operator 0.5.9 0.5.5 The greptimedb-operator Helm chart for Kubernetes.
greptime/greptimedb-remote-compaction 0.1.1 0.1.0 Remote compaction components (scheduler, compac...
greptime/greptimedb-standalone 0.4.2 1.0.1 A Helm chart for deploying standalone greptimedb
```
## Install the GreptimeDB Standalone
### Basic Installation
For a quick start with default configuration:
```bash
helm upgrade --install greptimedb-standalone greptime/greptimedb-standalone -n default
```
Expected Output
```bash
Release "greptimedb-standalone" does not exist. Installing it now.
NAME: greptimedb-standalone
LAST DEPLOYED: Sun Apr 26 20:30:51 2026
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
Welcome to use greptimedb-standalone
Chart version: 0.4.2
GreptimeDB Standalone version: 1.0.1
***********************************************************************
Installed components:
* greptimedb-standalone
The greptimedb-standalone is starting, use `kubectl get statefulset greptimedb-standalone -n default` to check its status.
```
```bash
kubectl get pod -n default
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
greptimedb-standalone-0 1/1 Running 0 40s
```
### Customized Installation
For production or customized deployments, create a `values.yaml` file:
```yaml
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "4"
memory: "8Gi"
```
For more configuration options, please refer to the [documentation](https://github.com/GreptimeTeam/helm-charts/tree/main/charts/greptimedb-standalone).
Then install with custom values:
```bash
helm upgrade --install greptimedb-standalone greptime/greptimedb-standalone \
--values values.yaml \
--namespace default
```
Expected Output
```bash
Release "greptimedb-standalone" has been upgraded. Happy Helming!
NAME: greptimedb-standalone
LAST DEPLOYED: Sun Apr 26 20:35:11 2026
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
***********************************************************************
Welcome to use greptimedb-standalone
Chart version: 0.4.2
GreptimeDB Standalone version: 1.0.1
***********************************************************************
Installed components:
* greptimedb-standalone
The greptimedb-standalone is starting, use `kubectl get statefulset greptimedb-standalone -n default` to check its status.
```
```bash
kubectl get pod -n default
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
greptimedb-standalone-0 1/1 Running 0 3s
```
## Access GreptimeDB
After installation, you can access GreptimeDB through:
### MySQL Protocol
```bash
kubectl port-forward svc/greptimedb-standalone 4002:4002 -n default > connections.out &
```
```bash
mysql -h 127.0.0.1 -P 4002
```
Expected Output
```bash
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.4.2 Greptime
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>
```
### PostgreSQL Protocol
```bash
kubectl port-forward svc/greptimedb-standalone 4003:4003 -n default > connections.out &
```
```bash
psql -h 127.0.0.1 -p 4003 -d public
```
Expected Output
```bash
psql (16.2, server 16.3-GreptimeDB-1.0.1)
Type "help" for help.
public=>
```
### HTTP API
```bash
kubectl port-forward svc/greptimedb-standalone 4000:4000 -n default > connections.out &
```
```bash
curl -X POST \
-d 'sql=show databases' \
http://localhost:4000/v1/sql | jq .
```
Expected Output
```json
{
"output": [
{
"records": {
"schema": {
"column_schemas": [
{
"name": "Database",
"data_type": "String"
}
]
},
"rows": [
[
"greptime_private"
],
[
"information_schema"
],
[
"public"
]
],
"total_rows": 3
}
}
],
"execution_time_ms": 2
}
```
## Uninstallation
To remove GreptimeDB standalone:
```bash
helm uninstall greptimedb-standalone -n default
```
```bash
kubectl delete pvc -l app.kubernetes.io/instance=greptimedb-standalone -n default
```
---
## Deploying Kafka Cluster
In this guide, you will learn how to deploy a Kafka cluster on Kubernetes using a Helm Chart.
## Prerequisites
- Kubernetes >= v1.18.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
## Configuration Management
Before installation, you need to create a configuration file for the Kafka cluster. Please adjust the following `kafka-values.yaml` reference configuration according to your Kubernetes environment:
```yaml
image:
registry: docker.io
repository: greptime/kafka
tag: 3.9.0-debian-12-r1
listeners:
client:
containerPort: 9092
protocol: PLAINTEXT
name: CLIENT
controller:
protocol: PLAINTEXT
heapOpts: "-Xmx512m -Xms512m -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+ExplicitGCInvokesConcurrent"
controller:
replicaCount: 3
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
persistence:
enabled: true
storageClass: ""
size: 50Gi
broker:
replicaCount: 3
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
persistence:
enabled: true
storageClass: ""
size: 50Gi
extraConfig: |
num.network.threads=3
num.io.threads=8
min.insync.replicas=1
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
allow.everyone.if.no.acl.found=true
auto.create.topics.enable=true
default.replication.factor=1
max.partition.fetch.bytes=1048576
max.request.size=1048576
message.max.bytes=20000000
log.dirs=/bitnami/kafka/data
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=4
log.roll.hours=3
log.segment.bytes=1073741824
```
:::warning Retention settings when this cluster backs Remote WAL
`log.retention.hours=4` is a testing value. Kafka deletes a segment as soon as
any retention condition matches, so a retention window shorter than your longest
flush, failure, or manual-recovery window can remove WAL that GreptimeDB has not
replayed yet.
Do not add a size-based retention such as `log.retention.bytes` to topics used
for Remote WAL. See [Required Settings and
Limitations](/user-guide/deployments-administration/wal/remote-wal/configuration.md#required-settings-and-limitations)
for the full set of constraints.
:::
## Installing Kafka Cluster
Install the Kafka cluster in the kafka namespace:
```bash
helm upgrade --install kafka \
--create-namespace \
oci://registry-1.docker.io/bitnamicharts/kafka \
--version 31.0.0 \
-n kafka --values kafka-values.yaml
```
Expected Output
```bash
Release "kafka" does not exist. Installing it now.
Pulled: greptime-registry.cn-hangzhou.cr.aliyuncs.com/charts/kafka:31.0.0
Digest: sha256:85b135981fd5d951ceef8b51cdcbc6917ebface50d0eb3367eb7ddc4a5db482b
NAME: kafka
LAST DEPLOYED: Tue May 12 00:57:32 2026
NAMESPACE: kafka
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 31.0.0
APP VERSION: 3.9.0
** Please be patient while the chart is being deployed **
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
kafka.kafka.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092
kafka-broker-0.kafka-broker-headless.kafka.svc.cluster.local:9092
To create a pod that you can use as a Kafka client run the following commands:
kubectl run kafka-client --restart='Never' --image greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/kafka:3.9.0-debian-12-r1 --namespace kafka --command -- sleep infinity
kubectl exec --tty -i kafka-client --namespace kafka -- bash
PRODUCER:
kafka-console-producer.sh \
--bootstrap-server kafka.kafka.svc.cluster.local:9092 \
--topic test
CONSUMER:
kafka-console-consumer.sh \
--bootstrap-server kafka.kafka.svc.cluster.local:9092 \
--topic test \
--from-beginning
Substituted images detected:
- greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/kafka:3.9.0-debian-12-r1
```
## Verifying Kafka Cluster Installation
Check the status of Kafka components (Broker and Controller):
```bash
kubectl get pod -n kafka
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
kafka-broker-0 1/1 Running 0 8m3s
kafka-broker-1 1/1 Running 0 8m2s
kafka-broker-2 1/1 Running 0 8m1s
kafka-controller-0 1/1 Running 0 8m3s
kafka-controller-1 1/1 Running 0 8m2s
kafka-controller-2 1/1 Running 0 8m1s
```
# Configuring Kafka Endpoints
After the Kafka cluster is deployed, GreptimeDB can enable Remote WAL by configuring Kafka endpoints. For more information, refer to [this documentation](/user-guide/deployments-administration/deploy-on-kubernetes/configure-remote-wal.md).
```yaml
remoteWal:
enabled: true
kafka:
brokerEndpoints:
- "kafka-broker-0.kafka-broker-headless.kafka.svc.cluster.local:9092"
- "kafka-broker-1.kafka-broker-headless.kafka.svc.cluster.local:9092"
- "kafka-broker-2.kafka-broker-headless.kafka.svc.cluster.local:9092"
```
# Monitoring
- Install Prometheus Operator (eg: [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack))。
- Install the servicemonitor CRD。
To monitor the Kafka cluster, you need to have a monitoring system (such as Prometheus and Grafana) deployed in advance. Then add the following content to `kafka-values.yaml` and re-run the command to update the Kafka configuration:
```yaml
metrics:
jmx:
enabled: true
image:
registry: greptime-registry.cn-hangzhou.cr.aliyuncs.com
repository: greptime/jmx-exporter
tag: 1.0.1-debian-12-r9
serviceMonitor:
enabled: true
namespace: "kafka"
interval: "10s"
labels:
release: kube-prometheus-stack
```
## Grafana dashboard
Use [Kubernetes Kafka](https://grafana.com/grafana/dashboards/12483-kubernetes-kafka/) (ID: 12483) to monitor Kafka metrics.
1. Log in to your Grafana.
2. Navigate to Dashboards -> New -> Import.
3. Enter Dashboard ID: 12483, select a data source and load the dashboard.
# Uninstalling Kafka Cluster
Use the following command to uninstall the Kafka cluster:
```bash
helm -n kafka uninstall kafka
```
## Deleting PVCs
Deleting PVCs will remove persistent data from the Kafka cluster. Please ensure you have backed up your data before proceeding.
```bash
kubectl -n kafka delete pvc -l app.kubernetes.io/instance=kafka
```
---
## Deploying MinIO Cluster
In this guide, you will learn how to deploy a MinIO cluster on Kubernetes using a Helm Chart.
## Prerequisites
- Kubernetes >= v1.18.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
## Configuration Management
Before installation, you need to create a `minio-values.yaml` configuration file. Please adjust the following configuration according to your Kubernetes environment:
```yaml
global:
security:
allowInsecureImages: true
image:
registry: docker.io
repository: greptime/minio
tag: 2025.4.22-debian-12-r1
resources:
requests:
cpu: 500m
memory: 500Mi
limits:
cpu: '2'
memory: 2Gi
extraEnvVars:
- name: MINIO_REGION
value: "ap-southeast-1"
statefulset:
replicaCount: 4
mode: distributed
persistence:
storageClass: null
size: 100Gi
```
The root credentials are deliberately absent from this file. Helm reads a values file as plain YAML and does not expand shell variables in it, so a `${MINIO_ROOT_USER}` placeholder would be installed verbatim.
Create a Secret and point the chart at it instead. Avoid `--set-string` for the password: Helm splits that value on commas itself, so a password containing one fails to parse no matter how the shell quotes it.
```bash
kubectl create namespace minio
kubectl -n minio create secret generic minio-root-credentials \
--from-literal=root-user="$MINIO_ROOT_USER" \
--from-literal=root-password="$MINIO_ROOT_PASSWORD"
```
Then reference it from `minio-values.yaml`:
```yaml
auth:
existingSecret: minio-root-credentials
rootUserSecretKey: root-user
rootPasswordSecretKey: root-password
```
These root credentials are for administering MinIO and signing in to its console. GreptimeDB does not use them — it connects with a separate Access Key that you create in [Generating Access Key](#generating-access-key), which you can scope with its own permission policy.
## Installing MinIO Cluster
Install the MinIO cluster in the minio namespace:
```bash
helm upgrade \
--install minio oci://registry-1.docker.io/bitnamicharts/minio \
--create-namespace \
--version 16.0.10 \
-n minio --values minio-values.yaml
```
Expected Output
```bash
Release "minio" does not exist. Installing it now.
Pulled: greptime-registry.cn-hangzhou.cr.aliyuncs.com/charts/minio:16.0.10
Digest: sha256:96e220fd7cf1596879a243453b39c96a95d34f0005fdd452da3d094a7b386eb4
NAME: minio
LAST DEPLOYED: Tue May 12 17:21:30 2026
NAMESPACE: minio
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: minio
CHART VERSION: 16.0.10
APP VERSION: 2025.4.22
Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami for more information.
** Please be patient while the chart is being deployed **
MinIO® can be accessed via port 9000 on the following DNS name from within your cluster:
minio.minio.svc.cluster.local
To get your credentials run:
export ROOT_USER=$(kubectl get secret --namespace minio minio -o jsonpath="{.data.root-user}" | base64 -d)
export ROOT_PASSWORD=$(kubectl get secret --namespace minio minio -o jsonpath="{.data.root-password}" | base64 -d)
To connect to your MinIO® server using a client:
- Run a MinIO® Client pod and append the desired command (e.g. 'admin info'):
kubectl run --namespace minio minio-client \
--rm --tty -i --restart='Never' \
--env MINIO_SERVER_ROOT_USER=$ROOT_USER \
--env MINIO_SERVER_ROOT_PASSWORD=$ROOT_PASSWORD \
--env MINIO_SERVER_HOST=minio \
--image docker.io/bitnami/minio-client:2025.4.16-debian-12-r1 -- admin info minio
To access the MinIO® web UI:
- Get the MinIO® URL:
echo "MinIO® web URL: http://127.0.0.1:9001/minio"
kubectl port-forward --namespace minio svc/minio 9001:9001
Substituted images detected:
- greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/minio:2025.4.22-debian-12-r1
```
## Verifying MinIO Cluster Installation
Check the status of MinIO Pods:
```bash
kubectl get pod -n minio
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
minio-0 1/1 Running 0 30s
minio-1 1/1 Running 0 30s
minio-2 1/1 Running 0 30s
minio-3 1/1 Running 0 30s
```
# Creating Bucket and Access Key
## Accessing MinIO Console
1. First, expose the MinIO console service. You can use the kubectl port-forward command:
```bash
kubectl port-forward -n minio svc/minio 9001:9001
```
2. Open your browser: http://localhost:9001/login
3. Log in with the root credentials held in the `minio-root-credentials` Secret.
## Creating a Bucket
After logging into the MinIO console, follow these steps to create a Bucket:
1. Click "Buckets" in the left sidebar
2. Click the "Create Bucket" button
3. Enter a Bucket name, for example：`greptimedb-bucket`
4. Click "Create Bucket" to confirm creation
## Generating Access Key
1. Click "Access Keys" in the left sidebar
2. Click the "Create Access Key" button
3. Optional: Set permission policies
4. Click "Create" to generate the Access Key and Secret Key
:::warning
⚠️ Important: Please save the following information securely. You will need it when deploying GreptimeDB.
- Bucket name：greptimedb-bucket
- Region：ap-southeast-1
- MinIO Endpoint：`http://minio.minio:9000`
- Access Key：The Access Key
- Secret Key：The Secret Key
:::
# Configuring GreptimeDB to Use MinIO
When deploying a GreptimeDB cluster, you can use MinIO as backend storage with the following configuration:
```yaml
objectStorage:
credentials:
accessKeyId: ""
secretAccessKey: ""
s3:
bucket: "greptimedb-bucket"
region: "ap-southeast-1"
root: "greptimedb-data"
endpoint: "http://minio.minio:9000"
```
# Monitoring
- Install Prometheus Operator (e.g: [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)).
- Install the ServiceMonitor CRD.
To monitor the MinIO cluster, you need to have a monitoring system (such as Prometheus and Grafana) deployed in advance. Then add the following content to minio-values.yaml and re-run the command to update the MinIO configuration:
```yaml
metrics:
enabled: true
serviceMonitor:
enabled: true
namespace: minio
labels:
release: kube-prometheus-stack
interval: 30s
```
## Grafana dashboard
Use the [MinIO Dashboard](https://grafana.com/grafana/dashboards/13502-minio-dashboard/) (ID: 13502) to monitor MinIO metrics.
1. Log in to your Grafana.
2. Navigate to Dashboards -> New -> Import.
3. Enter Dashboard ID: 13502, select a data source, and load the dashboard.
# Uninstalling MinIO Cluster
Use the following command to uninstall the MinIO cluster:
```bash
helm -n minio uninstall minio
```
## Deleting PVCs
Deleting PVCs will remove persistent data from the MinIO cluster. Please ensure you have backed up your data before proceeding.
```bash
kubectl -n minio delete pvc -l app.kubernetes.io/instance=minio
```
---
## GreptimeDB Operator Management
The GreptimeDB Operator manages the [GreptimeDB](https://github.com/GrepTimeTeam/greptimedb) resources on [Kubernetes](https://kubernetes.io/) using the [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/).
It is like an autopilot that automates the deployment, provisioning, and orchestration of the GreptimeDB cluster and standalone.
The GreptimeDB Operator includes, but is not limited to, the following features:
- **Automated Provisioning**: Automates the deployment of the GreptimeDB cluster and standalone on Kubernetes by providing CRD `GreptimeDBCluster` and `GreptimeDBStandalone`.
- **Multi-Cloud Support**: Users can deploy the GreptimeDB on any Kubernetes cluster, including on-premises and cloud environments(like AWS, GCP, Aliyun, etc.).
- **Scaling**: Scale the GreptimeDB cluster as easily as changing the `replicas` field in the `GreptimeDBCluster` CR.
- **Monitoring Bootstrap**: Bootstrap the GreptimeDB monitoring stack for the GreptimeDB cluster by providing the `monitoring` field in the `GreptimeDBCluster` CR.
This document will show you how to install, upgrade, configure, and uninstall the GreptimeDB Operator on Kubernetes.
:::note
The following output may have minor differences depending on the versions of the Helm charts and environment.
:::
## Prerequisites
- Kubernetes >= v1.18.0
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
## Production Deployment
For production deployments, it's recommended to use Helm to install the GreptimeDB Operator.
### Installation
You can refer [Install and verify the GreptimeDB Operator](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md#install-and-verify-the-greptimedb-operator) for detailed instructions.
:::note
If you are using [Argo CD](https://argo-cd.readthedocs.io/en/stable/) to deploy applications, please make sure that the `Application` has set the [`ServerSideApply=true`](https://argo-cd.readthedocs.io/en/latest/user-guide/sync-options/#server-side-apply) to enable the server-side apply (other GitOps tools may have similar settings).
:::
### Upgrade
We always publish the latest version of the GreptimeDB Operator as a Helm chart in our official Helm repository.
When the new version of the GreptimeDB Operator is released, you can upgrade the GreptimeDB Operator by running the following commands.
#### Update the Helm repository
```bash
helm repo update greptime
```
Expected Output
```bash
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "greptime" chart repository
Update Complete. ⎈Happy Helming!⎈
```
You can use the following command to search the latest version of the GreptimeDB Operator:
```bash
helm search repo greptime/greptimedb-operator
```
Expected Output
```bash
NAME CHART VERSION APP VERSION DESCRIPTION
greptime/greptimedb-operator 0.5.9 0.5.5 The greptimedb-operator Helm chart for Kubernetes.
```
You also can use the following command to list all the available versions:
```bash
helm search repo greptime/greptimedb-operator --versions
```
#### Upgrade the GreptimeDB Operator
You can upgrade to the latest released version of the GreptimeDB Operator by running the following command:
```bash
helm -n greptimedb-admin upgrade --install greptimedb-operator greptime/greptimedb-operator
```
Expected Output
```bash
NAME: greptimedb-operator
LAST DEPLOYED: Sun Apr 26 20:43:58 2026
NAMESPACE: greptimedb-admin
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
***********************************************************************
Welcome to use greptimedb-operator
Chart version: 0.5.9
GreptimeDB Operator version: 0.5.5
***********************************************************************
Installed components:
* greptimedb-operator
The greptimedb-operator is starting, use `kubectl get deployment greptimedb-operator -n greptimedb-admin` to check its status.
```
If you want to upgrade to a specific version, you can use the following command:
```bash
helm -n greptimedb-admin upgrade --install greptimedb-operator greptime/greptimedb-operator --version
```
After the upgrade is complete, you can use the following command to verify the installation:
```bash
helm list -n greptimedb-admin
```
Expected Output
```bash
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
greptimedb-operator greptimedb-admin 1 2026-04-26 20:43:58.003167 +0800 CST deployed greptimedb-operator-0.5.9 0.5.5
```
### CRDs
There are two kinds of CRD that are installed with the GreptimeDB Operator: `GreptimeDBCluster` and `GreptimeDBStandalone`.
You can use the following command to verify the installation:
```bash
kubectl get crd | grep greptimedb
```
Expected Output
```bash
greptimedbclusters.greptime.io 2026-04-26T12:43:58Z
greptimedbstandalones.greptime.io 2026-04-26T12:43:58Z
```
By default, the GreptimeDB Operator chart will manage the installation and upgrade of the CRDs and the users don't need to manage them manually. If you want to know the specific definitions of these two types of CRD, you can refer to the GreptimeDB Operator [API documentation](https://github.com/GreptimeTeam/greptimedb-operator/blob/main/docs/api-references/docs.md).
### Configuration
The GreptimeDB Operator chart provides a set of configuration options that allow you to customize the installation, you can refer to the [GreptimeDB Operator Helm Chart](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-operator/README.md#values) for more details.
You can create a `values.yaml` to configure the GreptimeDB Operator chart (the complete configuration of `values.yaml` can be found in the [chart](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-operator/values.yaml)), for example:
```yaml
image:
# -- The image registry
registry: docker.io
# -- The image repository
repository: greptime/greptimedb-operator
# -- The image pull policy for the controller
imagePullPolicy: IfNotPresent
# -- The image tag
tag: "v0.6.1"
# -- The image pull secrets
pullSecrets: []
replicas: 1
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
```
You also can use one command to install or upgrade the GreptimeDB Operator with the custom configuration:
```bash
helm -n greptimedb-admin upgrade --install greptimedb-operator greptime/greptimedb-operator -f values.yaml
```
### Uninstallation
You can use the `helm` command to uninstall the GreptimeDB Operator:
```bash
helm -n greptimedb-admin uninstall greptimedb-operator
```
We don't delete the CRDs by default when you uninstall the GreptimeDB Operator.
:::danger
If you really want to delete the CRDs, you can use the following command:
```bash
kubectl delete crd greptimedbclusters.greptime.io greptimedbstandalones.greptime.io
```
The related resources will be removed after you delete the CRDs.
:::
---
## Deploy GreptimeDB on Kubernetes
GreptimeDB runs on Kubernetes, either on-premises or on a cloud provider such as AWS, Alibaba Cloud, or Google Cloud.
## Deploy GreptimeDB Standalone
For development, testing, or small-scale production use cases, you can [deploy a standalone GreptimeDB instance](deploy-greptimedb-standalone.md) on Kubernetes, without running a full cluster.
## Deploy GreptimeDB Cluster
For production environments requiring high availability and horizontal scaling,
[deploy a GreptimeDB cluster](deploy-greptimedb-cluster.md) using the GreptimeDB Operator on Kubernetes.
## Configurations
You can apply custom configurations to GreptimeDB by creating a `values.yaml` file
when deploying either GreptimeDB clusters or standalone instances.
For a complete list of available configuration options, see [Common Helm Chart Configurations](./common-helm-chart-configurations.md).
## Manage GreptimeDB Operator
The GreptimeDB Operator automates the setup, provisioning, and management of GreptimeDB instances on Kubernetes.
Learn how to [manage the GreptimeDB Operator](./greptimedb-operator-management.md),
including installation and upgrades.
## Advanced Deployments
Once you are familiar with [the architecture and components of GreptimeDB](/user-guide/concepts/architecture.md), you can move on to these deployment scenarios:
- [Deploy GreptimeDB Infrastructure test](deploy-greptimedb-infra-test.md): Prerequisite infrastructure testing for installing GreptimeDB.
- [Deploy MinIO cluster](deploy-minio.md): Learn how to deploy, configure, and monitor a MinIO cluster.
- [Deploy Kafka cluster](deploy-kafka.md): Learn how to deploy, configure, and monitor a Kafka cluster.
- [Deploy GreptimeDB Cluster with Remote WAL](configure-remote-wal.md): Configure Kafka as a remote write-ahead log (WAL) for your GreptimeDB cluster, so writes that a Datanode has not yet flushed to storage can be replayed from Kafka after a failure.
- [Use MySQL/PostgreSQL as Metadata Store](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#configuring-metasrv-backend-storage): Store cluster metadata in a MySQL or PostgreSQL database.
- [Deploy Multi-Frontend GreptimeDB Cluster](configure-frontend-groups.md): Set up a GreptimeDB cluster on Kubernetes with a frontend group consisting of multiple frontend instances for improved load distribution and availability.
---
## Data Export & Import
This guide describes how to use GreptimeDB's Export and Import tools for database data backup and restoration operations.
For the resumable snapshot-based Export/Import V2 workflow, see [Export and Import Data with Export/Import V2](./export-import-v2.md).
For detailed command-line options and advanced configurations, please refer to [Data Export & Import](/reference/command-lines/utilities/data.md).
## Export Operations
### Full Databases Backup
Export every database into its own directory, including all tables and their data.
```bash
# Export all databases backup
greptime cli data export \
--addr localhost:4000 \
--output-dir /tmp/backup/greptimedb
```
The output directory structure is as follows:
```
/
└── greptime/
└── /
├── create_database.sql
├── create_tables.sql
├── copy_from.sql
└──
```
#### Export to S3
Export all databases backup to S3:
```bash
greptime cli data export \
--addr localhost:4000 \
--s3 \
--s3-bucket \
--s3-access-key-id \
--s3-secret-access-key \
--s3-region \
--s3-root \
--s3-endpoint
```
### Export with Basic Authentication
If the GreptimeDB instance has authentication enabled, pass the credentials with `--auth-basic`:
```bash
greptime cli data export \
--addr localhost:4000 \
--output-dir /tmp/backup/greptimedb \
--auth-basic :
```
### Schema-Only Operations
Export only schemas without data. This operation exports `CREATE TABLE` statements into SQL files, allowing you to backup table structures without the actual data.
```bash
# Export only schemas
greptime cli data export \
--addr localhost:4000 \
--output-dir /tmp/backup/schemas \
--target schema
```
### Time-Range Based Backup
```bash
# Export data within specific time range
greptime cli data export --addr localhost:4000 \
--output-dir /tmp/backup/timerange \
--start-time "2024-01-01 00:00:00" \
--end-time "2024-01-31 23:59:59"
```
### Specific Database Backup
```bash
# To export a specific database
greptime cli data export \
--addr localhost:4000 \
--output-dir /tmp/backup/greptimedb \
--database '{my_database_name}'
```
## Import Operations
### Full Databases Import
Import every database from a backup directory produced by `greptime cli data export`.
```bash
# Import all databases
greptime cli data import \
--addr localhost:4000 \
--input-dir /tmp/backup/greptimedb
```
### Import with Basic Authentication
If the GreptimeDB instance has authentication enabled, pass the credentials with `--auth-basic`:
```bash
greptime cli data import \
--addr localhost:4000 \
--input-dir /tmp/backup/greptimedb \
--auth-basic :
```
### Schema-Only Import
Import only schemas without data. This operation imports `CREATE TABLE` statements from SQL files, allowing you to restore table structures without the actual data.
```bash
# Import only schemas
greptime cli data import \
--addr localhost:4000 \
--input-dir /tmp/backup/schemas \
--target schema
```
### Specific Database Import
```bash
# Import a specific database
greptime cli data import \
--addr localhost:4000 \
--input-dir /tmp/backup/greptimedb \
--database '{my_database_name}'
```
## Best Practices
1. **Parallelism Configuration**
- Adjust `--export-jobs`/`--import-jobs` based on available system resources
- Start with a lower value and increase gradually
- Monitor system performance during operations
2. **Backup Strategy**
- Incremental data backups using time ranges
- Periodic backups for disaster recovery
3. **Error Handling**
- Use `--max-retry` for handling transient failures
- Keep logs for troubleshooting
## Performance Tips
1. **Export Performance**
- Use time ranges for large datasets
- Adjust parallel jobs based on CPU/Memory
- Consider network bandwidth limitations
2. **Import Performance**
- Monitor database resources
## Troubleshooting
1. **Connection Errors**
- Verify server address and port
- Check network connectivity
- Ensure authentication credentials are correct
2. **Permission Issues**
- Verify read/write permissions on output/input directories
3. **Resource Constraints**
- Reduce parallel jobs
- Ensure sufficient disk space
- Monitor system resources during operations
---
## Metadata Export & Import
This guide describes how to use GreptimeDB's metadata export and import tools for metadata backup and restoration operations.
For detailed command-line options and advanced configurations, please refer to [Metadata Export & Import](/reference/command-lines/utilities/metadata.md).
:::note
- If Metasrv uses a PostgreSQL backend with `meta_schema_name` set, pass the same schema with `--meta-schema-name`. Without it, the CLI reads and writes the metadata table in the `search_path` of the connection, which may not be the table Metasrv uses.
- `--backend-tls-mode` defaults to `disable`. If the metadata store requires TLS, set `--backend-tls-mode` and the related `--backend-tls-*` options.
:::
## Overview
## Export Operations
### Export to S3 Cloud Storage
Export metadata from PostgreSQL to S3 for cloud-based backup storage:
```bash
greptime cli meta snapshot save \
--store-addrs 'password=password dbname=postgres user=postgres host=localhost port=5432' \
--backend postgres-store \
--s3 \
--s3-bucket your-bucket-name \
--s3-region ap-southeast-1 \
--s3-access-key-id \
--s3-secret-access-key
```
**Output**: Creates `metadata_snapshot.metadata.fb` file in the specified S3 bucket.
### Export to Local Directory
#### From PostgreSQL Backend
Export metadata from PostgreSQL to local directory:
```bash
greptime cli meta snapshot save \
--store-addrs 'password=password dbname=postgres user=postgres host=localhost port=5432' \
--backend postgres-store
```
#### From MySQL Backend
Export metadata from MySQL to local directory:
```bash
greptime cli meta snapshot save \
--store-addrs 'mysql://user:password@127.0.0.1:3306/database' \
--backend mysql-store
```
#### From etcd Backend
Export metadata from etcd to local directory:
```bash
greptime cli meta snapshot save \
--store-addrs 127.0.0.1:2379 \
--backend etcd-store
```
**Output**: Creates `metadata_snapshot.metadata.fb` file in the current working directory.
#### From RaftEngine Backend
:::note
RaftEngine locks the metadata directory while the standalone instance is running. Stop the standalone instance before exporting.
:::
Export metadata from RaftEngine to local directory:
```bash
greptime cli meta snapshot save \
--store-addrs "raftengine:///path/to/metadata" \
--backend raft-engine-store
```
**Output**: Creates `metadata_snapshot.metadata.fb` file in the current working directory.
## Import Operations
:::warning
**Important**: Before importing metadata, ensure the target backend is in a **clean state** (contains no existing data). Importing to a non-empty backend may result in data corruption or conflicts.
If the target backend is not empty and you do **not** pass `--force`, the command logs a warning and exits without restoring anything. It still returns a success exit code, so check the command output rather than assuming the restore ran.
`--force` only bypasses the non-empty check. It does not clear the target first: the restore writes the keys contained in the snapshot and leaves any other existing keys in place, which is how a restored cluster ends up mixing two generations of metadata. Prefer restoring into an empty backend, then restart the whole cluster and verify catalogs, table routes, and procedures.
:::
### Import from S3 Cloud Storage
Restore metadata from S3 backup to PostgreSQL storage backend:
```bash
greptime cli meta snapshot restore \
--store-addrs 'password=password dbname=postgres user=postgres host=localhost port=5432' \
--backend postgres-store \
--s3 \
--s3-bucket your-bucket-name \
--s3-region ap-southeast-1 \
--s3-access-key-id \
--s3-secret-access-key
```
### Import from Local File
#### To PostgreSQL Backend
Restore metadata from local backup file to PostgreSQL:
```bash
greptime cli meta snapshot restore \
--store-addrs 'password=password dbname=postgres user=postgres host=localhost port=5432' \
--backend postgres-store
```
#### To MySQL Backend
Restore metadata from local backup file to MySQL:
```bash
greptime cli meta snapshot restore \
--store-addrs 'mysql://user:password@127.0.0.1:3306/database' \
--backend mysql-store
```
#### To etcd Backend
Restore metadata from local backup file to etcd:
```bash
greptime cli meta snapshot restore \
--store-addrs 127.0.0.1:2379 \
--backend etcd-store
```
#### To RaftEngine Backend
Restore metadata from local backup file to RaftEngine:
```bash
greptime cli meta snapshot restore \
--store-addrs "raftengine:///path/to/metadata" \
--backend raft-engine-store
```
---
## DR Solution Based on Cross-Region Deployment in a Single Cluster
## How disaster recovery works in GreptimeDB
GreptimeDB is well-suited for cross-region disaster recovery. You may have varying regional characteristics and business needs, and GreptimeDB offers tailored solutions to meet these diverse requirements.
GreptimeDB resource management involves the concept of Availability Zones (AZs). An AZ is a logical unit of disaster recovery.
It can be a Data Center (DC), a compartment of a DC. This depends on your specific DC conditions and deployment design.
In the cross region disaster recovery solutions, a GreptimeDB region is a city. When two DC are in the same region and one DC becomes unavailable, the other DC can take over the services of the unavailable DC. This is a localization strategy.
Before understanding the details of each DR solution, it is necessary to first understand the following knowledge:
1. The DR solution for the remote wal component is also very important. Essentially, it forms the foundation of the entire DR solution. Therefore, for each DR solution of GreptimeDB, we will let the remote wal component in the diagram. Currently, GreptimeDB's default remote wal component is implemented based on Kafka, and other implementations will be provided in the future; however, there won't be significant differences in deployment.
2. The table of GreptimeDB: Each table can be divided into multiple partitions according to a certain range, and each partition may be distributed on different datanodes. When writing or querying, the specified node will be called according to the corresponding rules. A table's partitions might look like this:
```
Table name: T1
Table partition count: 4
T1-1
T1-2
T1-3
T1-4
Table name: T2
Table partition count: 3
T2-1
T2-2
T2-3
```
:::note
The latencies listed under each solution below are example network round-trip
values used to illustrate the topology, not measured GreptimeDB results. Measure
the round-trip latency between your own AZs, DCs, and regions and substitute
those numbers when comparing the solutions.
:::
### Metadata across 2 regions, data in the same region
In this solution, the data is in one region (2 DCs), while the metadata across 2 regions.
DC1 and DC2 are used together to handle read and write services, while DC3 (located in region2) is a replica used to meet the majority protocol. This architecture is also called the "2-2-1" solution.
Both DC1 and DC2 must be able to handle all requests in extreme situations, so please ensure that sufficient resources are allocated.
Latencies:
- 2ms latency in the same region
- 30ms latency in two regions
Supports High Availability:
- A single AZ is unavailable with the same performance
- A single DC is unavailable with almost the same performance
If you want a regional-level disaster recovery solution, you can take it a step further by providing read and write services on DC3. So, the next solution is:
### Data across 2 regions
In this solution, the data across 2 regions.
Each DC must be able to handle all requests in extreme situations, so please ensure that sufficient resources are allocated.
Latencies:
- 2ms latency in the same region
- 30ms latency in two regions
Supports High Availability:
- A single AZ is unavailable with the same performance
- A single DC is unavailable with degraded performance
If you can't tolerate performance degradation from a single DC failure, consider upgrading to the five-DC and three-region solution.
### Metadata across 3 regions, data across 2 regions
In this solution, the data across 2 regions, while the metadata across 3 regions.
Region1 and region2 are used together to handle read and write services, while region3 is a replica used to meet the majority protocol. This architecture is also called the "2-2-1" solution.
Each of the two adjacent regions must be able to handle all requests in extreme situations, so please ensure that sufficient resources are allocated.
Latencies:
- 2ms latency in the same region
- 7ms latency in two adjacent regions
- 30ms latency in two distant regions
Supports High Availability:
- A single AZ is unavailable with the same performance
- A single DC is unavailable with the same performance
- A single region(city) is unavailable with slightly degraded performance
You can take it a step further by providing read and write services on both 3 regions. So, the next solution is:
(This solution may have higher latency, so if that's unacceptable, it's not recommended.)
### Data across 3 regions
In this solution, the data across 3 regions.
In the event of a failure in one region, the other two regions must be able to handle all requests, so please ensure sufficient resources are allocated.
Latencies:
- 2ms latency in the same region
- 7ms latency in two adjacent regions
- 30ms latency in two distant regions
Supports High Availability:
- A single AZ is unavailable with the same performance
- A single DC is unavailable with the same performance
- A single region(city) is unavailable with degraded performance
## Solution Comparison
The goal of the above solutions is to meet the high requirements for availability and reliability in medium to large-scale scenarios. However, in specific implementations, the cost and effectiveness of each solution may vary. The table below compares each solution to help you choose the final plan based on your specific scenario, needs, and costs.
Here is the content formatted into a table:
| Solution | Latencies | High Availability |
| --- | --- | --- |
| Metadata across 2 regions, data in the same region | - 2ms latency in the same region- 30ms latency in two regions | - A single AZ is unavailable with the same performance- A single DC is unavailable with almost the same performance |
| Data across 2 regions | - 2ms latency in the same region- 30ms latency in two regions | - A single AZ is unavailable with the same performance- A single DC is unavailable with degraded performance |
| Metadata across 3 regions, data across 2 regions | - 2ms latency in the same region- 7ms latency in two adjacent regions- 30ms latency in two distant regions | - A single AZ is unavailable with the same performance- A single DC is unavailable with the same performance- A single region(city) is unavailable with slightly degraded performance |
| Data across 3 regions | - 2ms latency in the same region- 7ms latency in two adjacent regions- 30ms latency in two distant regions | - A single AZ is unavailable with the same performance - A single DC is unavailable with the same performance- A single region(city) is unavailable with degraded performance |
---
## DR solution for GreptimeDB Standalone
---
## Export and Import Data with Export/Import V2
Export/Import V2 creates portable snapshots of GreptimeDB data. Use it to back up data, restore data, or migrate data between GreptimeDB instances.
A V2 snapshot contains schema metadata, a manifest, and data files. Data files are split into chunks so interrupted export and import jobs can resume from the previous progress.
This guide uses the following commands:
- `greptime cli data export-v2 create` to create a snapshot
- `greptime cli data export-v2 verify` to verify a snapshot
- `greptime cli data export-v2 list` to list snapshots under a parent location
- `greptime cli data export-v2 delete` to delete a snapshot
- `greptime cli data import-v2` to import from a snapshot
## Prerequisites
Before using Export/Import V2, make sure that:
- The GreptimeDB HTTP endpoint is reachable, for example `127.0.0.1:4000`.
- You have a `greptime` binary that includes the `cli data export-v2` and `cli data import-v2` commands.
- The snapshot storage location is readable and writable by both the CLI client and the GreptimeDB server.
For remote object storage, explicitly enable one supported backend and provide the backend options. Export/Import V2 supports S3-compatible storage, Alibaba Cloud OSS, Google Cloud Storage, and Azure Blob Storage. For example, use `--s3` with `--s3-region` for S3-compatible storage.
For `file://` snapshots, the path must be accessible from the GreptimeDB server as well as the CLI client. This usually means running the CLI on the same host as a standalone server, or mounting the same filesystem path into the GreptimeDB server. The parent directory of the snapshot must exist before you create the snapshot. For remote, Kubernetes, or distributed deployments, use object storage such as S3 or MinIO instead of a local `file://` path.
The GreptimeDB server writes snapshot data files with `COPY DATABASE ... TO`, so a `file://` snapshot path must also be inside the server's copy root, which is `storage.copy_root` and defaults to `/copy`. The server rejects a path outside the copy root. Object storage snapshots are not subject to the copy root, because the server writes them directly to the object store. See [Migrate Local SQL File Access](/user-guide/deployments-administration/migrate-local-sql-file-access.md) for details.
:::note
The snapshot URI, such as `s3://my-bucket/snapshots/prod`, identifies the snapshot location, including the bucket. The object store options, such as `--s3-region` and `--s3-endpoint`, configure how the CLI connects to the backend. `--s3-bucket` and `--s3-root` do not change where the snapshot is read or written; the URI is authoritative. Pass the same connection options to create, verify, import, list, and delete commands.
:::
## Quick start with local filesystem
Use a local filesystem snapshot only when the path is shared by the CLI client and the GreptimeDB server. For example, this works for a local standalone server when the CLI runs on the same host.
The examples in this guide store snapshots under `/tmp/greptime-snapshots`. Set this directory as the server's copy root in the GreptimeDB configuration file and restart the server:
```toml
[storage]
copy_root = "/tmp/greptime-snapshots"
```
Alternatively, keep the default copy root and use a snapshot path under `/copy`, such as `file:///path/to/data_home/copy/demo`.
Create the parent directory so that both the server and the CLI can access it:
```bash
mkdir -p /tmp/greptime-snapshots
```
Create a snapshot:
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/demo
```
Verify the snapshot before restoring it:
```bash
greptime cli data export-v2 verify \
--snapshot file:///tmp/greptime-snapshots/demo
```
Import from the snapshot:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/demo
```
By default, export and import use the `greptime` catalog. Add `--catalog ` if you need another catalog.
## Export to remote object storage
Export/Import V2 can store snapshots in AWS S3, S3-compatible services such as MinIO, Alibaba Cloud OSS, Google Cloud Storage, and Azure Blob Storage. The examples below use the S3 backend because S3 and MinIO are common choices. For other object stores, use the corresponding backend flag and options described in [Supported storage backends](#supported-storage-backends).
### S3 or MinIO
Use the S3 backend for AWS S3 and S3-compatible services such as MinIO.
### S3-compatible endpoint
For MinIO, pass `--s3-endpoint`:
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to s3://greptime/snapshots/local-test \
--s3 \
--s3-bucket greptime \
--s3-region us-west-2 \
--s3-access-key-id superpower_ci_user \
--s3-secret-access-key superpower_password \
--s3-endpoint http://127.0.0.1:9000
```
Use the same storage options when verifying, importing, listing, or deleting the snapshot:
```bash
greptime cli data export-v2 verify \
--snapshot s3://greptime/snapshots/local-test \
--s3 \
--s3-bucket greptime \
--s3-region us-west-2 \
--s3-access-key-id superpower_ci_user \
--s3-secret-access-key superpower_password \
--s3-endpoint http://127.0.0.1:9000
```
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from s3://greptime/snapshots/local-test \
--s3 \
--s3-bucket greptime \
--s3-region us-west-2 \
--s3-access-key-id superpower_ci_user \
--s3-secret-access-key superpower_password \
--s3-endpoint http://127.0.0.1:9000
```
For AWS S3, use the same `--s3` options but omit `--s3-endpoint` unless you use a custom endpoint. If your environment uses instance profiles or another credential provider, explicit access keys may not be required. The S3 backend still requires `--s3` and `--s3-region`.
## Export selected schemas
By default, `export-v2 create` exports every schema except `information_schema`. This includes the `greptime_private` schema, and importing such a snapshot restores `greptime_private` as well. To exclude it, list the schemas you want explicitly with `--schemas`.
Export only selected schemas with `--schemas`. You can pass a comma-separated list:
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/observability \
--schemas public,metrics
```
Or pass the option multiple times:
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/observability \
--schemas public \
--schemas metrics
```
When importing, `--schemas` selects a subset from the snapshot:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/observability \
--schemas public
```
If you request a schema that is not in the snapshot, the import fails.
## Export schema only
Use `--schema-only` to export table definitions without data:
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/schema-only \
--schema-only
```
Importing a schema-only snapshot restores the schema but does not import data chunks:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/schema-only
```
## Choose the data file format
Use `--format` to choose the data file format. Supported values are `parquet`, `csv`, and `json`. The default is `parquet`.
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/demo-csv \
--format csv
```
The format is recorded in the snapshot manifest. `import-v2` reads the format from the manifest and applies it automatically, so you do not need to specify the format again when importing.
## Export by time range and chunk window
Use a bounded time range and a chunk window for large datasets. Export/Import V2 creates one or more data chunks for each schema in the range.
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to s3://greptime/snapshots/prod-2026-06 \
--start-time 2026-06-01T00:00:00Z \
--end-time 2026-07-01T00:00:00Z \
--chunk-time-window 1d \
--s3 \
--s3-bucket greptime \
--s3-region us-west-2 \
--s3-access-key-id superpower_ci_user \
--s3-secret-access-key superpower_password \
--s3-endpoint http://127.0.0.1:9000
```
`--chunk-time-window` requires both `--start-time` and `--end-time`.
Chunks are created by time window. For example, a 15-minute range with `--chunk-time-window 5m` creates 3 chunks. During import, data tasks are scheduled per chunk and per schema. If the snapshot contains 3 chunks and 2 schemas with data, import schedules 6 data tasks.
Snapshot data files are also organized by schema and chunk. For example, a multi-chunk snapshot can contain paths such as:
```text
data/logs/1/app_logs.parquet
data/logs/2/app_logs.parquet
data/logs/3/app_logs.parquet
data/public/1/host_metrics.parquet
data/public/2/host_metrics.parquet
data/public/3/host_metrics.parquet
```
Choose the chunk window based on the data volume and density:
- Use a larger window for small or sparse datasets to avoid too many small chunks.
- Use a smaller window for large datasets to keep each chunk manageable.
- Start with `1d` and adjust after checking export duration and chunk sizes.
## Tune parallelism
Export has two parallelism options:
- `--parallelism` controls server-side `COPY DATABASE` parallelism per schema per chunk. The default is `1`.
- `--chunk-parallelism` controls how many export chunks the client runs concurrently. The default is `1` and the valid range is `1..=64`.
Import uses:
- `--task-parallelism` to control how many import data tasks the client runs concurrently. The default is `1` and the valid range is `1..=64`.
Increase these values gradually and monitor GreptimeDB, object storage, and network resource usage.
## Resume an interrupted export or import
Export/Import V2 is designed to resume interrupted work.
For export:
- If the target snapshot does not exist, `export-v2 create` creates it.
- If the target snapshot already exists, the same command resumes from existing progress.
- Completed chunks are skipped. Failed or pending chunks are retried.
- Use `--force` only when you want to delete the existing snapshot and recreate it.
```bash
greptime cli data export-v2 create \
--addr 127.0.0.1:4000 \
--to file:///tmp/greptime-snapshots/demo
```
Run the same command again to resume after interruption.
For import, progress is stored in a state file while the import is running. By default, the state file is under `~/.greptime/import_state`. Override it with `--state-path` when you want an explicit state location:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/demo \
--state-path /tmp/greptime-import-demo.state
```
`--dry-run` does not create an import state file. A real import keeps the state file if it fails or is interrupted, so the next run can resume completed data tasks. After a successful import, import-v2 automatically deletes the state file because there is no remaining work to resume. A `.lock` file with the same prefix can remain; it is a lock file, not the import state content.
If you intentionally want to rerun an import from scratch after a failed or interrupted run, use a different `--state-path` or remove the previous state file after confirming that doing so is safe.
## Dry-run an import
Use `--dry-run` to validate the snapshot and import plan without executing DDL or importing data:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/demo \
--dry-run
```
Dry-run is useful before a production migration or restore.
## Verify, list, and delete snapshots
Verify a snapshot:
```bash
greptime cli data export-v2 verify \
--snapshot file:///tmp/greptime-snapshots/demo
```
List snapshots under a parent location:
```bash
greptime cli data export-v2 list \
--location file:///tmp/greptime-snapshots
```
Delete a snapshot:
```bash
greptime cli data export-v2 delete \
--snapshot file:///tmp/greptime-snapshots/demo
```
For non-interactive deletion, pass `--no-confirm`:
```bash
greptime cli data export-v2 delete \
--snapshot s3://greptime/snapshots/local-test \
--no-confirm \
--s3 \
--s3-bucket greptime \
--s3-region us-west-2 \
--s3-access-key-id superpower_ci_user \
--s3-secret-access-key superpower_password \
--s3-endpoint http://127.0.0.1:9000
```
## Authentication, timeout, proxy, and progress
If the GreptimeDB instance has basic authentication enabled, pass credentials with `--auth-basic :`.
Set request timeout with `--timeout`, for example `--timeout 60s` or `--timeout 5m`.
For commands that connect to the GreptimeDB HTTP endpoint, such as `export-v2 create` and `import-v2`, use `--proxy` to set an HTTP proxy, or `--no-proxy` to disable proxy usage:
```bash
greptime cli data import-v2 \
--addr 127.0.0.1:4000 \
--from file:///tmp/greptime-snapshots/demo \
--no-proxy
```
Long-running create and import commands support `--progress` to control progress output:
- `auto`: show an interactive bar on a TTY, otherwise log progress. This is the default.
- `always`: always emit progress; it uses a bar on a TTY and lightweight logs otherwise.
- `never`: never emit progress.
## Supported storage backends
Export/Import V2 supports local filesystem snapshots and remote object storage snapshots.
For remote object storage, enable exactly one backend:
- `--s3` for AWS S3 or S3-compatible storage such as MinIO
- `--oss` for Alibaba Cloud OSS
- `--gcs` for Google Cloud Storage
- `--azblob` for Azure Blob Storage
Common S3 options are:
| Option | Description |
| --- | --- |
| `--s3-bucket` | S3 bucket name. Does not affect the snapshot location; the bucket in the snapshot URI is used. |
| `--s3-region` | S3 region. Required for S3. |
| `--s3-access-key-id` | Access key ID. Optional when the environment provides credentials. |
| `--s3-secret-access-key` | Secret access key. Optional when the environment provides credentials. |
| `--s3-endpoint` | Custom S3-compatible endpoint, such as a MinIO endpoint. |
| `--s3-enable-virtual-host-style` | Enable virtual-hosted-style requests. |
| `--s3-disable-ec2-metadata` | Disable EC2 metadata service credential lookup. |
## Troubleshooting
### Failed to connect to `/v1/sql`
Check that GreptimeDB is running and that `--addr` points to the HTTP endpoint:
```bash
curl http://127.0.0.1:4000/health
```
### `file://` path is outside the configured copy root
If `export-v2 create` fails with an error such as the following, the snapshot path is outside the server's copy root:
```text
Local filesystem path '/tmp/greptime-snapshots/demo/data/greptime_private/1/' is outside the configured copy root or is unsafe
```
Set `storage.copy_root` to a directory that contains the snapshot path, or use a snapshot path under `/copy`. See [Migrate Local SQL File Access](/user-guide/deployments-administration/migrate-local-sql-file-access.md) for details.
The failed run leaves a partial snapshot on disk, and `export-v2 verify` reports the failed chunk. After fixing the configuration, run the same `export-v2 create` command again to resume, or pass `--force` to recreate the snapshot.
### `--chunk-time-window` fails validation
`--chunk-time-window` requires both `--start-time` and `--end-time`. Add a bounded time range or remove `--chunk-time-window`.
### S3 or MinIO reports `NoSuchBucket` or `NoSuchKey`
Check that:
- The bucket and path in the snapshot URI are correct. `--s3-bucket` does not change the snapshot location.
- `--s3-endpoint` is set for MinIO or other S3-compatible services. Without it, the CLI sends requests to the public AWS S3 endpoint for `--s3-region`, which usually shows up as a long hang or a `301` response rather than a clear configuration error.
- The credentials can read and write the snapshot location.
- You pass the same storage options to create, verify, import, list, and delete commands.
### Import skips data unexpectedly
Import resumes from its state file. If a previous import marked tasks as completed, a later run with the same state identity may skip them. Use an explicit `--state-path` for each restore operation, or remove the old state file after confirming it is safe.
### A requested schema is not found
When using `--schemas` during import, make sure the schema exists in the snapshot. Use `export-v2 verify` first and check the snapshot contents.
### Verification fails
Do not import a snapshot that fails verification. The snapshot may be incomplete or corrupted. Re-run the export with the same command to resume missing chunks, or recreate the snapshot with `--force` if you intentionally want to discard the existing snapshot.
## Limitations
- Export/Import V2 snapshots are not the same format as the legacy export/import output.
- Schema filtering is schema-level. Table-level filtering is not covered by this command.
- Time-range chunking is designed for time-series data. Sparse data may need a larger chunk window.
- Schema-only snapshots do not contain data files.
- Large migrations require careful tuning of chunk window, parallelism, object storage throughput, and GreptimeDB resources.
---
## Disaster Recovery
GreptimeDB is a distributed database designed to withstand disasters. It provides different solutions for disaster recovery (DR).
This document contains:
* Basic concepts in DR.
* The deployment architecture of GreptimeDB and Backup & Restore (BR).
* GreptimeDB provides the DR solutions.
* Compares these DR solutions.
## Basic Concepts
* **Recovery Time Objective (RTO)**: refers to the maximum acceptable amount of time that a business process can be down after a disaster occurs before it negatively impacts the organization.
* **Recovery Point Objective (RPO)**: refers to the maximum acceptable amount of time since the last data recovery point. This determines what is considered an acceptable loss of data between the last recovery point and the interruption of service.
The following figure illustrates these two concepts:
* **Write-Ahead Logging (WAL)**: persistently records every data modification to ensure data integrity and consistency.
GreptimeDB storage engine is a typical [LSM Tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree) :
The data written is going firstly persisted into WAL, then applied into Memtable in memory. Under specific conditions (e.g., exceeding the memory threshold), the Memtable will be flushed and persisted as an SSTable. So the DR of WAL and SSTable is key to the DR of GreptimeDB.
* **Region**: a contiguous segment of a table, and also could be regarded as a partition in some relational databases. Read [Table Sharding](/contributor-guide/frontend/table-sharding.md#region) for more details.
## Component architecture
### GreptimeDB
Before digging into the specific DR solution, let's explain the architecture of GreptimeDB components in the perspective of DR:
GreptimeDB is designed with a cloud-native architecture based on storage-compute separation:
* **Frontend**: the ingestion and query service layer, which forwards requests to Datanode and processes, and merges responses from Datanode.
* **Datanode**: the storage layer of GreptimeDB, and is an LSM storage engine. Region is the basic unit for storing and scheduling data in Datanode. A region is a table partition, a collection of data rows. The data in region is saved into Object Storage (such as AWS S3). Unflushed Memtable data is written into WAL and can be recovered in DR.
* **WAL**: persists the unflushed Memtable data in memory. It will be truncated when the Memtable is flushed into SSTable files. It can be local disk-based (local WAL) or Kafka cluster-based (remote WAL).
* **Object Storage**: persists the SSTable data and index.
The GreptimeDB stores data in object storage such as [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) or its compatible services, which is designed to provide 99.999999999% durability and 99.99% availability of objects over a given year. And services such as S3 provide [replications in Single-Region or Cross-Region](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html), which is naturally capable of DR.
At the same time, the WAL component is pluggable, e.g. using Kafka as the WAL service that offers a mature [DR solution](https://www.confluent.io/blog/disaster-recovery-multi-datacenter-apache-kafka-deployments/).
### Backup and restore
The Backup & Restore (BR) tool can perform a full snapshot backup of databases or tables at a specific time and supports incremental backup.
When a cluster encounters a disaster, you can restore the cluster from backup data. Generally speaking, BR is the last resort for disaster recovery.
## Solutions introduction
### DR solution for GreptimeDB Standalone
If the Standalone is running on the local disk for WAL and data, then:
* RPO: depends on backup frequency.
* RTO: doesn't make sense in standalone mode, mostly depends on the size of the data to be restored, your failure response time, and the operational infrastructure.
A good start is to deploy GreptimeDB Standalone into an IaaS platform that has a backup and recovery solution. For example, Amazon EC2 with EBS volumes provides a comprehensive [Backup and Recovery solution](https://docs.aws.amazon.com/prescriptive-guidance/latest/backup-recovery/backup-recovery-ec2-ebs.html).
But if running the Standalone with remote WAL and object storage, there is a better DR solution:
Write the WAL to the Kafka cluster and store the data in object storage, so that the ingested data no longer depends on the node's local disk.
The node is not fully stateless, though: a standalone instance keeps its metadata — catalogs, schemas and table definitions — in a local key-value store under `/metadata` (see [Storage Location](/user-guide/concepts/storage-location.md)). Kafka and object storage cannot rebuild it. Losing the host together with its disk means losing that metadata unless you have backed it up separately, so include it in the plan with [Metadata Export & Import](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md).
**RPO=0** and an **RTO in minutes** are the design targets of this topology. They hold as long as the Kafka cluster and the object storage both survive the failure you are planning for, the WAL covering unflushed writes is still present, and the metadata can be restored. Verify them with a failure drill against your own deployment.
### DR solution based on single-region deployment in a single cluster
Before a cluster spans regions, it has to survive the loss of a single node or of a single AZ inside one region. An AZ here is the same logical unit of disaster recovery used in the cross-region solutions: a data center, or a compartment of a data center. This topology is the common production baseline, and the cross-region solutions below are built on top of it.
The cluster lives in one region, with its roles spread over the AZs that region offers:
* **Frontend** is stateless. Run several replicas behind a load balancer so that a failed replica only affects the requests in flight on it. See [Role Replicas Configuration](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#role-replicas-configuration).
* **Metasrv** runs as several replicas that elect a leader; losing a follower is transparent, and losing the leader costs a re-election. The metadata itself lives in an external backend — etcd, MySQL or PostgreSQL — that Metasrv does not replicate for you, so give that backend its own high availability and its own backups. See [Metadata Management](/user-guide/deployments-administration/manage-metadata/overview.md) and [Metadata Export & Import](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md).
* **Datanode** holds the regions. When one fails, [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) reopens its regions on the surviving Datanodes. It is **disabled by default**, and it requires [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) together with remote WAL. Enabling it on local WAL with `allow_region_failover_on_local_wal=true` is possible but may lose data, because the WAL of the failed node stays on that node's disk.
* **Kafka** (remote WAL) and the **object storage** hold the state that has to outlive the node. Give Kafka a replication factor that tolerates the broker loss you are planning for, and spread brokers and Datanodes over the AZs instead of packing them into one.
Latencies:
- Same-region round trips only; neither writes nor replication pay a cross-region penalty
Supports High Availability:
- A single node is unavailable with almost the same performance, once its regions are reopened elsewhere
- A single AZ is unavailable with degraded performance, unless the surviving AZs were sized to absorb its share of the load
- The region itself is not covered; it is a single failure domain in this topology
This solution targets zero RPO and a second- or minute-level RTO for a node or AZ failure. As with the other solutions, the numbers depend on conditions you have to check:
- Region Failover is **disabled by default** and must be enabled explicitly.
- The surviving Datanodes need spare capacity to carry the failed node's regions, otherwise failover only moves the overload.
- The recovery time is dominated by WAL replay: the more regions share one Kafka topic, the more redundant data has to be read before those regions serve again. Read [The recovery time of Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md#the-recovery-time-of-region-failover) for the model behind that.
Confirm the resulting RPO and RTO with a failure drill. If the region as a whole, the Kafka cluster or the object storage becomes unavailable, you need one of the solutions below, or regular backups kept in another region.
### DR solution based on Active-Active Failover
In some edge or small-to-medium scale scenarios, or if you lack the resources to deploy remote WAL or object storage, Active-Active Failover offers a better solution compared to Standalone DR. Two actively serving standalone nodes replicate data changes asynchronously. If a peer or the inter-site network fails, the healthy node continues serving, retains pending changes on its local storage, and sends them after the peer recovers.
Pending changes remain recoverable only while the source node's local storage is available. Losing a node and its local storage before replication completes can leave the peer without those changes.
Deploying nodes in different regions can also meet region-level DR requirements, but the scalability is limited.
:::tip NOTE
**Active-Active Failover is only available in GreptimeDB Enterprise.**
:::
For more information about this solution, see [DR solution based on Active-Active Failover](/enterprise/deployments-administration/disaster-recovery/dr-solution-based-on-active-active-failover.md).
### DR solution based on cross-region deployment in a single cluster
For medium-to-large scale scenarios requiring zero RPO, this solution is highly recommended. In this deployment architecture, the entire cluster spans across three regions, with each region capable of handling both read and write requests. Data replication is achieved using remote WAL and object storage, both of which must have cross-region DR enabled.
If Region 1 becomes completely unavailable due to a disaster, the table regions within it will be opened and recovered in the other regions.
In the event that Region 1 becomes completely unavailable due to a disaster, the table regions within it will be opened and recovered in the other regions. Region 3 serves as a replica to adhere to the majority protocol of Metasrv.
This solution targets region-level fault tolerance, scalable write capability, zero RPO, and a second- or minute-level RTO. Reaching those numbers depends on the whole dependency chain, not on the cluster layout alone:
- Region Failover is **disabled by default** and must be enabled explicitly.
- Kafka, the object storage, the metadata backend and the traffic entry point all have to span the failure domain you are protecting against. Metasrv running in three regions does not replicate an external MySQL or PostgreSQL metadata backend for you.
- The automatic Datanode selector picks targets by round-robin, lease or load. It is not an availability-zone-aware placement policy, so surviving regions need enough healthy capacity to take over.
Confirm the resulting RPO and RTO with an end-to-end failure drill. For more information about this solution, see [DR solution based on cross-region deployment in a single cluster](./dr-solution-based-on-cross-region-deployment-in-single-cluster.md).
### DR solution based on BR
In this architecture, GreptimeDB Cluster 1 is deployed in region 1. The BR process continuously and regularly backs up the data from Cluster 1 to region 2. If region 1 experiences a disaster rendering Cluster 1 unrecoverable, you can use the backup data to restore a new cluster (Cluster 2) in region 2 to resume services.
The DR solution based on BR provides an RPO depending on the backup frequency and an RTO that varies with the size of the data to be restored.
Read [Backup & restore data](./back-up-&-restore-data.md) for details.
### Solution Comparison
By comparing these DR solutions, you can decide on the final option based on their specific scenarios, requirements, and cost. The RPO and RTO columns are the design targets of each topology under the conditions described above, not values guaranteed for every deployment; confirm them with a failure drill.
| DR solution | Error Tolerance Objective | RPO | RTO | TCO | Scenarios | Remote WAL & Object Storage | Notes |
| ------------- | ------------------------- | ----- | ----- | ----- | ---------------- | --------- | --------|
| DR solution for Standalone| Single-Region | Backup Interval | Minute or Hour level | Low | Low requirements for availability and reliability in small scenarios | Optional | |
| DR solution based on single-region deployment in a single cluster | Single-Region, node and AZ level | 0 | Second or minute level | Medium | The common production baseline for a cluster that lives in one region | Required | Region Failover is disabled by default |
| DR solution based on Active-Active Failover | Cross-Region | Depends on pending changes and the failure mode | Depends on external failover | Low | High requirements for availability and reliability in small-to-medium scenarios | Optional | Commercial feature |
| DR solution based on cross-region deployment in a single cluster| Multi-Regions | 0 | Second or minute level | High | High requirements for availability and reliability in medium-to-large scenarios | Required | |
| DR solution based on BR | Single-Region | Backup Interval | Minute or Hour level | Low | Acceptable requirements for availability and reliability | Optional | |
## References
* [Backup & restore data](./back-up-&-restore-data.md)
* [DR solution based on Active-Active Failover ](/enterprise/deployments-administration/disaster-recovery/dr-solution-based-on-active-active-failover.md)
* [DR solution based on cross-region deployment in a single cluster](./dr-solution-based-on-cross-region-deployment-in-single-cluster.md)
* [S3 Replicating objects overview](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html)
---
## Cluster Maintenance Mode
Maintenance mode is a safety feature in GreptimeDB that temporarily disables automatic cluster management operations.
This mode is particularly useful during:
- Cluster deployment
- Cluster upgrades
- Planned downtime
- Any operation that might temporarily affect cluster stability
## When to Use Maintenance Mode
### With GreptimeDB Operator
If you are upgrading a cluster using GreptimeDB Operator, you don't need to enable the maintenance mode manually. The operator handles this automatically.
### Without GreptimeDB Operator
When upgrading a cluster without using GreptimeDB Operator, **you must manually enable Metasrv's maintenance mode before**:
1. Deploying a new cluster (maintenance mode should be enabled after metasrv nodes are ready)
2. Rolling upgrades of Datanode nodes
3. Metasrv nodes upgrades
4. Frontend nodes upgrades
5. Any operation that might cause temporary node unavailability
After the cluster is deployed/upgraded, you can disable the maintenance mode.
## Impact of Maintenance Mode
When maintenance mode is enabled:
- Auto Balancing (if enabled) will be paused
- Region Failover (if enabled) will be paused
- Manual region operations are still possible
- Read and write operations continue to work normally
- Monitoring and metrics collection continue to function
## Managing Maintenance Mode
The maintenance mode can be enabled and disabled through Metasrv's HTTP interface at: `http://{METASRV}:{HTTP_PORT}/admin/maintenance/enable` and `http://{METASRV}:{HTTP_PORT}/admin/maintenance/disable`. Note that this interface listens on Metasrv's `HTTP_PORT`, which defaults to `4000`.
### Enable Maintenance Mode
:::danger
After calling the maintenance mode interface,
ensure you check that the HTTP status code returned is 200 and confirm that the response content meets expectations.
If there are any exceptions or the interface behavior does not meet expectations,
proceed with caution and avoid continuing with high-risk operations such as cluster upgrades.
:::
Enable maintenance mode by sending a POST request to the `/admin/maintenance/enable` endpoint.
```bash
curl -X POST 'http://localhost:4000/admin/maintenance/enable'
```
The expected output is:
```bash
{"enabled":true}
```
If you encounter any issues or unexpected behavior, do not proceed with maintenance operations.
### Disable Maintenance Mode
:::danger
Before disabling maintenance mode,
you must confirm that **all components have returned to normal status**.
:::
Disable maintenance mode by sending a POST request to the `/admin/maintenance/disable` endpoint.
Before disabling maintenance mode:
1. Ensure all components are healthy and operational
2. Verify that all nodes are properly joined to the cluster
```bash
curl -X POST 'http://localhost:4000/admin/maintenance/disable'
```
The expected output is:
```bash
{"enabled":false}
```
### Check Maintenance Mode Status
Check maintenance mode status by sending a GET request to the `/admin/maintenance/status` endpoint.
```bash
curl -X GET http://localhost:4000/admin/maintenance/status
```
The expected output is:
```bash
{"enabled":false}
```
## Troubleshooting
### Common Issues
1. **Maintenance mode cannot be enabled**
- Verify Metasrv is running and accessible
- Check if you have the correct permissions
- Ensure the HTTP port is correct
### Best Practices
1. Always verify the maintenance mode status before and after operations
2. Have a rollback plan ready
3. Monitor cluster health during maintenance
4. Document all changes made during maintenance
---
## Prevent Metadata Changes
To prevent metadata changes, you can pause the Procedure Manager. This mechanism rejects all new procedures (i.e., new metadata-changing operations) while allowing existing procedures to continue running.
Once enabled, the Metasrv will reject the following procedures (including but not limited to):
**DDL procedures:**
- Create table
- Drop table
- Alter table
- Create database
- Drop database
- Create view
- Create flow
- Drop flow
**Region procedures:**
- Region Migration
- Region Failover (if enabled)
- Auto Balancing (if enabled)
You may see error messages if you or Metasrv try to perform these procedures after metadata changes are paused. For region procedures, you can enable [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) to temporarily disable them.
## Managing Procedure Manager
The Procedure Manager can be paused and resumed through Metasrv's HTTP interface at: `http://{METASRV}:{HTTP_PORT}/admin/procedure-manager/pause` and `http://{METASRV}:{HTTP_PORT}/admin/procedure-manager/resume`. Note that this interface listens on Metasrv's `HTTP_PORT`, which defaults to `4000`.
### Pause Procedure Manager
Pause Procedure Manager by sending a POST request to the `/admin/procedure-manager/pause` endpoint.
```bash
curl -X POST 'http://localhost:4000/admin/procedure-manager/pause'
```
The expected output is:
```bash
{"status":"paused"}
```
### Resume Procedure Manager
Resume Procedure Manager by sending a POST request to the `/admin/procedure-manager/resume` endpoint.
```bash
curl -X POST 'http://localhost:4000/admin/procedure-manager/resume'
```
The expected output is:
```bash
{"status":"running"}
```
### Check Procedure Manager Status
Check Procedure Manager status by sending a GET request to the `/admin/procedure-manager/status` endpoint.
```bash
curl -X GET 'http://localhost:4000/admin/procedure-manager/status'
```
The expected output is:
```bash
{"status":"running"}
```
---
## Cluster Recovery Mode
Recovery mode is a safety feature in GreptimeDB that allows developers to manually recover the cluster from a failed state.
## When to Use Recovery Mode
Recovery mode is particularly useful when the Datanode fails to start due to an "Empty region directory" error, often caused by:
- Data corruption (Missing region data directory)
- Recover the cluster from a metadata snapshot.
## Recovery Mode Management
Recovery mode can be enabled and disabled through Metasrv's HTTP interface at: `http://{METASRV}:{HTTP_PORT}/admin/recovery/enable` and `http://{METASRV}:{HTTP_PORT}/admin/recovery/disable`. Note that this interface listens on Metasrv's `HTTP_PORT`, which defaults to `4000`.
### Enable Recovery Mode
Enable recovery mode by sending a POST request to the `/admin/recovery/enable` endpoint.
```bash
curl -X POST 'http://localhost:4000/admin/recovery/enable'
```
The expected output is:
```bash
{"enabled":true}
```
### Disable Recovery Mode
Disable recovery mode by sending a POST request to the `/admin/recovery/disable` endpoint.
```bash
curl -X POST 'http://localhost:4000/admin/recovery/disable'
```
The expected output is:
```bash
{"enabled":false}
```
### Check Recovery Mode Status
Check recovery mode status by sending a GET request to the `/admin/recovery/status` endpoint.
```bash
curl -X GET 'http://localhost:4000/admin/recovery/status'
```
The expected output is:
```bash
{"enabled":false}
```
---
## Resource ID Management
Resource ID management is primarily used to manually set resource identifiers (such as table IDs) when restoring a cluster from a [Metadata Backup](/user-guide/deployments-administration/manage-metadata/restore-backup.md). This is because the backup file may not contain the latest `next table ID` value, which could lead to resource conflicts or data inconsistencies if not properly reset.
### Understanding the Relationship Between Table ID and Next Table ID
In GreptimeDB:
- **Table ID**: Each table has a unique numeric identifier used internally by the database to identify and manage tables
- **Next Table ID**: The system-reserved next available table ID value. When creating a new table, the system automatically assigns this ID to the new table and then increments the `next table ID`
**Example:**
- Suppose the current cluster has tables with IDs 1001, 1002, and 1003
- The `next table ID` should be 1004
- When creating a new table, the system assigns ID 1004 to the new table and updates the `next table ID` to 1005
- If during backup restoration, the `next table ID` is still 1002, it would conflict with existing table IDs 1002 and 1003 (you would encounter the error `Region 1024 is corrupted, reason: ` when starting the Datanode)
Under normal circumstances, resource IDs are automatically maintained by the database and require no manual intervention. However, in certain special scenarios (such as restoring a cluster from metadata backup where new tables were created after the backup), the `next table ID` in the backup may lag behind the actual cluster state, requiring manual adjustment.
**How to determine if manual `next table ID` setting is needed:**
1. Query all table IDs in the current cluster: `SELECT TABLE_ID FROM INFORMATION_SCHEMA.TABLES ORDER BY TABLE_ID DESC LIMIT 1;`
2. Get the current `next table ID` via API (see interface description below)
3. If the maximum existing table ID is greater than or equal to the current `next table ID`, you need to manually set the `next table ID` to a larger value, typically the maximum existing table ID plus 1.
You can get or set the `next table ID` using Metasrv's HTTP interface at the following endpoints: `http://{METASRV}:{HTTP_PORT}/admin/sequence/table/next-id` (to get) and `http://{METASRV}:{HTTP_PORT}/admin/sequence/table/set-next-id` (to set). This interface listens on Metasrv's `HTTP_PORT`, which defaults to `4000`.
### Set next table ID
To safely update the `next table ID`, follow this step-by-step process:
1. **Block new procedure submissions** - Pause the Procedure Manager. This rejects newly submitted procedures, including table creation. It does not suspend or cancel procedures that were already accepted, and their child procedures keep running. See [Prevent Metadata Changes](/user-guide/deployments-administration/maintenance/prevent-metadata-changes.md).
2. **Wait for in-flight procedures to finish** - Poll until no procedure is still in progress:
```sql
SELECT procedure_id, procedure_type, status
FROM INFORMATION_SCHEMA.PROCEDURE_INFO
WHERE status IN ('Running', 'Retrying', 'PrepareRollback', 'RollingBack');
```
Proceed only when this returns no rows. Terminal states such as `Done`, `Failed` and `Poisoned` are not included: their runners have already exited and released their locks, and the rows stay visible until background cleanup removes them. If procedures keep running past the window you allotted, stop here and investigate instead of changing the sequence. See [PROCEDURE_INFO](/reference/sql/information-schema/procedure-info.md).
3. **Enable cluster recovery mode** - Setting the `next table ID` is only allowed in recovery mode. See [Cluster Recovery Mode](/user-guide/deployments-administration/maintenance/recovery-mode.md) for more details.
4. **Set the next table ID** - Use the HTTP interface to set the `next table ID`.
5. **Restart metasrv nodes** - This ensures the new `next table ID` is properly applied. After the restart, confirm that recovery mode is still enabled and that the Procedure Manager is still paused, then verify the new `next table ID`.
6. **Disable cluster recovery mode** - Then resume the Procedure Manager to return to normal cluster operations.
:::warning
Recovery mode does not block DDL on its own. It only gates the `set-next-id` endpoint and relaxes Datanode startup checks. Pausing the Procedure Manager is what stops new metadata changes.
:::
Set the `next table ID` by sending a POST request to the `/admin/sequence/table/set-next-id` endpoint:
```bash
curl -X POST 'http://localhost:4000/admin/sequence/table/set-next-id' \
-H 'Content-Type: application/json' \
-d '{"next_table_id": 2048}'
```
The expected output is (the `next_table_id` may be different):
```bash
{"next_table_id":2048}
```
### Get next table ID
```bash
curl -X GET 'http://localhost:4000/admin/sequence/table/next-id'
```
The expected output is (the `next_table_id` may be different):
```bash
{"next_table_id":1254}
```
---
## Table Reconciliation
## Overview
GreptimeDB uses a layered metadata management architecture in distributed environments:
- **Metasrv**: Acts as the metadata management layer, responsible for maintaining metadata information for all tables in the cluster
- **Datanode**: Handles actual data storage and query execution, while also persisting some table metadata
Under ideal conditions, table metadata between Metasrv and Datanode should remain perfectly consistent. However, in real production environments, restore from a [metadata backup](/user-guide/deployments-administration/manage-metadata/restore-backup.md) may cause metadata inconsistencies.
**Table Reconciliation** is a table metadata repair mechanism provided by GreptimeDB that:
- Detects metadata differences between Metasrv and Datanode
- Repairs inconsistency issues according to predefined strategies
- Ensures the system can recover from abnormal states to a consistent, available state
## Before starting
Before starting the table reconciliation process, you need to:
1. Restore the cluster from a specific [metadata backup](/user-guide/deployments-administration/manage-metadata/restore-backup.md)
2. Set the [Next Table ID](/user-guide/deployments-administration/maintenance/sequence-management.md) to the original cluster's next table ID
## Repair Scenarios
### `Table not found` Error
After a cluster is restored from specific metadata, write and query operations may encounter `Table not found` errors.
- **Scenario 1**: The original cluster created new tables after the metadata backup, causing the new table metadata to not be included in the backup. This results in `Table not found` errors when querying these new tables. In this case, the new created table will be lost, and you must to manually set the [Next Table ID](/user-guide/deployments-administration/maintenance/sequence-management.md) to ensure that the restored cluster won't fail to create tables due to table ID conflicts when creating new tables.
- **Scenario 2**: The original cluster renamed existing tables after the metadata backup. In this case, the new table names will be lost.
### `Empty region directory` Error
After a cluster is restored from specific metadata, starting the Datanode may result in an `Empty region directory` error. This usually occurs because the original cluster deleted tables (executed `DROP TABLE`) after the metadata backup, causing the deleted table metadata to not be included in the backup, resulting in this error when starting the Datanode. For this situation, you need to enable [Recovery Mode](/user-guide/deployments-administration/maintenance/recovery-mode.md) after starting Metasrv when starting the cluster to ensure the Datanode can start normally.
- **Mito Engine tables**: Table metadata cannot be repaired, you need to manually execute the `DROP TABLE` command to delete the non-existent table.
- **Metric Engine tables**: Table metadata can be repaired, you need to manually execute the `ADMIN reconcile_table('table_name')` command to repair the table metadata.
### `No field named` Error
After a cluster is restored from specific metadata, write and query operations may encounter `No field named` errors. This usually occurs because the original cluster deleted columns (executed `DROP COLUMN`) after the metadata backup, causing the deleted column metadata to not be included in the backup, resulting in this error when querying these deleted columns. For this situation, you need to manually execute the `ADMIN reconcile_table('table_name')` command to repair the table metadata.
### `schema has a different type` Error
After a cluster is restored from specific metadata, write and query operations may encounter `schema has a different type` errors. This usually occurs because the original cluster modified column types (executed `MODIFY COLUMN [column_name] [type]`) after the metadata backup, causing the modified column type metadata to not be included in the backup, resulting in this error when querying these modified columns. For this situation, you need to manually execute the `ADMIN reconcile_table('table_name')` command to repair the table metadata.
### Missing Specific Columns
After a cluster is restored from specific metadata, write and query operations may run normally but not include some columns. This occurs because the original cluster added columns (executed `ADD COLUMN`) after the metadata backup, causing the new column metadata to not be included in the backup, making these columns unavailable during queries. For this situation, you need to manually execute the `ADMIN reconcile_table('table_name')` command to repair the table metadata.
### Missing Column Indexes
After a cluster is restored from specific metadata, write and query operations may run normally, but `SHOW CREATE TABLE`/`SHOW INDEX FROM [table_name]` shows that certain columns don't include expected indexes. This occurs because the original cluster modified indexes (executed `MODIFY INDEX [column_name] SET [index_type] INDEX`) after the metadata backup, causing the index change metadata to not be included in the backup. For this situation, you need to manually execute the `ADMIN reconcile_table('table_name')` command to repair the table metadata.
## Repair operations
GreptimeDB provides the following Admin functions to trigger table metadata repair:
### Repair All Tables
Repair the metadata inconsistency of all tables in the entire cluster:
```sql
ADMIN reconcile_catalog()
```
### Repair a Specific Database
Repair the metadata inconsistency of all tables in a specific database:
```sql
ADMIN reconcile_database('database_name')
```
### Repair a Specific Table
Repair the metadata inconsistency of a single table:
```sql
ADMIN reconcile_table('table_name')
```
### View Repair Progress
After the Admin function is executed, it will return a `ProcedureID`, you can use the following command to view the progress of the repair task:
```sql
ADMIN procedure_state('procedure_id')
```
When `procedure_state` returns Done, it indicates that the repair task has completed.
## Important Notes
When performing table metadata repair operations, please note the following:
- Repair operations are executed asynchronously, and you can check the execution progress through the `procedure_id`
- It is recommended to perform repair operations during low-traffic periods to reduce the impact on system performance
- For large-scale repair operations (such as `reconcile_catalog()`), it is recommended to validate in a test environment first
---
## Basic Table Operations
[Data Model](/user-guide/concepts/data-model.md) should be read before this guide.
GreptimeDB provides table management functionalities via SQL. The following guide
uses [MySQL Command-Line Client](https://dev.mysql.com/doc/refman/8.0/en/mysql.html) to demonstrate it.
For more explanations of the `SQL` syntax, please see the [SQL reference](/reference/sql/overview.md).
## Create a database
The default database is `public`. You can create a database manually.
```sql
CREATE DATABASE test;
```
```sql
Query OK, 1 row affected (0.05 sec)
```
Create a database with a `TTL` (Time-To-Live) of seven days, which means all the tables in this database will inherit this option if they don't have their own `TTL` setting:
```sql
CREATE DATABASE test with(ttl='7d');
```
You can list all the existing databases.
```sql
SHOW DATABASES;
```
```sql
+--------------------+
| Database |
+--------------------+
| greptime_private |
| information_schema |
| public |
| test |
+--------------------+
4 rows in set (0.00 sec)
```
Using `like` syntax:
```sql
SHOW DATABASES LIKE 'p%';
```
```sql
+----------+
| Database |
+----------+
| public |
+----------+
1 row in set (0.00 sec)
```
Then change the database:
```sql
USE test;
```
## Create a table
:::tip NOTE
GreptimeDB offers a schemaless approach to writing data that eliminates the need to manually create tables using additional protocols. See [Automatic Schema Generation](/user-guide/ingest-data/overview.md#automatic-schema-generation).
:::
You can still create a table manually via SQL if you have specific requirements.
Suppose we want to create a table named monitor with the following data model:
- `host` is the hostname of the collected standalone machine, which should be a `Tag` that used to filter data when querying.
- `ts` is the time when the data is collected, which should be the `Timestamp`. It can also used as a filter when querying data with a time range.
- `cpu` and `memory` are the CPU utilization and memory utilization of the machine, which should be `Field` columns that contain the actual data and are not indexed.
The SQL code for creating the table is shown below. In SQL, we use the primary key to specify `Tag`s and the `TIME INDEX` to specify the `Timestamp` column. The remaining columns are `Field`s.
```sql
CREATE TABLE monitor (
host STRING,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP() TIME INDEX,
cpu FLOAT64 DEFAULT 0,
memory FLOAT64,
PRIMARY KEY(host));
```
```sql
Query OK, 0 row affected (0.03 sec)
```
Create the table with a `TTL` (Time-To-Live) of seven days:
```sql
CREATE TABLE monitor (
host STRING,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP() TIME INDEX,
cpu FLOAT64 DEFAULT 0,
memory FLOAT64,
PRIMARY KEY(host)
) WITH (ttl='7d');
```
:::warning NOTE
GreptimeDB does not currently support changing the TIME INDEX after a table has been created.
Therefore, it is important to carefully choose your TIME INDEX column before creating tables.
:::
### `CREATE TABLE` syntax
- Time index: Every GreptimeDB table requires exactly one time index. Declare it with the
`TIME INDEX` keyword. The column must use a `TIMESTAMP` data type.
- Primary key: The columns in primary key are similar to tags in other other time-series systems like [InfluxDB][1]. The primary key columns with the time index column are used to uniquely define a series of data, which is similar
to time series like [InfluxDB][2].
- Table options: when creating a table, you can specify a set of table options, click [here](/reference/sql/create.md#table-options) for more details.
### Table name constraints
GreptimeDB supports a limited set of special characters in table names, but they must adhere to the following constraints:
- A valid GreptimeDB table name must start with a letter (either lowercase or uppercase) or `-` / `_` / `:`.
- The rest part of table name can be alphanumeric or special characters within: `-` / `_` / `:` / `@` / `#`.
- Any table name containing special characters must be quoted with backquotes.
- Any table name containing uppercase letters must be quoted with backquotes.
Here are some examples:
```sql
-- ✅ Ok
create table a (ts timestamp time index);
-- ✅ Ok
create table a0 (ts timestamp time index);
-- 🚫 Invalid table name
create table 0a (ts timestamp time index);
-- 🚫 Invalid table name
create table -a (ts timestamp time index);
-- ✅ Ok
create table `-a` (ts timestamp time index);
-- ✅ Ok
create table `a@b` (ts timestamp time index);
-- 🚫 Invalid table name
create table memory_HugePages (ts timestamp time index);
-- ✅ Ok
create table `memory_HugePages` (ts timestamp time index);
```
[1]: https://docs.influxdata.com/influxdb/v1.8/concepts/glossary/#tag-key
[2]: https://docs.influxdata.com/influxdb/v1/concepts/glossary/#series
## Describe a table
Show table information in detail:
```sql
DESC TABLE monitor;
```
```sql
+--------+----------------------+------+------+---------------------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------+----------------------+------+------+---------------------+---------------+
| host | String | PRI | YES | | TAG |
| ts | TimestampMillisecond | PRI | NO | current_timestamp() | TIMESTAMP |
| cpu | Float64 | | YES | 0 | FIELD |
| memory | Float64 | | YES | | FIELD |
+--------+----------------------+------+------+---------------------+---------------+
4 rows in set (0.01 sec)
```
The Semantic Type column describes the data model of the table. The `host` is a `Tag` column, `ts` is a `Timestamp` column, and cpu and memory are `Field` columns.
## Show Table Definition and Indexes
Use `SHOW CREATE TABLE table_name` to get the statement when creating the table:
```sql
SHOW CREATE TABLE monitor;
```
```
+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| monitor | CREATE TABLE IF NOT EXISTS `monitor` (
`host` STRING NULL,
`ts` TIMESTAMP(3) NOT NULL DEFAULT current_timestamp(),
`cpu` DOUBLE NULL DEFAULT 0,
`memory` DOUBLE NULL,
TIME INDEX (`ts`),
PRIMARY KEY (`host`)
)
ENGINE=mito
|
+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
List all indexes in a table:
```sql
SHOW INDEXES FROM monitor;
```
```sql
+---------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+-------------------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+---------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+-------------------------+---------+---------------+---------+------------+
| monitor | 1 | PRIMARY | 1 | host | A | NULL | NULL | NULL | YES | greptime-primary-key-v1 | | | YES | NULL |
| monitor | 1 | TIME INDEX | 1 | ts | A | NULL | NULL | NULL | NO | | | | YES | NULL |
+---------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+-------------------------+---------+---------------+---------+------------+
```
For more info about `SHOW` statement, please read the [SHOW reference](/reference/sql/show.md).
## List Existing Tables
You can use `show tables` statement to list existing tables
```sql
SHOW TABLES;
```
```sql
+----------------+
| Tables_in_test |
+----------------+
| monitor |
+----------------+
1 row in set (0.00 sec)
```
Currently only table name filtering is supported. You can filter existing tables by their names.
```sql
SHOW TABLES LIKE monitor;
```
```sql
+----------------+
| Tables_in_test |
+----------------+
| monitor |
+----------------+
1 row in set (0.00 sec)
```
List tables in a specific database:
```sql
SHOW TABLES FROM test;
```
```sql
+----------------+
| Tables_in_test |
+----------------+
| monitor |
+----------------+
1 row in set (0.01 sec)
```
## Alter a table
You can alter the schema of existing tables just like in MySQL database
```sql
ALTER TABLE monitor ADD COLUMN label VARCHAR;
```
```sql
Query OK, 0 rows affected (0.03 sec)
```
```sql
ALTER TABLE monitor DROP COLUMN label;
```
```sql
Query OK, 0 rows affected (0.03 sec)
```
The `ALTER TABLE` statement also supports adding, removing, and renaming columns, as well as modifying the column datatype, etc. Please refer to the [ALTER reference](/reference/sql/alter.md) for more information.
## Drop a table
:::danger danger
Without [soft-drop](/enterprise/soft-drop.md) (GreptimeDB Enterprise only), `DROP TABLE` cannot be undone. Use it with care!
:::
`DROP TABLE [db.]table` is used to drop the table in `db` or the current database in-use.Drop the table `monitor` in the current database:
```sql
DROP TABLE monitor;
```
```sql
Query OK, 1 row affected (0.01 sec)
```
## Drop a database
:::danger danger
`DROP DATABASE` cannot be undone. Use it with care!
:::
You can use `DROP DATABASE` to drop a database.
For example, change back to `public` database and drop the `test` database:
```sql
USE public;
DROP DATABASE test;
```
Please refer to the [DROP](/reference/sql/drop.md#drop-database) document for more details.
## HTTP API
You can execute the SQL statements through the HTTP API.
For example,
using the following code to create a table through POST method:
```shell
curl -X POST \
-H 'authorization: Basic ' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'sql=CREATE TABLE monitor (host STRING, ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP(), cpu FLOAT64 DEFAULT 0, memory FLOAT64, TIME INDEX (ts), PRIMARY KEY(host))' \
http://localhost:4000/v1/sql?db=public
```
```json
{ "output": [{ "affectedrows": 0 }], "execution_time_ms": 10 }
```
For more information about SQL HTTP request, please refer to [API document](/user-guide/protocols/http.md#post-sql-statements).
## Time zone
The specified time zone in the SQL client session will affect the default timestamp value when creating or altering a table.
If you set the default value of a timestamp column to a string without a time zone,
the client's time zone information will be automatically added.
For more information about the effect of the client time zone, please refer to the [time zone](/user-guide/ingest-data/for-iot/sql.md#time-zone) section in the write data document.
---
## Compaction
For databases based on the LSM Tree, compaction is extremely critical. It merges overlapping fragmented SST files into a single ordered file, discards deleted data while significantly improves query performance.
Since v0.9.1, GreptimeDB provides compaction strategies to control how SST files are compacted: Time Windowed Compaction Strategy (TWCS) and Strict Window Compaction Strategy (SWCS).
## Concepts
Let us start with those core concepts of compaction in GreptimeDB.
### SST files
SSTs are sorted files generated when memtables are flushed to persistent storage, such as disks and object stores.
In GreptimeDB, data rows in SST files are organized by [tags](/user-guide/concepts/data-model.md) and timestamps, as illustrated below. Each SST file covers a specific time range. When a query specifies a time range, GreptimeDB retrieves only the relevant SST files that may contain data within that range, rather than loading all persisted files.
Typically, the time ranges of SST files do not overlap during real-time writing workloads. However, due to factors like row deletions and out-of-order writes, SST files may have overlapping time ranges, which can affect query performance.
### Time window
Time-series workloads present prominent "windowed" pattern in that most recently inserted rows are more possible to be read. Thus GreptimeDB logically partitioned the time axis into different time windows and we are more interested in compacting those SST files fall into the same time window.
Time window for some specific table is usually inferred from the most-recently flushed SST files or users may manually specify the time window if TWCS is chosen.
GreptimeDB has a preset collection of window sizes that are:
- 1 hour
- 2 hours
- 12 hours
- 1 day
- 1 week
If time window is not specified, GreptimeDB will use `1hour` as the initial time window size and split all inserted rows to those time windows and will infer a proper time window size when the first compaction happens by selecting the minimum time window size for the above collection that can cover the whole time span of all files to be compacted.
For example, during the first compaction, the time span for all SST files is 4 hours, then GreptimeDB will choose 12 hours as the time window for that table and persist it for later compactions.
### Sorted runs
Sorted runs is a collection of SST files that have sorted and non-overlapping time ranges.
For example, a table contains 5 SSTs with following time ranges (all inclusive): `[0, 10]`, `[12, 23]`, `[22, 25]`,`[24, 30]`,`[26,33]` and we can find 2 sorted runs:
The number of sorted runs indicates the orderliness of SST files. More sorted runs typically lead to worse query performance. The main goal of compactions is to reduce the number of sorted runs.
### Levels
Databases based on LSM trees may also have levels, keys are merged level by level. GreptimeDB only has 2 levels which are 0 (uncompacted) and 1 (compacted).
## Compaction strategies
GreptimeDB provides two compaction strategies as mentioned above, but only Time Windowed Compaction Strategy (TWCS) can be chosen when creating tables. Strict Window Compaction Strategy (SWCS) is only available when executing manual compactions.
## Time Windowed Compaction Strategy (TWCS)
TWCS primarily aims to reduce read/write amplification during compaction.
It assigns files to be compacted into different time windows. For each window, TWCS identifies the sorted runs. If there are more than 1 sorted runs, TWCS finds a solution to merge the overlapping files in those runs with merging penalties taken into consideration. If there's only 1 sorted run, TWCS checks for excessive file fragmentation and merges fragmented files if necessary since SST file count also impacts query performance.
For window assignment, SST files may span multiple time windows. TWCS assigns SSTs based on their maximum timestamps to ensure they are not affected by stale data. In time-series workloads, out-of-order writes are infrequent, and even when they occur, recent data's query performance is more critical than that of stale data.
Common TWCS table options include:
- `trigger_file_num`: number of files in a specific time window to trigger a compaction (default 4).
- `time_window`: time window size for TWCS compaction.
- `max_output_file_size`: max allowed compaction output file size (default 512MB).
Following diagrams show how files in a window get compacted when `trigger_file_num = 3`:
- In A, there're two SST files `[0, 3]` and `[5, 6, 9]` but there's only one sorted run since those two files have disjoint time ranges.
- In B, a new SST file `[1, 4]` is flushed therefore forms two sorted runs that exceeds the threshold. Then `[0, 3]` and `[1, 4]` are merged to `[0, 1, 3, 4]`.
- In C, a new SST file `[9, 10]` is flushed, and it will be merged with `[5, 6, 10]` to create `[5, 6, 9, 10]`, and after compaction the files will be like D.
- In E, a new file `[11, 12]` is flushed. Though there's still only one sorted run, but the number of files reaches `trigger_file_num`(3), so `[5, 6, 9, 10]` will be merged with `[11, 12]` to form `[5, 6, 9, 10, 11, 12]`.
### Specifying TWCS parameters
TWCS parameters can be specified at two levels:
1. **Table level**: Explicitly set when creating or altering a table
2. **Database level**: Set as defaults for all tables in the database
The effective compaction settings are resolved dynamically at compaction scheduling time with the following priority:
- Table-level settings (if specified)
- Database-level settings (if specified and no table-level override)
- Built-in defaults
#### Table-level compaction settings
You can specify TWCS parameters when creating tables:
```sql
CREATE TABLE monitor (
host STRING,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP() TIME INDEX,
cpu FLOAT64 DEFAULT 0,
memory FLOAT64,
PRIMARY KEY(host))
WITH (
'compaction.type'='twcs',
'compaction.twcs.trigger_file_num'='8',
'compaction.twcs.time_window'='1h',
'compaction.twcs.max_output_file_size'='500MB'
);
```
#### Database-level compaction settings
You can also set default compaction parameters for all tables in a database. These settings will be inherited by tables that don't have explicit compaction settings. See [ALTER DATABASE](/reference/sql/alter.md#modify-compaction-options-of-database) for examples.
:::note
Unlike table-level settings which are stored with the table metadata, database-level compaction settings are dynamically resolved at runtime. Changing database-level compaction options will immediately affect all tables in the database that don't have their own explicit compaction settings. This behavior is similar to how [database-level TTL](/reference/sql/create.md#create-database) works.
:::
## Strict Window Compaction Strategy (SWCS) and manual compaction
Unlike TWCS, which assigns one window per SST file based on their maximum timestamps, SWCS assigns SST files to **all** overlapping windows. Consequently, a single SST file may be included in multiple compaction outputs, as its name suggests. Due to its high read amplification during compaction, SWCS is not the default compaction strategy. However, it is useful when you need to manually trigger compaction to reorganize the layout of SST files—especially if an individual SST file spans a large time range that significantly slows down queries. GreptimeDB offers a simple SQL function for triggering compaction:
```sql
ADMIN COMPACT_TABLE(
,
,
[]
);
```
The `` parameter can be `regular` (or `twcs`) for regular TWCS compaction, or `swcs` (or `strict_window`) for Strict Window Compaction Strategy. The value is case-insensitive.
For the `swcs` strategy, the `` can specify:
- The window size (in seconds) for splitting SST files
- The `parallelism` parameter to control the level of parallelism for compaction (defaults to 1)
For example, to trigger compaction with a 1-hour window:
```sql
ADMIN COMPACT_TABLE(
"monitor",
"swcs",
"3600"
);
+------------------------------------------------+
| ADMIN COMPACT_TABLE("monitor", "swcs", "3600") |
+------------------------------------------------+
| 0 |
+------------------------------------------------+
1 row in set (0.01 sec)
```
When executing this statement, GreptimeDB will split each SST file into segments with a time span of 1 hour (3600 seconds) and merge these segments into a single output, ensuring no overlapping files remain.
You can also specify the `parallelism` parameter to speed up compaction by processing multiple files concurrently:
```sql
-- SWCS compaction with default time window and parallelism set to 2
ADMIN COMPACT_TABLE("monitor", "swcs", "parallelism=2");
-- SWCS compaction with custom time window and parallelism
ADMIN COMPACT_TABLE("monitor", "swcs", "window=1800,parallelism=2");
```
The `parallelism` parameter is also available for regular compaction:
```sql
-- Regular compaction with parallelism set to 2
ADMIN COMPACT_TABLE("monitor", "regular", "parallelism=2");
```
### Limit manual compaction to a time range
Both regular and SWCS manual compactions can be limited to a time range by setting `start_time` and `end_time` in the options string:
```sql
-- Regular compaction for a time range
ADMIN COMPACT_TABLE(
"monitor",
"regular",
"parallelism=2,start_time=2026-01-01T00:00:00Z,end_time=2026-02-01T00:00:00Z"
);
-- SWCS compaction for a time range
ADMIN COMPACT_TABLE(
"monitor",
"strict_window",
"window=3600,start_time=2026-01-01T00:00:00Z,end_time=2026-02-01T00:00:00Z"
);
```
The time range has the following rules and behavior:
- `start_time` and `end_time` must be specified together, and `start_time` must be earlier than `end_time`.
- The range is half-open: `[start_time, end_time)`. It selects compaction windows that overlap the range rather than filtering individual rows.
- A timestamp with an explicit time zone or UTC offset uses that zone. A timestamp without one uses the current query session time zone.
- Regular compaction keeps the existing TWCS trigger rules and only considers candidate windows that overlap the range. Specifying a range does not force an otherwise ineligible window to be compacted.
- SWCS preserves its forced-compaction behavior. If an SST file in a selected window also spans other windows, GreptimeDB transitively includes those dependent windows. The actual compacted windows can therefore extend beyond the requested range; this ensures that all rows from shared input SST files are retained.
The following diagram shows the process of strict window compaction:
In Figure A, there are 3 overlapping SST files: `[0, 3]` (which includes timestamps 0, 1, 2, and 3), `[3, 8]`, and `[8, 10]`.
The strict window compaction strategy will assign the file `[3, 8]` that covers windows 0, 4, and 8 to three separate windows respectively. This allows it to merge with `[0, 3]` and `[8, 10]` separately.
Figure B shows the final compaction result with three files: `[0, 3]`, `[4, 7]`, and `[8, 10]`. These files do not overlap with each other.
---
## Garbage Collection (GC)
GreptimeDB GC delays physical deletion of SST/index files until all references (running queries, [repartition](./repartition.md) cross-region file refs) are released. The configuration contains two parts:
- Metasrv Configuration
- Datanode Configuration
## How it works
- **Roles**: Meta decides when/where to clean; datanodes perform the actual delete while keeping in-use files safe.
- **Safety windows**: `lingering_time` holds known-removed files a bit longer; `unknown_file_lingering_time` is a rare-case guard that only applies during full listing GC.
- **Listing modes**: Fast mode removes files the system already marked; full listing walks storage to catch stragglers/orphans.
## Metasrv Configuration
On the Metasrv side, GC schedules cleanup tasks for regions and coordinates when to run GC.
```toml
[gc]
enable = true # Enable meta GC scheduler. default to be false; must match datanode.
gc_cooldown_period = "5m" # Minimum gap before the same region is GCed again.
[gc.experimental_soft_drop]
enable = false # Enable soft-drop tables (GreptimeDB Enterprise only). Requires gc.enable = true.
retention = "7d" # How long soft-dropped tables are retained before purge.
```
### Options
| Configuration Option | Description |
| --- | --- |
| `enable` | Enable the meta GC scheduler. Must match datanode GC enablement. |
| `gc_cooldown_period` | Minimum interval before the same region is scheduled for GC again; keep datanode `lingering_time` longer than this. |
| `experimental_soft_drop.enable` | Enable [soft-drop tables](/enterprise/soft-drop.md) (GreptimeDB Enterprise only). Requires `gc.enable = true`. |
| `experimental_soft_drop.retention` | How long soft-dropped tables are retained before automatic purge. |
## Datanode Configuration
The Datanode side performs the actual deletion while protecting files still in use.
```toml
[[region_engine]]
[region_engine.mito]
[region_engine.mito.gc]
enable = true # Turn on datanode GC worker; must match meta.
lingering_time = "10m" # Keep known-removed files this long for active queries.
unknown_file_lingering_time = "1d" # Keep files without expel time; rare safeguard.
```
### Options
| Configuration Option | Description |
| --- | --- |
| `enable` | Enable the datanode GC worker. Must match meta GC `enable`. |
| `lingering_time` | How long to keep manifest-removed files before deletion to protect long follower-region queries/cross-region references; set longer than `gc_cooldown_period`. Use `"0s"` to delete immediately. |
| `unknown_file_lingering_time` | Safety hold for files without expel time (not tracked in manifest). Applies only during full listing GC. Defaults to `1d`. |
For an active or open region, an unknown file is deleted only once its object-store
last-modified timestamp is older than `now - unknown_file_lingering_time`. If the object
store does not report a last-modified timestamp, the file is kept. Unknown files belonging
to dropped regions are deleted immediately, without waiting for this window.
:::warning
Keep `unknown_file_lingering_time` generous. A long-running flush or compaction writes
files before they appear in the manifest, so a short window can delete data that is still
being written.
:::
:::warning
`gc.enable` must be set consistently on metasrv and all datanodes. Mismatched flags cause GC to be skipped or stuck.
:::
## When to enable
- GC only applies when tables use object storage; tables on local filesystems ignore GC settings.
- Turn on GC if need to repartition so cross-region references can drain safely before deletion.
- For clusters with long-running follower-region queries, turn on GC and set `lingering_time` longer than `gc_cooldown_period` so files created or referenced during a GC cycle stay alive (in-use or lingering) until at least the next cycle.
- Leave GC off if you are not repartitioning and do not need delayed deletion.
## Operational notes
- GC is designed for object storage backends (with list/delete support); ensure your store credentials and permissions allow listing and deletion.
- Deleted files live in object storage until GC removes them; ensure storage listing/deletion permissions are in place.
- After enabling, restart metasrv and datanodes to apply config changes.
---
## Table and Region Operations
---
## Region Failover
Region Failover provides the ability to recover regions from region failures. With Kafka WAL (Remote WAL) and shared storage, Region Failover can recover regions without losing data. This is implemented via [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md).
## Enable the Region Failover
This feature is only available on GreptimeDB running on distributed mode and
- Using Kafka WAL (Remote WAL) or Local WAL with `allow_region_failover_on_local_wal=true` (enabling region failover on Local WAL may lead to data loss during failover)
- Using [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) (e.g., AWS S3)
If you want to enable region failover on local WAL, you need to set `allow_region_failover_on_local_wal=true` in [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) configuration file. It's not recommended to enable this option, because it may lead to data loss.
### Via configuration file
Set `enable_region_failover=true` in the [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration) configuration file. Additionally, set `region_failure_detector_initialization_delay` to a larger value and enable [Cluster Maintenance Mode](/user-guide/deployments-administration/maintenance/maintenance-mode.md) within the `region_failure_detector_initialization_delay` period to avoid triggering unnecessary Region Failover during datanode startup or upgrades.
### Via GreptimeDB Operator
To enable region failover via GreptimeDB Operator, you can refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#enable-region-failover) for more details.
## The recovery time of Region Failover
The recovery time of Region Failover with Kafka WAL (Remote WAL) depends on:
- number of regions per Topic.
- the Kafka cluster read throughput performance.
### The read amplification
In best practices, [the number of topics/partitions supported by a Kafka cluster is limited](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.html) (exceeding this number can degrade Kafka cluster performance).
Therefore, we allow multiple regions to share a single topic as the WAL.
However, this may cause to the read amplification issue.
The data belonging to a specific region consists of data files plus data in the WAL (typically `WAL[LastCheckpoint...Latest]`). The failover of a specific region only requires reading the region's WAL data to reconstruct the memory state, which is called region replaying. However, If multiple regions share a single topic, replaying data for a specific region from the topic requires filtering out unrelated data (i.e., data from other regions). This means replaying data for a specific region from the topic requires reading more data than the actual size of the region's data in the topic, a phenomenon known as read amplification.
Although multiple regions share the same topic, allowing the Datanode to support more regions, the cost of this approach is read amplification during WAL replay.
For example, configure 128 topics for [metasrv](/user-guide/deployments-administration/configuration.md#metasrv-only-configuration), and if the whole cluster holds 1024 regions (physical regions), every 8 regions will share one topic.
(Figure1: recovery Region 3 need to read redundant data 7 times larger than the actual size)
A simple model to estimate the redundant read amplification factor (redundant replay data size/actual data size):
- For a single topic, if we try to replay all regions that belong to the topic, then the amplification factor would be 7+6+...+1 = 28 times. (The Region WAL data distribution is shown in the Figure 1. Replaying Region 3 will read 7 times redundant data larger than the actual size; Region 6 will read 6 times, and so on)
- When recovering 100 regions (requiring about 13 topics), the amplification factor is approximately 28 \* 13 = 364 times.
Assuming we have 100 regions to recover, and the actual data size of all regions is 0.5GB, the following table shows the replay data size based on the number of regions per topic.
| Number of regions per Topic | Number of topics required for 100 Regions | Single topic redundant read factor | Total redundant read factor | Replay data size (GB) |
| --------------------------- | ----------------------------------------- | -------------------------------------- | ---------------------------------- | ---------------- |
| 1 | 100 | 0 | 0 | 0.5 |
| 2 | 50 | 1 | 50 | 25.5 |
| 4 | 25 | 6 | 150 | 75.5 |
| 8 | 13 | 28 | 364 | 182.5 |
| 16 | 7 | 120 | 840 | 420.5 |
The following table shows the recovery time of 100 regions under different read throughput conditions of the Kafka cluster. For example, when providing a read throughput of 300MB/s, recovering 100 regions requires approximately 10 minutes (182.5GB/0.3GB = 10m).
| Number of regions per Topic | Replay data size (GB) | Kafka throughput 300MB/s- Recovery time (secs) | Kafka throughput 1000MB/s- Recovery time (secs) |
| --------------------------- | ---------------- | --------------------------------------------- | ---------------------------------------------- |
| 1 | 0.5 | 2 | 1 |
| 2 | 25.5 | 85 | 26 |
| 4 | 75.5 | 252 | 76 |
| 8 | 182.5 | 608 | 183 |
| 16 | 420.5 | 1402 | 421 |
### Suggestions for improving recovery time
In the above example, we calculated the recovery time based on the number of Regions contained in each Topic for reference.
We have calculated the recovery time under different Number of regions per Topic configuration for reference.
In actual scenarios, the read amplification may be larger than this model.
If you use Kafka WAL and are very sensitive to recovery time, we recommend that each region have its own topic (i.e., Number of regions per Topic is 1).
---
## Region Migration
Region Migration allows users to move regions between the Datanode.
:::warning Warning
This feature is only available on GreptimeDB running on distributed mode and
- Using [shared storage](/user-guide/deployments-administration/configuration.md#storage-options) (e.g., AWS S3)
Otherwise, you can't perform a region migration.
:::
## Figure out the region distribution of the table.
You need to first query the region distribution of the table, i.e., find out on which Datanode the Regions in the table are located.
```sql
select b.peer_id as datanode_id,
a.greptime_partition_id as region_id
from information_schema.partitions a left join information_schema.region_peers b
on a.greptime_partition_id = b.region_id where a.table_name='migration_target' order by datanode_id asc;
```
For example, have the following region distribution:
```sql
+-------------+---------------+
| datanode_id | region_id |
+-------------+---------------+
| 1 | 4398046511104 |
+-------------+---------------+
1 row in set (0.01 sec)
```
For more info about the `region_peers` table, please read the [REGION-PEERS](/reference/sql/information-schema/region-peers.md).
## Select a Datanode as the migration destination.
:::warning Warning
The region migration won't be performed if the `from_peer_id` equals the `to_peer_id`.
:::
Remember, if you deploy the cluster via the GreptimeDB operator, the `peer_id` of Datanode always starts from 0. For example, if you have a 3 Datanode GreptimeDB cluster, the available `peer_id` will be 0,1,2.
Finally, you can do a Region migration request via the following SQL:
```sql
ADMIN migrate_region(4398046511104, 1, 2, 60);
```
The parameters of `migrate_region`：
```sql
ADMIN migrate_region(region_id, from_peer_id, to_peer_id);
ADMIN migrate_region(region_id, from_peer_id, to_peer_id, replay_timeout);
```
| Option | Description | Required | |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | --- |
| `region_id` | The region id. | **Required** | |
| `from_peer_id` | The peer id of the migration source(Datanode). | **Required** | |
| `to_peer_id` | The peer id of the migration destination(Datanode). | **Required** | |
| `replay_timeout` | The timeout (in seconds) for replaying data. Defaults to 300 seconds if omitted. If the new Region fails to replay the data within the specified timeout, the migration will fail, however the data in the old Region will not be lost. | Optional | |
## Query the migration state
The `migrate_region` function returns the procedure id that executes the migration, queries the procedure state by it:
```sql
ADMIN procedure_state('538b7476-9f79-4e50-aa9c-b1de90710839')
```
If it's done, outputs the state in JSON:
```json
{"status":"Done"}
```
Of course, you can confirm the region distribution by querying from `region_peers` and `partitions` in `information_schema`.
---
## Repartition
Repartition lets you adjust partition rules after a table has been created.
GreptimeDB does this through `ALTER TABLE` partition, split, and merge operations; see [ALTER TABLE](/reference/sql/alter.md#split-or-merge-partitions) for the syntax.
Repartition is only supported in distributed clusters.
## How it works
The core idea is to adjust partition rules and Region routing online instead of manually moving data into a new table.
GreptimeDB switches to the new partition layout by updating manifest file references for each Region, so the rules can better match the current data distribution.
This approach is useful when traffic patterns change over time and you want to keep partition rules aligned with the workload without rebuilding the table.
During repartitioning, writes may be briefly affected, so client-side retries are recommended.
## When to repartition
Consider repartitioning when:
- Some regions are consistently hotter than others.
- Your data distribution changes and the current partition boundaries no longer fit.
- Some regions become very small and cold, and you want to reduce resource usage by merging them.
- You want to further split a partition to improve write concurrency or query performance.
In general, when partition rules no longer reflect the current data distribution well, it is worth considering repartitioning.
## How to identify hot partitions
Before repartitioning, confirm which regions are hot.
Join region statistics with partition metadata to find the hottest rules:
```sql
SELECT
t.table_name,
r.region_id,
r.region_number,
p.partition_name,
p.partition_description,
r.region_role,
r.written_bytes_since_open,
r.region_rows
FROM information_schema.region_statistics r
JOIN information_schema.tables t
ON r.table_id = t.table_id
JOIN information_schema.partitions p
ON p.table_schema = t.table_schema
AND p.table_name = t.table_name
AND p.greptime_partition_id = r.region_id
WHERE t.table_schema = 'public'
AND t.table_name = 'your_table'
ORDER BY r.written_bytes_since_open DESC
LIMIT 10;
```
If some regions have a much higher `written_bytes_since_open` value over time, that partition rule is usually a good candidate for splitting.
Also check whether the region peers are healthy so node issues are not mistaken for hotspots:
```sql
SELECT
p.region_id,
p.peer_addr,
p.status,
p.down_seconds
FROM information_schema.region_peers p
WHERE p.table_schema = 'public'
AND p.table_name = 'your_table'
ORDER BY p.region_id, p.peer_addr;
```
If the nodes are healthy and the hotspot signal persists, you can move on to designing the repartition plan.
## Prerequisites
:::warning Warning
This feature is only available in distributed clusters and requires:
- Using [shared object storage](/user-guide/deployments-administration/configuration.md#storage-options) (e.g., AWS S3)
- Using [GC](/user-guide/deployments-administration/manage-data/gc.md) on both metasrv and all datanodes
Otherwise, you can't perform repartitioning.
:::
GreptimeDB supports repartitioning an unpartitioned table with `PARTITION ON COLUMNS`, and adjusting an existing partitioned table through repeated `SPLIT PARTITION` and `MERGE PARTITION` operations.
The most common follow-up changes are 1-to-2 splits and 2-to-1 merges.
More complex changes can also be done step by step.
Object storage stores region files, while GC reclaims old files only after references are released.
This helps prevent accidental deletion of data still in use.
### Through GreptimeDB Operator
If you deploy GreptimeDB with the GreptimeDB Operator, refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) for GC and object storage setup.
## Example
You can first partition an unpartitioned table, and later keep tuning the layout by merging existing partitions and splitting them with new rules.
Partition an unpartitioned table:
```sql
ALTER TABLE sensor_readings PARTITION ON COLUMNS (device_id, area) (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South',
device_id >= 100 AND area <= 'East',
device_id >= 100 AND area > 'East'
);
```
Adjust an existing partitioned table. The following example changes the `area` partition key from `South` to `North` for devices with `device_id < 100`:
```sql
ALTER TABLE sensor_readings MERGE PARTITION (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South'
);
ALTER TABLE sensor_readings SPLIT PARTITION (
device_id < 100
) INTO (
device_id < 100 AND area < 'North',
device_id < 100 AND area >= 'North'
);
```
To introduce a new partition column, use `SPLIT PARTITION ... ON COLUMNS`. The following example starts with a table partitioned only by `device_id`, then introduces `area` as a new partition column:
```sql
ALTER TABLE sensor_readings PARTITION ON COLUMNS (device_id) (
device_id < 100,
device_id >= 100
);
ALTER TABLE sensor_readings SPLIT PARTITION (
device_id < 100
) ON COLUMNS (device_id, area) INTO (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South'
);
```
After the split, the table's partition columns are updated to `(device_id, area)`.
## Further reading
For a step-by-step tutorial with more background and examples, see [How to Split and Merge Partitions Online in GreptimeDB](https://greptime.com/blogs/2026-03-19-greptimedb-repartition-guide).
---
## Table Sharding
Table sharding is a technique used to distribute a large table into multiple smaller tables.
This practice is commonly employed to enhance the performance of database systems.
In GreptimeDB, data is logically sharded into partitions.
Since GreptimeDB uses "tables" to group data and SQL to query them,
we adopt the term "partition," a concept frequently used in OLTP databases.
## When to shard a table
Inside GreptimeDB, both data management and scheduling are based on the region level,
and each region is corresponding to a table partition.
Thus when you have a table that is too large to fit into a single node,
or the table is too hot to be served by a single node,
you should consider sharding it.
A region in GreptimeDB has a relative fixed throughput capacity,
and the number of regions in a table determines the total throughput capacity of the table.
If you want to increase the throughput capacity of a table,
you can increase the number of regions in the table.
Ideally the overall throughput of a table should be proportional to the number of regions.
As for which specific partition column to use or how many regions to create,
it depends on the data distribution and the query pattern.
A general goal is to make the data distribution among regions as even as possible.
And the query pattern should be considered when designing the partition rule set as one query can be processed in parallel among regions.
In other word the query latency is depends on the "slowest" region's latency.
But notice that the increase of regions will bring some basic consumption and increase the complexity of the system.
You need to consider the requirement of data ingest rate, the query performance,
the data distribution on storage system.
You should shard a table only when necessary.
For more information on the relationship between partitions and regions, refer to the [Table Sharding](/contributor-guide/frontend/table-sharding.md) section in the contributor guide.
## Partition
In GreptimeDB, a table can be horizontally partitioned by column value ranges.
Currently, GreptimeDB supports range partitioning with the `PARTITION ON COLUMNS` syntax.
Each partition includes only a portion of the data from the table, and is
grouped by some column(s) value range. For example, we can partition a table in GreptimeDB like
this:
```sql
CREATE TABLE (...)
PARTITION ON COLUMNS () (
);
```
The syntax mainly consists of two parts:
1. `PARTITION ON COLUMNS` followed by a comma-separated list of column names. This specifies which columns will be used for partitioning. Currently, any column defined in the table can be used as a partition column. Note that the ranges of all partitions must **not** overlap.
2. `RULE LIST` is a list of multiple partition rules. Each rule is a partition condition, and GreptimeDB generates partition names such as `p0`, `p1`, and so on for these rules. The expressions here can use `=`, `!=`, `>`, `>=`, `<`, `<=`, `AND`, `OR`, column names, and literals.
:::tip Choosing partition columns
We recommend partitioning on Tag columns (the primary key columns). This matters most for tables that are **not** append-only (deduplication is enabled): the partition columns should be a subset of the primary key columns. That way, rows with the same primary key always go to the same partition, so deduplication and merging keep working correctly.
:::
:::tip Note
The parentheses in `PARTITION ON COLUMNS (...)` only accept column names (e.g., `device_id`, `area`), not expressions like `device_id + 1`. GreptimeDB does not support MySQL's `PARTITION BY RANGE` syntax.
:::
## Create a partitioned table
You can use the MySQL CLI to [connect to GreptimeDB](/user-guide/protocols/mysql.md) and create a distributed table.
The following example creates a table and partitions it based on the `device_id` column.
```SQL
CREATE TABLE sensor_readings (
device_id INT16,
reading_value FLOAT64,
ts TIMESTAMP DEFAULT current_timestamp(),
PRIMARY KEY (device_id),
TIME INDEX (ts)
)
PARTITION ON COLUMNS (device_id) (
device_id < 100,
device_id >= 100 AND device_id < 200,
device_id >= 200
);
```
More partition columns can be used to create more complex partition rules:
```sql
CREATE TABLE sensor_readings (
device_id INT,
area STRING,
reading_value FLOAT64,
ts TIMESTAMP DEFAULT current_timestamp(),
PRIMARY KEY (device_id, area),
TIME INDEX (ts)
)
PARTITION ON COLUMNS (device_id, area) (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South',
device_id >= 100 AND area <= 'East',
device_id >= 100 AND area > 'East'
);
```
The following content uses the `sensor_readings` table with two partition columns as an example.
## Partition an existing table
You can also partition an existing unpartitioned table with `ALTER TABLE ... PARTITION ON COLUMNS`.
The syntax is similar to creating a partitioned table, except that the partition rule is added to an existing table:
```sql
ALTER TABLE sensor_readings PARTITION ON COLUMNS (device_id, area) (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South',
device_id >= 100 AND area <= 'East',
device_id >= 100 AND area > 'East'
);
```
This is useful when a table starts as a single-partition table and later needs to be distributed across multiple regions.
:::warning Warning
Partitioning an existing table is a repartitioning operation. It is only supported in distributed clusters and requires shared object storage and GC. For the full prerequisites, see [Repartition](/user-guide/deployments-administration/manage-data/repartition.md#prerequisites).
:::
## Adjust partition rules
After a table is partitioned, you can keep tuning its partition layout by splitting hot partitions or merging small cold partitions.
Use `SPLIT PARTITION` to split an existing partition into multiple partitions:
```sql
ALTER TABLE sensor_readings SPLIT PARTITION (
device_id < 100 AND area < 'South'
) INTO (
device_id < 50 AND area < 'South',
device_id >= 50 AND device_id < 100 AND area < 'South'
);
```
Use `MERGE PARTITION` to merge multiple partitions into one:
```sql
ALTER TABLE sensor_readings MERGE PARTITION (
device_id < 100 AND area < 'South',
device_id < 100 AND area >= 'South'
);
```
For prerequisites, hotspot diagnosis, and more step-by-step examples, see [Repartition](/user-guide/deployments-administration/manage-data/repartition.md).
## Insert data into the table
The following code inserts 3 rows into each partition of the `sensor_readings` table.
```sql
INSERT INTO sensor_readings (device_id, area, reading_value, ts)
VALUES (1, 'North', 22.5, '2023-09-19 08:30:00'),
(10, 'North', 21.8, '2023-09-19 09:45:00'),
(50, 'North', 23.4, '2023-09-19 10:00:00');
INSERT INTO sensor_readings (device_id, area, reading_value, ts)
VALUES (20, 'South', 20.1, '2023-09-19 11:15:00'),
(40, 'South', 19.7, '2023-09-19 12:30:00'),
(90, 'South', 18.9, '2023-09-19 13:45:00');
INSERT INTO sensor_readings (device_id, area, reading_value, ts)
VALUES (110, 'East', 25.3, '2023-09-19 14:00:00'),
(120, 'East', 26.5, '2023-09-19 15:30:00'),
(130, 'East', 27.8, '2023-09-19 16:45:00');
INSERT INTO sensor_readings (device_id, area, reading_value, ts)
VALUES (150, 'West', 24.1, '2023-09-19 17:00:00'),
(170, 'West', 22.9, '2023-09-19 18:15:00'),
(180, 'West', 23.7, '2023-09-19 19:30:00');
```
:::tip NOTE
Note that when the written data does not meet any of the rules in the partitioning scheme, it will be assigned to the default partition (i.e., the first partition 0 of the table).
:::
## Distributed Read
Simply use the `SELECT` statement to query the data:
```sql
SELECT * FROM sensor_readings order by reading_value desc LIMIT 5;
```
```sql
+-----------+------+---------------+---------------------+
| device_id | area | reading_value | ts |
+-----------+------+---------------+---------------------+
| 130 | East | 27.8 | 2023-09-19 16:45:00 |
| 120 | East | 26.5 | 2023-09-19 15:30:00 |
| 110 | East | 25.3 | 2023-09-19 14:00:00 |
| 150 | West | 24.1 | 2023-09-19 17:00:00 |
| 180 | West | 23.7 | 2023-09-19 19:30:00 |
+-----------+------+---------------+---------------------+
5 rows in set (0.02 sec)
```
```sql
SELECT MAX(reading_value) AS max_reading
FROM sensor_readings;
```
```sql
+-------------+
| max_reading |
+-------------+
| 27.8 |
+-------------+
1 row in set (0.03 sec)
```
```sql
SELECT * FROM sensor_readings
WHERE area = 'North' AND device_id < 50
ORDER BY device_id;
```
```sql
+-----------+-------+---------------+---------------------+
| device_id | area | reading_value | ts |
+-----------+-------+---------------+---------------------+
| 1 | North | 22.5 | 2023-09-19 08:30:00 |
| 10 | North | 21.8 | 2023-09-19 09:45:00 |
+-----------+-------+---------------+---------------------+
2 rows in set (0.03 sec)
```
## Inspect a sharded table
GreptimeDB provides severals system table to check DB's state. For table sharding information, you can query [`information_schema.partitions`](/reference/sql/information-schema/partitions.md) which gives the detail of partitions inside one table, and [`information_schema.region_peers`](/reference/sql/information-schema/region-peers.md) which gives the runtime distribution of regions.
---
## Metadata Storage Configuration
This section describes how to configure different metadata storage backends for the GreptimeDB Metasrv component. The metadata storage is used to store critical system information including catalogs, schemas, tables, regions, and other metadata that are essential for the operation of GreptimeDB.
## Available Storage Backends
GreptimeDB supports the following metadata storage backends:
- **etcd**: The default and recommended backend for development and testing environments, offering simplicity and ease of setup
- **MySQL/PostgreSQL**: Production-ready backend options that integrate well with existing database infrastructure and cloud RDS services
This documentation describes the TOML configuration for each backend. You can use these configurations when deploying GreptimeDB without Helm Chart.
If you are using Helm Chart to deploy GreptimeDB, please refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md#configuring-metasrv-backend-storage) for more details.
## Use etcd as metadata storage
While etcd is suitable for development and testing environments, it may not be ideal for production deployments requiring high availability and scalability.
Configure the metasrv component to use etcd as metadata storage:
```toml
# The metadata storage backend for metasrv
backend = "etcd_store"
# Store server addresses
# You can specify multiple etcd endpoints for high availability
store_addrs = ["127.0.0.1:2379"]
# Backend client options for etcd
[backend_client]
# The keep alive timeout for backend client
keep_alive_timeout = "3s"
# The keep alive interval for backend client
keep_alive_interval = "10s"
# The connect timeout for backend client
connect_timeout = "3s"
[backend_tls]
# - "disable" - No TLS
# - "require" - Require TLS
mode = "prefer"
# Path to client certificate file (for client authentication)
# Like "/path/to/client.crt"
cert_path = ""
# Path to client private key file (for client authentication)
# Like "/path/to/client.key"
key_path = ""
# Path to CA certificate file (for server certificate verification)
# Required when using custom CAs or self-signed certificates
# Leave empty to use system root certificates only
# Like "/path/to/ca.crt"
ca_cert_path = ""
```
### Best Practices
While etcd can be used as metadata storage, we recommend against using it in production environments unless you have extensive experience with etcd operations and maintenance. For detailed guidance on etcd management, including installation, backup, and maintenance procedures, please refer to [Manage etcd](/user-guide/deployments-administration/manage-metadata/manage-etcd.md).
When using etcd as metadata storage:
- Deploy multiple endpoints across different availability zones for high availability
- Configure appropriate auto-compaction settings to manage storage growth
- Implement regular maintenance procedures:
- Run `Defrag` command periodically to reclaim disk space
- Monitor etcd cluster health metrics
- Review and adjust resource limits based on usage patterns
## Use MySQL as metadata storage
MySQL serves as a viable metadata storage backend option. This choice is particularly beneficial when you need to integrate with existing MySQL infrastructure or have specific MySQL-related requirements. For production deployments, we strongly recommend utilizing cloud providers' Relational Database Service (RDS) solutions for enhanced reliability and managed service benefits.
Configure the metasrv component to use MySQL as metadata storage:
```toml
# The metadata storage backend for metasrv
backend = "mysql_store"
# Store server address
# Format: mysql://user:password@ip:port/dbname
store_addrs = ["mysql://user:password@ip:port/dbname"]
# Optional: Custom table name for storing metadata
# Default: greptime_metakv
meta_table_name = "greptime_metakv"
[backend_tls]
# - "disable" - No TLS
# - "prefer" (default) - Try TLS, fallback to plain
# - "require" - Require TLS
# - "verify_ca" - Require TLS and verify CA
# - "verify_full" - Require TLS and verify hostname
mode = "prefer"
# Path to client certificate file (for client authentication)
# Like "/path/to/client.crt"
cert_path = ""
# Path to client private key file (for client authentication)
# Like "/path/to/client.key"
key_path = ""
# Path to CA certificate file (for server certificate verification)
# Required when using custom CAs or self-signed certificates
# Leave empty to use system root certificates only
# Like "/path/to/ca.crt"
ca_cert_path = ""
```
When sharing a MySQL instance between multiple GreptimeDB clusters, you must set a unique `meta_table_name` for each GreptimeDB cluster to avoid metadata conflicts.
## Use PostgreSQL as metadata storage
PostgreSQL serves as a viable metadata storage backend option. This choice is particularly beneficial when you need to integrate with existing PostgreSQL infrastructure or have specific PostgreSQL-related requirements. For production deployments, we strongly recommend utilizing cloud providers' Relational Database Service (RDS) solutions for enhanced reliability and managed service benefits.
Configure the metasrv component to use PostgreSQL as metadata storage:
```toml
# The metadata storage backend for metasrv
backend = "postgres_store"
# Store server address
# Format: password=password dbname=postgres user=postgres host=localhost port=5432
store_addrs = ["password=password dbname=postgres user=postgres host=localhost port=5432"]
# Optional: Custom table name for storing metadata
# Default: greptime_metakv
meta_table_name = "greptime_metakv"
# Optional: The schema for metadata table and election table name.
# In PostgreSQL 15 and later, the default public schema is restricted by default,
# and non-superusers are no longer allowed to create tables in the public schema.
# When encountering permission restrictions, use this parameter to specify a writable schema.
meta_schema_name = "greptime_schema"
# Optional: Automatically create PostgreSQL schema if it doesn't exist.
# When enabled, the system will execute `CREATE SCHEMA IF NOT EXISTS `
# before creating metadata tables. This is useful in production environments where
# manual schema creation may be restricted.
# Default: true
# Note: The PostgreSQL user must have CREATE SCHEMA permission for this to work.
auto_create_schema = true
# Optional: Advisory lock ID for election
# Default: 1
meta_election_lock_id = 1
[backend_tls]
# - "disable" - No TLS
# - "prefer" (default) - Try TLS, fallback to plain
# - "require" - Require TLS
# - "verify_ca" - Require TLS and verify CA
# - "verify_full" - Require TLS and verify hostname
mode = "prefer"
# Path to client certificate file (for client authentication)
# Like "/path/to/client.crt"
cert_path = ""
# Path to client private key file (for client authentication)
# Like "/path/to/client.key"
key_path = ""
# Path to CA certificate file (for server certificate verification)
# Required when using custom CAs or self-signed certificates
# Leave empty to use system root certificates only
# Like "/path/to/ca.crt"
ca_cert_path = ""
```
When sharing a PostgreSQL instance between multiple GreptimeDB clusters or with other applications, you must configure two unique identifiers to prevent conflicts:
1. Set a unique `meta_table_name` for each GreptimeDB cluster to avoid metadata conflicts
2. Assign a unique `meta_election_lock_id` to each GreptimeDB cluster to prevent advisory lock conflicts with other applications using the same PostgreSQL instance
---
## Manage etcd
The GreptimeDB cluster requires an etcd cluster for [metadata storage](https://docs.greptime.com/nightly/contributor-guide/metasrv/overview) by default. Let's install an etcd cluster using Bitnami's etcd Helm [chart](https://github.com/bitnami/charts/tree/main/bitnami/etcd).
## Prerequisites
- [Kubernetes](https://kubernetes.io/docs/setup/) >= v1.23
- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) >= v1.18.0
- [Helm](https://helm.sh/docs/intro/install/) >= v3.0.0
## Install
Save the following configuration as a file `etcd.yaml`:
```yaml
global:
security:
allowInsecureImages: true
replicaCount: 3
image:
registry: docker.io
repository: greptime/etcd
tag: 3.6.1-debian-12-r3
auth:
rbac:
create: false
token:
enabled: false
persistence:
storageClass: null
size: 8Gi
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '2'
memory: 8Gi
autoCompactionMode: "revision"
autoCompactionRetention: "100"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "8589934592"
- name: ETCD_ELECTION_TIMEOUT
value: "2000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
```
This configuration uses revision-based auto compaction. Every 5 minutes, etcd checks the current revision and compacts up to `current revision - 100`. The retained history window depends on the write rate; clients that read or watch older revisions must handle compaction errors. Compaction makes space available for reuse within etcd; reclaiming disk space requires defragmentation.
Install etcd cluster:
```bash
helm upgrade --install etcd \
oci://registry-1.docker.io/bitnamicharts/etcd \
--create-namespace \
--version 12.0.8 \
-n etcd-cluster \
--values etcd.yaml
```
Wait for etcd cluster to be running:
```bash
kubectl get pod -n etcd-cluster
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 64s
etcd-1 1/1 Running 0 65s
etcd-2 1/1 Running 0 72s
```
When the etcd cluster is running, use the following command to check the health status of etcd cluster:
```bash
kubectl -n etcd-cluster \
exec etcd-0 -- etcdctl \
--endpoints etcd-0.etcd-headless.etcd-cluster:2379,etcd-1.etcd-headless.etcd-cluster:2379,etcd-2.etcd-headless.etcd-cluster:2379 \
endpoint status -w table
```
Expected Output
```bash
+----------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| etcd-0.etcd-headless.etcd-cluster:2379 | 680910587385ae31 | 3.5.15 | 20 kB | false | false | 4 | 73991 | 73991 | |
| etcd-1.etcd-headless.etcd-cluster:2379 | d6980d56f5e3d817 | 3.5.15 | 20 kB | false | false | 4 | 73991 | 73991 | |
| etcd-2.etcd-headless.etcd-cluster:2379 | 12664fc67659db0a | 3.5.15 | 20 kB | true | false | 4 | 73991 | 73991 | |
+----------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```
## Backup
In the bitnami etcd chart, a shared storage volume Network File System (NFS) is used to store etcd backup data. By using CronJob in Kubernetes to perform etcd snapshot backups and mount NFS PersistentVolumeClaim (PVC), snapshots can be transferred to NFS.
Add the following configuration and name it `etcd-backup.yaml` file, Note that you need to modify **existingClaim** to your NFS PVC name:
```yaml
global:
security:
allowInsecureImages: true
replicaCount: 3
image:
registry: docker.io
repository: greptime/etcd
tag: 3.6.1-debian-12-r3
auth:
rbac:
create: false
token:
enabled: false
persistence:
storageClass: null
size: 8Gi
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '2'
memory: 8Gi
autoCompactionMode: "revision"
autoCompactionRetention: "100"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "8589934592"
- name: ETCD_ELECTION_TIMEOUT
value: "2000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
# Backup settings
disasterRecovery:
enabled: true
cronjob:
schedule: "*/30 * * * *"
historyLimit: 2
snapshotHistoryLimit: 2
pvc:
existingClaim: "${YOUR_NFS_PVC_NAME_HERE}"
```
Redeploy etcd cluster:
```bash
helm upgrade --install etcd \
oci://registry-1.docker.io/bitnamicharts/etcd \
--create-namespace \
--version 12.0.8 \
-n etcd-cluster \
--values etcd-backup.yaml
```
You can see the etcd backup scheduled task:
```bash
kubectl get cronjob -n etcd-cluster
```
Expected Output
```bash
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
etcd-snapshotter */30 * * * * False 0 36s
```
```bash
kubectl get pod -n etcd-cluster
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 35m
etcd-1 1/1 Running 0 36m
etcd-2 0/1 Running 0 6m28s
etcd-snapshotter-28936038-tsck8 0/1 Completed 0 4m49s
```
```bash
kubectl logs etcd-snapshotter-28936038-tsck8 -n etcd-cluster
```
Expected Output
```log
etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 2.698457ms
etcd 11:18:07.47 INFO ==> Snapshotting the keyspace
{"level":"info","ts":"2025-01-06T11:18:07.579095Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/snapshots/db-2025-01-06_11-18.part"}
{"level":"info","ts":"2025-01-06T11:18:07.580335Z","logger":"client","caller":"v3@v3.5.15/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2025-01-06T11:18:07.580359Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379"}
{"level":"info","ts":"2025-01-06T11:18:07.582124Z","logger":"client","caller":"v3@v3.5.15/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2025-01-06T11:18:07.582688Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379","size":"20 kB","took":"now"}
{"level":"info","ts":"2025-01-06T11:18:07.583008Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/snapshots/db-2025-01-06_11-18"}
Snapshot saved at /snapshots/db-2025-01-06_11-18
```
Next, you can see the etcd backup snapshot in the NFS server:
```bash
ls ${NFS_SERVER_DIRECTORY}
```
Expected Output
```bash
db-2025-01-06_11-18 db-2025-01-06_11-20 db-2025-01-06_11-22
```
## Restore
When you encounter etcd data loss or corruption, such as critical information stored in etcd being accidentally deleted, or catastrophic cluster failure that prevents recovery, you need to perform an etcd restore. Additionally, restoring etcd can also be useful for development and testing purposes.
Before recovery, you need to stop writing data to the etcd cluster (stop GreptimeDB Metasrv writing) and create the latest snapshot file use for recovery.
Add the following configuration file and name it `etcd-restore.yaml`. Note that **existingClaim** is the name of your NFS PVC, and **snapshotFilename** is change to the etcd snapshot file name:
```yaml
global:
security:
allowInsecureImages: true
replicaCount: 3
image:
registry: docker.io
repository: greptime/etcd
tag: 3.6.1-debian-12-r3
auth:
rbac:
create: false
token:
enabled: false
persistence:
storageClass: null
size: 8Gi
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '2'
memory: 8Gi
autoCompactionMode: "revision"
autoCompactionRetention: "100"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "8589934592"
- name: ETCD_ELECTION_TIMEOUT
value: "2000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
# Restore settings
startFromSnapshot:
enabled: true
existingClaim: "${YOUR_NFS_PVC_NAME_HERE}"
snapshotFilename: "${YOUR_ETCD_SNAPSHOT_FILE_NAME}"
```
Deploy etcd recover cluster:
```bash
helm upgrade --install etcd-recover \
oci://registry-1.docker.io/bitnamicharts/etcd \
--create-namespace \
--version 12.0.8 \
-n etcd-cluster \
--values etcd-restore.yaml
```
After waiting for the etcd recover cluster to be Running:
```bash
kubectl get pod -n etcd-cluster -l app.kubernetes.io/instance=etcd-recover
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
etcd-recover-0 1/1 Running 0 91s
etcd-recover-1 1/1 Running 0 91s
etcd-recover-2 1/1 Running 0 91s
```
:::tip NOTE
The configuration structure has changed between chart versions:
- In older version: `meta.etcdEndpoints`
- In newer version: `meta.backendStorage.etcd.endpoints`
Always refer to the latest [values.yaml](https://github.com/GreptimeTeam/helm-charts/blob/main/charts/greptimedb-cluster/values.yaml) in the Helm chart repository for the most up-to-date configuration structure.
:::
Next, change Metasrv [backendStorage.etcd.endpoints](https://github.com/GreptimeTeam/helm-charts/tree/main/charts/greptimedb-cluster) to the new etcd recover cluster, in this example is `"etcd-recover.etcd-cluster.svc.cluster.local:2379"`:
```yaml
apiVersion: greptime.io/v1alpha1
kind: GreptimeDBCluster
metadata:
name: greptimedb
spec:
# Other configuration here
meta:
backendStorage:
etcd:
endpoints:
- "etcd-recover.etcd-cluster.svc.cluster.local:2379"
```
Restart GreptimeDB Metasrv to complete etcd restore.
## Monitoring
- Prometheus Operator installed (e.g. via [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)).
- podmonitor CRD installed (automatically installed with Prometheus Operator).
Add the following to your `etcd-monitoring.yaml` to enable monitoring:
```yaml
global:
security:
allowInsecureImages: true
replicaCount: 3
image:
registry: docker.io
repository: greptime/etcd
tag: 3.6.1-debian-12-r3
auth:
rbac:
create: false
token:
enabled: false
persistence:
storageClass: null
size: 8Gi
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '2'
memory: 8Gi
autoCompactionMode: "revision"
autoCompactionRetention: "100"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "8589934592"
- name: ETCD_ELECTION_TIMEOUT
value: "2000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
# Monitoring settings
metrics:
enabled: true
podMonitor:
enabled: true
namespace: etcd-cluster
interval: 10s
scrapeTimeout: 10s
additionalLabels:
release: prometheus
```
Deploy etcd with Monitoring:
```bash
helm upgrade --install etcd \
oci://registry-1.docker.io/bitnamicharts/etcd \
--create-namespace \
--version 12.0.8 \
-n etcd-cluster \
--values etcd-monitoring.yaml
```
### Grafana dashboard
Use the [ETCD Cluster Overview dashboard](https://grafana.com/grafana/dashboards/15308-etcd-cluster-overview/) (ID: 15308) for monitoring key metrics.
1. Log in your Grafana.
2. Navigate to Dashboards -> New -> Import.
3. Enter Dashboard ID: 15308, select the data source and load.
## ⚠️ Defrag - Critical Warning
Defragmentation is a HIGH-RISK operation that can severely impact your ETCD cluster and dependent systems (like GreptimeDB):
1. Blocks ALL read/write operations during execution (cluster becomes unavailable).
2. High I/O usage may cause timeouts in client applications.
3. May trigger leader elections if defrag takes too long.
4. Can cause OOM kills if not properly resourced.
5. May corrupt data if interrupted mid-process.
ETCD uses a multi-version concurrency control (MVCC) mechanism that stores multiple versions of KV. Over time, as data is updated and deleted, the backend database can become fragmented, leading to increased storage usage and reduced performance. Defragmentation is the process of compacting this storage to reclaim space and improve performance.
Add the following defrag-related configuration to `etcd-defrag.yaml` file:
```yaml
global:
security:
allowInsecureImages: true
replicaCount: 3
image:
registry: docker.io
repository: greptime/etcd
tag: 3.6.1-debian-12-r3
auth:
rbac:
create: false
token:
enabled: false
persistence:
storageClass: null
size: 8Gi
resources:
limits:
cpu: '2'
memory: 8Gi
requests:
cpu: '2'
memory: 8Gi
autoCompactionMode: "revision"
autoCompactionRetention: "100"
extraEnvVars:
- name: ETCD_QUOTA_BACKEND_BYTES
value: "8589934592"
- name: ETCD_ELECTION_TIMEOUT
value: "2000"
- name: ETCD_SNAPSHOT_COUNT
value: "10000"
# Defragmentation settings
defrag:
enabled: true
cronjob:
schedule: "0 3 * * *" # Daily at 3:00 AM
suspend: false
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
```
Deploying with Defrag Configuration:
```bash
helm upgrade --install etcd \
oci://registry-1.docker.io/bitnamicharts/etcd \
--create-namespace \
--version 12.0.8 \
-n etcd-cluster \
--values etcd-defrag.yaml
```
You can see the etcd defrag scheduled task:
```bash
kubectl get cronjob -n etcd-cluster
```
Expected Output
```bash
NAME SCHEDULE TIMEZONE SUSPEND ACTIVE LAST SCHEDULE AGE
etcd-defrag 0 3 * * * False 0 34s
```
```bash
kubectl get pod -n etcd-cluster
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 4m30s
etcd-1 1/1 Running 0 4m29s
etcd-2 1/1 Running 0 4m29s
etcd-defrag-29128518-sstbf 0/1 Completed 0 90s
```
```bash
kubectl logs etcd-defrag-29128518-sstbf -n etcd-cluster
```
Expected Output
```log
Finished defragmenting etcd member[http://etcd-0.etcd-headless.etcd-cluster.svc.cluster.local:2379]
Finished defragmenting etcd member[http://etcd-1.etcd-headless.etcd-cluster.svc.cluster.local:2379]
Finished defragmenting etcd member[http://etcd-2.etcd-headless.etcd-cluster.svc.cluster.local:2379]
```
---
## Metadata Storage
A GreptimeDB cluster stores its metadata in an external system, which the Metasrv component reads and writes. The supported backends are:
- **[etcd](https://etcd.io/)** — a distributed key-value store with a small operational footprint.
- **[MySQL](https://www.mysql.com/) and [PostgreSQL](https://www.postgresql.org/)** — relational databases with ACID transactions, replication, and established backup tooling. Both are available as managed services (RDS) on the major cloud platforms.
## Which to choose
For development and testing, etcd is sufficient.
For production, use a cloud provider's managed relational database. A managed service covers high availability, automated backups, and maintenance, which is where most of the operational cost of self-hosting a metadata store goes.
Whichever backend you run, it sits outside GreptimeDB's own data path: it needs its own backup schedule and its own health monitoring.
## Next steps
- [Configuration](/user-guide/deployments-administration/manage-metadata/configuration.md) — pointing Metasrv at a backend.
- [Manage etcd](/user-guide/deployments-administration/manage-metadata/manage-etcd.md) — running etcd for development and testing.
---
## Backup and Restore, Migrate
GreptimeDB provides metadata backup and restore capabilities through its CLI tool. This functionality supports all major metadata storage backends including etcd, MySQL, and PostgreSQL. For detailed instructions on using these features, refer to the [Backup and Restore](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md) guide.
## Backup
For optimal backup reliability, schedule metadata backups during periods of low DDL (Data Definition Language) activity. This helps ensure data consistency and reduces the risk of partial or incomplete backups.
To perform a backup:
1. Verify that your GreptimeDB cluster is operational
2. Execute the backup using the CLI tool, follows the export metadata steps in [Backup and Restore](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md) guide.
3. Ensure the backup output file is created, and the file size is greater than 0.
## Restore
To restore from a backup:
1. Use the CLI tool to restore the metadata, follows the import metadata steps in [Backup and Restore](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-meta-data.md) guide.
2. Restart the GreptimeDB cluster to apply the restored metadata.
3. Set the [Next Table ID](/user-guide/deployments-administration/maintenance/sequence-management.md) to the original cluster's next table ID.
4. Call the [Table Reconciliation](/user-guide/deployments-administration/maintenance/table-reconciliation.md) function to repair the table metadata inconsistency.
## Migrate
We recommend to migrate metadata during periods of low DDL (Data Definition Language) activity to ensure data consistency and minimize the risk of partial or incomplete migrations.
Migrate metadata from one metadata storage to another.
1. Backup the metadata from the source storage.
2. Restore the metadata to the target storage.
3. Restart the whole GreptimeDB cluster(all components) to apply the restored metadata.
---
## Migrate Local SQL File Access
SQL access to local files is sandboxed in standalone deployments and disabled in
distributed deployments.
## Standalone
The default sandbox is `/copy`. Relative paths in `COPY` and
external-table locations are resolved below this directory. Absolute paths work
only when they are inside the sandbox.
Before upgrading, identify existing `COPY` workflows and external tables that
use local paths outside the default sandbox. Choose one of these migrations:
- Move the files below `/copy` and update the SQL locations.
- Set `storage.copy_root` to a dedicated local directory containing the files.
- Move the files to S3, OSS, GCS, or AzBlob and update the SQL locations.
Do not set `storage.copy_root` to `storage.data_home` or to a directory that
contains GreptimeDB data, WAL, manifests, or configuration files. GreptimeDB
rejects copy roots that expose its internal data directory.
When `storage.data_home` is an object-storage URL, local SQL file access is
disabled unless `storage.copy_root` explicitly names a local directory.
## Distributed
Distributed frontend and datanode processes reject local paths for `COPY TABLE`,
`COPY QUERY`, `COPY DATABASE`, and external tables. Migrate these workflows and
tables to S3, OSS, GCS, or AzBlob before upgrading.
---
## Check GreptimeDB Status
GreptimeDB provides a series of HTTP endpoints to query the operational status of GreptimeDB.
The following HTTP requests assume that GreptimeDB is running on node `127.0.0.1` with the HTTP service listening on the default port `4000`.
## Check if GreptimeDB is running normally
You can use the `/health` endpoint to check if GreptimeDB is running normally.
An HTTP status code `200 OK` indicates that GreptimeDB is running normally.
Example:
```bash
curl -i -X GET http://127.0.0.1:4000/health
```
Output:
```text
HTTP/1.1 200 OK
content-type: application/json
vary: origin, access-control-request-method, access-control-request-headers
access-control-allow-origin: *
content-length: 2
date: Sun, 26 Apr 2026 13:46:41 GMT
{}
```
For more information about the health check endpoint, please refer to [the Health endpoint](/reference/http-endpoints.md#health-check).
## Check GreptimeDB status
You can use the `/status` endpoint to check the deployment status of GreptimeDB.
```bash
curl -X GET http://127.0.0.1:4000/status | jq
```
Output:
```json
{
"commit": "8d2f92c01ae762287a3cac587156519a536ae134",
"branch": "",
"rustc_version": "rustc 1.96.0-nightly (ac7f9ec7d 2026-03-20)",
"hostname": "127.0.0.1",
"version": "1.0.1"
}
```
Please refer to [the Status endpoint](/reference/http-endpoints.md#status) for more details.
---
## Self-Monitoring GreptimeDB Clusters
Before reading this document, ensure you understand how to [deploy a GreptimeDB cluster on Kubernetes](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md).
This guide will walk you through configuring monitoring when deploying a GreptimeDB cluster.
## Quick Start
You can enable monitoring and Grafana by adding configurations to the `values.yaml` file when deploying the GreptimeDB cluster using Helm Chart.
Here's a complete example of `values.yaml` for deploying a minimal GreptimeDB cluster with monitoring and Grafana:
```yaml
image:
# Image registry:
# Use `docker.io` for OSS GreptimeDB,
# consult staff for Enterprise GreptimeDB
registry:
# Image repository:
# Use `greptime/greptimedb` for OSS GreptimeDB
# Consult staff for GreptimeDB Enterprise
repository:
# Image tag:
# Use database version `v1.2.1` for OSS GreptimeDB
# Consult staff for GreptimeDB Enterprise
tag:
pullSecrets: []
initializer:
registry: docker.io
repository: greptime/greptimedb-initializer
tag: "v0.6.1"
monitoring:
# Enable monitoring
enabled: true
grafana:
# Enable Grafana deployment
# Requires monitoring to be enabled first (monitoring.enabled: true)
enabled: true
frontend:
replicas: 1
meta:
replicas: 1
backendStorage:
etcd:
endpoints: ["etcd.etcd-cluster.svc.cluster.local:2379"]
datanode:
replicas: 1
flownode:
replicas: 1
```
When `monitoring` is enabled, GreptimeDB Operator launches an additional GreptimeDB Standalone instance to collect metrics and logs from the GreptimeDB cluster.
To collect log data, GreptimeDB Operator starts a [Vector](https://vector.dev/) sidecar container in each Pod.
When `grafana` is enabled, a Grafana instance is deployed that uses the GreptimeDB Standalone instance configured for cluster monitoring as its data source.
This enables visualization of the GreptimeDB cluster's monitoring data out of the box using both Prometheus and MySQL protocols.
Then install the GreptimeDB cluster with the above `values.yaml` file:
```bash
helm upgrade --install mycluster \
greptime/greptimedb-cluster \
--values /path/to/values.yaml \
-n default
```
After installation, you can check the Pod status of the GreptimeDB cluster:
```bash
kubectl -n default get pods
```
Expected Output
```bash
NAME READY STATUS RESTARTS AGE
mycluster-datanode-0 2/2 Running 0 77s
mycluster-flownode-0 2/2 Running 0 2m26s
mycluster-frontend-6ffdd549b-9s7gx 2/2 Running 0 66s
mycluster-grafana-675b64786-ktqps 1/1 Running 0 6m35s
mycluster-meta-58bc88b597-ppzvj 2/2 Running 0 86s
mycluster-monitor-standalone-0 1/1 Running 0 6m35s
```
You can then access the Grafana dashboard by port-forwarding the Grafana service to your local machine:
```shell
kubectl -n default port-forward svc/mycluster-grafana 18080:80
```
Then refer to the [Access Grafana Dashboard](#access-grafana-dashboard) section below for details on accessing Grafana.
## Monitoring Configuration
This section covers the details of monitoring configurations.
### Enable Monitoring
Add the following configuration to [`values.yaml`](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md#setup-valuesyaml) to enable monitoring when deploying the GreptimeDB cluster:
```yaml
monitoring:
enabled: true
```
This deploys a `GreptimeDBStandalone` resource named `${cluster-name}-monitor` to collect metrics and logs. You can verify the deployment with:
```bash
kubectl get greptimedbstandalones.greptime.io ${cluster-name}-monitor -n ${namespace}
```
The Service and StatefulSet it creates are named `${cluster-name}-monitor-standalone`, and the Pods add the usual StatefulSet ordinal, for example `${cluster-name}-monitor-standalone-0`. You can use the following addresses to access monitoring data:
- **Prometheus metrics**: `http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/prometheus`
- **SQL logs**: `${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4002`. By default, cluster logs are stored in the `public._gt_logs` table.
### Customize Monitoring Storage
By default, the GreptimeDB Standalone instance stores monitoring data using the Kubernetes default StorageClass in local storage.
You can configure the GreptimeDB Standalone instance through the `monitoring.standalone` field in `values.yaml`. For example, the following configuration uses S3 object storage to store monitoring data:
```yaml
monitoring:
enabled: true
ttl: 30d
standalone:
base:
main:
# Configure GreptimeDB Standalone instance image
image: "greptime/greptimedb:v1.2.1"
# Configure GreptimeDB Standalone instance resources
resources:
requests:
cpu: "2"
memory: "4Gi"
limits:
cpu: "2"
memory: "4Gi"
# Configure object storage for GreptimeDB Standalone instance
objectStorage:
s3:
# Configure bucket
bucket: "monitoring"
# Configure region
region: "ap-southeast-1"
# Configure secret name
secretName: "s3-credentials"
# Configure root path
root: "standalone-with-s3-data"
```
### Customize Vector Sidecar
The Vector sidecar configuration for log collection can be customized via the `monitoring.vector` field.
For example, you can adjust the Vector image and resource limits as follows:
```yaml
monitoring:
enabled: true
ttl: 30d
vector:
# Configure Vector image registry
registry: docker.io
# Configure Vector image repository
repository: timberio/vector
# Configure Vector image tag
tag: "0.49.0-debian"
# Configure Vector resources
resources:
requests:
cpu: "50m"
memory: "64Mi"
limits:
cpu: "250m"
memory: "256Mi"
```
### YAML Configuration with `kubectl` Deployment
If you're not using Helm Chart, you can also use the `monitoring` field to manually configure self-monitoring mode in the `GreptimeDBCluster` YAML:
```yaml
monitoring:
enabled: true
```
For detailed configuration options, refer to the [`GreptimeDBCluster` API documentation](https://github.com/GreptimeTeam/greptimedb-operator/blob/main/docs/api-references/docs.md#monitoringspec).
## Grafana Configuration
### Enable Grafana
To enable Grafana deployment, add the following configuration to `values.yaml`.
Note that monitoring must be enabled first [(`monitoring.enabled: true`)](#enable-monitoring):
```yaml
monitoring:
enabled: true
grafana:
enabled: true
```
### Customize Grafana Data Sources
By default, Grafana uses `mycluster` and `default` as the cluster name and namespace to create data sources.
To monitor clusters with different names or namespaces, you need to create custom data source configurations based on the actual cluster names and namespaces.
Here's an example `values.yaml` configuration:
```yaml
monitoring:
enabled: true
grafana:
enabled: true
datasources:
datasources.yaml:
datasources:
# Query the cluster metrics.
- name: greptimedb-metrics
type: prometheus
url: http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/prometheus
access: proxy
# Query the cluster traces.
- name: greptimedb-traces
type: jaeger
url: http://${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4000/v1/jaeger
access: proxy
# Query the cluster logs and slow queries.
- name: greptimedb-logs
type: mysql
url: ${cluster-name}-monitor-standalone.${namespace}.svc.cluster.local:4002
access: proxy
database: public
# Query the information schema from the cluster.
- name: information_schema
type: mysql
url: ${cluster-name}-frontend.${namespace}.svc.cluster.local:4002
access: proxy
database: information_schema
```
This configuration creates the following data sources for GreptimeDB cluster monitoring in Grafana:
- **`greptimedb-metrics`**: Cluster metrics stored in the standalone monitoring database, exposed via Prometheus protocol (`type: prometheus`)
- **`greptimedb-logs`**: Cluster logs stored in the standalone monitoring database, exposed via MySQL protocol (`type: mysql`). Uses the `public` database by default
- **`greptimedb-traces`**: Cluster traces stored in the standalone monitoring database, exposed via Jaeger protocol (`type: jaeger`). Uses the `public` database by default
- **`information_schema`**: Cluster information stored in the frontend database, exposed via MySQL protocol (`type: mysql`). Uses the `information_schema` database by default
### Access Grafana Dashboard
You can access the Grafana dashboard by port-forwarding the Grafana service to your local machine:
```bash
kubectl -n ${namespace} port-forward svc/${cluster-name}-grafana 18080:80
```
Then open `http://localhost:18080` to access the Grafana dashboard.
The default login credentials for Grafana are:
- **Username**: `admin`
- **Password**: `gt-operator`
Navigate to the `Dashboards` section to explore the pre-configured dashboards for monitoring your GreptimeDB cluster.
## Cleanup the PVCs
:::danger
This removes the collected metrics and logs stored by the monitoring standalone instance. It does not touch the data of the GreptimeDB cluster itself, but the monitoring history is not recoverable afterwards.
:::
To uninstall the GreptimeDB cluster, please refer to the [Cleanup GreptimeDB Cluster](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md#cleanup) documentation.
To clean up the Persistent Volume Claims (PVCs) used by the GreptimeDB standalone monitoring instance, delete the PVCs using the following command:
```bash
kubectl -n ${namespace} delete pvc -l app.greptime.io/component=${cluster-name}-monitor-standalone
```
---
## DDL events
See [Event data model](/user-guide/deployments-administration/monitoring/events/event-data-model.md)
for shared columns and Procedure states.
## Database events
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
json_to_string(payload) AS payload
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND type = ''
ORDER BY timestamp;
```
Database event types are `create_database`, `alter_database`, and `drop_database`.
Event rows retain `schema_name`, which this query uses as a filter but does not
include in the result.
**`create_database`**
```sql
+-------------------------------+-----------------+-----------------+--------------+----------------------------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | payload |
+-------------------------------+-----------------+-----------------+--------------+----------------------------------------------------------------------------------+
| 2026-08-10 11:39:18.014737397 | create_database | Running | Submitted | {"create_if_not_exists":true,"options":[{"key":"ttl","value":"1h"}],"version":1} |
| 2026-08-10 11:39:18.052757527 | create_database | Done | Succeeded | null |
+-------------------------------+-----------------+-----------------+--------------+----------------------------------------------------------------------------------+
```
**`alter_database`**
```sql
+-------------------------------+----------------+-----------------+--------------+---------------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | payload |
+-------------------------------+----------------+-----------------+--------------+---------------------------------------------------------------------+
| 2026-08-10 11:39:18.060198497 | alter_database | Running | Submitted | {"action":"set","options":[{"key":"ttl","value":"2h"}],"version":1} |
| 2026-08-10 11:39:18.107764363 | alter_database | Done | Succeeded | null |
+-------------------------------+----------------+-----------------+--------------+---------------------------------------------------------------------+
```
**`drop_database`**
```sql
+-------------------------------+---------------+-----------------+--------------+-------------------------------------+
| timestamp | type | procedure_state | trigger_type | payload |
+-------------------------------+---------------+-----------------+--------------+-------------------------------------+
| 2026-08-10 11:39:18.113713574 | drop_database | Running | Submitted | {"drop_if_exists":true,"version":1} |
| 2026-08-10 11:39:18.197868233 | drop_database | Done | Succeeded | null |
+-------------------------------+---------------+-----------------+--------------+-------------------------------------+
```
## Table events
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
table_name, table_id, json_to_string(payload) AS payload
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND type = ''
ORDER BY timestamp;
```
This query returns table events of the selected type from the specified database
in the last hour. To query one table, add
`AND table_name = ''` to the `WHERE` clause.
Rows for the same table retain `table_name`. A successful `create_table` or
`create_logical_tables` event includes `table_id` only when the Procedure returns
the allocated ID. `alter_table`, `truncate_table`, and `drop_table` already know
the table ID when they are submitted, so their event rows include it.
GreptimeDB also records `create_logical_tables` and `alter_logical_tables` events.
Each emits one row per logical table for each lifecycle trigger type, so a query
by `procedure_id` can return multiple rows for the same trigger type. `undrop_table`
and `purge_dropped_table` are Enterprise-only.
**`create_table`**
```sql
+-------------------------------+--------------+-----------------+--------------+----------------+----------+------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | table_name | table_id | payload |
+-------------------------------+--------------+-----------------+--------------+----------------+----------+------------------------------------------------------------+
| 2026-08-10 11:39:53.544351589 | create_table | Running | Submitted | ddl_source | NULL | {"create_if_not_exists":false,"engine":"mito","version":1} |
| 2026-08-10 11:39:53.620080844 | create_table | Done | Succeeded | ddl_source | 1449 | null |
+-------------------------------+--------------+-----------------+--------------+----------------+----------+------------------------------------------------------------+
```
**`alter_table`**
```sql
+-------------------------------+-------------+-----------------+--------------+------------+----------+------------------------------------+
| timestamp | type | procedure_state | trigger_type | table_name | table_id | payload |
+-------------------------------+-------------+-----------------+--------------+------------+----------+------------------------------------+
| 2026-08-10 11:40:23.445906161 | alter_table | Running | Submitted | ddl_source | 1449 | {"kind":"add_columns","version":1} |
| 2026-08-10 11:40:23.539710789 | alter_table | Done | Succeeded | ddl_source | 1449 | null |
+-------------------------------+-------------+-----------------+--------------+------------+----------+------------------------------------+
```
**`truncate_table`**
```sql
+-------------------------------+----------------+-----------------+--------------+----------------+----------+------------------------------------+
| timestamp | type | procedure_state | trigger_type | table_name | table_id | payload |
+-------------------------------+----------------+-----------------+--------------+----------------+----------+------------------------------------+
| 2026-08-10 11:40:51.949349425 | truncate_table | Running | Submitted | ddl_drop_probe | 1451 | {"time_range_count":0,"version":1} |
| 2026-08-10 11:40:51.982366731 | truncate_table | Done | Succeeded | ddl_drop_probe | 1451 | null |
+-------------------------------+----------------+-----------------+--------------+----------------+----------+------------------------------------+
```
**`drop_table`**
```sql
+-------------------------------+------------+-----------------+--------------+----------------+----------+--------------------------------------+
| timestamp | type | procedure_state | trigger_type | table_name | table_id | payload |
+-------------------------------+------------+-----------------+--------------+----------------+----------+--------------------------------------+
| 2026-08-10 11:40:51.986072827 | drop_table | Running | Submitted | ddl_drop_probe | 1451 | {"drop_if_exists":false,"version":1} |
| 2026-08-10 11:40:52.061732144 | drop_table | Done | Succeeded | ddl_drop_probe | 1451 | null |
+-------------------------------+------------+-----------------+--------------+----------------+----------+--------------------------------------+
```
## Flow events
Flow rows retain `catalog_name` and `flow_name`. `schema_name` is SQL `NULL` for
Flow events, while `flow_name` remains available. `create_flow` adds `flow_id` when
available, and `drop_flow` retains the known ID.
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
catalog_name, schema_name, flow_name, flow_id,
json_to_string(payload) AS payload
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND flow_name = ''
AND type = ''
ORDER BY timestamp;
```
**`create_flow`**
```sql
+-------------------------------+-------------+-----------------+--------------+--------------+-------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | catalog_name | schema_name | flow_name | flow_id | payload |
+-------------------------------+-------------+-----------------+--------------+--------------+-------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
| 2026-08-10 11:41:47.833758897 | create_flow | Running | Submitted | greptime | NULL | ddl_flow | NULL | {"create_if_not_exists":false,"eval_interval_secs":10,"expire_after":null,"or_replace":false,"version":1} |
| 2026-08-10 11:41:47.857319802 | create_flow | Done | Succeeded | greptime | NULL | ddl_flow | 1025 | null |
+-------------------------------+-------------+-----------------+--------------+--------------+-------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
```
**`drop_flow`**
```sql
+-------------------------------+-----------+-----------------+--------------+--------------+-------------+-----------+---------+--------------------------------------+
| timestamp | type | procedure_state | trigger_type | catalog_name | schema_name | flow_name | flow_id | payload |
+-------------------------------+-----------+-----------------+--------------+--------------+-------------+-----------+---------+--------------------------------------+
| 2026-08-10 11:41:47.864231473 | drop_flow | Running | Submitted | greptime | NULL | ddl_flow | 1025 | {"drop_if_exists":false,"version":1} |
| 2026-08-10 11:41:47.926665304 | drop_flow | Done | Succeeded | greptime | NULL | ddl_flow | 1025 | null |
+-------------------------------+-----------+-----------------+--------------+--------------+-------------+-----------+---------+--------------------------------------+
```
## View events
View rows retain stable `view_name` values. `create_view` adds `view_id` only
when available, and `drop_view` retains the known ID.
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
view_name, view_id, json_to_string(payload) AS payload
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND view_name = ''
AND type = ''
ORDER BY timestamp;
```
The query uses `schema_name` as a filter and does not include it in the result.
**`create_view`**
```sql
+-------------------------------+-------------+-----------------+--------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | view_name | view_id | payload |
+-------------------------------+-------------+-----------------+--------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
| 2026-08-10 11:41:18.483028192 | create_view | Running | Submitted | ddl_view | NULL | {"column_count":0,"create_if_not_exists":false,"or_replace":false,"referenced_table_count":1,"version":1} |
| 2026-08-10 11:41:18.513571854 | create_view | Done | Succeeded | ddl_view | 1452 | null |
+-------------------------------+-------------+-----------------+--------------+-----------+---------+-----------------------------------------------------------------------------------------------------------+
```
**`drop_view`**
```sql
+-------------------------------+-----------+-----------------+--------------+-----------+---------+--------------------------------------+
| timestamp | type | procedure_state | trigger_type | view_name | view_id | payload |
+-------------------------------+-----------+-----------------+--------------+-----------+---------+--------------------------------------+
| 2026-08-10 11:41:18.520281932 | drop_view | Running | Submitted | ddl_view | 1452 | {"drop_if_exists":false,"version":1} |
| 2026-08-10 11:41:18.572632167 | drop_view | Done | Succeeded | ddl_view | 1452 | null |
+-------------------------------+-----------+-----------------+--------------+-----------+---------+--------------------------------------+
```
---
## Event data model
`greptime_private.events` has the following common columns.
| Column | Meaning |
| --------------- | ----------------------------------------------------------- |
| `type` | Event type, such as `create_table` or `region_migration`. |
| `timestamp` | Time at which the row was recorded. |
| `payload` | JSON data for the event type. |
| `actor` | Database user recorded as the operation's initiator; SQL `NULL` when the event is not associated with a user request. |
| `event_context` | JSON describing why the event was triggered when available. |
## Event actor {#actor}
The `actor` is the database user recorded as the initiator of an operation. Use
it to identify who ran an `ADMIN` statement or submitted a Procedure.
When an operation starts a Procedure, GreptimeDB records the database user as
its `actor`. The Procedure and any child Procedures it starts keep the same
`actor` throughout their lifecycle. Events that are not associated with a user
request have a SQL `NULL` actor.
When [authentication](/user-guide/deployments-administration/authentication/overview.md)
is enabled, `actor` is the authenticated database user. Without authentication,
`actor` does not verify who sent the request. MySQL and PostgreSQL record the
username supplied by the client, while HTTP and gRPC record the default user
`greptime`.
## Procedure event columns
Procedure events also have the following columns:
| Column | Meaning |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `procedure_id` | Unique Procedure ID. |
| `procedure_state` | Procedure state when the event was recorded. Values are `Running`, `Done`, `Retrying`, `PrepareRollback`, `RollingBack`, `Failed`, and `Poisoned`. |
| `procedure_trigger` | Procedure event trigger in JSON. Its `type` is `Submitted`, `Recovered`, `ChildSubmitted`, `Retrying`, `RollingBack`, `Succeeded`, `Failed`, or `Poisoned`. |
| `procedure_error` | Debug-formatted error when the Procedure fails; an empty string otherwise. |
Rows with trigger type `Submitted` normally have state `Running`. When a Procedure
succeeds, the completed event has state `Done` and trigger type `Succeeded`. The completed
row is generated from the Procedure's final state, so its fields can differ from
the submitted row. Events are recorded asynchronously; a recording failure does
not change the Procedure result.
`event_context` is written only on `Submitted` rows. When it is available, its
stable `reason` value is one of
`manual`, `auto_create`, `auto_alter`, `auto_repartition`, `auto_rebalance`,
`region_failover`, `scheduled_gc`, or `unknown`. For example, a MySQL-submitted
event can contain `{"protocol":"mysql","reason":"manual"}`.
In addition to `Submitted` and `Succeeded`, a Procedure can emit the following
trigger types when applicable. Not every Procedure emits every trigger type, and rows are
not guaranteed to appear in the order shown:
| `type` | Meaning and fields |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Recovered` | The root Procedure was recovered from persisted state. |
| `ChildSubmitted` | A child Procedure submission was attempted. `procedure_trigger` includes the child `procedure_id` and its `outcome` (`Accepted`, `AlreadyAccepted`, `ManagerStopped`, or `SpawnFailed`). |
| `Retrying` | Procedure execution or rollback is being retried. `procedure_trigger` includes the retry `phase` (`Execute` or `Rollback`) and `attempt`. |
| `RollingBack` | Procedure rollback is starting. |
| `Failed` | The Procedure reached a failed terminal state. Inspect `procedure_error` for failure details. |
| `Poisoned` | The Procedure cannot proceed. Inspect `procedure_error` for the failure details. |
## ADMIN function event columns
An `admin_function` event records the result returned by an `ADMIN` statement.
| Column | Meaning |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| `admin_function_name` | The name of the executed ADMIN function. |
| `admin_function_status` | The execution status: `Succeeded` or `Failed`. |
| `admin_function_output` | JSON containing `result` when the function succeeds or `error` when it fails. |
The event `payload` contains the function arguments. If the result is a
Procedure ID, query the Procedure events for that ID to track its progress.
## Query JSON fields
See the [JSON functions](/reference/sql/functions/json.md) reference for
details. In event queries, `json_to_string` converts a JSON value to readable
text, `json_get_string` extracts a value by path, `json_path_match` evaluates a
JSON predicate, and `json_is_null` checks whether a value is the JSON `null`
value. Use `IS NULL` separately to check for SQL `NULL`.
For example, this query extracts fields from a `create_table` row that contains
`event_context`:
```sql
SELECT procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
json_path_match(procedure_trigger, '$.type == "Submitted"') AS is_submitted,
json_get_string(event_context, 'reason') AS reason
FROM greptime_private.events
WHERE type = 'create_table'
AND timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
AND event_context IS NOT NULL
ORDER BY timestamp;
```
```sql
+-----------------+--------------+--------------+--------+
| procedure_state | trigger_type | is_submitted | reason |
+-----------------+--------------+--------------+--------+
| Running | Submitted | 1 | manual |
+-----------------+--------------+--------------+--------+
```
## JSON `null` and SQL `NULL`
In the `create_table` example and DDL/repartition events after a Procedure completes, a terminal
`payload` can be JSON `null` rather than SQL `NULL`:
```sql
SELECT procedure_state, json_to_string(payload) AS payload,
payload IS NULL AS payload_is_sql_null,
json_is_null(payload) AS payload_is_json_null,
json_get_string(procedure_trigger, 'type') AS trigger_type
FROM greptime_private.events
WHERE type = 'create_table'
AND timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
ORDER BY timestamp;
```
```sql
+-----------------+------------------------------------------------------------+---------------------+----------------------+--------------+
| procedure_state | payload | payload_is_sql_null | payload_is_json_null | trigger_type |
+-----------------+------------------------------------------------------------+---------------------+----------------------+--------------+
| Running | {"create_if_not_exists":false,"engine":"mito","version":1} | 0 | 0 | Submitted |
| Done | null | 0 | 1 | Succeeded |
+-----------------+------------------------------------------------------------+---------------------+----------------------+--------------+
```
## Procedure event type-specific columns
The following columns are populated only by the listed event types. An SQL `NULL`
value in one of these columns is normal when it does not apply.
- **Database, table, and view events:** `catalog_name`, `schema_name`, and the
applicable table or view name and ID columns identify the affected object.
Flow events use `catalog_name` and `flow_name`; their `schema_name` is SQL
`NULL`. `physical_table_id` applies only to `create_logical_tables` and
`alter_logical_tables`. See [DDL events](/user-guide/deployments-administration/monitoring/events/ddl-events.md).
- **Region migration:** `region_id` and `region_number` identify the Region.
`region_migration_trigger_reason`, `region_migration_src_node_id`,
`region_migration_src_peer_addr`, `region_migration_dst_node_id`, and
`region_migration_dst_peer_addr` describe why and where it moved.
- **Repartition:** `catalog_name`, `schema_name`, `table_name`, and `table_id`
identify the affected table. `parent_procedure_id` links a child procedure to
its parent; `repartition_group_id` identifies a group operation. `source_region_id`,
`source_region_number`, `source_partition_expr`, `target_region_id`,
`target_region_number`, and `target_partition_expr` describe the affected
Regions and partition expressions.
- **Batch GC:** `region_id`, `region_number`, and `gc_report` describe the
Regions processed and their GC result.
- **WAL pruning:** `topic_name`, `prunable_entry_id`, and `latest_offset`
identify the topic, requested prune boundary, and exclusive latest offset.
---
## Maintenance events
Metasrv records GC and Remote WAL pruning events in distributed deployments.
## GC events
Each row with a non-`NULL` `gc_report` records the files processed for one Region:
```sql
SELECT timestamp, procedure_id, procedure_state,
region_id, table_id, region_number,
json_to_string(gc_report) AS gc_report
FROM greptime_private.events
WHERE type = 'batch_gc'
AND gc_report IS NOT NULL
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 3;
```
`deleted_files` lists deleted SST/Parquet file IDs. `deleted_indexes` lists deleted index file IDs and their `index_version`. `need_retry` is `true` when the Region needs another GC run.
For example:
```json
{
"deleted_files": ["580653aa-252b-415b-aaf9-ce65e9d78249"],
"deleted_indexes": [
{"file_id":"580653aa-252b-415b-aaf9-ce65e9d78249","index_version":0}
],
"need_retry": false
}
```
Scheduled GC does not emit a `Submitted` row. A manual `Submitted` row contains the configuration. If no Region needs cleanup or retry, no `Done` row is written. `payload` is JSON `null` on a report row.
## Remote WAL pruning
```sql
SELECT timestamp, type, topic_name, prunable_entry_id, latest_offset,
procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
json_to_string(payload) AS payload
FROM greptime_private.events
WHERE type = 'wal_prune'
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 3;
```
```sql
+-------------------------------+-----------+-------------------------+-------------------+---------------+-----------------+--------------+--------------------------------------+
| timestamp | type | topic_name | prunable_entry_id | latest_offset | procedure_state | trigger_type | payload |
+-------------------------------+-----------+-------------------------+-------------------+---------------+-----------------+--------------+--------------------------------------+
| 2026-08-10 11:23:28.495676149 | wal_prune | greptimedb_wal_topic_20 | 248376 | 248381 | Done | Succeeded | {"logical_delete":false,"version":1} |
| 2026-08-10 10:53:28.478381545 | wal_prune | greptimedb_wal_topic_20 | 248375 | 248376 | Done | Succeeded | {"logical_delete":false,"version":1} |
| 2026-08-10 10:23:28.486761751 | wal_prune | greptimedb_wal_topic_20 | 248371 | 248375 | Done | Succeeded | {"logical_delete":false,"version":1} |
+-------------------------------+-----------+-------------------------+-------------------+---------------+-----------------+--------------+--------------------------------------+
```
`prunable_entry_id` is the pruning boundary for `topic_name`. It is the smallest boundary reported by Regions that use the topic, so none of those Regions needs entries before it. On a non-`Succeeded` row, it is only the attempted boundary.
When present, `latest_offset` is the Kafka latest offset observed during this attempt. It is an exclusive upper bound: the offset after the last record, not the offset of the last record.
The recorder omits `Submitted` and `Recovered` rows. A successful run writes `Done`; `Retrying` is not terminal, and a run can also write `Failed` or `Poisoned`.
---
## Events
The `greptime_private.events` table stores events recorded while GreptimeDB runs. It helps operators inspect DDL changes and background procedures without searching service logs.
GreptimeDB writes event records asynchronously on a best-effort basis. Records are
normally flushed every five seconds, so a recent operation might not be visible
immediately. Event recording should not be treated as the operation's success
acknowledgement.
## Configure event recording
See [Event recording](/user-guide/deployments-administration/configuration.md#event-recording) for configuration options and the event types supported by standalone, Frontend, and Metasrv.
Standalone deployments record supported local DDL Procedure events. Distributed deployments with Metasrv can additionally record operational event types.
## Query events
The table is created when the first event is recorded. If no event has been
recorded yet, or event recording is disabled, this query returns a table-not-found
error.
Start with a bounded query while investigating a recent operation:
```sql
SELECT timestamp, type
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
Use the following pages for focused queries and event details:
- [Query events](/user-guide/deployments-administration/monitoring/events/query-events.md)
- [Event data model](/user-guide/deployments-administration/monitoring/events/event-data-model.md)
- [DDL events](/user-guide/deployments-administration/monitoring/events/ddl-events.md)
- [Region events](/user-guide/deployments-administration/monitoring/events/region-events.md)
- [Maintenance events](/user-guide/deployments-administration/monitoring/events/maintenance-events.md)
---
## Query events
Query the `greptime_private.events` system table to investigate recent events.
Events are written asynchronously, so a newly submitted operation might not be
visible immediately. See [Event data model](/user-guide/deployments-administration/monitoring/events/event-data-model.md)
for event columns.
## Start with recent events
The following query returns all columns for up to 20 events recorded during the
last hour:
```sql
SELECT *
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
This is useful for exploration, but it returns every column. For routine checks,
select only the columns you need:
```sql
SELECT timestamp, type, json_to_string(payload) AS payload
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
The compact result keeps the time, type, and payload visible without pasting the
full output.
## Discover and filter event types
List types actually present in the cluster before choosing a filter:
```sql
SELECT type, COUNT(*) AS event_rows
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
GROUP BY type
ORDER BY type;
```
The result is a point-in-time view of the event types currently present in the
cluster. It varies with workload and does not define the configured or
source-supported types. See [DDL events](/user-guide/deployments-administration/monitoring/events/ddl-events.md)
for the supported local DDL event types.
## Filter events by actor
Use `actor` to list recent events recorded for a database user:
```sql
SELECT timestamp, type, actor, json_to_string(payload) AS payload
FROM greptime_private.events
WHERE actor = ''
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
See [Event actor](/user-guide/deployments-administration/monitoring/events/event-data-model.md#actor)
for how GreptimeDB determines this value.
## Query ADMIN function events
An `admin_function` event records the function name, the current database user,
the immediate status, the input arguments, and the immediate output:
```sql
SELECT timestamp,
actor,
admin_function_name,
admin_function_status,
json_to_string(payload) AS payload,
json_to_string(admin_function_output) AS output
FROM greptime_private.events
WHERE type = 'admin_function'
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
For a successful function, `output` contains a `result`. For a failed function,
it contains an `error`:
```text
| actor | admin_function_name | admin_function_status | output |
| root | flush_table | Succeeded | {"result":0} |
| root | unknown_function | Failed | {"error":"..."} |
```
Combine an event type with a database and object name to avoid unrelated rows:
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type
FROM greptime_private.events
WHERE type = 'create_table'
AND timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
ORDER BY timestamp;
```
Example output:
```sql
+-------------------------------+--------------+-----------------+--------------+
| timestamp | type | procedure_state | trigger_type |
+-------------------------------+--------------+-----------------+--------------+
| 2026-08-10 11:28:40.590240203 | create_table | Running | Submitted |
| 2026-08-10 11:28:40.659064297 | create_table | Done | Succeeded |
+-------------------------------+--------------+-----------------+--------------+
```
## Find the latest event for an object
Replace the placeholders with the object name and, where needed, the database.
Each query returns the newest matching event.
### Database
```sql
SELECT timestamp, type, schema_name, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND type IN ('create_database', 'alter_database', 'drop_database')
ORDER BY timestamp DESC
LIMIT 1;
```
### Table
```sql
SELECT timestamp, type, schema_name, table_name, table_id, procedure_state
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
ORDER BY timestamp DESC
LIMIT 1;
```
### Flow
```sql
SELECT timestamp, type, flow_name, flow_id, procedure_state
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND flow_name = ''
ORDER BY timestamp DESC
LIMIT 1;
```
### View
```sql
SELECT timestamp, type, schema_name, view_name, view_id, procedure_state
FROM greptime_private.events
WHERE timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND view_name = ''
ORDER BY timestamp DESC
LIMIT 1;
```
### Region
Operational events that reference a Region are global, not tied to a database.
Use this query to find the latest event for a Region. It requires
`region_migration`, `batch_gc`, and `repartition_group` to have each recorded at
least one event: the table adds an event type's columns when it first records
that type. A row fills only the fields for its event type; the other selected
fields are SQL `NULL`.
```sql
SELECT timestamp, type, procedure_state,
region_id, source_region_id, target_region_id,
region_migration_trigger_reason,
region_migration_src_node_id, region_migration_dst_node_id
FROM greptime_private.events
WHERE type IN ('region_migration', 'batch_gc', 'repartition_group')
AND timestamp >= now() - INTERVAL '1' hour
AND (region_id =
OR source_region_id =
OR target_region_id = )
ORDER BY timestamp DESC
LIMIT 1;
```
## Query Procedure events
Procedure events share a `procedure_id`.
### Get a procedure ID
For a table-creation procedure, find the row whose trigger type is `Submitted`
for the given database and table:
```sql
SELECT procedure_id
FROM greptime_private.events
WHERE type = 'create_table'
AND timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
AND json_path_match(procedure_trigger, '$.type == "Submitted"')
ORDER BY timestamp DESC
LIMIT 1;
```
Example result:
```sql
+--------------------------------------+
| procedure_id |
+--------------------------------------+
| a5788f51-5726-4db7-a85e-e9afc36da557 |
+--------------------------------------+
```
Use the returned ID to query the Procedure's event rows. Filtering by
`schema_name` and `table_name` avoids selecting a procedure for another object
with a similar name.
### Query a Procedure
Use the full-row query when you need to explore every available column:
```sql
SELECT *
FROM greptime_private.events
WHERE procedure_id = ''
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp ASC;
```
For routine checks, use a focused projection:
```sql
SELECT timestamp, type, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
procedure_error, json_to_string(payload) AS payload
FROM greptime_private.events
WHERE procedure_id = ''
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp;
```
Example output from a MySQL operation:
```sql
+-------------------------------+--------------+-----------------+--------------+-----------------+------------------------------------------------------------+
| timestamp | type | procedure_state | trigger_type | procedure_error | payload |
+-------------------------------+--------------+-----------------+--------------+-----------------+------------------------------------------------------------+
| 2026-08-10 11:23:14.388632208 | create_table | Running | Submitted | | {"create_if_not_exists":false,"engine":"mito","version":1} |
| 2026-08-10 11:23:14.463992155 | create_table | Done | Succeeded | | null |
+-------------------------------+--------------+-----------------+--------------+-----------------+------------------------------------------------------------+
```
### Find failed Procedures
To list recent failed Procedures:
```sql
SELECT timestamp, type, procedure_id, procedure_state,
json_get_string(procedure_trigger, 'type') AS trigger_type,
procedure_error
FROM greptime_private.events
WHERE procedure_state IN ('Failed', 'Poisoned')
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 20;
```
---
## Region events
Metasrv records Region events for repartitioning and Region migration in distributed deployments.
A `region_id` combines a high 32-bit `table_id` with a low 32-bit `region_number`. The queries on this page return all three values together.
## Repartition
Each repartition consists of a root `repartition` Procedure and one or more `repartition_group` child Procedures. The root `Submitted` row records the table and requested partition rules. The child `Submitted` rows show how each source Region is split or merged.
Find the root Procedure:
```sql
SELECT timestamp, procedure_id, procedure_state,
catalog_name, schema_name, table_name, table_id,
json_to_string(payload) AS payload
FROM greptime_private.events
WHERE type = 'repartition'
AND timestamp >= now() - INTERVAL '1' hour
AND schema_name = ''
AND table_name = ''
AND json_path_match(procedure_trigger, '$.type == "Submitted"')
ORDER BY timestamp DESC
LIMIT 1;
```
```sql
+-------------------------------+--------------------------------------+-----------------+--------------+-------------------+-------------------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| timestamp | procedure_id | procedure_state | catalog_name | schema_name | table_name | table_id | payload |
+-------------------------------+--------------------------------------+-----------------+--------------+-------------------+-------------------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 2026-08-09 08:29:30.771436255 | 95bb4e53-e73e-40b1-b878-208b0ae2b812 | Running | greptime | repartitiondebug1 | greptime_physical_table | 1332 | {"source_partition_exprs":["namespace >= app-2"],"source_type":"partitioned","target_partition_exprs":["namespace >= app-2 AND namespace < app-26","namespace >= app-26 AND namespace < app-5","namespace >= app-5"],"timeout":"2m","version":2} |
+-------------------------------+--------------------------------------+-----------------+--------------+-------------------+-------------------------+----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
The `payload` column contains `source_type` and `target_partition_exprs`. It also contains `source_partition_exprs` for a partitioned source. Use the returned `procedure_id` to query the Region mapping:
```sql
SELECT timestamp,
parent_procedure_id AS repartition_procedure_id,
procedure_id AS repartition_group_procedure_id,
source_region_id, table_id AS source_table_id, source_region_number, source_partition_expr,
target_region_id, table_id AS target_table_id, target_region_number, target_partition_expr
FROM greptime_private.events
WHERE type = 'repartition_group'
AND parent_procedure_id = ''
AND json_path_match(procedure_trigger, '$.type == "Submitted"')
ORDER BY source_region_id, target_region_id;
```
```sql
+-------------------------------+--------------------------------------+--------------------------------------+------------------+-----------------+----------------------+-----------------------+------------------+-----------------+----------------------+-------------------------------------------+
| timestamp | repartition_procedure_id | repartition_group_procedure_id | source_region_id | source_table_id | source_region_number | source_partition_expr | target_region_id | target_table_id | target_region_number | target_partition_expr |
+-------------------------------+--------------------------------------+--------------------------------------+------------------+-----------------+----------------------+-----------------------+------------------+-----------------+----------------------+-------------------------------------------+
| 2026-08-09 08:29:30.968778963 | 95bb4e53-e73e-40b1-b878-208b0ae2b812 | 4f132d57-c402-4c9f-b707-3b9d53b32cfa | 5720896438274 | 1332 | 2 | namespace >= app-2 | 5720896438274 | 1332 | 2 | namespace >= app-2 AND namespace < app-26 |
| 2026-08-09 08:29:30.968778963 | 95bb4e53-e73e-40b1-b878-208b0ae2b812 | 4f132d57-c402-4c9f-b707-3b9d53b32cfa | 5720896438274 | 1332 | 2 | namespace >= app-2 | 5720896438275 | 1332 | 3 | namespace >= app-26 AND namespace < app-5 |
| 2026-08-09 08:29:30.968778963 | 95bb4e53-e73e-40b1-b878-208b0ae2b812 | 4f132d57-c402-4c9f-b707-3b9d53b32cfa | 5720896438274 | 1332 | 2 | namespace >= app-2 | 5720896438276 | 1332 | 4 | namespace >= app-5 |
+-------------------------------+--------------------------------------+--------------------------------------+------------------+-----------------+----------------------+-----------------------+------------------+-----------------+----------------------+-------------------------------------------+
```
For an unpartitioned table, `source_partition_expr` is SQL `NULL`. The source and target table IDs are the same because every mapping belongs to one table. To inspect a child Procedure's lifecycle, query its `procedure_id` on [Query Procedure events](/user-guide/deployments-administration/monitoring/events/query-events.md).
## Region migration
Region migration can be started manually or by Region Balancer or Region Failover. This query returns all Region rows for the latest migration event:
```sql
WITH latest_migration AS (
SELECT procedure_id, timestamp
FROM greptime_private.events
WHERE type = 'region_migration'
AND timestamp >= now() - INTERVAL '1' hour
ORDER BY timestamp DESC
LIMIT 1
)
SELECT e.timestamp, e.procedure_id, e.procedure_state,
e.region_id, e.table_id, e.region_number,
e.region_migration_trigger_reason,
e.region_migration_src_node_id AS source_datanode_id,
e.region_migration_src_peer_addr AS source_datanode_addr,
e.region_migration_dst_node_id AS target_datanode_id,
e.region_migration_dst_peer_addr AS target_datanode_addr
FROM greptime_private.events AS e
JOIN latest_migration AS latest
ON e.procedure_id = latest.procedure_id
AND e.timestamp = latest.timestamp
WHERE e.type = 'region_migration'
ORDER BY e.region_id;
```
```sql
+-------------------------------+--------------------------------------+-----------------+---------------+----------+---------------+---------------------------------+--------------------+----------------------+--------------------+----------------------+
| timestamp | procedure_id | procedure_state | region_id | table_id | region_number | region_migration_trigger_reason | source_datanode_id | source_datanode_addr | target_datanode_id | target_datanode_addr |
+-------------------------------+--------------------------------------+-----------------+---------------+----------+---------------+---------------------------------+--------------------+----------------------+--------------------+----------------------+
| 2026-08-10 02:52:50.078350103 | 63167e21-0165-4964-afef-02271baa126b | Done | 5162550689808 | 1202 | 16 | AutoRebalance | 1 | 172.16.62.49:4001 | 2 | 172.16.232.158:4001 |
+-------------------------------+--------------------------------------+-----------------+---------------+----------+---------------+---------------------------------+--------------------+----------------------+--------------------+----------------------+
```
`region_migration_trigger_reason` identifies why the migration started. The source and target columns identify the Datanodes involved.
---
## Key Logs
During operation, GreptimeDB outputs key operations and unexpected error information to logs.
You can use these logs to understand GreptimeDB's operational status and troubleshoot errors.
## Log Location
GreptimeDB components default to outputting INFO level logs to the following locations:
- Standard output
- `greptimedb_data/logs` directory under GreptimeDB's current working directory
The log output directory can also be modified through the `[logging]` section in the configuration file or the `--log-dir` startup parameter:
```toml
[logging]
dir = "/path/to/logs"
```
Log file formats are:
- `greptimedb.YYYY-MM-DD-HH` contains logs of INFO level and above
- `greptimedb-err.YYYY-MM-DD-HH` contains error logs
For example:
```bash
greptimedb.2025-04-11-06
greptimedb-err.2025-04-11-06
```
Currently, GreptimeDB components include:
- frontend
- datanode
- metasrv
- flownode
If you need to adjust log levels, such as viewing DEBUG level logs for a component, you can refer to [this document](https://github.com/GreptimeTeam/greptimedb/blob/main/docs/how-to/how-to-change-log-level-on-the-fly.md) to modify them at runtime.
## Important Logs
The following lists recommended logs to monitor:
### Error Logs
When the database is running normally and stably, it will not output error logs. If database operations encounter exceptions or panics occur, error logs will be printed. It is recommended that users check error logs from all components.
#### Panic
If the database encounters a panic, it is recommended to collect the panic logs and report them to the official team. Typical panic logs look like this, with the keyword `panicked at`:
```bash
2025-04-02T14:44:24.485935Z ERROR common_telemetry::panic_hook: panicked at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/expr/src/logical_plan/plan.rs:1035:25:
with_new_exprs for Distinct does not support sort expressions backtrace= 0: backtrace::backtrace::libunwind::trace
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backtrace-0.3.74/src/backtrace/libunwind.rs:116:5
backtrace::backtrace::trace_unsynchronized
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backtrace-0.3.74/src/backtrace/mod.rs:66:5
1: backtrace::backtrace::trace
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backtrace-0.3.74/src/backtrace/mod.rs:53:14
2: backtrace::capture::Backtrace::create
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backtrace-0.3.74/src/capture.rs:292:9
3: backtrace::capture::Backtrace::new
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/backtrace-0.3.74/src/capture.rs:257:22
4: common_telemetry::panic_hook::set_panic_hook::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/common/telemetry/src/panic_hook.rs:37:25
5: as core::ops::function::Fn>::call
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/alloc/src/boxed.rs:1984:9
std::panicking::rust_panic_with_hook
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/std/src/panicking.rs:820:13
6: std::panicking::begin_panic_handler::{{closure}}
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/std/src/panicking.rs:678:13
7: std::sys::backtrace::__rust_end_short_backtrace
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/std/src/sys/backtrace.rs:168:18
8: rust_begin_unwind
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/std/src/panicking.rs:676:5
9: core::panicking::panic_fmt
at /rustc/409998c4e8cae45344fd434b358b697cc93870d0/library/core/src/panicking.rs:75:14
10: datafusion_expr::logical_plan::plan::LogicalPlan::with_new_exprs
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/expr/src/logical_plan/plan.rs:1035:25
11: ::analyze::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/optimizer/type_conversion.rs:105:17
12: core::ops::function::impls::::call_mut
at /greptime/.rustup/toolchains/nightly-2024-12-25-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:272:13
13: core::ops::function::impls::::call_once
at /greptime/.rustup/toolchains/nightly-2024-12-25-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:305:13
14: datafusion_common::tree_node::Transformed::transform_parent
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/common/src/tree_node.rs:764:44
15: datafusion_common::tree_node::TreeNode::transform_up::transform_up_impl::{{closure}}
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/common/src/tree_node.rs:265:13
16: stacker::maybe_grow
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/stacker-0.1.17/src/lib.rs:55:9
datafusion_common::tree_node::TreeNode::transform_up::transform_up_impl
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/common/src/tree_node.rs:260:9
17: datafusion_common::tree_node::TreeNode::transform_up
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/common/src/tree_node.rs:269:9
18: datafusion_common::tree_node::TreeNode::transform
at /greptime/.cargo/git/checkouts/datafusion-11a8b534adb6bd68-shallow/2464703/datafusion/common/src/tree_node.rs:220:9
19: ::analyze
at /greptime/codes/greptime/procedure-traits/src/query/src/optimizer/type_conversion.rs:46:9
20: query::query_engine::state::QueryEngineState::optimize_by_extension_rules::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/query_engine/state.rs:195:17
21: core::iter::traits::iterator::Iterator::try_fold
at /greptime/.rustup/toolchains/nightly-2024-12-25-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:2370:21
22: query::query_engine::state::QueryEngineState::optimize_by_extension_rules
at /greptime/codes/greptime/procedure-traits/src/query/src/query_engine/state.rs:192:9
23: query::planner::DfLogicalPlanner::plan_sql::{{closure}}::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/planner.rs:119:20
24: as core::future::future::Future>::poll
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9
25: query::planner::DfLogicalPlanner::plan_sql::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/planner.rs:71:5
26: ::plan::{{closure}}::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/planner.rs:198:73
27: as core::future::future::Future>::poll
at /greptime/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9
28: ::plan::{{closure}}
at /greptime/codes/greptime/procedure-traits/src/query/src/planner.rs:195:5
...
```
### Metasrv
When the GreptimeDB cluster experiences node online/offline events,
region migration, schema changes, etc.,
metasrv will record corresponding logs.
Therefore, in addition to error logs from each component,
it is also recommended to monitor the following metasrv log keywords:
#### Metasrv Leader Step Down/Election
```bash
// Error level, indicates current leader stepped down, new election will follow. Note the {:?} part is the leader identifier
"Leader :{:?} step down"
```
#### Region Lease Renewal Failure
```bash
// Warning level, indicates a region lease renewal was denied. Region lease requests will be rejected when the region is not properly closed/scheduled on a datanode.
Denied to renew region lease for datanode: {datanode_id}, region_id: {region_id}
```
```bash
// Info level, datanode receives lease renewal failure and attempts to close target region
Closing staled region:
```
#### Region Failover
```bash
// Warning level, detects some regions failed Phi health check, need to execute failover operation. The datanode ID and region IDs will be printed after the colon
Detects region failures on datanode:
// A region migration failed
Failed to wait region migration procedure
// Info level, when maintenance mode is enabled, failover procedure will be skipped
Maintenance mode is enabled, skip failover
```
#### Region Migration
```bash
// Info level, indicates a region starts migration. Region information will be printed after the log
Starting region migration procedure
// Error level, a region migration failed
Failed to wait region migration procedure
```
#### Procedure
Metasrv internally uses a component called "procedure" to execute distributed operations. You can monitor error logs from this component:
```bash
Failed to execute procedure
```
#### Flow Creation Failure
When flow creation fails, the failure reason can usually be seen in procedure error logs. Logs may contain the following keywords:
```bash
Failed to execute procedure metasrv-procedure:: CreateFlow
```
### Datanode
#### Compaction
When compaction starts and ends, datanode will log the following information:
```bash
2025-05-16T06:01:08.794415Z INFO mito2::compaction::task: Compacted SST files, region_id: 4612794875904(1074, 0), input: [FileMeta { region_id: 4612794875904(1074, 0), file_id: FileId(a29500fb-cae0-4f3f-8376-cb3f14653378), time_range: (1686455010000000000::Nanosecond, 1686468410000000000::Nanosecond), level: 0, file_size: 45893329, available_indexes: [], index_file_size: 0, num_rows: 5364000, num_row_groups: 53, sequence: Some(114408000) }, FileMeta { region_id: 4612794875904(1074, 0), file_id: FileId(a31dcb1b-19ae-432f-8482-9e1b7db7b53b), time_range: (1686468420000000000::Nanosecond, 1686481820000000000::Nanosecond), level: 0, file_size: 45900506, available_indexes: [], index_file_size: 0, num_rows: 5364000, num_row_groups: 53, sequence: Some(119772000) }], output: [FileMeta { region_id: 4612794875904(1074, 0), file_id: FileId(5d105ca7-9e3c-4298-afb3-e85baae3b2e8), time_range: (1686455010000000000::Nanosecond, 1686481820000000000::Nanosecond), level: 1, file_size: 91549797, available_indexes: [], index_file_size: 0, num_rows: 10728000, num_row_groups: 105, sequence: Some(119772000) }], window: Some(86400), waiter_num: 0, merge_time: 3.034328293s
```
```bash
2025-05-16T06:01:08.805366Z INFO mito2::request: Successfully compacted region: 4612794875904(1074, 0)
```
---
## Prometheus-Monitoring GreptimeDB Cluster
Before reading this document, ensure you understand how to [deploy a GreptimeDB cluster on Kubernetes](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md).
It is recommended to use [self-monitoring mode](cluster-monitoring-deployment.md) to monitor GreptimeDB cluster,
as it's simple to set up and provides out-of-the-box Grafana dashboards.
However, if you already have a Prometheus instance deployed in your Kubernetes cluster and want to integrate
GreptimeDB cluster metrics into it, follow the steps below.
## Check the Prometheus Instance Configuration
Ensure you have deployed the Prometheus Operator and created a Prometheus instance. For example, you can use [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to deploy the Prometheus stack. Refer to its [official documentation](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) for more details.
When deploying the Prometheus instance, ensure you set the labels used for scraping GreptimeDB cluster metrics.
For example, your existing Prometheus instance may contain the following configuration:
```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: greptime-podmonitor
namespace: default
spec:
selector:
matchLabels:
release: prometheus
# other configurations...
```
When the `PodMonitor` is deployed,
the Prometheus Operator continuously watches for pods in the `default` namespace that match all labels defined in `spec.selector.matchLabels` (in this example, `release: prometheus`).
## Enable `prometheusMonitor` for GreptimeDB Cluster
When deploying a GreptimeDB cluster using a Helm Chart,
enable the `prometheusMonitor` field in your `values.yaml` file. For example:
```yaml
prometheusMonitor:
# Enable Prometheus monitoring - this will create PodMonitor resources
enabled: true
# Configure scrape interval
interval: "30s"
# Configure labels
labels:
release: prometheus
```
**Important:** The `labels` field value (`release: prometheus`)
must match the `matchLabels` field used to create the Prometheus instance,
otherwise metrics collection won't work properly.
After configuring `prometheusMonitor`,
the GreptimeDB Operator will automatically create `PodMonitor` resources and import metrics into Prometheus at the specified `interval`.
You can check the `PodMonitor` resources with:
```
kubectl get podmonitors.monitoring.coreos.com -n ${namespace}
```
:::note
If you're not using a Helm Chart, you can manually configure Prometheus monitoring in the `GreptimeDBCluster` YAML:
```yaml
apiVersion: greptime.io/v1alpha1
kind: GreptimeDBCluster
metadata:
name: basic
spec:
base:
main:
image: greptime/greptimedb:v1.2.1
frontend:
replicas: 1
meta:
replicas: 1
backendStorage:
etcd:
endpoints:
- "etcd.etcd-cluster.svc.cluster.local:2379"
datanode:
replicas: 1
prometheusMonitor:
enabled: true
interval: "30s"
labels:
release: prometheus
```
:::
## Grafana Dashboards
You need to deploy Grafana by yourself,
then import the dashboards.
### Add Data Sources
After deploying Grafana,
refer to Grafana's [data sources](https://grafana.com/docs/grafana/latest/datasources/) documentation to add the following two type data sources:
- **Prometheus**: Name it `metrics`. This data source connects to your Prometheus instance, which collects GreptimeDB cluster monitoring metrics. Use your Prometheus instance URL as the connection URL.
- **MySQL**: Name it `information-schema`. This data source connects to your GreptimeDB cluster to access cluster metadata via the SQL protocol. If you have deployed GreptimeDB following the [Deploy a GreptimeDB Cluster on Kubernetes](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-greptimedb-cluster.md) guide, use `${cluster-name}-frontend.${namespace}.svc.cluster.local:4002` as the server address with database `information_schema`.
### Import Dashboards
The [GreptimeDB Cluster Metrics Dashboard](https://github.com/GreptimeTeam/greptimedb/tree/v1.2.1/grafana/dashboards/metrics/cluster) uses the `metrics` and `information-schema` data sources to display GreptimeDB cluster metrics.
Refer to Grafana's [Import dashboards](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/import-dashboards/) documentation to learn how to import dashboards.
---
## Monitoring
Effective database administration relies heavily on monitoring.
You can monitor GreptimeDB using the following methods:
---
## Runtime Information
The `INFORMATION_SCHEMA` database provides access to system metadata, such as the name of a database or table, the data type of a column, etc.
* Find the topology information of the cluster though [CLUSTER_INFO](/reference/sql/information-schema/cluster-info.md) table.
* Find the table regions distribution though [PARTITIONS](/reference/sql/information-schema/partitions.md) and [REGION_PEERS](/reference/sql/information-schema/region-peers.md) tables.
For example, find all the region id of a table:
```sql
SELECT greptime_partition_id FROM PARTITIONS WHERE table_name = 'monitor'
```
Find the distribution of all regions in a table:
```sql
SELECT b.peer_id as datanode_id,
a.greptime_partition_id as region_id
FROM information_schema.partitions a LEFT JOIN information_schema.region_peers b
ON a.greptime_partition_id = b.region_id
WHERE a.table_name='monitor'
ORDER BY datanode_id ASC
```
For more information about the `INFORMATION_SCHEMA` database,
Please read the [reference](/reference/sql/information-schema/overview.md).
---
## Slow Query (Experimental)
GreptimeDB provides a slow query log to help you find and fix slow queries. By default, the slow queries are output to the system table `greptime_private.slow_queries` with `30s` threshold and `1.0` sample ratio with `30d` TTL.
The schema of the `greptime_private.slow_queries` table is as follows:
```sql
+--------------+----------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------+----------------------+------+------+---------+---------------+
| cost | UInt64 | | NO | | FIELD |
| threshold | UInt64 | | NO | | FIELD |
| query | String | | NO | | FIELD |
| is_promql | Boolean | | NO | | FIELD |
| timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| promql_range | UInt64 | | NO | | FIELD |
| promql_step | UInt64 | | NO | | FIELD |
| promql_start | TimestampMillisecond | | NO | | FIELD |
| promql_end | TimestampMillisecond | | NO | | FIELD |
+--------------+----------------------+------+------+---------+---------------+
```
- `cost`: The cost of the query in milliseconds.
- `threshold`: The threshold of the query in milliseconds.
- `query`: The query string.
- `is_promql`: Whether the query is a PromQL query.
- `timestamp`: The timestamp of the query.
- `promql_range`: The range of the query. Only used when `is_promql` is `true`.
- `promql_step`: The step of the query. Only used when `is_promql` is `true`.
- `promql_start`: The start time of the query. Only used when `is_promql` is `true`.
- `promql_end`: The end time of the query. Only used when `is_promql` is `true`.
In cluster mode, you can configure the slow query in frontend configs (same as in standalone mode), for example:
```toml
[slow_query]
## Whether to enable slow query log.
enable = true
## The record type of slow queries. It can be `system_table` or `log`.
## If `system_table` is selected, the slow queries will be recorded in a system table `greptime_private.slow_queries`.
## If `log` is selected, the slow queries will be logged in a log file `greptimedb-slow-queries.*`.
record_type = "system_table"
## The threshold of slow query. It can be human readable time string, for example: `10s`, `100ms`, `1s`.
threshold = "30s"
## The sampling ratio of slow query log. The value should be in the range of (0, 1]. For example, `0.1` means 10% of the slow queries will be logged and `1.0` means all slow queries will be logged.
sample_ratio = 1.0
## The TTL of the `slow_queries` system table. Default is `30d` when `record_type` is `system_table`.
ttl = "30d"
```
If you use the Helm chart to deploy GreptimeDB, you can configure the slow query in the `values.yaml` file, for example:
```yaml
slowQuery:
enable: true
recordType: "system_table"
threshold: "30s"
sampleRatio: "1.0"
ttl: "30d"
```
If you use `log` as the record type, the slow queries will be logged in a log file `greptimedb-slow-queries.*`. By default, the log file is located in the `${data_home}/logs` directory.
---
## Standalone Monitoring
GreptimeDB standalone provides a `/metrics` endpoint on the HTTP port (default `4000`) that exposes [Prometheus metrics](/reference/http-endpoints.md#metrics).
You can use Prometheus to scrape these metrics and Grafana to visualize them.
## Grafana Dashboard Integration
GreptimeDB provides pre-built Grafana dashboards for monitoring standalone deployments.
You can access the dashboard JSON files from the [GreptimeDB repository](https://github.com/GreptimeTeam/greptimedb/tree/v1.2.1/grafana/dashboards/metrics/standalone).
---
## Tracing
GreptimeDB supports distributed tracing. GreptimeDB exports all collected spans using the gRPC-based OTLP protocol. Users can use [Jaeger](https://www.jaegertracing.io/), [Tempo](https://grafana.com/oss/tempo/) and other OTLP protocol backends that support gRPC to collect the span instrument by GreptimeDB.
In the [logging section](/user-guide/deployments-administration/configuration.md#logging-options) in the configuration, there are descriptions of configuration items related to tracing, [standalone.example.toml](https://github.com/GreptimeTeam/greptimedb/blob/v1.2.1/config/standalone.example.toml) provide a reference configuration in the logging section.
## Dynamic Tracing Control
GreptimeDB provides the ability to enable or disable tracing dynamically at runtime using the HTTP API without requiring a server restart. This is useful for troubleshooting production issues or temporarily enabling tracing for debugging purposes.
To enable tracing:
```bash
curl --data "true" http://127.0.0.1:4000/debug/enable_trace
# Output: trace enabled
```
To disable tracing:
```bash
curl --data "false" http://127.0.0.1:4000/debug/enable_trace
# Output: trace disabled
```
## Tutorial: Use Jaeger to trace GreptimeDB
[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed tracing system, originally developed and open sourced by Uber. Its goal is to help developers monitor and debug the request flow in complex microservice architectures.
Jaeger supports gRPC-based OTLP protocol, so GreptimeDB can export trace data to Jaeger. The following tutorial shows you how to deploy and use Jaeger to track GreptimeDB.
### Step 1: Deploy Jaeger
Start a Jaeger instance using the `all-in-one` docker image officially provided by jaeger:
```bash
docker run --rm -d --name jaeger \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 9411:9411 \
jaegertracing/all-in-one:latest
```
### Step 2: Deploy GreptimeDB
Write configuration files to allow GreptimeDB to perform tracing. Save the following configuration items as the file `config.toml`
```Toml
[logging]
enable_otlp_tracing = true
```
Then start GreptimeDB using standalone mode
```bash
greptime standalone start -c config.toml
```
Refer to the chapter [MySQL](/user-guide/protocols/mysql.md) on how to connect to GreptimeDB. Run the following SQL statement in MySQL Client:
```sql
CREATE TABLE host (
ts timestamp(3) time index,
host STRING PRIMARY KEY,
val BIGINT,
);
INSERT INTO TABLE host VALUES
(0, 'host1', 0),
(20000, 'host2', 5);
SELECT * FROM host ORDER BY ts;
DROP TABLE host;
```
### Step 3: Obtain trace information in Jaeger
1. Go to http://127.0.0.1:16686/ and select the Search tab.
2. Select the `greptime-standalone` service in the service drop-down list.
3. Click **Find Traces** to display trace information.
## Guide: How to configure tracing sampling rate
GreptimeDB provides many protocols and interfaces for data insertion, query and other functions. You can collect the calling chains of each operation through tracing. However, for some high-frequency operations, collecting all tracing of the operation may be unnecessary and waste storage space. At this time, you can use `tracing_sample_ratio` to set the sampling rate of tracing for various operations, which can greatly reduce the number of exported tracing and facilitate system observation.
All tracing within GreptimeDB is classified according to the protocol it is connected to and the corresponding operations of that protocol:
| **protocol** | **request_type** |
|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| grpc | inserts / query.sql / query.logical_plan / query.prom_range / query.empty / ddl.create_database / ddl.create_table / ddl.alter / ddl.drop_table / ddl.truncate_table / ddl.empty / deletes / row_inserts / row_deletes |
| mysql | |
| postgres | |
| otlp | metrics / traces / logs |
| opentsdb | |
| influxdb | write_v1 / write_v2 |
| prometheus | remote_read / remote_write / format_query / instant_query / range_query / labels_query / series_query / label_values_query |
| http | sql / promql |
You can configure different tracing sampling rates through `tracing_sample_ratio`.
```toml
[logging]
enable_otlp_tracing = true
[logging.tracing_sample_ratio]
default_ratio = 0.0
[[logging.tracing_sample_ratio.rules]]
protocol = "mysql"
ratio = 1.0
[[logging.tracing_sample_ratio.rules]]
protocol = "grpc"
request_types = ["inserts"]
ratio = 0.3
```
The above configuration formulates two sampling rules and sets a default sampling rate. GreptimeDB will start matching from the first one according to the sampling rules, and use the first matching sampling rule as the sampling rate of the tracing. If no rule matches, `default_ratio` will be used as the default sampling rate. The range of sampling rate is `[0.0, 1.0]`, `0.0` means not sampling, `1.0` means sampling all tracing.
For example, according to the rules provided above, all calls accessed using the mysql protocol will be sampled, data inserted using grpc will be sampled 30%, and all remaining tracing will not be sampled.
---
## Deployments & Administration
# Deployments & Administration
This section covers running GreptimeDB on your own infrastructure. Start with [Architecture](/user-guide/concepts/architecture.md) to see which components you will deploy and operate. GreptimeCloud runs the same engine as a managed service, with deployment and maintenance handled for you.
## Deploy
- [Configuration](configuration.md) — protocol, storage, and runtime settings to review before the first deployment.
- [Deploy on Kubernetes](./deploy-on-kubernetes/overview.md) — deployment through the GreptimeDB Operator.
- [Capacity Planning](./capacity-plan.md) — sizing compute, memory, and local cache for the expected ingestion rate and query mix.
- [Authentication](./authentication/overview.md) — not enabled by default.
- [Run on Android](run-on-android.md) — for edge deployments on Android devices.
## Operate
- [Monitoring](./monitoring/overview.md) — cluster health and performance through metrics, tracing, and runtime information.
- [Table and Region Operations](./manage-data/overview.md) — table operations, sharding, Region migration and failover, repartition, compaction, and garbage collection.
- [Metadata Storage](./manage-metadata/overview.md) — the metadata backend the cluster depends on.
- [Maintenance](./maintenance/maintenance-mode.md) — maintenance mode, recovery mode, table reconciliation, and sequence management.
- [Upgrade](./upgrade.md) — moving a deployment to a newer version.
- [Troubleshooting](./troubleshooting.md) — collecting the information needed to diagnose a problem.
## Durability and recovery
- [Write-Ahead Logging (WAL)](./wal/overview.md) — local and Remote WAL, including the [Remote WAL setup](./wal/remote-wal/configuration.md) that cluster failover depends on.
- [Disaster Recovery](./disaster-recovery/overview.md) — backup, restore, and cross-region options.
## Performance
- [Performance Tuning](./performance-tuning/performance-tuning-tips.md) — cache, query, and ingestion settings to adjust from runtime evidence.
- [Design Table Schema](./performance-tuning/design-table.md) — primary keys, indexes, append-only mode, and partitioning.
---
## Design Your Table Schema
The design of your table schema significantly impacts both write and query performance.
Before writing data,
it is crucial to understand the data types, scale, and common queries relevant to your business,
then model the data accordingly.
Designing a table comes down to a few decisions:
- which columns form the **primary key** (data ordering and time-series identity);
- whether the table is **append-only or deduplicates**, and which **merge mode** to use;
- which columns to **index**;
- how to lay out your data across tables (**one wide table vs. multiple tables**) and **how to partition** them for scale.
This guide first explains how GreptimeDB stores and reads data, then walks through each of these decisions in turn.
## Basic Concepts
Before proceeding, please review the GreptimeDB [Data Model Documentation](/user-guide/concepts/data-model.md).
### Cardinality
**Cardinality**: Refers to the number of unique values in a dataset. It can be classified as "high cardinality" or "low cardinality":
- **Low Cardinality**: Low cardinality columns usually have constant values.
The total number of unique values usually no more than 10 thousand.
For example, `namespace`, `cluster`, `http_method` are usually low cardinality.
- **High Cardinality**: High cardinality columns contain a large number of unique values.
For example, `trace_id`, `span_id`, `user_id`, `uri`, `ip`, `uuid`, `request_id`, table auto increment id, timestamps are usually high cardinality.
### Semantic Types
In GreptimeDB, columns are categorized into three semantic types: `Tag`, `Field`, and `Timestamp`.
The timestamp usually represents the time of data sampling or the occurrence time of logs/events.
GreptimeDB uses the `TIME INDEX` constraint to identify the `Timestamp` column.
So the `Timestamp` column is also called the `TIME INDEX` column.
If you have multiple columns with timestamp data type, you can only define one as `TIME INDEX` and others as `Field` columns.
In GreptimeDB, tag columns are optional.
GreptimeDB reuses the `PRIMARY KEY` constraint to define tag columns; together they identify a time series and define how rows are ordered in storage (see [How GreptimeDB Stores and Reads Data](#how-greptimedb-stores-and-reads-data)).
The main purposes of tag columns are:
1. Defining the storage ordering of data, which improves the locality of data with the same tags. If there are no tag columns, GreptimeDB sorts rows by timestamp.
2. Identifying a unique time series, so GreptimeDB can deduplicate rows under the same time series (primary key) when the table is not append-only.
3. Smoothing migration from other TSDBs that use tags or labels.
## How GreptimeDB Stores and Reads Data
Understanding how GreptimeDB stores data and executes a query makes the rest of this guide easier to follow.
Later sections refer back to the ideas introduced here.
For the engine-level details — the on-disk SST file format and the full pruning pipeline — see [Data Layout in SST Files](/contributor-guide/datanode/storage-engine.md#data-layout-in-sst-files) and [Scan Pruning](/contributor-guide/datanode/storage-engine.md#scan-pruning) in the storage engine documentation.
### Data is sorted by primary key and time
GreptimeDB stores rows sorted by `(primary key, timestamp)`.
Rows that share the same primary key form a single time series and are stored next to each other, ordered by time.
This locality is what makes scanning a single time series cheap and improves compression.
For a concrete example of how rows are laid out on disk, see [Data Layout in SST Files](/contributor-guide/datanode/storage-engine.md#data-layout-in-sst-files).
### A scan prunes data in stages
GreptimeDB stores data in immutable files (SST files).
Each file is split into row groups, and each row group keeps statistics such as the minimum and maximum value of every column.
When you run a query, GreptimeDB avoids reading data that cannot match, in increasingly precise stages:
1. **Time range**: skip whole files (and in-memory buffers) whose time range does not overlap the query's time range. This is usually the cheapest and most effective step for time-series data.
2. **Row-group min/max statistics**: within a file, skip row groups whose statistics prove that no row can match a filter.
3. **Index**: if a filtered column has an index, use it to further narrow down to specific row groups or rows.
4. **Read and filter**: read the remaining data and apply the exact filters.
Take a query over a `host_metrics` table with `PRIMARY KEY (host, region)` as an example:
```sql
SELECT ts, cpu
FROM host_metrics
WHERE host = 'host-a'
AND region = 'us-east'
AND ts >= '2024-01-01 10:00:00'
AND ts < '2024-01-01 11:00:00'
AND cpu > 0.7;
```
**Time range** removes any file whose data falls outside the one-hour window.
**Primary-key ordering** keeps all `host-a` / `us-east` rows together, so the scan reads a small contiguous slice of each remaining file instead of the whole file.
**Row-group statistics on the first primary-key column** are especially effective. Because rows are sorted by primary key, the leading primary-key column (`host`) has a tight, non-overlapping range in each row group:
| Row group | host (min..max) |
| --- | --- |
| 0 | host-a .. host-b |
| 1 | host-c .. host-f |
The filter `host = 'host-a'` can only match row group 0, so row group 1 is skipped without being read.
Choosing and ordering the primary key well turns the leading primary-key column into an effective coarse pruning key — no extra index required.
This pruning is coarse, though: GreptimeDB keeps no global index over the primary key, so it cannot jump straight to the rows for a given key. It only skips row groups by their min/max values, which works well for the leading primary-key column but does little for columns deeper in the key or for point lookups on high cardinality columns — that is what indexes are for.
Statistics on field columns help too: a row group whose `cpu` maximum is `0.6` is skipped for `cpu > 0.7`.
**Indexes** handle selective filters that ordering and statistics cannot.
For example, an index on a column lets the scan jump to the matching row groups or rows before decoding the data columns. See [Index](#index).
### Deduplication happens during the scan
GreptimeDB uses an LSM-tree storage engine: it never overwrites data in place, so multiple versions of the same row can exist across files.
For tables that are not append-only, the scan **merges and deduplicates** rows that share the same `(primary key, timestamp)` on the fly, keeping only the newest version (see [Data updating and merging](#data-updating-and-merging)).
Append-only tables skip this work.
They don't need to deduplicate, and when a query doesn't require ordered output they can be scanned in any order, which is faster.
### Takeaways
- Filtering by **time range** and by the **leading primary-key columns** is the cheapest way to make queries fast.
- **Indexes** help selective filters on columns that ordering doesn't cover.
- **Deduplication** (on non-append-only tables) adds extra work at query time.
## Primary key
The primary key is the most impactful schema decision: it defines how rows are ordered on disk and identifies a time series for deduplication.
This section helps you decide whether to set one and which columns to use.
### Primary key is optional
Bad primary key or index may significantly degrade performance.
Generally you can create an append-only table without a primary key since ordering data by timestamp is sufficient for many workloads.
It can also serve as a baseline.
```sql
CREATE TABLE http_logs (
access_time TIMESTAMP TIME INDEX,
application STRING,
remote_addr STRING,
http_status STRING,
http_method STRING,
http_refer STRING,
user_agent STRING,
request_id STRING,
request STRING,
) with ('append_mode'='true');
```
The `http_logs` table is an example for storing HTTP server logs.
- The `'append_mode'='true'` option creates the table as an append-only table.
This ensures a log doesn't override another one with the same timestamp.
- The table sorts logs by time so it is efficient to search logs by time.
### Primary key design
You can use primary key when there are suitable columns and one of the following conditions is met:
- Most queries can benefit from the ordering.
- You need to deduplicate (including delete) rows by the primary key and time index.
For example, if you always only query logs of a specific application, you may set the `application` column as primary key (tag).
```sql
SELECT request FROM http_logs WHERE application = 'greptimedb' AND access_time > now() - '5 minute'::INTERVAL;
```
The number of applications is usually limited. Table `http_logs_v2` uses `application` as the primary key.
It sorts logs by application so querying logs under the same application is faster as it only has to scan a small number of rows. Setting tags may also reduce disk space usage as it improves the locality of data.
```sql
CREATE TABLE http_logs_v2 (
access_time TIMESTAMP TIME INDEX,
application STRING,
remote_addr STRING,
http_status STRING,
http_method STRING,
http_refer STRING,
user_agent STRING,
request_id STRING,
request STRING,
PRIMARY KEY(application),
) with ('append_mode'='true');
```
A long primary key will negatively affect the insert performance and enlarge the memory footprint. It's recommended to define a primary key with no more than 5 columns.
You can put high cardinality columns such as `trace_id`, `span_id`, or `user_id` into the primary key, but it is often not the best way to filter on them.
Adding a high cardinality column to the primary key has a cost but limited benefit:
- It makes the primary key longer, which slows inserts and uses more memory.
- On deduplicating tables, deduplication by a high cardinality key is expensive. If you can tolerate duplication, use an append-only table instead.
- Unlike a [skipping index](#skipping-index), it doesn't always speed up queries much.
A skipping index targets the same filters with much lower overhead, so it is usually the better default. Add a high cardinality column to the primary key only when you genuinely need to order or deduplicate by it. Because the best layout depends on your data and queries, benchmark on your own dataset to decide.
Recommendations for tags:
- Low cardinality columns that occur in `WHERE`/`GROUP BY`/`ORDER BY` frequently.
These columns usually remain constant.
For example, `namespace`, `cluster`, or an AWS `region`.
- No need to set all low cardinality columns as tags since this may impact the performance of ingestion and querying.
- Typically use short strings and integers for tags, avoiding `FLOAT`, `DOUBLE`, `TIMESTAMP`.
- For high cardinality columns such as `trace_id`, `span_id`, and `user_id`, prefer a [skipping index](#skipping-index); add them to the primary key only when you need to order or deduplicate by them.
- Order primary key columns from the most frequently filtered and most selective leading column. The leading column benefits the most from ordering and row-group pruning (see [How GreptimeDB Stores and Reads Data](#how-greptimedb-stores-and-reads-data)).
## Deduplication and Append-Only
Next, decide how the table handles rows that share the same primary key and timestamp:
- **Append-only** (`append_mode = 'true'`): keep every row, with no deduplication or deletes. This is the fastest option.
- **Deduplicating** (the default): keep a single row per `(primary key, timestamp)`, with a **merge mode** (`last_row` or `last_non_null`) controlling how updates combine.
Choose append-only when you don't need updates or deletes (for example, logs).
Otherwise, use the default deduplicating mode, which keeps `append_mode` at `false` and removes duplicate rows by primary key and timestamp.
Because deduplication groups rows by `(primary key, timestamp)`, the primary key decides what counts as a duplicate.
Make sure the primary key uniquely identifies a time series, so that rows you want to keep separate are not merged together (see [Primary key](#primary-key)).
For example, the `system_metrics` table deduplicates rows by `host` and `ts`:
```sql
CREATE TABLE IF NOT EXISTS system_metrics (
host STRING,
cpu_util DOUBLE,
memory_util DOUBLE,
disk_util DOUBLE,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY(host),
TIME INDEX(ts)
);
```
### Data updating and merging
GreptimeDB supports two different strategies for deduplication: `last_row` and `last_non_null`.
You can specify the strategy by the `merge_mode` table option.
`merge_mode` only takes effect on deduplicating tables. Append-only tables don't merge rows, so you normally leave `merge_mode` unset on them.
GreptimeDB uses an LSM Tree-based storage engine,
which does not overwrite old data in place but allows multiple versions of data to coexist.
These versions are merged during the query process.
The default merge behavior is `last_row`, meaning the most recently inserted row takes precedence.
In `last_row` merge mode,
the latest row is returned for queries with the same primary key and time value,
requiring all Field values to be provided during updates.
For scenarios where only specific Field values need updating while others remain unchanged,
the `merge_mode` option can be set to `last_non_null`.
This mode retains the latest non-null value for each field during queries,
allowing updates to provide only the values that need to change.
Tables auto-created via the InfluxDB line protocol default to `last_non_null` to align with InfluxDB's update behavior, configurable via the [`influxdb.default_merge_mode`](/user-guide/deployments-administration/configuration.md) option.
The `last_row` merge mode doesn't have to check each individual field value so it is usually faster than the `last_non_null` mode.
:::warning Deduplication and partitioning
Deduplication and merging happen **within a single partition**.
If you partition a deduplicating table (any table that is not append-only) by a column that is **not** part of the primary key, rows with the same primary key can be spread across different partitions and won't be deduplicated against each other.
To keep deduplication correct, only use primary key columns as partition columns, so that rows with the same primary key always go to the same partition.
GreptimeDB does not enforce this — it is your responsibility.
See [Distributed Tables](#distributed-tables).
:::
### When to use append-only tables
If you don't need the following features, you can use append-only tables:
- Deduplication
- Deletion
GreptimeDB implements `DELETE` via deduplicating rows so append-only tables don't support deletion now.
Deduplication requires more computation and restricts the parallelism of ingestion and querying.
Using append-only tables usually has better query performance.
## Index
With the primary key and table mode decided, you can add indexes to speed up specific filters.
Indexes are optional — add one only when a filter is common and not already fast enough, and choose the index type based on the column (see the comparison in [When to use index](#when-to-use-index)).
A primary key and an index are complementary, not alternatives:
- The **primary key** gives every table one physical ordering. It helps range scans and locality, and is required for deduplication and deletion.
- An **index** is auxiliary and can be added to any column, whether a tag or a field. It targets selective filters (for example a point lookup on a high cardinality column) that ordering alone can't accelerate.
A single query can use both at once. For example, with a primary key on `application` and a [skipping index](#skipping-index) on `request_id`, GreptimeDB uses the time range and the `application` ordering to read a small slice of data, then uses the index on `request_id` to pinpoint the matching rows.
### Inverted Index
GreptimeDB supports inverted index that may speed up filtering low cardinality columns.
When creating a table, you can specify the [inverted index](/contributor-guide/datanode/data-persistence-indexing.md#inverted-index) columns using the `INVERTED INDEX` clause.
For example, `http_logs_v3` adds an inverted index for the `http_method` column.
```sql
CREATE TABLE http_logs_v3 (
access_time TIMESTAMP TIME INDEX,
application STRING,
remote_addr STRING,
http_status STRING,
http_method STRING INVERTED INDEX,
http_refer STRING,
user_agent STRING,
request_id STRING,
request STRING,
PRIMARY KEY(application),
) with ('append_mode'='true');
```
The following query can use the inverted index on the `http_method` column.
```sql
SELECT request FROM http_logs_v3 WHERE application = 'greptimedb' AND http_method = 'GET' AND access_time > now() - '5 minute'::INTERVAL;
```
Inverted index supports the following operators:
- `=`
- `<`
- `<=`
- `>`
- `>=`
- `IN`
- `BETWEEN`
- `~`
### Skipping Index
For high cardinality columns like `trace_id`, `request_id`, a [skipping index](/user-guide/manage-data/data-index.md#skipping-index) is often a good fit.
This method has lower storage overhead and resource usage, particularly in terms of memory and disk consumption.
Example:
```sql
CREATE TABLE http_logs_v4 (
access_time TIMESTAMP TIME INDEX,
application STRING,
remote_addr STRING,
http_status STRING,
http_method STRING INVERTED INDEX,
http_refer STRING,
user_agent STRING,
request_id STRING SKIPPING INDEX,
request STRING,
PRIMARY KEY(application),
) with ('append_mode'='true');
```
The following query can use the skipping index to filter the `request_id` column.
```sql
SELECT request FROM http_logs_v4 WHERE application = 'greptimedb' AND request_id = '25b6f398-41cf-4965-aa19-e1c63a88a7a9' AND access_time > now() - '5 minute'::INTERVAL;
```
However, note that the query capabilities of the skipping index are generally inferior to those of the inverted index.
Skipping index can't handle complex filter conditions and may have a lower filtering performance on low cardinality columns. It applies to equality predicates and to predicates that reduce to a membership test, such as `IN` and a chain of `OR` equalities. Other operators do not use the index.
### Full-Text Index
For unstructured log messages that require tokenization and searching by terms, GreptimeDB provides full-text index.
For example, the `raw_logs` table stores unstructured logs in the `message` field.
```sql
CREATE TABLE IF NOT EXISTS `raw_logs` (
message STRING NULL FULLTEXT INDEX WITH(analyzer = 'English', case_sensitive = 'false'),
ts TIMESTAMP(9) NOT NULL,
TIME INDEX (ts),
) with ('append_mode'='true');
```
The `message` field is full-text indexed using the `FULLTEXT INDEX` option.
See [fulltext column options](/user-guide/manage-data/data-index.md#full-text-index) for more information.
Storing and querying structured logs usually have better performance than unstructured logs with full-text index.
It's recommended to [use Pipeline](/user-guide/logs/use-custom-pipelines.md#create-a-custom-pipeline) to convert logs into structured logs.
### When to use index
Index in GreptimeDB is flexible and powerful.
You can create an index for any column, no matter if the column is a tag or a field.
It's meaningless to create additional index for the timestamp column.
Generally you don't need to create indexes for all columns.
Maintaining indexes may introduce additional cost and stall ingestion.
A bad index may occupy too much disk space and make queries slower.
You can use a table without additional index as a baseline.
There is no need to create an index for the table if the query performance is already acceptable.
You can create an index for a column when:
- The column occurs in the filter frequently.
- Filtering the column without an index isn't fast enough.
- There is a suitable index for the column.
The following table lists the suitable scenarios of all index types.
| | Inverted Index | Full-Text Index | Skip Index|
| ----- | ----------- | ------------- |------------- |
| Suitable Scenarios | - Filtering low cardinality columns | - Text content search | - Precise filtering high cardinality columns |
| Creation Method | - Specified using `INVERTED INDEX` |- Specified using `FULLTEXT INDEX` in column options | - Specified using `SKIPPING INDEX` in column options |
## Wide Table vs. Multiple Tables
With each table's schema settled, decide how to spread your data across tables: group columns collected together into one wide table, use separate tables for logically distinct data, and partition a table only when a single node can no longer serve it.
In monitoring or IoT scenarios, multiple metrics are often collected simultaneously.
We recommend placing metrics collected simultaneously into a single table to improve read/write throughput and data compression efficiency.
### Prometheus metrics and the metric engine
For Prometheus-style metrics, GreptimeDB relies on the [Metric Engine](/contributor-guide/datanode/metric-engine.md).
If you ingest data through the Prometheus remote write API, GreptimeDB routes it to the metric engine automatically: tables are created for you and no configuration is needed.
Under the hood, the metric engine handles Prometheus's single-value model efficiently by storing many small metric tables on a shared physical wide table.
Each metric stays its own logical table, while the shared storage improves read/write throughput and compression.
When you scale out to a cluster, one detail matters: partitioning.
By default, the metric engine uses a single physical table with **only one partition**, which is enough for most workloads.
In a cluster, however, that means a single datanode handles all ingestion.
To spread the load across nodes, create your own partitioned physical table on a suitable label such as `namespace`.
See [GreptimeDB cluster with metric engine](/user-guide/ingest-data/for-observability/prometheus.md#greptimedb-cluster-with-metric-engine) for an example.
### Multiple tables vs. multiple partitions
Splitting data into multiple tables and partitioning a single table solve different problems and can be combined:
- Use **multiple tables** when data is logically distinct: different schemas, different sets of columns, or different retention (TTL) requirements. Separate tables also let each table use its own primary key, indexes, and merge mode, whereas a single wide table has to share one of each across all its columns. They keep each schema clean and let you tune and retain each table independently. The cost is more tables: because each table maintains its own files, splitting data across many small tables produces many small files.
- Use **multiple partitions** (a distributed table) when a single table grows too large for one node to serve. Partitioning splits one table's rows across nodes for horizontal scaling while keeping the data in a single table, so it spreads load without creating as many small files as many separate tables would. See [Distributed Tables](#distributed-tables).
In short: split into separate tables for logical separation, and into partitions for scale.
## Distributed Tables
GreptimeDB supports partitioning data tables to distribute read/write hotspots and achieve horizontal scaling.
This section continues from the layout discussion above and helps you decide whether to partition a table, how many partitions to create, and which columns to partition on.
### Two misunderstandings about distributed tables
At the storage layer, GreptimeDB automatically partitions data based on the TIME INDEX column.
Therefore, it is unnecessary and not recommended for you to partition data by time
(e.g., one partition per day or one table per week).
GreptimeDB uses columnar storage,
so partitioning a table refers to horizontal partitioning by rows,
with each partition containing all columns.
### When to Partition and Determining the Number of Partitions
A table can utilize all the resources in the machine, especially during query.
Partitioning a table may not always improve the performance:
- A distributed query plan isn't always as efficient as a local query plan.
- Distributed query may introduce additional data transmission across the network.
There is no need to partition a table unless a single machine isn't enough to serve the table.
For example:
- There is not enough local disk space to store the data or to cache the data when using object stores.
- You need more CPU cores to improve the query performance or more memory for costly queries.
- The disk throughput becomes the bottleneck.
- The ingestion rate is larger than the throughput of a single node.
GreptimeDB releases a [benchmark report](https://github.com/GreptimeTeam/greptimedb/tree/v1.2.1/docs/benchmarks/tsbs) with each major version update,
detailing the ingestion throughput of a single partition.
Use this report alongside your target scenario to estimate if the write volume approaches the single partition's limit.
To estimate the total number of partitions,
consider the write throughput and reserve an additional 50% resource of CPU
to ensure query performance and stability. Adjust this ratio as necessary.
You can reserve more CPU cores if there are more queries.
### How to choose partition columns
GreptimeDB employs expressions to define partitioning rules.
For optimal performance, select partition columns that are:
- **Evenly distributed**: each partition should receive a similar share of data, so no single partition becomes a hotspot. The column should have enough distinct values to divide the data.
- **Stable**: the value should not change for a given entity over time.
- **Aligned with query conditions**: the column should appear in the filters of your common queries, so a query can be routed to a small number of partitions instead of all of them.
Examples include:
- Partitioning by the prefix of a trace id.
- Partitioning by data center name.
- Partitioning by business name.
For instance, if most queries target data from a specific data center,
using the data center name as a partition column is appropriate.
If the data distribution is not well understood, perform aggregate queries on existing data to gather relevant information.
:::warning Partition columns and deduplication
If your table is **not append-only** (deduplication is enabled), the partition columns **must be a subset of the primary key columns**.
Deduplication and merging only happen within a partition, so rows with the same primary key must always be routed to the same partition. Partitioning on a non-primary-key column would scatter rows with the same primary key across partitions and break deduplication.
Append-only tables don't deduplicate, so they can be partitioned by any column.
See [Data updating and merging](#data-updating-and-merging).
:::
For more details, refer to the [table partition guide](/user-guide/deployments-administration/manage-data/table-sharding.md#partition).
## SST format
This is an upgrade-only note; you don't normally need it when designing a new table.
GreptimeDB stores data in SST files using the `flat` format by default. It works well across all primary key cardinalities — including high cardinality ones such as `trace_id` or `uuid` — so you are **not expected to set the `sst_format` option manually**.
The only reason to set `sst_format` is when upgrading from an old version of GreptimeDB. Older versions used a legacy `primary_key` format as the default. If you upgraded from such a version, or are not sure which format a table currently uses, switch it to `flat`:
```sql
ALTER TABLE http_logs_v2 SET 'sst_format' = 'flat';
```
See [SST formats](/reference/sql/create.md#create-a-table-with-sst-format) for more details.
---
## Performance Tuning Tips
A GreptimeDB instance's default configuration may not fit all use cases. It's important to tune the database configurations and usage according to the scenario.
Performance tuning is not limited to server configuration. Table schema and index design directly affect ingestion and query efficiency. See [Design Your Table Schema](/user-guide/deployments-administration/performance-tuning/design-table.md) for guidance on primary keys, append-only and deduplicating tables, index selection, table layout, and partitioning.
GreptimeDB provides various metrics to help monitor and troubleshoot performance issues. The official repository provides [Grafana dashboard templates](https://github.com/GreptimeTeam/greptimedb/tree/main/grafana) for both standalone and cluster modes.
## Query
### Analyze query performance
To debug a slow query, run it with `EXPLAIN ANALYZE VERBOSE`:
```sql
EXPLAIN ANALYZE VERBOSE ;
```
`EXPLAIN ANALYZE` executes the query and collects runtime metrics from each execution stage. The `VERBOSE` option adds scan-level metrics that are useful for identifying why the query is slow and what can improve it. See [EXPLAIN](/reference/sql/explain.md) for the meaning of the metrics in the output.
When comparing repeated runs, remember that cache state may affect the timing and result of the second run. If you want to avoid measuring a fully cached repeat, change the query condition slightly while keeping the query shape equivalent, for example by shifting the time range.
If the cause is still unclear, include the full `EXPLAIN ANALYZE VERBOSE` output when creating an issue.
Common findings:
- **Many small files to search**: A high `num_file_ranges` value, especially when many files contain less than one full row group, can mean the scan has to open many small SST files. This often comes from backfilling many different time windows or from high compaction pressure. Consider adjusting `compaction.twcs.time_window`, reviewing the write or backfill pattern so rows fill time windows more naturally, and checking compaction status. Small files do not always hurt performance if the number of files is not large.
- **Many rows filtered after scanning**: If `scan_cost` is high and `rows_precise_filtered` is also high, the scan reads many candidate rows and then removes them with exact filters. See [Add indexes to improve filtering](#add-indexes-to-improve-filtering).
- **Local cache misses during data fetch**: If `fetch_metrics.cache_miss`, `fetch_metrics.pages_to_fetch_store`, or `fetch_metrics.store_fetch_elapsed` is high, GreptimeDB is reading page data from object storage instead of local caches. Check the cache metrics below and consider increasing `write_cache_size` or `page_cache_size`.
- **High scan preparation cost**: High `build_parts_cost` or `build_reader_cost` means GreptimeDB spends significant time building scan ranges or readers. Correlate this with `num_file_ranges`, metadata cache misses, and cache pressure.
- **High metadata load time**: If `metadata_cache_metrics.metadata_load_cost` or `metadata_cache_metrics.cache_miss` is high, the SST metadata cache may be too small. Check `greptime_mito_cache_bytes{type="sst_meta"}` and consider increasing `sst_meta_cache_size`.
### Add indexes to improve filtering
Indexes can reduce the amount of data that GreptimeDB reads and filters. If `EXPLAIN ANALYZE VERBOSE` shows high `scan_cost` and many `rows_precise_filtered`, compare the query's `WHERE` predicates with the table schema. Columns that occur frequently in selective filters and are not already covered by the leading primary key columns are good index candidates.
Choose the index type based on the query and the column:
- Use an inverted index for filters on low-cardinality columns, including equality, range, and `IN` predicates.
- Use a skipping index for equality filters on high-cardinality columns such as trace IDs and request IDs.
- Use a full-text index for searches over unstructured text.
Indexes have storage and memory costs and can increase flush and compaction latency, so avoid indexing every column. See [Index in the table design guide](/user-guide/deployments-administration/performance-tuning/design-table.md#index) for how indexes complement primary key ordering and how to select an index type. See [Data Index](/user-guide/manage-data/data-index.md) for index syntax and management.
After adding an index, rerun the same query with `EXPLAIN ANALYZE VERBOSE` and compare `scan_cost` and `rows_precise_filtered`. Index pruning also appears in the `rg_*_filtered` and `rows_*_filtered` counters for the corresponding index type. Currently, a newly added index only takes effect for newly flushed data: the index is built for SST files produced by subsequent flushes, but is not added to existing SST files.
### Metrics
The following metrics help diagnose query performance issues:
| Metric | Type | Description |
|---|---|---|
| greptime_mito_read_stage_elapsed_bucket | histogram | The elapsed time of different phases of a query in the storage engine. |
| greptime_mito_cache_bytes | gauge | Size of cached contents. The `type` label indicates the cache type. |
| greptime_mito_cache_hit | counter | Total count of cache hit. The `type` label indicates the cache type. |
| greptime_mito_cache_miss | counter | Total count of cache miss. The `type` label indicates the cache type. |
| greptime_mito_cache_eviction | counter | Total count of cache eviction. The `type` label indicates the cache type. |
### Enlarging cache size
Use `greptime_mito_cache_bytes{type="..."}` as the primary signal for cache pressure. If the metric is often close to the configured capacity of the corresponding cache, consider increasing that cache size. If the metric stays far below capacity, consider reducing that cache size to free memory or disk space for other caches. Use `greptime_mito_cache_hit`, `greptime_mito_cache_miss`, and `greptime_mito_cache_eviction` as supporting signals to judge whether resizing a cache is useful.
The following example lists the main cache size configurations. Some default cache sizes are adjusted according to system memory. See [Configuration](/user-guide/deployments-administration/configuration.md#region-engine-options) for the default values of these options.
```toml
[[region_engine]]
[region_engine.mito]
# Cache size for the write cache. The `type` label value is `file` for data files.
write_cache_size = "10G"
# Percentage of write cache capacity allocated to index (puffin) files.
# The `type` label value for this cache is `index`.
index_cache_percent = 20
# Cache size for SST metadata. The `type` label value for this cache is `sst_meta`.
sst_meta_cache_size = "128MB"
# Cache size for vectors and arrow arrays. The `type` label value for this cache is `vector`.
vector_cache_size = "512MB"
# Cache size for pages of SST row groups. The `type` label value for this cache is `page`.
page_cache_size = "512MB"
# Cache size for time series selector (e.g. `last_value()`). The `type` label value for this cache is `selector_result`.
selector_result_cache_size = "512MB"
# Cache size for flat range scan results. The `type` label value for this cache is `range_result`.
range_result_cache_size = "512MB"
# Cache size for prefilter results. The `type` label value for this cache is `prefilter_result`.
prefilter_result_cache_size = "128MB"
# Cache size for manifest files. The `type` label value for this cache is `manifest`.
manifest_cache_size = "256MB"
[region_engine.mito.index]
# Cache size for index metadata. The `type` label value for this cache is `index_metadata`.
metadata_cache_size = "64MiB"
# Cache size for index content. The `type` label value for this cache is `index_content`.
content_cache_size = "128MiB"
# Page size for the index content cache.
content_cache_page_size = "64KiB"
# Cache size for index query results. The `type` label value for this cache is `index_result`.
result_cache_size = "128MiB"
# The max capacity of the index staging directory. The `type` label value is `index_staging`.
staging_size = "2GB"
```
Some tips:
- 1/10 of disk space for the write cache at least. It's recommended to use a large write cache when using object storage.
- The write cache is split between data files and index (puffin) files by `index_cache_percent`. If `greptime_mito_cache_bytes{type="index"}` is full but `greptime_mito_cache_bytes{type="file"}` is not, consider increasing `index_cache_percent`.
- If a cache is consistently small compared with its configured capacity, reduce that cache size and allocate the resource to caches with higher pressure.
- 1/4 of total memory for the `page_cache_size` at least if the memory usage is under 20%
- Double the cache size if the cache hit ratio is less than 50%
- Tune `index.staging_size` only for index search workloads where `greptime_mito_cache_bytes{type="index_staging"}` approaches capacity, or staging misses and evictions show pressure. It is not necessary to enlarge this setting for every deployment.
### SST format
GreptimeDB stores data in SST files using the `flat` format by default. It works well across all primary key cardinalities, including high cardinality ones such as `trace_id` or `uuid`, so you are not expected to set the `sst_format` option manually. For tables with high cardinality primary keys, also consider using an [append-only table](/reference/sql/create.md#create-an-append-only-table) to further improve performance.
The only reason to set `sst_format` is when upgrading from an old version of GreptimeDB, whose default was a legacy `primary_key` format. In that case, switch such tables to `flat`. See the [SST format](/user-guide/deployments-administration/performance-tuning/design-table.md#sst-format) guide for how to switch the format on existing tables, and [SST formats](/reference/sql/create.md#create-a-table-with-sst-format) for the reference.
### Using append-only table if possible
In general, append-only tables have a higher scan performance as the storage engine can skip merging and deduplication. What's more, the query engine can use statistics to speed up some queries if the table is append-only.
We recommend enabling the [append_mode](/reference/sql/create.md#create-an-append-only-table) for the table if it doesn't require deduplication or performance is prioritized over deduplication. For example, a log table should be append-only as log messages may have the same timestamp.
### Disable Write-Ahead-Log(WAL)
If you are consuming and writing to GreptimeDB from replayable data sources such as Kafka, you can further improve write throughput by disabling WAL.
Please note that when WAL is disabled, unflushed data to disk or object storage will not be recoverable and will need to be restored from the original data source, such as re-reading from Kafka or re-fetching logs.
Disable WAL by setting the table option `skip_wal='true'`:
```sql
CREATE TABLE logs(
message STRING,
ts TIMESTAMP TIME INDEX
) WITH (skip_wal = 'true');
```
## Ingestion
### Batching rows
Batching means sending multiple rows to the database over the same request. This can significantly improve ingestion throughput. A recommended starting point is 1000 rows per batch. You can enlarge the batch size if latency and resource usage are still acceptable.
### Writing by time window
Although GreptimeDB can handle out-of-order data, it still affects performance. GreptimeDB infers a time window size from ingested data and partitions the data into multiple time windows according to their timestamps. If the written rows are not within the same time window, GreptimeDB needs to split them, which affects write performance.
Generally, real-time data doesn't have the issues mentioned above as they always use the latest timestamp. If you need to import data with a long time range into the database, we recommend creating the table in advance and [specifying the compaction.twcs.time_window option](/reference/sql/create.md#create-a-table-with-custom-compaction-options).
### Metrics
The following metrics help diagnose ingestion issues. Most of these metrics are available in the official Grafana dashboards. Use the metric names below when you need custom PromQL or deeper investigation.
| Metric | Type | Description |
| -------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------- |
| greptime_table_operator_ingest_rows | counter | The number of rows ingested by the table operator. Use the rate of this metric to track total write load. |
| greptime_servers_http_requests_elapsed_bucket | histogram | HTTP request latency. Use labels such as `path`, `method`, and `code` to find write-related latency. |
| greptime_servers_grpc_requests_elapsed_bucket | histogram | gRPC request latency. Use labels such as `path` and `code` to find write-related latency. |
| greptime_mito_handle_request_elapsed_bucket | histogram | The elapsed time of handling storage engine requests on datanodes. |
| greptime_mito_write_stage_elapsed_bucket | histogram | The elapsed time of different phases of processing a write request in the storage engine. |
| greptime_mito_write_buffer_bytes | gauge | The current estimated bytes allocated for the write buffer (memtables). |
| greptime_mito_write_rows_total | counter | The number of rows written to the storage engine. Use it to compare write load across datanodes. |
| greptime_mito_write_stalling_count | gauge | The number of write requests currently stalled in each worker. |
| greptime_mito_write_stall_total | counter | The total number of write requests stalled due to high memory pressure or transient region states. |
| greptime_mito_write_reject_total | counter | The number of write requests rejected due to high memory pressure. |
| raft_engine_sync_log_duration_seconds_bucket | histogram | The elapsed time of flushing the WAL to the disk. |
| greptime_mito_flush_requests_total | counter | The number of scheduled flush requests. |
| greptime_mito_flush_elapsed | histogram | The elapsed time of flushing SST files. |
| greptime_mito_flush_bytes_total | counter | The number of bytes flushed to SST files. |
| greptime_mito_flush_file_total | counter | The number of SST files produced by flush jobs. |
### Check ingestion throughput and request latency
Use `rate(greptime_table_operator_ingest_rows[$__rate_interval])` to observe the total ingestion rate. For protocol-level latency, check `greptime_servers_http_requests_elapsed_bucket` and `greptime_servers_grpc_requests_elapsed_bucket`. Filter by labels such as `path`, `method`, and `code` to separate write requests from health checks, metrics scraping, and query requests.
If frontend protocol latency is high, compare it with datanode-side metrics such as `greptime_mito_handle_request_elapsed_bucket` and `greptime_mito_write_stage_elapsed_bucket`. This helps determine whether the bottleneck is in request handling before the storage engine or inside the datanode write path.
### Diagnose datanode write pressure
When datanode writes are slow or ingestion latency is high, first check whether the write buffer is under pressure:
- `greptime_mito_write_buffer_bytes`
- `greptime_mito_write_stall_total`
- `greptime_mito_write_stalling_count`
- `greptime_mito_write_reject_total`
Write stalls mean GreptimeDB is applying backpressure instead of accepting writes immediately. Stalls can happen when the global write buffer reaches `global_write_buffer_size`, when a region reaches its effective per-region stall threshold, or when a region is temporarily not writable during internal state changes. If clients receive an error like `Engine write buffer is full, rejecting write requests`, the datanode has reached either the global reject threshold controlled by `global_write_buffer_reject_size` or the per-region reject threshold controlled by `write_buffer_size` or `default_region_write_buffer_size`.
When a datanode is under write pressure, check flush performance and write distribution before tuning the write buffer size. Increasing the write buffer only gives the datanode more memory headroom. It does not fix slow flushes or an unbalanced table that sends most writes to one region.
### Check flush performance
If the write buffer keeps growing or writes stall frequently, check whether flush is the bottleneck. Use `greptime_mito_flush_elapsed` to inspect flush latency, and use `greptime_mito_flush_requests_total`, `greptime_mito_flush_bytes_total`, and `greptime_mito_flush_file_total` to understand flush frequency and throughput.
Datanode logs also help identify slow flushes and hot regions. Search for `Successfully flush memtables`. The log line includes the region, flush reason, SST files, `total_rows`, `total_bytes`, `cost`, encoded part count, and flush metrics. You can estimate flush throughput by dividing `total_rows` or `total_bytes` by `cost`.
If flush elapsed time is high, for example 30 seconds or more, or the flush throughput is lower than the ingestion speed, increasing the write buffer size usually only delays stalls or rejects. In this case, review the table partitioning and region distribution so flush work can be spread across more regions and datanodes.
If flush jobs are fast, usually finishing in several seconds, but the flush request rate is high and write buffer pressure appears often, increasing the write buffer size can help reduce flush frequency.
### Check write distribution
For partitioned tables, check whether writes are evenly distributed across datanodes and regions. Use `greptime_mito_write_rows_total` to compare datanode-level write rows, and `greptime_mito_handle_request_elapsed_bucket` to compare datanode request latency.
You can also query [`REGION_STATISTICS`](/reference/sql/information-schema/region-statistics.md) to inspect region-level statistics such as `written_bytes_since_open` and `memtable_size`. If one datanode or one region receives most of the write load, adjust the table partitioning so the workload is spread across more regions and datanodes before increasing the write buffer size.
### Tune write buffer size
If stalls or rejects are caused by write buffer pressure, and flush performance and write distribution are healthy, consider increasing `region_engine.mito.global_write_buffer_size`. This is most useful for workloads with large rows, such as logs or traces. By default, `global_write_buffer_size` is auto-sized to 1/8 of OS memory with a maximum of 1GB. Increase it gradually, for example by 2x to 4x, and watch datanode memory usage.
In most cases, leave `region_engine.mito.global_write_buffer_reject_size` unset so GreptimeDB uses the default reject threshold of 2x `global_write_buffer_size`. If you need writes to fail earlier under memory pressure, set it manually to a deliberate margin, commonly 1.5x to 2x `global_write_buffer_size`, based on available datanode memory and how early you want requests to be rejected. The value must be greater than `global_write_buffer_size`; otherwise, GreptimeDB sanitizes it back to 2x.
For hot-region protection, configure a per-table `write_buffer_size` or set `region_engine.mito.default_region_write_buffer_size` as the cluster default. The effective value is the stall threshold: GreptimeDB schedules a flush when mutable memtable usage reaches half the value, stalls writes at the value, and rejects writes at twice the value, while other regions on the same worker can continue accepting writes. A table-level `write_buffer_size` takes precedence over the engine default. An explicit table value of `0` disables the per-region limit, while unsetting the table option falls back to the engine default. The default value of `default_region_write_buffer_size` is `0`, which disables default per-region limits.
Example:
```toml
[[region_engine]]
[region_engine.mito]
global_write_buffer_size = "2GB"
# Optional. Leave unset unless you need a custom reject margin.
global_write_buffer_reject_size = "3GB"
# Optional. Use 0 to disable the default per-region limit.
default_region_write_buffer_size = "512MB"
```
To override the region limit for a specific table:
```sql
ALTER TABLE monitor SET 'write_buffer_size'='1GB';
```
## Schema
Schema design affects both query and ingestion performance. This section describes one optimization for metric tables. For guidance on primary keys, indexes, append-only and deduplicating tables, and partitioning, see [Design Your Table Schema](/user-guide/deployments-administration/performance-tuning/design-table.md).
### Using multiple fields
While designing the schema, we recommend putting related metrics that can be collected together in the same table. This can also improve the write throughput and compression ratio.
For example, the following three tables collect the CPU usage metrics.
```sql
CREATE TABLE IF NOT EXISTS cpu_usage_user (
hostname STRING NULL,
usage_value BIGINT NULL,
ts TIMESTAMP(9) NOT NULL,
TIME INDEX (ts),
PRIMARY KEY (hostname)
);
CREATE TABLE IF NOT EXISTS cpu_usage_system (
hostname STRING NULL,
usage_value BIGINT NULL,
ts TIMESTAMP(9) NOT NULL,
TIME INDEX (ts),
PRIMARY KEY (hostname)
);
CREATE TABLE IF NOT EXISTS cpu_usage_idle (
hostname STRING NULL,
usage_value BIGINT NULL,
ts TIMESTAMP(9) NOT NULL,
TIME INDEX (ts),
PRIMARY KEY (hostname)
);
```
We can merge them into one table with three fields.
```sql
CREATE TABLE IF NOT EXISTS cpu (
hostname STRING NULL,
usage_user BIGINT NULL,
usage_system BIGINT NULL,
usage_idle BIGINT NULL,
ts TIMESTAMP(9) NOT NULL,
TIME INDEX (ts),
PRIMARY KEY (hostname)
);
```
## Collect profiling data
When GreptimeDB has high CPU or memory usage and the cause is unclear, collect profiling data from the affected node and provide it to developers for further analysis.
Profiling is platform-specific and may be unavailable on some platforms or builds. CPU profiling is available on most Unix platforms. Memory profiling is available on Linux, but not available on macOS or Windows. Generating flamegraphs can consume extra memory because GreptimeDB needs to parse the profile and symbolize stack information in process. Be careful when collecting flamegraphs from nodes that are close to their memory limit.
Profiling is node-local. Replace `127.0.0.1:4000` in the following examples with the HTTP address of the GreptimeDB node that has high CPU or memory usage.
### CPU profiling
Collect a 10-second CPU flamegraph while CPU usage is high:
```bash
curl -X POST -s 'http://127.0.0.1:4000/debug/prof/cpu?seconds=10&output=flamegraph' > greptime-cpu.svg
```
### Memory profiling
Memory profiling requires heap profiling to be enabled. Official Docker images enable heap profiling by default. For self-built or non-Docker deployments, we recommend starting GreptimeDB with heap profiling enabled by default:
```bash
MALLOC_CONF=prof:true ./greptime standalone start
```
Collect a memory flamegraph whenever memory usage is high, especially if it is still increasing or has reached a new peak:
```bash
curl -X POST -s 'http://127.0.0.1:4000/debug/prof/mem?output=flamegraph' > greptime-mem.svg
```
You can also collect two memory profiles at different times and compare them to identify increased allocations:
```bash
curl -X POST -s 'http://127.0.0.1:4000/debug/prof/mem?output=proto' > greptime-mem-before.pprof
curl -X POST -s 'http://127.0.0.1:4000/debug/prof/mem?output=proto' > greptime-mem-after.pprof
```
See [Profiling Tools](/reference/http-endpoints.md#profiling-tools) for the endpoint reference.
When reporting the issue, include:
- GreptimeDB version and deployment mode.
- Affected component or node.
- CPU flamegraph or memory flamegraph/profile files.
- Approximate time range and workload when the issue happened.
- Relevant metrics and logs if available.
---
## Run on Android Platforms
Since v0.4.0, GreptimeDB supports running on Android platforms with ARM64 CPU and Android API level >= 23.
## Install terminal emulator on Android
You can install [Termux](https://termux.dev/) from [GitHub release page](https://github.com/termux/termux-app/releases/latest).
## Download GreptimeDB Android binary.
```bash
VERSION=$(curl -sL \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/repos/GreptimeTeam/greptimedb/releases/latest" | sed -n 's/.*"tag_name": "\([^"]*\)".*/\1/p')
curl -sOL "https://github.com/GreptimeTeam/greptimedb/releases/download/${VERSION}/greptime-android-arm64-${VERSION}.tar.gz"
tar zxvf ./greptime-android-arm64-${VERSION}.tar.gz
./greptime -V
```
If binary's downloaded correctly, the command is expected to print the version of downloaded binary.
## Create GreptimeDB configuration file
You can create a minimal configuration file that allows access from local network.
```bash
DATA_DIR="$(pwd)/greptimedb-data"
mkdir -p ${DATA_DIR}
cat < ./config.toml
[http]
addr = "0.0.0.0:4000"
[grpc]
bind_addr = "0.0.0.0:4001"
[mysql]
addr = "0.0.0.0:4002"
[storage]
data_home = "${DATA_DIR}"
type = "File"
EOF
```
## Start GreptimeDB from command line
```bash
./greptime --log-dir=$(pwd)/logs standalone start -c ./config.toml
```
## Connect to GreptimeDB running on Android
> You can find the IP address of your Android device from system settings or `ifconfig`.
You can now connect to the GreptimeDB instance running on Android from another device with MySQL client installed.
```bash
mysql -h -P 4002
```
And play like on any other platforms!
```
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.1.10-alpha-msql-proxy Greptime
Copyright (c) 2000, 2023, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CREATE TABLE monitor (env STRING, host STRING, ts TIMESTAMP, cpu DOUBLE DEFAULT 0, memory DOUBLE, TIME INDEX (ts), PRIMARY KEY(env,host));
Query OK, 0 rows affected (0.14 sec)
mysql> INSERT INTO monitor(ts, env, host, cpu, memory) VALUES
-> (1655276557000,'prod', 'host1', 66.6, 1024),
-> (1655276557000,'prod', 'host2', 66.6, 1024),
-> (1655276557000,'prod', 'host3', 66.6, 1024),
-> (1655276558000,'prod', 'host1', 77.7, 2048),
-> (1655276558000,'prod', 'host2', 77.7, 2048),
-> (1655276558000,'test', 'host3', 77.7, 2048),
-> (1655276559000,'test', 'host1', 88.8, 4096),
-> (1655276559000,'test', 'host2', 88.8, 4096),
-> (1655276559000,'test', 'host3', 88.8, 4096);
Query OK, 9 rows affected (0.14 sec)
mysql>
mysql> select * from monitor where env='test' and host='host3';
+------+-------+---------------------+------+--------+
| env | host | ts | cpu | memory |
+------+-------+---------------------+------+--------+
| test | host3 | 2022-06-15 15:02:38 | 77.7 | 2048 |
| test | host3 | 2022-06-15 15:02:39 | 88.8 | 4096 |
+------+-------+---------------------+------+--------+
2 rows in set (0.20 sec)
```
---
## Troubleshooting
When encountering errors or performance issues,
you can understand GreptimeDB's status through metrics and logs.
This information can also help further investigate the root causes of problems.
The following lists troubleshooting methods for some common abnormal situations.
For cases where the cause cannot be easily identified,
providing metrics and logs to the official team can also improve the efficiency of official troubleshooting.
## Checking CPU and Memory Load
You can directly view the CPU and Memory load of corresponding components from the Dashboard.
CPU is displayed in millicores, while Memory shows the current process RSS.
You need to pay attention to whether the corresponding CPU and Memory load exceeds the Pod's Limit.
If CPU has reached the Pod's Limit,
throttling will be triggered, and clients will experience slower request processing.
If Memory reaches over 70% of the Limit, it may result in OOM (Out of Memory).
## Flow Creation Failure
When flow creation fails,
one common scenario is that flownode is not deployed.
You can check:
- Whether flownode is deployed in the cluster
- Whether the flownode status in the cluster is READY
If flownode is already deployed,
you can further investigate by examining metasrv and flownode logs,
or check whether the flow node was successfully created through internal tables:
```sql
select * from information_schema.cluster_info;
```
## Object Storage Configuration Issues
When object storage is misconfigured,
GreptimeDB will encounter exceptions when accessing object storage.
If GreptimeDB hasn't stored any data,
it generally doesn't need to access object storage,
so errors might not be observable immediately after deployment.
Once you create a table or start writing data to GreptimeDB through write protocols, you can observe request errors.
Typically, DB error messages will include object storage error details.
You can find specific object storage error information through DB error logs.
Some common error causes include:
- Incorrect Access Key or Secret Access Key
- Improper object storage permission configuration
- If using Tencent Cloud COS's S3-compatible API, since Tencent Cloud has [disabled path-style domains](https://cloud.tencent.com/document/product/436/102489), you need to set `enable_virtual_host_style = true` in GreptimeDB's S3 configuration
For S3 as an example, GreptimeDB requires the following permissions:
```txt
"s3:PutObject",
"s3:ListBucket",
"s3:GetObject",
"s3:DeleteObject"
```
## Low Write Throughput
You can observe write throughput through the Ingestion-related panels in the dashboard:
If write throughput is lower than expected load
it might be due to high write latency causing DB backlog.
The dashboard provides panels in the `Frontend Requests` area to observe request latency:
You can check if any nodes are experiencing stall through the `Write Stall` panel in the `Mito Engine` area (stall values remaining above 0 for extended periods).
If so, it indicates that node writes have encountered bottlenecks.
By observing the `Write Buffer` panel, you can also monitor size changes in buffers used for writing. If buffers fill up quickly, consider proportionally increasing `global_write_buffer_size` and `global_write_buffer_reject_size`:
The Write Stage panel shows which stages have high write latency:
You can check background task status through Compaction/Flush related panels:
- Whether frequent flush operations occur; if so, consider proportionally increasing `global_write_buffer_size` and `global_write_buffer_reject_size`
- Whether there are long-running compaction and flush operations; if so, writes might be affected by these background tasks
Additionally, logs related to `flush memtables` can show the latency of individual flush operations.
## High Memory Consumption During Writes
During DB write operations,
the total size of all memtables is estimated.
You can view memtable memory usage through the `Write Buffer` panel.
Note that these values might be smaller than the actual allocated memory.
If you notice that DB memory grows too quickly during writes,
or if writes frequently encounter OOM situations,
this might be related to unreasonable table schema design.
A common cause is improper Primary Key design.
If there are too many primary keys,
it will consume excessive memory during writes,
potentially causing database memory usage to become too high.
In such cases, `Write Buffer` often remains at high levels.
Increasing write buffer size generally won't improve this issue;
you need to reduce the number of primary key columns or remove high-cardinality primary key columns.
## High Query Latency
If query latency is high,
you can add `EXPLAIN ANALYZE` or `EXPLAIN ANALYZE VERBOSE` before the query and re-execute it.
The query results will include execution time for each stage to assist in troubleshooting.
Additionally, the `Read Stage` panel can help understand query latency across different stages:
## Cache Full
To understand the size of various caches,
you can check the used size of different caches on nodes through the `Cached Bytes` panel:
For nodes with high query latency,
you can also determine if caches are full by checking cache sizes.
## Object Storage Latency
You can view object storage operation latency through panels in the `OpenDAL` area of the dashboard.
For example, the following panel shows response latency for object storage write operations:
If you suspect object storage operations are experiencing jitter,
you can observe the related panels.
---
## Upgrade
## Overview
This guide provides upgrade instructions for GreptimeDB, including compatibility information and breaking changes for each version. Before upgrading, ensure you review the relevant breaking changes for your upgrade path.
For each upgrade, review the changes introduced after your current version up
to and including your target version. Reviewing intermediate releases does not
mean you need to install each one; follow the supported upgrade paths below.
For complete version history and feature additions, see the [Release Notes](/release-notes/).
## Upgrade Paths to v1.2
### From v1.0 or v1.1 to v1.2
If you are currently running v1.0 or v1.1, you can upgrade directly to v1.2.
Review [Upgrading from v1.0 or v1.1 to v1.2](#upgrading-from-v10-or-v11-to-v12)
and complete the related checks in the [Upgrade Checklist](#upgrade-checklist)
before the rollout.
If you are starting from v1.0, also review [Metric Engine changes in v1.1](/1.1/user-guide/deployments-administration/upgrade/#metric-engine-changes-in-v11).
### From v0.17 or Earlier
If you are upgrading to v1.2 from v0.17 or an earlier release, first review the
relevant v1.0 upgrade path below and then apply the
[v1.2 breaking changes](#upgrading-from-v10-or-v11-to-v12).
For Metric Engine, also review the [v1.0 compaction guidance](/1.0/user-guide/deployments-administration/upgrade/#metric-engine-compaction-changes-in-v10)
and [Metric Engine changes in v1.1](/1.1/user-guide/deployments-administration/upgrade/#metric-engine-changes-in-v11).
## Upgrade Paths to v1.0
### From v0.16 to v1.0
If you are currently running v0.16, you can upgrade directly to v1.0. See [Upgrading from v0.16 to v1.0](#upgrading-from-v016-to-v10) for all relevant breaking changes.
### From v0.17 to v1.0
If you are currently running v0.17, you can upgrade directly to v1.0. See [Upgrading from v0.17 to v1.0](#upgrading-from-v017-to-v10) for breaking changes.
### From Earlier Versions
**Important:** This guide only covers upgrades from v0.16 and later versions.
If you are running a version earlier than v0.16, you must first upgrade to v0.16 by following the upgrade documentation for your current version. Once you have successfully upgraded to v0.16, you can then use this guide to upgrade to v1.0.
## Breaking Changes by Version
### Upgrading from v1.0 or v1.1 to v1.2
#### Removed PromQL `holt_winters`
**Impact:** PromQL query parsing
The `holt_winters` name has been removed. It was an alias kept for backward
compatibility; the function itself is unchanged and remains available as
`double_exponential_smoothing`, which GreptimeDB supported before v1.2 as well.
**Action Required:**
- Search dashboards, recording rules, alert rules, and API callers for
`holt_winters(`
- Rename each call to `double_exponential_smoothing(...)`. The arguments and
the results are identical:
```promql
# Before v1.2
holt_winters(prom_series[10s], 0.5, 0.1)
# v1.2 and later
double_exponential_smoothing(prom_series[10s], 0.5, 0.1)
```
- Re-run the affected queries in a staging environment after the change
#### Rejected `fill`, `fill_left`, and `fill_right` PromQL Modifiers
**Impact:** PromQL compatibility tightening to avoid incorrect query plans
v1.2 upgrades the PromQL parser, which makes the binary-operator modifiers
`fill`, `fill_left`, and `fill_right` syntactically valid for the first time,
as in `metric_a + fill(0) metric_b`. GreptimeDB does not implement the
outer-join semantics they require, so it rejects them, whether used directly or
nested, rather than planning them as ordinary inner joins.
Earlier releases could not parse these modifiers at all, so no query that
worked before v1.2 is affected. This is also unrelated to the SQL
`RANGE ... FILL` clause, which is unchanged.
**Action Required:**
- Search PromQL expressions for `fill(`, `fill_left(`, and `fill_right(`
- Rewrite any affected query so it does not depend on these modifiers
#### Pipeline Integer Narrowing Now Uses `on_failure`
**Impact:** Pipeline type coercion correctness
When a pipeline converts an integral input or a numeric string into a declared
integer type that cannot represent it, the value no longer wraps with modulo
semantics. It now follows the transform's `on_failure` policy. Previously `-1`
written to `uint8` was stored as `255`, and `256` written to `int8` was stored
as `0`.
This applies to `int8`, `int16`, `int32`, `uint8`, `uint16`, and `uint32`, and
to cross-sign conversions into `int64` and `uint64`. Float-to-integer behavior
is unchanged in this release.
**Action Required:**
- Review pipelines that write into narrow integer targets, or that may write
negative values into unsigned targets
- If those pipelines relied on wraparound behavior, widen the target type or
configure an explicit `on_failure` policy before upgrading
- Use `POST /v1/pipelines/_dryrun` or representative staging data to verify
that boundary values now produce the expected error, default, or null outcome
#### Local SQL File Access Is Now Sandboxed
**Impact:** `COPY` and external-table workflows that read or write local files
Standalone deployments now resolve local SQL file paths inside
`storage.copy_root` (default `/copy`). Distributed deployments reject
local-file SQL access entirely.
**Action Required:**
- Identify `COPY` statements and external tables that reference local paths
- If you run standalone, move those files under the sandbox, set
`storage.copy_root` to a dedicated directory, or switch the workflow to
object storage
- If you run distributed services, migrate local-file workflows to S3, OSS,
GCS, or AzBlob before upgrading
- Follow [Migrate Local SQL File Access](/user-guide/deployments-administration/migrate-local-sql-file-access.md)
for the detailed migration procedure
#### Removed `sparse_primary_key_encoding` Configuration
**Impact:** Metric engine configuration cleanup
GreptimeDB now always uses sparse primary key encoding for metric tables, and
the `sparse_primary_key_encoding` option has been removed. A configuration file
that still sets the key loads without error; the key is ignored.
**Action Required:**
- Remove `sparse_primary_key_encoding` from the `[region_engine.metric]` block
in your configuration files, Helm values, and automation templates. Drop the
older `experimental_sparse_primary_key_encoding` key as well if it is still
present.
- If you previously set `sparse_primary_key_encoding = false`, note that v1.2
no longer provides an opt-out
- Restart a staging environment with the cleaned configuration to confirm your
deployment no longer depends on the removed setting
### Upgrading from v1.0 to v1.1
#### OSS RRSA credential precedence {#oss-rrsa-credential-precedence}
**Impact:** OSS authentication can use a different RAM role after an upgrade.
When upgrading from v1.0.2 to v1.1.0, ACK deployments using RRSA/OIDC can select
ECS node RAM Role credentials before OIDC credentials if the Pod can obtain
credentials from node metadata. If the node role lacks the required OSS
permissions, table creation or writes can fail with `403 PermissionDenied`
(for example, while writing a manifest). The same credential order is present
in v1.1.4, v1.2.1, and `v1.3.0-alpha.1-nightly-20260907`; also review this guidance
when upgrading directly from v1.0 to one of those versions.
**Action Required:**
1. If the deployment uses RRSA/OIDC, set
`ALIBABA_CLOUD_ECS_METADATA_DISABLED=true` in the environment of the standalone
instance or all datanodes accessing OSS before rolling out the upgrade.
Keep the existing RRSA environment variables and token mount. See
[OSS RRSA/OIDC configuration](/user-guide/deployments-administration/configuration.md#alibaba-cloud-oss-authentication-with-rrsaoidc)
for the container configuration example and credential precedence details.
2. Restart the affected instances so the setting takes effect. Do not apply it
to deployments that rely on ECS RAM Role credentials.
3. Test the upgrade in a non-production environment. Verify table creation,
writes, and reads, and check that OSS operations no longer return permission
errors. If errors persist, check the selected identity and its bucket
permissions; a `403` alone does not identify this credential-order issue.
### Upgrading from v0.17 to v1.0
For Metric Engine tables, review the [compaction guidance](/1.0/user-guide/deployments-administration/upgrade/#metric-engine-compaction-changes-in-v10) before
upgrading from v0.17 to v1.0 or later.
#### Jaeger HTTP Header Removal
**Impact:** HTTP header deprecation
The HTTP header `x-greptime-jaeger-time-range-for-operations` has been deprecated and removed.
**Action Required:**
- If you configured this header in your Jaeger data source or proxy, remove it from your configuration
- The header will no longer have any effect
#### Metric Engine Default Sparse Primary Key Encoding
**Impact:** Default configuration change with performance improvements
Metric Engine now enables **sparse primary key encoding** by default to improve storage efficiency and query performance for metric scenarios.
**Configuration Changes:**
- **NEW default:** `sparse_primary_key_encoding = true`
- **DEPRECATED:** `experimental_sparse_primary_key_encoding` (use `sparse_primary_key_encoding` instead)
**Action Required:**
- This change does not cause data format compatibility issues
- All metric tables will automatically use sparse encoding by default
- If you want to continue using the old encoding method, explicitly set:
```toml
[[region_engine]]
[region_engine.metric]
sparse_primary_key_encoding = false
```
#### `greptime_identity` Pipeline JSON Behavior Change
**Impact:** JSON processing logic change
The JSON processing logic in `greptime_identity` pipeline has changed significantly:
**New Behavior:**
- Nested JSON objects are automatically flattened into separate columns using dot notation (e.g., `object.a`, `object.b`)
- Arrays are stored as JSON strings instead of JSON objects
- The `flatten_json_object` parameter has been removed
- A new `max_nested_levels` parameter controls flattening depth (default: 10 levels)
- When the depth limit is exceeded, remaining nested structures are serialized as JSON strings
**Action Required:**
1. Review your pipeline configurations that use `greptime_identity`
2. Remove any usage of the deprecated `flatten_json_object` parameter
3. Adjust queries that reference nested JSON fields to use the new dot notation
4. If you have deeply nested JSON (>10 levels), consider setting `max_nested_levels` appropriately
**Example:**
Before (v0.17):
```json
{ "user": { "name": "Alice", "age": 30 } }
```
Stored as a single JSON column.
After (v1.0):
```
user.name = "Alice"
user.age = 30
```
Stored as separate columns.
#### Metric Engine TSID Generation Algorithm Change
**Impact:** Time Series ID generation optimization with query implications
The TSID (Time Series ID) generation algorithm has been optimized by replacing `mur3::Hasher128` with the higher-performance `fxhash::FxHasher`, including a fast-path for series without NULL labels.
**Performance Improvements:**
- Regular scenarios: 5-6x faster
- Scenarios with NULL labels: ~2.5x faster
**Breaking Change Impact:**
This is a **breaking change** that affects time series identification:
- **Before upgrade (time < t):** Data uses the old algorithm to generate TSIDs
- **After upgrade (time > t):** Data uses the new algorithm to generate TSIDs
**Query Behavior:**
- Queries with time ranges that **span the upgrade time `t`** may experience slight discrepancies in time series matching near time `t`
- Queries with time ranges that **do not include `t`** are not affected
**Action Required:**
Choose one of the following upgrade strategies:
1. **Direct Upgrade (Recommended for most users):**
- Accept minor query discrepancies near the upgrade time
- Suitable if approximate results near the upgrade time are acceptable
2. **Export-Upgrade-Import (For zero tolerance):**
- If you cannot accept any discrepancies, use this fully compatible upgrade method:
1. Export all data before upgrading
2. Upgrade to v1.0
3. Import data back into the new version
- Refer to [Backup & Restore Documentation](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-data/)
### Upgrading from v0.16 to v1.0
If you are upgrading from v0.16, you need to review:
1. **All breaking changes from v0.17 to v1.0** (listed above)
2. **v0.17.0 breaking changes** (listed below)
This ensures you're aware of all changes that occurred between v0.16 and v1.0.
### v0.17.0 Breaking Changes
#### Ordered-Set Aggregate Functions
**Impact:** SQL syntax change
Ordered-set aggregate functions now require a `WITHIN GROUP (ORDER BY …)` clause.
**Before:**
```sql
SELECT approx_percentile_cont(latency, 0.95) FROM metrics;
```
**After:**
```sql
SELECT approx_percentile_cont(0.95) WITHIN GROUP (ORDER BY latency) FROM metrics;
```
**Action Required:** Update all queries using ordered-set aggregate functions (`approx_percentile_cont`, `approx_percentile_cont_weight`, etc.) to include the `WITHIN GROUP (ORDER BY …)` clause.
#### MySQL Protocol Comment Styling
**Impact:** Comment syntax strictness
Incorrect comment styling is no longer allowed in MySQL protocol. Comments must start with `--` instead of `---`.
**Before:**
```sql
--- This is a comment
SELECT * FROM table;
```
**After:**
```sql
-- This is a comment
SELECT * FROM table;
```
**Action Required:** Update any SQL scripts or queries that use `---` style comments to use the standard `--` format.
## Additional v1.0 Changes (Non-Breaking)
### v1.0.0-beta.3
#### Cache Configuration Improvements
The cache architecture has been refactored for better performance:
**New Configuration:**
- `region_engine.mito.manifest_cache_size` (default: 256MB) - specialized manifest file cache
**Removed Configuration:**
- `storage.cache_path`
- `storage.enable_read_cache`
- `storage.cache_capacity`
**Action Required:** Update your configuration files to use the new `manifest_cache_size` setting and remove the deprecated storage cache options.
### v1.0.0-beta.2
#### Improved Database Compatibility
- Numeric type aliases aligned with PostgreSQL and MySQL standards
- Better PostgreSQL extended query support
- Improved MySQL binary protocol handling
**Action Required:** Test your applications to ensure compatibility with the improved behavior.
## Minimizing Business Impact During Upgrade
Before upgrading GreptimeDB, it is essential to perform a comprehensive backup of your data to safeguard against potential data loss. This backup acts as a safety measure in the event of any issues during the upgrade process.
### Best Practices
#### Rolling Upgrade
Utilize [rolling upgrades](https://kubernetes.io/docs/tutorials/kubernetes-basics/update/update-intro/) on Kubernetes to update GreptimeDB instances incrementally. This approach replaces old instances with new ones while maintaining service availability and minimizing downtime.
#### Automatic Retries
Configure client applications to enable automatic retries with exponential backoff. This helps handle temporary service interruptions gracefully.
#### Temporary Pause of Write Operations
For applications that can tolerate brief maintenance windows, consider pausing write operations during the upgrade to ensure data consistency.
#### Double Writing
Implement double writing to both the old and new versions of GreptimeDB, then switch to the new version once you have verified that it is functioning correctly. This allows you to verify data consistency and gradually redirect read traffic to the upgraded version.
## Upgrade Checklist
Before upgrading to your target version, complete the following checklist:
### Pre-Upgrade
- [ ] If using OSS RRSA/OIDC, review the [credential precedence change](#oss-rrsa-credential-precedence) and configure the ECS metadata opt-out where needed
- [ ] Review all breaking changes relevant to your upgrade path
- [ ] **Backup all data and configurations**
- [ ] If upgrading to v1.2, search PromQL assets for `holt_winters(`, `fill(`, `fill_left(`, and `fill_right(`
- [ ] If upgrading to v1.2, identify `COPY` workflows and external tables that reference local file paths
- [ ] If upgrading to v1.2, review pipelines that narrow values into `int8`/`int16`/`int32`/`uint8`/`uint16`/`uint32`, or that write negative values into unsigned targets
- [ ] Identify queries using ordered-set aggregate functions (if upgrading from v0.16 or earlier)
- [ ] Identify pipelines using `greptime_identity` with JSON data
- [ ] Check for usage of deprecated Jaeger HTTP header (if upgrading from v0.17 or earlier)
- [ ] Review metric tables if using Metric Engine
- [ ] If upgrading from v0.17 to v1.0 or later, follow the [Metric Engine compaction guidance](/1.0/user-guide/deployments-administration/upgrade/#metric-engine-compaction-changes-in-v10) to choose a window and decide whether to run SWCS before upgrading
- [ ] If upgrading from a version earlier than v1.1 to v1.1 or later, review [Metric Engine changes in v1.1](/1.1/user-guide/deployments-administration/upgrade/#metric-engine-changes-in-v11)
### Configuration Updates
- [ ] Update configuration files (remove deprecated cache settings)
- [ ] If upgrading to v1.0, update metric engine configuration if needed (`sparse_primary_key_encoding`)
- [ ] If upgrading to v1.2, remove the now-ignored `sparse_primary_key_encoding` and `experimental_sparse_primary_key_encoding` overrides
- [ ] Update pipeline configurations (remove `flatten_json_object`, add `max_nested_levels` if needed)
- [ ] If upgrading to v1.2 standalone deployments, set `storage.copy_root` if the default sandbox path does not fit your workflow
### Code Updates
- [ ] If upgrading to v1.2, rename PromQL `holt_winters(...)` calls to `double_exponential_smoothing(...)`
- [ ] If upgrading to v1.2, rewrite PromQL queries that use `fill`, `fill_left`, or `fill_right`
- [ ] If upgrading to v1.2, update pipelines that relied on integer wraparound and set an explicit `on_failure` policy when needed
- [ ] Update SQL queries with ordered-set aggregates to use `WITHIN GROUP (ORDER BY ...)`
- [ ] Update SQL scripts using `---` comments to use `--`
- [ ] Update queries that access nested JSON fields to use dot notation
- [ ] Remove Jaeger header configuration if present
### Testing & Deployment
- [ ] Test the upgrade in a non-production environment
- [ ] If upgrading from v0.17 to v1.0 or later, test with historical SSTs and monitor memory usage, compaction progress, and query failures
- [ ] If upgrading to v1.2, dry-run representative pipeline inputs that hit integer boundaries and verify the expected `on_failure` result
- [ ] If upgrading to v1.2, validate the updated PromQL queries in staging
- [ ] If upgrading to v1.2, verify local-file `COPY` and external-table workflows after moving them into the sandbox or object storage
- [ ] Verify query results, especially for:
- Ordered-set aggregate functions
- Nested JSON data access
- Metric queries (if affected by TSID change)
- [ ] Plan for rolling upgrade or maintenance window
- [ ] Prepare rollback plan in case of issues
- [ ] Monitor system behavior after upgrade
### Special Considerations for Metric Engine Users
If you cannot accept query discrepancies near the upgrade time due to TSID algorithm change:
- [ ] Plan for export-upgrade-import process
- [ ] Allocate sufficient time for data export and import
- [ ] Refer to [Backup & Restore Documentation](/user-guide/deployments-administration/disaster-recovery/back-up-&-restore-data/)
---
## Local WAL
This section describes how to configure the local WAL for GreptimeDB Datanode component.
```toml
[wal]
provider = "raft_engine"
file_size = "128MB"
purge_threshold = "1GB"
purge_interval = "1m"
read_batch_size = 128
sync_write = false
sync_period = "5s"
```
## Options
If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) to learn how to configure the Datanode by injecting configuration files.
| Configuration Option | Description | Default Value |
| -------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `provider` | The provider of the WAL. Options: `raft_engine` (local file system storage), `kafka` (remote WAL storage in Kafka), or `noop` (no-op WAL provider) | `"raft_engine"` |
| `dir` | The directory where to write logs | `{data_home}/wal` |
| `file_size` | The size of single WAL log file | `128MB` |
| `purge_threshold` | The threshold of the WAL size to trigger purging | `1GB` |
| `purge_interval` | The interval to trigger purging | `1m` |
| `read_batch_size` | The read batch size | `128` |
| `sync_write` | Whether to call fsync when writing every log | `false` |
| `sync_period` | The interval between periodic fsync operations | `5s` |
## Best practices
### Using a separate High-Performance Volume for WAL
It is beneficial to configure a separate volume for the WAL (Write-Ahead Log) directory when deploying GreptimeDB. This setup allows you to:
- Leverage a high-performance disk—either a dedicated physical volume or one provisioned via a custom `StorageClass`.
- Isolate WAL I/O from cache file access, reducing I/O contention and enhancing overall system performance.
If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) to learn how to configure a dedicated WAL volume.
---
## Noop WAL
Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data.
## Availability
Noop WAL is **only available in cluster mode**, not in standalone mode.
## Use Cases
- **Temporary WAL Unavailability**: When the WAL provider (e.g., Kafka) is temporarily unavailable, switch the Datanode to Noop WAL to keep the cluster running.
- **Testing and Development**: Useful for testing scenarios where WAL persistence is not required.
## Data Loss Warning
**When using Noop WAL, all unflushed data will be lost when the Datanode is shutdown or restarted.** Only use this provider temporarily when the normal WAL provider is unavailable. Not recommended for production use except in emergency situations.
## Configuration
To configure Noop WAL for a Datanode:
```toml
[wal]
provider = "noop"
```
In a GreptimeDB cluster, WAL configuration has two parts:
- **Metasrv** - Generates WAL metadata for new regions. Should be set to `raft_engine` or `kafka`.
- **Datanode** - Reads and writes WAL data. Configure as `noop` when the WAL provider is unavailable.
Note: Noop WAL can only be configured on Datanode, not on Metasrv. When using Noop WAL on Datanode, Metasrv continues using its configured WAL provider.
## Best Practices
- Flush regions regularly using `admin flush_table()` or `admin flush_region()` to minimize data loss.
- Switch back to the normal WAL provider as soon as it becomes available.
---
## GreptimeDB WAL Overview
# Write-Ahead Logging (WAL)
The [Write-Ahead Logging](/contributor-guide/datanode/wal.md#introduction)(WAL) in GreptimeDB persistently records every data modification to ensure no memory-cached data loss. GreptimeDB provides three WAL storage options:
- **Local WAL**: Uses an embedded storage engine([raft-engine](https://github.com/tikv/raft-engine)) within the [Datanode](/user-guide/concepts/architecture.md).
- **Remote WAL**: Uses [Apache Kafka](https://kafka.apache.org/) as the external(remote) WAL storage component.
- **Noop WAL**: A no-op WAL provider for emergency situations when WAL becomes unavailable. Does not store any data.
## Local WAL
### Advantages
- **Low latency**: The local WAL is stored within the same process as the Datanode, eliminating network overhead and providing low write latency.
- **No extra components**: The WAL is co-located with the Datanode, so nothing else has to be deployed or operated.
- **Zero RPO**: When deploying GreptimeDB in the cloud, you can configure persistent storage for WAL data using cloud storage services such as AWS EBS or GCP Persistent Disk. This ensures zero [Recovery Point Objective](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective) (RPO), meaning no data loss, even in the event of system failure.
### Disadvantages
- **High RTO**: Because the WAL resides on the same node as the Datanode, the [Recovery Time Objective](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) (RTO) is relatively high. After a Datanode restarts, it must replay the WAL to restore the latest data, during which time the node remains unavailable.
- **Single-Point Access Limitation**: The local WAL is tightly coupled with the Datanode process and only supports a single consumer, which limits features such as region hot standby and [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md).
## Remote WAL
### Advantages
- **Low RTO**: By decoupling WAL from the Datanode, [Recovery Time Objective](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) (RTO) is minimized. If a Datanode crashes, the Metasrv can quickly trigger a [Region Failover](/user-guide/deployments-administration/manage-data/region-failover.md) to migrate affected regions to healthy nodes—without the need to replay WAL locally.
- **Multi-Consumer Subscriptions**: Remote WAL supports multiple consumers subscribing to WAL logs simultaneously, enabling features such as region hot standby and [Region Migration](/user-guide/deployments-administration/manage-data/region-migration.md), thereby enhancing system availability and flexibility.
### Disadvantages
- **External dependencies**: Remote WAL relies on an external Kafka cluster, which increases the complexity of deployment, operation, and maintenance.
- **Network overhead**: Since WAL data needs to be transmitted over the network, careful planning of cluster network bandwidth is required to ensure low latency and high throughput, especially under write-heavy workloads.
## Noop WAL
Noop WAL is a special WAL provider for emergency situations when the configured WAL provider becomes temporarily unavailable. It does not store any WAL data and is only available in cluster mode.
For detailed configuration, see the [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) page.
## Next steps
- [Local WAL](/user-guide/deployments-administration/wal/local-wal.md) — configuring the embedded WAL.
- [Remote WAL](/user-guide/deployments-administration/wal/remote-wal/configuration.md) — configuring Kafka as the WAL backend.
- [Noop WAL](/user-guide/deployments-administration/wal/noop-wal.md) — the emergency no-op provider.
---
## GreptimeDB Remote WAL Configuration
# Configuration
The configuration of Remote WAL contains two parts:
- Metasrv Configuration
- Datanode Configuration
If you are using Helm Chart to deploy GreptimeDB, you can refer to [Common Helm Chart Configurations](/user-guide/deployments-administration/deploy-on-kubernetes/common-helm-chart-configurations.md) to learn how to configure Remote WAL.
## Metasrv Configuration
On the Metasrv side, Remote WAL is primarily responsible for managing Kafka topics and periodically pruning stale WAL data.
```toml
[wal]
provider = "kafka"
broker_endpoints = ["kafka.kafka-cluster.svc.cluster.local:9092"]
# WAL data pruning options
auto_prune_interval = "30m"
auto_prune_logical_delete = false # set to true if your Kafka deployment does not support DeleteRecords
auto_prune_parallelism = 10
flush_trigger_size = "512MB"
checkpoint_trigger_size = "128MB"
# Topic creation options
auto_create_topics = true
num_topics = 64
# Testing value: no redundant copy. Set this from your durability policy in production.
replication_factor = 1
topic_name_prefix = "greptimedb_wal_topic"
create_topic_timeout = "30s"
```
### Options
| Configuration Option | Description |
|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `provider` | The WAL provider to use. Set to `"kafka"` to enable Remote WAL with Kafka. |
| `broker_endpoints` | List of Kafka broker addresses to connect to. Example: `["kafka.kafka-cluster.svc:9092"]`. |
| `auto_prune_interval` | How often to automatically prune (delete) stale WAL data. Specify as a duration string (e.g., `"30m"`). Set to `"0s"` to disable automatic pruning. |
| `auto_prune_logical_delete` | If `true`, automatic WAL pruning only advances GreptimeDB's metadata marker and skips Kafka `DeleteRecords`. Set this to `true` for Kafka deployments that do not support `DeleteRecords`. When enabled, actual data removal from Kafka topics relies on Kafka's own retention and cleanup policies. |
| `auto_prune_parallelism` | Maximum number of concurrent pruning tasks. Increasing this value may speed up pruning but will use more resources. |
| `auto_create_topics` | If `true`, Metasrv will automatically create required Kafka topics. If `false`, you must manually create all topics before starting Metasrv. |
| `num_topics` | Number of Kafka topics to use for WAL storage. More topics can improve scalability and performance. |
| `replication_factor` | Replication factor for Kafka topics. Determines how many Kafka brokers will store copies of each topic's data. The example value `1` keeps no redundant copy and suits testing only; for production pick a value from your durability policy. It must not exceed the number of brokers. |
| `topic_name_prefix` | Prefix for Kafka topic names. WAL topics will be named as `{topic_name_prefix}_{index}` (e.g., `greptimedb_wal_topic_0`). The prefix must match the regex `[a-zA-Z_:-][a-zA-Z0-9_:\-\.@#]*`. |
| `flush_trigger_size` | Estimated size threshold (e.g., `"512MB"`) for triggering a flush operation in a region. Calculated as `(latest_entry_id - flushed_entry_id) * avg_record_size`. When this value exceeds `flush_trigger_size`, MetaSrv initiates a flush. Set to `"0"` to let the system automatically determine the flush trigger size. This also controls the maximum replay size from a topic during region replay; using a smaller value can help reduce region replay time during Datanode startup. |
| `checkpoint_trigger_size` | Estimated size threshold (e.g., `"128MB"`) for triggering a checkpoint operation in a region. Calculated as `(latest_entry_id - last_checkpoint_entry_id) * avg_record_size`. When this value exceeds `checkpoint_trigger_size`, MetaSrv initiates a checkpoint. Set to `"0"` to let the system automatically determine the checkpoint trigger size. Using a smaller value can help reduce region replay time during Datanode startup. |
| `create_topic_timeout` | The timeout for creating a Kafka topic. Default is `"30s"`. |
#### Topic Setup and Kafka Permissions
To ensure Remote WAL works correctly with Kafka, please check the following:
- If `auto_create_topics = false`:
- All required topics must be created manually **before** starting Metasrv.
- Topic names must follow the pattern `{topic_name_prefix}_{index}` where `index` ranges from `0` to `{num_topics - 1}`. For example, with the default prefix `greptimedb_wal_topic` and `num_topics = 64`, you need to create topics from `greptimedb_wal_topic_0` to `greptimedb_wal_topic_63`.
- Topics must be configured to support **LZ4 compression**.
- The Kafka user must have the following permissions:
- **Append** records to WAL topics (requires LZ4 compression support).
- **Read** records from WAL topics (requires LZ4 compression support).
- **Delete** records from WAL topics (not required when `auto_prune_logical_delete = true`).
- **Create** topics (only required if `auto_create_topics = true`).
## Datanode Configuration
The Datanode side is mainly used to write the data to the Kafka topics and read the data from the Kafka topics.
```toml
[wal]
provider = "kafka"
broker_endpoints = ["kafka.kafka-cluster.svc:9092"]
max_batch_bytes = "1MB"
create_index = false
overwrite_entry_start_id = true
connect_timeout = "3s"
timeout = "3s"
```
### Options
| Configuration Option | Description |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `provider` | Set to "kafka" to enable Remote WAL via Kafka. |
| `broker_endpoints` | List of Kafka broker addresses. |
| `max_batch_bytes` | Maximum size for each Kafka producer batch. |
| `create_index` | Whether to create per-region Kafka WAL indexes. The default is `false`, and the option only takes effect in distributed mode. The index can reduce Kafka reads during recovery, but enable it only after confirming significant read amplification during single-region recovery. The Datanode persists the index to object storage every `dump_index_interval` (`"60s"` by default), adding continuous object-store I/O. |
| `overwrite_entry_start_id` | If true, the Datanode will skip over missing entries during WAL replay. Prevents out-of-range errors, but may hide data loss. |
| `connect_timeout` | The connect timeout for Kafka client. Default is `"3s"`. |
| `timeout` | The timeout for Kafka client operations. Default is `"3s"`. |
#### Required Settings and Limitations
:::warning IMPORTANT: Kafka Retention Policy Configuration
Please configure Kafka retention policy very carefully to avoid data loss. GreptimeDB will automatically recycle unneeded WAL data, so in most cases you don't need to set the retention policy. However, if you do set it, please ensure the following:
- **Size-based retention**: Typically not needed, as the database manages its own data lifecycle
- **Time-based retention**: If you choose to set this, ensure it's **significantly greater than the auto-flush-interval** to prevent premature data deletion
Improper retention settings can lead to data loss if WAL data is deleted before GreptimeDB has processed it.
:::
- If you set `overwrite_entry_start_id = true`:
- Ensure that `auto_prune_interval` is enabled in Metasrv to allow automatic WAL pruning.
- Kafka topics **must not use size-based retention policies**.
- If time-based retention is enabled, the retention period should be set to a value significantly greater than auto-flush-interval, preferably at least 2 times its value.
- Ensure the Kafka user used by Datanode has the following permissions:
- **Append** records to WAL topics (requires LZ4 compression support).
- **Read** records from WAL topics (requires LZ4 compression support).
- Ensure that `max_batch_bytes` does not exceed Kafka’s maximum message size (typically 1MB by default).
## Kafka Authentication Configuration
Kafka authentication settings apply to both Metasrv and Datanode under the `[wal]` section.
### SASL
Kafka supports several SASL mechanisms: `PLAIN`, `SCRAM-SHA-256`, and `SCRAM-SHA-512`.
```toml
[wal]
broker_endpoints = ["kafka.kafka-cluster.svc.cluster.local:9092"]
[wal.sasl]
type = "SCRAM-SHA-512"
username = "user"
password = "secret"
```
### TLS
You can enable TLS encryption for Kafka connections by configuring the `[wal.tls]` section. There are three common modes:
#### Using System CA Certificate
To use system-wide trusted CAs, enable TLS without providing any certificate paths:
```toml
[wal]
broker_endpoints = ["kafka.kafka-cluster.svc.cluster.local:9092"]
[wal.tls]
```
#### Using Custom CA Certificate
If your Kafka cluster uses a private CA, specify the server CA certificate explicitly:
```toml
[wal]
broker_endpoints = ["kafka.kafka-cluster.svc.cluster.local:9092"]
[wal.tls]
server_ca_cert_path = "/path/to/server.crt"
```
#### Using Mutual TLS (mTLS)
To enable mutual authentication, provide both the client certificate and private key along with the server CA:
```toml
[wal]
broker_endpoints = ["kafka.kafka-cluster.svc.cluster.local:9092"]
[wal.tls]
server_ca_cert_path = "/path/to/server_cert"
client_cert_path = "/path/to/client_cert"
client_key_path = "/path/to/key"
```
---
## Kafka
When [Remote WAL](/user-guide/deployments-administration/wal/overview.md#remote-wal) is enabled, the GreptimeDB cluster writes its write-ahead log to Kafka instead of local disks. Datanodes append WAL entries to Kafka topics, and Metasrv prunes entries that are no longer needed for recovery.
Until a region flushes to object storage, Kafka may hold the only durable copy of those writes. Treat it as a stateful dependency of the database, not as a transport buffer. Getting the retention policy wrong is the most common way to lose data here: Kafka deletes segments on its own schedule, and a WAL entry deleted before GreptimeDB replays it cannot be recovered.
- [Deploying Kafka Cluster](/user-guide/deployments-administration/deploy-on-kubernetes/deploy-kafka.md) — installation, verification, monitoring, and uninstallation with the Bitnami Kafka Helm chart.
- [Remote WAL Configuration](/user-guide/deployments-administration/wal/remote-wal/configuration.md) — the Metasrv and Datanode `[wal]` options, plus the [required settings and limitations](/user-guide/deployments-administration/wal/remote-wal/configuration.md#required-settings-and-limitations) the Kafka cluster must satisfy.
- [Configure Remote WAL](/user-guide/deployments-administration/deploy-on-kubernetes/configure-remote-wal.md) — a complete Helm chart example on Kubernetes.
---
## Continuous Aggregation
The Flow engine maintains aggregates such as sums, averages, and counts over time windows, updating the sink table as new rows arrive. Queries then read the sink table instead of scanning the source table.
The three examples below cover the common cases:
1. **Real-time Analytics**: downsample a high-frequency event stream into a lower resolution. For instance, compile per-millisecond log events into requests per minute, average response times, and error rates per minute.
2. **Real-time Monitoring**: aggregate a stream and use the aggregated rows as alerts, for example emitting a row when the maximum temperature in a window exceeds a threshold.
3. **Real-time Dashboard**: precompute the counters a dashboard renders, such as request counts, average response time, and error counts per minute.
## Real-time Analytics Example
### Calculate the Log Statistics
This use case is to calculate the total number of logs, the minimum size, the maximum size, the average size, and the number of packets with the size greater than 550 for each status code in a 1-minute fixed window for access logs.
First, create a source table `ngx_access_log` and a sink table `ngx_statistics` with following clauses:
```sql
CREATE TABLE `ngx_access_log` (
`client` STRING NULL,
`ua_platform` STRING NULL,
`referer` STRING NULL,
`method` STRING NULL,
`endpoint` STRING NULL,
`trace_id` STRING NULL FULLTEXT INDEX,
`protocol` STRING NULL,
`status` SMALLINT UNSIGNED NULL,
`size` DOUBLE NULL,
`agent` STRING NULL,
`access_time` TIMESTAMP(3) NOT NULL,
TIME INDEX (`access_time`)
)
WITH(
append_mode = 'true'
);
```
```sql
CREATE TABLE `ngx_statistics` (
`status` SMALLINT UNSIGNED NULL,
`total_logs` BIGINT NULL,
`min_size` DOUBLE NULL,
`max_size` DOUBLE NULL,
`avg_size` DOUBLE NULL,
`high_size_count` BIGINT NULL,
`time_window` TIMESTAMP time index,
`update_at` TIMESTAMP NULL,
PRIMARY KEY (`status`)
);
```
Then create the flow `ngx_aggregation` to aggregate a series of aggregate functions, including `count`, `min`, `max`, `avg` of the `size` column, and the number of packets with a size greater than 550. The aggregation is calculated in 1-minute fixed windows of `access_time` column and also grouped by the `status` column. A spike in `high_size_count` or `max_size` within a single window then points you at the minute to inspect.
The `EXPIRE AFTER '6h'` in the following SQL ensures that the flow computation only uses source data from the last 6 hours. Data older than 6 hours in the sink table will not be modified by this flow. For more details, see [Manage Flow](manage-flow.md#expire-after).
```sql
CREATE FLOW ngx_aggregation
SINK TO ngx_statistics
EXPIRE AFTER '6h'
COMMENT 'aggregate nginx access logs'
AS
SELECT
status,
count(client) AS total_logs,
min(size) as min_size,
max(size) as max_size,
avg(size) as avg_size,
sum(case when `size` > 550 then 1 else 0 end) as high_size_count,
date_bin('1 minutes'::INTERVAL, access_time) as time_window
FROM ngx_access_log
GROUP BY
status,
time_window;
```
To observe the outcome of the continuous aggregation in the `ngx_statistics` table, insert some data into the source table `ngx_access_log`.
```sql
INSERT INTO ngx_access_log
VALUES
('android', 'Android', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 200, 1000, 'agent', now() - INTERVAL '1' minute),
('ios', 'iOS', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 200, 500, 'agent', now() - INTERVAL '1' minute),
('android', 'Android', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 200, 600, 'agent', now()),
('ios', 'iOS', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 404, 700, 'agent', now());
```
Then the sink table `ngx_statistics` will be incrementally updated and contain the following data:
The time-window and `update_at` timestamps in these result tables are illustrative and vary with execution time.
```sql
SELECT * FROM ngx_statistics;
```
```sql
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
| status | total_logs | min_size | max_size | avg_size | high_size_count | time_window | update_at |
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
| 200 | 2 | 500 | 1000 | 750 | 1 | 2025-04-24 06:46:00 | 2025-04-24 06:47:06.680000 |
| 200 | 1 | 600 | 600 | 600 | 1 | 2025-04-24 06:47:00 | 2025-04-24 06:47:06.680000 |
| 404 | 1 | 700 | 700 | 700 | 1 | 2025-04-24 06:47:00 | 2025-04-24 06:47:06.680000 |
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
3 rows in set (0.01 sec)
```
Try to insert more data into the `ngx_access_log` table:
```sql
INSERT INTO ngx_access_log
VALUES
('android', 'Android', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 200, 500, 'agent', now()),
('ios', 'iOS', 'referer', 'GET', '/api/v1', 'trace_id', 'HTTP', 404, 800, 'agent', now());
```
The sink table `ngx_statistics` now has corresponding rows updated; note how `max_size`, `avg_size` and `high_size_count` are updated:
```sql
SELECT * FROM ngx_statistics;
```
```sql
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
| status | total_logs | min_size | max_size | avg_size | high_size_count | time_window | update_at |
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
| 200 | 2 | 500 | 1000 | 750 | 1 | 2025-04-24 06:46:00 | 2025-04-24 06:47:06.680000 |
| 200 | 2 | 500 | 600 | 550 | 1 | 2025-04-24 06:47:00 | 2025-04-24 06:47:21.720000 |
| 404 | 2 | 700 | 800 | 750 | 2 | 2025-04-24 06:47:00 | 2025-04-24 06:47:21.720000 |
+--------+------------+----------+----------+----------+-----------------+---------------------+----------------------------+
3 rows in set (0.01 sec)
```
Here is the explanation of the columns in the `ngx_statistics` table:
- `status`: The status code of the HTTP response.
- `total_logs`: The total number of logs with the same status code.
- `min_size`: The minimum size of the packets with the same status code.
- `max_size`: The maximum size of the packets with the same status code.
- `avg_size`: The average size of the packets with the same status code.
- `high_size_count`: The number of packets with the size greater than 550.
- `time_window`: The time window of the aggregation.
- `update_at`: The time when the aggregation is updated.
### Retrieve Distinct Countries by Time Window
Another example of real-time analytics is to retrieve all distinct countries from the `ngx_access_log` table.
You can use the following query to group countries by time window:
```sql
/* source table */
CREATE TABLE ngx_access_log (
client STRING,
country STRING,
access_time TIMESTAMP TIME INDEX,
PRIMARY KEY(client)
)
WITH(
append_mode = 'true'
);
/* sink table */
CREATE TABLE ngx_country (
country STRING,
time_window TIMESTAMP TIME INDEX,
update_at TIMESTAMP,
PRIMARY KEY(country)
);
/* create flow task to calculate the distinct country */
CREATE FLOW calc_ngx_country
SINK TO ngx_country
EXPIRE AFTER '7days'::INTERVAL
COMMENT 'aggregate for distinct country'
AS
SELECT
DISTINCT country,
date_bin('1 hour'::INTERVAL, access_time) as time_window
FROM ngx_access_log
GROUP BY
country,
time_window;
```
The above query puts the data from the `ngx_access_log` table into the `ngx_country` table.
It calculates the distinct country for each time window.
The `date_bin` function is used to group the data into one-hour intervals.
The `ngx_country` table will be continuously updated with the aggregated data,
providing real-time insights into the distinct countries that are accessing the system. The `EXPIRE AFTER` clause makes the Flow ignore data with `access_time` older than 7 days; see [Manage Flow](manage-flow.md#expire-after) for details.
You can insert some data into the source table `ngx_access_log`:
```sql
INSERT INTO ngx_access_log VALUES
('client1', 'US', now() - '2 hour'::INTERVAL),
('client2', 'US', now() - '2 hour'::INTERVAL),
('client3', 'UK', now() - '2 hour'::INTERVAL),
('client4', 'UK', now() - '1 hour'::INTERVAL),
('client5', 'CN', now() - '1 hour'::INTERVAL),
('client6', 'CN', now() - '1 hour'::INTERVAL),
('client7', 'JP', now()),
('client8', 'JP', now()),
('client9', 'KR', now()),
('client10', 'KR', now());
```
Wait for few seconds for the Flow to write the result to the sink table and then query:
```sql
select * from ngx_country;
```
```sql
+---------+---------------------+----------------------------+
| country | time_window | update_at |
+---------+---------------------+----------------------------+
| CN | 2025-04-24 05:00:00 | 2025-04-24 06:55:17.217000 |
| JP | 2025-04-24 06:00:00 | 2025-04-24 06:55:17.217000 |
| KR | 2025-04-24 06:00:00 | 2025-04-24 06:55:17.217000 |
| UK | 2025-04-24 04:00:00 | 2025-04-24 06:55:17.217000 |
| UK | 2025-04-24 05:00:00 | 2025-04-24 06:55:17.217000 |
| US | 2025-04-24 04:00:00 | 2025-04-24 06:55:17.217000 |
+---------+---------------------+----------------------------+
6 rows in set (0.00 sec)
```
## Real-Time Monitoring Example
Consider a usecase where you have a stream of sensor events from a network of temperature sensors that you want to monitor in real-time. The sensor events contain information such as the sensor ID, the temperature reading, the timestamp of the reading, and the location of the sensor. You want to continuously aggregate this data to provide real-time alerts when the temperature exceeds a certain threshold. Then the query for continuous aggregation would be:
```sql
/* create source table */
CREATE TABLE temp_sensor_data (
sensor_id INT,
loc STRING,
temperature DOUBLE,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY(sensor_id, loc)
)
WITH(
append_mode = 'true'
);
/* create sink table */
CREATE TABLE temp_alerts (
sensor_id INT,
loc STRING,
max_temp DOUBLE,
time_window TIMESTAMP TIME INDEX,
update_at TIMESTAMP,
PRIMARY KEY(sensor_id, loc)
);
CREATE FLOW temp_monitoring
SINK TO temp_alerts
EXPIRE AFTER '1h'
AS
SELECT
sensor_id,
loc,
max(temperature) as max_temp,
date_bin('10 seconds'::INTERVAL, ts) as time_window
FROM temp_sensor_data
GROUP BY
sensor_id,
loc,
time_window
HAVING max_temp > 100;
```
The above query continuously aggregates data from the `temp_sensor_data` table into the `temp_alerts` table.
It calculates the maximum temperature reading for each sensor and location,
filtering out data where the maximum temperature exceeds 100 degrees.
The `temp_alerts` table will be continuously updated with the aggregated data,
providing real-time alerts (in the form of new rows in the `temp_alerts` table) when the temperature exceeds the threshold. The `EXPIRE AFTER '1h'` makes flow only calculate source data with `ts` in `(now - 1h, now)` range, see [Manage Flow](manage-flow.md#expire-after) for details.
Now that we have created the flow task, we can insert some data into the source table `temp_sensor_data`:
```sql
INSERT INTO temp_sensor_data VALUES
(1, 'room1', 98.5, now() - '10 second'::INTERVAL),
(2, 'room2', 99.5, now());
```
The table should be empty now; wait a few seconds for the Flow to update the sink table:
```sql
SELECT * FROM temp_alerts;
```
```sql
Empty set (0.00 sec)
```
Now insert some data that will trigger the alert:
```sql
INSERT INTO temp_sensor_data VALUES
(1, 'room1', 101.5, now()),
(2, 'room2', 102.5, now());
```
Wait a few seconds for the Flow to update the sink table:
```sql
SELECT * FROM temp_alerts;
```
```sql
+-----------+-------+----------+---------------------+----------------------------+
| sensor_id | loc | max_temp | time_window | update_at |
+-----------+-------+----------+---------------------+----------------------------+
| 1 | room1 | 101.5 | 2025-04-24 06:58:20 | 2025-04-24 06:58:32.379000 |
| 2 | room2 | 102.5 | 2025-04-24 06:58:20 | 2025-04-24 06:58:32.379000 |
+-----------+-------+----------+---------------------+----------------------------+
2 rows in set (0.01 sec)
```
## Real-Time Dashboard
Consider a usecase in which you need a bar graph that show the distribution of packet sizes for each status code to monitor the health of the system. The query for continuous aggregation would be:
```sql
/* create source table */
CREATE TABLE ngx_access_log (
client STRING,
stat INT,
size INT,
access_time TIMESTAMP TIME INDEX
)
WITH(
append_mode = 'true'
);
/* create sink table */
CREATE TABLE ngx_distribution (
stat INT,
bucket_size INT,
total_logs BIGINT,
time_window TIMESTAMP TIME INDEX,
update_at TIMESTAMP,
PRIMARY KEY(stat, bucket_size)
);
/* create flow task to calculate the distribution of packet sizes for each status code */
CREATE FLOW calc_ngx_distribution SINK TO ngx_distribution
EXPIRE AFTER '6h'
AS
SELECT
stat,
trunc(size, -1)::INT as bucket_size,
count(client) AS total_logs,
date_bin('1 minutes'::INTERVAL, access_time) as time_window
FROM
ngx_access_log
GROUP BY
stat,
time_window,
bucket_size;
```
The query aggregates data from the `ngx_access_log` table into the `ngx_distribution` table.
It computes the total number of logs for each status code and packet size bucket (bucket size of 10, as specified by `trunc` with a second argument of -1) within each time window.
The `date_bin` function groups the data into one-minute intervals.
The `EXPIRE AFTER '6h'` ensures that the flow computation only uses source data from the last 6 hours. See more details in [Manage Flow](manage-flow.md#expire-after).
Now that we have created the flow task, we can insert some data into the source table `ngx_access_log`:
```sql
INSERT INTO ngx_access_log VALUES
('cli1', 200, 100, now()),
('cli2', 200, 104, now()),
('cli3', 200, 120, now()),
('cli4', 200, 124, now()),
('cli5', 200, 140, now()),
('cli6', 404, 144, now()),
('cli7', 404, 160, now()),
('cli8', 404, 164, now()),
('cli9', 404, 180, now()),
('cli10', 404, 184, now());
```
Wait a few seconds for the Flow to update the sink table:
```sql
SELECT * FROM ngx_distribution;
```
```sql
+------+-------------+------------+---------------------+----------------------------+
| stat | bucket_size | total_logs | time_window | update_at |
+------+-------------+------------+---------------------+----------------------------+
| 200 | 100 | 2 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
| 200 | 120 | 2 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
| 200 | 140 | 1 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
| 404 | 140 | 1 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
| 404 | 160 | 2 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
| 404 | 180 | 2 | 2025-04-24 07:05:00 | 2025-04-24 07:05:56.308000 |
+------+-------------+------------+---------------------+----------------------------+
6 rows in set (0.00 sec)
```
## Using TQL with Flow for Advanced Time-Series Analysis
:::warning Experimental Feature
This experimental feature may contain unexpected behavior and have its functionality change in the future.
:::
A Flow query can wrap TQL (Time Query Language), which gives you rate calculations, moving averages, and other time-window operations that are awkward to express in plain SQL.
### Understanding TQL Flow Components
The TQL integration with Flow provides several advantages:
1. **Time Range Specification**: The `EVAL (start_time, end_time, step)` syntax allows precise control over the evaluation window, see [TQL](/reference/sql/tql.md).
2. **Automatic Schema Generation**: GreptimeDB creates appropriate sink tables based on TQL function outputs
3. **Continuous Processing**: Combined with Flow's scheduling, TQL functions run continuously on incoming data
4. **Advanced Analytics**: Access to sophisticated time-series functions like `rate()`, `increase()`, and statistical aggregations
### Setting Up the Source Table
First, let's create a source table to store HTTP request metrics:
```sql
CREATE TABLE http_requests_total (
host STRING,
job STRING,
instance STRING,
byte DOUBLE,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY (host, job, instance)
);
```
This table will serve as the data source for our TQL-based Flow computations. The `ts` column acts as the time index, while `byte` represents the metric values we want to analyze.
### Creating a Rate Calculation Flow
Now we'll create a Flow that uses TQL to calculate the rate of `byte` over time:
```sql
CREATE FLOW calc_rate
SINK TO rate_reqs
EVAL INTERVAL '1m' AS
TQL EVAL (now() - '1m'::interval, now(), '30s') rate(http_requests_total{job="my_service"}[1m]);
```
This Flow definition includes several key components:
- **EVAL INTERVAL '1m'**: Executes the Flow every minute for continuous updates
- **TQL EVAL**: Specifies the time range for evaluation from 1 minute ago to now, see [TQL](/reference/sql/tql.md).
- **rate()**: TQL function that calculates the rate of change
- **[1m]**: Defines a 1-minute lookback window for the rate calculation
### Wrapping TQL with a CTE
If you want a cleaner flow definition or need stable output column names, you can wrap `TQL EVAL` in a simple CTE inside `CREATE FLOW`:
```sql
CREATE FLOW calc_rate_cte
SINK TO rate_reqs_cte
EVAL INTERVAL '1m' AS
WITH rate_data (ts, req_rate, host, job, instance) AS (
TQL EVAL (now() - '1m'::interval, now(), '30s')
rate(http_requests_total{job="my_service"}[1m])
AS req_rate
)
SELECT * FROM rate_data;
```
This pattern is useful when you want to rename columns before GreptimeDB infers the sink table schema. It is especially handy for TQL expressions whose generated value column names are verbose.
The supported shape is intentionally limited:
- Use exactly one TQL CTE.
- End the flow query with `SELECT * FROM `.
- Do not add `WHERE`, joins, extra projections, or additional CTEs around it.
- If you quote the CTE name, keep the same quoted form in the outer query.
### Examining the Generated Sink Table
You can inspect the automatically created sink table structure:
```sql
SHOW CREATE TABLE rate_reqs;
```
```sql
+-----------+-------------------------------------+
| Table | Create Table |
+-----------+-------------------------------------+
| rate_reqs | CREATE TABLE IF NOT EXISTS `rate_reqs` (
`ts` TIMESTAMP(3) NOT NULL,
`prom_rate(ts_range,byte,ts,Int64(60000))` DOUBLE NULL,
`host` STRING NULL,
`job` STRING NULL,
`instance` STRING NULL,
TIME INDEX (`ts`),
PRIMARY KEY (`host`, `job`, `instance`)
)
ENGINE=mito
|
+-----------+-------------------------------------+
```
This shows how GreptimeDB automatically generates the appropriate schema for storing TQL computation results, creating a table with the same structure as the prom ql query result.
### Testing with Sample Data
Let's insert some test data to see the Flow in action:
```sql
INSERT INTO TABLE http_requests_total VALUES
('localhost', 'my_service', 'instance1', 100, now() - INTERVAL '2' minute),
('localhost', 'my_service', 'instance1', 200, now() - INTERVAL '1' minute),
('remotehost', 'my_service', 'instance1', 300, now() - INTERVAL '30' second),
('remotehost', 'their_service', 'instance1', 300, now() - INTERVAL '30' second),
('localhost', 'my_service', 'instance1', 400, now());
```
This creates a simple increasing sequence of values over time, which will produce a measurable rate when processed by our TQL Flow.
### Triggering Flow Execution
To manually trigger the Flow computation and see immediate results:
```sql
ADMIN FLUSH_FLOW('calc_rate');
```
This command forces the Flow to process all available data immediately, rather than waiting for the next scheduled interval.
### Verifying Results
Finally, verify that the Flow has successfully processed the data:
```sql
SELECT count(*) > 0 FROM rate_reqs;
```
```sql
+---------------------+
| count(*) > Int64(0) |
+---------------------+
| true |
+---------------------+
```
This query confirms that the rate calculation has produced results and populated the sink table with computed rate values.
You can also examine the actual computed rate values:
```sql
SELECT * FROM rate_reqs;
```
```sql
+---------------------+------------------------------------------+-----------+------------+-----------+
| ts | prom_rate(ts_range,byte,ts,Int64(60000)) | host | job | instance |
+---------------------+------------------------------------------+-----------+------------+-----------+
| 2025-09-01 13:14:34 | 4.166666666666666 | localhost | my_service | instance1 |
| 2025-09-01 13:15:04 | 4.444444444444444 | localhost | my_service | instance1 |
+---------------------+------------------------------------------+-----------+------------+-----------+
2 rows in set (0.03 sec)
```
Note that the timestamps and exact rate values may vary depending on when you run the example, but you should see similar rate calculations based on the input data pattern.
### Cleanup
When you're done experimenting, clean up the resources:
```sql
DROP FLOW calc_rate;
DROP FLOW calc_rate_cte;
DROP TABLE http_requests_total;
DROP TABLE rate_reqs;
DROP TABLE rate_reqs_cte;
```
## Next Steps
- [Manage Flow](manage-flow.md): Gain insights into the mechanisms of the Flow engine and the SQL syntax for defining a Flow.
- [Expressions](expressions.md): Learn about the expressions supported by the Flow engine for data transformation.
---
## Expressions
## Aggregate functions
Flow supports aggregate functions that are supported by the SQL query engine and the Flow plan, such as `COUNT`, `SUM`, `MIN`, and `MAX`. Unsupported query plans fail when the Flow is created. For a detailed list, please refer to [Aggregate Functions](/reference/sql/functions/df-functions.md#aggregate-functions).
## Scalar functions
Flow supports scalar functions that are supported by the SQL query engine and the Flow plan. Unsupported query plans fail when the Flow is created. See our [SQL reference](/reference/sql/functions/overview.md) for the function catalogue.
Here are some commonly used scalar functions in Flow:
- [`date_bin`](/reference/sql/functions/df-functions.md#date_bin): calculates time intervals and returns the start of the interval nearest to the specified timestamp.
- [`date_trunc`](/reference/sql/functions/df-functions.md#date_trunc): truncate a timestamp value to a specified precision.
- [`trunc`](/reference/sql/functions/df-functions.md#trunc): truncate a number to a whole number or to the specified decimal places.
---
## Manage Flows
Each `flow` is a continuous aggregation query in GreptimeDB.
It continuously updates the aggregated data based on the incoming data.
This document describes how to create, and delete a flow.
:::note
`EVAL INTERVAL` schedules batching evaluations but does not select batching mode. TQL workloads require it. See [Create a flow](#create-a-flow) for execution routing and instant-TTL restrictions.
:::
## Create a Source Table
Before creating a flow, you need to create a source table to store the raw data. Like this:
```sql
CREATE TABLE temp_sensor_data (
sensor_id INT,
loc STRING,
temperature DOUBLE,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY(sensor_id, loc)
);
```
For new workloads, avoid `WITH ('ttl' = 'instant')` on Flow source tables. This is a legacy pattern and is not recommended for new aggregation or TQL workloads. Keep source data with an appropriate retention policy instead.
## Create a Sink Table
A flow stores its aggregated data in a sink table. When the sink table does not exist, `CREATE FLOW`
automatically creates it when the query result is sufficient to infer its schema. Pre-create the sink when you
need control over its schema or layout, or when inference is complex. An existing sink table is validated against
the flow's query result. The source and sink tables must be different tables.
The sink table has to be compatible with the flow's query result:
- **Column order and type**: For a pre-created SQL sink, match the query output columns in order and type.
- **Time index**: Specify the `TIME INDEX` for the sink table, typically using the time window column generated by the time window function.
- **Update time**: For an auto-created batching SQL sink, Flow adds an `update_at` column for the update time. TQL sinks follow the query output and do not automatically add `update_at`. A pre-created SQL sink can either match the query output width or include one extra trailing timestamp column for update time.
- **Tags**: Use `PRIMARY KEY` to specify Tags, which together with the time index serve as a unique identifier for row data and optimize query performance.
For example:
```sql
/* Create sink table */
CREATE TABLE temp_alerts (
sensor_id INT,
loc STRING,
max_temp DOUBLE,
time_window TIMESTAMP TIME INDEX,
update_at TIMESTAMP,
PRIMARY KEY(sensor_id, loc)
);
CREATE FLOW temp_monitoring
SINK TO temp_alerts
AS
SELECT
sensor_id,
loc,
max(temperature) AS max_temp,
date_bin('10 seconds'::INTERVAL, ts) AS time_window
FROM temp_sensor_data
GROUP BY
sensor_id,
loc,
time_window
HAVING max_temp > 100;
```
The sink table has the columns `sensor_id`, `loc`, `max_temp`, `time_window`, and `update_at`.
- The first four columns correspond to the query result columns of flow: `sensor_id`, `loc`, `max(temperature)` and `date_bin('10 seconds'::INTERVAL, ts)` respectively.
- The `time_window` column is specified as the `TIME INDEX` for the sink table.
- The `update_at` column is the last one in the schema to store the update time of the data.
- The `PRIMARY KEY` at the end of the schema definition specifies `sensor_id` and `loc` as the tag columns.
This means the flow will insert or update data based on the tags `sensor_id` and `loc` along with the time index `time_window`.
## Create a flow
The grammar to create a flow is:
```sql
CREATE [ OR REPLACE ] FLOW [ IF NOT EXISTS ]
SINK TO
[ EXPIRE AFTER ]
[ EVAL INTERVAL ]
[ COMMENT '' ]
[ WITH ( = [, ...]) ]
AS
;
```
The clauses must appear in the order shown: `EXPIRE AFTER` comes before `EVAL INTERVAL`.
`EVAL INTERVAL` schedules batching evaluations. TQL flows require it. SQL plans containing `Aggregate` or `Distinct` use
batching, unless an instant-TTL source selects legacy streaming; ordinary projections and non-aggregate joins also use
legacy streaming. Streaming ignores `EVAL INTERVAL`. For batching time-window SQL, an evaluation can be incremental
rather than a full-query evaluation. Batching SQL without a usable time window requires `EVAL INTERVAL` and executes the full query; time-window aggregates can run without it.
When `OR REPLACE` is specified, any existing flow with the same name will be updated to the new version. It's important to note that this only affects the flow task itself; the source and sink tables will remain unchanged.
Conversely, when `IF NOT EXISTS` is specified, the command will have no effect if the flow already exists, rather than reporting an error. Additionally, please note that `OR REPLACE` cannot be used in conjunction with `IF NOT EXISTS`.
- `flow-name` is a unique identifier at the catalog level.
- `sink-table-name` is the table name where the materialized aggregated data is stored.
It can be an existing table or a new one; see [Create a Sink Table](#create-a-sink-table) for creation and validation behavior.
- `EXPIRE AFTER` is an optional interval to expire data from the Flow engine. For more details, please refer to the [`EXPIRE AFTER`](#expire-after) section.
- `EVAL INTERVAL` is an optional interval for batching evaluations; streaming ignores it.
- `COMMENT` is the description of the flow.
- `WITH` specifies flow options.
The user-facing options documented below are `defer_on_missing_source` and the experimental `experimental_enable_incremental_read`.
- `SQL` part defines the continuous aggregation query.
It defines the source tables that provide data for the flow.
Each flow can have multiple source tables.
Please refer to [Write a SQL query](#write-a-sql-query) for details.
A simple example to create a flow:
```sql
CREATE FLOW IF NOT EXISTS my_flow
SINK TO my_sink_table
EXPIRE AFTER '1 hour'::INTERVAL
COMMENT 'My first flow in GreptimeDB'
AS
SELECT
max(temperature) as max_temp,
date_bin('10 seconds'::INTERVAL, ts) as time_window
FROM temp_sensor_data
GROUP BY time_window;
```
The created flow groups `max(temperature)` into 10-second windows and stores the result in `my_sink_table`. Data within the last hour is used in the flow.
### EXPIRE AFTER
The `EXPIRE AFTER` clause specifies the interval after which data will expire from the flow engine.
For a Flow with a usable time-window expression, data in the source table older than the specified interval is excluded from calculations, and older sink rows are not updated. This limits the state and recomputation range for time-window flows, including stateful queries such as those involving `GROUP BY`.
Batching plans without a usable time-window expression require `EVAL INTERVAL` and execute unfiltered snapshots unless their query has a time predicate; `EXPIRE AFTER` does not add a time filter. It does not delete data from either table. If you want to delete data from the source or sink table, please [set the `TTL` option](/user-guide/manage-data/overview.md#manage-data-retention-with-ttl-policies) when creating tables.
For example, if the flow engine processes the aggregation at 10:00:00 and the `'1 hour'::INTERVAL` is set,
any input data that arrive now with a time index older than 1 hour (before 09:00:00) will expire and be ignored.
Only data timestamped from 09:00:00 onwards will be used in the aggregation and to update the sink table.
### Defer creation when a source is missing
By default, creating a Flow fails if one of its source tables does not exist. Set
`defer_on_missing_source` to `true` to persist a pending Flow instead of failing. The Flow is not scheduled while its
sources remain unresolved, and it is not activated automatically when those tables are created. `CREATE OR REPLACE`
cannot activate a pending Flow.
```sql
CREATE FLOW pending_flow
SINK TO pending_sink
WITH (defer_on_missing_source = 'true')
AS
SELECT * FROM source_created_later;
```
After all source tables have been created, drop and recreate the Flow to activate it.
```sql
DROP FLOW pending_flow;
CREATE FLOW pending_flow
SINK TO pending_sink
AS
SELECT * FROM source_created_later;
```
### Experimental incremental source reads
:::warning Experimental feature
The `experimental_enable_incremental_read` option is experimental.
Its behavior and limitations may change in future releases.
:::
For batching SQL flows whose source tables are append-only, you can enable incremental source reads:
```sql
CREATE TABLE temp_sensor_data (
sensor_id INT,
loc STRING,
temperature DOUBLE,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY(sensor_id, loc)
) WITH ('append_mode' = 'true');
CREATE FLOW temp_monitoring
SINK TO temp_alerts
WITH (experimental_enable_incremental_read = 'true')
AS
SELECT
sensor_id,
loc,
max(temperature) AS max_temp,
date_bin('10 seconds'::INTERVAL, ts) AS time_window
FROM temp_sensor_data
GROUP BY
sensor_id,
loc,
time_window;
```
When enabled, Flow attempts to read only newly appended source rows after the initial full snapshot.
This is an execution optimization and does not change the query result. The optimization is not a persistence
contract: the first run, and a run after a restart or when incremental reading is not safe, may use a full snapshot.
The current limitations are:
- All source tables must be append-only tables created with `append_mode = 'true'`.
Flow creation fails if any source table is not append-only.
- The optimization applies only to eligible batching SQL flows. TQL flows and plans that do not support incremental
reads use the normal full-snapshot behavior.
### Write a SQL query
The SQL after `AS` is planned as a standard SQL query. A typical batching time-window aggregate has this shape:
```sql
SELECT AGGR_FUNCTION(column1, column2,..) [, TIME_WINDOW_FUNCTION() as time_window]
FROM
GROUP BY {time_window | column1, column2,.. };
```
The query engine and Flow plan determine which SQL expressions and clauses are supported. For batching SQL flows
with `Aggregate` or `Distinct` plans, `EVAL INTERVAL` schedules evaluations; TQL flows require it. Planner-valid
joins, subqueries, and SQL CTEs are supported for batching SQL flows with `EVAL INTERVAL`. The query planner must
still produce a valid plan; unsupported queries fail when the Flow is created. For batching time-window aggregates,
`GROUP BY` commonly includes the time-window expression. See [Expressions](expressions.md)
for functions commonly used in Flow queries, and [Define time window](#define-time-window) for fixed windows.
Refer to [Continuous Aggregation](continuous-aggregation.md) for more examples of how to use continuous aggregation in real-time analytics, monitoring, and dashboards.
### Define time window
A time window is a crucial attribute of your continuous aggregation query.
It determines how data is aggregated within the flow.
These time windows are left-closed and right-open intervals.
A time window represents a specific range of time.
Data from the source table is mapped to the corresponding window based on the time index column.
The time window also defines the scope for each calculation of an aggregation expression,
resulting in one row per time window in the result table.
You can use `date_bin()` after the `SELECT` keyword to define fixed time windows.
For example:
```sql
SELECT
max(temperature) as max_temp,
date_bin('10 seconds'::INTERVAL, ts) as time_window
FROM temp_sensor_data
GROUP BY time_window;
```
In this example, the `date_bin('10 seconds'::INTERVAL, ts)` function creates 10-second time windows starting from UTC 00:00:00.
The `max(temperature)` function calculates the maximum temperature value within each time window.
For more details on the behavior of the function,
please refer to [`date_bin`](/reference/sql/functions/df-functions.md#date_bin).
:::tip NOTE
The time-window expression helps Flow determine how to update results incrementally. The appropriate window size
depends on the workload and query semantics.
:::
## Inspect flows
Use the following commands and system tables to inspect Flow definitions and runtime information:
| Command | Purpose |
| --- | --- |
| `SHOW FLOWS;` | List flows. |
| `SHOW CREATE FLOW my_flow;` | Return a Flow definition. |
| `SHOW FLOW STATUS LIKE 'my%';` | Return runtime statistics for matching flows. |
| `SELECT * FROM information_schema.flows;` | View Flow definitions. |
| `SELECT * FROM information_schema.flow_statistics;` | View Flow runtime statistics. |
Runtime fields can initially be `NULL`, and values can lag behind the latest state in distributed deployments. See [`SHOW FLOW STATUS`](/reference/sql/show.md#show-flow-status) and [`INFORMATION_SCHEMA.FLOW_STATISTICS`](/reference/sql/information-schema/flow-statistics.md) for the returned columns.
## Flush a flow
The flow engine automatically processes aggregation operations within a short period(i.e. few seconds) when new data arrives in the source table.
However, you can manually trigger the flow engine to process the aggregation operation immediately using the `ADMIN FLUSH_FLOW` command.
```sql
ADMIN FLUSH_FLOW('')
```
## Delete a flow
To delete a flow, use the following `DROP FLOW` clause:
```sql
DROP FLOW [IF EXISTS]
```
For example:
```sql
DROP FLOW IF EXISTS my_flow;
```
---
## Flow Computation
GreptimeDB's Flow engine enables real-time computation on incoming data.
It is particularly beneficial for Extract-Transform-Load (ETL) processes or for performing continuous aggregations such as sum, average, and other time-window calculations.
Flow materializes computation results in the sink table as source data is processed, so queries can read the computed results instead of recalculating them from raw data.
Use cases include:
- Keeping aggregates ready for dashboards and alerts, so queries read the sink table instead of scanning raw events.
- Downsampling data points, such as using average pooling, to reduce the volume of data for storage and analysis.
## Programming Model
:::note
Flow uses batching mode for aggregation and TQL workloads unless an instant-TTL source selects the deprecated streaming mode. Simple non-aggregation Flow queries also use streaming mode and are not recommended for new workloads.
:::
Upon data insertion into the source table, the data is made available to the Flow engine.
Flow then processes the specified computation and updates the sink table with the results.
Both the source and sink tables are time-series tables within GreptimeDB.
Before creating a Flow,
it is crucial to define the schemas for these tables and design the Flow to specify the computation logic.
This process is visually represented in the following image:
## Quick Start Example
As an example, count user agents in nginx logs.
The source table is `ngx_http_log`,
and the sink table is `user_agent_statistics`.
First, create the source table `ngx_http_log`.
To optimize performance for counting the `user_agent` field,
specify it as a `TAG` column type using the `PRIMARY KEY` keyword.
```sql
CREATE TABLE ngx_http_log (
ip_address STRING,
http_method STRING,
request STRING,
status_code INT16,
body_bytes_sent INT32,
user_agent STRING,
response_size INT32,
ts TIMESTAMP TIME INDEX,
PRIMARY KEY (ip_address, http_method, user_agent, status_code)
) WITH ('append_mode'='true');
```
Next, create the sink table `user_agent_statistics`.
The `update_at` column tracks the last update time of the record, which is automatically updated by the Flow engine.
Although all tables in GreptimeDB are time-series tables, this computation does not require time windows.
Therefore, the `__ts_placeholder` column is included as a time index placeholder.
```sql
CREATE TABLE user_agent_statistics (
user_agent STRING,
total_count INT64,
update_at TIMESTAMP,
__ts_placeholder TIMESTAMP TIME INDEX,
PRIMARY KEY (user_agent)
);
```
Finally, create the Flow `user_agent_flow` to count the occurrences of each user agent in the `ngx_http_log` table.
```sql
CREATE FLOW user_agent_flow
SINK TO user_agent_statistics
EVAL INTERVAL '1s'
AS
SELECT
user_agent,
COUNT(user_agent) AS total_count
FROM
ngx_http_log
GROUP BY
user_agent;
```
Once the Flow is created,
the Flow engine will continuously process data from the `ngx_http_log` table and update the `user_agent_statistics` table with the computed results.
To observe the results,
insert sample data into the `ngx_http_log` table.
```sql
INSERT INTO ngx_http_log
VALUES
('192.168.1.1', 'GET', '/index.html', 200, 512, 'Mozilla/5.0', 1024, '2023-10-01T10:00:00Z'),
('192.168.1.2', 'POST', '/submit', 201, 256, 'curl/7.68.0', 512, '2023-10-01T10:01:00Z'),
('192.168.1.1', 'GET', '/about.html', 200, 128, 'Mozilla/5.0', 256, '2023-10-01T10:02:00Z'),
('192.168.1.3', 'GET', '/contact', 404, 64, 'curl/7.68.0', 128, '2023-10-01T10:03:00Z');
```
After inserting the data,
query the `user_agent_statistics` table to view the results.
```sql
SELECT * FROM user_agent_statistics;
```
The query results will display the total count of each user agent in the `user_agent_statistics` table.
```sql
+-------------+-------------+----------------------------+---------------------+
| user_agent | total_count | update_at | __ts_placeholder |
+-------------+-------------+----------------------------+---------------------+
| Mozilla/5.0 | 2 | 2024-12-12 06:45:33.228000 | 1970-01-01 00:00:00 |
| curl/7.68.0 | 2 | 2024-12-12 06:45:33.228000 | 1970-01-01 00:00:00 |
+-------------+-------------+----------------------------+---------------------+
```
## Next Steps
- [Continuous Aggregation](./continuous-aggregation.md): Explore the primary scenario in time-series data processing, with three common use cases for continuous aggregation.
- [Manage Flow](manage-flow.md): Gain insights into the mechanisms of the Flow engine and the SQL syntax for defining a Flow.
- [Expressions](expressions.md): Learn about the expressions supported by the Flow engine for data transformation.
---
## EMQX
[EMQX](https://www.emqx.io/) is an open-source, highly scalable, and feature-rich MQTT broker designed for IoT and real-time messaging applications. It supports various protocols, including MQTT (3.1, 3.1.1, and 5.0), HTTP, QUIC, and WebSocket.
GreptimeDB can be used as a data system for EMQX. To learn how to integrate GreptimeDB with EMQX, please refer to [Ingest MQTT Data into GreptimeDB](https://docs.emqx.com/en/emqx/latest/data-integration/data-bridge-greptimedb.html).
---
## Go
GreptimeDB offers ingester libraries for high-throughput data writing.
It utilizes the gRPC protocol,
which supports schemaless writing and eliminates the need to create tables before writing data.
For more information, refer to [Automatic Schema Generation](/user-guide/ingest-data/overview.md#automatic-schema-generation).
The Go ingester SDK provided by GreptimeDB is a lightweight,
concurrent-safe library that is easy to use with the metric struct.
## Quick start demos
To quickly get started, you can explore the [quick start demos](https://github.com/GreptimeTeam/greptimedb-ingester-go/tree/main/examples) to understand how to use the GreptimeDB Go ingester SDK.
## Installation
Use the following command to install the GreptimeDB client library for Go:
```shell
go get -u github.com/GreptimeTeam/greptimedb-ingester-go@v0.7.2
```
Import the library in your code:
```go
greptime "github.com/GreptimeTeam/greptimedb-ingester-go"
ingesterContext "github.com/GreptimeTeam/greptimedb-ingester-go/context"
"github.com/GreptimeTeam/greptimedb-ingester-go/table"
"github.com/GreptimeTeam/greptimedb-ingester-go/table/types"
)
```
## Connect to database
If you have set the [`--user-provider` configuration](/user-guide/deployments-administration/authentication/overview.md) when starting GreptimeDB,
you will need to provide a username and password to connect to GreptimeDB.
The following example shows how to set the username and password when using the library to connect to GreptimeDB.
```go
cfg := greptime.NewConfig("127.0.0.1").
// change the database name to your database name
WithDatabase("public").
// Default port 4001
// WithPort(4001).
// Enable secure connection if your server is secured by TLS
// WithInsecure(false).
// set authentication information
// If the database doesn't require authentication, just remove the WithAuth method
WithAuth("username", "password")
cli, _ := greptime.NewClient(cfg)
defer cli.Close()
```
## Data model
Each row item in a table consists of three types of columns: `Tag`, `Timestamp`, and `Field`. For more information, see [Data Model](/user-guide/concepts/data-model.md).
The types of column values could be `String`, `Float`, `Int`, `Timestamp`, `JSON` etc. For more information, see [Data Types](/reference/sql/data-types.md).
## Set table options
Although the time series table is created automatically when writing data to GreptimeDB via the SDK,
you can still configure table options.
The SDK supports the following table options:
- `auto_create_table`: Default is `True`. If set to `False`, it indicates that the table already exists and does not need automatic creation, which can improve write performance.
- `ttl`, `append_mode`, `merge_mode`: For more details, refer to the [table options](/reference/sql/create.md#table-options).
You can set table options using the `ingesterContext` context.
For example, to set the `ttl` option, use the following code:
```go
hints := []*ingesterContext.Hint{
{
Key: "ttl",
Value: "3d",
},
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*3)
ctx = ingesterContext.New(ctx, ingesterContext.WithHint(hints))
// Use the ingesterContext when writing data to GreptimeDB.
// The `data` object is described in the following sections.
resp, err := cli.Write(ctx, data)
if err != nil {
return err
}
```
For how to write data to GreptimeDB, see the following sections.
## Low-level API
The GreptimeDB low-level API provides a straightforward method to write data to GreptimeDB
by adding rows to the table object with a predefined schema.
### Create row objects
This following code snippet begins by constructing a table named `cpu_metric`,
which includes columns `host`, `cpu_user`, `cpu_sys`, and `ts`.
Subsequently, it inserts a single row into the table.
The table consists of three types of columns:
- `Tag`: The `host` column, with values of type `String`.
- `Field`: The `cpu_user` and `cpu_sys` columns, with values of type `Float`.
- `Timestamp`: The `ts` column, with values of type `Timestamp`.
```go
// Construct the table schema for CPU metrics
cpuMetric, err := table.New("cpu_metric")
if err != nil {
// Handle error appropriately
}
// Add a 'Tag' column for host identifiers
cpuMetric.AddTagColumn("host", types.STRING)
// Add a 'Timestamp' column for recording the time of data collection
cpuMetric.AddTimestampColumn("ts", types.TIMESTAMP_MILLISECOND)
// Add 'Field' columns for user and system CPU usage measurements
cpuMetric.AddFieldColumn("cpu_user", types.FLOAT)
cpuMetric.AddFieldColumn("cpu_sys", types.FLOAT)
// Insert example data
// NOTE: The arguments must be in the same order as the columns in the defined schema: host, ts, cpu_user, cpu_sys
err = cpuMetric.AddRow("127.0.0.1", time.Now(), 0.1, 0.12)
err = cpuMetric.AddRow("127.0.0.1", time.Now(), 0.11, 0.13)
if err != nil {
// Handle error appropriately
}
```
To improve the efficiency of writing data, you can create multiple rows at once to write to GreptimeDB.
```go
cpuMetric, err := table.New("cpu_metric")
if err != nil {
// Handle error appropriately
}
cpuMetric.AddTagColumn("host", types.STRING)
cpuMetric.AddTimestampColumn("ts", types.TIMESTAMP_MILLISECOND)
cpuMetric.AddFieldColumn("cpu_user", types.FLOAT)
cpuMetric.AddFieldColumn("cpu_sys", types.FLOAT)
err = cpuMetric.AddRow("127.0.0.1", time.Now(), 0.1, 0.12)
if err != nil {
// Handle error appropriately
}
memMetric, err := table.New("mem_metric")
if err != nil {
// Handle error appropriately
}
memMetric.AddTagColumn("host", types.STRING)
memMetric.AddTimestampColumn("ts", types.TIMESTAMP_MILLISECOND)
memMetric.AddFieldColumn("mem_usage", types.FLOAT)
err = memMetric.AddRow("127.0.0.1", time.Now(), 112)
if err != nil {
// Handle error appropriately
}
```
### Insert data
The following example shows how to insert rows to tables in GreptimeDB.
```go
resp, err := cli.Write(ctx, cpuMetric, memMetric)
if err != nil {
// Handle error appropriately
}
log.Printf("affected rows: %d\n", resp.GetAffectedRows().GetValue())
```
### Streaming insert
Streaming insert is useful when you want to insert a large amount of data such as importing historical data.
```go
err := cli.StreamWrite(ctx, cpuMetric, memMetric)
if err != nil {
// Handle error appropriately
}
```
Close the stream writing after all data has been written.
In general, you do not need to close the stream writing when continuously writing data.
```go
affected, err := cli.CloseStream(ctx)
```
## High-level API
The high-level API uses an ORM style object to write data to GreptimeDB.
It allows you to create, insert, and update data in a more object-oriented way,
providing developers with a friendlier experience.
However, it is not as efficient as the low-level API.
This is because the ORM style object may consume more resources and time when converting the objects.
### Create row objects
```go
type CpuMetric struct {
Host string `greptime:"tag;column:host;type:string"`
CpuUser float64 `greptime:"field;column:cpu_user;type:float64"`
CpuSys float64 `greptime:"field;column:cpu_sys;type:float64"`
Ts time.Time `greptime:"timestamp;column:ts;type:timestamp;precision:millisecond"`
}
func (CpuMetric) TableName() string {
return "cpu_metric"
}
cpuMetrics := []CpuMetric{
{
Host: "127.0.0.1",
CpuUser: 0.10,
CpuSys: 0.12,
Ts: time.Now(),
}
}
```
### Insert data
```go
resp, err := cli.WriteObject(ctx, cpuMetrics)
log.Printf("affected rows: %d\n", resp.GetAffectedRows().GetValue())
```
### Streaming insert
Streaming insert is useful when you want to insert a large amount of data such as importing historical data.
```go
err := cli.StreamWriteObject(ctx, cpuMetrics)
```
Close the stream writing after all data has been written.
In general, you do not need to close the stream writing when continuously writing data.
```go
affected, err := cli.CloseStream(ctx)
```
## Insert data in JSON type
GreptimeDB supports storing complex data structures using [JSON type data](/reference/sql/data-types.md#json-type-experimental).
With this ingester library, you can insert JSON data using string values.
For instance, if you have a table named `sensor_readings` and wish to add a JSON column named `attributes`,
refer to the following code snippet.
In the [low-level API](#low-level-api),
you can specify the column type as `types.JSON` using the `AddFieldColumn` method to add a JSON column.
Then, use a `struct` or `map` to insert JSON data.
```go
sensorReadings, err := table.New("sensor_readings")
// The code for creating other columns is omitted
// ...
// specify the column type as JSON
sensorReadings.AddFieldColumn("attributes", types.JSON)
// Use struct to insert JSON data
type Attributes struct {
Location string `json:"location"`
Action string `json:"action"`
}
attributes := Attributes{ Location: "factory-1" }
sensorReadings.AddRow(... , attributes)
// The following code for writing data is omitted
// ...
```
In the [high-level API](#high-level-api), you can specify the column type as JSON using the `greptime:"field;column:details;type:json"` tag.
```go
type SensorReadings struct {
// The code for creating other columns is omitted
// ...
// specify the column type as JSON
Attributes string `greptime:"field;column:details;type:json"`
// ...
}
// Use struct to insert JSON data
type Attributes struct {
Location string `json:"location"`
Action string `json:"action"`
}
attributes := Attributes{ Action: "running" }
sensor := SensorReadings{
// ...
Attributes: attributes,
}
// The following code for writing data is omitted
// ...
```
For the executable code for inserting JSON data, please refer to the [example](https://github.com/GreptimeTeam/greptimedb-ingester-go/tree/main/examples/jsondata) in the SDK repository.
## Bulk Write
When you need to import data in bulk and require high ingestion throughput, you can use the Bulk Write API. It is designed for batch loading scenarios, where large volumes of data are written efficiently.
By using Arrow IPC encoding to aggregate multiple tables or objects on the client side and sending them in a single gRPC request, it significantly reduces network overhead and improves overall write performance compared to the regular Write API.
### Adding Data
You can build data objects and add them to the bulk writer.
```go
// Build the CPU metrics table
cpuMetric, err := table.New("cpu_metric")
if err != nil {
// Handle error appropriately
}
cpuMetric.AddTagColumn("host", types.STRING)
cpuMetric.AddTimestampColumn("ts", types.TIMESTAMP_MILLISECOND)
cpuMetric.AddFieldColumn("cpu_user", types.FLOAT64)
cpuMetric.AddFieldColumn("cpu_sys", types.FLOAT64)
// Add multiple rows of data
cpuMetric.AddRow("127.0.0.1", time.Now(), 0.1, 0.12)
cpuMetric.AddRow("127.0.0.2", time.Now(), 0.2, 0.15)
```
### Insert data
```go
resp, err := cli.BulkWrite(context.TODO(), cpuMetric)
if err != nil {
// Handle error appropriately
}
log.Printf("Bulk write affected rows: %d\n", resp.GetAffectedRows().GetValue())
```
Please refer to the [example](https://github.com/GreptimeTeam/greptimedb-ingester-go/tree/main/examples/bulkwrite) in the SDK repository for runnable code demonstrating bulk write.
## Ingester library reference
- [API Documentation](https://pkg.go.dev/github.com/GreptimeTeam/greptimedb-ingester-go)
---
## Java Ingester for GreptimeDB
GreptimeDB offers ingester libraries for high-throughput data writing.
It utilizes the gRPC protocol,
which supports schemaless writing and eliminates the need to create tables before writing data.
For more information, refer to [Automatic Schema Generation](/user-guide/ingest-data/overview.md#automatic-schema-generation).
The Java ingester SDK provided by GreptimeDB is a lightweight, high-performance client designed for efficient time-series data ingestion. It leverages the gRPC protocol to provide a non-blocking, purely asynchronous API that delivers exceptional throughput while maintaining seamless integration with your applications.
This client offers multiple ingestion methods optimized for various performance requirements and use cases. You can select the approach that best suits your specific needs—whether you require simple unary writes for low-latency operations or high-throughput bulk streaming for maximum efficiency when handling large volumes of time-series data.
## High Level Architecture
```
+-----------------------------------+
| Client Applications |
| +------------------+ |
| | Application Code | |
| +------------------+ |
+-------------+---------------------+
|
v
+-------------+---------------------+
| API Layer |
| +---------------+ |
| | GreptimeDB | |
| +---------------+ |
| / \ |
| v v |
| +-------------+ +-------------+ | +------------------+
| | BulkWrite | | Write | | | Data Model |
| | Interface | | Interface | |------->| |
| +-------------+ +-------------+ | | +------------+ |
+-------|----------------|----------+ | | Table | |
| | | +------------+ |
v v | | |
+-------|----------------|----------+ | v |
| Transport Layer | | +------------+ |
| +-------------+ +-------------+ | | | TableSchema| |
| | BulkWrite | | Write | | | +------------+ |
| | Client | | Client | | +------------------+
| +-------------+ +-------------+ |
| | \ / | |
| | \ / | |
| | v v | |
| | +-------------+ | |
| | |RouterClient | | |
+-----|--+-------------|---+--------+
| | | |
| | | |
v v v |
+-----|----------------|---|--------+
| Network Layer |
| +-------------+ +-------------+ |
| | Arrow Flight| | gRPC Client | |
| | Client | | | |
| +-------------+ +-------------+ |
| | | |
+-----|----------------|------------+
| |
v v
+-------------------------+
| GreptimeDB Server |
+-------------------------+
```
- **API Layer**: Provides high-level interfaces for client applications to interact with GreptimeDB
- **Data Model**: Defines the structure and organization of time series data with tables and schemas
- **Transport Layer**: Handles communication logistics, request routing, and client management
- **Network Layer**: Manages low-level protocol communications using Arrow Flight and gRPC
## How To Use
### Installation
1. Install the Java Development Kit(JDK)
Make sure that your system has JDK 8 or later installed. For more information on how to
check your version of Java and install the JDK, see the [Oracle Overview of JDK Installation documentation](https://www.oracle.com/java/technologies/javase-downloads.html)
2. Add GreptimeDB Java SDK as a Dependency
If you are using [Maven](https://maven.apache.org/), add the following to your pom.xml
dependencies list:
```xml
io.greptime
ingester-all
0.15.0
```
The latest version can be viewed [here](https://central.sonatype.com/search?q=io.greptime&name=ingester-all).
After configuring your dependencies, make sure they are available to your project. This may require refreshing the project in your IDE or running the dependency manager.
### Client Initialization
The entry point to the GreptimeDB Ingester Java client is the `GreptimeDB` class. You create a client instance by calling the static create method with appropriate configuration options.
```java
// GreptimeDB has a default database named "public" in the default catalog "greptime",
// we can use it as the test database
String database = "public";
// By default, GreptimeDB listens on port 4001 using the gRPC protocol.
// We can provide multiple endpoints that point to the same GreptimeDB cluster.
// The client will make calls to these endpoints based on a load balancing strategy.
// The client performs regular health checks and automatically routes requests to healthy nodes,
// providing fault tolerance and improved reliability for your application.
String[] endpoints = {"127.0.0.1:4001"};
// Sets authentication information.
AuthInfo authInfo = new AuthInfo("username", "password");
GreptimeOptions opts = GreptimeOptions.newBuilder(endpoints, database)
// If the database does not require authentication, we can use `AuthInfo.noAuthorization()` as the parameter.
.authInfo(authInfo)
// Enable secure connection if your server is secured by TLS
//.tlsOptions(new TlsOptions())
// A good start ^_^
.build();
// Initialize the client
// NOTE: The client instance is thread-safe and should be reused as a global singleton
// for better performance and resource utilization.
GreptimeDB client = GreptimeDB.create(opts);
```
### Writing Data
The ingester provides a unified approach for writing data to GreptimeDB through the `Table` abstraction. All data writing operations, including high-level APIs, are built on top of this fundamental structure. To write data, you create a `Table` with your time series data and write it to the database.
#### Creating and Writing Tables
Define a table schema and create a table:
```java
// Create a table schema
TableSchema schema = TableSchema.newBuilder("metrics")
.addTag("host", DataType.String)
.addTag("region", DataType.String)
.addField("cpu_util", DataType.Float64)
.addField("memory_util", DataType.Float64)
.addTimestamp("ts", DataType.TimestampMillisecond)
.build();
// Create a table from the schema
Table table = Table.from(schema);
// Add rows to the table
// The values must be provided in the same order as defined in the schema
// In this case: addRow(host, region, cpu_util, memory_util, ts)
table.addRow("host1", "us-west-1", 0.42, 0.78, System.currentTimeMillis());
table.addRow("host2", "us-west-2", 0.46, 0.66, System.currentTimeMillis());
// Add more rows
// ..
// Complete the table to make it immutable. This finalizes the table for writing.
// If users forget to call this method, it will automatically be called internally
// before the table data is written.
table.complete();
// Write the table to the database
CompletableFuture> future = client.write(table);
```
GreptimeDB supports storing complex data structures using [JSON type data](/reference/sql/data-types.md#json-type-experimental). You can define JSON columns in your table schema and insert data using Map objects:
```java
// Construct the table schema for sensor_readings
TableSchema sensorReadings = TableSchema.newBuilder("sensor_readings")
// The code for creating other columns is omitted
// ...
// specify the column type as JSON
.addField("attributes", DataType.Json)
.build();
// ...
// Use map to insert JSON data
Map attr = new HashMap<>();
attr.put("location", "factory-1");
Table table = Table.from(sensorReadings);
table.addRow(... , attr);
```
##### TableSchema
The `TableSchema` defines the structure for writing data to GreptimeDB. It specifies the table structure including column names, semantic types, and data types. For detailed information about column semantic types (`Tag`, `Timestamp`, `Field`), refer to the [Data Model](/user-guide/concepts/data-model.md) documentation.
##### Table
The `Table` interface represents data that can be written to GreptimeDB. It provides methods for adding rows and manipulating the data. Essentially, `Table` temporarily stores data in memory, allowing you to accumulate multiple rows for batch processing before sending them to the database, which significantly improves write efficiency compared to writing individual rows.
A table goes through several distinct lifecycle stages:
1. **Creation**: Initialize a table from a schema using `Table.from(schema)`
2. **Data Addition**: Populate the table with rows using `addRow()` method
3. **Completion**: Finalize the table with `complete()` when all rows have been added
4. **Writing**: Send the completed table to the database
Important considerations:
- Tables are not thread-safe and should be accessed from a single thread
- Tables cannot be reused after writing - create a new instance for each write operation
- The associated `TableSchema` is immutable and can be safely reused across multiple operations
### Write Operations
Although the time series table is created automatically when writing data to GreptimeDB via the SDK,
you can still configure table options.
The SDK supports the following table options:
- `auto_create_table`: Default is `True`. If set to `False`, it indicates that the table already exists and does not need automatic creation, which can improve write performance.
- `ttl`, `append_mode`, `merge_mode`: For more details, refer to the [table options](/reference/sql/create.md#table-options).
You can set table options using the `Context`.
For example, to set the `ttl` option, use the following code:
```java
Context ctx = Context.newDefault();
// Add a hint to make the database create a table with the specified TTL (time-to-live)
ctx = ctx.withHint("ttl", "3d");
// Set the compression algorithm to Zstd.
ctx = ctx.withCompression(Compression.Zstd);
// Use the ctx when writing data to GreptimeDB
CompletableFuture> future = client.write(Arrays.asList(table1, table2), WriteOp.Insert, ctx);
```
For how to write data to GreptimeDB, see the following sections.
### Batching Write
Batching write allows you to write data to multiple tables in a single request. It returns a `CompletableFuture>` and provides good performance through asynchronous execution.
This is the recommended way to write data to GreptimeDB for most use cases.
```java
// The batching write API
CompletableFuture> future = client.write(table1, table2, table3);
// For performance reasons, the SDK is designed to be purely asynchronous.
// The return value is a CompletableFuture object. If you want to immediately obtain
// the result, you can call `future.get()`, which will block until the operation completes.
// For production environments, consider using non-blocking approaches with callbacks or
// the CompletableFuture API.
Result result = future.get();
```
### Streaming Write
The streaming write API maintains a persistent connection to GreptimeDB for continuous data ingestion with rate limiting. It allows writing data from multiple tables through a single stream.
Use this API when you need:
- Continuous data collection with moderate volume
- Writing to multiple tables via one connection
- Cases where simplicity and convenience are more important than maximum throughput
```java
// Create a stream writer
StreamWriter writer = client.streamWriter();
// Write multiple tables
writer.write(table1)
.write(table2)
.write(table3);
// Complete the stream and get the result
CompletableFuture result = writer.completed();
```
You can also set a rate limit for stream writing:
```java
// Limit to 1000 points per second
StreamWriter writer = client.streamWriter(1000);
```
### Bulk Write
The Bulk Write API provides a high-performance, memory-efficient mechanism for ingesting large volumes of time-series data into GreptimeDB. It leverages off-heap memory management to achieve optimal throughput when writing batches of data.
**Important**:
1. **Manual Table Creation Required**: Bulk API does **not** create tables automatically. You must create the table beforehand using either:
- Insert API (which supports auto table creation), or
- SQL DDL statements (CREATE TABLE)
2. **Schema Matching**: The table template in bulk API must exactly match the existing table schema.
This API supports writing to one table per stream and handles large data volumes (up to 200MB per write) with adaptive flow control. Performance advantages include:
- Off-heap memory management with Arrow buffers
- Efficient binary serialization and data transfer
- Optional compression
- Batched operations
This approach is particularly well-suited for:
- Large-scale batch processing and data migrations
- High-throughput log and sensor data ingestion
- Time-series applications with demanding performance requirements
- Systems processing high-frequency data collection
Here's a typical pattern for using the Bulk Write API:
```java
// Create a BulkStreamWriter with the table schema
try (BulkStreamWriter writer = greptimeDB.bulkStreamWriter(schema)) {
// Write multiple batches
for (int batch = 0; batch < batchCount; batch++) {
// Get a TableBufferRoot for this batch
Table.TableBufferRoot table = writer.tableBufferRoot(1000); // column buffer size
// Add rows to the batch
for (int row = 0; row < rowsPerBatch; row++) {
Object[] rowData = generateRow(batch, row);
table.addRow(rowData);
}
// Complete the table to prepare for transmission
table.complete();
// Send the batch and get a future for completion
CompletableFuture future = writer.writeNext();
// Wait for the batch to be processed (optional)
Integer affectedRows = future.get();
System.out.println("Batch " + batch + " wrote " + affectedRows + " rows");
}
// Signal completion of the stream
writer.completed();
}
```
#### Configuration
The Bulk Write API can be configured with several options to optimize performance:
```java
BulkWrite.Config cfg = BulkWrite.Config.newBuilder()
.allocatorInitReservation(64 * 1024 * 1024L) // Customize memory allocation: 64MB initial reservation
.allocatorMaxAllocation(4 * 1024 * 1024 * 1024L) // Customize memory allocation: 4GB max allocation
.timeoutMsPerMessage(60 * 1000) // 60 seconds timeout per request
.maxRequestsInFlight(8) // Concurrency Control: Configure with 8 maximum in-flight requests
.build();
// Enable Zstd compression
Context ctx = Context.newDefault().withCompression(Compression.Zstd);
BulkStreamWriter writer = greptimeDB.bulkStreamWriter(schema, cfg, ctx);
```
### Resource Management
It's important to properly shut down the client when you're finished using it:
```java
// Gracefully shut down the client
client.shutdownGracefully();
```
### Performance Tuning
#### Compression Options
The ingester supports various compression algorithms to reduce network bandwidth and improve throughput.
```java
// Set the compression algorithm to Zstd
Context ctx = Context.newDefault().withCompression(Compression.Zstd);
```
#### Write Operation Comparison
Understanding the performance characteristics of different write methods is crucial for optimizing data ingestion.
| Write Method | API | Throughput | Latency | Memory Efficiency | CPU Usage | Best For | Limitations |
|--------------|-----|------------|---------|-------------------|-----------|----------|-------------|
| Batching Write | `write(tables)` | Better | Good | High | Higher | Simple applications, low latency requirements | Lower throughput for large volumes |
| Streaming Write | `streamWriter()` | Moderate | Good | Moderate | Moderate | Continuous data streams, moderate throughput | More complex to use than regular writes |
| Bulk Write | `bulkStreamWriter()` | Best | Higher | Best | Moderate | Maximum throughput, large batch operations | Higher latency, requires manual table creation |
#### Buffer Size Optimization
When using `BulkStreamWriter`, you can configure the column buffer size:
```java
// Get the table buffer with a specific column buffer size
Table.TableBufferRoot table = bulkStreamWriter.tableBufferRoot(columnBufferSize);
```
This option can significantly improve the speed of data conversion to the underlying format. For optimal performance, we recommend setting the column buffer size to 1024 or larger, depending on your specific workload characteristics and available memory.
### Export Metrics
The ingester exposes comprehensive metrics that enable you to monitor its performance, health, and operational status.
For detailed information about available metrics and their usage, refer to the [Ingester Prometheus Metrics](https://github.com/GreptimeTeam/greptimedb-ingester-java/tree/main/ingester-prometheus-metrics) documentation.
## Key Configuration Options
`GreptimeOptions` is the main configuration class for the GreptimeDB Java client, used to configure client connections, write options, RPC settings, and various other parameters.
For production environments, you may need to configure these commonly used options. Complete reference: [GreptimeOptions JavaDoc](https://javadoc.io/static/io.greptime/ingester-protocol/0.15.0/io/greptime/options/GreptimeOptions.html).
**Key Options:**
- `database`: Target database name, format `[catalog-]schema` (default: `public`)
- `authInfo`: Authentication credentials for production environments
- `rpcOptions.defaultRpcTimeout`: RPC request timeout (default: 60 seconds)
- `writeMaxRetries`: Maximum retries for failed writes (default: 1)
- `maxInFlightWritePoints`: Maximum data points in-flight for write flow control (default: 655360)
- `writeLimitedPolicy`: Policy when write flow limit exceeded (default: AbortOnBlockingTimeoutPolicy 3s)
- `defaultStreamMaxWritePointsPerSecond`: Rate limit for StreamWriter (default: 655360)
```java
// Production-ready configuration
RpcOptions rpcOpts = RpcOptions.newDefault();
rpcOpts.setDefaultRpcTimeout(30000); // 30 seconds timeout
AuthInfo authInfo = new AuthInfo("username", "password");
GreptimeOptions options = GreptimeOptions.newBuilder("127.0.0.1:4001", "production_db")
.authInfo(authInfo)
.rpcOptions(rpcOpts)
.writeMaxRetries(3)
.maxInFlightWritePoints(1000000)
.writeLimitedPolicy(new LimitedPolicy.AbortOnBlockingTimeoutPolicy(5, TimeUnit.SECONDS))
.defaultStreamMaxWritePointsPerSecond(50000)
.build();
```
## FAQ
### Why am I getting some connection exceptions?
When using the GreptimeDB Java ingester SDK, you may encounter some connection exceptions.
For example, exceptions that are "`Caused by: java.nio.channels.UnsupportedAddressTypeException`",
"`Caused by: java.net.ConnectException: connect(..) failed: Address family not supported by protocol`", or
"`Caused by: java.net.ConnectException: connect(..) failed: Invalid argument`". While you are certain that the
GreptimeDB server is running, and its endpoint is reachable.
These connection exceptions could be all because the gRPC's `io.grpc.NameResolverProvider` service provider is not
packaged into the final JAR, during the assembling process. So the fix can be:
- If you are using Maven Assembly plugin, add the `metaInf-services` container descriptor handler to your assembly
file, like this:
```xml
...
metaInf-services
```
- And if you are using Maven Shade plugin, you can add the `ServicesResourceTransformer` instead:
```xml
...
org.apache.maven.plugins
maven-shade-plugin
3.6.0
shade
...
```
## API Documentation and Examples
- [API Reference](https://javadoc.io/doc/io.greptime/ingester-protocol/latest/index.html)
- [Examples](https://github.com/GreptimeTeam/greptimedb-ingester-java/tree/main/ingester-example/)
---
## gRPC SDKs
This guide will demonstrate how to use client libraries to write data in GreptimeDB.
- [Go](go.md)
- [Java](java.md)
- [Rust](https://github.com/GreptimeTeam/greptimedb-ingester-rust)
- [.NET](https://github.com/GreptimeTeam/greptimedb-ingester-dotnet)
- [Erlang](https://github.com/GreptimeTeam/greptimedb-ingester-erl)
- [TypeScript](https://github.com/GreptimeTeam/greptimedb-ingester-ts)
---
## InfluxDB Line Protocol
GreptimeDB supports HTTP InfluxDB Line protocol.
## Ingest data
### Protocols
#### Post metrics
You can write data to GreptimeDB using the InfluxDB-compatible write APIs.
Here's an example of how to use these APIs:
```shell
curl -i -XPOST "http://localhost:4000/v1/influxdb/api/v2/write?bucket=public&precision=ms" \
-H "authorization: token " \
--data-binary \
'monitor,host=127.0.0.1 cpu=0.1,memory=0.4 1667446797450
monitor,host=127.0.0.2 cpu=0.2,memory=0.3 1667446798450
monitor,host=127.0.0.1 cpu=0.5,memory=0.2 1667446798450'
```
```shell
curl -i -XPOST "http://localhost:4000/v1/influxdb/write?db=public&precision=ms&u=&p=" \
--data-binary \
'monitor,host=127.0.0.1 cpu=0.1,memory=0.4 1667446797450
monitor,host=127.0.0.2 cpu=0.2,memory=0.3 1667446798450
monitor,host=127.0.0.1 cpu=0.5,memory=0.2 1667446798450'
```
The InfluxDB-compatible write APIs support the following query parameters:
* `bucket`: Specifies the database to write to when using the InfluxDB line protocol V2 API. The default value is `public`.
* `db`: Specifies the database to write to. This is the standard parameter for the InfluxDB line protocol V1 API. The V2 API also accepts `db` as a GreptimeDB compatibility alias, but `bucket` is recommended for V2.
* `precision`: Defines the precision of the timestamp provided in the request body. Accepted values are `ns` (nanoseconds), `us` (microseconds), `ms` (milliseconds), and `s` (seconds). The data type of timestamps written by this API is `TimestampNanosecond`, so the default precision is `ns` (nanoseconds). If you use timestamps with other precisions in the request body, you need to specify the precision using this parameter. This parameter ensures that timestamp values are accurately interpreted and stored with nanosecond precision.
You can pass HTTP hints to set options on auto-created tables.
For example, set the `append_mode` hint to `true` to create an append-only table:
```shell
curl -i -XPOST "http://localhost:4000/v1/influxdb/write?db=public&precision=ms&u=&p=" \
-H 'x-greptime-hint-append_mode: true' \
--data-binary \
'monitor,host=127.0.0.1 cpu=0.1,memory=0.4 1667446797450'
```
When `append_mode=true` is explicitly provided, GreptimeDB creates the table with `append_mode = 'true'` and `merge_mode = 'last_row'`.
Otherwise, auto-created InfluxDB line protocol tables use the merge mode from the `merge_mode` HTTP hint. If no hint is provided, GreptimeDB uses the configured [`influxdb.default_merge_mode`](/user-guide/deployments-administration/configuration.md) value, which defaults to `last_non_null`.
To change the cluster-wide default for InfluxDB line protocol auto-created tables, set this option in the standalone or frontend configuration file:
```toml
[influxdb]
default_merge_mode = "last_row"
```
Available values are `last_non_null` and `last_row`.
You can also omit the timestamp when sending requests. GreptimeDB will use the current system time (in UTC) of the host machine as the timestamp. For example:
```shell
curl -i -XPOST "http://localhost:4000/v1/influxdb/api/v2/write?bucket=public" \
-H "authorization: token " \
--data-binary \
'monitor,host=127.0.0.1 cpu=0.1,memory=0.4
monitor,host=127.0.0.2 cpu=0.2,memory=0.3
monitor,host=127.0.0.1 cpu=0.5,memory=0.2'
```
```shell
curl -i -XPOST "http://localhost:4000/v1/influxdb/write?db=public&u=&p=" \
--data-binary \
'monitor,host=127.0.0.1 cpu=0.1,memory=0.4
monitor,host=127.0.0.2 cpu=0.2,memory=0.3
monitor,host=127.0.0.1 cpu=0.5,memory=0.2'
```
#### Authentication
GreptimeDB is compatible with InfluxDB's line protocol authentication format, both V1 and V2.
If you have [configured authentication](/user-guide/deployments-administration/authentication/overview.md) in GreptimeDB, you need to provide the username and password in the HTTP request.
For the InfluxDB line protocol V2 API, pass the username and password in the `Authorization` header using the `token :` format.
```shell
curl 'http://localhost:4000/v1/influxdb/api/v2/write?bucket=public' \
-H 'authorization: token ' \
-d 'monitor,host=127.0.0.1 cpu=0.1,memory=0.4'
```
For the InfluxDB line protocol V1 API, add `u` for user and `p` for password to the HTTP query string:
```shell
curl 'http://localhost:4000/v1/influxdb/write?db=public&u=&p=' \
-d 'monitor,host=127.0.0.1 cpu=0.1,memory=0.4'
```
### Telegraf
GreptimeDB's support for the [InfluxDB line protocol](../for-iot/influxdb-line-protocol.md) ensures its compatibility with Telegraf.
To configure Telegraf, simply add GreptimeDB URL into Telegraf configurations:
```toml
[[outputs.influxdb_v2]]
urls = ["http://:4000/v1/influxdb"]
token = ":"
bucket = ""
## Leave empty
organization = ""
```
```toml
[[outputs.influxdb]]
urls = ["http://:4000/v1/influxdb"]
database = ""
username = ""
password = ""
```
## Data model
While you may already be familiar with [InfluxDB key concepts](https://docs.influxdata.com/influxdb/v2/reference/key-concepts/), the [data model](/user-guide/concepts/data-model.md) of GreptimeDB is something new to explore.
Here are the similarities and differences between the data models of GreptimeDB and InfluxDB:
- Both solutions are [schemaless](/user-guide/ingest-data/overview.md#automatic-schema-generation), eliminating the need to define a schema before writing data.
- The GreptimeDB table is automatically created with the [`merge_mode` option](/reference/sql/create.md#create-a-table-with-merge-mode) set from the `merge_mode` HTTP hint or from [`influxdb.default_merge_mode`](/user-guide/deployments-administration/configuration.md) when the hint is absent. The configuration defaults to `last_non_null`.
With `last_non_null`, the table merges rows with the same tags and timestamp by keeping the latest value of each field, which is the same behavior as InfluxDB.
If the request explicitly sets the HTTP `append_mode` hint to `true`, the table is created with `append_mode = 'true'` and `merge_mode = 'last_row'`.
- In InfluxDB, a point represents a single data record with a measurement, tag set, field set, and a timestamp.
In GreptimeDB, it is represented as a row of data in the time-series table,
where the table name aligns with the measurement,
and the columns are divided into three types: Tag, Field, and Timestamp.
- GreptimeDB uses `TimestampNanosecond` as the data type for timestamp data from the [InfluxDB line protocol API](/user-guide/ingest-data/for-iot/influxdb-line-protocol.md).
- GreptimeDB uses `Float64` as the data type for numeric data from the InfluxDB line protocol API.
Consider the following [sample data](https://docs.influxdata.com/influxdb/v2/reference/key-concepts/data-elements/#sample-data) borrowed from InfluxDB docs as an example:
|_time|_measurement|location|scientist|_field|_value|
|---|---|---|---|---|---|
|2019-08-18T00:00:00Z|census|klamath|anderson|bees|23|
|2019-08-18T00:00:00Z|census|portland|mullen|ants|30|
|2019-08-18T00:06:00Z|census|klamath|anderson|bees|28|
|2019-08-18T00:06:00Z|census|portland|mullen|ants|32|
The data mentioned above is formatted as follows in the InfluxDB line protocol:
```shell
census,location=klamath,scientist=anderson bees=23 1566086400000000000
census,location=portland,scientist=mullen ants=30 1566086400000000000
census,location=klamath,scientist=anderson bees=28 1566086760000000000
census,location=portland,scientist=mullen ants=32 1566086760000000000
```
In the GreptimeDB data model, the data is represented as follows in the `census` table:
```sql
+---------------------+----------+-----------+------+------+
| greptime_timestamp | location | scientist | bees | ants |
+---------------------+----------+-----------+------+------+
| 2019-08-18 00:00:00 | klamath | anderson | 23 | NULL |
| 2019-08-18 00:06:00 | klamath | anderson | 28 | NULL |
| 2019-08-18 00:00:00 | portland | mullen | NULL | 30 |
| 2019-08-18 00:06:00 | portland | mullen | NULL | 32 |
+---------------------+----------+-----------+------+------+
```
The schema of the `census` table is as follows:
```sql
+--------------------+----------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+----------------------+------+------+---------+---------------+
| location | String | PRI | YES | | TAG |
| scientist | String | PRI | YES | | TAG |
| bees | Float64 | | YES | | FIELD |
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| ants | Float64 | | YES | | FIELD |
+--------------------+----------------------+------+------+---------+---------------+
```
## Reference
- [InfluxDB Line protocol](https://docs.influxdata.com/influxdb/v2.7/reference/syntax/line-protocol/)
---
## Kafka(For-iot)
Please refer to the [Kafka documentation](/user-guide/ingest-data/for-observability/kafka.md) for instructions on how to ingest data from Kafka into GreptimeDB.
---
## OpenTSDB
GreptimeDB supports ingesting OpenTSDB via HTTP API.
GreptimeDB also supports inserting OpenTSDB metrics via HTTP endpoints. We use the request and
response format described in OpenTSDB's `/api/put`.
The HTTP endpoint in GreptimeDB for handling metrics is `/v1/opentsdb/api/put`.
Starting GreptimeDB, the HTTP server is listening on port `4000` by default.
Use curl to insert one metric point:
```shell
curl -X POST http://127.0.0.1:4000/v1/opentsdb/api/put \
-H 'Authorization: Basic ' \
-d '
{
"metric": "sys.cpu.nice",
"timestamp": 1667898896,
"value": 18,
"tags": {
"host": "web01",
"dc": "hz"
}
}
'
```
Or insert multiple metric points:
```shell
curl -X POST http://127.0.0.1:4000/v1/opentsdb/api/put \
-H 'Authorization: Basic ' \
-d '
[
{
"metric": "sys.cpu.nice",
"timestamp": 1667898896,
"value": 1,
"tags": {
"host": "web02",
"dc": "hz"
}
},
{
"metric": "sys.cpu.nice",
"timestamp": 1667898897,
"value": 9,
"tags": {
"host": "web03",
"dc": "sh"
}
}
]
'
```
---
## Ingest Data for IoT
IoT data reaches GreptimeDB from devices, brokers, and applications. [Ingest Data](../overview.md) lists every source in one table; this page covers the IoT paths.
- [gRPC SDKs](./grpc-sdks/overview.md) — clients for Go, Java, Rust, .NET, Erlang, and TypeScript.
- [EMQX](emqx.md) — MQTT devices, written through an EMQX data integration.
- [InfluxDB Line Protocol](influxdb-line-protocol.md) — the path for migrating off InfluxDB, and for Telegraf. This page also covers the Telegraf configuration.
- [OpenTSDB](opentsdb.md) — the `/v1/opentsdb/api/put` HTTP endpoint.
- [SQL](sql.md) — `INSERT` statements over the MySQL or PostgreSQL protocol.
Tables and columns are created as data arrives, except on the SQL path. See [Automatic Schema Generation](../overview.md#automatic-schema-generation).
---
## SQL
You can execute SQL statements using [MySQL](/user-guide/protocols/mysql.md) or [PostgreSQL](/user-guide/protocols/postgresql.md) clients,
and access GreptimeDB using the MySQL or PostgreSQL protocol with any programming language of your preference, such as Java JDBC.
We will use the `monitor` table as an example to show how to write data.
## Create a table
Before inserting data, you need to create a table. For example, create a table named `monitor`:
```sql
CREATE TABLE monitor (
host STRING,
ts TIMESTAMP DEFAULT CURRENT_TIMESTAMP() TIME INDEX,
cpu FLOAT64 DEFAULT 0,
memory FLOAT64,
PRIMARY KEY(host));
```
The above statement will create a table with the following schema:
```sql
+--------+----------------------+------+------+---------------------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------+----------------------+------+------+---------------------+---------------+
| host | String | PRI | YES | | TAG |
| ts | TimestampMillisecond | PRI | NO | current_timestamp() | TIMESTAMP |
| cpu | Float64 | | YES | 0 | FIELD |
| memory | Float64 | | YES | | FIELD |
+--------+----------------------+------+------+---------------------+---------------+
4 rows in set (0.01 sec)
```
For more information about the `CREATE TABLE` statement,
please refer to [table management](/user-guide/deployments-administration/manage-data/basic-table-operations.md#create-a-table).
## Insert data
Let's insert some testing data to the `monitor` table. You can use the `INSERT INTO` SQL statements:
```sql
INSERT INTO monitor
VALUES
("127.0.0.1", 1702433141000, 0.5, 0.2),
("127.0.0.2", 1702433141000, 0.3, 0.1),
("127.0.0.1", 1702433146000, 0.3, 0.2),
("127.0.0.2", 1702433146000, 0.2, 0.4),
("127.0.0.1", 1702433151000, 0.4, 0.3),
("127.0.0.2", 1702433151000, 0.2, 0.4);
```
```sql
Query OK, 6 rows affected (0.01 sec)
```
You can also insert data by specifying the column names:
```sql
INSERT INTO monitor (host, ts, cpu, memory)
VALUES
("127.0.0.1", 1702433141000, 0.5, 0.2),
("127.0.0.2", 1702433141000, 0.3, 0.1),
("127.0.0.1", 1702433146000, 0.3, 0.2),
("127.0.0.2", 1702433146000, 0.2, 0.4),
("127.0.0.1", 1702433151000, 0.4, 0.3),
("127.0.0.2", 1702433151000, 0.2, 0.4);
```
Through the above statement, we have inserted six rows into the `monitor` table.
For more information about the `INSERT` statement, please refer to [`INSERT`](/reference/sql/insert.md).
## Time zone
The time zone specified in the SQL client will affect the timestamp with a string format that does not have time zone information.
The timestamp value will automatically have the client's time zone information added.
For example, the following SQL set the time zone to `+8:00`:
```sql
SET time_zone = '+8:00';
```
Then insert values into the `monitor` table:
```sql
INSERT INTO monitor (host, ts, cpu, memory)
VALUES
("127.0.0.1", "2024-01-01 00:00:00", 0.4, 0.1),
("127.0.0.2", "2024-01-01 00:00:00+08:00", 0.5, 0.1);
```
The first timestamp value `2024-01-01 00:00:00` does not have time zone information, so it will automatically have the client's time zone information added.
After inserting, it will be equivalent to the second value `2024-01-01 00:00:00+08:00`.
The result in the `+8:00` time zone client is as follows:
```sql
+-----------+---------------------+------+--------+
| host | ts | cpu | memory |
+-----------+---------------------+------+--------+
| 127.0.0.1 | 2024-01-01 00:00:00 | 0.4 | 0.1 |
| 127.0.0.2 | 2024-01-01 00:00:00 | 0.5 | 0.1 |
+-----------+---------------------+------+--------+
```
The result in the `UTC` time zone client is as follows:
```sql
+-----------+---------------------+------+--------+
| host | ts | cpu | memory |
+-----------+---------------------+------+--------+
| 127.0.0.1 | 2023-12-31 16:00:00 | 0.4 | 0.1 |
| 127.0.0.2 | 2023-12-31 16:00:00 | 0.5 | 0.1 |
+-----------+---------------------+------+--------+
```
---
## Grafana Alloy
[Grafana Alloy](https://grafana.com/docs/alloy/latest/) is an observability data pipeline for OpenTelemetry (OTel), Prometheus, Pyroscope, Loki, and many other metrics, logs, traces, and profiling tools.
You can integrate GreptimeDB as a data sink for Alloy.
## Prometheus Remote Write
Configure GreptimeDB as remote write target:
```hcl
prometheus.remote_write "greptimedb" {
endpoint {
url = coalesce(sys.env("GREPTIME_SCHEME"), "http") + "://" +
coalesce(sys.env("GREPTIME_HOST"), "greptimedb") + ":" +
coalesce(sys.env("GREPTIME_PORT"), "4000") +
"/v1/prometheus/write?db=" + coalesce(sys.env("GREPTIME_DB"), "public")
basic_auth {
username = sys.env("GREPTIME_USERNAME")
password = sys.env("GREPTIME_PASSWORD")
}
}
}
```
- `GREPTIME_HOST`: GreptimeDB host address, e.g., `localhost`.
- `GREPTIME_DB`: GreptimeDB database name, default is `public`.
- `GREPTIME_USERNAME` and `GREPTIME_PASSWORD`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
For details on the data model transformation from Prometheus to GreptimeDB, refer to the [Data Model](/user-guide/ingest-data/for-observability/prometheus.md#data-model) section in the Prometheus Remote Write guide.
## OpenTelemetry
GreptimeDB can also be configured as OpenTelemetry collector.
### Metrics
```hcl
otelcol.exporter.otlphttp "greptimedb" {
client {
endpoint = coalesce(sys.env("GREPTIME_SCHEME"), "http") + "://" +
coalesce(sys.env("GREPTIME_HOST"), "greptimedb") + ":" +
coalesce(sys.env("GREPTIME_PORT"), "4000") + "/v1/otlp/"
headers = {
"X-Greptime-DB-Name" = coalesce(sys.env("GREPTIME_DB"), "public"),
}
auth = otelcol.auth.basic.credentials.handler
}
}
otelcol.auth.basic "credentials" {
username = sys.env("GREPTIME_USERNAME")
password = sys.env("GREPTIME_PASSWORD")
}
```
- `GREPTIME_HOST`: GreptimeDB host address, e.g., `localhost`.
- `GREPTIME_DB`: GreptimeDB database name, default is `public`.
- `GREPTIME_USERNAME` and `GREPTIME_PASSWORD`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
For details on the metrics data model transformation from OpenTelemetry to GreptimeDB, refer to the [Data Model](/user-guide/ingest-data/for-observability/opentelemetry.md#data-model) section in the OpenTelemetry guide.
### Logs
This example sends logs to GreptimeDB through an OpenTelemetry pipeline.
For production log pipelines, add an explicit batch processor before the exporter. See the [Batching](#batching) section for details.
```hcl
loki.source.file "greptime" {
targets = [
{__path__ = "/tmp/foo.txt"},
]
forward_to = [otelcol.receiver.loki.greptime.receiver]
}
otelcol.receiver.loki "greptime" {
output {
logs = [otelcol.processor.batch.greptimedb_logs.input]
}
}
otelcol.processor.batch "greptimedb_logs" {
send_batch_size = 5000
send_batch_max_size = 10000
timeout = "1s"
output {
logs = [otelcol.exporter.otlphttp.greptimedb_logs.input]
}
}
otelcol.auth.basic "credentials" {
username = sys.env("GREPTIME_USERNAME")
password = sys.env("GREPTIME_PASSWORD")
}
otelcol.exporter.otlphttp "greptimedb_logs" {
client {
endpoint = coalesce(sys.env("GREPTIME_SCHEME"), "http") + "://" +
coalesce(sys.env("GREPTIME_HOST"), "greptimedb") + ":" +
coalesce(sys.env("GREPTIME_PORT"), "4000") + "/v1/otlp/"
headers = {
"X-Greptime-DB-Name" = coalesce(sys.env("GREPTIME_DB"), "public"),
"X-Greptime-Log-Table-Name" = coalesce(sys.env("GREPTIME_LOG_TABLE_NAME"), "demo_logs"),
"X-Greptime-Log-Extract-Keys" = "filename,log.file.name,loki.attribute.labels",
}
auth = otelcol.auth.basic.credentials.handler
}
sending_queue {
queue_size = 10000
num_consumers = 10
}
}
```
- `GREPTIME_HOST`: GreptimeDB host address, such as `localhost`.
- `GREPTIME_DB`: GreptimeDB database name, default is `public`.
- `GREPTIME_LOG_TABLE_NAME`: Target log table name, default is `demo_logs`.
- `GREPTIME_USERNAME` and `GREPTIME_PASSWORD`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
- `X-Greptime-Log-Extract-Keys`: Keys extracted from OTLP log attributes. See the [OTLP/HTTP API documentation](/user-guide/ingest-data/for-observability/opentelemetry.md#otlphttp-api-1) for details.
For details on the log data model transformation from OpenTelemetry to GreptimeDB, refer to the [Data Model](/user-guide/ingest-data/for-observability/opentelemetry.md#data-model-1) section in the OpenTelemetry guide.
## Loki
GreptimeDB also supports the Loki push protocol for logs.
If your Alloy pipeline is already built with Loki components, we recommend using the native Loki ingestion path first.
See the [Loki guide](/user-guide/ingest-data/for-observability/loki.md) for the Loki protocol details and data model mapping.
### Logs
This example uses only Loki components to read, process, and send logs to GreptimeDB through the Loki push API:
```hcl
loki.source.file "greptime" {
targets = [
{__path__ = "/tmp/foo.txt"},
]
forward_to = [loki.process.greptime.receiver]
}
loki.process "greptime" {
forward_to = [loki.write.greptimedb.receiver]
stage.static_labels {
values = {
job = "greptime",
from = "alloy",
}
}
}
loki.write "greptimedb" {
endpoint {
url = coalesce(sys.env("GREPTIME_SCHEME"), "http") + "://" +
coalesce(sys.env("GREPTIME_HOST"), "greptimedb") + ":" +
coalesce(sys.env("GREPTIME_PORT"), "4000") + "/v1/loki/api/v1/push"
headers = {
"X-Greptime-DB-Name" = coalesce(sys.env("GREPTIME_DB"), "public"),
"X-Greptime-Log-Table-Name" = coalesce(sys.env("GREPTIME_LOG_TABLE_NAME"), "loki_demo_logs"),
}
basic_auth {
username = sys.env("GREPTIME_USERNAME")
password = sys.env("GREPTIME_PASSWORD")
}
}
}
```
- `GREPTIME_HOST`: GreptimeDB host address, such as `localhost`.
- `GREPTIME_DB`: GreptimeDB database name, default is `public`.
- `GREPTIME_LOG_TABLE_NAME`: Target log table name, default is `loki_demo_logs`.
- `GREPTIME_USERNAME` and `GREPTIME_PASSWORD`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
This configuration tails `/tmp/foo.txt`, adds two static labels, and sends the logs directly to GreptimeDB with `loki.write`.
## Batching
`otelcol.exporter.otlphttp` does not enable batching by default.
When Alloy reads a burst of logs, such as a large backlog from files or Docker containers, the exporter queue can fill before enough records are grouped together, which leads to errors like `sending queue is full`.
Based on the behavior reported in production testing, enabling the exporter's internal `sending_queue.batch` block alone may still be insufficient for bursty log workloads.
Putting `otelcol.processor.batch` in front of the exporter is a more reliable pattern for logs because the exporter receives larger bundled batches instead of many individual log records.
If you use OTLP/HTTP for logs, we recommend this order:
1. `loki.source.*`
2. `otelcol.receiver.loki`
3. `otelcol.processor.batch`
4. `otelcol.exporter.otlphttp`
If your pipeline is already Loki-native and you do not need OpenTelemetry processing, prefer `loki.write` and the Loki push protocol instead.
:::tip NOTE
Refer to the official Grafana Alloy documentation for the latest component behavior and tuning guidance, especially for `loki.write`, `otelcol.processor.batch`, and `otelcol.exporter.otlphttp`.
:::
---
## Elasticsearch
## Overview
GreptimeDB supports data ingestion through Elasticsearch's [`_bulk` API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). We map Elasticsearch's Index concept to GreptimeDB's Table, and users can specify the database name using the `db` URL parameter. **Unlike native Elasticsearch, this API only supports data insertion, not modification or deletion**. At the implementation level, both `index` and `create` commands in native Elasticsearch `_bulk` API requests are treated as **creation operations** by GreptimeDB. Additionally, GreptimeDB only parses the `_index` field from native `_bulk` API command requests while ignoring other fields.
## HTTP API
In most log collectors(such as Logstash and Filebeat mentioned below), you only need to configure the HTTP endpoint like `http://${db_host}:${db_http_port}/v1/elasticsearch`, for example `http://localhost:4000/v1/elasticsearch`.
GreptimeDB supports data ingestion through Elasticsearch protocol by implementing the following two HTTP endpoints:
- **`/v1/elasticsearch/_bulk`**: Users can use the POST method to write data in NDJSON format to GreptimeDB.
For example, the following request will create a table named `test` and insert two records:
```json
POST /v1/elasticsearch/_bulk
{"create": {"_index": "test", "_id": "1"}}
{"name": "John", "age": 30}
{"create": {"_index": "test", "_id": "2"}}
{"name": "Jane", "age": 25}
```
- **`/v1/elasticsearch/${index}/_bulk`**: Users can use the POST method to write data in NDJSON format to the `${index}` table in GreptimeDB. If the POST request also contains an `_index` field, the `${index}` in the URL will be ignored.
For example, the following request will create tables named `test` and `another_index`, and insert corresponding data:
```json
POST /v1/elasticsearch/test/_bulk
{"create": {"_id": "1"}}
{"name": "John", "age": 30}
{"create": {"_index": "another_index", "_id": "2"}}
{"name": "Jane", "age": 25}
```
### HTTP Headers
- `x-greptime-db-name`: Specifies the database name. Defaults to `public` if not specified.
- `x-greptime-pipeline-name`: Specifies the pipeline name. Defaults to GreptimeDB's internal pipeline `greptime_identity` if not specified.
For more details about Pipeline, please refer to the [Manage Pipelines](/user-guide/logs/manage-pipelines.md) documentation.
### URL Parameters
You can use the following HTTP URL parameters:
- `db`: Specifies the database name. Defaults to `public` if not specified.
- `pipeline_name`: Specifies the pipeline name. Defaults to GreptimeDB's internal pipeline `greptime_identity` if not specified.
- `msg_field`: Specifies the JSON field name containing the original log data. For example, in Logstash and Filebeat, this field is typically `message`. If specified, GreptimeDB will attempt to parse the data in this field as JSON format. If parsing fails, the field will be treated as a string. This configuration option currently only takes effect in URL parameters.
### Authentication Header
For detailed information about the authentication header, please refer to the [Authorization](/user-guide/protocols/http.md#authentication) documentation.
## Usage
### Use HTTP API to ingest data
You can create a `request.json` file with the following content:
```json
{"create": {"_index": "es_test", "_id": "1"}}
{"name": "John", "age": 30}
{"create": {"_index": "es_test", "_id": "2"}}
{"name": "Jane", "age": 25}
```
Then use the `curl` command to send this file as a request body to GreptimeDB:
```bash
curl -XPOST http://localhost:4000/v1/elasticsearch/_bulk \
-H "Authorization: Basic " \
-H "Content-Type: application/x-ndjson" \
--data-binary @request.json
```
We can use a `mysql` client to connect to GreptimeDB and execute the following SQL to view the inserted data:
```sql
SELECT * FROM es_test;
```
We will see the following results:
```
mysql> SELECT * FROM es_test;
+------+------+----------------------------+
| age | name | greptime_timestamp |
+------+------+----------------------------+
| 30 | John | 2025-01-15 08:26:06.516665 |
| 25 | Jane | 2025-01-15 08:26:06.521510 |
+------+------+----------------------------+
2 rows in set (0.13 sec)
```
### Logstash
If you are using [Logstash](https://www.elastic.co/logstash) to collect logs, you can use the following configuration to write data to GreptimeDB:
```
output {
elasticsearch {
hosts => ["http://localhost:4000/v1/elasticsearch"]
index => "my_index"
manage_template => false
ilm_enabled => false
data_stream => false
parameters => {
"msg_field" => "message"
}
}
}
```
Please pay attention to the following GreptimeDB-related configurations:
- `hosts`: Specifies the HTTP address of GreptimeDB's Elasticsearch protocol, which is `http://${db_host}:${db_http_port}/v1/elasticsearch`.
- `index`: Specifies the table name that will be written to.
- `manage_template`, `ilm_enabled`, and `data_stream`: Disable Elasticsearch index management features. GreptimeDB's Elasticsearch-compatible API supports log ingestion through `_bulk`, but does not manage Elasticsearch templates, ILM policies, or data streams.
- `parameters`: Specifies the URL parameters for writing. The example above uses `msg_field=message`, which tells GreptimeDB to parse the original log content from the `message` field. You can also set `pipeline_name` here if you want to use an existing GreptimeDB pipeline. If `pipeline_name` is omitted, GreptimeDB uses the built-in `greptime_identity` pipeline.
- If Logstash prints warnings about X-Pack monitoring trying to connect to `elasticsearch:9200`, disable Logstash monitoring by setting `xpack.monitoring.enabled: false` in `logstash.yml`.
### Filebeat
If you are using [Filebeat](https://github.com/elastic/beats/tree/main/filebeat) to collect logs, you can use the following configuration to write data to GreptimeDB:
```
output.elasticsearch:
hosts: ["http://localhost:4000/v1/elasticsearch"]
index: "my_index"
parameters:
msg_field: message
setup.ilm.enabled: false
setup.template.enabled: false
```
Please pay attention to the following GreptimeDB-related configurations:
- `hosts`: Specifies the HTTP address of GreptimeDB's Elasticsearch protocol, which is `http://${db_host}:${db_http_port}/v1/elasticsearch`.
- `index`: Specifies the table name that will be written to.
- `parameters`: Specifies the URL parameters for writing. The example above uses `msg_field: message`, which tells GreptimeDB to parse the original log content from the `message` field. You can also set `pipeline_name` here if you want to use an existing GreptimeDB pipeline. If `pipeline_name` is omitted, GreptimeDB uses the built-in `greptime_identity` pipeline.
- `setup.ilm.enabled` and `setup.template.enabled`: Disable Elasticsearch index lifecycle management and template setup. GreptimeDB's Elasticsearch-compatible API supports log ingestion through `_bulk`, but does not manage Elasticsearch templates or ILM policies.
### Telegraf
If you are using [Telegraf](https://github.com/influxdata/telegraf) to collect logs, you can use its Elasticsearch plugin to write data to GreptimeDB, as shown below:
```toml
[[outputs.elasticsearch]]
urls = [ "http://localhost:4000/v1/elasticsearch" ]
index_name = "test_table"
health_check_interval = "0s"
enable_sniffer = false
flush_interval = "1s"
manage_template = false
template_name = "telegraf"
overwrite_template = false
namepass = ["tail"]
[outputs.elasticsearch.headers]
"X-GREPTIME-DB-NAME" = "public"
"X-GREPTIME-PIPELINE-NAME" = "greptime_identity"
[[inputs.tail]]
files = ["/tmp/test.log"]
from_beginning = true
data_format = "value"
data_type = "string"
character_encoding = "utf-8"
interval = "1s"
pipe = false
watch_method = "inotify"
```
Please pay attention to the following GreptimeDB-related configurations:
- `urls`: Specifies the HTTP address of GreptimeDB's Elasticsearch protocol, which is `http://${db_host}:${db_http_port}/v1/elasticsearch`.
- `index_name`: Specifies the table name that will be written to.
- `outputs.elasticsearch.header`: Specifies the HTTP Header for writing, where the example above specifies two parameters: `X-GREPTIME-DB-NAME` and `X-GREPTIME-PIPELINE-NAME`.
---
## Fluent Bit
[Fluent Bit](http://fluentbit.io/) is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.
You can forward Fluent Bit data to GreptimeDB. This document describes how to configure Fluent Bit to send logs, metrics, and traces to GreptimeDB.
## HTTP
Using Fluent Bit's [HTTP Output Plugin](https://docs.fluentbit.io/manual/pipeline/outputs/http), you can send logs to GreptimeDB.
Before configuring Fluent Bit, ensure that you understand the [log ingestion flow](/user-guide/logs/overview.md) and [how to use pipelines](/user-guide/logs/use-custom-pipelines.md).
```conf
[OUTPUT]
Name http
Match *
Host greptimedb
Port 4000
Uri /v1/ingest?db=public&table=your_table&pipeline_name=greptime_identity
Format json
Json_date_key scrape_timestamp
Json_date_format iso8601
compress gzip
http_User
http_Passwd
```
- `host`: GreptimeDB host address, e.g., `localhost`.
- `port`: GreptimeDB port, default is `4000`.
- `uri`: The endpoint to send logs to.
- `format`: The format of the logs, needs to be `json`.
- `json_date_key`: The key in the JSON object that contains the timestamp.
- `json_date_format`: The format of the timestamp.
- `compress`: The compression method to use, e.g., `gzip`.
- `header`: The header to send with the request, e.g., `Authorization` for authentication.
- `http_user` and `http_passwd`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
In params Uri,
- `db` is the database name you want to write logs to.
- `table` is the table name you want to write logs to.
- `pipeline_name` is the pipeline name you want to use for processing logs.
- `custom_time_index` is optional. Use it when a field in the input data should become the GreptimeDB time index. The supported formats are `;epoch;` and `;datestr;`.
The `greptime_identity` pipeline creates the table directly from the JSON fields. If you want a field such as `scrape_timestamp` to become the GreptimeDB time index instead of a regular column, set `custom_time_index` in the request URI or use a custom pipeline to parse and map that field.
## OpenTelemetry
GreptimeDB can also be configured as OpenTelemetry collector. Using Fluent Bit's [OpenTelemetry Output Plugin](https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry), you can send metrics, logs, and traces to GreptimeDB.
Use a separate output for each signal because GreptimeDB uses signal-specific headers. Assign a distinct tag to each signal in the Fluent Bit inputs, then match that tag in the corresponding output:
```
# Only for metrics
[OUTPUT]
Name opentelemetry
Alias opentelemetry_metrics
Match
Host 127.0.0.1
Port 4000
Metrics_uri /v1/otlp/v1/metrics
Log_response_payload True
Tls Off
Tls.verify Off
Header X-Greptime-DB-Name ""
# Only for logs
[OUTPUT]
Name opentelemetry
Alias opentelemetry_logs
Match
Host 127.0.0.1
Port 4000
Logs_uri /v1/otlp/v1/logs
Log_response_payload True
Tls Off
Tls.verify Off
Header X-Greptime-Log-Table-Name ""
Header X-Greptime-Pipeline-Name ""
Header X-Greptime-DB-Name ""
# Only for traces
[OUTPUT]
Name opentelemetry
Alias opentelemetry_traces
Match
Host 127.0.0.1
Port 4000
Traces_uri /v1/otlp/v1/traces
Log_response_payload True
Tls Off
Tls.verify Off
Header X-Greptime-Pipeline-Name "greptime_trace_v1"
Header X-Greptime-DB-Name ""
```
- `Metrics_uri`, `Logs_uri`, and `Traces_uri`: The endpoints for metrics, logs, and traces.
- `Match`: The signal-specific tag configured on the corresponding Fluent Bit input.
- `http_user` and `http_passwd`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB. Add them to each output when authentication is enabled.
In this example, the OpenTelemetry OTLP/HTTP API is used. For signal-specific headers and options, see the OpenTelemetry API sections for [metrics](/user-guide/ingest-data/for-observability/opentelemetry.md#otlphttp-api), [logs](/user-guide/ingest-data/for-observability/opentelemetry.md#otlphttp-api-1), and [traces](/user-guide/ingest-data/for-observability/opentelemetry.md#otlphttp-api-2).
## Prometheus Remote Write
Configure GreptimeDB as remote write target:
```
[OUTPUT]
Name prometheus_remote_write
Match internal_metrics
Host 127.0.0.1
Port 4000
Uri /v1/prometheus/write?db=
Tls Off
http_user
http_passwd
```
- `Uri`: The endpoint to send metrics to.
- `http_user` and `http_passwd`: The [authentication credentials](/user-guide/deployments-administration/authentication/static.md) for GreptimeDB.
In params Uri,
- `db` is the database name you want to write metrics to.
For details on the data model transformation from Prometheus to GreptimeDB, refer to the [Data Model](/user-guide/ingest-data/for-observability/prometheus.md#data-model) section in the Prometheus Remote Write guide.
---
## InfluxDB Line Protocol(For-observability)
Please refer to the [InfluxDB Line Protocol documentation](../for-iot/influxdb-line-protocol.md) for instructions on how to write data to GreptimeDB using the InfluxDB Line Protocol.
---
## Kafka(For-observability)
If you are using Kafka or Kafka-compatible message queue for observability data
transporting, it's possible to ingest data into GreptimeDB directly.
Here we are using Vector as the tool to transport data from Kafka to GreptimeDB.
## Metrics
When ingesting metrics from Kafka into GreptimeDB, messages should be formatted in InfluxDB line protocol. For example:
```txt
census,location=klamath,scientist=anderson bees=23 1566086400000000000
```
Then configure Vector to use the `influxdb` decoding codec to process these messages.
```toml
[sources.metrics_mq]
# Specifies that the source type is Kafka
type = "kafka"
# The consumer group ID for Kafka
group_id = "vector0"
# The list of Kafka topics to consume messages from
topics = ["test_metric_topic"]
# The address of the Kafka broker to connect to
bootstrap_servers = "kafka:9092"
# The `influxdb` means the messages are expected to be in InfluxDB line protocol format.
decoding.codec = "influxdb"
[sinks.metrics_in]
inputs = ["metrics_mq"]
# Specifies that the sink type is `greptimedb_metrics`
type = "greptimedb_metrics"
# The endpoint of the GreptimeDB server.
# Replace with the actual hostname or IP address.
endpoint = ":4001"
dbname = ""
username = ""
password = ""
tls = {}
```
For details on how InfluxDB line protocol metrics are mapped to GreptimeDB data, please refer to the [Data Model](/user-guide/ingest-data/for-iot/influxdb-line-protocol.md#data-model) section in the InfluxDB line protocol documentation.
When Vector decodes Kafka messages with `decoding.codec = "influxdb"` and writes them through the `greptimedb_metrics` sink, each InfluxDB field is converted into a metric name. For example, `census bees=23` is written to a table named `census_bees`, with the value stored in the `val` column.
## Logs
Developers commonly work with two types of logs: JSON logs and plain text logs.
Consider the following examples sent from Kafka.
A plain text log:
```txt
127.0.0.1 - - [25/May/2024:20:16:37 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
```
Or a JSON log:
```json
{
"timestamp": "2024-12-23T10:00:00Z",
"level": "INFO",
"message": "Service started"
}
```
GreptimeDB transforms these logs into structured data with multiple columns and automatically creates the necessary tables.
A pipeline processes the logs into structured data before ingestion into GreptimeDB. Different log formats require different [Pipelines](/user-guide/logs/use-custom-pipelines.md) for parsing. See the following sections for details.
### Logs with JSON format
For logs in JSON format (e.g., `{"timestamp": "2024-12-23T10:00:00Z", "level": "INFO", "message": "Service started"}`),
you can use the built-in [`greptime_identity`](/reference/pipeline/built-in-pipelines.md#greptime_identity) pipeline for direct ingestion.
This pipeline creates columns automatically based on the fields in your JSON log message.
Simply configure Vector's `transforms` settings to parse the JSON message and use the `greptime_identity` pipeline as shown in the following example:
```toml
[sources.logs_in]
type = "kafka"
# The consumer group ID for Kafka
group_id = "vector0"
# The list of Kafka topics to consume messages from
topics = ["test_log_topic"]
# The address of the Kafka broker to connect to
bootstrap_servers = "kafka:9092"
# transform the log to JSON format
[transforms.logs_json]
type = "remap"
inputs = ["logs_in"]
source = '''
. = parse_json!(.message)
'''
[sinks.logs_out]
# Specifies that this sink will receive data from the `logs_json` source
inputs = ["logs_json"]
# Specifies that the sink type is `greptimedb_logs`
type = "greptimedb_logs"
# The endpoint of the GreptimeDB server
endpoint = "http://:4000"
compression = "gzip"
# Replace , , and with the actual values
dbname = ""
username = ""
password = ""
# The table name in GreptimeDB, if it doesn't exist, it will be created automatically
table = "demo_logs"
# Use the built-in `greptime_identity` pipeline
pipeline_name = "greptime_identity"
```
### Logs with text format
For logs in text format, such as the access log format below, you'll need to create a custom pipeline to parse them:
```
127.0.0.1 - - [25/May/2024:20:16:37 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
```
#### Create a pipeline
To create a custom pipeline,
please refer to the [using custom pipelines](/user-guide/logs/use-custom-pipelines.md)
documentation for detailed instructions.
#### Ingest data
After creating the pipeline, configure it to the `pipeline_name` field in the Vector configuration file.
```toml
# sample.toml
[sources.log_mq]
# Specifies that the source type is Kafka
type = "kafka"
# The consumer group ID for Kafka
group_id = "vector0"
# The list of Kafka topics to consume messages from
topics = ["test_log_topic"]
# The address of the Kafka broker to connect to
bootstrap_servers = "kafka:9092"
[sinks.sink_greptime_logs]
# Specifies that the sink type is `greptimedb_logs`
type = "greptimedb_logs"
# Specifies that this sink will receive data from the `log_mq` source
inputs = [ "log_mq" ]
# Use `gzip` compression to save bandwidth
compression = "gzip"
# The endpoint of the GreptimeDB server
# Replace with the actual hostname or IP address
endpoint = "http://:4000"
dbname = ""
username = ""
password = ""
# The table name in GreptimeDB, if it doesn't exist, it will be created automatically
table = "demo_logs"
# The custom pipeline name that you created
pipeline_name = "your_custom_pipeline"
```
## Demo
For a runnable demo of data transformation and ingestion, please refer to the [Kafka Ingestion Demo](https://github.com/GreptimeTeam/demo-scene/tree/main/kafka-ingestion).
---
## Loki
## Usage
### API
To send logs to GreptimeDB through the raw HTTP API, use the following information:
* **URL**: `http{s}:///v1/loki/api/v1/push`
* **Headers**:
* `X-Greptime-DB-Name`: ``
* `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments-administration/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication).
* `X-Greptime-Log-Table-Name`: `` (optional) - The table name to store the logs. If not provided, the default table name is `loki_logs`.
GreptimeDB accepts either of the Loki Push API payload encodings:
* **Protobuf**: Send a Snappy-compressed `PushRequest` matching [logproto.proto](https://github.com/grafana/loki/blob/main/pkg/logproto/logproto.proto) with `Content-Type: application/x-protobuf`.
* **JSON**: Send a Loki JSON push body with `Content-Type: application/json`.
### Example Code
[Grafana Alloy](https://grafana.com/docs/alloy/latest/) is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Alloy uniquely combines the very best OSS observability signals in the community.
It supplies a Loki exporter that can be used to send logs to GreptimeDB. Here is an example configuration:
```
loki.source.file "greptime" {
targets = [
{__path__ = "/tmp/foo.txt"},
]
forward_to = [loki.write.greptime_loki.receiver]
}
loki.write "greptime_loki" {
endpoint {
url = coalesce(sys.env("GREPTIME_SCHEME"), "http") + "://" +
coalesce(sys.env("GREPTIME_HOST"), "greptimedb") + ":" +
coalesce(sys.env("GREPTIME_PORT"), "4000") + "/v1/loki/api/v1/push"
headers = {
"x-greptime-db-name" = coalesce(sys.env("GREPTIME_DB"), "public"),
"x-greptime-log-table-name" = coalesce(sys.env("GREPTIME_LOG_TABLE_NAME"), "loki_demo_logs"),
}
}
external_labels = {
"job" = "greptime",
"from" = "alloy",
}
}
```
This configuration reads logs from the file `/tmp/foo.txt` and sends them to GreptimeDB. The logs are stored in the table `loki_demo_logs` with the external labels `job` and `from`.
For more information, please refer to the [Grafana Alloy loki.write documentation](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.write/).
You can run the following command to check the data in the table:
```sql
SELECT * FROM loki_demo_logs;
+----------------------------+------------------------+---------------------+--------------+-------+----------+
| greptime_timestamp | line | structured_metadata | filename | from | job |
+----------------------------+------------------------+---------------------+--------------+-------+----------+
| 2024-11-25 11:02:31.256251 | Greptime is very cool! | {} | /tmp/foo.txt | alloy | greptime |
+----------------------------+------------------------+---------------------+--------------+-------+----------+
1 row in set (0.01 sec)
```
## Data Model
The Loki logs data model is mapped to the GreptimeDB data model according to the following rules:
**Default table schema without external labels:**
```sql
DESC loki_demo_logs;
+---------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+---------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
| structured_metadata | Json | | YES | | FIELD |
+---------------------+---------------------+------+------+---------+---------------+
3 rows in set (0.00 sec)
```
- `greptime_timestamp`: The timestamp of the log entry
- `line`: The log message content
- `structured_metadata`: The structured metadata of the log entry in JSON format
If the Loki Push request contains labels, they will be added as tags to the table schema (like `job` and `from` in the above example).
**Important notes:**
- All labels are treated as tags with string type
- Do not attempt to pre-create the table using SQL to specify tag columns, as this will cause a type mismatch and write failure
### Example Table Schema
The following is an example of the table schema with external labels:
```sql
DESC loki_demo_logs;
+---------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+---------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| line | String | | YES | | FIELD |
| structured_metadata | Json | | YES | | FIELD |
| filename | String | PRI | YES | | TAG |
| from | String | PRI | YES | | TAG |
| job | String | PRI | YES | | TAG |
+---------------------+---------------------+------+------+---------+---------------+
6 rows in set (0.00 sec)
```
```sql
SHOW CREATE TABLE loki_demo_logs\G
*************************** 1. row ***************************
Table: loki_demo_logs
Create Table: CREATE TABLE IF NOT EXISTS `loki_demo_logs` (
`greptime_timestamp` TIMESTAMP(9) NOT NULL,
`line` STRING NULL,
`structured_metadata` JSON NULL,
`filename` STRING NULL,
`from` STRING NULL,
`job` STRING NULL,
TIME INDEX (`greptime_timestamp`),
PRIMARY KEY (`filename`, `from`, `job`)
)
ENGINE=mito
WITH(
'comment' = 'Created on insertion',
append_mode = 'true',
'greptime.semantic.signal_type' = 'log',
'greptime.semantic.source' = 'loki'
)
1 row in set (0.00 sec)
```
## Using Pipeline with Loki Push API
GreptimeDB supports using pipelines to process Loki push requests.
You can simply set the HTTP header `x-greptime-pipeline-name` to the target pipeline name to enable pipeline processing.
**Note:** When request data goes through the pipeline engine, GreptimeDB adds prefixes to the label and metadata column names:
- `loki_label_` before each label name
- `loki_metadata_` before each structured metadata name
- The original Loki log line is named `loki_line`
### Pipeline Example
Here is a complete example demonstrating how to use pipelines with Loki push API.
**Step 1: Prepare the log file**
Suppose we have a log file named `logs.json` with JSON-formatted log entries:
```json
{"timestamp":"2025-08-21 14:23:17.892","logger":"sdk.tool.DatabaseUtil","level":"ERROR","message":"Connection timeout exceeded for database pool","trace_id":"a7f8c92d1e4b4c6f9d2e5a8b3f7c1d9e","source":"application"}
{"timestamp":"2025-08-21 14:23:18.156","logger":"core.scheduler.TaskManager","level":"WARN","message":"Task queue capacity reached 85% threshold","trace_id":"b3e9f4a6c8d2e5f7a1b4c7d9e2f5a8b3","source":"scheduler"}
{"timestamp":"2025-08-21 14:23:18.423","logger":"sdk.tool.NetworkUtil","level":"INFO","message":"Successfully established connection to remote endpoint","trace_id":"c5d8e7f2a9b4c6d8e1f4a7b9c2e5f8d1","source":"network"}
```
Each line is a separate JSON object containing log information.
**Step 2: Create the pipeline configuration**
Here is a pipeline configuration that parses the JSON log entries:
```yaml
# pipeline.yaml
version: 2
processors:
- vrl:
source: |
message = parse_json!(.loki_line)
target = {
"log_time": parse_timestamp!(message.timestamp, "%Y-%m-%d %T%.3f"),
"log_level": message.level,
"log_source": message.source,
"logger": message.logger,
"message": message.message,
"trace_id": message.trace_id,
}
. = target
transform:
- field: log_time
type: time, ms
index: timestamp
```
The pipeline content is straightforward: we use `vrl` processor to parse the line into a JSON object, then extract the fields to the root level.
`log_time` is specified as the time index in the transform section, other fields will be auto-inferred by the pipeline engine, see [pipeline version 2](/reference/pipeline/pipeline-config.md#transform-in-version-2) for details.
Note that the input field name is `loki_line`, which contains the original log line from Loki.
**Step 3: Configure Grafana Alloy**
Prepare an Alloy configuration file to read the log file and send it to GreptimeDB:
```
loki.source.file "greptime" {
targets = [
{__path__ = "/logs.json"},
]
forward_to = [loki.write.greptime_loki.receiver]
}
loki.write "greptime_loki" {
endpoint {
url = "http://127.0.0.1:4000/v1/loki/api/v1/push"
headers = {
"x-greptime-pipeline-name" = "pp",
}
}
external_labels = {
"job" = "greptime",
"from" = "alloy",
}
}
```
In the `greptime_loki`, the `x-greptime-pipeline-name` header is added to indicating the input data should be processed by the pipeline engine.
**Step 4: Deploy and run**
1. First, start your GreptimeDB instance. See [here](/getting-started/installation/overview.md) for quick startup.
2. [Upload](/user-guide/logs/manage-pipelines.md#upload-a-pipeline) the pipeline configuration to the database using `curl`:
```bash
curl -X "POST" "http://localhost:4000/v1/pipelines/pp" -F "file=@pipeline.yaml"
```
3. Start the Alloy Docker container to process the logs:
```shell
docker run --rm \
-v ./config.alloy:/etc/alloy/config.alloy \
-v ./logs.json:/logs.json \
--network host \
grafana/alloy:latest \
run --server.http.listen-addr=0.0.0.0:12345 --storage.path=/var/lib/alloy/data \
/etc/alloy/config.alloy
```
**Step 5: Verify the results**
After the logs are processed, you can verify that they have been successfully ingested and parsed. The database logs will show the ingestion activity.
Query the table using a MySQL client to see the parsed log data:
```sql
mysql> show tables;
+-----------+
| Tables |
+-----------+
| loki_logs |
| numbers |
+-----------+
2 rows in set (0.00 sec)
mysql> select * from loki_logs limit 1 \G
*************************** 1. row ***************************
log_time: 2025-08-21 14:23:17.892000
log_level: ERROR
log_source: application
logger: sdk.tool.DatabaseUtil
message: Connection timeout exceeded for database pool
trace_id: a7f8c92d1e4b4c6f9d2e5a8b3f7c1d9e
1 row in set (0.01 sec)
```
This output demonstrates that the pipeline engine has successfully parsed the original JSON log lines and extracted the structured data into separate columns.
For more details about pipeline configuration and features, refer to the [pipeline documentation](/reference/pipeline/pipeline-config.md).
---
## OpenTelemetry Protocol (OTLP)
[OpenTelemetry](https://opentelemetry.io/) is a vendor-neutral open-source observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, logs. The OpenTelemetry Protocol (OTLP) defines the encoding, transport, and delivery mechanism of telemetry data between telemetry sources, intermediate processes such as collectors and telemetry backends.
## OpenTelemetry Collectors
You can easily configure GreptimeDB as the target for your OpenTelemetry collector.
For more information, please refer to the [OTel Collector](otel-collector.md) and [Grafana Alloy](alloy.md) example.
## HTTP Base Endpoint
[Base endpoint URL](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/#otel_exporter_otlp_endpoint) for all signal types: `http{s}:///v1/otlp`
This unified endpoint is useful when sending multiple signal types (metrics, logs, and traces) to the same destination, simplifying your OpenTelemetry configuration.
## Metrics
GreptimeDB is an observability backend to consume OpenTelemetry Metrics natively via [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol.
### OTLP/HTTP API
To send OpenTelemetry Metrics to GreptimeDB through OpenTelemetry SDK libraries, use the following information:
- URL: `http{s}:///v1/otlp/v1/metrics`
- Headers:
- `X-Greptime-DB-Name`: ``
- `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](https://docs.greptime.com/user-guide/deployments-administration/authentication/static/) and [HTTP API](https://docs.greptime.com/user-guide/protocols/http#authentication)
The request uses binary protobuf to encode the payload, so you need to use packages that support `HTTP/protobuf`. For example, in Node.js, you can use [`@opentelemetry/exporter-metrics-otlp-proto`](https://www.npmjs.com/package/@opentelemetry/exporter-metrics-otlp-proto); in Go, you can use [`go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`](https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp); in Java, you can use [`io.opentelemetry:opentelemetry-exporter-otlp`](https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-exporter-otlp); and in Python, you can use [`opentelemetry-exporter-otlp-proto-http`](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/).
:::tip NOTE
The package names may change according to OpenTelemetry, so we recommend that you refer to the official OpenTelemetry documentation for the most up-to-date information.
:::
For more information about the OpenTelemetry SDK, please refer to the official documentation for your preferred programming language.
### Example Code
Here are some example codes about how to setup the request in different languages:
```ts
const auth = Buffer.from(`${username}:${password}`).toString('base64')
const exporter = new OTLPMetricExporter({
url: `https://${dbHost}/v1/otlp/v1/metrics`,
headers: {
Authorization: `Basic ${auth}`,
'X-Greptime-DB-Name': db,
},
timeoutMillis: 5000,
})
```
```Go
auth := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", *username, *password)))
exporter, err := otlpmetrichttp.New(
context.Background(),
otlpmetrichttp.WithEndpoint(*dbHost),
otlpmetrichttp.WithURLPath("/v1/otlp/v1/metrics"),
otlpmetrichttp.WithHeaders(map[string]string{
"X-Greptime-DB-Name": *dbName,
"Authorization": "Basic " + auth,
}),
otlpmetrichttp.WithTimeout(time.Second*5),
)
```
```Java
String endpoint = String.format("https://%s/v1/otlp/v1/metrics", dbHost);
String auth = username + ":" + password;
String b64Auth = new String(Base64.getEncoder().encode(auth.getBytes()));
OtlpHttpMetricExporter exporter = OtlpHttpMetricExporter.builder()
.setEndpoint(endpoint)
.addHeader("X-Greptime-DB-Name", db)
.addHeader("Authorization", String.format("Basic %s", b64Auth))
.setTimeout(Duration.ofSeconds(5))
.build();
```
```python
auth = f"{username}:{password}"
b64_auth = base64.b64encode(auth.encode()).decode("ascii")
endpoint = f"https://{host}/v1/otlp/v1/metrics"
exporter = OTLPMetricExporter(
endpoint=endpoint,
headers={"Authorization": f"Basic {b64_auth}", "X-Greptime-DB-Name": db},
timeout=5)
```
For more information on the example code, please refer to the official documentation for your preferred programming language.
### Prometheus Compatibility
GreptimeDB supports a Prometheus-compatible mode for OTLP metrics ingestion.
If the metrics data is persisted using the Prometheus-compatible format, you should be able to query them using PromQL, just like any Prometheus metrics.
GreptimeDB selects the ingestion format once for each OTLP export request:
- If none of the referenced metric tables exists, GreptimeDB uses the Prometheus-compatible format.
- If all referenced existing tables use the same format, GreptimeDB uses that format for every metric in the request, including any tables created by the request.
- If the request references existing tables in both formats, GreptimeDB rejects the request.
Send legacy and Prometheus-compatible metrics in separate OTLP export requests.
GreptimeDB pre-processes the incoming data before persisting them, including:
1. Converting the metric names(table names) and the label names to the Prometheus style(e.g: replace `.` with `_`). By default, GreptimeDB also adds Prometheus-style suffixes based on the metric unit and type. See [here](https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#metric-metadata-1) for details.
Here are some examples of the conversion:
| OTLP Metric / Attribute | OTLP Type / Unit | Prometheus Equivalent |
| :--- | :--- | :--- |
| `cache.hit_ratio` | Gauge / `1` | `cache_hit_ratio` |
| `memory.usage` | Gauge / `By` | `memory_usage_bytes` |
| `queue.length` | Gauge / `{item}` | `queue_length` |
| `http.server.request.duration` | Histogram / `s` | `http_server_request_duration_seconds` |
| `rpc.server.duration` | Histogram / `ms` | `rpc_server_duration_milliseconds` |
| `http.client.request.size` | Sum (Monotonic) / `By` | `http_client_request_size_bytes_total` |
| `system.network.io` | Sum (Monotonic) / `By` | `system_network_io_bytes_total` |
| `http.status_code` (Attribute) | - | `http_status_code` |
| `service.name` (Attribute) | - | `service_name` |
2. Discarding some resource attributes and all scope attributes by default. The kept resource attributes name list can be found [here](https://prometheus.io/docs/guides/opentelemetry/#promoting-resource-attributes). This behavior is configurable.
Note, `Sum` and `Histogram` data in OTLP can have delta temporality.
GreptimeDB saves their value directly without calculating the cumulative value.
See [here](https://grafana.com/blog/2023/09/26/opentelemetry-metrics-a-guide-to-delta-vs.-cumulative-temporality-trade-offs/) for some context.
You can set the HTTP headers to configure the pre-processing behaviors. Here are the options:
1. `x-greptime-otlp-metric-promote-all-resource-attrs`: Persist all resource attributes. Default to `false`.
2. `x-greptime-otlp-metric-promote-resource-attrs`: If not persisting all resource attributes, the attribute name list to be kept. Use `;` to join the name list.
3. `x-greptime-otlp-metric-ignore-resource-attrs`: If persisting all resource attributes, the attribute name list to be ignored. Use `;` to join the name list.
4. `x-greptime-otlp-metric-promote-scope-attrs`: Whether to persist the scope attributes. Default to `false`.
5. `x-greptime-otlp-metric-translation-strategy`: How to translate OTLP metric names(table names) and label names(tag columns) before persisting them. Default to `UnderscoreEscapingWithSuffixes`.
The `x-greptime-otlp-metric-translation-strategy` header accepts the following values:
| Value | Metric name behavior | Label name behavior | Original names | Translated names |
| :--- | :--- | :--- | :--- | :--- |
| `UnderscoreEscapingWithSuffixes` | Convert unsupported characters to `_`, and add Prometheus-style unit/type suffixes. | Convert unsupported characters to `_`. | Metric: `http.server.request-duration_total` (monotonic sum, unit `ms`)Label: `_http.status-code` | Metric: `http_server_request_duration_milliseconds_total`Label: `key_http_status_code` |
| `UnderscoreEscapingWithoutSuffixes` | Convert unsupported characters to `_`, but do not add unit/type suffixes. | Convert unsupported characters to `_`. | Metric: `http.server.request-duration_total` (monotonic sum, unit `ms`)Label: `_http.status-code` | Metric: `http_server_request_duration_total`Label: `key_http_status_code` |
| `NoUTF8EscapingWithSuffixes` | Keep the original metric name characters, and add Prometheus-style unit/type suffixes. | Keep the original label name characters. | Metric: `http.server.request-duration_total` (monotonic sum, unit `ms`)Label: `_http.status-code` | Metric: `http.server.request-duration_milliseconds_total`Label: `_http.status-code` |
| `NoTranslation` | Keep the original metric name without adding suffixes. | Keep the original label name characters. | Metric: `http.server.request-duration_total` (monotonic sum, unit `ms`)Label: `_http.status-code` | Metric: `http.server.request-duration_total`Label: `_http.status-code` |
The header value is case-sensitive. Invalid values are rejected with a `400 Bad Request` response.
See [OTel specs](https://opentelemetry.io/docs/specs/otel/metrics/sdk_exporters/prometheus/#translation-strategy) and [Prometheus docs](https://prometheus.io/docs/guides/opentelemetry/#utf-8) for more details.
### Data Model
The Prometheus-compatible OTLP metrics data model is mapped to the GreptimeDB data model according to the following rules:
- The name of the Metric will be used as the name of the GreptimeDB table, and the table will be automatically created if it does not exist.
- Only selected resource attributes are kept by default. See above for details and configuration options. Attributes are used as tag columns in the GreptimeDB table.
- You can refer to the [Prometheus Data Model](./prometheus.md#data-model) for other details.
- ExponentialHistogram is not supported yet.
If you are using a table that was created by an older OTLP metrics ingestion format, the existing table keeps that format. Here are some mapping differences:
- All attributes, including resource attributes, scope attributes, and data point attributes, will be used as tag columns of the GreptimeDB table.
- Each quantile of the Summary data type will be used as a separated data column of GreptimeDB, and the column name is `greptime_pxx`, where xx is the quantile, such as 90/99, etc.
## Logs
GreptimeDB consumes OpenTelemetry Logs natively via [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol.
### OTLP/HTTP API
To send OpenTelemetry Logs to GreptimeDB through OpenTelemetry SDK libraries, use the following information:
- URL: `http{s}:///v1/otlp/v1/logs`
- Headers:
- `X-Greptime-DB-Name`: ``
- `Authorization`: `Basic` authentication, which is a Base64 encoded string of `:`. For more information, please refer to [Authentication](/user-guide/deployments-administration/authentication/static.md) and [HTTP API](/user-guide/protocols/http.md#authentication).
- `X-Greptime-Log-Table-Name`: `` (optional) - The table name to store the logs. If not provided, the default table name is `opentelemetry_logs`.
- `X-Greptime-Log-Extract-Keys`: `` (optional) - The keys to extract from the attributes. The keys should be separated by commas (`,`). For example, `key1,key2,key3` will extract the keys `key1`, `key2`, and `key3` from the attributes and promote them to the top level of the log, setting them as tags. Key matching is case-insensitive. If the same key exists in log attributes, scope attributes, and resource attributes, the value from log attributes takes precedence, followed by scope attributes and then resource attributes. Supported extracted value types are string, signed integer, unsigned integer, and boolean. If the field type is array, float, or object, an error will be returned. If a pipeline is provided, this setting will be ignored.
- `X-Greptime-Pipeline-Name`: `` (optional) - The pipeline name to process the logs. If not provided, GreptimeDB uses the built-in OTLP log mapping and applies `X-Greptime-Log-Extract-Keys` if it is provided.
- `X-Greptime-Pipeline-Version`: `` (optional) - The pipeline version to process the logs. If not provided, the latest version of the pipeline will be used.
- `X-Greptime-Pipeline-Params`: `` (optional) - The pipeline parameters to use when processing logs with a custom pipeline.
`X-Greptime-Log-Pipeline-Name` and `X-Greptime-Log-Pipeline-Version` are also accepted as legacy aliases for the generic pipeline headers. Use `X-Greptime-Pipeline-Name` and `X-Greptime-Pipeline-Version` for new configurations.
The request uses binary protobuf to encode the payload, so you need to use packages that support `HTTP/protobuf`.
:::tip NOTE
The package names may change according to OpenTelemetry, so we recommend that you refer to the official OpenTelemetry documentation for the most up-to-date information.
:::
For more information about the OpenTelemetry SDK, please refer to the official documentation for your preferred programming language.
### Example Code
Please refer to the sample code in the [OpenTelemetry Collector documentation](otel-collector.md), which includes how to send OpenTelemetry logs to GreptimeDB.
You can also refer to the sample code in the [Alloy documentation](alloy.md#logs) to learn how to send OpenTelemetry logs to GreptimeDB.
### Custom Pipeline Input
When `X-Greptime-Pipeline-Name` is provided, GreptimeDB converts each OTLP log record to one pipeline event. The event uses the following fields:
| Field | Description |
| --- | --- |
| `Timestamp` | The OTLP `time_unix_nano`. |
| `ObservedTimestamp` | The OTLP `observed_time_unix_nano`. |
| `TraceId`, `SpanId` | Hex-encoded IDs. |
| `TraceFlags`, `SeverityText`, `SeverityNumber`, `Body` | The corresponding OTLP log fields. `Body` is converted to a string. |
| `ResourceSchemaUrl`, `ScopeSchemaUrl`, `ScopeName`, `ScopeVersion` | The corresponding resource and scope fields. |
| `ResourceAttributes`, `ScopeAttributes`, `LogAttributes` | Objects containing the corresponding OTLP attributes. |
### Data Model
The OTLP logs data model is mapped to the GreptimeDB data model according to the following rules:
Default table schema:
```sql
+-----------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+-----------------------+---------------------+------+------+---------+---------------+
| timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| trace_id | String | | YES | | FIELD |
| span_id | String | | YES | | FIELD |
| severity_text | String | | YES | | FIELD |
| severity_number | Int32 | | YES | | FIELD |
| body | String | | YES | | FIELD |
| log_attributes | Json | | YES | | FIELD |
| trace_flags | UInt32 | | YES | | FIELD |
| scope_name | String | PRI | YES | | TAG |
| scope_version | String | | YES | | FIELD |
| scope_attributes | Json | | YES | | FIELD |
| scope_schema_url | String | | YES | | FIELD |
| resource_attributes | Json | | YES | | FIELD |
| resource_schema_url | String | | YES | | FIELD |
+-----------------------+---------------------+------+------+---------+---------------+
14 rows in set (0.00 sec)
```
- You can use `X-Greptime-Log-Table-Name` to specify the table name for storing the logs. If not provided, the default table name is `opentelemetry_logs`.
- All attributes, including resource attributes, scope attributes, and log attributes, will be stored as a JSON column in the GreptimeDB table.
- The `body` column is created with a fulltext index by default. The index is
enabled with the default fulltext settings: `analyzer=English`,
`case_sensitive=false`, and `backend=bloom`. For the Bloom backend, the
default `granularity` is `10240` and the default `false_positive_rate` is
`0.01`. For details, see the [fulltext index
documentation](/user-guide/manage-data/data-index.md#full-text-index).
- The timestamp of the log will be used as the timestamp index in GreptimeDB, with the column name `timestamp`. It is preferred to use `time_unix_nano` as the timestamp column. If `time_unix_nano` is not provided, `observed_time_unix_nano` will be used instead.
### Append Only
By default, log table created by OpenTelemetry API are in [append only
mode](/user-guide/deployments-administration/performance-tuning/design-table.md#when-to-use-append-only-tables).
## Traces
GreptimeDB supports writing OpenTelemetry traces data directly via the [OTLP/HTTP](https://opentelemetry.io/docs/specs/otlp/#otlphttp) protocol, and it also provides a table model of OpenTelemetry traces for users to query and analyze traces data conveniently.
### OTLP/HTTP API
You can use [OpenTelemetry SDK](https://opentelemetry.io/docs/languages/) or other similar technologies to add traces data to your application. You can also use [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) to collect traces data and use GreptimeDB as the backend storage.
To send OpenTelemetry traces data to GreptimeDB through OpenTelemetry SDK libraries, please use the following information:
- URL: `http{s}:///v1/otlp/v1/traces`
- Headers:
- `Content-Type`: `application/x-protobuf`
- `Authorization`: `Basic` authentication.
- `X-Greptime-DB-Name`: ``
- `X-Greptime-Trace-Table-Name`: `` (optional) - The table name to store the traces. If not provided, the default table name is `opentelemetry_traces`.
- `X-Greptime-Pipeline-Name`: `greptime_trace_v1` (required) - The pipeline name to process the traces.
GreptimeDB accepts **protobuf encoded traces data** via **HTTP protocol**.
### Example Code
You can directly send OpenTelemetry traces data to GreptimeDB, or use OpenTelemetry Collector to collect traces data and use GreptimeDB as the backend storage. Please refer to the example code in the [OpenTelemetry Collector documentation](/user-guide/traces/read-write.md#opentelemetry-collector) to learn how to send OpenTelemetry traces data to GreptimeDB.
### Data Model
GreptimeDB maps the OTLP traces data model to a table schema. By default, trace data is stored in the `opentelemetry_traces` table.
```sql
+------------------------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+------------------------------------+---------------------+------+------+---------+---------------+
| timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| timestamp_end | TimestampNanosecond | | YES | | FIELD |
| duration_nano | UInt64 | | YES | | FIELD |
| parent_span_id | String | | YES | | FIELD |
| trace_id | String | | YES | | FIELD |
| span_id | String | | YES | | FIELD |
| span_kind | String | | YES | | FIELD |
| span_name | String | | YES | | FIELD |
| span_status_code | String | | YES | | FIELD |
| span_status_message | String | | YES | | FIELD |
| trace_state | String | | YES | | FIELD |
| scope_name | String | | YES | | FIELD |
| scope_version | String | | YES | | FIELD |
| service_name | String | PRI | YES | | TAG |
| span_attributes.net.sock.peer.addr | String | | YES | | FIELD |
| span_attributes.peer.service | String | | YES | | FIELD |
| span_events | Json | | YES | | FIELD |
| span_links | Json | | YES | | FIELD |
+------------------------------------+---------------------+------+------+---------+---------------+
```
- Each row represents a single span.
- `service_name` is used as a **Tag** (part of the **Primary Key**).
- `timestamp` is used as the **Time Index**.
- Resource attributes, scope attributes, and span attributes are automatically flattened into separate columns.
- Note: `resource_attributes.service.name` is excluded from flattening as it is already stored in the `service_name` column.
- `span_events` and `span_links` are stored as `JSON` data types by default.
For more details on the data model and auxiliary tables, please refer to [Trace Data Modeling](/user-guide/traces/data-model.md).
Note:
1. The `greptime_trace_v1` process uses the `trace_id` field to divide data into partitions for better performance. **Please make sure the first letter of the `trace_id` is evenly distributed**.
2. For non-test scenarios, you might want to set a `ttl` to the trace table to avoid data overload. Set the HTTP header `x-greptime-hints: ttl=7d` would set a `ttl` of 7 days during the table creation, see [here](/reference/sql/create.md#table-options) for more details about `ttl` in table option.
### Schema Evolution and Partial Success
The `greptime_trace_v1` model automatically adds new flattened attribute columns. When a trace table already exists, GreptimeDB reconciles incoming attribute value types with the existing table schema. Compatible scalar values can be coerced to the existing column type, and existing `Int64` attribute columns may be widened to `Float64` when incoming values contain both integers and floats.
If some spans in a trace request cannot be written because of incompatible schema or values, GreptimeDB may still accept the other spans and return an OTLP `partial_success` response with `rejected_spans` and `error_message`. If all spans are rejected, the endpoint returns a `400 Bad Request` response.
### Auxiliary Tables
GreptimeDB automatically creates auxiliary tables (e.g., `opentelemetry_traces_services` and `opentelemetry_traces_operations`) to facilitate searching for services and operations. See [Auxiliary Tables](/user-guide/traces/data-model.md#auxiliary-tables) for details.
### Append-only Mode
By default, trace tables created by the OpenTelemetry API are in [append-only mode](/user-guide/deployments-administration/performance-tuning/design-table.md#when-to-use-append-only-tables).
---
## OTel Collector
[OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) offers a vendor-agnostic implementation of how to receive, process and export telemetry data. It can act as an intermediate layer to send data from different sources to GreptimeDB.
Below is a sample configuration for sending data to GreptimeDB using OpenTelemetry Collector.
```yaml
extensions:
basicauth/client:
client_auth:
username:
password:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
exporters:
otlp_http/traces:
endpoint: 'http://127.0.0.1:4000/v1/otlp'
# auth:
# authenticator: basicauth/client
headers:
# x-greptime-db-name: ''
x-greptime-pipeline-name: 'greptime_trace_v1'
tls:
insecure: true
otlp_http/logs:
endpoint: 'http://127.0.0.1:4000/v1/otlp'
# auth:
# authenticator: basicauth/client
headers:
# x-greptime-db-name: ''
# x-greptime-log-table-name: ''
# x-greptime-pipeline-name: ''
# x-greptime-pipeline-params: ''
tls:
insecure: true
otlp_http/metrics:
endpoint: 'http://127.0.0.1:4000/v1/otlp'
# auth:
# authenticator: basicauth/client
headers:
# x-greptime-db-name: ''
tls:
insecure: true
service:
# extensions: [basicauth/client]
pipelines:
traces:
receivers: [otlp]
exporters: [otlp_http/traces]
logs:
receivers: [otlp]
exporters: [otlp_http/logs]
metrics:
receivers: [otlp]
exporters: [otlp_http/metrics]
```
In the above configuration, we define a receiver `otlp` that can receive data from OpenTelemetry. We also define three exporters: `otlp_http/traces`, `otlp_http/logs`, and `otlp_http/metrics`, which send data to the OTLP endpoint of GreptimeDB.
Based on the OTLP/HTTP protocol, we have added some headers to specify certain parameters, such as `x-greptime-pipeline-name` and `x-greptime-log-table-name`:
* The `x-greptime-pipeline-name` header is used to specify the pipeline name to use, and,
* the `x-greptime-pipeline-params` header is used to pass parameters to the pipeline, and,
* the `x-greptime-log-table-name` header is used to specify the table name in GreptimeDB where the data will be written.
If you have enabled [authentication](/user-guide/deployments-administration/authentication/overview.md) in GreptimeDB, you need to use the `basicauth/client` extension to handle basic authentication.
Here, we strongly recommend using different exporters to handle traces, logs, and metrics separately. On one hand, GreptimeDB supports some specific headers to customize processing flows; on the other hand, this also helps with data isolation.
For more about OpenTelemetry protocols, please read the [doc](/user-guide/ingest-data/for-observability/opentelemetry.md).
---
## Ingest Data for Observability
Observability data reaches GreptimeDB through the protocol its source already speaks. [Ingest Data](../overview.md) lists every source in one table; this page groups the observability paths by signal.
## Metrics
- [Prometheus Remote Write](prometheus.md) — for an existing Prometheus server or Prometheus Agent.
- [OpenTelemetry](opentelemetry.md) — OTLP/HTTP from an OpenTelemetry SDK.
- [InfluxDB Line Protocol](../for-iot/influxdb-line-protocol.md) — the path for migrating off InfluxDB, and for Telegraf. This page also covers the Telegraf configuration.
## Logs
- [Loki](loki.md) — the Loki Push API, the path for migrating off Loki without changing the clients already writing to it. This page also covers the Alloy configuration.
- [Elasticsearch](elasticsearch.md) — the `_bulk` API, compatible with Logstash, Filebeat, and Telegraf.
- [Splunk](splunk.md) — the HTTP Event Collector (HEC) protocol, compatible with Vector and the OpenTelemetry Collector.
Parsing and transforming text logs before they are stored is covered in [Logs](/user-guide/logs/overview.md).
## Traces
- [OpenTelemetry](opentelemetry.md) — OTLP/HTTP from an SDK or the Collector.
Storing traces and querying them with SQL or the Jaeger-compatible API is covered in [Traces](/user-guide/traces/overview.md).
## Agents and pipelines
These sit between the source and GreptimeDB, writing through an exporter, sink, or output that you configure. The signals each one carries differ:
- [OpenTelemetry Collector](otel-collector.md) — routes metrics, logs, and traces through an OTLP/HTTP exporter.
- [Vector](vector.md) — metrics through the `greptimedb_metrics` sink, logs through `greptimedb_logs`. No traces sink.
- [Fluent Bit](fluent-bit.md) — logs through the HTTP output; metrics, logs, and traces through the OpenTelemetry output; metrics also through Prometheus Remote Write.
- [Grafana Alloy](alloy.md) — writes through Remote Write, OpenTelemetry, or Loki.
- [Kafka](kafka.md) — consumed into GreptimeDB with Vector as the transport.
---
## Prometheus
GreptimeDB can serve as a long-term storage solution for Prometheus,
providing a seamless integration experience.
## Remote write configuration
### Prometheus configuration file
To configure Prometheus with GreptimeDB,
update your [Prometheus configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file) (`prometheus.yml`) as follows:
```yaml
remote_write:
- url: http://localhost:4000/v1/prometheus/write?db=public
# Uncomment and set credentials if authentication is enabled
# basic_auth:
# username: greptime_user
# password: greptime_pwd
remote_read:
- url: http://localhost:4000/v1/prometheus/read?db=public
# Uncomment and set credentials if authentication is enabled
# basic_auth:
# username: greptime_user
# password: greptime_pwd
```
- The host and port in the URL represent the GreptimeDB server. In this example, the server is running on `localhost:4000`. You can replace it with your own server address. For the HTTP protocol configuration in GreptimeDB, please refer to the [protocol options](/user-guide/deployments-administration/configuration.md#protocol-options).
- The `db` parameter in the URL represents the database to which we want to write data. It is optional. By default, the database is set to `public`.
- `basic_auth` is the authentication configuration. Fill in the username and password if GreptimeDB authentication is enabled. Please refer to the [authentication document](/user-guide/deployments-administration/authentication/overview.md).
### Grafana Alloy configuration file
If you are using Grafana Alloy, configure the remote write endpoint in the Alloy configuration file (`config.alloy`). For more information, refer to the [Alloy documentation](alloy.md#prometheus-remote-write).
### Vector configuration file
If you use Vector, configure Remote Write in Vector's configuration file (vector.toml). For more information, see the [Vector documentation](vector.md#using-prometheus-remote-write-protocol).
## Data Model
In the [data model](/user-guide/concepts/data-model.md) of GreptimeDB, data is organized into tables with columns for tags, time index, and fields.
GreptimeDB can be thought of as a multi-value data model,
automatically grouping multiple Prometheus metrics into corresponding tables.
This allows for efficient data management and querying.
When the metrics are written into GreptimeDB by remote write endpoint, they will be transformed as
follows:
| Sample Metrics | In GreptimeDB | GreptimeDB Data Types |
| -------------- | ------------------------- | --------------------- |
| Name | Table (Auto-created) Name | String |
| Value | Column (Field) | Double |
| Timestamp | Column (Time Index) | Timestamp |
| Label | Column (Tag) | String |
For example, the following Prometheus metric:
```txt
prometheus_remote_storage_samples_total{instance="localhost:9090", job="prometheus",
remote_name="648f0c", url="http://localhost:4000/v1/prometheus/write"} 500
```
will be transformed as a row in the table `prometheus_remote_storage_samples_total`：
| Column | Value | Column Data Type |
| :----------------- | :------------------------------------------ | :----------------- |
| instance | localhost:9090 | String |
| job | prometheus | String |
| remote_name | 648f0c | String |
| url | `http://localhost:4000/v1/prometheus/write` | String |
| greptime_value | 500 | Double |
| greptime_timestamp | The sample's unix timestamp | Timestamp |
## Improve efficiency by using metric engine
The Prometheus remote writing always creates a large number of small tables.
These tables are classified as logical tables in GreptimeDB.
However, having a large number of small tables can be inefficient for both data storage and query performance.
To address this, GreptimeDB introduces the [metric engine](/contributor-guide/datanode/metric-engine.md) feature,
which stores the data represented by the logical tables in a single physical table.
This approach reduces storage overhead and improves columnar compression efficiency.
The metric engine is enabled by default in GreptimeDB,
and you don't need to specify any additional configuration.
By default, the physical table used is `greptime_physical_table`.
If you want to use a specific physical table, you can specify the `physical_table` parameter in the remote write URL.
If the specified physical table doesn't exist, it will be automatically created.
```yaml
remote_write:
- url: http://localhost:4000/v1/prometheus/write?db=public&physical_table=greptime_physical_table
```
Data is stored in the physical table,
while queries are performed on logical tables to provide an intuitive view from a metric perspective.
For instance, when successfully writing data, you can use the following command to display the logical tables:
```sql
show tables;
```
```sql
+---------------------------------------------------------------+
| Tables |
+---------------------------------------------------------------+
| prometheus_remote_storage_enqueue_retries_total |
| prometheus_remote_storage_exemplars_pending |
| prometheus_remote_storage_read_request_duration_seconds_count |
| prometheus_rule_group_duration_seconds |
| ...... |
+---------------------------------------------------------------+
```
The physical table itself can also be queried.
It contains columns from all the logical tables,
making it convenient for multi-join analysis and computation.
To view the schema of the physical table, use the `DESC TABLE` command:
```sql
DESC TABLE greptime_physical_table;
```
The physical table includes all the columns from the logical tables:
```sql
+--------------------+----------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+----------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampMillisecond | PRI | NO | | TIMESTAMP |
| greptime_value | Float64 | | YES | | FIELD |
| __table_id | UInt32 | PRI | NO | | TAG |
| __tsid | UInt64 | PRI | NO | | TAG |
| device | String | PRI | YES | | TAG |
| instance | String | PRI | YES | | TAG |
| job | String | PRI | YES | | TAG |
| error | String | PRI | YES | | TAG |
...
```
You can use the `SELECT` statement to filter data from the physical table as needed.
For example, you can filter data based on the `device` condition from logical table A and the `job` condition from logical table B:
```sql
SELECT *
FROM greptime_physical_table
WHERE greptime_timestamp > '2024-08-07 03:27:26.964000'
AND device = 'device1'
AND job = 'job1';
```
### GreptimeDB cluster with metric engine
If you are using GreptimeDB cluster for Prometheus remote write, you may notice
that only 1 datanode taking all the workload and other datanodes receives no
traffic. This is because with default settings, there is only 1 metric engine
physical table, and only 1 partition(region) in that table. The datanode that
serves the partition will take all the data ingestion.
Why we are not creating more partitions by default? [GreptimeDB's table
partition](/user-guide/deployments-administration/manage-data/table-sharding.md)
is based pre-configured partition columns. However, in Prometheus ecosystem,
there is no common column (label, as in Prometheus) that is suitable for good
partition rules.
To fix this, we recommend you to define your own partition rule based on your
data model. For example, it can be `namespace` if your are monitoring a
kubernetes cluster. The partition columns should have enough cardinality to
divide data. Also we recommend you to create 2x-3x partitions on initial
datanode count, so when you scaling more datanodes in your cluster, just migrate
those partitions to new ones.
An example DDL of partitioned physical table based on `namespace` label:
```sql
CREATE TABLE greptime_physical_table (
greptime_timestamp TIMESTAMP(3) NOT NULL,
greptime_value DOUBLE NULL,
namespace STRING PRIMARY KEY,
TIME INDEX (greptime_timestamp),
)
PARTITION ON COLUMNS (namespace) (
namespace <'g',
namespace >= 'g' AND namespace < 'n',
namespace >= 'n' AND namespace < 't',
namespace >= 't'
)
ENGINE = metric
with (
"physical_metric_table" = "",
);
```
Note that you won't have need add all possible *PRIMARY KEY* (label) here,
metric engine will add them automatically. Only labels you use for partitioning
are required to be defined ahead of time.
## Special labels for ingestion options
:::warning Experimental Feature
This experimental feature may contain unexpected behavior, have its functionality change in the future.
:::
Normally, the complete dataset of a remote write request is ingested into the database under the same option, for example, a default physical table with metric engine enabled.
All the logical tables (i.e, the metrics) is backed with the same physical table, even when the number of metrics grows.
It's probably fine for data ingestion. However, this set-up might slow down the query speed if you just want to query for a small group of metrics, but the database have to scan the complete dataset because they are all in the same physical table.
If you can foresee a large data volume and incremental queries upon a small group of metrics each time, then it might be useful to split the storage during the ingestion to reduce the query overhead later. This fine-grade level of control can be achieved using ingest options for each metric within a remote request.
GreptimeDB supports special labels.
These labels (along with their values) turn into ingest options during the parsing phase, allowing each individual metric within a request to be more precisely controlled.
The labels are not mutually exclusive, so they can be combined together to produce more versatile controlling.
Here is a representative diagram of special labels for a metric. Note this is not the actual data model of a metric.
| `__name__` | `x_greptime_database` | `x_greptime_physical_table` | `pod_name_label` | `__normal_label_with_underscore_prefix__` | `timestamp` | `value` |
|--------------------|-------------------------|-------------------------------|-----------------------|---------------------------------------------|---------------------------|-----------|
| `some_metric_name` | `public` | `p_1` | `random_k8s_pod_name` | `true` | `2025-06-17 16:31:52.000` | `12.1` |
The special labels you see above are just normal valid labels in Prometheus.
GreptimeDB recognizes some of the label names and turns them into ingest options.
It's much like the custom HTTP headers, where you just set a valid HTTP header and its value to indicate following operations, only the header pair means nothing outside your program.
Here is a list of supported label names:
- `x_greptime_database`
- `x_greptime_physical_table`
### Setting labels
How to set labels to the metrics is very dependent on the tools (or code) that collects the metrics and send them over to the database.
If you're using Prometheus to scrape metrics from the source and send them to GreptimeDB using remote write, you can add `external_labels` in the global config.
Refer to the [docs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file) here.
It's the same for other collecting tools. You may have to find the relevant configuration for your tool of choice.
### `x_greptime_database`
This option decides which database the metric goes into. Note, the database should be created in advance(for instance, using `create database xxx` SQL).
Metrics from the same tech stack can have same metric names.
For example, if you have two Kubernetes clusters with very different content, and you deploy a single collector on both clusters, they will generate metrics with same names but with different labels or values.
If the metrics are collected and ingested into the same database, then on a Grafana dashboard you will have to manually set every label selection on every diagram to view the two clusters' metrics separately. This can be very tedious and painful.
In this case, you might want to store the metrics on different databases during ingestion and use two dashboards to view the metrics separately.
### `x_greptime_physical_table`
If the metric is storing using the metric engine, then there is a physical table behind each metric's logical table.
By default, all metrics using the same physical table.
With the number of the metrics growing, this physical table becomes a super wide table.
If the metric frequency is different, then the table will be sparse.
Finding a certain metric or a certain label in the complete metrics dataset would be very time-consuming since the database have to scan all the 'irrelevant' data.
In this case, you might want to split the metrics into different physical tables to ease the pressure on a single physical table.
It can also be helpful to group metrics by their frequency.
Note, each metric's logical table is bound to a physical table upon creation.
So setting different physical table for the same metric within the same database won't work.
## Using pipeline in remote write
GreptimeDB supports using a pipeline to process Prometheus remote write requests.
You can simply set the HTTP header `x-greptime-pipeline-name` to the target pipeline name to enable pipeline processing.
Here is a very simple pipeline configuration, using `vrl` processor to add a `source` label to each metric:
```YAML
version: 2
processors:
- vrl:
source: |
.source = "local_laptop"
.
transform:
- field: greptime_timestamp
type: time, ms
index: timestamp
```
The result looks something like this
```
mysql> select * from `go_memstats_mcache_inuse_bytes`;
+----------------------------+----------------+--------------------+---------------+--------------+
| greptime_timestamp | greptime_value | instance | job | source |
+----------------------------+----------------+--------------------+---------------+--------------+
| 2025-07-11 07:42:03.064000 | 1200 | node_exporter:9100 | node-exporter | local_laptop |
| 2025-07-11 07:42:18.069000 | 1200 | node_exporter:9100 | node-exporter | local_laptop |
+----------------------------+----------------+--------------------+---------------+--------------+
2 rows in set (0.01 sec)
```
You can refer to the [pipeline's documentation](/user-guide/logs/use-custom-pipelines.md) for more details.
## Performance tuning
### Batching mode
When the metric engine is enabled, GreptimeDB supports a batching mode for Prometheus Remote Write
that reduces per-request overhead and improves ingestion throughput.
In batching mode, incoming rows are accumulated and flushed to the metric engine in larger batches,
which can yield up to **2x throughput improvement** in clustered deployments with multiple regions,
but requires careful tuning of batch parameters to match the ingestion workload and prevent increased
latency.
Batching mode is **disabled by default**.
To enable it, set `pending_rows_flush_interval` to a non-zero duration in the `[prom_store]` section of the configuration file:
```toml
[prom_store]
enable = true
with_metric_engine = true
pending_rows_flush_interval = "500ms"
```
The following table describes the batching-related options:
| Key | Type | Default | Description |
| ---------------------------- | ------- | -------- | --------------------------------------------------------------------------- |
| pending_rows_flush_interval | String | `"0s"` | Interval between batch flushes. `"0s"` disables batching. |
| max_batch_rows | Integer | `100000` | Maximum number of rows per batch before a flush is triggered. |
| max_concurrent_flushes | Integer | `256` | Maximum number of flush operations that can run concurrently. |
| max_inflight_requests | Integer | `3000` | Maximum number of in-flight write requests waiting for batch completion. |
:::tip
Batching mode only takes effect when both `with_metric_engine` is `true` and `pending_rows_flush_interval` is set to a non-zero duration.
:::
### Request timeout and retries
When [`http.timeout`](/user-guide/deployments-administration/configuration.md#protocol-options) is set to a non-zero duration and a remote write request to `/v1/prometheus/write` exceeds it,
GreptimeDB responds with `504 Gateway Timeout` instead of `408 Request Timeout`.
Prometheus and other remote write senders retry on `5xx` responses,
so a timed-out request is retried automatically instead of being dropped.
In batching mode, rows that have been accepted into a pending batch continue to flush in the background even after the request times out.
To ensure a request can wait long enough for its batch to flush,
GreptimeDB raises a non-zero `http.timeout` that is less than or equal to `pending_rows_flush_interval` plus 1 second to that value and logs a warning.
Setting `http.timeout = "0s"` (the default) disables the HTTP timeout entirely.
### Customized physical table
By default, the metric engine will automatically create a physical table named `greptime_physical_table` if it does not already exist. For performance optimization, you may choose to create a physical table with customized configurations.
### Enable skipping index
By default, the metric engine won't create indexes for columns. You can enable it by setting the `index.type` to `skipping`.
Create a physical table with a skipping index; all automatically added columns will have this index applied.
```sql
CREATE TABLE greptime_physical_table (
greptime_timestamp TIMESTAMP(3) NOT NULL,
greptime_value DOUBLE NULL,
TIME INDEX (greptime_timestamp),
)
engine = metric
with (
"physical_metric_table" = "",
"index.type" = "skipping"
);
```
For more configurations, please refer to the [create table](/reference/sql/create.md#create-table) section.
## VictoriaMetrics remote write
VictoriaMetrics slightly modified Prometheus remote write protocol for better
compression. The protocol is automatically enabled when you are using `vmagent`
to send data to a compatible backend.
GreptimeDB has this variant supported, too. Just configure GreptimeDB's remote
write url for `vmagent`. For example, if you have GreptimeDB installed locally:
```shell
vmagent -remoteWrite.url=http://localhost:4000/v1/prometheus/write
```
---
## Splunk
## Overview
GreptimeDB implements a subset of the [Splunk HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector) protocol, so shippers that already speak HEC — such as Vector and the OpenTelemetry Collector — can write to GreptimeDB by changing the endpoint URL and the token.
Two ingestion endpoints are available:
- `/services/collector/event` takes structured JSON events.
- `/services/collector/raw` takes plain text and stores it verbatim.
Splunk's `index` maps to a GreptimeDB table. `host`, `source`, `sourcetype` and the keys under `fields` become tag columns, which GreptimeDB adds to the table's primary key.
Indexer acknowledgment (`/services/collector/ack`) is not implemented. The `channel` parameter is accepted and ignored.
:::tip NOTE
Fluent Bit's native `splunk` output cannot be used here. It hardcodes the request path to `/services/collector/event` and exposes no path setting, so it cannot reach GreptimeDB's `/v1/splunk` mount without a reverse proxy. To send data from Fluent Bit, use [its HTTP output](/user-guide/ingest-data/for-observability/fluent-bit.md) instead.
:::
## HTTP API
The base path is `/v1/splunk`; clients append the collector path themselves. Point your shipper at `http://${db_host}:${db_http_port}/v1/splunk`, for example `http://localhost:4000/v1/splunk`.
| Endpoint | Method | Purpose |
| --- | --- | --- |
| `/v1/splunk/services/collector/event` | `POST` | Ingest JSON events. Aliases: `/services/collector`, `/services/collector/event/1.0` |
| `/v1/splunk/services/collector/raw` | `POST` | Ingest plain text. Alias: `/services/collector/raw/1.0` |
| `/v1/splunk/services/collector/health` | `GET` | Health probe. Alias: `/services/collector/health/1.0` |
Request bodies may be gzip-compressed (`Content-Encoding: gzip`).
### Authentication
Authentication only applies when the server runs with a [user provider](/user-guide/deployments-administration/authentication/overview.md). Without one, requests are accepted unauthenticated.
HEC clients send `Authorization: Splunk `. GreptimeDB reads the token as `username:password`:
```
Authorization: Splunk greptime_user:greptime_pwd
```
`Authorization: Basic ` is also accepted.
:::warning
Most Splunk clients default to a single opaque token with no colon in it. GreptimeDB rejects such a token with HEC code `4`. Configure the client's token as `:`.
:::
The health endpoint is public and never requires a token.
## Ingest JSON events
`POST /v1/splunk/services/collector/event`
The body is one or more HEC event objects. They may be concatenated directly, separated by JSON whitespace such as a newline or a space, or wrapped in a top-level JSON array. Any other separator, a comma or a semicolon for example, is a parse error and fails the request with HEC code `6`.
### Field mapping
| HEC field | Maps to |
| --- | --- |
| `time` | The `greptime_timestamp` time index |
| `index` | The table name |
| `host`, `source`, `sourcetype` | Tag columns |
| keys under `fields` | Tag columns, one per key |
| `event` and any remaining keys | Data columns |
`time` is epoch seconds and may be fractional. Values at or above `1e12` are read as milliseconds instead. If `time` is absent or `null`, the server's ingest time is used.
:::warning
A `time` that is present but unparsable is an error, not a fallback — the request fails with HEC code `6`. The same applies to `?time=` on the raw endpoint.
:::
### Table routing
The table is resolved in this order:
1. The event's `index`, coerced into a valid table name. Characters outside `[A-Za-z0-9_:\-.@#]` become `_`, and an underscore is prepended when the first character is not valid. For example `my index/2026` becomes `my_index_2026`.
2. The `table` query parameter.
3. The default table `splunk_logs`.
### Query parameters
| Parameter | Description |
| --- | --- |
| `db` | Target database. Defaults to `public`. |
| `table` | Fallback table name, used when the event carries no `index`. |
| `pipeline_name` | Pipeline to run. Defaults to the built-in `greptime_identity`. |
| `version` | Pipeline version. Defaults to the latest version. |
:::tip NOTE
Only these parameters are honored on this endpoint. Other log-ingestion parameters such as `msg_field`, `custom_time_index`, and `ignore_errors` have no effect here.
:::
### Example
```bash
curl -X POST "http://localhost:4000/v1/splunk/services/collector/event" \
-H "Authorization: Splunk greptime_user:greptime_pwd" \
-H "Content-Type: application/json" \
-d '{"time":1785312000,"host":"web-01","source":"/var/log/nginx/access.log","sourcetype":"nginx:access","index":"splunk_events","fields":{"env":"prod","region":"us-west"},"event":{"status":200,"path":"/api/v1/health","latency_ms":12}}
{"time":1785312001,"host":"web-02","source":"/var/log/nginx/access.log","sourcetype":"nginx:access","index":"splunk_events","fields":{"env":"prod","region":"eu-central"},"event":{"status":503,"path":"/api/v1/orders","latency_ms":841}}'
```
A successful write returns:
```json
{"code":0,"text":"Success"}
```
The table `splunk_events` is created on the first write:
```sql
DESC TABLE splunk_events;
```
```
+--------------------+---------------------+------+------+---------+---------------+
| Column | Type | Key | Null | Default | Semantic Type |
+--------------------+---------------------+------+------+---------+---------------+
| greptime_timestamp | TimestampNanosecond | PRI | NO | | TIMESTAMP |
| env | String | PRI | YES | | TAG |
| event.latency_ms | Int64 | | YES | | FIELD |
| event.path | String | | YES | | FIELD |
| event.status | Int64 | | YES | | FIELD |
| host | String | PRI | YES | | TAG |
| region | String | PRI | YES | | TAG |
| source | String | PRI | YES | | TAG |
| sourcetype | String | PRI | YES | | TAG |
+--------------------+---------------------+------+------+---------+---------------+
```
The default pipeline flattens the nested `event` object, which is why the data columns are named `event.status`, `event.path`, and `event.latency_ms`.
```sql
SELECT * FROM splunk_events ORDER BY greptime_timestamp;
```
```
+---------------------+------+------------------+----------------+--------------+--------+------------+---------------------------+--------------+
| greptime_timestamp | env | event.latency_ms | event.path | event.status | host | region | source | sourcetype |
+---------------------+------+------------------+----------------+--------------+--------+------------+---------------------------+--------------+
| 2026-07-29 08:00:00 | prod | 12 | /api/v1/health | 200 | web-01 | us-west | /var/log/nginx/access.log | nginx:access |
| 2026-07-29 08:00:01 | prod | 841 | /api/v1/orders | 503 | web-02 | eu-central | /var/log/nginx/access.log | nginx:access |
+---------------------+------+------------------+----------------+--------------+--------+------------+---------------------------+--------------+
```
New keys appearing under `fields` on a later write are added as tag columns and joined to the primary key automatically.
## Ingest raw text
`POST /v1/splunk/services/collector/raw`
The body is stored verbatim in a `message` column. GreptimeDB does not parse its content — use a [custom pipeline](#using-a-pipeline) reading `message` if you need structure.
:::warning
By default the **entire body is one event**, so multi-line payloads such as stack traces stay intact. To split the body into several events, pass `linebreaker` with the delimiter you want, percent-encoded — `?linebreaker=%0A` splits on newlines. Whitespace-only segments are dropped.
:::
Metadata is request-level on this endpoint: it comes from query parameters and applies to every event in the request.
### Query parameters
| Parameter | Description |
| --- | --- |
| `host`, `source`, `sourcetype` | Tag columns applied to every event in the request. |
| `index` | Table name, sanitized the same way as the `index` field on `/event`. |
| `time` | Epoch timestamp applied to every event. Defaults to ingest time. |
| `table` | Fallback table name when `index` is absent. |
| `linebreaker` | Percent-encoded literal delimiter used to split the body. Without it, the whole body is one event. |
| `pipeline_name` | Pipeline to run. Defaults to `greptime_identity`. |
| `version` | Pipeline version. |
| `db` | Target database. Defaults to `public`. |
| `channel` | Accepted and ignored. Also accepted as the `x-splunk-request-channel` header. |
### Example
```bash
printf 'line one\nline two\nline three\n' | curl -X POST \
"http://localhost:4000/v1/splunk/services/collector/raw?index=splunk_raw&host=web-01&source=app.log&sourcetype=applog&linebreaker=%0A" \
-H "Authorization: Splunk greptime_user:greptime_pwd" \
--data-binary @-
```
This writes three rows. Dropping `linebreaker` from the same request writes one row containing all three lines.
```sql
SHOW CREATE TABLE splunk_raw;
```
```sql
CREATE TABLE IF NOT EXISTS `splunk_raw` (
`greptime_timestamp` TIMESTAMP(9) NOT NULL,
`host` STRING NULL,
`message` STRING NULL,
`source` STRING NULL,
`sourcetype` STRING NULL,
TIME INDEX (`greptime_timestamp`),
PRIMARY KEY (`host`, `source`, `sourcetype`)
)
ENGINE=mito
WITH(
'comment' = 'Created on insertion',
append_mode = 'true'
)
```
## Using a pipeline
Both endpoints run every event through a [pipeline](/user-guide/logs/manage-pipelines.md). The default is the built-in `greptime_identity`, which maps each input key to a column of the same name.
Override it with the `pipeline_name` query parameter or the `x-greptime-pipeline-name` header. When both are present, the query parameter wins. Use `version` to pin a specific pipeline version.
:::warning
Promoting `host`, `source`, `sourcetype`, and the `fields` keys to tag columns only happens on the default `greptime_identity` path. A custom pipeline owns its output schema, so declare the columns you want indexed in the pipeline itself.
:::
## Response codes
The response body is `{"text": ..., "code": ...}`, matching HEC. Clients branch on `code`.
| Code | Meaning |
| --- | --- |
| `0` | Success |
| `2` | Token is required |
| `4` | Invalid token |
| `5` | No data |
| `6` | Invalid data format, including an unparsable `time` or `version` |
| `7` | Incorrect index, meaning the resolved table name is invalid |
| `8` | Internal server error |
| `12` | Event field is required |
| `13` | Event field cannot be blank |
| `17` | Returned by the health endpoint to report a healthy collector |
:::warning
Validation rejects the whole request. If any event in a batch has a missing or blank `event`, or a `time` that is present but unparsable, none of the events in that batch are written — the endpoint does not skip bad events and keep the rest. Optional metadata such as `host` or `source` may be absent without any effect.
:::
## Vector
Vector's [`splunk_hec_logs`](https://vector.dev/docs/reference/configuration/sinks/splunk_hec_logs/) sink writes to GreptimeDB with no other changes:
```toml
[sinks.greptime]
type = "splunk_hec_logs"
inputs = ["my_source"]
endpoint = "http://localhost:4000/v1/splunk"
default_token = "greptime_user:greptime_pwd"
index = "vector_logs"
compression = "gzip"
encoding.codec = "json"
```
- `endpoint`: the `/v1/splunk` base path, without a collector path. Vector appends that itself.
- `default_token`: must be `:`, not an opaque token.
- `index`: the target table.
- `encoding.codec`: required by this sink.
- `endpoint_target`: `event` by default, which uses `/services/collector/event`. Set it to `raw` for the raw endpoint.
To use the raw endpoint instead:
```toml
[sinks.greptime]
type = "splunk_hec_logs"
inputs = ["my_source"]
endpoint = "http://localhost:4000/v1/splunk"
default_token = "greptime_user:greptime_pwd"
index = "vector_raw_logs"
endpoint_target = "raw"
encoding.codec = "text"
batch.max_events = 1
```
Vector concatenates batched events with no separator, so on the raw endpoint a whole batch arrives as one event. Set `batch.max_events = 1` as above, or stay on the default `event` target.
:::tip NOTE
`linebreaker` would split such a batch server-side, but this particular sink cannot send it: it has no option for arbitrary query parameters, and appending one to `endpoint` does not work because Vector joins `endpoint` and the collector path as plain strings — `http://localhost:4000/v1/splunk?linebreaker=%0A` becomes `/v1/splunk?linebreaker=%0A/services/collector/raw?index=...`, which no longer matches the route. HEC clients that can set query parameters use `linebreaker` normally.
:::
## OpenTelemetry Collector
The [`splunk_hec` exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/splunkhecexporter/README.md) is configured the same way:
```yaml
exporters:
splunk_hec:
endpoint: "http://localhost:4000/v1/splunk/services/collector"
token: "greptime_user:greptime_pwd"
index: "otel_logs"
tls:
insecure: true
service:
pipelines:
logs:
receivers: [otlp]
exporters: [splunk_hec]
```
Unlike Vector, this exporter expects the full collector path in `endpoint`.
---
## Vector
:::tip NOTE
This document is based on Vector v0.49.0. All example configurations below are based on this version. Please adjust the host and port configurations for each sink according to your actual GreptimeDB instance. All port values below are defaults.
:::
Vector is a high-performance observability data pipeline. It natively supports GreptimeDB as a metrics data receiver. Through Vector, you can receive metrics data from various sources including Prometheus, OpenTelemetry, StatsD, etc. GreptimeDB can serve as a sink component for Vector to receive metrics data.
## Writing Metrics Data
GreptimeDB supports multiple ways to write metrics data:
- Using [`greptimedb_metrics` sink](https://vector.dev/docs/reference/configuration/sinks/greptimedb_metrics/)
- Using InfluxDB line protocol format
- Using Prometheus Remote Write protocol
### Using `greptimedb_metrics` sink
#### Example
Below is an example configuration using `greptimedb_metrics` sink to write host metrics:
```toml
# sample.toml
[sources.in]
type = "host_metrics"
[sinks.my_sink_id]
inputs = ["in"]
type = "greptimedb_metrics"
endpoint = ":4001"
dbname = ""
username = ""
password = ""
new_naming = true
```
Vector uses gRPC to communicate with GreptimeDB, so the default port for Vector sink is `4001`. If you changed the default gRPC port when starting GreptimeDB with [custom configuration](/user-guide/deployments-administration/configuration.md#configuration-file-options), please use your own port.
For more requirements, please visit [Vector GreptimeDB Configuration](https://vector.dev/docs/reference/configuration/sinks/greptimedb_metrics/) to view more configuration options.
### Data Model
The following rules are used when storing Vector metrics into GreptimeDB:
- Use `_` as the table name in GreptimeDB, for example, `host_cpu_seconds_total`;
- Use the timestamp of the metric as the time index of GreptimeDB, the column name is `ts`;
- Use the tags of the metric as GreptimeDB tags;
- For Vector metrics which have multiple subtypes:
- For Counter and Gauge metrics, the values are stored in the `val` column;
- For Set metrics, the number of data points are stored in the `val` column;
- For Distribution metrics, the values of each percentile are stored in the `pxx` column, where xx is the percentile, and the `min/max/avg/sum/count` columns are also stored;
- For AggregatedHistogram metrics, the values of each bucket are stored in the `bxx` column, where xx is the upper limit of the bucket, and the `sum/count` columns are also stored;
- For AggregatedSummary metrics, the values of each percentile are stored in the `pxx` column, where xx is the percentile, and the `sum/count` columns are also stored;
- For Sketch metrics, the values of each percentile are stored in the `pxx` column, where xx is the percentile, and the `min/max/avg/sum` columns are also stored;
### Using InfluxDB Line Protocol Format
You can use the `influx` sink to write metrics data. We recommend using v2 version of InfluxDB line protocol format.
Below is an example configuration using `influx` sink to write host metrics:
```toml
# sample.toml
[sources.my_source_id]
type = "internal_metrics"
[sinks.my_sink_id]
type = "influxdb_metrics"
inputs = [ "my_source_id" ]
bucket = "public"
endpoint = "http://:4000/v1/influxdb"
org = ""
token = ""
```
The above configuration uses v2 version of InfluxDB line protocol. Vector determines the InfluxDB protocol version based on fields in the TOML configuration, so please ensure the configuration contains `bucket`, `org`, and `token` fields. Specific field explanations:
- `type`: Value for InfluxDB line protocol is `influxdb_metrics`.
- `bucket`: Database name in GreptimeDB.
- `org`: Organization name in GreptimeDB (needs to be empty).
- `token`: Token for authentication (needs to be empty). Since Influx line protocol token has special format and must start with `Token `, this differs from GreptimeDB's authentication method and is currently not compatible. If using GreptimeDB instance with authentication, please use `greptimedb_metrics`.
For more details, please refer to [InfluxDB Line Protocol documentation](../for-iot/influxdb-line-protocol.md) to learn how to write data to GreptimeDB using InfluxDB Line Protocol.
### Using Prometheus Remote Write Protocol
Below is an example configuration using Prometheus Remote Write protocol to write host metrics:
```toml
# sample.toml
[sources.my_source_id]
type = "internal_metrics"
[sinks.prometheus_remote_write]
type = "prometheus_remote_write"
inputs = [ "my_source_id" ]
endpoint = "http://:4000/v1/prometheus/write?db="
compression = "snappy"
auth = { strategy = "basic", username = "", password = "" }
```
## Writing Logs Data
GreptimeDB supports multiple ways to write logs data:
- Using [`greptimedb_logs` sink](https://vector.dev/docs/reference/configuration/sinks/greptimedb_logs/) to write logs data to GreptimeDB.
- Using Loki protocol to write logs data to GreptimeDB.
We strongly recommend all users to use `greptimedb_logs` sink to write logs data, as it is optimized for GreptimeDB and better supports GreptimeDB features. We also recommend enabling compression for various protocols to improve data transmission efficiency.
### Using `greptimedb_logs` sink (recommended)
```toml
# sample.toml
[sources.my_source_id]
type = "demo_logs"
count = 10
format = "apache_common"
interval = 1
[sinks.my_sink_id]
type = "greptimedb_logs"
inputs = [ "my_source_id" ]
compression = "gzip"
dbname = "public"
endpoint = "http://:4000"
extra_headers = { "x-greptime-pipeline-params" = "max_nested_levels=10" }
pipeline_name = "greptime_identity"
table = ""
username = ""
password = ""
[sinks.my_sink_id.extra_params]
source = "vector"
skip_error = "true"
```
This example demonstrates how to use `greptimedb_logs` sink to write generated demo logs data to GreptimeDB. For more information, please refer to [Vector greptimedb_logs sink](https://vector.dev/docs/reference/configuration/sinks/greptimedb_logs/) documentation.
### Using Loki Protocol
#### Example
```toml
[sources.generate_syslog]
type = "demo_logs"
format = "syslog"
count = 100
interval = 1
[transforms.remap_syslog]
inputs = ["generate_syslog"]
type = "remap"
source = """
.labels = {
"host": .host,
"service": .service,
}
.structured_metadata = {
"source_type": .source_type
}
"""
[sinks.my_sink_id]
type = "loki"
inputs = ["remap_syslog"]
compression = "snappy"
endpoint = "http://:4000"
out_of_order_action = "accept"
path = "/v1/loki/api/v1/push"
encoding = { codec = "raw_message" }
labels = { "*" = "{{labels}}" }
structured_metadata = { "*" = "{{structured_metadata}}" }
auth = {strategy = "basic", user = "", password = ""}
```
The above configuration writes logs data to GreptimeDB using Loki protocol. Specific configuration item explanations:
- `compression`: Sets compression algorithm for data transmission, using `snappy` here.
- `endpoint`: Specifies Loki's receiving address.
- `out_of_order_action`: Sets how to handle out-of-order logs, choosing `accept` here to accept out-of-order logs. GreptimeDB supports writing out-of-order logs.
- `path`: Specifies Loki's API path.
- `encoding`: Sets data encoding method, using `raw_message` here.
- `labels`: Specifies log labels, mapping `labels` content to `{{labels}}` here. That is the `labels` field in remap_syslog.
- `structured_metadata`: Specifies structured metadata, mapping `structured_metadata` content to `{{structured_metadata}}` here. That is the `structured_metadata` field in remap_syslog.
For meanings of `labels` and `structured_metadata`, please refer to [Loki documentation](https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/).
For Loki protocol, `labels` will use Tag type in time series scenarios by default, please avoid using high-cardinality fields for these fields. `structured_metadata` will be stored as a complete JSON field.
Note that since Vector's Loki sink configuration doesn't allow setting headers, you cannot specify a custom GreptimeDB log table or pipeline through `X-Greptime-*` headers. Logs are written to GreptimeDB's default Loki table, `loki_logs`. If you need to specify a table or use pipeline functionality, please consider using the `greptimedb_logs` sink.
---
## Ingest Data
GreptimeDB accepts data through established observability and database protocols, so most sources write to it without a custom client. This page maps each source to the protocol it writes with and the guide that covers it.