Fluent Bit

Fluent Bit is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.

You can forward Fluent Bit data to GreptimeDB. This document describes how to configure Fluent Bit to send logs, metrics, and traces to GreptimeDB.

HTTP

Using Fluent Bit's HTTP Output Plugin, you can send logs to GreptimeDB. Before configuring Fluent Bit, ensure that you understand the log ingestion flow and how to use pipelines.

[OUTPUT]
    Name             http
    Match            *
    Host             greptimedb
    Port             4000
    Uri              /v1/ingest?db=public&table=your_table&pipeline_name=greptime_identity
    Format           json
    Json_date_key    scrape_timestamp
    Json_date_format iso8601
    compress         gzip
    http_User        <username>
    http_Passwd      <password>

• host: GreptimeDB host address, e.g., localhost.
• port: GreptimeDB port, default is 4000.
• uri: The endpoint to send logs to.
• format: The format of the logs, needs to be json.
• json_date_key: The key in the JSON object that contains the timestamp.
• json_date_format: The format of the timestamp.
• compress: The compression method to use, e.g., gzip.
• header: The header to send with the request, e.g., Authorization for authentication.
• http_user and http_passwd: The authentication credentials for GreptimeDB.

In params Uri,

• db is the database name you want to write logs to.
• table is the table name you want to write logs to.
• pipeline_name is the pipeline name you want to use for processing logs.
• custom_time_index is optional. Use it when a field in the input data should become the GreptimeDB time index. The supported formats are <field_name>;epoch;<resolution> and <field_name>;datestr;<format>.

The greptime_identity pipeline creates the table directly from the JSON fields. If you want a field such as scrape_timestamp to become the GreptimeDB time index instead of a regular column, set custom_time_index in the request URI or use a custom pipeline to parse and map that field.

OpenTelemetry

GreptimeDB can also be configured as OpenTelemetry collector. Using Fluent Bit's OpenTelemetry Output Plugin, you can send metrics, logs, and traces to GreptimeDB.

[OUTPUT]
    Name                 opentelemetry
    Match                *
    Host                 127.0.0.1
    Port                 4000
    Metrics_uri          /v1/otlp/v1/metrics
    Logs_uri             /v1/otlp/v1/logs
    Traces_uri           /v1/otlp/v1/traces
    Log_response_payload True
    Tls                  Off
    Tls.verify           Off
    logs_body_key message
    http_User <username>
    http_Passwd <password>

• Metrics_uri, Logs_uri, and Traces_uri: The endpoint to send metrics, logs, and traces to.
• http_user and http_passwd: The authentication credentials for GreptimeDB.

We recommend not writing metrics, logs, and traces to a single output simultaneously, as each has specific header options for specifying parameters. We suggest creating a separate OpenTelemetry output for metrics, logs, and traces. for example:

# Only for metrics
[OUTPUT]
    Name                 opentelemetry
    Alias                opentelemetry_metrics
    Match                *
    Host                 127.0.0.1
    Port                 4000
    Metrics_uri          /v1/otlp/v1/metrics
    Log_response_payload True
    Tls                  Off
    Tls.verify           Off

# Only for logs
[OUTPUT]
    Name                 opentelemetry
    Alias                opentelemetry_logs
    Match                *
    Host                 127.0.0.1
    Port                 4000
    Logs_uri             /v1/otlp/v1/logs
    Log_response_payload True
    Tls                  Off
    Tls.verify           Off
    Header X-Greptime-Log-Table-Name "<log_table_name>"
    Header X-Greptime-Log-Pipeline-Name "<pipeline_name>"
    Header X-Greptime-DB-Name "<dbname>"

In this example, the OpenTelemetry OTLP/HTTP API is used. For signal-specific headers and options, see the OpenTelemetry API sections for metrics, logs, and traces.

Prometheus Remote Write

Configure GreptimeDB as remote write target:

[OUTPUT]
    Name                 prometheus_remote_write
    Match                internal_metrics
    Host                 127.0.0.1
    Port                 4000
    Uri                  /v1/prometheus/write?db=<dbname>
    Tls                  Off
    http_user            <username>
    http_passwd          <password>

• Uri: The endpoint to send metrics to.
• http_user and http_passwd: The authentication credentials for GreptimeDB.

In params Uri,

• db is the database name you want to write metrics to.

For details on the data model transformation from Prometheus to GreptimeDB, refer to the Data Model section in the Prometheus Remote Write guide.