discord9
34761265d5
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Rebased onto `narthex` at `bd6be2cf8f83b04280d3113467563ae81005e2f3`, including merged #9868, #9937, #9914, #9870, #9871 and #9896. This PR no longer depends on the old #9896 feature branch. - Builds on the existing native Complete aggregate runtime from #9873. #9936 can be restacked after this PR merges. ## What's changed and what's your intention? Connect Complete aggregation to the current experimental native PromQL execution path while preserving actual partition-local evaluation: ```text SeriesScanExec N → FusedSeriesExec N → complete partial accumulator states N → CoalescePartitionsExec → FloatMerger → AggregateFinalizeExec 1 → ordinary RecordBatch ``` ### Execution and current API adaptations - Keep #9896's `FusedSeries.children`, physical children, `BoundInput.source_index`, and existing ordinary multi-input execution. An optional `partial_fold` changes only the output mode: ordinary executions still emit query rows; aggregate producers fold evaluated chunks into complete accumulator state and export it only after successful input EOF. The aggregate path currently has one actual scan child, with a separate evaluator for each of its N partitions; no gather is inserted before evaluation. - Split at the innermost supported Complete aggregate. The existing `FloatMerger` combines full state, not finalized partition answers: average retains count and compensated sum, and variance retains count, mean and M2. The finalizer waits for all producer input to end successfully before running the global suffix or returning rows. A late source error fails the query rather than returning an incomplete aggregate. - Build the suffix as a separate `NativeTree`. Its `source_index = 0` names its own evaluated input, not the original scan. Obtain the actual input from `NativeExecutionState::input_ids(0)` and call `NativeExecutor::push_evaluated(InputId, chunk)`; do not equate a source index with a runtime input ID. No `NodePath` or flat executor is restored. - Use current `SeriesLabels` and the raw field layout from Native Input output samples, without restoring a separate scan-schema mirror. Reuse the existing row converter for final output. State and output reservations are execution-local and released on errors and stream drop, including when a physical plan is retained for another execution. - Preserve `sum`, `avg`, `count`, `min`, `max`, `group`, `stddev` and `stdvar`, `by` / `without`, maps and nested aggregate suffixes. Literal scalar broadcasts work below the cut (`sum(8 * rate(...))`, `max(10 - rate(...))`) and above it (`100 - avg(...)`, `sum(...) * 2`, `100 / sum(...)`, `-sum(...)`), keeping subtraction and division operand order. - Compose the current selector matcher/time-bound filters with aggregate planning. Add `sum(rate(counter{job="a"}[5m]))` to the existing standalone fixture; the independently expected answer is 9. **The private state-row `RecordBatch` codec remains a temporary bridge.** Producer, codec and finalizer retain the explanation that future native final aggregation should consume series frames directly, eliminating the intermediate state-row encoding, decoding and reconstruction. This PR does not implement that replacement. ### Scope and compatibility The supported target is standalone, one Region, multiple nonempty scan partitions, with in-process region transport and the existing engine/per-query native opt-ins. Ordinary Fused input traversal is not redesigned. There is no cross-Region series merge, generic Exchange, multi-input Aggregate Join, distributed Partial/Final architecture, new public configuration, persisted format or wire format. No performance or CPU-overlap improvement is claimed. Unsupported aggregate topologies remain outside this PoC. ### Verification All commands below passed on exact head `9720e1bf88a776bd017b304152a055e1735d4ba1`, over the narthex base above, using `RUST_MIN_STACK=8388608`, a dedicated external Cargo target, and no test retries: - `cargo fmt -p promql -p query -p tests-integration -- --check` - `cargo check --locked -p query -p promql --all-targets` - `cargo check --locked -p tests-integration --all-targets` - `cargo nextest run --locked -p query -p promql --no-fail-fast --retries 0` — **1967 passed, 2 skipped**. - `cargo nextest run --locked -p tests-integration --lib --no-fail-fast --retries 0 -E 'test(native_aggregate_complete_over_four_nonempty_mito_partitions) | test(native_promql_)'` — **16 passed, 251 skipped**. - `cargo clippy --locked -p query -p promql --all-targets --all-features -- -D warnings` - `cargo clippy --locked -p tests-integration --all-targets --all-features -- -D warnings` - `git diff --check bd6be2cf8f83b04280d3113467563ae81005e2f3..HEAD` - `hawkeye format --dry-run` and `hawkeye check` The standalone aggregate case checks typed physical-plan structure as well as results: `SeriesScanExec` and partial Fused each retain four partitions, with no intervening exchange; the finalizer consumes the producer's state schema and answers one partition. For unfiltered counter cases all four evaluators must report positive contributions, totaling five and at most two per partition, forcing the three-member group across partitions. Results are checked for labels, timestamps, columns, values and multiplicity against independent expectations, in addition to legacy execution. The five nonzero counter rates are 1/3/5/2/4: total sum 15; per-job sums 9/6, means 3/3, counts 3/2 and standard deviations sqrt(8/3)/1. Scalar cases include `sum(8 * rate(...)) = 120`, `max(10 - rate(...)) = 9`, `100 - avg(...) = 97`, `100 / sum(...) = 100/15` and unary minus = -15. The subtraction prefix distinguishes an incorrectly moved suffix (5) or reversed operands (-5). The matcher selects three series, so only that case does not require four nonempty partitions; all unfiltered assertions remain intact. Unit coverage includes all eight accumulator kinds, unequal producer counts (one value versus two, so the merged mean is 6 rather than the mean-of-means 5), nested/map/scalar suffixes, empty input, producer read barriers, late errors, and reservation/drop boundaries. Local full-workspace tests, an HTTP smoke run and native SQLness have not been run; the SQLness runner does not carry the per-query native hint. Final-head CI for `9720e1bf88a776bd017b304152a055e1735d4ba1` has completed: [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361500), [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361419), [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361402) all succeeded. The final check rollup contains 40 successful checks and 2 skipped checks, with no pending or failed checks. An earlier Semantic Pull Request run was cancelled and superseded by a successful run; the current Semantic PR title check passed. No workflow reruns were requested. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`).
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.