mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
1cab5f1624
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - **Stacked on #9942** (PR A, the unified SeriesFrame codec). - GitHub cannot target a fork branch, so this PR's base reads `narthex`. **Review #9942 first.** - The commits unique to this PR are `b32573f9cbf`, `16b3e428951`, `d5547c55bdf` and `f2041d8d9de` (test-only lint cleanup). The chain is `narthex (771347df5d) → #9942 (8bc53c4871) → this PR (CLEANUP_SHA)`. - Merge order remains **#9942 → #9943**. Once #9942 lands, rebase this PR onto the resulting `narthex`; do not merge the codec changes twice. - Spec: Narthex Unified SeriesFrame refactor, PR B of two. ## What's changed and what's your intention? The fused native node could only answer one way: a row table with the grid timestamps, one column per series field and one per declared tag. That is what a query returns, but it is not what a consumer of the native path needs when it has to tell series and positions apart — the labels of a series, and which grid positions it actually holds, are both flattened away by the row shape. This adds a second output format, **fixed when the plan is built** rather than chosen at run time: ```rust pub enum FusedOutputFormat { PlainRows, SeriesFrame } ``` | | `PlainRows` (default) | `SeriesFrame` | | --- | --- | --- | | output schema | the existing row table (time / fields / tags) | the codec's declared evaluated frame | | encoding | `series_to_record_batch` | straight from `EvaluatedChunk` via `SeriesFrameCursor` | | reachable from a query | yes | no | | `EXPLAIN` | byte-identical | appends ` (output: series_frame)` | The format is part of the plan because it decides the node's output schema, so it is part of the node's identity (`PartialEq`/`Hash`/`PartialOrd`), its `EXPLAIN` text, and its rebuild over a new child. A format error is refused while the plan is built, not when the first poll happens. The `SeriesFrame` schema is derived from the **`NativeTree.root().output` evaluated `SeriesSchema`**, never from the original scan schema or the input tags: after a label rewrite or an aggregate, the result's labels and fields are what count. ### Independent chains remain independent `is_last` closes the current chain, not the entire label set. Separate chains may therefore carry identical complete labels and each close independently. This PR adds no plan-owned global label identity management; combining contributions across partitions remains the consumer's responsibility. Tests check exact positions, payload bits and separate finality for both consecutive same-label chains in one encoder and chains in separate partitions. ### Parallelism is unchanged For the supported single-source Frame path, output partition count equals the input scan's partition count. No `CoalescePartitionsExec`, `RepartitionExec` or round-robin is introduced in either format. `series_frame_preserves_four_input_partitions` checks the in-memory boundary. The standalone integration test additionally captures a real single-region `SeriesScanExec` at `parallelism=4` and executes every partition in each output format. ### Implementation notes worth knowing 1. **Stream and batch schemas agree.** The codec-derived output schema includes the explicit `contiguous` ordering declaration. Published batches are checked against that declared schema. 2. **The end of the execution must flush.** The frame builder buffers up to a batch across chunks, so missing the final flush loses the tail rows. 3. **`Poll::Pending` also flushes.** Otherwise a closing row stays in the builder behind an input that may never become ready — the same convention PR A's `PackStream` already follows. ### Key code locations - `src/promql/src/extension_plan.rs`: exports `FusedOutputFormat`. - `src/promql/src/extension_plan/fused_series.rs`: default/explicit format constructors, root-output schema derivation, plan identity/rebuild/EXPLAIN, direct chunk encoding and stream flushing. - `src/promql/src/extension_plan/fused_series/tests.rs` and `tests/frame_output.rs`: internal logical/physical execution and lifecycle/semantic regressions. - `tests-integration/tests/http.rs`: real standalone scan replay in both output formats. ### Review fixes and acceptance - **Batch-level backpressure:** the stream retains the existing `SeriesFrameCursor` and advances only until one batch is ready, then returns control to the consumer. It no longer encodes an entire evaluated chunk into a `ready` queue before returning the first batch. Pending native output is drained before more upstream input; a pre-EOF stall does not close the executor. PlainRows and partial-state conversion keep their original timing. - **Real scan acceptance:** the standalone File integration fixture has one physical region and `parallelism=4`. The test captures its actual `SeriesScanExec`, replays the internal fused plan in both formats, executes all four nonempty partitions, and checks the exact 24 expected rows, eight Frame chain closings and per-partition native output metrics. Frame is not exposed as a user query's final result to perform this test. - Replaced the vacuous sparse fixture with exact nonempty positions `0,2,5`, two fields with partial NULLs and preserved ordinary-NaN bits, and a PlainRows comparison. Added fixed-value `abs(rate(...))` and label-rewrite comparisons, including a new label outside the input tag projection and two independent same-label chains in one encoder. Cross-partition chains also check exact positions and value bits. - Added retained-cursor/one-batch checks, buffered upstream-error termination, and early-drop ownership checks. Reintroducing eager encoding compiles but makes the backpressure regression fail at its queue assertion; restoring the reviewed source makes it pass. - Removed the redundant physical scalar/Frame guard; the shared root-output validator remains. - Independent correctness and anti-over-design review: **APPROVE** for `8bc53c48714..d5547c55bdf`. Review approval is separate from new-head CI. ### Limitations - `SeriesFrame` is **not reachable from a user query** yet (production defaults to `PlainRows`). Per spec §8.7 the acceptance executes the internal physical plan and decodes its output, instead of exposing the frame as a query's final result format. - No sqlness result changes: the production planner still constructs `PlainRows`. The existing standalone integration case is extended to execute and decode the internal Frame plan. - Rebased onto `narthex` at `771347df5d9`, including #9936, #9882 and #9886. The native zero-source scalar, `timestamp()`/instant-`@`, multi-input and partial-state paths are preserved. A Frame over a scalar root is rejected because this format requires evaluated samples. - The multi-source case is refused for frames rather than implemented: this node still answers only partition 0 when it has more than one raw source, so a frame over several sources would have to change that first. - Consumer contract: a sparse chunk decodes into **several** `Series` (one frame row per grid run), so a consumer must concatenate by position and cannot assume "one chunk → one decoded result". Nothing consumes frames in this PR, but Complete Aggregate's SeriesFrame input must respect it. - Not verified: Windows; the results below are Linux. ### Does this break API or data compatibility? `PlainRows` remains the default and the only format a user query reaches. This increment leaves its conversion and plan display paths in place. The added tests compare exact Frame semantics without changing the user-facing result format. No persisted format is touched. ### Tests Coverage includes codec-derived schema, logical rebuild and EXPLAIN, dense and genuinely sparse output, multi-field NULL/NaN values, composed math and label rewrites, four in-memory and four real scan partitions, independent same-label chains, partial-state schema, constructor refusals, zero-sample closing, Pending/EOF flush, buffered error, cancellation and incremental encoding. <!-- CURRENT_VERIFICATION_START --> Main verification on `d5547c55bdf` (Linux; default features): - `make fmt-check`: passed. - `cargo nextest run -p datatypes -p common-query -p table -p promql -p query --retries 0`: **2698 passed, 2 skipped**. - `cargo nextest run -p tests-integration --test main -E 'test(test_native_promql_route_plans_the_query) & test(file)' --retries 0 --success-output immediate`: **1 passed, 771 skipped**. Both formats produced per-partition decoded row counts **[15, 3, 3, 3]**; exact 24-row results and eight Frame chain closings were asserted. - `cargo check --workspace --all-targets`: passed. - The final strict Clippy pass found two test-only hygiene issues in the new tests: an unnecessary `f64` cast and two `println!` calls that the workspace's `print_stdout` lint rejects. The follow-up types the existing float literal explicitly, removes the cast, and drops the diagnostics, whose values were already asserted by the surrounding exact checks. It changes no production code or assertions. - Final test-only cleanup `f2041d8d9de` on top of the reviewed `d5547c55bdf`: `make fmt-check` passed; `cargo clippy -p promql -p query -p tests-integration --all-targets -- -D warnings` passed; the affected `abs_rate_frame_matches_plain_rows_with_exact_values_and_labels` (1 passed) and real File scan replay (1 passed, 771 skipped) passed again. The full five-crate suite above was run at `d5547c55bdf`; this last commit touches tests only and was not followed by another full-suite run. Other focused evidence: the eager-encoding mutation compiled and then failed the queue-boundary assertion; restored production source passed. License/TOML/diff checks passed. These are local Linux results, not a claim of Windows or new-head CI success; full all-feature workspace Clippy and the full workspace test suite were not run. <!-- CURRENT_VERIFICATION_END --> ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`).
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.