mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-13 22:36:57 +00:00
discord9
7e0f6e29aa
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Depends on: #9603 (`feat/narthex-raw-contract`). - Base of: #9611 (`feat/narthex-raw-frame-stream`). - The finite raw SeriesFrame codec remains independent of the Mito producer in #9608. ## What's changed and what's your intention? Adds an Arrow `RecordBatch` representation for the raw series chunks defined in the stack, so subsequent execution and transport code can exchange typed samples without rebuilding every sample, plus the bounded multirow encoder that packs raw series chunks into multi-row batches directly. **Codec.** The new `series::frame` module provides raw-frame schema construction, chunk packing and whole-batch unpacking, in encoded-key and explicit-label modes, preserving explicit timestamps, float/histogram/mixed payloads, nulls, field order and `is_last`. Different fields may have different chunk boundaries: packing splits at their union and retains Arrow views rather than densifying the inputs. `Samples::segments()` exposes the zero-copy dense segment view the codec walks, with a public-call doctest, so the codec can consume existing dense segments without a new segment type or allocation. Decode validates schema metadata, payload lengths, indices, histogram structure and timestamps, and any malformed row fails the entire unpacking call rather than returning a successful prefix. **Bounded multirow encoding.** `RawSeriesFrameBuilder` buffers rows and emits one batch when either bound is reached (`DEFAULT_MAX_ROWS` 256, `DEFAULT_MAX_BYTES` 1 MiB); the effective byte cap is stored as `min(requested, i32::MAX)`, which keeps every cumulative Arrow offset domain representable for any admitted group. `RawSeriesChunkCursor` walks one already validated chunk at the union of the field segment boundaries, holds at most one pending measured row, and measures each row exactly once; the pending row is retried after a flush, so no row is dropped or split, and per-field offsets advance against each field's original segment length, which keeps nonzero-offset (sliced) sample arrays correct. The byte measure dominates the encoder's offset domains: key and timestamp bytes, list element counts and bitmap bytes, and the visible extent of each histogram table list column read from the rebased `value_offsets` view rather than from retained capacity. It is an internal batching work estimate — not an exact IPC size and not a resident-memory or RSS cap — so one individually legal row that exceeds the requested cap is still accepted and emitted alone, because that slice cannot be split into smaller legal rows. **Histogram layout and sharing.** Row-local histogram indices are never remapped; physical tables are concatenated under cumulative row list offsets, so each row still addresses its own table through a nonzero slice of the child. A row group with a single Arrow constituent reuses it by clone, which shares buffers and keeps the single-row zero-copy and owner-drop behavior; a group with several constituents is concatenated, which copies. Both cases are asserted. The `Samples::into_dense` sharing check compares the Arrow table view (`ArrayData` identity) in addition to `Arc` identity, and the first histogram batch's `ArrayData` is computed lazily and cached: that is pointer-identity caching, not content deduplication, and distinct buffers with equal content are never merged. Local buffer sharing does not prevent a full table from being encoded in every batch; any future trimming or remapping belongs at the actual IPC/network/spill boundary, reusing Arrow slicing and selection and the existing writer, and preserving duplicate indices without content hashing. **Preserved interfaces.** `pack_raw_series_chunk` keeps its `Vec<RecordBatch>` return type and `unpack_raw_series_frame` keeps its signature; `RawSeriesFrameDecoder` prepares the expected frame schema once for a stream of batches, while the wrapper still builds a decoder per call. The raw contract's `RawTimestamps` validates strictly increasing source timestamps once at construction behind a private buffer. The frozen `v1` contract is untouched: the four reserved metadata entries, the column names (`read_key`, `is_last`, `time`, `fields`, plus `name`/`labels` in explicit mode) and the wire column literal `read_key` stay as they were (`primary_key` is only the Rust-side name of that buffer). Histogram-only rows decode through `Samples::try_from_histogram` directly, without a duplicate bounds check. **Tests.** The frame tests cover schema and metadata acceptance and rejection, bit-exact payloads (including NaN payloads and `-0.0`), sliced-payload owner lifetimes, row and byte bounds (bounds taken from real measured rows, flush-before-exceed with pending retry, legal oversized row), row-local histogram indices and complete tables with a real Arrow IPC round trip, decoder schema caching and label-mode separation. The `samples` tests pin both sides of the sharing rule: distinct wrappers of one Arrow table view are reused, while distinct buffers, distinct table slices, and shared value buffers with different null-bitmap offsets are never merged. **Limitations and boundaries.** Each histogram-bearing row carries its complete histogram table, including entries no index references, and a large table referenced by several segments can be repeated per row; there is no deduplication and no global reindex, so an encoded batch can exceed the source sample bytes. These entry points are exercised by unit tests, and the builder, cursor and decoder are also used by the in-process stream adapter of #9611, which is not yet wired into a production query path; there is no transport, persistence or query-performance change. This PR handles finite raw frames only: it does not add stream-wide key/order/EOF validation (that is #9611), evaluated frames, a Flight/IPC execution path, logical-name authentication, a native query consumer, a production query-path switch, distributed or cross-region merging, or a prepared producer. The frame preserves ordered logical field names and rejects mismatched supplied schemas; that is not authentication of label binding or data provenance. Existing persisted storage formats and query entrypoints are unchanged, and this is not a cross-version wire-compatibility claim. Open and unmerged; current-head CI has completed successfully. ### Bounded local Frame-boundary measurement (B baseline single-row, C selected multirow) Provenance: measured at the frozen local candidate `eddd595b82888452b3c066678fabc6c1200c74f7`, whose bounded-multirow finite codec content is `3406a5334574a518924c67c319e2d92b6cac943b`, against the old public baseline `36843e8cebeded1328dd7005ed668300a606f182`. The later normal parent synchronization and the strict-lint syntax-only fix are not separately rebenchmarked, so these numbers describe that frozen local content, not the pushed head; no remote CI or performance result is implied by this local measurement. This PR now implements the selected C direction and native A is unchanged: the native direct path remains the control and is not replaced. A bounded local-only experiment measured this codec's role as the pack/unpack kernel of that direction. Fixture: six synthetic raw-series Frame-boundary cases in two sequential campaigns (100000 series in the first, 1000 in the second, one chunk per series in both), CPU affinity 2, seven repetitions, default warmup, plain-timing and allocator-instrumented runs kept separate, no retries and no excluded rounds. Stage wall time from the plain runner, milliseconds, median only, for the six cases in table order: | Case | Series / rows per series | A0 native, baseline build | A1 native, candidate build | B baseline single-row | C selected multirow | |---|---:|---:|---:|---:|---:| | float2_key16 | 100000 / 2 | 42.313 | 43.520 | 2549.316 | 206.888 | | float3_key16 | 100000 / 3 | 59.434 | 60.111 | 3261.311 | 260.933 | | float3_key128 | 100000 / 3 | 64.968 | 62.999 | 3240.701 | 291.846 | | long_float1024_key16 | 1000 / 1024 | 7.556 | 7.113 | 26.311 | 10.468 | | histogram3_key16 | 1000 / 3 | 0.452 | 0.448 | 17.250 | 5.632 | | multifield_float_hist3_key16 | 1000 / 3 | 0.593 | 0.588 | 25.194 | 6.328 | The same run's full min–max, MAD, all seven rounds per role, allocation counters, RSS, frame counts and acceptance checks are in #9611, which carries the canonical full results — including the frame totals counted from the runs (B emitted 100000 / 100000 / 100000 / 1000 / 1000 / 1000 batches and C 391 / 391 / 391 / 17 / 4 / 4, with frame rows equal to series × chunks in every case and no frames on the native path). Reading: the A0 and A1 native control ranges overlap in every case, so the native controls establish no candidate-build difference, and the C/B range disjointness in this bounded seven-round sample is descriptive for these synthetic fixtures only. Relative to B, C lowers stage wall time and allocation request counts while raising phase peak-live bytes and whole-process RSS; that batching-versus-retained-memory tradeoff of the selected direction is accepted once, as part of that direction, and is not a veto or an open design question (largest observed increase: the long-float case, peak live Δ 2156848 B versus 50438 B and lifetime RSS 9176 kB versus 6080 kB). Scope limits, as in #9611: every SOURCE_KEY group uses one identical explicit metric label set, so this is not real PromQL label diversity or cardinality evidence; the histogram table template is built outside the measured phase; no `Samples::Mixed` inside one field. The measured scope is local Frame generation, the codec boundary, digests and phase-owned cleanup — not a Flight/IPC network path, not a whole-query or query-regression benchmark, not a storage or service measurement, and not a CI result. No query speedup is claimed from this codec or from the selected direction. ### Verification Pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`, synchronized normally with #9603 at `794389688084ee97a7760d1ae993bcf1c109489e`. Checks on the exact frozen source recorded by this head: - `cargo +1.98.1 check --locked -p datatypes --all-targets` — passed. - `cargo +1.98.1 clippy --locked -p datatypes --all-targets --all-features -- -D warnings` — passed with the official pinned 1.98.1 Clippy component. - `cargo +1.98.1 nextest run --locked -p datatypes --retries 0 --no-fail-fast` — **363 passed, 0 skipped**. - `cargo +1.98.1 test --locked -p datatypes --doc series::samples::Samples::segments` — **1 passed**. - Rust 1.99 formatting, TOML/license/typo checks, `git diff --check`, and normal signed-off commit/push hooks passed. Source hashes were verified unchanged after the hooks. The final closeout only collapses the full-builder flush guard into a short-circuit-preserving let-chain for strict Clippy; it changes no batching behavior or assertion. The local measurements above remain attributed to their frozen candidate. Current-head remote CI completed successfully in all three workflows below. The PR remains open and unmerged until the normal merge gates and parent-first integration are checked. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Verified current-head remote CI Exact pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`. - [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922544) — **success**. - [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922504) — **success**. - [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922665) — **success**. Linux and Windows test jobs, strict Clippy, builds, SQLness and the executed integration/fuzz jobs succeeded. Coverage and recent-release compatibility jobs were skipped by their workflow conditions; they are not claimed as executed. These CI results are independent of the frozen local experiment provenance above.
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.