mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-13 22:36:57 +00:00
discord9
3fd69e312e
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Related: #9608, which builds on this PR's branch and adds the engine-side raw-series producer; the contract reviewed here is this PR's own diff. - Base branch: `narthex`, the shared integration branch rather than `main` (it already contains the merged Narthex PRs #9598, #9599, and #9602). ## What's changed and what's your intention? Define the raw-series read contract shared by storage engines and their consumers, and separate "which series are read" from "how a reader streams them". The contract lands on its own so its vocabulary is reviewed once; the engine-side producer arrives in a follow-up PR. ### Raw series values and streams - `RawSeriesChunk` is one raw source batch: the complete encoded `primary_key` (not a TSID or hash alone), an optional `metric`, a strictly increasing `time` axis, fields aligned to that axis, and `is_last`. `try_new` rejects an evaluation grid, mismatched field count or length, and an empty time axis; validation is per chunk and does not claim ordering between chunks. - `RawTimestamps` keeps its buffer private, so "strictly increasing, no duplicates" is validated once at construction and cannot be bypassed by a struct literal (pinned by a `compile_fail` doctest). Slices are zero-copy and keep the underlying buffer alive. - `RawSeriesStream` is the reader side: a `Stream` of `Result<RawSeriesChunk, BoxedError>` items exposing `schema`, `label_source`, and `ordering`. `SendableRawSeriesStream` is its pinned boxed form. - `RawLabelSource` carries the binding: `PrimaryKey` means labels are encoded in the complete primary key and decoded with scan-bound metadata; `Explicit` means the stream supplies the complete name/label set itself, never a partial projection. Chunks under `PrimaryKey` carry `metric: None` because labels are undecoded, not because the metric name is missing, while `Some(metric)` is always complete. - `PreparedSeriesSource` is the selection side and is deliberately not a reader: `num_series()`, `series_info(ordinal)`, `label_source()`, and `read_series(range)`. Both it and `RawSeriesStream` return the same `label_source()`, so a holder can interpret labels before polling anything. ### Shared ordering vocabulary - `datatypes::series::SeriesOrdering` is a common vocabulary that raw and evaluated series can express; in this PR the only user is `RawSeriesStream::ordering`, and no evaluated consumer or implementation is added. - Modes: `Interleaved` (a series may resume after another series), `Contiguous` (each series' chunks are contiguous), and `KeyThenTime` (this stream's series identity, then position). Key order uses the complete bound-compatible identity, not a TSID or hash alone. - All modes are per-stream guarantees only: positions strictly increase within a series, including across chunks, with no duplicates, and there is no query-global ordering or finality claim. `is_last` marks the last chunk of that series in this one stream; merge code must recompute finalization across all related inputs. ### Prepared series selection - A source describes a finite, stable selected set: `num_series()` is exact, `series_info(ordinal)` returns identity metadata without polling samples, `label_source()` is available without opening a reader, and `read_series(range)` covers a half-open ordinal range. - Ordinals are stable local indexes, not protocol identifiers, and each ordinal's identity matches the chunks produced for it. Readers are independent: repeated or overlapping reads do not consume shared iteration state, dropping one reader does not cancel siblings, and a reader owns what it needs to continue after the source is dropped. - Invalid ordinals and ranges return errors; an empty in-bounds range is an ordinary empty stream. `SeriesInfo::metric` follows chunk semantics (`None` = primary-key label binding, `Some` = complete). - The documented requirements — no sample payload loading or hidden blocking I/O during preparation, identity lookup, or reader creation, and binding to the partition's immutable MVCC view, predicates, authorization, projection, and labels — are requirements for future implementations. - This PR adds no engine implementation, so it makes no I/O, performance, or MVCC behavior claim. - The store-api tests include a small in-memory `PreparedSeriesSource` that pins these guarantees: binding available before readers, empty ranges, invalid ordinals and ranges, independent overlapping readers across source and reader drops, and a selection that does not change when a later mutable selection does. ### Scanner entrypoints - `RegionScanner::prepare_series_partition` is the optional capability and defaults to `Ok(None)`, meaning "unsupported before execution, so the caller may use the legacy record-batch scan". No scanner overrides it in this PR, and the only `PreparedSeriesSource` implementation here is the store-api test fixture. - `RegionScanner::scan_series_partition` keeps its existing default panic (`todo!()`), now documented as must-override; a store-api test pins that panic and shows the legacy `scan_partition` result is unchanged. - Callers must use exactly one entrypoint per scanner partition and execution, and must not replay a failed raw scan through the legacy path. ### Compatibility The legacy record-batch scan and existing queries are unchanged, and nothing in planning or execution calls the new entrypoints yet. No persisted format, wire protocol, storage format, or configuration change is made: the added types are in-memory Rust types with no serialized representation. Within this stack the raw-series types are unreleased, and the branch replaces the interim raw-only ordering enum with the shared `SeriesOrdering` plus the source-level label binding. For released code the change is additive: new trait methods with default implementations, and no released method signature changes. Note for reviewers: the malformed-range case is built with `std::ops::Range { start: 2, end: 1 }` rather than a reversed range literal, because `clippy::reversed_empty_ranges` rejects the literal form; the assertion itself is unchanged and no lint allow is added. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Verification Local checks below ran on the exact source now pushed to this PR, including the test-only range fix noted above. - `cargo +1.98.1 check --locked -p datatypes -p store-api --all-targets` — passed. - `cargo nextest run --locked -p datatypes -p store-api --retries 0 --no-fail-fast` — 432 passed, 0 skipped. - `cargo test --locked -p datatypes --doc RawTimestamps` — 1 passed. - `cargo clippy --locked -p datatypes -p store-api --all-targets --all-features -- -D warnings` — passed on the pinned 1.98.1 toolchain, covering the clippy lint that the fixed test file previously tripped. - Exact-head remote CI for `794389688084ee97a7760d1ae993bcf1c109489e` completed successfully: [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322418) (including workspace Clippy, Linux tests, Windows build/tests and riscv64 build), [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322439), and [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322830) (SQL, SDK, import/export and fuzz jobs). The workflow marked coverage and recent-release compatibility jobs skipped; those two jobs are not claimed as executed.
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.