I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Pre-merge rehearsal for keeping `narthex` current with `main`. The proto pin below points at the branch of GreptimeTeam/greptime-proto#355. ## What's changed and what's your intention? **This is a merge rehearsal, opened as a draft on purpose.** It merges `main` (`62dad907d0f`) into `narthex` (`115d85b6699`), resolves all 31 conflicts, and lands on a tree that builds and passes the suites below — so that the next real back-merge is mechanical instead of a week of archaeology. Three known blockers are listed at the end; **do not merge before they are cleared**. The branch is `main` + narthex's own work: 293 files, +241,450/−2,699 relative to `main`. | Check | Result | |---|---| | `cargo check --workspace` | 0 errors | | `cargo check --workspace --all-targets` | 0 errors | | `cargo nextest run -p promql` | 1215/1215 passed | | `cargo nextest run -p mito2 -E 'test(/series_index/)'` | 185/185 passed | | `cargo nextest run -p mito2 -E 'test(/cache/)'` | 257/257 passed | Not run: full workspace test suite, sqlness (`tql-explain-analyze` needs re-recording), `tests-integration`, `tests/compatibility`. ### The conflicts that actually matter Six places where "take both sides" compiles but is wrong. Each is marked `TODO(narthex-merge)` in the tree. #### 1. `query.experimental_native_promql` is rejected by main's query-option snapshot — the native route silently disables main (#9494) added `QueryContext::query_option_snapshot`, which runs every `query.*` extension through `canonical_query_option_name` and fails on unknown keys. narthex's native gate reads `query.experimental_native_promql` from that same extension map, so after a naive merge **every native request is rejected before reaching `plan_pql`**. Adding the key to `RESERVED_EXTENSION_KEYS` instead is also wrong: that list makes the HTTP/gRPC hint layer *drop* the key, which silently falls back to the legacy planner. Resolution: register it in `src/session/src/query_options.rs::canonical_query_option_name`, so the snapshot accepts it and keeps it on the context. Review question: should this be a publicly settable option at all, or should the gate read a typed extension written by `apply_query_options` (which would also let `SET` work consistently)? #### 2. `cache.rs`: two independent redesigns of the same decoded-metadata cache - main: `SstDecodedMetaKey::new(file_id, PageIndexPolicy)` → `SstMetaLoadResult`, async `get_decoded_sst_meta`, `CacheSizeRegistration`, and it **deleted** the `PageRangeCache` / `PageFragmentKey` / `PageFragmentGroupKey` family in favour of `ParquetPageCache` + `RangeResultCache`. - narthex: `DecodedKey`/`DecodedValue` over `SstMetaKey`, `Cache::builder().weigher(shared_weight)`, and `CacheManager::page_cache: Option<Arc<PageRangeCache>>` consumed by `cache/columnar.rs`. Resolution: main's architecture is the baseline; narthex's columnar metadata gets its own `CacheManager::columnar_meta_cache` tier, funded by a quarter of `sst_meta_cache_size` with the decoded tier reduced from half to a quarter, so the total budget stays exactly what the user configured. The budget split is **gated on the format**: `enable_columnar_meta_cache` is only set when `experimental_series_index_format == Columnar`, so the default deployment keeps main's exact `1/2 + 1/2` split and never constructs the extra tier. A first cut of this merge reserved the quarter unconditionally, which would have halved main's decoded tier for everyone; that is fixed in the second commit of this PR. **Deferred cost: narthex's native payload byte cache is now disabled** (`columnar_payload()` returns `None`). main's `ParquetPageCache` only serves uniform-size pages through a `RangeReader` and cannot hold these 16 KiB payload blocks, and reviving a second page cache would double-count `greptime_mito_cache_bytes`. Two tests that assert warm-payload reuse are `#[ignore]`d. Unlike the budget split this one *does* change behaviour when the columnar format is on, and needs an explicit decision. #### 3. `series_index`: main's memtable-pin fix vs narthex's format selection main (#9588) rewrote `reconcile_series_indexes` around a memtable-free `ReconcileSnapshot` so that a `VersionRef` is never held across `.await` (holding it pins flushed memtables). narthex needs `version.options.ttl`, `version.metadata`, `series_index_version_control.formats` and `backend.reader(..).await` to pick a series-index format. Resolution: keep the snapshot invariant; clone `ttl` / `metadata` / `formats` out of it before any await, and introduce `build_series_index(store, region, metadata, bucket, entry, backend, purger)` — a signature neither side had. `build_samples` / `build_sample_windows` now take `&RegionMetadataRef` instead of `&VersionRef`. #### 4. `ScanInput` no longer carries a scan memory pool main (#9101) moved scan memory accounting to the request-scoped `QueryScanContext::memory_pool` and deleted `ScanRegion::scan_memory_pool`. narthex's columnar reader registers four `MemoryConsumer`s against the deleted field. Resolution: `SeriesScanExec::execute` now threads `context.memory_pool()` into its `QueryScanContext`, but the columnar path inside `read/series_scan/prepared.rs` still uses an unbounded pool, so **`scan_memory_limit` does not cover native columnar reads**. Note the trap: main's default is now 25% of system memory where the old pool was unlimited, so a naive "just reuse the old pool" fix would silently tighten it ~4x. #### 5. Sparse primary-key validation vs narthex's fixtures main enforces (in `fix!: enforce sparse primary key schema`) that a sparse primary key starts with `(__table_id, __tsid)` **and** that every PK column id exists in the schema. Five `promql::source::tests` fixtures satisfied neither. Resolution: the test helper registers `__table_id` / `__tsid` as columns and prefixes the key with them. narthex's production code never constructs a sparse PK, so this is fixture adaptation — but it does confirm that both branches extended the same persisted sparse key encoding independently, which is worth a second look in review. #### 6. `.github`: duplicated cherry-picks, plus two silent-coverage traps - The `metric-snapshot-s3-bench` family (6 files) is a **byte-identical cherry-pick of #9573 that missed main's #10157**; `scope-rules.toml`, `scope-cases.toml`, `github-script-tests/action.yml`, `heavy-ci.md` are a squashed backport (#10053) that is a strict subset of main. All resolved to main. `heavy-ci-policy.test.cjs` resolves to narthex (it adds 13 lines of passing cases). - `integration.yml` resolved **whole-file to main**. main renamed the env to `FUZZ_VERSION`, narthex to `CARGO_FUZZ_VERSION`, which is referenced by four cache-key strings — mixing hunks leaves it undefined and silently changes every fuzz cache key. - Two traps worth a reviewer's attention. `tests-integration/tests/http.rs` registers tests through a **hard-coded name list** inside `macro_rules! http_tests`; a hunk-level resolution silently un-registers tests without a compile error. Verified in this branch: all 11 narthex names and 6 main names are present, 89 total, no duplicates. And `.github/ci/scope-rules.toml`'s fuzz gate matches nothing for `.github/scripts/prepare-fuzz-corpus.sh`, because the `src_tree = "src/"` fail-open net does not cover that path — a PR touching only that script would skip fuzz CI silently. Resolved to main's ruleset, which lists it. ### The other 25 conflicts - **Cargo**: `Cargo.toml` had exactly one real conflict (`greptime-proto` rev); everything else is a union (main's `src/common/cache`, `src/relational-engine`, `datafusion-execution`, `sha2`, pprof pin, `sqlx 0.9`, `simd-json 0.19` + narthex's `postcard`, `stacker`, `twox-hash`). `Cargo.lock` was regenerated, not hand-merged. `src/datatypes/Cargo.toml` keeps main's `enterprise = []` feature and drops `regex` (main removed its last use). - **mito2**: `metrics.rs` union (narthex's `CACHE_LOAD` duplicates main's `cache_loading_metrics()` — TODO), `read/series_scan.rs` import union, `series_index/tests.rs` follows the new signature. - **query**: `dist_plan/merge_scan.rs` conflicted only in a doc comment (function body auto-merged); the new comment states the two invariants — `greptime.series.*` schemas are compared for exact identity, everything else keeps main's lenient nullability rule, and the exact branch runs first. `optimizer.rs` / `query_engine/state.rs` are adjacent insertions, both kept. - **datatypes / store-api / session**: re-export, trait-item and `QueryContext` literal unions. - **compatibility case**: `fulltext_bloom_tokens/case.toml` — both sides fixed the same thing (`from_range`), so exactly one may survive; kept main's `<v1.3.0` (concatenating both yields a duplicate TOML key). ### Default-configuration differences from `main` (audit) An audit of "does a default deployment ever execute narthex code" found that the native path itself is properly gated (engine option `false` **and** per-request extension absent; `plan_pql` is the only entry point; SQL has no native branch; `metric-engine`'s diff is test-only; no table-level override exists). But **three behaviour changes still apply on the default path**, and they need to be either gated or consciously accepted: 1. **PromQL instant output cap removed** — `InstantManipulate` no longer enforces the previous output-point limit. With `memory_pool_size` set to a finite value this can turn a previously rejected instant query into a `ResourcesExhausted` failure somewhere else, or into a much larger allocation. 2. **topk/bottomk/limitk response `metric_name`** — `servers/src/http/prometheus.rs:1527-1545` rewrites the metric name in the HTTP response with no switch. 3. **ACL-scoped metric discovery** — `frontend/src/instance.rs:1374` reads the per-request extension to derive discovery authorisation targets; with an ACL checker installed this changes which targets are submitted. There are also per-query overheads that do not change results: an eager `time_pruned_files` build, an unconditional `series_disjoint` computation, and two whole-tree traversals (`native_raw_plan_bounds_region_reads`, `rewrite_hybrid_binary_operands`). Each is cheap but runs for every default query. These are tracked for a follow-up PR whose acceptance criterion is: **a default-configured deployment produces byte-identical plans to `main`**. That PR will also gate the three items above, short-circuit the traversals, downgrade `pub mod native_compile`, and decide whether `query.experimental_native_promql` should be a public option at all. ### Blockers before this leaves draft 1. **proto pin** — currently points at the fork branch for GreptimeTeam/greptime-proto#355 (`b9159cd6`). Replace with the upstream rev once #355 lands. 2. **Native payload byte cache** — restore it with a dedicated page-cache budget (and un-ignore the two samples tests), or accept the loss explicitly. 3. **`query.experimental_native_promql`** — decide public option vs typed extension. ### Compatibility - No wire/persisted format change here beyond the proto bump carried by #355; both sides left `proto/` untouched. - `samples.json` / `GTSERIES` / `GTSAMPLE` artifacts are unchanged by the merge. Note the pre-existing (not introduced here) behaviour: narthex's `load_version_control` prefers `samples.json` and discards the legacy catalogs when it exists. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [ ] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ## Merge timing - [ ] Merge during the nightly window <!-- nightly-merge --> <!-- Left unchecked on purpose: this is a rehearsal and still has three open blockers. -->
Metrics, logs, and traces.
One engine, on your infrastructure.
User Guide · API Docs · Live Demo · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Or skip the install and try the Live Demo.
Next: Quickstart, the full Install Guide, Demo Scenes, and the FAQ.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write (with native histograms over Remote Write 2.0), Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Pipelines that parse and transform logs at ingestion, retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo, 168 investigations per interface across six models. 40% fewer wrong diagnoses, about 48% fewer input tokens, about 45% lower estimated cost (write-up)
- JSONBench billion-record cold run: first place as of March 2025
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning and region migration are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison. For GreptimeDB Enterprise inquiries, contact us.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol endpoints and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage.
- Metasrv — metadata, region routing, and cluster coordination (DDL procedures, failover, region migration, repartitioning). Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Releases follow Semantic Versioning. For where the project is going, read the v1.0 highlights, the 2026 roadmap, and what we mean by an observability database.
Community
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.
