mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
f2f8043fdd
* feat(promql): add the bound tree and batch protocol for native execution A native PromQL evaluation needs a compiled form whose operations are already bound to kernels, fields and grids. Recognising an opportunity from a finished physical plan cannot provide that: the opportunity then depends on how the optimizer happened to arrange a subtree, and every new shape widens a fixed template. `NativeExpr` instead covers input, scalar, instant, window, map, aggregate and binary nodes, each carrying a bound operation and a declared output type. Legality becomes a question about types and data dependencies, which `NativeNode::validate` answers, so a new combination of supported operations is an ordinary tree rather than another matcher. Two properties are checked there rather than assumed: a window's declared input grid must be the grid its input produces, so a subquery's inner grid is consumed as it is instead of being resampled, and an aggregate's output type must match its stage, so intermediate state cannot pass for a query value. `EvaluatedChunk` is the batch a node hands to the next one: an ordinary Rust value, addressed on absolute grid positions, closing a series exactly once. An empty final chunk is still meaningful — it is how a closed prefix is reported, which a consumer needs in order to tell "nothing more here" from "no more input". Nodes are identified by their position in the tree rather than by an id. The position is derived by the one traversal both `DisplayAs` and run-time construction use, so what is shown and what is registered cannot disagree. It stays in the metric name because the collector aggregates by name first, which would otherwise merge two nodes of the same operation. This adds the tree and the batch protocol only. The evaluator's chunked interface, memory reservation and execution are not part of it. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): make the bound tree's type contracts match what each stage consumes Four review findings on the bound tree, all about checks that passed the wrong things. An aggregate's stage decided its output but not its input. Every stage first required its input to be evaluated, which aggregate state is not, so `Final(Partial(...))` was rejected and `Final(Window(...))` was accepted. Each stage now states its pair: `Complete` takes a vector and produces one on the same grid, `Partial` takes a vector and produces state on that grid with the bound grouping, `Final` takes state with the bound grouping and produces a vector on the state's grid. Window and instant selections accepted aggregate state. The cause was asking `is_evaluated()`, which is false both for raw samples and for state, since state has no grid: `Instant(Partial(...))` and a window bound to `input_grid: None` over `Partial(...)` both passed and would have read state as raw samples. Both now match on `NativeType::Samples` first and only then on the grid, so state is rejected by what it is rather than by lacking a grid. Scalars were modelled as a grid-less evaluated sample, and `BoundScalar`'s doc claimed a value varying per grid point is not a scalar. That is wrong for PromQL: `time()` and `scalar(v)` produce a scalar per evaluation time. A scalar is now its own `NativeType::Scalar(grid)`, so a binary operation can tell a broadcast from label matching -- `sum by(job)(rate(x[5m])) + time()` broadcasts rather than matching on the empty label set -- and `Map`, `Aggregate`, `Instant` and `Window` reject it by type. The `time()` kernel itself is not in scope. An input could bind only samples, so an exchange boundary could not bind the state a `Final` aggregate consumes. `BoundInput` holds a `NativeType`, and input nodes check that their output is the same binding rather than the same schema. A peak is the one figure the collector must not add up: it groups metrics by name, clears the partition and sums gauges, so partition peaks of 1024 and 2048 reported 3072, which is neither peak. `PEAK_RESERVED_BYTES` now carries the partition in its name (`native.p0.root.peak_reserved_bytes`), while counts and the other gauges still sum across partitions, where a sum is meaningful. The test that claimed the peak kept its partition only checked a name prefix; it now registers two partitions, runs the real `aggregate_by_name`, and asserts both peaks survive with their own values. The walkthrough's `+1` test was wrong twice over: it took a `Final` aggregate directly over a `Window` -- the case the first finding now rejects -- and applied `MathOp::Abs`, which is not `+1`. It now builds a real binary `+` with a scalar operand. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): use the evaluator's chunk type for native batches `native/batch.rs` defined its own `EvaluatedChunk` with the same two fields as `promql::evaluated::EvaluatedChunk`. Two definitions of one protocol drift: the evaluator's chunk could not be handed to a native node without translation, and a change to either would have to be mirrored. Use `crate::evaluated::EvaluatedChunk` instead. The type moves here with the evaluator's documentation, which states the same two rules this module did: a grid position is absolute, and `is_last` is the only thing that closes a series. The module keeps its own doc explaining the rules native nodes rely on and re-exports the type, so `native::batch::EvaluatedChunk` still names it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reject a node that declares a different field count than it consumes A window and a map both report one value per input field, so they keep the layout of their input. Validation checked that both sides were samples and that the grids matched, but not that the field counts did, so a tree could declare two output fields over a one-field input and still pass `NativeTree::try_new`. The mismatch then surfaced at run time, against a schema the tree was supposed to have validated. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate the field layout of every node that declares one Field counts were checked for a window and a map and nothing else, so an instant selection, either aggregate stage and a binary operation could each declare a layout their kernel does not produce and still pass validation, surfacing at run time against a schema the tree was supposed to have validated. Each node is now checked against the count its own kernel produces, which for a binary over two vectors is the shorter side. Grids are compared by their specification, as the kernels compare them, instead of by allocation. A compiler that allocates one grid per window -- `rate(a[5m]) + rate(b[5m])` -- built a tree the runtime would run and validation rejected. Binding identity is still compared by allocation: an input has to report the one binding it was given, which is a different question from what two equal grids mean. A rejected tree is a compiler defect, so it is reported as an internal error rather than as a mistake in the client's statement, which `Plan` would have implied. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): check a window's input ordering and range, and drop the tests that only repeat another module `validate` accepted an interleaved input for a window, although the range evaluator reads one series to its last chunk before it starts the next, so the tree this layer builds was guaranteed to fail at run time. A window now rejects both an interleaved input and a range that is not positive. An instant selection reads one series at a time, so it keeps accepting an interleaved input and now says why. The grid comment claimed the input grid is checked to cover the window's read range. It is not: only a subquery feeds an evaluated input to a window, and this compiler does not accept subqueries yet. The binary operation's catch-all arm is unreachable at run time -- state has no grid and the grid checks return before the match -- but the match still has to cover the state pairs, so they are spelled out and say so instead of claiming a value arrived. Rejection branches had no tests, so deleting any of them kept the suite green. Each now has one. The four tests in `native/batch.rs` repeated the ones in `evaluated.rs` through the re-export, so they could not fail on their own; they are gone, and the coverage stays in `evaluated.rs`. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): bind a grouping as `by` or `without`, not as a label list A label list can only spell `by`: `sum without (instance) (rate(x[5m]))` has no list of group labels, because its key is every label the series has except the ones named. Spelling that as a list would group the query by labels it meant to remove, and the kernel takes both shapes already (`aggregate::Grouping::{By, Without}`), so the tree had no way to say what the engine can already do. State carries its grouping in the same terms now, so a `Final` also refuses state built for the other shape, not only state built from a different label list. The `groups` gauge counts per partition: a `Partial` aggregate sees one group in every partition it reaches, so the summed gauge is the total of per-partition entries rather than distinct groups. Both the metric's doc and the comment that claimed otherwise now say so. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com>
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.