mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
fe2e364d23
* feat(datatypes): add the shared aggregate-state representation A frame that carries partial aggregate state has to say which aggregate it is state of and which accumulator components it carries, or a consumer can read a sum state as an average one. These types are that statement: the kind, the ordered component list, one position's values, and the schema that validates the two against each other. Nothing uses them yet; the frame content that carries them follows. Signed-off-by: discord9 <discord9@163.com> * feat(datatypes): carry aggregate state as a third frame content A state frame keeps the series-major columns a frame already has — the key, finality, grid spans and labels — and puts each aggregate field's accumulators in one list of component structs under `fields`, the shape a classic histogram payload already uses. An absent list item is that field having no contribution at that position, so a hole stays a hole instead of becoming a zero, and a group state — which carries no component at all — still says per field where it was folded. The reader keeps typed handles into the batch rather than expanding the state into rows and rebuilding it, so the merge above it reads the arrays where they lie. Signed-off-by: discord9 <discord9@163.com> * feat(common-query): pack and unpack aggregate-state series frames The packer takes what a partial aggregate produces — one group, its carried positions and its per-field state — and the unpacker hands back the row with typed handles into the batch. That asymmetry is deliberate: export columnarises once, import must not rebuild the state to read it. Validation mirrors the evaluated pair: the declaration is parsed, contiguity required, the empty batch decoded before the stream is returned, and kind, component order, grid and field count are each checked with the part that disagreed named. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run native partial and final aggregates An aggregate node used to compute only `Complete`, refusing the stages a split aggregate needs. It now folds evaluated chunks into kernel state for `Partial` and merges that state into values for `Final`, so the compiler's `Partial -> Final` plan has a runtime behind it. The executor gains a third chunk kind: an input leaf accepts raw, evaluated or state chunks according to what its binding produces, never guessing from a schema, and a state chunk cannot be handed to a node that reads values. The state binding now carries the aggregate kind and its ordered components, derived from the kernel's own state fields, so a sum state cannot be merged as an average one. The sibling router refuses aggregate state explicitly for now; the node that routes it follows in its own change. Signed-off-by: discord9 <discord9@163.com> * feat(promql): route aggregate state by its group labels A state row is one group, complete as one producer wrote it: there is nothing to chain and nothing to merge, so the existing repartition node — whose invariants are all about assembling a key's chain and waiting for every input to close — is the wrong place for it. This node does one thing: send each row to the output its canonical labels name, so every contribution of a group reaches one owner and the merge above sees all of them. The read belongs to the outputs that were claimed, not to the plan, so dropping every output stops it and returns what it reserved. Signed-off-by: discord9 <discord9@163.com> * feat(promql): execute partial aggregates as state frames A fused node now reads its inputs by what they produce — raw samples, evaluated values, or aggregate state — instead of inferring the kind from a schema, and a partial root answers state rather than rows: the executor's state chunks are packed into the aggregate-state frame the merge above consumes. The temporary partial-fold bridge goes with it: no marker field, no side channel fold, no compatibility accessor. A tree that folds on the side is exactly what this change exists to remove, and the aggregate merge that replaces it is a separate node. The state path derives its mode and its binding from one reading of the root, so the two cannot disagree, and no production path panics on a malformed tree. Signed-off-by: discord9 <discord9@163.com> * feat(promql): split native aggregates into partial and final trees The previous commit made a `Partial` aggregate export its state, but left `AggregateFinalize` in place: it still called `FusedSeries::try_new_partial` and `partial_fold`, which that commit deleted, so the crate did not compile. Replace the bridge with the two trees that run a split aggregate: - `native/split.rs` owns the cut. It moves out of `aggregate_finalize.rs` unchanged in structure, and `Split::into_trees` binds the two halves: the prefix under a `Partial` aggregate answering aggregate state, and a `Final` aggregate consuming that state with the suffix above it. - The suffix is narrower than the old merge admitted. Only point-wise maps and a binary operation with a constant side may sit above the cut; `topk`, the classic-histogram functions, a second aggregate and a vector matching are declined, because the groups a `Final` aggregate owns are split across partitions and each of those reads a whole vector. - `FusedSeries` gains `try_new_state_export` (a `Partial` node declares its own state frame schema rather than taking a caller's), `split_aggregate` (declining shapes become `None`, not an error), `reads_aggregate_state` and the state-input path of `to_execution_plan_with_parallelism_option`, which routes state through `StateRepartitionExec` and lets a cross-series tree read state from more than one partition. - The runtime executes the `Final` stage: the guard that refused it is gone. `AggregateFinalize`, `aggregate_exchange` and their tests are deleted; nothing in the crate referenced the exchange module. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): route aggregate state through the series repartitioner Replace the dedicated state-repartition node with the existing series repartition execution: state batches are routed by their canonical labels without being rebuilt as StateSeries, and each row slice is charged for the Arrow wrappers it newly allocates while the parent buffers stay charged once. Carry a producer's state charge with the chunk it hands over, so the exporter that packs rows across polls remains accountable for the batch it still holds instead of releasing it at handoff. Read a Final aggregate's value and tag columns positionally, which is the order the split's output schema is built in, and name them through the legacy expression only at the outer boundary; a bare sum over a metric whose value column is named after the whole expression no longer fails planning. Pack aggregate-state frames in one pass over positions and validity instead of scanning the contributing rows for every position. Cover the changed behaviour with execution tests for bare global and grouped sums over one- and two-value-field tables, for the handed-over state charge, and for the export lifecycle on drain and early drop. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bound tree depth on both sides and validate merge inputs The depth limit was decode-only and set to a number nobody had measured. A 129-deep tree could therefore be built, serialized and shipped, while the receiver refused it; and in CI the boundary test itself aborted with a stack overflow, because a 2 MiB thread was asked to build, serialize, decode and drop a chain of that depth, and the recursive drop of `Box<NativeNode>` is not covered by a guard that only wraps `Deserialize`. Measured instead of guessed. A probe walked build, serialize, decode, validate and drop at increasing depth inside a named 2 MiB thread: 32 through 128 completed, 160 aborted. The cap is now 64, half the highest passing depth, and the same number is enforced in `NativeTree::try_new` and in the deserializer. Verified through the real constructor: a 64-node tree round-trips, and a 65-node tree is refused when it is built as well as when it is decoded, so sender and receiver accept the same set. The boundary test runs on a named thread, which is how a future failure will identify itself. The merge node accepted any single input. `with_exprs_and_inputs` rebuilt `AggregateFinalize` over whatever it was handed, on the reasoning that a rewritten input only changes where the state comes from. Nothing checked that it was state at all -- and until now nothing could produce a counterexample, because a values-producing producer could not be serialized. The serializer was standing in for a validation that belonged to the node. `with_exprs_and_inputs` now rejects an input whose schema is not `state_schema(binding.tags)`, so substituting a computed-values producer fails at the logical node with a message naming the expected schema, rather than depending on an encoding limitation. `AggregateFinalizeExec` already had the same check at physical planning; the serializer stays permissive. The three placement tests in `commutativity.rs` asserted the old refusal as their mechanism -- an ordinary fused node "fails to encode", a partial producer "cannot be pushed down". Those expectations were the encoding limitation wearing a test's clothes. They now assert the responsibilities that remain: an ordinary fused node round-trips through the encoder, a partial producer rides to the region while its finalize stays on the frontend, and a values producer handed to a merge is refused by the schema contract. The test-only `NPRQ` envelope is gone. It framed trees with its own version, a 1 MiB cap and its own rejections, none of which production had, so `native_tree_envelope_rejects_version_size_trailing_and_truncation` was proving a helper's protocol while the PR description claimed a production guarantee. Trees are tested through plain postcard, transport through the real `FusedSeries` interface. The redundant clone-and-revalidate in `FusedSeries::deserialize` is also gone: `NativeTree::Deserialize` already routes through `try_new`. Signed-off-by: discord9 <discord9@163.com> * style: let taplo order the workspace dependency entry `taplo format --check` runs over the whole workspace in CI, not over the two manifests I had checked. It sorts the `[workspace.dependencies]` table, so `postcard` belongs in alphabetical position inside the external-dependencies block rather than immediately after its comment. Local checking was narrower than the CI command, which is why this only surfaced there. Signed-off-by: discord9 <discord9@163.com> * style: place postcard in the sorted position the review asked for Inside the external-dependencies block, `taplo` was willing to leave the entry before the `axum` comment, because a comment line ends a reorder group. The review asked for the alphabetical position, which `taplo format --check` also accepts, so use it. Signed-off-by: discord9 <discord9@163.com> * docs(promql): name the real reason a multi-source name grouping declines The comments still said the fused node's declared columns cannot name `__name__`. Since #10106 a discovered source declares it, so a nameless selector's `sum by (__name__)` survives; what actually declines is a tree reading more than one *named* source, where the table name is the metric name and no source declares the tag. Same conditions, accurate reason. Signed-off-by: discord9 <discord9@163.com> * test(promql): find the topk producer by behaviour, not by node count `native_aggregate_topk_over_global_partitions` asserted that the whole plan holds exactly one fused node. Once native frames cross the region boundary a plan legitimately holds the frontend root and the region producers at the same time, so that count is no longer an invariant. The test now selects the segmented producer and the ranking node by what their plans do: at least one producer contains `segment` and none of them runs `topk`, and exactly one node across the fused and final captures ranks. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.