mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
discord9
72bee4ae9c
* feat(promql): name a fused tree's tags from the existing plan A fused node declares exactly the tags the planner gives it, and reads each one back off the series, so a label set this route cannot derive is a label set the answer would lose. The walk over the tree derives labels for a match, a set operation and the point-wise wrappers over them, and names none for an aggregate's groups or a whole-vector operation - which today declines every such statement. Take those columns from the existing planner instead, the way the aggregate split already does: its tag columns are what it projects for the same statement, and they are now shared through one helper rather than written out twice. Only the tag columns are copied, because the node names its field columns after its own tree while the existing planner can name a computed column after its expression. A label rewrite is a shape the walk cannot derive at all: label_replace adds its destination label and an empty replacement removes it, so passing the input's labels through is an inference the rewrite contradicts. Report no label set there. The rewrite itself keeps the existing plan, and this does not change that: the native runtime refuses two series a rewrite gives one label set, where that plan answers them, so admitting the shape would turn an answer into an error. The test that holds this now covers the two-occurrence shape as well. An aggregate or whole-vector root does not reach this branch today: both read across series, so they are planned by the aggregate split or fall back. It is the prerequisite for the placement that runs such a tree in one execution, and reaches the branch when that lands. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a classic histogram function above the aggregate cut The split accepts a suffix that runs once over the fully merged series, which is why TopK/BottomK were already admitted: the merge has folded every partition's contribution to each group, so the series the suffix reads are the ones the statement's input would have produced in one execution. A classic histogram function is that kind of operation. It reads each histogram off the bucket series that share its other labels and drops le, and the merge has already folded each (..., le) group into one complete series on the evaluated grid, so the buckets it reads are exactly the ones the input would have had. Reading le off those merged series is the same computation, not an approximation. LimitK still declines: it depends on the storage input order, which the merge does not preserve. This is what the o11ybench query `histogram_quantile(0.99, sum(rate(...[5m])) by (instance, le))` needs: it was planned by the existing path only because the split refused a whole-vector operation above the cut. The acceptance test runs the shape over real data and compares the rows two executions report: the fixture spreads six bucket series over several files and a memtable range, so the partial sums genuinely have partitions to merge, and the native answer must equal the complete input's for quantiles at both ends of the interpolation and for histogram_fraction. It also asserts the native run really merged state, so a silent fall back cannot pass it. Signed-off-by: discord9 <discord9@163.com> * feat(promql): read two metric logical tables as two inputs A two-selector statement whose two sides are aggregates kept the existing plan, even where both sides read a metric-engine logical table. The o11ybench query `sum(max by (...) (a)) by (namespace) / sum(b) by (namespace)` is that shape, and it fell back for three separate reasons. First, the engine gate refused every multi-selector metric-engine statement. The clause after it already requires each such selector to be a recognized logical rewrite, so the removed clause refused only the case this change handles. Second, a metric logical table is rewritten to a scan of the physical table restricted by `__table_id = <id>`, so two logical tables over one physical table prepare two scans of the same table. The check deciding whether the node binds one metric or one input per occurrence compared table names only, so the two metrics read as one and both inputs would carry the first one's name - a wrong answer, not a fall back. The id the rewrite already requires is now part of that identity. Third, a tree over two aggregates reads across series, so it is not answered per partition, and the branch for such trees was the aggregate split, which takes one child. A node with more than one input needs no split: it declares one partition and reads all of every input's partitions in one execution, so every group is folded in one place. Such a tree now falls through to the node. The one-selector path is unchanged: it still requires `can_split` and still declines a tree the split cannot cut. So is the multi-selector case whose inputs stay direct scans of one table - such a node is not region-local, as `Categorizer::check_extension_plan` refuses a cross-series plan over more than one scan - which is why the condition is the metric-logical-table one. The acceptance test runs both shapes over two logical tables of one physical table, with values that make a swapped input visibly wrong, and compares the rows the two routes report. Signed-off-by: discord9 <discord9@163.com> * test(promql): log the metric pair's rows through telemetry, not stdout The acceptance test printed its evidence with println!, which the crate lints as an error under -D warnings, and echoed a Display value through to_string inside the format arguments. Every other test in the file logs through common_telemetry::info!, so this one does too. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a tree the runtime cannot execute The multi-input route skipped the aggregate split, and the split was what refused a tree the runtime cannot execute: `check_cut` declines a `quantile` and a grouping by a value field. Skipping it turned queries the existing plan answers into errors, because the node is chosen while planning the logical plan and a refusal the physical layer raises later has nowhere to fall back to. `quantile(0.5, a) / sum(b)` over two metric logical tables reported `NotImplemented: the native runtime does not execute the Quantile { .. } aggregate yet` where the existing plan answers it, and `count(count by (val) (a)) / sum(b)` failed on the grouping by a value field, which the outer `count` had hidden from the column check. Ask the runtime before taking the route, with the same construction the physical layer validates by, and keep the existing plan for whatever it refuses. Asking it rather than listing the unsupported kinds is what makes the check cover refusals this planner does not know about. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a grouping and a selection the node cannot answer A multi-input node skips the aggregate split, so the refusals the split makes for reasons that are not about what the runtime computes were skipped with it. Two of them matter here. A grouping by the metric name keeps a label no column of the existing plan can name. The native aggregate folds its groups by `__name__` as by any other label, but the label set the node declares comes from the existing planner's columns, and `__name__` is the name a raw frame carries rather than a column of any scan - so the existing planner's `by(__name__)` finds nothing and reports no label. The schema would then report one label set for two groups the tree keeps apart: `sum by (__name__) (a or b)` over two metrics would answer two rows a reader cannot tell apart, which is a series identity lost rather than an answer given. A selection that depends on the order its input arrives in is the other. `limitk` keeps the first `k` series of each group in input order, while a node with more than one input reads its inputs' partitions through a gather that hands them over in the order they became ready: the schedule would decide which series a query keeps. Prometheus defines the selection over a series set. The split declines `LimitK` for the same reason. Every other vector operation orders its input itself - `topk` by label, `sort` by value, `limit_ratio` by a label hash - which is why only this one is asked about. Both shapes keep the existing plan. `limitk` is not answerable by either route today, since the existing planner has no kernel for the operator; the point is that the native route is not the first to answer a selection the schedule would decide. Each guard is checked with the guard removed, so both tests are known to fail without it. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline unsupported count_values groupings before execution Signed-off-by: discord9 <discord9@163.com> * docs(promql): scope the native capability probe guarantee Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the same-table pair the multi-selector exclusion used to keep out State that a Mito input joins a multi-input tree only when a second metric logical id makes the tree multi-input, and that only LimitK's selection depends on arrival order. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline ranking grouped by a value field before native execution topk, bottomk and limitk key their groups by series labels, while the existing plan also accepts a value field there and forms one group per sample value. Admitted multi-input trees could therefore keep one row where the existing plan keeps one per group. Validate the grouping when the vector runtime is built so the statement keeps the existing plan. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.