mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-25 13:22:30 +00:00
Lei, HUANG
bb3f07b482
* refactor(mito2): pick TWCS L1 merges by closed units TWCS could keep rewriting L1 files without new writes. Overlap-only picks could split their inputs into one more file than they read, and L1 file reduction then rewrote up to 16 files to remove that single file. The outputs, split by key, created new overlaps and the cycle repeated. Replace the three global pick phases with one pick per window, scheduled by class across windows. Every pick now decreases a termination measure when the window receives no writes. - Active windows only take merges that remove files. L1 files are grouped by conflicts and merged in key order, so size-split tails are merged together. A reduction rewrites at most 4 output files' worth of bytes per removed file. - Inactive windows absorb L0 files, then take closed merges: no other file intrudes on the inputs' bounding box, so outputs cannot add overlaps. Each merge resolves conflicts or removes files, decreasing (conflicts, files). Files that do not take part in an overlap are not rewritten. - Overlaps that only share a boundary key no longer count as conflicts, so a size-split sorted run does not look overlapping. - Closed merges may take 4x the input limit to resolve a whole window once. - Cache primary key ranges per pick instead of resolving them per comparison. - Log the pick branch of each compaction output. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): state the compactor assumptions behind TWCS termination Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): resolve active window overlaps on manual compaction A window that stops receiving writes keeps being the active window, whose picks only remove files, so its overlaps were never resolved. Manual compactions now pick every window as inactive, so a closed merge can resolve them. Automatic compactions are unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): defer overlap resolution in the window before the active one Late data keeps landing in the window right before the active window. A closed merge that resolves its overlaps early rewrites the whole window, then the late data brings the overlaps back, and the balanced run check keeps them there. Overlap-only merges now skip that window until it is older. File-reducing merges and manual compactions are unchanged. On the 12h/1GB out-of-order bench this reaches overlap depth 1.0 instead of 2.08 at the same write amplification (3.01 vs 3.02). Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): only check earlier-starting files when closing units Closing units only checks files that start before a unit ends, and the overlap checks compare key bounds without cloning them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep settling windows from rewriting L1 files The window before the active one still receives late data. Skipping only its overlap-only merges was not enough: file-reducing merges of conflicting files and mixed L0/L1 merges still rewrote its L1 files, and a mixed merge could rewrite a nearly full output for each small late file. A settling window now only merges L0 files among themselves and conflict-free files; the window resolves its overlaps once it is older. On the out-of-order bench, write amplification drops from 3.88 to 3.30 at 2GB/6h, 3.05 to 2.94 at 1GB/6h and 3.01 to 2.96 at 1GB/12h, with overlap depth 1.0 in all three. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve late overlaps once a window is quiet Late data that lands in a window older than the settling one is merged into a small file covering the whole key range. Resolving it rewrites the window's dominant sorted run, which the balanced run check refuses, so the overlap stayed. Removing the check instead rewrites the window for every late flush. A window is quiet once the region has written at least as many rows as the window holds since its newest file. Overlap merges in a quiet window may rewrite a dominant run, so each quiet period pays for at most one such rewrite. Sequences are per row and kept by compaction, so this needs no state. On the bench with late data up to 2 hours, mean overlap depth drops from 1.67 to 1.17 at 2GB/6h and from 2.33 to 1.83 at 1GB/6h, at write amplification 3.70 to 4.25 and 3.26 to 3.74; all inactive windows of 2GB/6h realtime reach depth 1 (write amplification 2.80 to 3.30). The remaining depth is in the final active window. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): check closed merges of quiet and settling windows terminate Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): split TWCS closures over the input limit A file spanning the keys of a large sorted run joins every run file into one closure. Once a closure held more files than one compaction accepts, closed merges skipped it, so the window never merged those files again and late files kept piling up. Split such closures into single files, like active-window picks do, and replace the closed check with a conflict bound: outputs only conflict with files intruding on the inputs' box, so a merge is taken if `outputs · I` does not exceed the conflicts its inputs had with other files. Intervals of whole closures have no such conflicts and still need to be closed, so other windows pick as before, and `(conflicts, files)` still strictly decreases. Small run files and late files over a large run can be merged again. A late file over a run that cannot shrink stays: resolving it would rewrite the whole run. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): address TWCS picker review findings - Manual compactions may rewrite a dominant sorted run to resolve overlaps. The active window is never quiet, so manual compactions could not resolve its late-data overlaps before. - Single-key intervals predict one output: the writer never splits a series, so large files of one series were never merged. - The active window falls back to single files when no group of conflicting files qualifies, so a large file no longer keeps its small peers unmerged. - Compute the settling window distance with checked_sub, which overflowed for windows at the timestamp limits. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): count inner keys as TWCS conflicts and check balance per group - A single key strictly inside another file's key range now conflicts, so a late file of one series inside a wide file is resolved like other overlaps. Files sharing only a boundary key still do not conflict. Intrusion covers the new pairs, which a randomized test checks. - Check the dominant-run guard per group of conflicting files. Runs of separate groups never overlap, so pairing one group's run with another group's peer made an interval of two dominant runs look balanced. - A window with unknown row counts, written before they were recorded, is not quiet: its threshold would drop to a single row. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): index TWCS files by key to avoid quadratic grouping Grouping a window's files compared every file with all files sharing its time range, and closing groups and checking intrusions scanned the whole window. A sorted run split by key shares one time range, so a window of N such files took O(N²) checks without any conflict. The 1024-file fallback only covers the sorted-run path, not these groups. Index the files by lower primary key in a segment tree that keeps each subtree's largest upper key and its time bounds. Grouping, closing, and the conflict bound only check the files whose key and time ranges meet, so a file of a key-sliced run only meets its neighbors. Conflicts among split files are precomputed per file and counted per interval with a binary search instead of pairwise. On a run of 20,000 key-sliced files with 5 late files spanning all keys, a closed merge pick takes ~62 ms instead of ~3.9 s and an active pick ~19 ms instead of ~3.5 s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(object-store): retry transient S3 transport failures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound TWCS closed-merge conflict graph work Budget the candidate files visited while building conflict units and split-unit adjacency. When the budget is exceeded, treat every file as a split unit and pick a bounded file reduction instead of materializing a quadratic graph. The fallback candidate is admitted only through adds_no_conflicts, so outputs do not add conflict pairs and the (P, N) termination invariant still holds. Dense picks are logged with their own branch. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): charge TWCS index nodes to the conflict-graph budget The conflict-graph work budget only counted files that reached a callback, so a traversal with poor pruning could still visit a quadratic number of internal nodes without triggering the dense fallback. Charge every ExtentIndex node, including pruned nodes, and stop the traversal as soon as the shared budget is exhausted. Add a regression test where the root straddles the query time but neither leaf matches, so a callback-only budget would never fire. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): retry smaller TWCS dense compaction candidates If a dense fallback candidate fails admission, try one single-output subinterval within it before giving up. Reuse the reduction guards and check both candidates against the full window's external conflicts, preserving bounded work and termination.\n\nCover the chain regression through both fallback unit constructions and reject one-output retries that introduce conflicts across time gaps. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): compact shared primary-key boundaries Count shared primary-key endpoints as conflicts and apply the same overlap rule to bounding-box intrusions. Existing compaction writers keep series intact, so fresh output ranges remain disjoint. Align simulated output splits with the writer and cover boundary-only cleanup in inactive and manual windows, plus stable real output files after a subsequent TWCS request. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): compact small L1 files in inactive windows Allow bounded small-L1 reductions in settling and older inactive windows when normal picks cannot proceed. Preserve full-window conflict checks, large-file protection, and LastNonNull closure semantics.\n\nCover selection limits, termination, and real SST compaction with tombstones and reopen checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify TWCS unit grouping guarantees Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Metrics, logs, and traces.
One engine, on your infrastructure.
A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.
User Guide · API Docs · Roadmap 2026 · Slack
stable for production · canary includes pre-releases · nightly is a weekly snapshot of main
Introduction
GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.
Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.
One Query Across Signals
OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to
opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:
-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
t.service_name,
t.span_name,
t.duration_nano / 1000000 AS duration_ms,
l.timestamp AS log_time,
l.severity_text,
l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;
Metrics join the same way, on any tag the tables share, such as service,
host, or pod.
Why You Might Use It
- You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
- You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
- You are hitting Loki's query performance limits as log volume grows
- You need long retention on object storage without a separate analytics stack
- You want to query telemetry with SQL, not only a domain query language
- You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
Learn more in Why GreptimeDB.
What's Supported
| Ingest | OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC |
| Query | SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols |
| Storage | S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches |
| Built in | Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes |
Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.
Benchmarks
- Agent RCA Bench: LLM agents doing root cause analysis over GreptimeDB versus Prometheus + Loki + Tempo. 40% fewer wrong diagnoses, 48% fewer input tokens, 45% lower cost (write-up)
- GreptimeDB tops JSONBench's billion-record cold run test
- TSBS Benchmark
- More benchmark reports
Compatibility and Migration
Compatibility is per protocol, and query-side coverage is narrower than ingestion.
| Compatible | Not compatible | |
|---|---|---|
| Prometheus | Remote Write ingestion; PromQL queries | Gaps are listed in PromQL compatibility |
| Loki | Push ingestion; dual-write through Grafana Alloy makes the cutover gradual | LogQL and the rest of the Loki query API |
| Elasticsearch | _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise |
Most other Elasticsearch APIs |
Limitations and Edition Boundary
Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.
Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.
Architecture
GreptimeDB can run in two modes:
- Standalone — single binary for development and small deployments.
- Distributed — four components, each independently scalable:
- Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
- Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
- Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
- Flownode (optional) — continuous flow computation (streaming and materialized views).
For deeper coverage, see the architecture doc or DeepWiki.
Try GreptimeDB
For AI agents — paste this prompt into your agent:
Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
-v "$(pwd)/greptimedb_data:/greptimedb_data" \
--name greptime --rm \
greptime/greptimedb:latest standalone start \
--http-addr 0.0.0.0:4000 \
--grpc-bind-addr 0.0.0.0:4001 \
--mysql-addr 0.0.0.0:4002 \
--postgres-addr 0.0.0.0:4003
Dashboard: http://localhost:4000/dashboard
Read more in the full Install Guide.
Troubleshooting:
- Cannot connect to the database? Ensure that ports
4000,4001,4002, and4003are not blocked by a firewall or used by other services. - Failed to start? Check the container logs with
docker logs greptimefor further details.
Getting Started
Build From Source
Prerequisites:
- Rust toolchain — stable, pinned by
rust-toolchain.toml - Protobuf compiler (>= 3.15)
- C/C++ building essentials:
gcc/g++/autoconfand the glibc dev package (libc6-devon Ubuntu,glibc-develon Fedora) - Python toolchain (optional, only for some test scripts)
Build and run:
make # build greptime binary
cargo run -- standalone start # start in standalone mode
Common dev commands:
make fmt # format Rust code
make clippy # lint (fails on warnings)
make test # unit + integration tests (uses cargo-nextest)
make sqlness-test # SQL regression tests
See the Contribution Guidelines for the full developer workflow.
Tools & Extensions
- Kubernetes: GreptimeDB Operator
- Helm Charts: Greptime Helm Charts
- Dashboard: Web UI
- gRPC Ingester: Go, Java, C++, Erlang, Rust, .NET, TypeScript
- Grafana Data Source: GreptimeDB Grafana data source plugin
- Grafana Dashboard: Official Dashboard for monitoring
Project Status
GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.
Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.
Community
We invite you to engage and contribute!
If GreptimeDB is useful to you, please star the repo.
License
GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.
A small set of peripheral, enterprise-only features are gated behind the
enterprise Cargo feature (not built by default) and are governed by the
separate GreptimeDB Enterprise License. Source files under
that license carry an explicit Enterprise License header.
Commercial Support
Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.
Contributing
- Read our Contribution Guidelines.
- Explore Internal Concepts and DeepWiki.
- Pick up a good first issue and join the #contributors Slack channel.
Acknowledgement
Special thanks to all contributors! See AUTHOR.md.
- Uses Apache Arrow™ (memory model)
- Apache Parquet™ (file storage)
- Apache DataFusion™ (query engine)
- Apache OpenDAL™ (data access abstraction)
All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.