Yingwen 01a73105b8 feat: use partition range cache in scan (#7873)
* feat: use range cache in scan

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename dedup to skip_dedup

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: use background concat for buffered batches

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: store permits

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: fix potential panic

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: skip range-cache wrapping when cache is disabled

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: avoid potential deadlock

Deadlock Chain

1. Range-level merge tasks: Each concurrent build_flat_partition_range_read (line 494-506) calls
build_flat_reader_from_sources → create_parallel_flat_sources → spawn_flat_scan_task. These
background tasks loop: acquire permit → input.next() → release permit.
2. Final merge tasks: After all range tasks return streams (line 509-511), the distributor calls
build_flat_reader_from_sources again (line 520-527) → create_parallel_flat_sources → more
spawn_flat_scan_task tasks. These also loop: acquire permit → input.next() → release permit.
3. Circular wait: The final merge tasks' input.next() reads from ReceiverStreams backed by
range-level merge tasks. If all num_partitions permits are held by final merge tasks blocked on
input.next(), the range-level merge tasks can't acquire permits to produce data → deadlock.

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add test for small permits

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: use avg batch size for channel size

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: fix test

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address review comments

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-04-13 08:27:53 +00:00
2023-08-10 08:08:37 +00:00
2026-04-01 01:59:37 +00:00
2026-03-19 21:26:41 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00
2026-03-11 07:29:35 +00:00
2023-03-28 19:14:29 +08:00

GreptimeDB Logo

One database for metrics, logs, and traces
replacing Prometheus, Loki, and Elasticsearch

The unified OpenTelemetry backend — with SQL + PromQL on object storage.

Introduction

GreptimeDB is an open-source observability database built for Observability 2.0 — treating metrics, logs, and traces as one unified data model (wide events) instead of three separate pillars.

Use it as the single OpenTelemetry backend — replacing Prometheus, Loki, and Elasticsearch with one database built on object storage. Query with SQL and PromQL, scale without pain, cut costs up to 50x.

Features

Feature Description
Drop-in replacement PromQL, Prometheus remote write, Jaeger, and OpenTelemetry native. Use as your single backend for all three signals, or migrate one at a time.
50x lower cost Object storage (S3, GCS, Azure Blob etc.) as primary storage. Compute-storage separation scales without pain.
SQL + PromQL Monitor with PromQL, analyze with SQL. One database replaces Prometheus + your data warehouse.
Sub-second at PB-EB scale Columnar engine with fulltext, inverted, and skipping indexes. Written in Rust.

Perfect for:

  • Replacing Prometheus + Loki + Elasticsearch with one database
  • Scaling past Prometheus — high cardinality, long-term storage, no Thanos/Mimir overhead
  • Cutting observability costs with object storage (up to 50x savings on traces, 30% on logs)
  • AI/LLM observability — store and analyze high-volume conversation data, agent traces, and token metrics via OpenTelemetry GenAI conventions
  • Edge-to-cloud observability with unified APIs on resource-constrained devices

Why Observability 2.0? The three-pillar model (separate databases for metrics, logs, traces) creates data silos and operational complexity. GreptimeDB treats all observability data as timestamped wide events in a single columnar engine — enabling cross-signal SQL JOINs, eliminating redundant infrastructure, and naturally supporting emerging workloads like AI agent observability. Read more: Observability 2.0 and the Database for It.

Learn more in Why GreptimeDB.

How GreptimeDB Compares

Feature GreptimeDB Prometheus / Thanos / Mimir Grafana Loki Elasticsearch
Data types Metrics, logs, traces Metrics only Logs only Logs, traces
Query language SQL + PromQL PromQL LogQL Query DSL
Storage Native object storage (S3, etc.) Local disk + object storage (Thanos/Mimir) Object storage (chunks) Local disk
Scaling Compute-storage separation, stateless nodes Federation / Thanos / Mimir — multi-component, ops heavy Stateless + object storage Shard-based, ops heavy
Cost efficiency Up to 50x lower storage High at scale Moderate High (inverted index overhead)
OpenTelemetry Native (metrics + logs + traces) Partial (metrics only) Partial (logs only) Via instrumentation

Benchmarks:

Architecture

GreptimeDB can run in two modes:

  • Standalone Mode - Single binary for development and small deployments
  • Distributed Mode - Separate components for production scale:
    • Frontend: Query processing and protocol handling
    • Datanode: Data storage and retrieval
    • Metasrv: Metadata management and coordination

Read the architecture document. DeepWiki provides an in-depth look at GreptimeDB: GreptimeDB System Overview

Try GreptimeDB

docker pull greptime/greptimedb
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --rpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard

Read more in the full Install Guide.

Troubleshooting:

  • Cannot connect to the database? Ensure that ports 4000, 4001, 4002, and 4003 are not blocked by a firewall or used by other services.
  • Failed to start? Check the container logs with docker logs greptime for further details.

Getting Started

Build From Source

Prerequisites:

  • Rust toolchain (nightly)
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials, including gcc/g++/autoconf and glibc library (eg. libc6-dev on Ubuntu and glibc-devel on Fedora)
  • Python toolchain (optional): Required only if using some test scripts.

Build and Run:

make
cargo run -- standalone start

Tools & Extensions

Project Status

Status: RC — marching toward v1.0 GA! GA (v1.0): March 2026

  • Deployed in production handling billions of data points daily
  • Stable APIs, actively maintained, with regular releases (version info)

GreptimeDB v1.0 represents a major milestone toward maturity — marking stable APIs, production readiness, and proven performance.

Roadmap: v1.0 highlights and release plan and 2026 roadmap.

For production use, we recommend using the latest stable release.

If you find this project useful, a would mean a lot to us!

Star History Chart

Known Users

Community

We invite you to engage and contribute!

License

GreptimeDB is licensed under the Apache License 2.0.

Commercial Support

Running GreptimeDB in your organization? We offer enterprise add-ons, services, training, and consulting. Contact us for details.

Contributing

Acknowledgement

Special thanks to all contributors! See AUTHORS.md.

Description
Languages
Rust 99.6%