jeremyhi 60655a933b fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases

A region's release lived only in memory: a restart after DROP TABLE and
before the next object never re-established the dropped region's
watermark, so its segments counted as live and pinned garbage collection
of the whole prefix.

A release now writes a release record (region id and the entry id the
region is released through) into an object sealed at once, and returns
once that object is durable and indexed, in both acknowledgement modes.
Recovery seeds the obsolete watermarks from the release records of the
indexed objects before it computes any reclaim boundary. obsolete_all
releases in full; the new LogStore::release_region, a no-op by default,
releases a dropped region through the largest entry id the store handed
out for it, and Mito calls it after obsolete when it drops a region.

Objects with release records use format version 2, the version 1 layout
with the records at the end of the footer; every other object is still
written as version 1. A counter of release records made durable is added.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe persisted object store WAL region releases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): carry a failed region release into the next object

A release whose batch failed left its record only in an orphan object,
so a drop that failed there could still leave the region unreleased
after a restart. The store now keeps the release records of failed
batches and seals them with its next batch, on a timer tick when no
entry is admitted, so the release becomes durable once a create
succeeds again.

The version 1 and version 2 object fixtures are frozen under
tests/compatibility/wal/object_store and loaded by the format tests,
and the RFC no longer says that state is persisted only in headers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): complete a region drop whose WAL release fails

A drop request is not sent again once the region was removed, so a drop
whose WAL release failed used to leave the region half dropped. The
drop now completes, cleaning up the region's runtime state and
scheduling its files for deletion, and reports the release failure.

In the durable mode the object store WAL now creates one more object at
stop with the release records of failed and dropped batches, so a
graceful stop does not lose a release that failed. The release record
counter is described as counting the records of indexed objects.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe the stop-time release create of the WAL actor

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): state the accepted window of unreleased dropped regions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe release waiters in the WAL actor's outcome table

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): fail region releases while the object store WAL is failing

A failing store keeps its backlog and rejects appends until a create
succeeds, so a release would otherwise wait for an upload that is not
coming. A release that arrives in that state now fails at once with the
retryable error, and one whose batch is still pending when the store
enters it fails as well. Neither poisons the store: the record of the
first waits for the next sealed batch, like that of a batch that failed,
and the record of the second stays in the backlog, so both become durable
once a create succeeds.

Rejected releases count in rejected_appends_total.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe region releases while the object store WAL is failing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): place the drop crash window in the drop procedure

The crash window between the deletion of a table's metadata and a
durable release record is a gap of the drop procedure, not of the WAL.
Describe it as far as the code shows, name the remedy that belongs in
the procedure, cleaning up offline a region a resumed region drop finds
not open, and list it under future work.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep rejected release records coalesced while failing

A release rejected while the store is failing kept its record for the
next sealed batch, but every timer tick then sealed that record alone
behind the refused backlog, so a release repeated across ticks grew the
backlog without bound. A failing store now keeps such records coalesced
and seals them on the first tick after a create succeeds.

The fault-matrix case of a release while the backlog is refused now runs
with a 10 ms flush interval, repeats the release, and checks that the
backlog does not grow and that recovery reads the record back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): keep rejected release records unsealed while failing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): cite the drop procedure issue for the crash window

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): let a stop seal rejected release records of a failing store

Keeping the records of rejected releases out of sealed batches while the
store is failing also kept them out of the seal a stop requests, so a
stop whose last attempt of the backlog succeeded dropped them and the
released region pinned collection after the restart. A stop now seals
them with the backlog.

A fault-matrix case rejects a release while the store is failing, stops,
lets both creates succeed, and checks the record after a reopen.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the stop of a failing store with rejected releases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep release records readable by older object store WAL readers

Release records were written as a version 2 footer, which a build without
it refuses to read, so rolling a node back after a drop failed to open
the whole WAL prefix. They are now the single entry of a segment of the
reserved region u64::MAX at the first position of a version 1 object:
an older reader indexes that segment as a region it never opens, and
recovery reads the segment back with one range read per object that
holds one. The segment never holds the reclaim boundary and names no
namespace.

A durable stop whose last create of release records fails now reports
the error instead of returning Ok and losing the records silently.

The tests are trimmed: the release unit tests and fault-matrix cases
that repeated each other are merged, and the Mito drop tests share one
parameterized case.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe release records as a segment of a reserved region

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): report release records a poisoning stop drops

A durable stop whose last create of release records met a conflict
poisoned the store and returned Ok, losing the records silently. Stop
now reports the error whenever release records are dropped after it
began, by a rollback or by poisoning. The test of a failed release
covers that conflict, and seals its record on a tick after a one-shot
create failure, so the next tick cannot fail too.

The documentation now says that recovery reads the segments of release
records, and that one that does not decode fails recovery.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe how recovery reads segments of release records

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): say that stop reports release records a conflict drops

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): restore the transiently failing release create in enqueued mode

The fault matrix again covers a release whose create fails transiently in
the enqueued mode: the release waits for the repeat under the same
sequence and returns once it succeeds. The encode_batch doc now says the
release segment is the last segment of the object.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): place the release segment and count the restored case

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): do not fail a region drop on a failed WAL release

Returning the release error from the drop left a metric physical region
half dropped: the metric engine stops after the data region and never
drops the metadata region or forgets the physical region, and a retried
drop finds the data region gone and counts it as done. The release error
gave the caller nothing to retry, since the region is already removed
and the log store keeps the record for its next object. The drop now
logs the error and succeeds.

A metric engine test drops a physical region while the WAL releases of
both mito regions fail, and checks that both regions and the physical
region state are gone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a failed release no longer fails the drop

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): do not fail a region drop on a failed WAL obsolete

A store that the release of another region poisoned fails obsolete as
well, which left the metric engine with its metadata region undropped.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): drop a physical region whose WAL obsolete or release fails

Cover a failed data release, two failed releases and a conflicting
release that poisons the store, after the regions wrote to the WAL, and
check that a store opened later keeps no segment of either region.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a failed obsolete does not fail the drop either

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): fail the drop on an obsolete error of other WAL providers

Only the object store WAL is shared by the regions of a store that the
release of one can poison; Raft Engine and Kafka keep failing the drop
when obsolete fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a stopped or poisoned store keeps no failed release

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-15 03:27:43 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00

GreptimeDB Logo

Metrics, logs, and traces.
One engine, on your infrastructure.

A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.

User Guide  ·  API Docs  ·  Roadmap 2026  ·  Slack

Stable Canary Nightly Docker Pulls License

stable for production  ·  canary includes pre-releases  ·  nightly is a weekly snapshot of main

Introduction

GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.

Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.

One Query Across Signals

OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:

-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
    t.service_name,
    t.span_name,
    t.duration_nano / 1000000 AS duration_ms,
    l.timestamp AS log_time,
    l.severity_text,
    l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
  AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;

Metrics join the same way, on any tag the tables share, such as service, host, or pod.

Why You Might Use It

  • You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
  • You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
  • You are hitting Loki's query performance limits as log volume grows
  • You need long retention on object storage without a separate analytics stack
  • You want to query telemetry with SQL, not only a domain query language
  • You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals

Learn more in Why GreptimeDB.

What's Supported

Ingest OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC
Query SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols
Storage S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches
Built in Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes

Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.

GreptimeDB Overview

Benchmarks

Compatibility and Migration

Compatibility is per protocol, and query-side coverage is narrower than ingestion.

Compatible Not compatible
Prometheus Remote Write ingestion; PromQL queries Gaps are listed in PromQL compatibility
Loki Push ingestion; dual-write through Grafana Alloy makes the cutover gradual LogQL and the rest of the Loki query API
Elasticsearch _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise Most other Elasticsearch APIs

Limitations and Edition Boundary

Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.

Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.

Architecture

GreptimeDB can run in two modes:

  • Standalone — single binary for development and small deployments.
  • Distributed — four components, each independently scalable:
    • Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
    • Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
    • Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
    • Flownode (optional) — continuous flow computation (streaming and materialized views).

For deeper coverage, see the architecture doc or DeepWiki.

GreptimeDB System Overview

Try GreptimeDB

For AI agents — paste this prompt into your agent:

Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard

Read more in the full Install Guide.

Troubleshooting:

  • Cannot connect to the database? Ensure that ports 4000, 4001, 4002, and 4003 are not blocked by a firewall or used by other services.
  • Failed to start? Check the container logs with docker logs greptime for further details.

Getting Started

Build From Source

Prerequisites:

  • Rust toolchain — stable, pinned by rust-toolchain.toml
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials: gcc / g++ / autoconf and the glibc dev package (libc6-dev on Ubuntu, glibc-devel on Fedora)
  • Python toolchain (optional, only for some test scripts)

Build and run:

make                          # build greptime binary
cargo run -- standalone start # start in standalone mode

Common dev commands:

make fmt            # format Rust code
make clippy         # lint (fails on warnings)
make test           # unit + integration tests (uses cargo-nextest)
make sqlness-test   # SQL regression tests

See the Contribution Guidelines for the full developer workflow.

Tools & Extensions

Project Status

GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.

Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.

Community

We invite you to engage and contribute!

If GreptimeDB is useful to you, please star the repo.

Known Users

License

GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.

A small set of peripheral, enterprise-only features are gated behind the enterprise Cargo feature (not built by default) and are governed by the separate GreptimeDB Enterprise License. Source files under that license carry an explicit Enterprise License header.

Commercial Support

Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.

Contributing

Integration CI Codecov

Acknowledgement

Special thanks to all contributors! See AUTHOR.md.


All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.

S
Description
Open-source, cloud-native, unified observability database for metrics, logs and traces, supporting SQL/PromQL/Streaming.
Readme Apache-2.0
1.4 GiB
Languages
Rust 97.9%
Python 1.4%
Shell 0.3%
JavaScript 0.2%