jeremyhi 9f6c957e2b fix(mito2): commit staging edits once when staging ends (#10322)
* fix(mito2): commit staging edits once when staging ends

Staging edits (the new partition expression, the imported files) were saved
as deltas in the staging directory, each taking a manifest version. The end of
staging listed that directory and merged every delta it found, including
deltas a stale writer wrote late.

Staged actions now stay in memory and take no version. The end of staging
commits them once, as the next version, in a new EndingStaging state that
stalls writes and holds role changes back until the commit returns. If the
commit may have been written and the region stays staging, the encoded commit
is kept: a retried request stages nothing new and commits the same content,
and other staged edits are refused with a retryable error meanwhile.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle an uncertain end of staging before later commits

Under overwrite publication a save that may have landed must not be
overwritten, so every commit takes its version before the save and a failed
save leaves that version unused. For the end of staging, the version is now
taken only once the result is known: a failed save is read back at once, and
if the read-back cannot tell, the commit stays unsettled. Every later commit
first reads it back again, applies it to the manifest and the region version if
it landed, and is refused with a retryable error while it still cannot tell.
A follower sync past that version forgets it.

Committed staging actions are applied to the region version through the
manifest context, so the end of staging and a later commit that settles it
share one path.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle before checking a commit and keep the lock into the end of staging

- Settle an unknown end of staging when taking the manifest lock, before the
  write is checked against the manifest, and fire the hook of a landed one
  with the lock released, so a failing commit cannot drop it. Entering staging
  settles too; a staging region settles only when staging ends.
- Pass the held manifest lock from the graceful Leader path into the end of
  staging, so a queued close cannot abort staging in between.
- Wait for the worker to drain the stalled writes before clearing the staging
  directory.
- Let an empty import retry a frozen end commit.
- Count the size of an end delta found by reading it back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep settling out of staging and before a new staging

- Do not settle while the region is staging or a staged edit is running; only
  the end of staging settles there, as it also leaves staging.
- Settle before entering staging through a graceful role change, and fire the
  hook of a landed result even if entering staging then fails.
- List the end-of-staging outcomes in the order they are decided.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle before the metadata backfill and fence the late staging clear

- Settle an unknown end of staging before a graceful Leader on a writable
  region checks whether to backfill metadata; a landed result is merged into
  the backfill notification.
- Skip the staging directory clear after the end of staging if the region was
  closed meanwhile, as it may be open and staging again.
- Clear the staging directory after an end of staging with nothing to commit.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): take the overwrite version before the end of staging is saved

Under overwrite publication the end of staging takes its version before the
save again, as before this change, so a later commit never overwrites a save
that may have landed. After an error that may have written it, the region goes
back to staging with its staged actions and encoded bytes kept, and the retry
writes the same bytes at the next version; a reopen replays both as one commit.

This removes the immediate read-back of a failed end commit, the unsettled mark
and the settling before later commits, the ManifestUnsettled error, and the
region version handle on the manifest context. The end of staging applies the
committed actions to the region version itself again.

A graceful Leader from staging takes the manifest lock again after the end of
staging and runs the metadata backfill, firing one notification for both, as
before this change.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): check the lease and fence conflicts in the end of staging commit

The end of staging commits its staged actions on its own path, so it
skipped the checks other manifest commits make: in conditional
publication it saved after the Leader deadline had passed, and a version
conflict left the region unfenced, so the worker never reloaded it. Run
both commits through one core that makes those checks.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): let the worker pass the drain to the end of staging

The end of staging built and sent the worker's StagingEnded notification
itself, so the region module depended on the worker's dispatch types.
The worker now passes the drain as a future that the end of staging
awaits after it drops the manifest lock.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the write order across the end of staging

The end of staging stores the region's next state under the manifest
lock, and the worker handles the writes it stalled meanwhile later, on
StagingEnded. A write that the worker took in between was applied
before the earlier stalled ones. The worker now stalls a region's later
writes behind those until StagingEnded drains them.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): read and encode the staged actions without copying them

The end of staging copied the staged actions to read them, to encode
them and to commit them. Read and encode them in place, and copy them
for the region hook only when one is registered. The commit keeps its
own copy, so the staged actions survive a failed commit for the retry.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover a flush while staging ends and the staging directory clear

Check that a migration flush gets a retryable error while the region
ends staging and succeeds after it. Plant a delta from an older binary
in the staging directory before applying the staging manifest, so the
check that the directory is cleared looks at the target region and
fails if the end of staging leaves it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): tidy the end of staging wording and a test helper

Nothing is merged at the end of staging any more, so its log, error
reasons and bindings say staged or committed. Fix the region hook doc
sentence and table cells for actions staged in memory, and move the
import helper of the conditional manifest tests next to the other
helpers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the staged files of a region that left staging

An end of staging that leaves for Follower or Downgrading kept the staged
actions in the manager, and the SST listing and all_manifest_files merged
them into the files of a region that never committed them. The end of
staging now drops them when it leaves staging, and both readers merge the
staging manifest only while the region is staging.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): check a resent import against the staged files in linear time

The check that a resent import only adds files that are already staged
walked every staged edit for every file, quadratic in the file count. It
now collects the staged file ids into a set once.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): apply the committed staged actions without another copy

The end of staging cloned the staged actions to split them for the region
version before the commit, on top of the copy the commit applies. The
commit now hands the staged list back once it succeeds, and the end of
staging checks the staged Change by reference before the commit and splits
the returned list after it. One copy stays, because a successful commit
uses the actions twice: a copy builds the new manifest, and the returned
list updates the region version. The region hook gets its copy only when
one is registered, as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): return the normal version from all_manifest_files and drop a dead fallback

all_manifest_files preferred the staging manifest's version. It now returns
the normal version, as its doc says. A test helper's unreachable
`unwrap_or(path)` after `rsplit` becomes `unwrap_or_default()`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the staging helpers after what they do

- Rename is_staging_or_staged_edit to in_staging_session. The Staging and
  EndingStaging pair lives in RegionRoleState::in_staging_session, which the
  compaction scheduler, region_role, should_abort_index and the region-level
  helper use.
- Rename finish_ending_staging to leave_ending_staging, since it can also
  land on Follower, Downgrading or Staging, and tie end_staging,
  EndingStaging and StagingEnded together in the EndingStaging doc.
- Rename contains_import to only_adds_staged_files and the staged list
  field to actions.
- Move the staging paragraph to the public update(), state on end_staging
  that the drain is never awaited under the manifest lock, say when
  all_manifest_files merges staged files, say that the listings do not rely
  on the clear at the end of staging, and count the end of staging in the
  flush rule of the mito2 AGENTS.md, naming the predicates the flush paths
  test.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): keep the end-of-staging drain next to the stalled requests

Move drain_ended_staging into worker.rs, and the set of regions whose writes
wait for it into StalledRequests, which adds, checks and removes them, so
the stall bookkeeping and its release live together.

Rejecting all stalled writes now also forgets these regions. Their stalled
writes are rejected with them, so a later write has nothing to keep order
behind and is handled under the region's current state.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): keep the staged view with the staged actions

The manifest with the staged actions applied was a separate field of the
manager, set and cleared next to the actions by hand. It is now a field of
StagedActions, so clearing the staged actions clears it as well, and its
accessor says what it returns.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): split EndStagingPending out of RegionBusy

A staged append that arrives while a possibly written end commit waits for
its retry was refused with RegionBusy, which therefore had to be in the
not-written arm of may_have_persisted_manifest_update for every producer.
It now gets EndStagingPending (status RegionBusy, retryable), which alone
takes that place, and RegionBusy keeps its earlier classification.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover regions that are ending staging

Check that a region ending staging reports the StagingLeader role and skips
compaction scheduling, and that its bulk writes stall, as do later ones
behind them after staging ended, until the worker drains them. Share the
stall wait and the StagingEnded drain between the tests.

The tests wait for these stalls through a listener on their own engine, so
the end-of-staging stalls notify the test listener as the write buffer
stalls do, and CountingStallListener moves to the shared test listeners.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-12 13:47:01 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00

GreptimeDB Logo

Metrics, logs, and traces.
One engine, on your infrastructure.

User Guide  ·  API Docs  ·  Live Demo  ·  Roadmap 2026  ·  Slack

Stable Canary Nightly Docker Pulls License

stable for production  ·  canary includes pre-releases  ·  nightly is a weekly snapshot of main

Introduction

GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.

Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.

One Query Across Signals

OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:

-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
    t.service_name,
    t.span_name,
    t.duration_nano / 1000000 AS duration_ms,
    l.timestamp AS log_time,
    l.severity_text,
    l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
  AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;

Metrics join the same way, on any tag the tables share, such as service, host, or pod.

Try GreptimeDB

For AI agents — paste this prompt into your agent:

Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard
Or skip the install and try the Live Demo.

Next: Quickstart, the full Install Guide, Demo Scenes, and the FAQ.

Why You Might Use It

  • You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
  • You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
  • You are hitting Loki's query performance limits as log volume grows
  • You need long retention on object storage without a separate analytics stack
  • You want to query telemetry with SQL, not only a domain query language
  • You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals

Learn more in Why GreptimeDB.

What's Supported

Ingest OpenTelemetry (OTLP), Prometheus Remote Write (with native histograms over Remote Write 2.0), Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC
Query SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols
Storage S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches
Built in Pipelines that parse and transform logs at ingestion, retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes

Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.

GreptimeDB Overview

Benchmarks

Compatibility and Migration

Compatibility is per protocol, and query-side coverage is narrower than ingestion.

Compatible Not compatible
Prometheus Remote Write ingestion; PromQL queries Gaps are listed in PromQL compatibility
Loki Push ingestion; dual-write through Grafana Alloy makes the cutover gradual LogQL and the rest of the Loki query API
Elasticsearch _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise Most other Elasticsearch APIs

Limitations and Edition Boundary

Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning and region migration are manual operations there.

Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison. For GreptimeDB Enterprise inquiries, contact us.

Architecture

GreptimeDB can run in two modes:

  • Standalone — single binary for development and small deployments.
  • Distributed — four components, each independently scalable:
    • Frontend — protocol endpoints and the distributed query engine. Stateless, scales horizontally.
    • Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage.
    • Metasrv — metadata, region routing, and cluster coordination (DDL procedures, failover, region migration, repartitioning). Backed by a pluggable KV layer (etcd or RDS).
    • Flownode (optional) — continuous flow computation (streaming and materialized views).

For deeper coverage, see the architecture doc or DeepWiki.

GreptimeDB System Overview

Build From Source

Prerequisites:

  • Rust toolchain — stable, pinned by rust-toolchain.toml
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials: gcc / g++ / autoconf and the glibc dev package (libc6-dev on Ubuntu, glibc-devel on Fedora)
  • Python toolchain (optional, only for some test scripts)

Build and run:

make                          # build greptime binary
cargo run -- standalone start # start in standalone mode

Common dev commands:

make fmt            # format Rust code
make clippy         # lint (fails on warnings)
make test           # unit + integration tests (uses cargo-nextest)
make sqlness-test   # SQL regression tests

See the Contribution Guidelines for the full developer workflow.

Tools & Extensions

Project Status

GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.

Releases follow Semantic Versioning. For where the project is going, read the v1.0 highlights, the 2026 roadmap, and what we mean by an observability database.

Community

If GreptimeDB is useful to you, please star the repo.

Star History Chart Known Users

License

GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.

A small set of peripheral, enterprise-only features are gated behind the enterprise Cargo feature (not built by default) and are governed by the separate GreptimeDB Enterprise License. Source files under that license carry an explicit Enterprise License header.

Contributing

Integration CI Codecov

Acknowledgement

Special thanks to all contributors! See AUTHOR.md.


All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.

S
Description
Open-source, cloud-native, unified observability database for metrics, logs and traces, supporting SQL/PromQL/Streaming.
Readme Apache-2.0
2 GiB
Languages
Rust 97.4%
Python 1.8%
Shell 0.4%
JavaScript 0.2%