discord9 27a7047f31 feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): cover preserve row sequence table alter

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): clear preserve_row_sequence marker on copy_region_from

copy_region_from copies source FileMeta into the target region, which has an
independent sequence domain. The physical per-row sequences in the copied
file belong to the source region only; trusting them in the target would let
an exact sequence-range request replay source-domain rows as if they were
target sequences. Clear the preserve_row_sequence marker on copied files so
the target fails closed with SequenceRangeUnsupported until the scan provably
cannot intersect the copied rows.

Add a regression test: copying from a preserve-enabled source into a
preserve-enabled target clears the marker, and an exact (2, 7] request on the
target returns SequenceRangeUnsupported instead of replaying source rows.

Fixes #8865

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: remove redundant doc comments for exact sequence range options

Approved comment-cleanup-only changes for #8865: drop outdated doc
summaries duplicated on the exact_sequence_range wrapper and the
preserve_row_sequence field, drop pure-restatement doc comments on the
SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and
remove the four structural SQL comments from the alter_preserve_row_sequence
case. No behavior changes; .result regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): fail closed exact reads on copied files and extension ranges

Address review feedback on #8865:

- copy_region_from: clear the source-domain FileMeta::sequence along with
  the preserve_row_sequence marker. An unmarked file retaining a stale
  source-domain max sequence could be silently skipped by
  files_allow_exact_sequence_range() as 'proven disjoint' in the target's
  independent sequence domain, dropping rows on exact (C, H] reads. With
  sequence=None the capability check fails closed (SequenceRangeUnsupported)
  until the copied rows are provably disjoint.
- Engine/reader: reject exact sequence-range reads whenever a follower
  region has an extension range provider attached. Extension streams are
  returned without a row-level sequence filter, so exactness cannot be
  proven; treat the capability as missing (fail closed) instead of emitting
  out-of-range rows. The reader also fails closed as defense in depth.

Tests: extend copy_region_from regression to assert the copied file's
sequence hint is cleared; mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): use doc comments for test function descriptions

Elevate the block comments describing test functions (in scan_test and
copy_region_from_test) to /// doc comments, matching the convention used
elsewhere in the exact sequence range change. No logic change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): extract helpers and trim comment noise in exact sequence reads

PR finalization for #8865 (zero behavior change, full mito2 suite green):

- engine: extract validate_sequence_fences and
  sequence_range_unsupported_reason, keeping error variants, check order
  and reason strings identical; OSS binds the extension blocker to false.
- handle_copy_region: extract remap_copied_file_meta and
  file_descriptors_for_meta; rename file_ids -> source_file_ids and
  files_to_copy -> new_file_metas.
- compactor: rename max_input_sequence -> known_max_input_sequence,
  document the None semantics (empty input vs unknown sequence).
- Remove restating/outdated comments (ScanInput::sequence_range doc
  first line, outdated file-pruning note, options test restatements),
  compress verbatim comments while keeping why/invariants/contracts.

Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt,
git diff --check, mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): reject foreign-region SSTs in exact sequence reads

Reading an SST whose FileMeta.region_id differs from the scanned region
means the region's sequence domain is broken (manifest corruption or a
repartition/copy path that leaked a source-domain file). Treat this as
an explicit RegionSequenceDomainBroken error instead of silently
ignoring the file's sequence or falling back to a full scan: the region
is unusable for exact sequence-range reads until the foreign lineage is
compacted away or repaired.

- files_allow_exact_sequence_range / exact_sequence_range now return
  Result and propagate the error through engine fence validation and
  scan construction (StatusCode::Internal, distinct from the
  fallback-capable SequenceRangeUnsupported).
- Row-level flat-batch sequence filtering rejects foreign-region files
  as defense in depth.
- Engine test asserts the broken-domain error rather than
  Unsupported/fallback.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping

An unmarked file's FileMeta.sequence may be synthesized by the
region-edit or repartition paths (committed+1 import barrier), not a
physical max of its rows. Treating it as a whole-file disjoint proof
could permanently skip rows that were never incrementally consumed
once the flow checkpoint passes that value.

Exact sequence-range capability now requires every SST in the region to
carry the preserve_row_sequence marker; any unmarked file disables
exactness (fallback), and the (C, H] file-selection skip also only
applies to marked files. Foreign-region files still raise
RegionSequenceDomainBroken as before.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): scope exact sequence-range capability to the time-selected read set

The exact capability check used to walk the entire SstVersion, so a
single unmarked or foreign-region SST anywhere in the region disabled
exact reads or raised RegionSequenceDomainBroken even when the
request's time range could never touch that file.

Both the engine fence and the scan builder now derive the read set with
shared time-pruning + exact-min/sst-min selection and validate
capability only over the files actually selected: a time-pruned file
cannot contribute a row to (C, H], so it cannot affect exactness. The
existing fail-loud semantics are unchanged for every selected file
(foreign region id -> RegionSequenceDomainBroken; unmarked -> exact
unavailable).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): wash untrusted sequences in compaction and restore barrier skipping

Compaction with any non-preserved input now writes a sequence-less
output: the physical __sequence column is zeroed (the flat format
requires the internal columns) and FileMeta.sequence records the
region-local admission barrier committed_sequence + 1 (falling back to
the flushed frontier). preserve_row_sequence stays false.

Exact sequence-range scans interpret an unmarked file's sequence as an
admission barrier: barrier <= C means flow has already consumed the
whole file, so it is skipped at file level; a missing or newer barrier
fails closed. Foreign-region files stay in the selected read set so the
capability fence still raises RegionSequenceDomainBroken.

This closes the recovery loop: after a region repartition, one
time-scoped fallback consumes the migrated rows, then compaction washes
the untrusted per-row sequences away and exact incremental reads resume
via file-level barrier skipping.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): drop restating comments in known_max_input_sequence tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): trim SQLness result EOF whitespace

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): reuse exact scan file selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): strengthen sequence scan coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): trim ALTER option comments

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): remove no-op bulk compaction check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve trusted row sequences when reading SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve target sequence domain for imported SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): add trailing blank line to SQLness result EOF

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): trim exact sequence scan plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): fold exact SST selection checks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make legacy compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make PK compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-03 06:20:09 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00
2023-03-28 19:14:29 +08:00

GreptimeDB Logo

Metrics, logs, and traces.
One engine, on your infrastructure.

A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.

User Guide | API Docs | Roadmap 2026

Stable Canary Nightly

stable for production  ·  canary includes pre-releases  ·  nightly is a weekly snapshot of main

Docker Pulls GitHub Actions Codecov License
Slack Twitter LinkedIn

Introduction

GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.

Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.

Why You Might Use It

  • You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
  • You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
  • You need long retention on object storage without a separate analytics stack
  • You want to query telemetry with SQL, not only a domain query language
  • You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals
  • You need the same engine and semantics on resource-constrained devices

Learn more in Why GreptimeDB.

Overview

A quick overview of what GreptimeDB ingests, how it connects to other systems, and what its distributed engine lets you do.

GreptimeDB Overview

What's Supported

Ingest OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC
Query SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols
Storage S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches
Built in Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes

Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.

Compatibility and Migration

Compatibility is per protocol, and query-side coverage is narrower than ingestion.

Compatible Not compatible
Prometheus Remote Write ingestion; PromQL queries Gaps are listed in PromQL compatibility
Loki Push ingestion; dual-write through Grafana Alloy makes the cutover gradual LogQL and the rest of the Loki query API
Elasticsearch _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise Most other Elasticsearch APIs

Benchmarks:

Limitations and Edition Boundary

Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.

Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.

Architecture

GreptimeDB can run in two modes:

  • Standalone — single binary for development and small deployments.
  • Distributed — four components, each independently scalable:
    • Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
    • Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
    • Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
    • Flownode (optional) — continuous flow computation (streaming and materialized views).

For deeper coverage, see the architecture doc or DeepWiki.

GreptimeDB System Overview

Try GreptimeDB

For AI agents — paste this prompt into your agent:

Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard

Read more in the full Install Guide.

Troubleshooting:

  • Cannot connect to the database? Ensure that ports 4000, 4001, 4002, and 4003 are not blocked by a firewall or used by other services.
  • Failed to start? Check the container logs with docker logs greptime for further details.

Getting Started

Build From Source

Prerequisites:

  • Rust toolchain — nightly, pinned by rust-toolchain.toml
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials: gcc / g++ / autoconf and the glibc dev package (libc6-dev on Ubuntu, glibc-devel on Fedora)
  • Python toolchain (optional, only for some test scripts)

Build and run:

make                          # build greptime binary
cargo run -- standalone start # start in standalone mode

Common dev commands:

make fmt            # format Rust code
make clippy         # lint (fails on warnings)
make test           # unit + integration tests (uses cargo-nextest)
make sqlness-test   # SQL regression tests

See the Contribution Guidelines for the full developer workflow.

Tools & Extensions

Project Status

GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.

Read the v1.0 highlights and 2026 roadmap, or browse the version reference.

If GreptimeDB is useful to you, please star the repo.

Known Users

Community

We invite you to engage and contribute!

License

GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.

A small set of peripheral, enterprise-only features are gated behind the enterprise Cargo feature (not built by default) and are governed by the separate GreptimeDB Enterprise License. Source files under that license carry an explicit Enterprise License header.

Commercial Support

Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.

Contributing

Acknowledgement

Special thanks to all contributors! See AUTHOR.md.


All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.

S
Description
Open-source, cloud-native, unified observability database for metrics, logs and traces, supporting SQL/PromQL/Streaming.
Readme Apache-2.0
1,006 MiB
Languages
Rust 98.6%
Python 0.7%
Shell 0.4%
JavaScript 0.1%