fengyue-xve 1e29e8d4b6 fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions

A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.

Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.

Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.

Fixes #9454

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix: access ColumnSchema fields directly and use plain format args

ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: access ColumnSchema data_type field in alter table tests

Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop borrows now that data_type is read as a field

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop useless format! in nested expression test

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: cover the partition tail in change_col_type sqlness case

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop spurious blank line in change_col_type result

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(meta): compare partition bounds with the scan comparison coercion

The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.

Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: update the sqlness fixture for the new incompatibility message

The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency

common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(meta): reject every non-widening type change of a partitioned column

The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.

Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.

Signed-off-by: fengyue-xve <1759519137@qq.com>

---------

Signed-off-by: fengyue-xve <1759519137@qq.com>
2026-10-19 14:49:14 +00:00
2023-08-10 08:08:37 +00:00
2023-06-25 11:05:46 +08:00
2023-11-09 10:38:12 +00:00

GreptimeDB Logo

Metrics, logs, and traces.
One engine, on your infrastructure.

A columnar database for metrics, logs, and traces on object storage. Apache-2.0 licensed core.

User Guide  ·  API Docs  ·  Roadmap 2026  ·  Slack

Stable Canary Nightly Docker Pulls License

stable for production  ·  canary includes pre-releases  ·  nightly is a weekly snapshot of main

Introduction

GreptimeDB is an open-source observability database. Metrics, logs, and traces run on one columnar engine over object storage and share one table model: tags, timestamp, and fields. When signals carry common identifiers such as service, host, or trace ID, you can correlate them in SQL without moving data between databases.

Ingest through OpenTelemetry, Prometheus Remote Write, Loki Push, or Elasticsearch Bulk. Use SQL across observability data and PromQL for metrics. Migrate ingestion one signal at a time without rebuilding your collectors.

One Query Across Signals

OpenTelemetry ingestion writes spans to opentelemetry_traces and log records to opentelemetry_logs. Both tables carry trace_id, so correlating them is a join:

-- The slowest failed spans in the last hour,
-- with the log lines emitted inside those same traces.
SELECT
    t.service_name,
    t.span_name,
    t.duration_nano / 1000000 AS duration_ms,
    l.timestamp AS log_time,
    l.severity_text,
    l.body
FROM opentelemetry_traces t
JOIN opentelemetry_logs l ON l.trace_id = t.trace_id
WHERE t.timestamp > now() - INTERVAL '1' HOUR
  AND t.span_status_code = 'STATUS_CODE_ERROR'
ORDER BY t.duration_nano DESC
LIMIT 20;

Metrics join the same way, on any tag the tables share, such as service, host, or pod.

Why You Might Use It

  • You run Prometheus plus Loki or Elasticsearch and want one backend instead of three
  • You have outgrown Prometheus on cardinality or retention and don't want the Thanos/Mimir operational surface
  • You are hitting Loki's query performance limits as log volume grows
  • You need long retention on object storage without a separate analytics stack
  • You want to query telemetry with SQL, not only a domain query language
  • You are storing GenAI or agent telemetry (OTel GenAI conventions) alongside infrastructure signals

Learn more in Why GreptimeDB.

What's Supported

Ingest OpenTelemetry (OTLP), Prometheus Remote Write, Loki Push, Elasticsearch Bulk, InfluxDB line protocol, gRPC
Query SQL, PromQL, Jaeger-compatible trace queries, MySQL and PostgreSQL wire protocols
Storage S3, GCS, Azure Blob and S3-compatible endpoints as primary storage, with memory and local-disk caches
Built in Retention policies, downsampling, continuous aggregation, explicit table partitioning, and inverted / skipping / fulltext indexes

Compute and storage are disaggregated: object storage holds the data, while memory and local-disk caches keep recent and frequently queried data close to compute.

GreptimeDB Overview

Benchmarks

Compatibility and Migration

Compatibility is per protocol, and query-side coverage is narrower than ingestion.

Compatible Not compatible
Prometheus Remote Write ingestion; PromQL queries Gaps are listed in PromQL compatibility
Loki Push ingestion; dual-write through Grafana Alloy makes the cutover gradual LogQL and the rest of the Loki query API
Elasticsearch _bulk ingestion in the open-source core; QueryDSL partially, in Enterprise Most other Elasticsearch APIs

Limitations and Edition Boundary

Cluster deployment, object storage, the Flow engine, and every ingestion protocol listed above are in the Apache-2.0 build. Repartitioning, region migration, and index creation are manual operations there.

Read replicas, workload isolation, and automated repartitioning are GreptimeDB Enterprise features, along with enterprise security and governance. The Enterprise overview has the current list, and pricing has the edition comparison.

Architecture

GreptimeDB can run in two modes:

  • Standalone — single binary for development and small deployments.
  • Distributed — four components, each independently scalable:
    • Frontend — protocol entry (OTel, Prometheus, MySQL/PostgreSQL, gRPC, ingestion APIs for Elasticsearch/InfluxDB/Loki) and the distributed query engine. Stateless, scales horizontally.
    • Datanode — region engine with WAL, memtable, SST, cache, compaction, and indexes. Persists data to object storage. Elastic.
    • Metasrv — metadata, routing, repartitioning, and security. Backed by a pluggable KV layer (etcd or RDS).
    • Flownode (optional) — continuous flow computation (streaming and materialized views).

For deeper coverage, see the architecture doc or DeepWiki.

GreptimeDB System Overview

Try GreptimeDB

For AI agents — paste this prompt into your agent:

Read https://docs.greptime.com/SKILL.md and follow the instructions
to deploy, configure, ingest, and query GreptimeDB.
docker run -p 127.0.0.1:4000-4003:4000-4003 \
  -v "$(pwd)/greptimedb_data:/greptimedb_data" \
  --name greptime --rm \
  greptime/greptimedb:latest standalone start \
  --http-addr 0.0.0.0:4000 \
  --grpc-bind-addr 0.0.0.0:4001 \
  --mysql-addr 0.0.0.0:4002 \
  --postgres-addr 0.0.0.0:4003

Dashboard: http://localhost:4000/dashboard

Read more in the full Install Guide.

Troubleshooting:

  • Cannot connect to the database? Ensure that ports 4000, 4001, 4002, and 4003 are not blocked by a firewall or used by other services.
  • Failed to start? Check the container logs with docker logs greptime for further details.

Getting Started

Build From Source

Prerequisites:

  • Rust toolchain — stable, pinned by rust-toolchain.toml
  • Protobuf compiler (>= 3.15)
  • C/C++ building essentials: gcc / g++ / autoconf and the glibc dev package (libc6-dev on Ubuntu, glibc-devel on Fedora)
  • Python toolchain (optional, only for some test scripts)

Build and run:

make                          # build greptime binary
cargo run -- standalone start # start in standalone mode

Common dev commands:

make fmt            # format Rust code
make clippy         # lint (fails on warnings)
make test           # unit + integration tests (uses cargo-nextest)
make sqlness-test   # SQL regression tests

See the Contribution Guidelines for the full developer workflow.

Tools & Extensions

Project Status

GreptimeDB is generally available, with stable APIs and regular releases. It runs in production at scale — OceanBase Cloud operates 80+ GreptimeDB clusters managing 300 TB of logs, cutting log storage cost by 60%+ after migrating from Grafana Loki. See more in case studies.

Release lines and support windows are in the version reference. For where the project is going, read the v1.0 highlights and the 2026 roadmap.

Community

We invite you to engage and contribute!

If GreptimeDB is useful to you, please star the repo.

Known Users

License

GreptimeDB is an open-core project. Its core is licensed under the Apache License 2.0.

A small set of peripheral, enterprise-only features are gated behind the enterprise Cargo feature (not built by default) and are governed by the separate GreptimeDB Enterprise License. Source files under that license carry an explicit Enterprise License header.

Commercial Support

Scaling observability on your infrastructure? GreptimeDB Enterprise adds the operational, security, and support layer for production deployments. Contact us for details.

Contributing

Integration CI Codecov

Acknowledgement

Special thanks to all contributors! See AUTHOR.md.


All trademarks, logos, and brand names referenced in this README and in the Overview diagram are the property of their respective owners. Their use is for identification purposes only and does not imply endorsement or affiliation.

S
Description
Open-source, cloud-native, unified observability database for metrics, logs and traces, supporting SQL/PromQL/Streaming.
Readme Apache-2.0
1.5 GiB
Languages
Rust 97.5%
Python 1.7%
Shell 0.4%
JavaScript 0.2%