Files
greptimedb/tests/compatibility
jeremyhi c6962802a9 feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration

A region migration now gives its candidate region registration a random non-zero
candidate lease id. Leases granted because of that registration carry it, but only
while the table route exists, and the upgrade instruction carries the same id. The
datanode passes the id of each lease to the engine, and Mito records the lease of
the latest heartbeat per region outside the region instance, so it survives a
reopen during catchup.

The migration also drops the candidate registration before it restores the route
on rollback, and before the procedure leaves the migration tracker when it ends
without a rollback.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a lease only while the region stays open

Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* build: point greptime-proto at the candidate lease id change

Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop lease records when workers stop

Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): drop a redundant helper comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(store-api): pin the candidate lease id wire bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: document candidate registration and lease record lifecycles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep candidate lease ids through customized lease renewers

The heartbeat handler now sets the candidate lease id of each lease a
customized renewer returns from the renewed region with the same id.

The metric engine records a lease for the data region only when the
metadata region ignored it, since the data region records its own
leases. The countdown reset command carries a named struct instead of
a tuple.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): cover a lease the data region ignores

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tighten lease id lookups and lease test assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: align lease naming and share the txn-observing kv backend

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tidy lease naming, docs and shared test helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop field comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop method comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: say which heartbeat a lease's send time refers to

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop the reset deadline field comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: state when an operating region's role and id reach a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: pin greptime-proto to the merged candidate lease id change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: add the candidate lease id argument to the direct demotion test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: order new keeper helpers after public items and test equal send times

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 18:37:17 +00:00
..

GreptimeDB Compatibility Test Framework

Compatibility tests verify that one GreptimeDB version can restart on state written by another version.

SQL restart tests are run via cargo sqlness compat and reuse the sqlness-runner infrastructure.

Wire fixtures

Frozen protobuf fixtures are exercised by the Rust tests that own their context conversion, not by SQLness case discovery. The legacy query-context fixture checks omitted read-preference defaulting and unrelated metadata preservation:

cargo nextest run -p session -E 'test(test_legacy_query_context_wire_defaults_to_leader)'

The same legacy fixture also checks that logical ALTER requests without pinned table IDs retain their original behavior, and that new pinned IDs survive the query-context protobuf conversion:

cargo nextest run -p common-meta -E 'test(test_pinned_logical_table_ids_wire_compatibility)'

The ID guard requires an updated metasrv; older metasrvs ignore the new internal extension. Update metasrv before enabling exemplar source-schema reconciliation on frontends. No procedure-state or protobuf fields are added.

The native histogram Remote Read fixture checks the sampled encoder against independent Prometheus protobuf values, including exact integer counts, custom bounds, and NaN payloads:

cargo nextest run -p servers -E 'test(native_histogram_remote_read_wire_fixture)'

The legacy region lease fixture checks that a lease without a candidate lease id decodes as such and encodes to the same bytes as before the field existed:

cargo nextest run -p store-api -E 'test(test_legacy_granted_region_wire)'

Object store WAL fixtures

Frozen objects of the object store WAL are checked byte for byte by the format tests of log-store. The SQL runner cannot cover the object store WAL: it owns wal.provider. See the fixture notes.

Manifest fixtures

Table-compaction fixtures are exercised by mito2's Rust tests. They cover ordinary file-edit compatibility, supported SST levels, and explicit rejection of unpublished prototype task-state records.

The inline sketch-index fixtures in src/mito2/src/manifest/action.rs cover both file-level and column-level descriptors and mixed builtin/sketch indexes. They run without the enterprise feature:

cargo nextest run -p mito2 -E 'test(test_sketch_index_manifest_fixture)'

These fixtures verify supported-reader decoding, not downgrade support for binaries that predate the new index types. Upgrade all manifest readers before enabling a writer that emits these descriptors; downgrade only to readers that recognize them. The OSS build does not generate sketch payloads, so SQL restart cases cannot create these descriptors through an OSS writer.

Frozen region manifests that carry pending WAL tails are checked by the manifest tests of mito2. No SQL statement records a pending tail. See the fixture notes.

Quick Start

# Self-compat smoke test (current binary only):
cargo run -p sqlness-runner -- compat

# Test from a specific released version to current:
cargo run -p sqlness-runner -- compat --from-version v0.9.5

# Test between two local binary directories:
cargo run -p sqlness-runner -- compat --from-bins-dir ./bins/old --to-bins-dir ./bins/new

# Test a downgrade from the current build to a released binary:
cargo run -p sqlness-runner -- compat --from-bins-dir ./bins/current --to-version v1.1.4

# Run a compatibility case in standalone mode:
cargo run -p sqlness-runner -- compat --topology standalone --test-filter "downgrade_compatibility"

# Run a specific case:
cargo run -p sqlness-runner -- compat --test-filter "basic_table"

# Preview which cases would run (no services started):
cargo run -p sqlness-runner -- compat --dry-run --from-version v0.9.5

# See all options:
cargo run -p sqlness-runner -- compat --help

Prerequisites

  • Docker (for etcd): PR1 always uses Docker etcd for distributed metadata. External metadata stores are future work.
  • From binary: Either --from-version <version> to auto-pull a release, or --from-bins-dir <path> to use a local build. The binary greptime must exist directly inside the given directory.
  • To binary: Defaults to the current debug build (target/debug/greptime). Override with --to-bins-dir <path> or fetch a release with --to-version <version>.
  • Custom target-dir: If you use a non-default CARGO_TARGET_DIR, the debug binary won't be at target/debug/greptime. Pass --from-bins-dir / --to-bins-dir explicitly pointing to your custom target directory. Alternatively, run cargo build -p greptime without a custom target-dir.

Case Format

Each compat case is a directory under tests/compatibility/cases/ containing three required files plus an expected output file:

my_case/
  case.toml       # Metadata (required)
  setup.sql       # SQL to run on the from version (required)
  verify.sql      # SQL to run on the to version (required)
  verify.result   # Expected output from verify.sql

case.toml — Required Metadata

name = "my_case"
reason = "Why this compatibility case exists"
introduced_by = "PR #1234 or feature name"
topologies = ["distributed", "standalone"]
from_range = ["*"]
to_range = ["*"]
features = ["table"]
owner = "team-name"
# optional:
namespace = "my_explicit_namespace"   # defaults to sanitized directory name

Required fields: name, reason, introduced_by, topologies, from_range, to_range, features, owner.

Old-Stage Datanode Configuration Overlay

To apply a datanode configuration overlay while running the old stage, add this strict optional table to case.toml:

[old_config]
datanode = "old-datanode.overlay.toml"

datanode is required whenever [old_config] is present; empty tables and unknown keys are rejected. The reference is relative to the case directory and must remain confined to that directory. The sidecar is native datanode TOML, which the runner loads and preflights before starting services or creating state.

The runner first applies the datanode baseline, then merges the sidecar. Tables merge recursively only when both values are tables. Scalars, type mismatches, arrays, and arrays of tables replace the baseline value atomically. In particular, region_engine has no special merge behavior.

Runner-owned fields cannot be changed by an overlay: mode, node_id, storage.data_home, meta_client_options.metasrv_addrs, and wal.provider, plus wal.dir for Raft WAL or wal.broker_endpoints for Kafka WAL. The runner restores these fields to its baseline values, or deletes them when the baseline has no value. It warns about protected-field overrides without printing their values.

Version-Range Filtering

from_range and to_range control which binary versions a case applies to:

Entry Meaning
"*" Matches any version (including unknown).
"vX.Y.Z" or "=vX.Y.Z" Matches exactly version X.Y.Z.
">=vX.Y.Z" Matches X.Y.Z or later.
">vX.Y.Z" Matches versions strictly later than X.Y.Z.
"<=vX.Y.Z" Matches X.Y.Z or earlier.
"<vX.Y.Z" Matches versions strictly earlier than X.Y.Z.

The range list is OR: a case matches if any entry matches.

Best-effort enforcement: The runner tries to determine the effective version:

  • --from-version is used directly.
  • --from-bins-dir / --to-bins-dir (or the default debug build) runs <binary> --version to infer the version.
  • When the version cannot be determined (e.g. binary missing or --version fails), non-wildcard ranges are skipped with a message; wildcard (*) ranges still match.

Example (legacy_jsonb):

from_range = ["<=v1.1.0"]
to_range = [">=v1.1.1"]

This case only runs when the old binary is <= v1.1.0 and the new binary is >= v1.1.1.

CI Version Window

The CI job uses tests/compatibility/ci.toml to choose the small sliding window of recent released from versions to test against the PR-built to binary:

from_versions = ["v1.0.0", "v1.1.0"]

Keep this window small for PR and merge-queue latency: the goal is to catch upgrade compatibility issues from recent releases to the latest build, not to retest every historical version on every PR. Case-level from_range/to_range still decides which cases run for each version pair; the CI window only decides which old binaries are sampled. Broader historical windows belong in nightly or release-validation workflows.

The GitHub Actions workflow delegates the window loading and compat invocation to .github/scripts/run-compat.py; the workflow YAML should stay as a thin wrapper around artifact download/extraction and this script.

downgrade_to_versions optionally lists releases that CI restarts after the PR-built cluster. Those runs select only the downgrade_compatibility case in both distributed and standalone topologies.

setup.sql — Setup Phase (From Version)

SQL statements executed on the from version cluster. These must succeed (any error fails the case). Setup output is NOT compared against any result file.

Rules:

  • Statements are semicolon-terminated
  • -- prefix for ordinary comments
  • -- SQLNESS ... interceptor comments follow ordinary sqlness semantics

verify.sql — Verify Phase (To Version)

SQL statements executed on the to version cluster. Output is compared against verify.result in sqlness snapshot style.

verify.result — Expected Output

Expected output in sqlness format. If this file is missing, the runner generates it from actual output and fails — the author must review, commit the generated file, and rerun.

<statement>;

<output>

<next statement>;

<output>

If output differs from expected, the run fails and verify.result is updated with actual output.

PR1 Limitations

  • Sqlness interceptors: -- SQLNESS ... comments are applied per statement using the same interceptor registry as the ordinary sqlness runner, including the GreptimeDB PROTOCOL interceptor. For PROTOCOL POSTGRES, the namespace prelude uses SET search_path instead of USE. Avoid unqualified PostgreSQL-protocol table names starting with pg_: GreptimeDB's current PostgreSQL compatibility parser rewrites them to pg_catalog.<table>.
  • Distributed topology: The compat runner starts 1 metasrv + 3 datanodes + 1 frontend + 1 flownode. Standalone compatibility runs need no external metadata store.
  • No comment-based compat config: The compat runner does not define extra compatibility configuration in SQL comments; sqlness comments keep their normal sqlness meaning.

Namespace Isolation

Each case runs in its own database namespace to prevent cross-case interference:

  • Default namespace is derived from the case directory name (sanitized to [a-z][a-z0-9_]*)
  • Override with namespace in case.toml
  • Duplicate namespaces are rejected at discovery time (before version filtering)
  • Before each statement, the runner executes a namespace prelude (not written to verify.result): CREATE DATABASE IF NOT EXISTS <ns> via gRPC; then USE <ns> for gRPC/MySQL statements or SET search_path TO '<ns>' for PostgreSQL statements.

Batch Behavior

  • The baseline (no-overlay) profile runs first. Cases whose old datanode TOML is semantically equivalent share one profile; profiles run serially and in isolation.
  • Each profile has its own state and etcd lifecycle. Its overlay is applied only to old-stage datanodes and remains in effect through old-stage setup restarts. The current stage always uses a clean configuration.
  • Cases run serially (no parallelism in PR1). Namespace state is session/protocol state and cannot be shared concurrently.
  • Same namespace across cases is rejected.
  • Without fail-fast, the runner verifies only cases whose setup succeeded. With fail-fast, it cleans up the active profile before stopping.
  • --dry-run displays the selected profiles, cases, and sidecar paths without printing configuration values; it starts no services.

xfail Policy (Future)

For PR1, all cases are expected to pass. Future PRs will add xfail support with required issue and expiry fields.

Cross-Job Distributed State

PR1 runs setup and verify in the same job (same process). Cross-job artifact restore for distributed state is not supported in PR1 due to port randomization and etcd lease expiration.