greptimedb/tests/compatibility

GreptimeDB Compatibility Test Framework

Compatibility tests verify that one GreptimeDB version can restart on state written by another version.

Tests are run via cargo sqlness compat and reuse the sqlness-runner infrastructure.

Quick Start

# Self-compat smoke test (current binary only):
cargo run -p sqlness-runner -- compat

# Test from a specific released version to current:
cargo run -p sqlness-runner -- compat --from-version v0.9.5

# Test between two local binary directories:
cargo run -p sqlness-runner -- compat --from-bins-dir ./bins/old --to-bins-dir ./bins/new

# Test a downgrade from the current build to a released binary:
cargo run -p sqlness-runner -- compat --from-bins-dir ./bins/current --to-version v1.1.4

# Run a compatibility case in standalone mode:
cargo run -p sqlness-runner -- compat --topology standalone --test-filter "downgrade_compatibility"

# Run a specific case:
cargo run -p sqlness-runner -- compat --test-filter "basic_table"

# Preview which cases would run (no services started):
cargo run -p sqlness-runner -- compat --dry-run --from-version v0.9.5

# See all options:
cargo run -p sqlness-runner -- compat --help

Prerequisites

• Docker (for etcd): PR1 always uses Docker etcd for distributed metadata. External metadata stores are future work.
• From binary: Either --from-version <version> to auto-pull a release, or --from-bins-dir <path> to use a local build. The binary greptime must exist directly inside the given directory.
• To binary: Defaults to the current debug build (target/debug/greptime). Override with --to-bins-dir <path> or fetch a release with --to-version <version>.
• Custom target-dir: If you use a non-default CARGO_TARGET_DIR, the debug binary won't be at target/debug/greptime. Pass --from-bins-dir / --to-bins-dir explicitly pointing to your custom target directory. Alternatively, run cargo build -p greptime without a custom target-dir.

Case Format

Each compat case is a directory under tests/compatibility/cases/ containing three required files plus an expected output file:

my_case/
  case.toml       # Metadata (required)
  setup.sql       # SQL to run on the from version (required)
  verify.sql      # SQL to run on the to version (required)
  verify.result   # Expected output from verify.sql

case.toml — Required Metadata

name = "my_case"
reason = "Why this compatibility case exists"
introduced_by = "PR #1234 or feature name"
topologies = ["distributed", "standalone"]
from_range = ["*"]
to_range = ["*"]
features = ["table"]
owner = "team-name"
# optional:
namespace = "my_explicit_namespace"   # defaults to sanitized directory name

Required fields: name, reason, introduced_by, topologies, from_range, to_range, features, owner.

Old-Stage Datanode Configuration Overlay

To apply a datanode configuration overlay while running the old stage, add this strict optional table to case.toml:

[old_config]
datanode = "old-datanode.overlay.toml"

datanode is required whenever [old_config] is present; empty tables and unknown keys are rejected. The reference is relative to the case directory and must remain confined to that directory. The sidecar is native datanode TOML, which the runner loads and preflights before starting services or creating state.

The runner first applies the datanode baseline, then merges the sidecar. Tables merge recursively only when both values are tables. Scalars, type mismatches, arrays, and arrays of tables replace the baseline value atomically. In particular, region_engine has no special merge behavior.

Runner-owned fields cannot be changed by an overlay: mode, node_id, storage.data_home, meta_client_options.metasrv_addrs, and wal.provider, plus wal.dir for Raft WAL or wal.broker_endpoints for Kafka WAL. The runner restores these fields to its baseline values, or deletes them when the baseline has no value. It warns about protected-field overrides without printing their values.

Version-Range Filtering

from_range and to_range control which binary versions a case applies to:

Entry	Meaning
"*"	Matches any version (including unknown).
"vX.Y.Z" or "=vX.Y.Z"	Matches exactly version X.Y.Z.
">=vX.Y.Z"	Matches X.Y.Z or later.
">vX.Y.Z"	Matches versions strictly later than X.Y.Z.
"<=vX.Y.Z"	Matches X.Y.Z or earlier.
"<vX.Y.Z"	Matches versions strictly earlier than X.Y.Z.

The range list is OR: a case matches if any entry matches.

Best-effort enforcement: The runner tries to determine the effective version:

• --from-version is used directly.
• --from-bins-dir / --to-bins-dir (or the default debug build) runs <binary> --version to infer the version.
• When the version cannot be determined (e.g. binary missing or --version fails), non-wildcard ranges are skipped with a message; wildcard (*) ranges still match.

Example (legacy_jsonb):

from_range = ["<=v1.1.0"]
to_range = [">=v1.1.1"]

This case only runs when the old binary is <= v1.1.0 and the new binary is >= v1.1.1.

CI Version Window

The CI job uses tests/compatibility/ci.toml to choose the small sliding window of recent released from versions to test against the PR-built to binary:

from_versions = ["v1.0.0", "v1.1.0"]

Keep this window small for PR and merge-queue latency: the goal is to catch upgrade compatibility issues from recent releases to the latest build, not to retest every historical version on every PR. Case-level from_range/to_range still decides which cases run for each version pair; the CI window only decides which old binaries are sampled. Broader historical windows belong in nightly or release-validation workflows.

The GitHub Actions workflow delegates the window loading and compat invocation to .github/scripts/run-compat.py; the workflow YAML should stay as a thin wrapper around artifact download/extraction and this script.

downgrade_to_versions optionally lists releases that CI restarts after the PR-built cluster. Those runs select only the downgrade_compatibility case in both distributed and standalone topologies.

setup.sql — Setup Phase (From Version)

SQL statements executed on the from version cluster. These must succeed (any error fails the case). Setup output is NOT compared against any result file.

Rules:

• Statements are semicolon-terminated
• -- prefix for ordinary comments
• -- SQLNESS ... interceptor comments follow ordinary sqlness semantics

verify.sql — Verify Phase (To Version)

SQL statements executed on the to version cluster. Output is compared against verify.result in sqlness snapshot style.

verify.result — Expected Output

Expected output in sqlness format. If this file is missing, the runner generates it from actual output and fails — the author must review, commit the generated file, and rerun.

<statement>;

<output>

<next statement>;

<output>

If output differs from expected, the run fails and verify.result is updated with actual output.

PR1 Limitations

• Sqlness interceptors: -- SQLNESS ... comments are applied per statement using the same interceptor registry as the ordinary sqlness runner, including the GreptimeDB PROTOCOL interceptor. For PROTOCOL POSTGRES, the namespace prelude uses SET search_path instead of USE. Avoid unqualified PostgreSQL-protocol table names starting with pg_: GreptimeDB's current PostgreSQL compatibility parser rewrites them to pg_catalog.<table>.
• Distributed topology: The compat runner starts 1 metasrv + 3 datanodes + 1 frontend + 1 flownode. Standalone compatibility runs need no external metadata store.
• No comment-based compat config: The compat runner does not define extra compatibility configuration in SQL comments; sqlness comments keep their normal sqlness meaning.

Namespace Isolation

Each case runs in its own database namespace to prevent cross-case interference:

• Default namespace is derived from the case directory name (sanitized to [a-z][a-z0-9_]*)
• Override with namespace in case.toml
• Duplicate namespaces are rejected at discovery time (before version filtering)
• Before each statement, the runner executes a namespace prelude (not written to verify.result): CREATE DATABASE IF NOT EXISTS <ns> via gRPC; then USE <ns> for gRPC/MySQL statements or SET search_path TO '<ns>' for PostgreSQL statements.

Batch Behavior

• The baseline (no-overlay) profile runs first. Cases whose old datanode TOML is semantically equivalent share one profile; profiles run serially and in isolation.
• Each profile has its own state and etcd lifecycle. Its overlay is applied only to old-stage datanodes and remains in effect through old-stage setup restarts. The current stage always uses a clean configuration.
• Cases run serially (no parallelism in PR1). Namespace state is session/protocol state and cannot be shared concurrently.
• Same namespace across cases is rejected.
• Without fail-fast, the runner verifies only cases whose setup succeeded. With fail-fast, it cleans up the active profile before stopping.
• --dry-run displays the selected profiles, cases, and sidecar paths without printing configuration values; it starts no services.

xfail Policy (Future)

For PR1, all cases are expected to pass. Future PRs will add xfail support with required issue and expiry fields.

Cross-Job Distributed State

PR1 runs setup and verify in the same job (same process). Cross-job artifact restore for distributed state is not supported in PR1 due to port randomization and etcd lease expiration.