greptimedb/tests/perf/fixture-format.md

Direct-SST fixture format

The phase-1 generator should be a reusable fixture generator, not a collection of issue-specific data loaders.

Inputs

Each case describes:

[case]
name = "example_metric"
description = "example direct-readable-SST regression"

[scenario]
kind = "direct_readable_sst"
seed = 12345

[[scenario.tables]]
database = "public"
name = "example_metric"
engine = "mito"
append_mode = true
sst_format = "flat"
primary_key = ["host", "instance"]
time_index = "ts"

[[scenario.tables.columns]]
name = "host"
type = "STRING"
semantic = "tag"
distribution = { kind = "cardinality", values = 100, prefix = "host" }

[[scenario.tables.columns]]
name = "value"
type = "DOUBLE"
semantic = "field"
distribution = { kind = "deterministic_wave", min = 0.0, max = 100.0 }

[[scenario.tables.columns]]
name = "ts"
type = "TIMESTAMP(9)"
semantic = "timestamp"

[scenario.layout]
regions = 1
sst_count = 1024
rows_per_sst = 4096
row_group_size = 512
time_range_layout = "non_overlapping_per_sst"
series_layout = "round_robin"

[[scenario.queries]]
name = "count_all"
kind = "sql"
query = "SELECT count(*) FROM example_metric"
warmup = 0
iterations = 1

Value distributions

Tag columns use cardinality and produce one label per series. Double field columns accept either a NULL-free wave:

distribution = { kind = "deterministic_wave", min = 0.0, max = 100.0 }

or a wave with periodic NULLs:

distribution = { kind = "nullable_wave", min = 0.0, max = 100.0, null_every = 4 }

nullable_wave also accepts null_every = 0 to disable NULLs. It writes NULL on every null_every-th sample of each series and wave values everywhere else, so a NULL is one missing sample of one series rather than a missing row. The cadence is counted in samples of the series, not in generated rows, so every series carries the same null ratio under any series_layout; when one SST holds a multiple of null_every samples per series (rows_per_sst / series_count), the pattern also repeats with the same period in every SST. Counting the cadence in generated rows instead would make it constant per series whenever series_count is a multiple of null_every (128 and 4, for example), leaving series either entirely NULL or entirely NULL-free. Counting per series keeps NULL and non-NULL samples mixed inside range-function windows, which is what the nullable branches of the PromQL range functions observe.

Query kinds

kind = "prom_http" runs a Prometheus range query by POSTing form fields to /v1/prometheus/api/v1/query_range. query, start, end, and step are sent as form fields; start and end define the range, and step defines its evaluation interval. start, end, and step are optional in the case model and default to empty form values when omitted. The table database is sent as the db URL query parameter, rather than as a form field.

The measurement starts before the HTTP request is sent and ends after the full response body is read and parsed as JSON. It therefore includes request send, server-side Prometheus JSON response building, and response-body read and JSON parsing; it is not server-side-only latency.

series_layout = "round_robin" advances the timestamp once per generated row and cycles series labels across rows. series_layout = "timestamp_major" writes all series for one timestamp before advancing to the next timestamp; use it for Prometheus-like high-cardinality scrape fixtures where short query windows should still contain many raw samples. timestamp_major requires rows_per_sst to be divisible by series_count. per_sst uses one series selected from the SST index. These describe logical generation order. Before either flat or primary-key SST format is written, each completed generated batch is physically sorted by encoded primary key ascending, timestamp ascending, and sequence descending, as required by the Mito Parquet writer; sorting preserves every generated column and row, including deterministic-wave values.

[scenario] is required. Other scenario variants are intentionally unsupported for now, but scenario.kind leaves room for future write_then_query and cache_warm_query configuration.

The generator should use these declarations to produce:

• object-store SST files written through the real Mito SST writer
• manifest checkpoint and _last_checkpoint
• fixture summary with file IDs, row counts, time ranges, and generated schema

Why this is generic

The same fixture format should support different query-regression families:

• PromQL/TQL time-index pushdown
• SQL predicate pruning
• projection and row-group pruning
• series scan behavior
• joins or aggregation over controlled layouts

Issue-specific behavior belongs in case configs and thresholds, not in the generator implementation.

Direct generation vs realism

Direct SST generation is phase-1 because it is fast and reproducible:

• SST count, file time ranges, row groups, and label distributions are fixed by the case config.
• The same fixture can be used for base and candidate builds.
• Large pruning-sensitive datasets can be created without spending CI time on ingestion, memtable flush, or compaction.

It is less realistic than ingestion-path data because it bypasses writes, memtables, flush scheduling, and compaction. That tradeoff is intentional for PR-level query regression. A later nightly/release suite can add ingestion-based cases for end-to-end realism.

Multi-table cases are supported by generating one fixture directory per table. Each table is still limited to one region, and the runner passes --table, the discovered --region-id, and the discovered --table-dir for each table before materializing all generated region subtrees into the same datanode data home. This supports JOIN regression cases without changing existing single-table case files.

Multi-table cases must use unique table names and unique (database, name) pairs. The runner derives each fixture subdirectory from table index, database, and table name, sanitizing path-unsafe characters to avoid collisions and unsafe paths.

The preferred compatibility path is:

1. create an empty table with the target build to seed catalog/table metadata;
2. stop the process;
3. generate readable SSTs and replacement manifest checkpoints offline using the seeded region metadata;
4. restart and query the fixture.

Fully synthetic metadata is useful for generator smoke tests, but seeded metadata is safer for end-to-end query performance cases.