mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-07-10 07:50:39 +00:00
12f83828b1
* feat: add Prom remote-write query regression scenario Signed-off-by: discord9 <discord9@163.com> * test: add high-cardinality remote-write query case Signed-off-by: discord9 <discord9@163.com> * feat: chunk remote-write query regression loads Signed-off-by: discord9 <discord9@163.com> * test: use multi-day remote-write regression case Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: allow large query regression comments Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
128 lines
4.0 KiB
Markdown
128 lines
4.0 KiB
Markdown
# Direct-SST fixture format
|
|
|
|
The phase-1 generator should be a reusable fixture generator, not a collection
|
|
of issue-specific data loaders.
|
|
|
|
## Inputs
|
|
|
|
Each case describes:
|
|
|
|
```toml
|
|
[case]
|
|
name = "example_metric"
|
|
description = "example direct-readable-SST regression"
|
|
|
|
[scenario]
|
|
kind = "direct_readable_sst"
|
|
seed = 12345
|
|
|
|
[[scenario.tables]]
|
|
database = "public"
|
|
name = "example_metric"
|
|
engine = "mito"
|
|
append_mode = true
|
|
sst_format = "flat"
|
|
primary_key = ["host", "instance"]
|
|
time_index = "ts"
|
|
|
|
[[scenario.tables.columns]]
|
|
name = "host"
|
|
type = "STRING"
|
|
semantic = "tag"
|
|
distribution = { kind = "cardinality", values = 100, prefix = "host" }
|
|
|
|
[[scenario.tables.columns]]
|
|
name = "value"
|
|
type = "DOUBLE"
|
|
semantic = "field"
|
|
distribution = { kind = "deterministic_wave", min = 0.0, max = 100.0 }
|
|
|
|
[[scenario.tables.columns]]
|
|
name = "ts"
|
|
type = "TIMESTAMP(9)"
|
|
semantic = "timestamp"
|
|
|
|
[scenario.layout]
|
|
regions = 1
|
|
sst_count = 1024
|
|
rows_per_sst = 4096
|
|
row_group_size = 512
|
|
time_range_layout = "non_overlapping_per_sst"
|
|
series_layout = "round_robin"
|
|
|
|
[[scenario.queries]]
|
|
name = "count_all"
|
|
kind = "sql"
|
|
query = "SELECT count(*) FROM example_metric"
|
|
warmup = 0
|
|
iterations = 1
|
|
```
|
|
|
|
`series_layout = "round_robin"` advances the timestamp once per generated row and
|
|
cycles series labels across rows. `series_layout = "timestamp_major"` writes all
|
|
series for one timestamp before advancing to the next timestamp; use it for
|
|
Prometheus-like high-cardinality scrape fixtures where short query windows should
|
|
still contain many raw samples. `timestamp_major` requires `rows_per_sst` to be
|
|
divisible by `series_count`.
|
|
|
|
`[scenario]` is required. Other scenario variants are intentionally unsupported
|
|
for now, but `scenario.kind` leaves room for future `write_then_query` and
|
|
`cache_warm_query` configuration.
|
|
|
|
The generator should use these declarations to produce:
|
|
|
|
- object-store SST files written through the real Mito SST writer
|
|
- manifest checkpoint and `_last_checkpoint`
|
|
- fixture summary with file IDs, row counts, time ranges, and generated schema
|
|
|
|
## Why this is generic
|
|
|
|
The same fixture format should support different query-regression families:
|
|
|
|
- PromQL/TQL time-index pushdown
|
|
- SQL predicate pruning
|
|
- projection and row-group pruning
|
|
- series scan behavior
|
|
- joins or aggregation over controlled layouts
|
|
|
|
Issue-specific behavior belongs in case configs and thresholds, not in the
|
|
generator implementation.
|
|
|
|
## Direct generation vs realism
|
|
|
|
Direct SST generation is phase-1 because it is fast and reproducible:
|
|
|
|
- SST count, file time ranges, row groups, and label distributions are fixed by
|
|
the case config.
|
|
- The same fixture can be used for base and candidate builds.
|
|
- Large pruning-sensitive datasets can be created without spending CI time on
|
|
ingestion, memtable flush, or compaction.
|
|
|
|
It is less realistic than ingestion-path data because it bypasses writes,
|
|
memtables, flush scheduling, and compaction. That tradeoff is intentional for
|
|
PR-level query regression. A later nightly/release suite can add ingestion-based
|
|
cases for end-to-end realism.
|
|
|
|
Multi-table cases are supported by generating one fixture directory per table.
|
|
Each table is still limited to one region, and the runner passes `--table`, the
|
|
discovered `--region-id`, and the discovered `--table-dir` for each table before
|
|
materializing all generated region subtrees into the same datanode data home.
|
|
This supports JOIN regression cases without changing existing single-table case
|
|
files.
|
|
|
|
Multi-table cases must use unique table names and unique `(database, name)`
|
|
pairs. The runner derives each fixture subdirectory from table index, database,
|
|
and table name, sanitizing path-unsafe characters to avoid collisions and unsafe
|
|
paths.
|
|
|
|
The preferred compatibility path is:
|
|
|
|
1. create an empty table with the target build to seed catalog/table metadata;
|
|
2. stop the process;
|
|
3. generate readable SSTs and replacement manifest checkpoints offline using the
|
|
seeded region metadata;
|
|
4. restart and query the fixture.
|
|
|
|
Fully synthetic metadata is useful for generator smoke tests, but seeded metadata
|
|
is safer for end-to-end query performance cases.
|