rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-17 17:32:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(perf): report bounded mixed BSS measurements and query regressions

Browse Source 
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
This commit is contained in:
discord9
2026-09-08 18:12:13 +08:00
parent 149b49de5f
commit 38c105c14f
2 changed files with 130 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/perf/README.md
+16 -12
View File
@@ -400,8 +400,11 @@ byte-stream-split (BSS) physical table. It writes 1,000 series × 4,320 samples
unique values. This deliberately matches a 95/5 design; it is not an empirical
claim about any production population. A separate survey sample found 94.79%
integral values, but that observation does not make its values globally unique.
Historical unique-value findings are preserved, but superseded pending a mixed
workload rerun, in [`query_cases/sst_float_bss/RESULTS.md`](query_cases/sst_float_bss/RESULTS.md).
Current mixed-data evidence, including SST inspection, exact-bit row verification,
warmed endpoint SQL, and both warm-reader projections, is recorded in
[`query_cases/sst_float_bss/RESULTS.md`](query_cases/sst_float_bss/RESULTS.md).
The historical unique-integer workload is preserved only in Git history and does
not apply to this case.
Both targets must use the exact same release `greptime` binary; only the
per-target table setup SQL differs. Run the existing driver from a checkout
containing that binary, with absolute paths and fresh data directories for every
@@ -440,8 +443,8 @@ frontend/cache measurement. The case restricts storage inspection to
stop, it also runs seven iterations of value-only `parquetbench` for all
inspected SST files and sequential `scanbench` over the corresponding regions,
with parallelism one. The three explicit flushes yielded six SSTs in the observed
run—three 1,105,920-row files and three 368,640-row files—because storage split
each flush; this is an observation, not a guaranteed file layout. Their per-run
mixed runs—three 1,105,920-row files and three 334,080-row files—because storage split each
flush; this is an observation, not a guaranteed file layout. Their per-run
output and aggregate `parquetbench_median_average_ms` and
`scanbench_median_average_ms` are under
`targets[].read_bench`; they are quiescent local-file read/scan diagnostics, not
@@ -449,14 +452,15 @@ warmed frontend-query latency measurements. Bench averages include iteration one
no OS cache is dropped, and the driver runs base before candidate.
Historical [PR #8548](https://github.com/GreptimeTeam/greptimedb/pull/8548)
reported storage savings alongside warm-read slowdowns. Its mixed counter/gauge
study used value-only and all-column projections, discarded the first iteration,
and alternated target order. This smaller integer-counter case is not an exact
reproduction: do not compare its inclusive averages directly with that study's
warm medians. For a matching reader comparison, reuse the recorded bench commands
against the stopped data directories, run both projections with eight iterations,
alternate target order, and compare medians of iterations 2–8 from raw output.
Keep post-flush and post-compaction measurements separate.
reported storage savings alongside warm-read slowdowns for a different mixed
counter/gauge study. It used value-only and all-column projections, discarded the
first iteration, and alternated target order. This case reuses that reader
measurement approach, not its dataset or results. For the supplementary warm
comparison, reuse recorded commands against stopped data directories with eight
iterations, discard iteration one, and alternate target order for eight rounds.
For parquetbench, sum all per-file warm medians before taking the outer median;
for scanbench, take the outer median of whole-region warm medians. Keep
post-flush and post-compaction measurements separate.
The case's `-5.0` storage target and `25` query-latency guardrail are experimental
acceptance targets, not observed-benefit claims; do not relax them if a run
tests/perf/query_cases/sst_float_bss/RESULTS.md
+114 -120
View File
@@ -1,144 +1,138 @@
# SST float BSS: historical unique-integer results
# SST float BSS: mixed-data post-flush evidence
> Superseded workload: the current case now generates bounded, fluctuating
> 95% integer-valued / 5% fractional series. The numbers below belong only to
> the earlier global-unique-integer experiment, not the current case. Mixed
> workload measurements are pending.
This report replaces the earlier unique-integer experiment as the primary evidence
for `sst_float_bss`. The detailed historical numbers remain in Git history; they
apply only to that superseded workload and must not be used for the current mixed
case.
## Decision context
## Case and data shape
This evidence supports a **minimal opt-in configuration, default off**. It shows a
large storage reduction for this controlled case, a SQL trade-off, and faster
stopped-directory warm readers; it does not establish a universal float benefit.
- Generator/case revision: `149b49de5f3`.
- Base and candidate used the same release `greptime` binary. The only physical
table difference was `experimental_sst_float_field_encoding`: `default` versus
`byte_stream_split`; base used dictionary encoding and candidate used BSS
without dictionaries. This is not a dictionary-policy-held-constant comparison.
- Each of three fresh post-flush runs wrote 1,000 series × 4,320 samples at
60-second intervals: 4,320,000 rows per target. Three explicit flushes produced
six SSTs per target: three × 1,105,920 rows and three × 334,080 rows, with 45
`greptime_value` row groups total. This is observed layout, not a layout
guarantee.
- `bounded_mixed` is a synthetic, uncalibrated design: a series hash selects its
baseline and span band, and 60-sample anchor interpolation produces bounded
fluctuation. It does not represent a measured production distribution.
- Exactly 950 equal-length series are integer-only and 50 are fractional-only,
yielding 4,104,000 integer rows and 216,000 fractional rows per target. The
equal-length 1,000-series design deliberately makes both the series and row
shares 95%/5%.
- The 95% design is not an empirical claim. A [local preview survey project](http://192.168.50.85:8765/view/prometheus-metrics-186-project-preliminary)
observed 94.79% integral **samples**; it does not establish a global rate,
global uniqueness, or a 95% series distribution.
## Compared build and case
## Initial SST inspection and value verification
- Same release `greptime` binary for base and candidate: SHA-256
`2ce0ab1670cd4bdb0f60c87af3e1f494ee66c3c6ab83bce943a073638514eb2b`.
- GreptimeDB `1.3.0-alpha.1`, source commit `dd4734d7b2ca4468f2c8a50e063fc9e7db53794b`.
- BSS case/harness source: `54f26ea8153`.
- The only DDL difference was `experimental_sst_float_field_encoding`:
`default` (base) versus `byte_stream_split` (candidate). Both used
`compaction.twcs.trigger_file_num = 100`.
- Three fresh runs wrote 1,024 series × 4,320 samples at 60 s intervals:
4,423,680 rows per target. Values were unique monotonic integer-valued
`DOUBLE`s.
- Each run used three 1,440-sample chunks and an explicit flush after each.
The observed layout was six SSTs: three × 1,105,920 rows and three ×
368,640 rows, compressed with ZSTD level 1. This is an observed post-flush
layout, not a promised layout.
Inspection is limited to `data/greptime/public`; bytes exclude WAL, indexes,
metadata, and internal/private-table storage. Each of the three runs had the
same totals:
## SST and correctness results
The inspection covered `data/greptime/public` only. Each target had six selected
SSTs, 4,423,680 rows, and 45 inspected `greptime_value` chunks. Footer checks
found no BSS encoding in base and BSS in candidate; row-count, logical-schema,
and SQL result checks passed. The expected and returned `sum(greptime_value)`
was `9,784,470,159,360` for both targets. This verifies counts and aggregate-query
results, not an exhaustive row-by-row comparison. Base chunks used dictionary
encoding; candidate chunks used BSS without dictionaries, so this is not a
codec-only comparison with dictionary policy held constant. The bytes below
exclude WAL, indexes, metadata, and internal-table storage.
| Fresh run | Base SST bytes | Candidate SST bytes | Candidate change |
| Measure | Default | BSS | BSS change |
| --- | ---: | ---: | ---: |
| formal-1 | 13,830,799 | 612,899 | -95.5686% |
| formal-2 | 13,830,799 | 612,884 | -95.5687% |
| formal-3 | 13,830,799 | 612,884 | -95.5687% |
| Selected SST bytes | 20,443,131 | 8,884,743 | -56.5392% |
| `greptime_value` compressed bytes | 20,201,188 | 8,643,964 | -57.2106% |
| SST files / rows / value row groups | 6 / 4,320,000 / 45 | 6 / 4,320,000 / 45 | — |
The configured storage threshold was at least a 5% reduction (`-5%`); all three
runs passed it.
Footer inspection found `PLAIN`/`RLE`/`RLE_DICTIONARY` for the default value
column and `RLE`/`BYTE_STREAM_SPLIT` for BSS (both ZSTD level 1). The configured
storage threshold remains a 5% reduction; this repeated SST-size result passes
that threshold. It is post-flush evidence only, not a compaction result or a
general float-compression claim.
For every base/candidate group in all three runs, rows were compared by the same
primary key and timestamp and their Float64 bits were exact matches: 950
integer-only and 50 fractional-only series, 4,104,000 and 216,000 rows
respectively. The six canonical sorted-row SHA-256 results are identical:
```
48f87d7d01bc8a0ce9a9cdb3033f322e787784fcae61f1417ac25b05c4a40375
```
This verifies exact cross-target stored rows, not independent reconstruction of
the generator output. All 1,000 series had both rises and falls, and every one
of the 36 inspected SSTs contained both integer-valued and fractional values;
these were not stored as two disjoint file sets.
## Warmed endpoint SQL
Each query had three warmups followed by 15 measured endpoint requests. Values
below are base/candidate median and p95 latency in milliseconds; delta is
`(candidate median - base median) / base median × 100%`.
Each query used three warmups followed by 15 measured endpoint requests. Values
are median/p95 milliseconds; delta is `(BSS median - default median) / default
median × 100%`. The guardrail remains a maximum 25% candidate median regression and
was not changed for these runs.
| Run | Query | Base median / p95 | Candidate median / p95 | Delta |
| --- | --- | ---: | ---: | ---: |
| formal-1 | `sum_all_values` | 7.23 / 8.36 | 6.55 / 7.52 | -9.4% |
| formal-1 | `hourly_sum_values` | 23.11 / 28.71 | 28.76 / 34.74 | +24.4% |
| formal-2 | `sum_all_values` | 6.77 / 7.70 | 7.61 / 8.76 | +12.4% |
| formal-2 | `hourly_sum_values` | 25.48 / 33.30 | 29.26 / 30.61 | +14.8% |
| formal-3 | `sum_all_values` | 8.97 / 10.38 | 9.12 / 9.68 | +1.6% |
| formal-3 | `hourly_sum_values` | 33.44 / 43.80 | 36.89 / 44.14 | +10.3% |
| Run | Query | Default median / p95 | BSS median / p95 | Delta | Guardrail |
| --- | --- | ---: | ---: | ---: | --- |
| mixed-1 | `sum_all_values` | 27.13 / 37.38 | 25.45 / 38.99 | -6.21% | pass |
| mixed-1 | `hourly_sum_values` | 118.78 / 136.63 | 130.93 / 185.26 | +10.23% | pass |
| mixed-2 | `sum_all_values` | 24.02 / 35.02 | 36.34 / 56.01 | +51.31% | **fail** |
| mixed-2 | `hourly_sum_values` | 95.58 / 120.40 | 85.45 / 148.03 | -10.60% | pass |
| mixed-3 | `sum_all_values` | 38.25 / 55.41 | 19.68 / 29.49 | -48.55% | pass |
| mixed-3 | `hourly_sum_values` | 56.77 / 148.29 | 121.87 / 143.16 | +114.66% | **fail** |
The SQL guardrail was a maximum 25% candidate median regression; all six checks
passed. `hourly_sum_values` was slower in all three runs despite passing.
The two failures mean this evidence does **not** establish performance acceptance
or a safe degradation. These are noisy measurements on a local development host;
there was no CPU isolation or cache flush, and no cause of the variation has been
proved.
## Stopped-directory warm readers
These are real `greptime datanode parquetbench` and sequential `scanbench`
measurements against the complete landed SST sets, not endpoint SQL latency.
For each of three formal datasets, the reader script used eight outer rounds,
eight inner iterations per command, discarded iteration 1, computed the median
of iterations 2–8, then summed the six per-file parquet medians before taking
the median across outer rounds. Thus the parquet statistic is a **sum of
per-SST warm medians**, not a whole-region latency. `scanbench` instead covered
the whole region, taking the outer median of its eight warm medians.
For each of the three datasets, eight outer rounds alternate base/candidate order.
Each command runs eight iterations, discards the first, and takes the median of
iterations 2–8. Parquetbench reads every SST: sum the six per-file medians, then
take the median of the eight round totals. Scanbench scans the whole region and
reports the outer median of its eight warm medians. These are distinct metrics,
not interchangeable whole-query latencies. Deltas are ratios of separately
aggregated medians. Value projection selects `greptime_value`; all-column
parquetbench returns five physical columns, while scanbench uses `{}` (all
region columns). Projection order is fixed; target order alternates.
Both projections were measured: `value` reads only `greptime_value`;
`allcolumns` selects all columns (five physical columns in direct parquetbench;
all region columns in scanbench). The targets alternated order. No
arbitrary base/candidate pair was selected. Across the three datasets this was
672 commands × 8 iterations = 5,376 timed iterations.
| Dataset | Projection | Reader | Base ms | Candidate ms | Delta |
| Dataset | Projection | Reader | Base ms | BSS ms | Change |
| --- | --- | --- | ---: | ---: | ---: |
| formal-1 | value | parquetbench | 55.946 | 18.995 | -66.05% |
| formal-1 | value | scanbench | 75.772 | 40.161 | -47.00% |
| formal-1 | allcolumns | parquetbench | 65.515 | 28.804 | -56.03% |
| formal-1 | allcolumns | scanbench | 120.528 | 85.226 | -29.29% |
| formal-2 | value | parquetbench | 111.607 | 35.158 | -68.50% |
| formal-2 | value | scanbench | 140.960 | 77.098 | -45.31% |
| formal-2 | allcolumns | parquetbench | 120.564 | 72.044 | -40.24% |
| formal-2 | allcolumns | scanbench | 226.479 | 224.709 | -0.78% |
| formal-3 | value | parquetbench | 71.076 | 20.752 | -70.80% |
| formal-3 | value | scanbench | 79.159 | 42.435 | -46.39% |
| formal-3 | allcolumns | parquetbench | 72.522 | 34.300 | -52.70% |
| formal-3 | allcolumns | scanbench | 137.906 | 89.151 | -35.35% |
| mixed-1 | value | parquetbench | 95.957 | 68.442 | -28.67% |
| mixed-1 | value | scanbench | 144.293 | 115.421 | -20.01% |
| mixed-1 | allcolumns | parquetbench | 176.620 | 103.761 | -41.25% |
| mixed-1 | allcolumns | scanbench | 200.881 | 202.660 | +0.89% |
| mixed-2 | value | parquetbench | 112.151 | 74.275 | -33.77% |
| mixed-2 | value | scanbench | 109.824 | 78.189 | -28.80% |
| mixed-2 | allcolumns | parquetbench | 141.944 | 102.681 | -27.66% |
| mixed-2 | allcolumns | scanbench | 211.414 | 163.261 | -22.78% |
| mixed-3 | value | parquetbench | 121.827 | 67.565 | -44.54% |
| mixed-3 | value | scanbench | 100.968 | 79.878 | -20.89% |
| mixed-3 | allcolumns | parquetbench | 113.984 | 94.361 | -17.22% |
| mixed-3 | allcolumns | scanbench | 193.101 | 164.175 | -14.98% |
Warm results varied substantially, especially formal-2 all-column scanbench:
its eight round medians ranged from 120.158–422.669 ms (base) and
85.224–243.650 ms (candidate); candidate was slower in rounds 7 and 8.
These deltas are ratios of separately aggregated medians, not medians of paired
percentage changes. Scanbench's projection is verified from commands/configs,
not a printed schema. Target order alternated, but projection order stayed fixed.
The runs used a non-isolated local development host, with no CPU affinity or
cache drop. No specific cause of the variability was established.
All 672 commands / 5,376 raw iterations returned the expected rows; timing,
projection configurations, parquet schemas and order metadata were checked.
Run 1 all-column scanbench was **0.89% slower**. Its round medians ranged from
167.197–663.465 ms (base) and 133.339–557.352 ms (BSS). There is substantial
variability; these measurements do not establish a stable gain on every read
path. Runs used a non-isolated development host without CPU affinity or cache
drops. The source of the variability was not established.
## Scope, comparison, and artifacts
## Scope and recorded checks
This is a post-flush test of monotonic integer-valued `DOUBLE`s. It is not a
general float-compression result, and no compaction result was tested.
This case compares default and BSS post-flush data only. It adds no benchmark
framework and no public artifact upload. Historical unique-integer results and
other mixed counter/gauge studies have different data and methods; they are
context only and do not apply to this workload.
[PR #8548](https://github.com/GreptimeTeam/greptimedb/pull/8548) used mixed
counters and gauges and reported 25.66% storage savings with 11.36–41.34% warm
slowdown. Its different dataset and methodology make these results neither a
direct comparison nor a contradiction.
Recorded checks for this change set: 37 targeted Rust tests, 20 Python tooling
tests, and 15 plans passed. Full-workspace tests were not run.
The local artifacts were not uploaded. They are rooted at
`/mnt/nvme_rust/rust-targets/metric-bss-perf/experiments/`:
Local artifacts are under
`/mnt/nvme_rust/rust-targets/metric-bss-perf/experiments/mixed-{1,2,3}/`,
including `value-verification.json`, `verification.txt`, and each run's
`sst_float_bss/query-regression-report.json`.
- `formal-{1,2,3}/sst_float_bss/query-regression-report.json` and
`prepare-remote.json`: SST, schema/data, endpoint-SQL, and read-bench data.
- `warm-comparison.json`: the 12-row warm-reader table above.
- `warm-formal-{1,2,3}/summary.json`: raw warm-reader summaries.
- `environment.txt`: host and binary checksums.
Reproduce with the existing commands in `tests/perf/README.md` and the external
artifact scripts `tools/warm_readers.py` and `tools/verify.py`; this result adds
no new benchmark framework.
## Verification recorded for this case
- Release build/check: passed.
- Targeted `cargo nextest`: 35 tests passed.
- Python tooling tests: 20 tests passed.
- Plan validation: 15 plans passed.
- Existing default smoke driver: recorded status 0; both targets returned 16 rows.
- All three formal drivers: recorded status 0 and report `ok`; artifact verification passed.
The outer Python driver always exits 0, so its exit code alone is not validation.
No full-workspace test suite or Clippy run is claimed here.
Warm artifacts are in sibling `warm-mixed-{1,2,3}/` directories;
`mixed-warm-comparison.json` contains the table and per-round ranges.
`mixed-file-composition.json` records integer/fractional counts for each SST.
The shared release `greptime` SHA-256 is
`2ce0ab1670cd4bdb0f60c87af3e1f494ee66c3c6ab83bce943a073638514eb2b`.