Lei, HUANG
da14ee21cf
* fix(mito2): look up row group stats by parquet leaf index for nested columns On flat-format tables a logical column can expand to multiple parquet leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per promoted path). ParquetFlat used the logical column index in the SST schema directly as the leaf index when reading row group statistics, so min/max/null stats of every column after a nested column were read from wrong leaves. When the misplaced leaf held order-compatible statistics (e.g. a small Int64 JSON path vs. the timestamp window predicate), min-max pruning dropped whole row groups by mistake. SWCS compaction reads inputs with a time window predicate, so it silently lost all rows of such files; plain queries with time-range predicates were affected as well. Map each column to its first parquet leaf column and report NoStats for columns with multiple leaves, which makes pruning conservative for them. Add a unit test and a sqlness regression case that reproduces the data loss on the unfixed binary. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): skip nested root stats and correct SWCS regression baseline Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve statistics leaves for primary-key SST readers Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): distinguish known null counts from unknown statistics Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): check JSON2 time pruning before SWCS compaction Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): validate all business columns after JSON2 SWCS Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> (cherry picked from commit743261f05e) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit113db823e8) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness bare -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness bare
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!