Lei, HUANG
045cf74c3f
* perf(mito2): merge compaction inputs on a separate task Compaction read, merged and encoded each output on a single task, so parquet encoding waited for the merge and the merge waited for encoding. The merge now runs on its own compaction runtime task and hands merged batches to the SST writer through a bounded channel. The channel also closes when that task panics, so the reading side awaits the task before reporting the end of the input. A panic fails the compaction instead of producing a truncated SST. One output of 4.8M rows took 1.28s instead of 1.65s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): parallelize decoding and encoding of compaction outputs Parquet encoding took about two thirds of a compaction output's time and ran on a single thread. When `experimental_compaction_output_parallelism` is above 1, the SST writer encodes the columns of each row group on up to that many blocking threads of the compaction runtime while the next row group is buffered, and the merge decodes up to that many input files ahead of it on compaction runtime tasks. The parallel writer feeds every column writer the same input as `ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group reach the object store writer one row group later, so `max_file_size` may split a file one row group later. The option defaults to 1, which keeps today's CPU usage. One output of 4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): cancel compaction producers before consumer polling Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): support per-table compaction output parallelism Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoding tasks before returning failures Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime. Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoders on SST write errors Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error. Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain compaction outputs on cancellation Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures. Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness bare -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness bare
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!