Weny Xu
f26e128b8a
* ci: update Agent and Traces benchmark runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use unrestricted benchmark runtimes by default
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: retry benchmark image pulls before generating datasets
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: increase tracesbench defaults to 8 CPU and 32 GiB
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: configure tracesbench query timeout and update runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime and configure import scan workers
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use shared parallel tracesbench import preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime to preserve Tempo trace requests
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: summarize tracesbench query results and lifecycle
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: show benchmark resource usage and update parallel loader runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): resolve shared summary module in existing test harness
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: select tracesbench engines and GreptimeDB image version
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update benchmark runtime with parallel Agent preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: select benchmark runtime registry by cloud provider
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: retry long-range benchmark image pulls
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: compare baseline and candidate GreptimeDB benchmarks
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: pin published dual-version benchmark runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: prepare agent benchmark indexes with updated runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: validate agent admission and render traces version comparison
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use runtime with traces index preparation
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use consolidated runtime for benchmark version reports
Signed-off-by: WenyXu <wenymedia@gmail.com>
* test: add license header to benchmark comparison tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): read legacy Tempo ingestion metrics in traces summaries
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: require tracesbench query plan artifacts
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use optimized tracesbench report runtime
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: update tracesbench runtime to merged report validation
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 6e1a137411)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness bare -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness bare
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!