mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-06 19:06:58 +00:00
discord9
58a67a47c3
* feat(promql): serialize raw series source bindings through Substrait Signed-off-by: discord9 <discord9@163.com> * feat(query): transport interleaved raw series fragments Signed-off-by: discord9 <discord9@163.com> * feat(promql): reconstruct interleaved raw series at repartition Signed-off-by: discord9 <discord9@163.com> * feat(query): plan distributed native raw series Signed-off-by: discord9 <discord9@163.com> * chore: satisfy clippy for the raw series transport The workspace clippy run rejects a redundant rebinding, a constant-size `chunks_exact`, and an implicit `String` clone in the tests these commits carry. No behavior changes. Signed-off-by: discord9 <discord9@163.com> * feat(query): execute native raw promql over real datanodes Plan a whole native statement as one fused node over raw series sources when the frontend reaches regions over the wire, including single-selector statements, while keeping the in-process split and segment shapes unchanged. Validate the whole tree with the runtime constructor before committing to that plan, so unsupported shapes still fall back to the legacy route. Bound the region reads of that topology with a query-wide permit budget of target_partitions, held until each region body is drained; other plan shapes keep unbounded reads rather than risk a body budget they could deadlock. Acceptance on a real two-datanode cluster with TCP frontend clients: time-partitioned series are reconciled before the fused node and match the unpartitioned control, with the DoGet peak at the target bound; series that no region shares keep contiguous per-series order and reach the fused node without a reconciling router. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the raw series transport PR The raw series Substrait message lives in greptime-proto PR #349, which is not merged, so pin its head commit instead of carrying a local path patch. Move the rev to the merge commit once #349 lands. https://github.com/GreptimeTeam/greptime-proto/pull/349 Signed-off-by: discord9 <discord9@163.com> * fix(promql): reconcile a discovered union of raw series A nameless selector reads its matches as one `Union`, and `UnionExec` reports the frame metadata of its last child. A time-partitioned metric sorted before a host-partitioned one therefore looked contiguous, and the fused node skipped the reconciliation of the fragmented member's series, silently folding region fragments as separate series. Lower the declaration on the fused edge to the weakest ordering any member declares, before the fused node reads it. The union itself is unchanged; the fused node's existing interleaved-input path inserts the router. A projection carries the lowered declaration because the pinned physical planner ignores a union's own schema. Cover it with a planner regression for both member orders and an all-complete control, and with a real two-datanode acceptance case that mixes a time-partitioned and a host-partitioned metric and checks the extrapolated increase per series. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover aggregates over discovered metrics A nameless selector's metric comes from its own frames, so a folding aggregate is a multi-input tree even with one selector: apply the same preflight and order-dependent refusals on the remote route. Cover `sum(increase(...))` and grouping by the metric name over the mixed discovery fixture. A grouping that names only `__name__` stays declined, since the output schema cannot name that group: here the metrics also differ by `site`, so grouping by the name and `site` exercises the two groups. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep remote multi-source refusals the split made Skipping the aggregate split and `segment_tree` over the wire must not drop their admission refusals. Two sources that fold groups by metric name would report groups that differ only by name under one label set, and an order-dependent operation over a discovered selector's union would answer in arrival order. Scope both refusals to the actual multi-source inputs rather than to the logical-table-id heuristic: a discovered selector reads several metrics through one selector, and a Mito table beside a metric one carries no logical id. Cover the Mito-plus-metric grouping refusal on a real two-datanode cluster, and cover a discovered `sum by (__name__)` over a raw selector (the name survives it) and a folding `sum(increase(...))`. Signed-off-by: discord9 <discord9@163.com> * docs: drop the local proto patch prerequisite The dependency is pinned to greptime-proto PR #349's commit, so the wire fixture no longer needs a local path patch. Signed-off-by: discord9 <discord9@163.com> * chore: finish the narthex rebase resolution Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and pass the engine state through the native planner's callers, while the remote route still declines the split it cannot carry. Signed-off-by: discord9 <discord9@163.com> * test: pass the engine state to the native operand test The native operand planner now takes the engine state, which decides whether a statement is planned for remote regions. Keep the histogram operand contract test on that signature. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the split and segment plans over the wire Remote planning disabled the aggregate split and segmentation because the split's producer read a bare scan. The producer reads whatever the statement prepared, and over the wire that is the raw series source: the partial half runs per partition on the frontend, the final half folds the exported states, and no aggregate state crosses the wire. Keep the split and let segmentation build evaluated segments from the raw sources instead of bare scans, so per-series window work keeps the scan width. Admit the resulting fused chain to the query-wide read budget, since each fused node polls its input while it is drained. Remove the recovery-projection wrapper: the analyzer no longer inserts that projection, and the physical planner consumes the frame boundary as it is. Serialization is now only the node's encoding: the codec no longer decides placement, and the metric-name binding is checked against the declared frames rather than the node types below. Signed-off-by: discord9 <discord9@163.com> * refactor(query): drop the in-process encodability exception A split aggregate encodes now, so the analyzer no longer needs a special case that skipped the encoding gate for it. The gate itself stays: every plan a region receives still has to encode. Signed-off-by: discord9 <discord9@163.com> * test: cover a repeated selector over one partitioned table Two occurrences of one time-partitioned metric keep their own raw source, time bounds and identity over the wire, so a ratio of two windows and a binary expression over two identical ones stay native and reconcile each occurrence's fragments before the fused node reads them. Compare both against the unpartitioned control. Signed-off-by: discord9 <discord9@163.com> * fix(table): refuse a scan that plans interleaved ordering A scan's planned ordering is a guarantee its own partitions keep. Interleaved is the transport annotation for fragments several producers contribute, so a scanner that planned it has no raw-series read whose frame would honour that declaration. Reject it in the scan conversion, while the frame codec keeps accepting interleaved raw frames from the router that reconciles them. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert the metric name beside the value `sum by (__name__)` over a discovered selector was asserted as a bare sorted list of values, so a plan that grouped by the wrong metric name - or dropped the name and folded everything into one series - could produce those same numbers. Read `__name__` and `val` by column name (the result puts `__name__` last, and a fold that drops the name drops the column rather than emptying it) and compare the `(name, value)` pairs against the fixture's own metric names. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the hybrid route for a partitioned pair A multi-selector tree over a partitioned table was admitted as one fused node for any pair of tables, not only for scans of one table. That preempted the frontend hybrid route, which copies the matcher's labels into each operand's scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08` depends on exactly that plan: its partitioned RHS holds a duplicated `job="other"` group, so the whole-native binary reports the duplicate instead of the value the hybrid route returns. Refuse the whole-fused plan for that shape in process again; the remote raw-source admission is unchanged, since a remote multi-selector tree over a partitioned table is the point of this branch and the distributed acceptance tests read it. The hybrid route then has to behave the same way it does without native operands: `matching_filters::propagate` now runs for them too, and a rewritten operand is re-planned through `try_plan_native_operand` with its original `selector_matcher` preserved, falling back to the ordinary plan when the native route declines. The unpartitioned case, whose duplicate group the whole-native binary is expected to report, is unchanged. The two new re-plan call sites put the native route's large future inside the recursive planner frame at every nesting level (~178 KB became ~209 KB), which overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where the new branch never ran. Box the future in a non-async helper and move the re-plan into `replan_rewritten_operand`, which brings the frame back to the old size without changing the behaviour. The partitioned-pair unit test asserted the removed shape. It now asks for the remote engine state, where a cross-table pair really is one fused node over two raw sources, and asserts that the same pair in process keeps the hybrid route. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the merged raw series transport GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as `f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit, in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an unmerged ref. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness bare -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness bare
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!