mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 16:54:09 +00:00
63dfaf8915cfa4f0fa76ba2d643e3da0752cb802
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
735401adc0 |
feat(mito2): consolidate columnar samples into bounded daily generations (#10129)
* perf(mito2): consolidate columnar samples into daily windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): isolate daily source coverage and version the local cache Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): use the exported index purge channel Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): exercise daily source groups across the merge limit Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): build stable daily groups from populated source windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): anchor daily coverage once per source group Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): validate partial-day reads and grouped rebuild cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): skip daily sample readers outside query bounds Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
58a67a47c3 |
feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait Signed-off-by: discord9 <discord9@163.com> * feat(query): transport interleaved raw series fragments Signed-off-by: discord9 <discord9@163.com> * feat(promql): reconstruct interleaved raw series at repartition Signed-off-by: discord9 <discord9@163.com> * feat(query): plan distributed native raw series Signed-off-by: discord9 <discord9@163.com> * chore: satisfy clippy for the raw series transport The workspace clippy run rejects a redundant rebinding, a constant-size `chunks_exact`, and an implicit `String` clone in the tests these commits carry. No behavior changes. Signed-off-by: discord9 <discord9@163.com> * feat(query): execute native raw promql over real datanodes Plan a whole native statement as one fused node over raw series sources when the frontend reaches regions over the wire, including single-selector statements, while keeping the in-process split and segment shapes unchanged. Validate the whole tree with the runtime constructor before committing to that plan, so unsupported shapes still fall back to the legacy route. Bound the region reads of that topology with a query-wide permit budget of target_partitions, held until each region body is drained; other plan shapes keep unbounded reads rather than risk a body budget they could deadlock. Acceptance on a real two-datanode cluster with TCP frontend clients: time-partitioned series are reconciled before the fused node and match the unpartitioned control, with the DoGet peak at the target bound; series that no region shares keep contiguous per-series order and reach the fused node without a reconciling router. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the raw series transport PR The raw series Substrait message lives in greptime-proto PR #349, which is not merged, so pin its head commit instead of carrying a local path patch. Move the rev to the merge commit once #349 lands. https://github.com/GreptimeTeam/greptime-proto/pull/349 Signed-off-by: discord9 <discord9@163.com> * fix(promql): reconcile a discovered union of raw series A nameless selector reads its matches as one `Union`, and `UnionExec` reports the frame metadata of its last child. A time-partitioned metric sorted before a host-partitioned one therefore looked contiguous, and the fused node skipped the reconciliation of the fragmented member's series, silently folding region fragments as separate series. Lower the declaration on the fused edge to the weakest ordering any member declares, before the fused node reads it. The union itself is unchanged; the fused node's existing interleaved-input path inserts the router. A projection carries the lowered declaration because the pinned physical planner ignores a union's own schema. Cover it with a planner regression for both member orders and an all-complete control, and with a real two-datanode acceptance case that mixes a time-partitioned and a host-partitioned metric and checks the extrapolated increase per series. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover aggregates over discovered metrics A nameless selector's metric comes from its own frames, so a folding aggregate is a multi-input tree even with one selector: apply the same preflight and order-dependent refusals on the remote route. Cover `sum(increase(...))` and grouping by the metric name over the mixed discovery fixture. A grouping that names only `__name__` stays declined, since the output schema cannot name that group: here the metrics also differ by `site`, so grouping by the name and `site` exercises the two groups. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep remote multi-source refusals the split made Skipping the aggregate split and `segment_tree` over the wire must not drop their admission refusals. Two sources that fold groups by metric name would report groups that differ only by name under one label set, and an order-dependent operation over a discovered selector's union would answer in arrival order. Scope both refusals to the actual multi-source inputs rather than to the logical-table-id heuristic: a discovered selector reads several metrics through one selector, and a Mito table beside a metric one carries no logical id. Cover the Mito-plus-metric grouping refusal on a real two-datanode cluster, and cover a discovered `sum by (__name__)` over a raw selector (the name survives it) and a folding `sum(increase(...))`. Signed-off-by: discord9 <discord9@163.com> * docs: drop the local proto patch prerequisite The dependency is pinned to greptime-proto PR #349's commit, so the wire fixture no longer needs a local path patch. Signed-off-by: discord9 <discord9@163.com> * chore: finish the narthex rebase resolution Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and pass the engine state through the native planner's callers, while the remote route still declines the split it cannot carry. Signed-off-by: discord9 <discord9@163.com> * test: pass the engine state to the native operand test The native operand planner now takes the engine state, which decides whether a statement is planned for remote regions. Keep the histogram operand contract test on that signature. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the split and segment plans over the wire Remote planning disabled the aggregate split and segmentation because the split's producer read a bare scan. The producer reads whatever the statement prepared, and over the wire that is the raw series source: the partial half runs per partition on the frontend, the final half folds the exported states, and no aggregate state crosses the wire. Keep the split and let segmentation build evaluated segments from the raw sources instead of bare scans, so per-series window work keeps the scan width. Admit the resulting fused chain to the query-wide read budget, since each fused node polls its input while it is drained. Remove the recovery-projection wrapper: the analyzer no longer inserts that projection, and the physical planner consumes the frame boundary as it is. Serialization is now only the node's encoding: the codec no longer decides placement, and the metric-name binding is checked against the declared frames rather than the node types below. Signed-off-by: discord9 <discord9@163.com> * refactor(query): drop the in-process encodability exception A split aggregate encodes now, so the analyzer no longer needs a special case that skipped the encoding gate for it. The gate itself stays: every plan a region receives still has to encode. Signed-off-by: discord9 <discord9@163.com> * test: cover a repeated selector over one partitioned table Two occurrences of one time-partitioned metric keep their own raw source, time bounds and identity over the wire, so a ratio of two windows and a binary expression over two identical ones stay native and reconcile each occurrence's fragments before the fused node reads them. Compare both against the unpartitioned control. Signed-off-by: discord9 <discord9@163.com> * fix(table): refuse a scan that plans interleaved ordering A scan's planned ordering is a guarantee its own partitions keep. Interleaved is the transport annotation for fragments several producers contribute, so a scanner that planned it has no raw-series read whose frame would honour that declaration. Reject it in the scan conversion, while the frame codec keeps accepting interleaved raw frames from the router that reconciles them. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert the metric name beside the value `sum by (__name__)` over a discovered selector was asserted as a bare sorted list of values, so a plan that grouped by the wrong metric name - or dropped the name and folded everything into one series - could produce those same numbers. Read `__name__` and `val` by column name (the result puts `__name__` last, and a fold that drops the name drops the column rather than emptying it) and compare the `(name, value)` pairs against the fixture's own metric names. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the hybrid route for a partitioned pair A multi-selector tree over a partitioned table was admitted as one fused node for any pair of tables, not only for scans of one table. That preempted the frontend hybrid route, which copies the matcher's labels into each operand's scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08` depends on exactly that plan: its partitioned RHS holds a duplicated `job="other"` group, so the whole-native binary reports the duplicate instead of the value the hybrid route returns. Refuse the whole-fused plan for that shape in process again; the remote raw-source admission is unchanged, since a remote multi-selector tree over a partitioned table is the point of this branch and the distributed acceptance tests read it. The hybrid route then has to behave the same way it does without native operands: `matching_filters::propagate` now runs for them too, and a rewritten operand is re-planned through `try_plan_native_operand` with its original `selector_matcher` preserved, falling back to the ordinary plan when the native route declines. The unpartitioned case, whose duplicate group the whole-native binary is expected to report, is unchanged. The two new re-plan call sites put the native route's large future inside the recursive planner frame at every nesting level (~178 KB became ~209 KB), which overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where the new branch never ran. Box the future in a non-async helper and move the re-plan into `replan_rewritten_operand`, which brings the frame back to the old size without changing the behaviour. The partitioned-pair unit test asserted the removed shape. It now asks for the remote engine state, where a cross-table pair really is one fused node over two raw sources, and asserts that the same pair in process keeps the hybrid route. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the merged raw series transport GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as `f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit, in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an unmerged ref. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
|
6a40c0b811 |
feat(mito): implement columnar samples format (#9887)
* docs(mito): specify columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): implement columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): address columnar samples review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use crate-rooted samples imports Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
7309800aef |
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
c879564671 |
chore(common-query): drop the raw-series request extension transport (#9815)
* chore(query): drop the prepared raw-series RPC skeleton The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(common-query): drop the raw-series request extension transport The prepared raw-series RPC carried a `RawSeriesRequest` through a query-context extension, and with that RPC gone (`607de914464`) nothing in `src/` writes or reads the extension any more. What remains is the binding the table scan actually uses: `RawSeriesRequest`, `RawSeriesTag`, `bind`, `validate` and `same_encoding_identity`, unchanged. Deleted: the extension key, `take_raw_series_request`, and the two serialization methods that only existed to encode and decode that transport, plus the wire fixture whose only reader was the test for them. The fixture never shipped — the commit that introduced it is in no tag. The tests that covered `validate` are kept, with their rejections now asserted directly instead of through a JSON round-trip: an unsupported version, an empty field list, a duplicate field, and a non-field column id are all still refused. The binding test is untouched, including its order, identity and `Arc` assertions. `pub` items in a library crate never trip `dead_code`, so the compiler could not have found this; the greps above are the evidence. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
1156cfbe0c |
chore(query): drop the prepared raw-series RPC skeleton (#9797)
The raw-series preparation RPC was merged into `narthex` for a plan that never landed: no server or frontend dispatches `greptime.prepare_raw_series.v1`, and the three `Datanode` methods, `bind_raw_series_query` and the `QueryEngine` trio have no call sites and no implementors. `PreparedNamedRawSeries` had no user outside them. The wire envelope never shipped -- no tag contains the commit that added it -- so the compatibility fixtures go with it. The harness drives `wire/` fixtures from the Rust tests that own their context conversion rather than from SQLness discovery, and has no retirement mechanism to record instead. Left in place: `RawSeriesRequest` and its extension key, `raw_series_frame_ordering`, `unpack_raw_series` and `RawLabelSource`, all of which the surviving series-frame path still uses. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
997cbf87f7 |
feat(common-query): declare raw series frame ordering in schema metadata
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |