mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-14 07:52:34 +00:00
fa5f44ac6094af69f50d77cf07fae03b78ec2691
762
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  DeviousCardiandClaude Opus 5.5
|
d6474b961c |
fix(promql): apply offset to subquery evaluation window (#9364)
* fix(promql): apply offset to subquery evaluation window
`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.
Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.
`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.
Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.
Closes #9330
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>
* docs(promql): correct the subquery-offset rationale and pin the histogram shift
Follow-up to
|
||
 dennis zhuang
|
079ffec116 |
chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove unused RouteInfoCorrupted error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 discord9
|
28e415a103 |
feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors The planner previously dropped the `at` field of selectors (`at: _`), so `some_metric @ 300` silently returned step-following values instead of the samples anchored at the fixed timestamp. Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch` semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply `offset` to the anchor, rewrite the selector offset to `eval_start - anchor`, scan only the anchored window, then report the same window at every evaluation step via a grid-wide-replay `InstantManipulate`. `@ start()` / `@ end()` resolve against the statement's evaluation range (`stmt_start` / `stmt_end`), which subquery planning does not rewrite. Timestamps before the Unix epoch are accepted; unrepresentable ones are rejected with `AtModifierTimestampOutOfRange` instead of wrapping. Selectors without `@` are planned exactly as before. Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): center predict_linear on the evaluation timestamp predict_linear_impl used the window's last sample time as the evaluation timestamp, and the UDF took only (ts_range, value_range, t) with no channel for the current instant. After a range selector is folded for @ (or with an offset) the same window is replayed at every step, so the prediction stayed constant at the anchor's answer; even a plain window ended before the step produced a stale value. Give prom_predict_linear a 4th argument carrying the step's evaluation instant (ms timestamp), derived in the planner from the row's time index plus the offset the window was folded with (at_offset for @, offset_ms otherwise, recorded on PromPlannerContext). The regression is centered on that instant, matching Prometheus' use of enh.Ts. The mixed float/native-histogram path forwards the same argument. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): tighten @ modifier planner and predict_linear - range_fold_offset is always a concrete millisecond offset, not an Option; the single-step replay helper no longer wraps an infallible plan in Result. - predict_linear's eval timestamp is always cast to Timestamp(ms) at the call site, so the UDF drops its dead Int64 branch and the extra func_name argument. - Drop two redundant comparison cases from the at_modifier sqlness test and fix the lookback window comment to the half-open (0s, 300s]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): accept parse-time rejection of @ on Windows `@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime holds it and the planner rejects the anchor it cannot represent, but a Windows SystemTime tops out near 1.8e12 seconds, so the parser's checked_add fails first and the same literal is rejected while parsing. Accept either rejection path so the test passes on both platforms. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): avoid replaying label_join over rewritten series keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): narrow anchored range call promotion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address @ modifier review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): satisfy super import format check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): address at modifier review follow-ups Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f956f4da30 |
refactor(promql): split planner into focused modules (#9354)
* refactor(promql): move planner tests into separate module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move function-specific planner methods into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move OR planner method into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): move binary island planner into module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): keep binary result labels in planner root Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): limit planner helper visibility to parent module Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): group set operator planning and localize imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): use crate-rooted planner imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
678aa81cae |
fix(client): complete transport lane isolation (#9030)
* fix(client): complete transport lane isolation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(client): deprecate legacy single-manager constructors Per review: mark the legacy single-manager constructors and helper as deprecated so callers move to the isolated query/control manager pair. Tests intentionally exercising the legacy path are annotated with `#[allow(deprecated)]`. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(client): update deprecated constructor callers for CI Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4ffd93d8dc |
fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch) below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then dropped or rewrote that fetched coalesce, losing the global limit. Root-caused and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo pins to that fix commit and adds regression coverage. - pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix) - global_limit.rs: adapt to new DF API (input_distribution_requirements, child_distribution, replace_children+Recompute); accept HashPartitioned and KeyPartitioned as partitioning to restore; add regression unit tests - sqlness: extend standalone common ssts and refresh distributed ssts_limit result for the preserved fetch plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: drop dead HashPartitioned compat arms and test wrapper - remove HashPartitioned compatibility matches (pinned DF only emits KeyPartitioned); drop their #[expect(deprecated)] attributes - remove single-use agg_with_limit helper whose seed limit is always overwritten by the soft-limit transform - restore Cargo.lock dependency edges unrelated to the DataFusion pin bump (cargo update --precise had downgraded 26 unrelated packages) Addresses oracle ablation review. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(deps): pin DataFusion to merged fetch preservation fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Ning Sun
|
20619273ee |
chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates Move the workspace from the pinned nightly-2026-03-21 to stable 1.96.1 and drop all 23 '#![feature]' gates across 13 crates, rewriting the still-unstable API usages with stable equivalents: - try_blocks: closures / an async block (table, query, common-function, servers) - duration_constructors: Duration::from_secs(n * 86400) / (n * 60) - iterator_try_collect: collect::<Result<Vec<_>, _>>() - box_patterns: as_deref() + matches! chains (sql) - error_iter: error_chain_root() source-chain walker (common-error); sources() includes the error itself, so the walker never panics - int_roundings: div_floor -> div_euclid (equal for positive divisors) - iter_partition_in_place: stable sort_by_key partition helper (index) - hash_set_entry: HashSet::insert bool / contains+insert - trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin) - string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned() - never_type: Infallible (common-recordbatch) - debug_closure_helpers: closure-backed DebugFmt newtype (mito2) - binary_heap_pop_if: peek().is_some_and() + pop() - exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds - stmt_expr_attributes: stale gate, no usages Also fix release-dev-builder-images.yaml, which parsed rust-toolchain.toml with a date-only regex and would produce empty image versions with a stable channel; it now extracts the full channel token. Dev-builder images verified against stable 1.96.1 (image build, default-toolchain behavior, binstall/nextest, riscv64 and android targets, in-image cargo check). Validated on 1.96.1: cargo check --workspace --all-targets, clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, and nextest on all 13 affected crates (4586 passed). Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging first. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: update flake checksum * chore: use wild for linker in flake --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
discord9
|
abedeb2bea |
fix(query): keep count_values generated label in enclosing expressions (#9223)
* fix(query): keep count_values generated label in enclosing expressions
`count_values("v", m)` projects the generated label as a real output
column, but did not register it in `ctx.tag_columns`. Enclosing
expressions (abs/round/+1/topk/label_replace/vector join) rebuild their
projection from `ctx.tag_columns` and silently drop the label.
Register the generated label in `ctx.tag_columns` after the projection,
and give it the same qualifier as other tag columns so qualified
references resolve. PromQL overwrites an input label with the same name,
so drop any existing tag with that name first to avoid duplicate column
ambiguity.
Fixes https://github.com/GreptimeTeam/greptimedb/issues/9181
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-1
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): use total_cmp in count_values test helper
Silence clippy::needless_borrow on partial_cmp(&right.1); f64 sorting
uses total_cmp, matching the other planner test helpers.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): keep count_values generated label as sink table primary key
FindGroupByFinalName::f_up only renamed a group key when the projection
aliased the key column directly. count_values now projects its generated
label as a unary UDF over the sampled column
(prom_float_to_string(value) AS label) while the aggregate still groups
by the raw column, so the name match failed and the sink table lost the
label from its PRIMARY KEY, demoting it to a DOUBLE value column.
Allow a projection above the aggregate to rename a group key by deriving
its output from a single group-key column (unary scalar function or cast
over that column only). Multi-column expressions, case, aggregates,
windows, subqueries and literals are still rejected so a computed column
cannot be mistaken for the group key.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): group count_values by the formatted sample value
The numeric count_values branch grouped by the raw sample column and only
formatted the value into the generated label in the post-aggregate
projection. Two distinct raw values that collapse to one label text
(e.g. BIGINT 9007199254740992 and 9007199254740993, both 9007199254740992
in Float64) were split into two groups, each emitting the same label set
at one timestamp, violating Prometheus' unique-label-set-per-timestamp
invariant.
Return the formatted value expression (prom_float_to_string, with a
CAST to Float64 for non-Float64 inputs) as previous_field_expressions so
the aggregate groups by the same expression that produces the label,
mirroring the existing mixed float/native-histogram precedent.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): drop needless borrow in count_values test helper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): don't replace group key with derived unary expr
FindGroupByFinalName treated any unary expression of a single group
column (scalar fn, Cast, TryCast) as a rename of that group key and
swapped it in as the sink primary key. A derived expression such as
lower(host) is many-to-one, so distinct groups (HOST_A vs host_a) would
collapse to the same primary key and be silently deduplicated.
Narrow the matching to direct name-matched aliases of the actual group
expression only, and drop the derived-unary path (is_unary_expr_of_column)
and the allow_derived flag. count_values already groups by the formatted
expression name, so it is unaffected.
Add a regression asserting host survives and host_lc does not replace it
under both optimizer settings.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
dennis zhuang
|
def0c2ec5c |
fix: only push down aggregates grouped by the partition columns themselves (#9337)
* fix: only push down aggregates grouped by the partition columns themselves Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep grouping sets on the frontend for partitioned tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
88197f4019 |
fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings A vector-vector binary operation projected one operand's whole tag set and inner-joined without any cardinality check, so `on()`/`ignoring()` did not reduce the result labels, `group_left`/`group_right` changed nothing, and a non-unique match group produced a cross product that PromQL cannot represent. Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the matching labels, `ignoring(...)` drops them, and a group modifier keeps the many side's labels plus the `group_x(...)` labels taken from the one side. A label the one side does not carry is deleted from the result. The reduced label set no longer identifies the operand series, so `__tsid` is dropped from the context on this path. Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)` window and a scalar UDF that fails the query on a repeated group: on the one side before the join, and on the result labels after it, matching where Prometheus raises each of its three errors. Series are unique by their whole tag set, so the window is only planted when the match keys drop a tag; plain arithmetic and `on(<all tags>)` plan exactly as before. Closes #9207, closes #9208, closes #9209. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): keep __tsid when the result labels are an operand's whole tag set Deriving the result labels dropped `__tsid` from the context unconditionally, so an enclosing operation fell back to joining on the tag columns even where the column still identified the result series. Keep it when every result label comes from one operand and covers that operand's whole tag set: no other operand value reaches the labels, and the matching gives each of its rows a single partner, so its `__tsid` is still one per result series. That is the common `on(<all tags>)` and bare `group_left` shape; a matching that actually drops a tag still clears it. The column is re-qualified as the result's own, which is how the enclosing expression and the context look it up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the match group count column unambiguous The cardinality check aliased its row count to a fixed `__promql_match_group_count`. An operand carrying a label of that name made the window output two fields with the same name, and planning failed with "Schema contains qualified field name collide_right.__promql_match_group_count and unqualified field name __promql_match_group_count which would be ambiguous". Pick a name the operand does not already have, the way the `or` operator allocates its match key columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover a match group spread over several regions The cardinality check runs above the merge of the region scans, so it counts a match group globally. Nothing pinned that: every table in these cases holds a single region, and a check evaluated per region would pass them all. Partition the operand on a column outside the match keys, which puts the two series of one match group in different regions, and assert both the pre-join and the post-join check still reject it. The case runs in the distributed environment too, where the regions sit on different datanodes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): group an outer aggregate on the labels the operand kept `by`/`without` planning topped up a missing grouping column by walking down to the table scan and re-projecting it. That is right for a column the scan pruned for efficiency, but the labels a matching modifier deletes are also absent from the operand's output, and they were restored the same way: sum without(host) (a / on(host) b) `on(host)` leaves the operand with `host` alone, so the sum covers everything and Prometheus answers `{} 10`. Instead `device` came back from the scan under `a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for `sum by(device)` of that operand, which has no `device` to group on at all. Take the grouping labels from the operand's own label set rather than from the row keys of the scan beneath it. A label pruned from the plan is still in that set and still gets restored; a label the operand dropped is not. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): drop the unreachable aggregation tag top-up `by`/`without` planning could restore a grouping column that the plan no longer carried by rewriting the scan underneath it. Once the grouping labels come from the operand's own label set, there is nothing left for it to restore: a scan projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds matcher columns to that set), so a label in the set is always in the schema. Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the standalone and the distributed environment. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): drop cases that guarded the removed tag top-up Three plain selector aggregates were there to show that restoring a pruned grouping column still worked. With the restore gone they only repeat what the aggregate cases already cover. Also fix a comment that still said the metric engine scan prunes tag columns: it projects every label of the operand. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): note the duplicate a propagated matcher hides A matcher copied onto the one-side operand removes groups without a partner before the cardinality check sees them, so a duplicate in such a group is not reported. Prometheus checks every group of the one side and fails the query. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Weny Xu
|
a35f9d5c78 |
fix: address Windows test failures and run full Windows CI (#9305)
* fix: use relative object keys for Windows filesystem access Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: cover Windows path and time limits in full test CI Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: use relative keys in metadata snapshot tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
   
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
25bca4609f |
perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
233e23b352 |
fix(promql): keep value-field grouping labels out of matching filter propagation (#9242)
* fix(promql): keep value-field grouping labels out of matching filter propagation #9202 propagates matching-label matchers to the scanned selector using the planned operands' tag columns. For an aggregated operand those are its grouping labels, and agg_modifier_to_col resolves by(...) names against the input schema only, so a value field named in by(...) is reported as a tag. A value field varies between the samples of one series, so lowering a matcher on it below sample selection (PromInstantManipulate) can drop the newest sample, promote a stale one from the lookback window, and fabricate a match the un-rewritten query does not produce. Track the by(...) labels that name value fields of the aggregated operand's input in PromPlannerContext::aggregation_field_labels, and refuse to propagate matchers on them while still requiring a matcher to name a grouping label to cross an aggregation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: sync matching_filter result fixture comment with the PR number Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9e9a8cac20 |
fix(query): insert MergeScan into nested scalar subqueries (#9261)
* fix(query): insert MergeScan into nested scalar subqueries DataFusion 55 keeps uncorrelated scalar subqueries as expression subqueries (enable_physical_uncorrelated_scalar_subquery, default true) instead of decorrelating them into joins, and executes them via the new physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked the plan with a plain TreeNode transform that does not descend into expression subqueries, so MergeScan was only inserted for depth-1 subqueries. A scalar subquery nested inside another scalar subquery kept a bare frontend DistTable TableScan and failed at execution with "Unsupported operation: get stream from a distributed table". Use the subquery-aware transform so handle_subquery (PlanRewriter / MergeScan insertion) runs for subquery plans at every nesting depth. Fixes #9260. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen nested scalar subquery regression coverage Address review on #9261: - Replace the ineffective 'no bare TableScan' string check with a real subquery-aware plan walk (apply_with_subqueries); MergeScan hides its remote input from traversal, so any TableScan the walk reaches was genuinely left unwrapped. - Add a distributed regression case on a range-partitioned table so the nested inner aggregate must merge partial results across regions (global AVG feeding an outer SUM filter). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
fys
|
e82cb0af74 |
feat(json2): respect type hints during JSON2 type concretization (#9222)
* fix: prefer JSON2 type hints for uncast read pushdown * feat(query): materialize JSON get result types before planning * fix(query): apply JSON2 type hints to parsed paths * fix(query): apply JSON2 type hints before distributed planning * chore: remove f * refactor: the code style of json_get_type_hint * chore: add sqlness case * fix: add json expr planner, remove json get type hint * chore: add sqlness test * fix(query): respect JSON2 type hints in query planning * chore: add more sqlness cases * fix: cargo clippy * fix: cargo clippy * chore: update sqlness test result |
||
|
dennis zhuang
|
007836ba7a |
fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query
`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.
Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.
Only metric engine tables are covered; other engines share no column space
to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): batch-resolve metric names by table id
Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.
The catalog walk stays, but only to find the physical tables to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): read an absent label as the empty string
A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.
Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.
This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): don't panic on pre-epoch label value bounds
`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.
Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): drop applicable_matchers, share the distinct scan
With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.
Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
dennis zhuang
|
b1106a9c5a |
perf(promql): propagate matching-label filters between binary operands (#9202)
* perf(promql): propagate matching-label filters between binary operands A one-to-one arithmetic binary expression inner-joins its operands on the matching labels, so every row that survives the join already satisfies the other operand's equality matchers on those labels. Copy those matchers to the other operand so both scans drop non-joining series before execution instead of feeding them to the join. Both operands are planned before the rewrite: a selector matcher can also constrain a value field, and only the planned contexts tell tags and fields apart. A matcher is copied only when its name is a tag column on both sides, its value is non-empty, and it is one of the matching labels. Operands are limited to vector selectors, parentheses, and label-preserving range functions applied directly to a matrix selector; `ignoring(...)`, group modifiers, fill values, set and comparison operators, regex matchers and `or` matcher groups are left alone. Selector matchers are now deduplicated in place instead of through a `HashSet`, so the generated scan filter keeps a stable order once a selector carries more than one matcher. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): propagate ignoring, non-equality and aggregated matchers Widens the matcher propagation added in the previous commit to the shapes it was leaving on the table. The join compares matching labels with plain column equality (`normalized_match_key_expr` and its coalescing are confined to `or`), so any predicate on a matching label is already enforced on both sides for every surviving pair: it originates on one operand, and the join carries it to the pairs it forms. Copying it to the other operand can only drop rows that had no surviving partner. That argument does not depend on the matcher kind, so regular expressions, negations and empty values now propagate too. `=~".*"` stays out: it lowers to no filter at all, so copying it would only force a re-plan. `ignoring(...)` is no longer rejected. A label is a join key exactly when it is a tag on both sides and not named in `ignoring`, which is what `binary_join_key_columns` computes and what the caller can now answer from the two planned contexts. Operands may now be aggregations that partition by their grouping labels, which is the shape most real queries use. `agg_modifier_to_col` rewrites `ctx.tag_columns` to the grouping labels, so a label found in an aggregated operand's tags is a group key, and filtering the aggregate's input by it drops exactly the corresponding output groups. `topk`, `bottomk` and `limitk` are excluded because they select across a group and carry input labels through -- `prom_topk_bottomk_to_plan` leaves `ctx.tag_columns` alone, so the tag check cannot catch them. `count_values` is excluded because it adds an output label that does not exist in its input. The rollup whitelist now covers every label-preserving range function the planner implements, and finds the matrix argument by position so multi-argument rollups such as `quantile_over_time` and `predict_linear` qualify. `absent_over_time` stays out: it synthesizes a series from the matchers when its input has none. Re-running the sqlness case against an unmodified planner produces a byte-identical `.result` for all 27 queries. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin that excluded matching labels stay on their own operand `ignoring(device)` and `on(host)` both leave `device` out of the join keys, so a `device` matcher must not reach the other operand. Neither case was covered end to end: a regression there silently drops the left operand's `eth1` series instead of returning them, which the two added queries now catch. Also corrects the comment at the rewrite site, which still described re-planning as touching a leaf selector. Aggregated operands are re-planned as a whole; what holds is that the rewrite only ever adds matchers to a selector, so the enclosing operand's table reference, time index and field columns are unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): log why matching-filter propagation was skipped Splits the guard chain into `try_propagate`, whose `Err` names the reason the rewrite does not apply, and logs it once in `propagate`. Debugging a query that did not get the filter no longer means stepping through the guards. The reasons also replace the comments that used to explain the same conditions, and the remaining comments lose the parts that restated the code or repeated each other. Two of them were wrong rather than verbose. Saying `topk`'s input "must not be filtered" reads as a claim about PromQL: `topk(1, m{host="x"})` is perfectly legal, and what matters is that filtering before `topk` changes the candidate set it ranks. Saying the subset-matching case "keeps its many-to-many result" read as an endorsement of behaviour Prometheus rejects outright; it now records that the query is a cross product here and points at #9209. The propagation cannot change whether such a check would fire: series in one match group agree on every matching label, so a matcher over one of those labels keeps all of them or drops all of them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep copied matchers out of the operand's selector metadata Re-planning an operand with a copied matcher also rebuilt its `PromPlannerContext::selector_matcher`, and that context is what an enclosing expression reads. `create_absent_plan` turns the equality matchers found there into the labels `absent()` reports, so absent(counter_metric{host="missing"} / on(host, device) gauge_metric) gained a `host="missing"` label that `main` does not produce. The inner expression was empty either way; only the reported label set changed. The copied matcher belongs to the scan, not to the operand's identity, so both re-planned contexts now keep the matchers their operand was written with. `selector_matcher` has one other reader, `create_table_scan_plan`, which consumes it while the operand is being planned and is unaffected. Reported by @discord9. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 jeremyhi
|
dca01654bc |
feat: prepare and execute database Metric exports (#9180)
* feat: prepare and authorize captured database exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: bound database export jobs and drain failures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: validate database export identity and restore equivalence Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate database export directory URLs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve Windows export directory paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject local database export filename aliases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: consolidate database export planning policies Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: restore escaped database export filenames Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: align database restore assertions with shared policies Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify database export boundaries and names Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 LFC
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
39e34c0b74 |
refactor: migrate partition_statistics to statistics_from_inputs (#9175)
DataFusion 55 deprecated ExecutionPlan::partition_statistics in favor of statistics_from_inputs with StatisticsContext::compute. Migrate the two remaining GreptimeDB overrides, RegionScanExec and MergeScanExec, and update the RegionScanExec test to the new API, dropping its allow(deprecated). Behavior is unchanged: RegionScanExec keeps the append-mode-only, exact-source-rows statistics gate from #9154, and MergeScanExec keeps reporting unknown statistics for per-partition requests. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
d185a8bf76 |
feat: use exact sequence ranges for incremental Flow reads (#9165)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
8af3a04ed7 |
fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
dennis zhuang
|
8e45809b6d |
perf(query): check Substrait encodability once per plan (#9095)
`PlanRewriter::should_expand` encoded every node it ascended past to Substrait. Each call clones the plan, runs an analyzer over it and builds a fresh `SessionState` with all default features, so a deep PromQL plan pays that cost once per level. Encode the root once in `PlanRewriter::new` and skip the per-node call when it succeeds. `to_substrait_plan` recurses from the root to the leaves, and `should_expand` only receives sub-trees that `f_down` pushed on the stack untouched, so a root that encodes proves the whole descent encodes. A root that does not encode falls back to the per-node check, which is what locates the node that has to stay on the frontend. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
7152ca9264 |
fix(promql): skip NULL samples and fix counter extrapolation order (#9118)
* fix(promql): treat NULL field values as absent samples in range functions Range functions read the value column through `Float64Array::values()`, which returns the raw buffer and ignores the null bitmap. A NULL field value means the series has no sample at that timestamp, so the padding under a null slot (0.0 in practice) was counted as a real sample. `rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`, `quantile_over_time` and `avg_over_time` now work on samples instead of slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL slot, and tokio swallowed the panic so the query returned success with that series missing. Null handling is gated on `null_count() == 0` over the whole backing array, checked once per batch, so tables without NULLs keep the existing code path. `deriv` and `predict_linear` already had this guard through `linear_regression_slices` and are untouched. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep other fields when one has no sample in the window Review follow-up. Two things surfaced once range functions started returning NULL for a window without samples. The filter after a function call required every field column to be non-NULL, so on a multi-field table one field with no samples in a window would drop the other fields' results with it. It now keeps a row when any field has a sample, which is the shape a selector already emits. On a single field column the two predicates are identical. `quantile_over_time` returned NaN rather than NULL for a window without samples, so the row survived that filter. Prometheus returns an empty vector there, so the emptiness check now sits in the range UDF; the shared quantile kernel still yields NaN for an empty slice, matching upstream's `quantile()` helper that `quantile_aggr` depends on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): correct two comments on the null handling The planner one did not say why `preserve_any_value` is hardcoded at that call site, which is the question a reader arrives with. The `quantile_over_time` one described the empty-window behaviour while sitting on the `has_nulls` line, and that behaviour had moved into `window_quantile`. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): clamp extrapolation before snapping a counter to zero Prometheus clamps `durationToStart` to half an average interval once the first sample is past the extrapolation threshold, and only then lets the counter zero-snap shorten it further, so the snap can never lengthen the leading extrapolation. Running the snap first let it rescue a duration the clamp should have cut, and `rate` and `increase` over-extrapolated to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream gives 0.375 and this returned 0.5. `extrapolation_matches_prometheus_on_seeded_windows` carries a line-by-line port of upstream `extrapolatedRate` as an oracle and diffs it against the UDF over seeded windows, so the order stays pinned. `factor` also picked up upstream's guard against a zero sampled interval, which previously divided by zero. Two sqlness results move, both verified against the upstream algorithm. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): fold range_presence_null into null_samples #9104 landed its own NULL-sample case whose data is the same series this one already used: one host with interior NULLs, one host with nothing but NULLs. Keeping both means two files asserting the same semantics on the same rows. The merged case keeps every query from both, so the presence functions still cover the trailing-NULL window, the count of two, the empty left-open window at t=7, and the all-NULL windows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
7f949f48c0 |
fix(promql): preserve native timestamps through sample selection (#9070)
* fix(promql): preserve native timestamps through sample selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): retain column indices in instant plan ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): update native precision plan expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): preserve selector output column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): verify preserved selector output order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): refresh native timestamp explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply PromQL offsets without native timestamp overflow Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover negative PromQL offsets at native timestamp bounds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: allow native precision instant LastRow selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: discard unused bounds for empty range intersections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve native time bounds independently for LastRow Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify native LastRow predicates and overflow through SQL Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain native PromQL selection and scan invariants Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): address timestamp helper and stream review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): pass selector offsets explicitly from planner Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): retain explicit offset in payload overflow regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): record inner-offset subquery SQL results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
38577c8854 |
refactor: support plugin-backed admin functions (#9083)
* feat: add bulk load SQL extension points Add `CREATE BULK LOAD` parsing and `BulkLoadHandler` dispatch. Expose private system table factories through `src/catalog`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: support plugin-backed admin functions Replace the custom bulk-load SQL hooks with access to process `Plugins` through `FunctionState`. Preserve typed external errors from ADMIN functions in `src/operator/src/error.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
da5cb1a190 |
perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: guard instant last row correctness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update instant query explain results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply last row after source deduplication Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: scope post-merge last row selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover instant PromQL last row Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): sort generated SST rows before writing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover instant last row selection in sqlness Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): avoid last row hints for lossy timestamp casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): preserve stale marker semantics across flushes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): decode dictionary labels in stale regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): avoid reserved column name in stale fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): exercise LastRow hints and filtered results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: keep after-merge mode in LastRow selector Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: reject instant LastRow across residual filters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: expect after-merge selector in instant vector guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain instant LastRow filter eligibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restrict instant LastRow to safe selector nodes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: show LastRow merge mode directly in diagnostics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh LastRow display in explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
ad7b0ace64 |
feat(flow): support eval schedule offsets (#8878)
* feat(flow): support eval schedule offsets Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): remove redundant schedule assertion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(flow): trim eval offset compatibility scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): trim eval offset edge coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(flow): trim eval offset comment noise Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): address eval offset review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover Flow eval offset persistence Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
84bd993131 |
refactor(json2): concretize JSON2 schemas at merge scan boundaries (#9016)
* refactor(json2): concretize JSON2 schemas at merge scan boundaries Infer concrete JSON2 output types from remote plans and expose them on MergeScanLogicalPlan before physical planning. Recompute affected local schemas and remove the JSON2-specific rewrite from MergeScanExec. Add SQLness coverage for whole JSON2 columns in windows and joins. Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 shuiyisong
|
b86da3d35f |
feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): update sqlness results for normalized label matching Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: derive temporality label from default column prefix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(promql): add analyze coverage for delta temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): scope label alignment to temporality marker Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: handle count-only histograms and vector broadcasts Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude temporality marker from entity descriptions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: use a fixed label for OTLP aggregation temporality Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): preserve mixed-range semantics for raw delta Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
LFC
|
d62a5a990a |
feat(json2): support list indexing for JSON2 columns (#9013)
feat(query): support list indexing for JSON2 columns Extend JSON2 paths through DataFusion field-access planning, including nested list indexes and object fields following an index. Preserve Variant reads for bracket JSONPath expressions and normalize dot accesses after subscripts to work around the current DataFusion planner limitation. Add unit and sqlness coverage for nested indexes, type conflicts, missing paths, flushes, and compacted SSTs. Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
LFC
|
15317a131b |
feat(json2): support JSON2 paths in SQL functions (#9007)
feat(query): support JSON2 paths in SQL functions Update the DataFusion fork to expose scalar function planning hooks. Infer JSON2 path output types from scalar, aggregate, and window function signatures, while preserving the default Utf8View behavior for functions that accept arbitrary inputs. Add unit and sqlness coverage for type conflicts, mixed typed and untyped JSON paths, filters, aggregates, and window functions. Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
9c135ebcb3 |
feat!: stabilize streaming analyze metrics (#8966)
* feat: stabilize streaming analyze metrics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: expose analyze memory usage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: simplify analyze stream handling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve analyze stream sequence on panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: log analyze stream worker panic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
529f046110 |
refactor: json2 v2 storage layout (#8979)
* refactor: json2 v2 storage layout Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * rethinking when "needs_remainder" Signed-off-by: luofucong <luofc@foxmail.com> * restore "ReadColumns" Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
shuiyisong
|
c4dafb5826 |
fix(promql): resolve derived labels in aggregation arithmetic (#8994)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
00d43b29ad |
feat(query): add experimental DataFusion spill-to-disk controls (#8884)
* feat(query): add experimental DataFusion spill-to-disk controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(config): regenerate configuration reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update config API for spill defaults Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address spill configuration review Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): preserve spill settings with runtime plugins Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
d32cd77505 |
fix: postgres describe for more statements (#8974)
* fix: postgres describe for more statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: cover more show statements Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments - add missing `clippy::too_many_arguments` allow on `query_from_information_schema_dataframe` (CI clippy failure) - take `&ShowKind` in the information-schema dataframe helper so `kind` is no longer cloned at every call site; only the WHERE arm (which needs an owned expression for `sql_to_expr`) clones internally - document why re-applying TQL explain formats never overwrites an existing value (per-query context state) Signed-off-by: Ning Sun <sunning@greptime.com> * chore: trim comments to essentials Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
discord9
|
beded6e232 |
fix(flow): avoid insert select HTTP/2 stalls (#8962)
* fix(flow): avoid insert select HTTP/2 stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): share record batch forwarding for DML Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
b31f05eb59 |
fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955)
* chore(deps): update tokio-postgres * fix: describing fetch cursor and analyze |
||
|
dennis zhuang
|
1851f6bf4d |
fix(query): keep INSERT timestamp conversion out of the source query (#8911)
* fix(query): keep INSERT timestamp conversion out of the source query Interpreting an INSERT's string timestamps used to work by pushing the conversion down into the source query, which changed what that query means. Two consequences: - Pushing through a UNION's DISTINCT moved the dedup key from the raw strings to parsed instants, so rows spelling the same instant differently collapsed into one. On an append-only table that is a silently dropped row. - A UNION branch that needed no conversion (a NULL, or an explicit cast) made the whole column give up, leaving sibling branches on UTC while the rest of the row used the session timezone. Convert at the assignment instead, by routing its cast through a timezone-carrying timestamp type and back. Arrow applies the timezone when a cast target carries one, and stripping it afterwards preserves the value. The source query is no longer touched, so both cases go away and the tree-walking rewrite (roughly 160 lines) is deleted. The rewrite reads source types, so it now runs TypeCoercion first: a UNION still carries its loose per-branch schema before coercion, and retargeting a cast whose input later becomes a timestamp would shift the value rather than reinterpret it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): address review on INSERT assignment rewrite - Clone the input `Arc` instead of the whole subtree, and only rebuild it when a `Values` row actually changes. - Defer cloning the cast source until the literal-folding path has been ruled out. - Move the UTC check onto `Timezone::is_utc`, replacing a bare string compare. - Cover a prepared `INSERT ... VALUES (?)`: an untyped placeholder types as `Null`, so the assignment cast is left for parameter substitution. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
LFC
|
932f87f7a8 |
refactor(json2): support querying v2 storage layout (#8940)
* feat(json2): support querying v2 storage layout - route missing JSON2 paths to the v2 remainder field - reconstruct complete values from explicit fields and remainder data - preserve root JSON2 columns across projections and filters - support nested JSON values in json_get string results - add and reorganize JSON2 sqlness coverage Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
shuiyisong
|
82444635f5 |
feat: support quantile and fraction queries on mixed histograms (#8874)
* fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove try_build_float_literal Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove NonCommutative Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: NaN Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test(query): cover frontend-only histogram fold planning Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(promql): ignore unparseable histogram bucket labels Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: typo Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
dennis zhuang
|
10f587bc30 |
fix(query): preserve timestamp literal semantics in inserts (#8889)
* fix(query): follow timestamp insert assignment lineage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): fold constant insert timestamp literals at the assignment Following lineage by retyping the source column changed every output column that reads it: a string column sharing the literal was silently rewritten to a formatted timestamp, and a nanosecond column was truncated to the precision of whichever column was converted first. Resolve the constant read-only and fold it into the assignment expression instead, which leaves the source query untouched and also covers literals behind WHERE, ORDER BY and DISTINCT. VALUES rows and UNION branches carry per-row values, so they keep the in-place rewrite, now guarded against columns with more than one consumer. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): strengthen insert lineage regression Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): trim redundant insert coverage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): rebuild insert unions loosely and cover UNION distinct Per review: rebuilding a rewritten union with the strict constructor rejected legal pre-coercion plans whose untouched columns still differ across branches. Use try_new_with_loose_types, matching the SQL planner. Distinct::All joins the rewrite passthrough so UNION (distinct) literals get session-timezone parsing like UNION ALL; deduplication then keys on parsed instants instead of raw strings. The top-level rewrite path gains the same single-consumer guard as rewrite_projection for hand-built DML plans that share a source column between targets. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style(query): tighten insert assignment comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Morax
|
c3cd186997 |
fix(query): respect query timezone in timestamp casts (#8859)
* fix(query): respect timezone in insert values Signed-off-by: Morax <james20081204@gmail.com> * fix(query): normalize timestamp casts with query timezone Signed-off-by: Morax <james20081204@gmail.com> * fix(query): scope timestamp conversion to insert assignments Signed-off-by: Morax <james20081204@gmail.com> * style(query): simplify Arc usage Signed-off-by: Morax <james20081204@gmail.com> --------- Signed-off-by: Morax <james20081204@gmail.com> |
||
|
fys
|
6538db6d61 |
refactor(json2): push down json2 type hints to parquet reads (#8833)
* refactor(mito): push down json2 type hints to parquet reads Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share json2 target types with arc Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): derive json2 output schema from target types Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): simplify read columns construction Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo check Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): reject JSON hints for non-JSON2 read columns Signed-off-by: fys <fengys1996@gmail.com> * fix: do not pushdown json type hint of non-json2-col Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> * refactor(query): simplify JSON type hint application Signed-off-by: fys <fengys1996@gmail.com> * refactor: clean up JSON2 type hint handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): keep JSON2 hints with flat read format Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): use raw parquet projection for output schema Signed-off-by: fys <fengys1996@gmail.com> * fix(query): note JSON2 hint scope limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): store JSON target types as native types Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover join hint qualifier limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): remove JSON2 fallback from compat cast Signed-off-by: fys <fengys1996@gmail.com> * docs(mito): document ReadColumns ordering contract Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo clippy Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |