mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-13 07:22:32 +00:00
5fe2a57adbe2f022009d8564ad2e459d7b90439e
927
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
5fe2a57adb |
fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step
`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover the default subquery step in a range query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
 discord9
|
1cdd4b54ed |
fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
discord9
|
00e4666391 |
fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): use current DataFusion plan downcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): match null rendering in join regression
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh union plan for unknown scan partitioning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): refresh shared plans after distribution rule removal
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): cover semi joins with remote dynamic filtering
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): strengthen partitioned merge scan regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): remove unused merge scan test helpers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(perf): measure merge scan hash repartition cost
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): adapt merge scan deserialization to two-arg constructor
Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): adapt nested merge scan integration constructor
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
6115559b64 |
fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries
An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): keep encode_substrait on a valid range query
A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): plan wrapped range-vector roots and reject a zero subquery step
TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover wrapped range-vector roots and a zero subquery step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): default subquery step and reject range vectors in range queries
A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover root subquery default step
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): align root subqueries with subquery_child_window
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): return millisecond timestamps for range-vector results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): keep the time index timezone and name its table in errors
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
discord9
|
293551cd45 |
fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds
Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.
The metasrv-side validate_flow_options remains as a defensive check.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): use DISTINCT output columns as sink key for streaming flows
When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.
Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.
Signed-off-by: discord9 <discord9@163.com>
* fix(flow): preserve extension options and complete DISTINCT sink identity
Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): align meta unknown option status assertions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): use valid aliases in DISTINCT sink regression SQL
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(flow): record complete DISTINCT sink SQL results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
f68b296669 |
fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch
Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: update flow_tql error for a negative TQL end time
A negative end now parses and fails the end >= start check instead of
the timestamp conversion.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reject evaluation times outside the millisecond range
The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: drop an unused import
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round query times to milliseconds like Prometheus
Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): round the float fraction before converting to nanoseconds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: skip the overflow case where SystemTime cannot hold it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
 Niladri Adhikary
|
98d9012f00 |
fix(promql): return no point for zero-interval irate/idelta (#9455)
fix(promql): irate/idelta return no point for a zero-ms sample gap
Signed-off-by: Niladri Adhikary <niladrix719@gmail.com>
(cherry picked from commit
|
||
 fengyue-xve
|
0ce6729b40 |
test(promql): cover label_replace normalization with distinct series (#9485)
The duplicate-label-set rejection must not fire when a label rewrite keeps
every series distinct through another label. Normalizing mixed histogram
boundary representations (`10.0` -> `10`) while `instance` still differs is
the false positive reported in #9444. The check already accounts for it
since it became a per-timestamp row count instead of a planning-time
rejection, but no fixture covered the positive case. Add one, together
with the aggregation variant that produces a genuine duplicate and must
still fail.
Signed-off-by: fengyue-xve <1759519137@qq.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
d08589e779 |
fix(promql): keep vector labels when a computed scalar is on the right (#9478)
* fix(promql): keep vector labels when a computed scalar is on the right
`metric + time()` or `metric * scalar(other)` planned the join with the
scalar side's context, which has no tags, so every series collapsed into
one unlabeled series. Use the vector side's context when the right operand
is scalar-typed.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): update results for labels kept with a computed scalar on the right
v * pi() and v + time() now keep the vector's labels and time index name,
as in Prometheus.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): qualify columns of an unnamed operand in vector joins
A comparison filter's projection has no table name, so after the join its
time index resolved unqualified and was ambiguous when the other operand had
the same column. Also keep the empty-metric branch unchanged for scalar
left operands.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover comparison filters joined with a scalar
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
discord9
|
5fb226fece |
fix(promql): align subqueries to the absolute evaluation grid (#9400)
* fix(promql): align subqueries to the absolute evaluation grid
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* docs(promql): streamline subquery comments
Remove redundant arithmetic narration and restated test comments. Keep grid, fallback, validation and overflow contracts, and preserve all regression tests and assertions. Regenerate SQLness output only for comment echoes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
ab00444d9a |
fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: verify compound filter semantics and last-row selection
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: consolidate label filter regression coverage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: separate filter evaluation and timestamp adaptation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: inline predicate and timestamp tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: keep direct IN predicates outside simple filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: coerce partition bounds before simple filtering
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: cover partition bound overflow after timestamp widening
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
83014e20a4 |
test(sqlness): wait for mirrored inserts before flushing streaming flows (#9424)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
 Yingwen
|
c52be28ff9 |
fix: coalesce equal histogram quantile bucket boundaries (#9451)
* fix: coalesce equal histogram quantile bucket boundaries
Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.
Fixes #9443
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: keep duplicate infinity buckets in one histogram group
Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.
Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: avoid buffering histogram groups with invalid trailing bounds
Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
cedd078440 |
fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics
- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): align label_replace and label_join edge cases with Prometheus
- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): join absent labels as empty strings in label_join
concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): skip rows without a sample in timestamp()
timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
e8918ae4b7 |
fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns
col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(servers): resolve remote read matcher labels as unqualified columns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover same-name columns differing in case and without()
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
 DeviousCardi
|
ffbbae7acf |
fix(promql): apply offset to subquery evaluation window (#9364)
* fix(promql): apply offset to subquery evaluation window `prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading `offset`, so `<subquery>[range:step] offset <d>` planned exactly the same window as the un-offset form and silently returned data for the wrong time range. The plain vector/matrix-selector paths already threaded the offset through `selector_to_series_normalize_plan` and `RangeManipulate`. Shift the inner evaluation window back by the offset and pass the offset to the subquery's `RangeManipulate`, which maps the inner samples forward onto the evaluation timeline before bucketing them into ranges. This matches Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner expression over `(start - offset - range, end - offset]` and whose `evalSubquery` then hands the samples to the outer range-vector function as a `MatrixSelector` that still carries the subquery offset. An offset on the inner selector composes additively, as `subqueryTimes` documents. `RangeManipulate`'s protobuf message has no offset field and recovers it on decode from an immediately underlying `SeriesNormalize`. Since `RangeManipulate` is commutative in `dist_plan` and can be pushed below a `MergeScan`, insert that carrier node so the offset survives distributed planning instead of decoding as zero. Known divergence, unchanged by this commit: Prometheus anchors subquery step points on absolute epoch multiples of the step, while GreptimeDB anchors them on the evaluation start. The two agree whenever the offset is a multiple of the subquery step; the added sqlness cases stay within that range. Closes #9330 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Aarav <aaravsjadav@gmail.com> * docs(promql): correct the subquery-offset rationale and pin the histogram shift Follow-up to |
||
  LFCandshuiyisong
|
456babbd18 |
feat(catalog): declare information-schema table scopes [Backport release/v1.3] (#9534)
feat(catalog): declare information-schema table scopes (#9504)
* feat(catalog): declare information-schema table scopes
* chore: rename cluster to global
* fix: update sqlness result
* refactor(catalog): require information-schema table scopes
---------
(cherry picked from commit
|
||
 fys
|
ad7a9c3518 |
fix(json2): support settings in ALTER TABLE ADD COLUMN [Backport release/v1.3] (#9513)
* fix(json2): preserve metadata when adding columns via alter table (#8696) * fix(json2): support adding columns via alter table Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): scope missing-column null handling to JSON2 --------- Signed-off-by: fys <fengys1996@gmail.com> (cherry picked from commit |
||
|
LFCandfys
|
1a25a02f21 |
feat(json2): json_get respect type hints [Backport release/v1.3] (#9511)
feat(json2): json_get respect type hints (#9349)
(cherry picked from commit
|
||
|
Yingwen
|
000ab3abbf |
fix: temporarily disable v2 series scan by default [Backport release/v1.3] (#9495)
fix: temporarily disable v2 series scan by default (#9436)
* fix: disable v2 series scan by default temporarily
* test: explicitly enable v2 series scan in sqlness configs
* test: regenerate sqlness results for legacy series scan default
---------
(cherry picked from commit
|
||
|
discord9
|
28e415a103 |
feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors The planner previously dropped the `at` field of selectors (`at: _`), so `some_metric @ 300` silently returned step-following values instead of the samples anchored at the fixed timestamp. Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch` semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply `offset` to the anchor, rewrite the selector offset to `eval_start - anchor`, scan only the anchored window, then report the same window at every evaluation step via a grid-wide-replay `InstantManipulate`. `@ start()` / `@ end()` resolve against the statement's evaluation range (`stmt_start` / `stmt_end`), which subquery planning does not rewrite. Timestamps before the Unix epoch are accepted; unrepresentable ones are rejected with `AtModifierTimestampOutOfRange` instead of wrapping. Selectors without `@` are planned exactly as before. Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): center predict_linear on the evaluation timestamp predict_linear_impl used the window's last sample time as the evaluation timestamp, and the UDF took only (ts_range, value_range, t) with no channel for the current instant. After a range selector is folded for @ (or with an offset) the same window is replayed at every step, so the prediction stayed constant at the anchor's answer; even a plain window ended before the step produced a stale value. Give prom_predict_linear a 4th argument carrying the step's evaluation instant (ms timestamp), derived in the planner from the row's time index plus the offset the window was folded with (at_offset for @, offset_ms otherwise, recorded on PromPlannerContext). The regression is centered on that instant, matching Prometheus' use of enh.Ts. The mixed float/native-histogram path forwards the same argument. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): tighten @ modifier planner and predict_linear - range_fold_offset is always a concrete millisecond offset, not an Option; the single-step replay helper no longer wraps an infallible plan in Result. - predict_linear's eval timestamp is always cast to Timestamp(ms) at the call site, so the UDF drops its dead Int64 branch and the extra func_name argument. - Drop two redundant comparison cases from the at_modifier sqlness test and fix the lookback window comment to the half-open (0s, 300s]. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): accept parse-time rejection of @ on Windows `@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime holds it and the planner rejects the anchor it cannot represent, but a Windows SystemTime tops out near 1.8e12 seconds, so the parser's checked_add fails first and the same literal is rejected while parsing. Accept either rejection path so the test passes on both platforms. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): avoid replaying label_join over rewritten series keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): narrow anchored range call promotion Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address @ modifier review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): satisfy super import format check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): address at modifier review follow-ups Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4ffd93d8dc |
fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch) below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then dropped or rewrote that fetched coalesce, losing the global limit. Root-caused and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo pins to that fix commit and adds regression coverage. - pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix) - global_limit.rs: adapt to new DF API (input_distribution_requirements, child_distribution, replace_children+Recompute); accept HashPartitioned and KeyPartitioned as partitioning to restore; add regression unit tests - sqlness: extend standalone common ssts and refresh distributed ssts_limit result for the preserved fetch plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: drop dead HashPartitioned compat arms and test wrapper - remove HashPartitioned compatibility matches (pinned DF only emits KeyPartitioned); drop their #[expect(deprecated)] attributes - remove single-use agg_with_limit helper whose seed limit is always overwritten by the soft-limit transform - restore Cargo.lock dependency edges unrelated to the DataFusion pin bump (cargo update --precise had downgraded 26 unrelated packages) Addresses oracle ablation review. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(deps): pin DataFusion to merged fetch preservation fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
357c935804 |
fix: make vector aggregates work with GROUP BY and partial aggregation (#9338)
* fix: make vector aggregates work with GROUP BY and partial aggregation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: make partitioned vec_avg case distinguish weighted averages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
abedeb2bea |
fix(query): keep count_values generated label in enclosing expressions (#9223)
* fix(query): keep count_values generated label in enclosing expressions
`count_values("v", m)` projects the generated label as a real output
column, but did not register it in `ctx.tag_columns`. Enclosing
expressions (abs/round/+1/topk/label_replace/vector join) rebuild their
projection from `ctx.tag_columns` and silently drop the label.
Register the generated label in `ctx.tag_columns` after the projection,
and give it the same qualifier as other tag columns so qualified
references resolve. PromQL overwrites an input label with the same name,
so drop any existing tag with that name first to avoid duplicate column
ambiguity.
Fixes https://github.com/GreptimeTeam/greptimedb/issues/9181
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-1
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): use total_cmp in count_values test helper
Silence clippy::needless_borrow on partial_cmp(&right.1); f64 sorting
uses total_cmp, matching the other planner test helpers.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): keep count_values generated label as sink table primary key
FindGroupByFinalName::f_up only renamed a group key when the projection
aliased the key column directly. count_values now projects its generated
label as a unary UDF over the sampled column
(prom_float_to_string(value) AS label) while the aggregate still groups
by the raw column, so the name match failed and the sink table lost the
label from its PRIMARY KEY, demoting it to a DOUBLE value column.
Allow a projection above the aggregate to rename a group key by deriving
its output from a single group-key column (unary scalar function or cast
over that column only). Multi-column expressions, case, aggregates,
windows, subqueries and literals are still rejected so a computed column
cannot be mistaken for the group key.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): group count_values by the formatted sample value
The numeric count_values branch grouped by the raw sample column and only
formatted the value into the generated label in the post-aggregate
projection. Two distinct raw values that collapse to one label text
(e.g. BIGINT 9007199254740992 and 9007199254740993, both 9007199254740992
in Float64) were split into two groups, each emitting the same label set
at one timestamp, violating Prometheus' unique-label-set-per-timestamp
invariant.
Return the formatted value expression (prom_float_to_string, with a
CAST to Float64 for non-Float64 inputs) as previous_field_expressions so
the aggregate groups by the same expression that produces the label,
mirroring the existing mixed float/native-histogram precedent.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): drop needless borrow in count_values test helper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): don't replace group key with derived unary expr
FindGroupByFinalName treated any unary expression of a single group
column (scalar fn, Cast, TryCast) as a rename of that group key and
swapped it in as the sink primary key. A derived expression such as
lower(host) is many-to-one, so distinct groups (HOST_A vs host_a) would
collapse to the same primary key and be silently deduplicated.
Narrow the matching to direct name-matched aliases of the actual group
expression only, and drop the derived-unary path (is_unary_expr_of_column)
and the allow_derived flag. count_values already groups by the formatted
expression name, so it is unaffected.
Add a regression asserting host survives and host_lc does not replace it
under both optimizer settings.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
dennis zhuang
|
d59725b04a |
test(sqlness): run environments concurrently and drop redundant restarts and sleeps (#9333)
* test(sqlness): run environments concurrently with external backends The runner forced both environment and instance parallelism to 1 whenever etcd/PG/MySQL or an external Kafka was set up, so the standalone and distributed environments ran one after the other. Only distributed uses the kv backend, and the two environments use different Kafka topic prefixes, so they can run at the same time. Keep one instance per environment, since instances would share the metadata table and Kafka topics. Runs with a runner-managed Kafka, an external server, a test filter, or `-j 1` stay fully serial. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): drop redundant restarts and sleeps - flow_rebuild: drop the R2 block, which repeats R1 from the same state, and the restart in R5, whose result R1 already asserts. - flow_advance_ttl, flow_view, ttl_instant: drop sleeps that no assertion depends on. - flow_basic, flow_null, flow_call_df_func: drop the bytes_log section (covered by flow_insert), a duplicate state_size query, an unchecked insert, and flushes that run with no new data. - alter_table_options, skip_wal: share restarts between independent tables. - session_skip_wal, copy_skip_wal: check every case after a single restart. Tables copied or inserted with skip_wal = true first get a WAL row that is truncated, so the check that truncated WAL entries are not replayed stays. - region_statistics: wait once for the statistics of all three tables. - build_index_table: fold its index_size checks into build_index_table_restart, which runs the same fixture and waits. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore session_skip_wal and copy_skip_wal Checking every case after a single restart dropped two state transitions the original cases cover: recovering a region whose memtable only held skipped rows, and writing to the recovered region before restarting again. Restore the original cases. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the one-restart check in flow_auto_sink_table #5112 checked the auto-created sink before a restart and the flow and sink after it. #5987 moved the restart in front of the first SHOW and added a second one. Flow recovery only creates the sink when it is missing, so the second restart recovers from the same persisted state as the first. Restore the original before/after check with one restart. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): keep one instance per environment with external store addresses Instances of the distributed environment share the etcd behind --store-addrs, so run one instance per environment when it is set, same as --setup-etcd. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(sqlness): restore the false-branch batch in flow_basic and assert it The kept batch (20, 22) is all above the threshold. The second batch (10, 23) is the only input that hits the false branch of the CASE and makes the flow compute again. Restore it and check the result, which the original case never did. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
affc0a1b1d |
fix: align ordered aggregate state type with the accumulator output (#9340)
* fix: align ordered aggregate state type with the accumulator output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject mismatched aggregate states and keep hard-ordered aggregates unsplit Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep WITHIN GROUP aggregates splittable and relabel state fields by position Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
def0c2ec5c |
fix: only push down aggregates grouped by the partition columns themselves (#9337)
* fix: only push down aggregates grouped by the partition columns themselves Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep grouping sets on the frontend for partitioned tables Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
b9f991502c |
refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cc1be82858 |
fix: merge every state row in geo_path and json_encode_path (#9339)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
88197f4019 |
fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings A vector-vector binary operation projected one operand's whole tag set and inner-joined without any cardinality check, so `on()`/`ignoring()` did not reduce the result labels, `group_left`/`group_right` changed nothing, and a non-unique match group produced a cross product that PromQL cannot represent. Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the matching labels, `ignoring(...)` drops them, and a group modifier keeps the many side's labels plus the `group_x(...)` labels taken from the one side. A label the one side does not carry is deleted from the result. The reduced label set no longer identifies the operand series, so `__tsid` is dropped from the context on this path. Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)` window and a scalar UDF that fails the query on a repeated group: on the one side before the join, and on the result labels after it, matching where Prometheus raises each of its three errors. Series are unique by their whole tag set, so the window is only planted when the match keys drop a tag; plain arithmetic and `on(<all tags>)` plan exactly as before. Closes #9207, closes #9208, closes #9209. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): keep __tsid when the result labels are an operand's whole tag set Deriving the result labels dropped `__tsid` from the context unconditionally, so an enclosing operation fell back to joining on the tag columns even where the column still identified the result series. Keep it when every result label comes from one operand and covers that operand's whole tag set: no other operand value reaches the labels, and the matching gives each of its rows a single partner, so its `__tsid` is still one per result series. That is the common `on(<all tags>)` and bare `group_left` shape; a matching that actually drops a tag still clears it. The column is re-qualified as the result's own, which is how the enclosing expression and the context look it up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the match group count column unambiguous The cardinality check aliased its row count to a fixed `__promql_match_group_count`. An operand carrying a label of that name made the window output two fields with the same name, and planning failed with "Schema contains qualified field name collide_right.__promql_match_group_count and unqualified field name __promql_match_group_count which would be ambiguous". Pick a name the operand does not already have, the way the `or` operator allocates its match key columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover a match group spread over several regions The cardinality check runs above the merge of the region scans, so it counts a match group globally. Nothing pinned that: every table in these cases holds a single region, and a check evaluated per region would pass them all. Partition the operand on a column outside the match keys, which puts the two series of one match group in different regions, and assert both the pre-join and the post-join check still reject it. The case runs in the distributed environment too, where the regions sit on different datanodes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): group an outer aggregate on the labels the operand kept `by`/`without` planning topped up a missing grouping column by walking down to the table scan and re-projecting it. That is right for a column the scan pruned for efficiency, but the labels a matching modifier deletes are also absent from the operand's output, and they were restored the same way: sum without(host) (a / on(host) b) `on(host)` leaves the operand with `host` alone, so the sum covers everything and Prometheus answers `{} 10`. Instead `device` came back from the scan under `a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for `sum by(device)` of that operand, which has no `device` to group on at all. Take the grouping labels from the operand's own label set rather than from the row keys of the scan beneath it. A label pruned from the plan is still in that set and still gets restored; a label the operand dropped is not. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): drop the unreachable aggregation tag top-up `by`/`without` planning could restore a grouping column that the plan no longer carried by rewriting the scan underneath it. Once the grouping labels come from the operand's own label set, there is nothing left for it to restore: a scan projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds matcher columns to that set), so a label in the set is always in the schema. Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the standalone and the distributed environment. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): drop cases that guarded the removed tag top-up Three plain selector aggregates were there to show that restoring a pruned grouping column still worked. With the restore gone they only repeat what the aggregate cases already cover. Also fix a comment that still said the metric engine scan prunes tag columns: it projects every label of the operand. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): note the duplicate a propagated matcher hides A matcher copied onto the one-side operand removes groups without a partner before the cardinality check sees them, so a duplicate in such a group is not reported. Prometheus checks every group of the one side and fails the query. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
fys
|
5a78e07be6 |
fix(json2): restrict JSON2 type hints (#9316)
* fix(sql): restrict JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): allow equivalent 64-bit aliases in JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): support UInt64 conversion and preserve JSON2 hint aliases Signed-off-by: fys <fengys1996@gmail.com> * refactor(sql): remove redundant JSON2 type hint normalization Signed-off-by: fys <fengys1996@gmail.com> * fix(datatypes): restrict JSON2 hint types in JsonSettings::try_new Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 Lei, HUANG
|
d0f8f4b80c |
feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
25bca4609f |
perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
LFC
|
16978cf6c2 |
feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2 Signed-off-by: luofucong <luofc@foxmail.com> * fix: remove unused annotation context import Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
233e23b352 |
fix(promql): keep value-field grouping labels out of matching filter propagation (#9242)
* fix(promql): keep value-field grouping labels out of matching filter propagation #9202 propagates matching-label matchers to the scanned selector using the planned operands' tag columns. For an aggregated operand those are its grouping labels, and agg_modifier_to_col resolves by(...) names against the input schema only, so a value field named in by(...) is reported as a tag. A value field varies between the samples of one series, so lowering a matcher on it below sample selection (PromInstantManipulate) can drop the newest sample, promote a stale one from the lookback window, and fabricate a match the un-rewritten query does not produce. Track the by(...) labels that name value fields of the aggregated operand's input in PromPlannerContext::aggregation_field_labels, and refuse to propagate matchers on them while still requiring a matcher to name a grouping label to cross an aggregation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: sync matching_filter result fixture comment with the PR number Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9e9a8cac20 |
fix(query): insert MergeScan into nested scalar subqueries (#9261)
* fix(query): insert MergeScan into nested scalar subqueries DataFusion 55 keeps uncorrelated scalar subqueries as expression subqueries (enable_physical_uncorrelated_scalar_subquery, default true) instead of decorrelating them into joins, and executes them via the new physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked the plan with a plain TreeNode transform that does not descend into expression subqueries, so MergeScan was only inserted for depth-1 subqueries. A scalar subquery nested inside another scalar subquery kept a bare frontend DistTable TableScan and failed at execution with "Unsupported operation: get stream from a distributed table". Use the subquery-aware transform so handle_subquery (PlanRewriter / MergeScan insertion) runs for subquery plans at every nesting depth. Fixes #9260. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen nested scalar subquery regression coverage Address review on #9261: - Replace the ineffective 'no bare TableScan' string check with a real subquery-aware plan walk (apply_with_subqueries); MergeScan hides its remote input from traversal, so any TableScan the walk reaches was genuinely left unwrapped. - Add a distributed regression case on a range-partitioned table so the nested inner aggregate must merge partial results across regions (global AVG feeding an outer SUM filter). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
fys
|
e82cb0af74 |
feat(json2): respect type hints during JSON2 type concretization (#9222)
* fix: prefer JSON2 type hints for uncast read pushdown * feat(query): materialize JSON get result types before planning * fix(query): apply JSON2 type hints to parsed paths * fix(query): apply JSON2 type hints before distributed planning * chore: remove f * refactor: the code style of json_get_type_hint * chore: add sqlness case * fix: add json expr planner, remove json get type hint * chore: add sqlness test * fix(query): respect JSON2 type hints in query planning * chore: add more sqlness cases * fix: cargo clippy * fix: cargo clippy * chore: update sqlness test result |
||
|
dennis zhuang
|
007836ba7a |
fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query
`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.
Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.
Only metric engine tables are covered; other engines share no column space
to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): batch-resolve metric names by table id
Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.
The catalog walk stays, but only to find the physical tables to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): read an absent label as the empty string
A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.
Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.
This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): don't panic on pre-epoch label value bounds
`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.
Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): drop applicable_matchers, share the distinct scan
With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.
Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
 Sainath Singineedi
|
fbbc017be4 |
feat: expose region min/max timestamp in region_statistics (#9060)
* feat: expose region min/max timestamp in region_statistics Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> * test: cover region_statistic time range assembly and projected values Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> --------- Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> |
||
 discord9anddiscord9
|
983738101a |
feat(flow): admit mergeable average states in incremental plans (#9235)
Incremental batching flows already merge sink state for scalar aggregates and the HLL/UDDSketch/stddev state families. AVG now exposes a mergeable Binary state (avg_state / avg_merge with __avg_state_delta_merge), so admit those aggregates too instead of forcing a full snapshot for flows whose only state column is an average. merge_op_for_aggregate_expr takes the aggregate input schema so the avg_merge arm can require a Binary state argument; the state form is the aggregate result persisted by the sink, so no extra coercion is needed. Other input types keep being rejected. Also cover avg_state, avg_merge and duplicate AVG projections in the incremental plan analysis tests, extend the mixed state-family rewrite test with an average column, and extend the standalone partitioned state-merge SQLness case with avg_state/avg_merge compared against a direct avg over the source. Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
discord9
|
27090496d1 |
test(flow): stabilize FLUSH_FLOW assertions after async source-table mirrors (#9230)
* test(flow): wait for async source-table mirrors before FLUSH_FLOW Streaming flows execute inline in the flownode insert handler since #8976, and source-table inserts are mirrored from the frontend as detached tasks. Tests that INSERT then ADMIN FLUSH_FLOW then SELECT intermittently miss rows on slow CI runners because the flush no longer implies the mirror reached the flownode (flow_no_aggr lost row 'l', flow_advance_ttl lost row '23' after the post-restart reinsert). Add SQLNESS SLEEP 3s between those mirror inserts and the flush, the established pattern in the flow suite, so the assertions are stable. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): also guard first-section flushes in flow_advance_ttl Local reproduction without sleeps failed in a window the previous commit did not cover: the first INSERT (20,20,22) of each section is followed immediately by ADMIN FLUSH_FLOW, and the distributed run observed an empty sink on that SELECT (~1 in 8 iterations). Add the same SLEEP 3s guard before those two flushes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
b1106a9c5a |
perf(promql): propagate matching-label filters between binary operands (#9202)
* perf(promql): propagate matching-label filters between binary operands A one-to-one arithmetic binary expression inner-joins its operands on the matching labels, so every row that survives the join already satisfies the other operand's equality matchers on those labels. Copy those matchers to the other operand so both scans drop non-joining series before execution instead of feeding them to the join. Both operands are planned before the rewrite: a selector matcher can also constrain a value field, and only the planned contexts tell tags and fields apart. A matcher is copied only when its name is a tag column on both sides, its value is non-empty, and it is one of the matching labels. Operands are limited to vector selectors, parentheses, and label-preserving range functions applied directly to a matrix selector; `ignoring(...)`, group modifiers, fill values, set and comparison operators, regex matchers and `or` matcher groups are left alone. Selector matchers are now deduplicated in place instead of through a `HashSet`, so the generated scan filter keeps a stable order once a selector carries more than one matcher. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): propagate ignoring, non-equality and aggregated matchers Widens the matcher propagation added in the previous commit to the shapes it was leaving on the table. The join compares matching labels with plain column equality (`normalized_match_key_expr` and its coalescing are confined to `or`), so any predicate on a matching label is already enforced on both sides for every surviving pair: it originates on one operand, and the join carries it to the pairs it forms. Copying it to the other operand can only drop rows that had no surviving partner. That argument does not depend on the matcher kind, so regular expressions, negations and empty values now propagate too. `=~".*"` stays out: it lowers to no filter at all, so copying it would only force a re-plan. `ignoring(...)` is no longer rejected. A label is a join key exactly when it is a tag on both sides and not named in `ignoring`, which is what `binary_join_key_columns` computes and what the caller can now answer from the two planned contexts. Operands may now be aggregations that partition by their grouping labels, which is the shape most real queries use. `agg_modifier_to_col` rewrites `ctx.tag_columns` to the grouping labels, so a label found in an aggregated operand's tags is a group key, and filtering the aggregate's input by it drops exactly the corresponding output groups. `topk`, `bottomk` and `limitk` are excluded because they select across a group and carry input labels through -- `prom_topk_bottomk_to_plan` leaves `ctx.tag_columns` alone, so the tag check cannot catch them. `count_values` is excluded because it adds an output label that does not exist in its input. The rollup whitelist now covers every label-preserving range function the planner implements, and finds the matrix argument by position so multi-argument rollups such as `quantile_over_time` and `predict_linear` qualify. `absent_over_time` stays out: it synthesizes a series from the matchers when its input has none. Re-running the sqlness case against an unmodified planner produces a byte-identical `.result` for all 27 queries. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin that excluded matching labels stay on their own operand `ignoring(device)` and `on(host)` both leave `device` out of the join keys, so a `device` matcher must not reach the other operand. Neither case was covered end to end: a regression there silently drops the left operand's `eth1` series instead of returning them, which the two added queries now catch. Also corrects the comment at the rewrite site, which still described re-planning as touching a leaf selector. Aggregated operands are re-planned as a whole; what holds is that the rewrite only ever adds matchers to a selector, so the enclosing operand's table reference, time index and field columns are unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): log why matching-filter propagation was skipped Splits the guard chain into `try_propagate`, whose `Err` names the reason the rewrite does not apply, and logs it once in `propagate`. Debugging a query that did not get the filter no longer means stepping through the guards. The reasons also replace the comments that used to explain the same conditions, and the remaining comments lose the parts that restated the code or repeated each other. Two of them were wrong rather than verbose. Saying `topk`'s input "must not be filtered" reads as a claim about PromQL: `topk(1, m{host="x"})` is perfectly legal, and what matters is that filtering before `topk` changes the candidate set it ranks. Saying the subset-matching case "keeps its many-to-many result" read as an endorsement of behaviour Prometheus rejects outright; it now records that the query is a cross product here and points at #9209. The propagation cannot change whether such a check would fire: series in one match group agree on every matching label, so a matcher over one of those labels keeps all of them or drops all of them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep copied matchers out of the operand's selector metadata Re-planning an operand with a copied matcher also rebuilt its `PromPlannerContext::selector_matcher`, and that context is what an enclosing expression reads. `create_absent_plan` turns the equality matchers found there into the labels `absent()` reports, so absent(counter_metric{host="missing"} / on(host, device) gauge_metric) gained a `host="missing"` label that `main` does not produce. The inner expression was empty either way; only the reported label set changed. The copied matcher belongs to the scan, not to the operand's identity, so both re-planned contexts now keep the matchers their operand was written with. `selector_matcher` has one other reader, `create_table_scan_plan`, which consumes it while the operand is being planned and is unaffected. Reported by @discord9. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
fys
|
5b631fd8eb |
feat(json2)!: remove nullable and default type hint options (#9213)
feat(json2): remove nullable and default type hint options |
||
|
discord9anddiscord9
|
5ef8a46e3f |
feat(function): add mergeable binary average states (#9062)
* feat(function): add mergeable binary average states Signed-off-by: discord9 <discord9@163.com> * feat(function): expose avg_calc as an OSS scalar Signed-off-by: discord9 <discord9@163.com> * fix(function): borrow invalid AVG scalar argument type Signed-off-by: discord9 <discord9@163.com> * test(function): verify public AVG state SQL finalization Signed-off-by: discord9 <discord9@163.com> * fix(function): return canonical AVG1 state for empty window frames DataFusion's plain-aggregate window executor bypasses the accumulator and calls default_value() directly when a window frame contains no rows. create_udaf's SimpleAggregateUDF derives default_value from the return type, which yields SQL NULL for Binary output instead of the canonical AVG1 empty state, so empty frames and frames over only NULL inputs became observable differently (IS NULL, direct state saves, state comparisons). Register avg_state/avg_merge through a small AggregateUDFImpl (AvgUdaf) that keeps the existing accumulator and declares the canonical AVG1 empty state as default_value, restoring the documented contract. Add a unit test asserting default_value equals the empty accumulator's evaluate() and an sqlness case covering the empty-frame window. Signed-off-by: discord9 <discord9@outlook.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
LFC
|
697cc5fa29 |
fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic Signed-off-by: luofucong <luofc@foxmail.com> * fix(json): align integer extraction with JSON2 conversions Signed-off-by: luofucong <luofc@foxmail.com> * style: format jsonb dependency declaration Signed-off-by: luofucong <luofc@foxmail.com> * fix(otlp): report concrete unsupported JSONB type names Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
6fa375021f |
feat(flow): expose extension-owned batching execution hooks (#9171)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: merge aggregate states in incremental flows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to state merge execution tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): expose extension-owned batching execution hooks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): correct extension matcher borrowing and regression assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: prove executed empty exact deltas keep incremental mode Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to empty exact delta regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover aggregate state merge through the full flow path Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to flow batching task regression calls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
7c7132ea65 |
refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): execute streaming flows with DataFusion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): remove legacy streaming runtime
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid retrying stateless sink inserts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): align stateless writes with sink schema
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): reject stale stateless source schemas
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): validate stateless flow routing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(mito2): cover regex inverted index pruning"
This reverts commit
|