Commit Graph
6367 Commits
Author SHA1 Message Date
discord9 8acc57ea2b fix(promql): drop needless question mark in post-execute
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 19:13:46 +08:00
discord9 f9cb5eddb5 test(promql): refresh identity boundary and matching snapshots
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 18:30:45 +08:00
discord9 a53d4471a7 fix(promql): retain deterministic set matching labels
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 18:15:43 +08:00
discord9 29e448813f fix(promql): project metric identity at native query boundaries
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 17:49:27 +08:00
discord9 432a92f529 chore(promql): sync capability with current main
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 17:27:05 +08:00
discord9 b33dcc4b02 chore(promql): sync prerequisite with current main
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 16:42:46 +08:00
discord9 c750cc1fd2 chore(promql): stack capability on independently verified prerequisite
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 16:38:58 +08:00
discord9 43ffbdebbe test(promql): split comprehensive coverage from the capability layer
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 16:38:44 +08:00
discord9 0692d92765 fix(promql): keep feature-specific annotations with metric-name changes
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 15:59:30 +08:00
discord9 7941800c9f fix(promql): retain projection dependencies and dropped histogram annotations
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 15:39:23 +08:00
jeremyhi 18726c7602 fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory

Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(snapshot): bound writes and reduce logical export overhead

Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify metric snapshot acceptance criteria

Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): split batches only for merged readers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(snapshot): cover supported types in both layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: consolidate metric snapshot experiment conclusions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): honor series statistics when splitting L0 scans

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito): explain memory risk of splitting short L0 series

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address snapshot export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): accept source errors in lazy snapshot contexts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-12 06:17:03 +00:00
discord9 483d1e49a3 fix(query): keep filter fetch when rewriting matches term (#9521)
* test: cover matches term limit dropped by constant rewrite

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep filter fetch when rewriting matches term

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 06:12:56 +00:00
discord9 5fe8740fc0 test(promql): align avg_over_time expectation with Prometheus name semantics
#9477 pinned the old behavior where the HTTP response writer stamped
every row with __name__=<table name>. Under this PR's identity
lifecycle, value-changing functions such as avg_over_time drop the
metric-name identity, so the response carries job only. Also take the
upstream UnionExec snapshot drift (#9491) into the regenerated
tql-cte result.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 13:11:50 +08:00
dennis zhuang 2d99a5cf69 feat(jaeger): add query endpoints for Jaeger UI v2 (#9409)
* feat(jaeger): support v3 HTTP query APIs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(jaeger): preserve full traces with v3 time hints

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(jaeger): clarify validation of v3 lookup hints

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(jaeger): bound v3 searches and preserve v1 binary attributes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(jaeger): match resource attributes in v0 tag filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:11:03 +00:00
dennis zhuang f3a0841f4f fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations

- Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL
  comparisons with NaN are false except !=.
- min/max: skip NaN unless the whole group is NaN. DataFusion's grouped
  min/max let NaN replace and be replaced by neighbours, so the result
  depended on input order.
- topk/bottomk rank NaN last.
- round() rounds ties up and returns NaN for a zero step, like Prometheus.
- sqrt() of a negative number is NaN instead of an error.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round() without a step rounds to integers

The planner passed a zero step for single-argument round(), which now
yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding

sqrt resolves through the session's builtin functions, not the UDF branch.
Distributed plans are decoded with the serializer's session, which did not
know prom_min/prom_max.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover NaN and rounding semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the nested count rewrite for PromQL min/max

count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin
min/max names. prom_min/prom_max return NULL for the same groups, so the
rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt,
topk and min/max changes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): register prom_min/prom_max in the function registry

Distributed plans split an aggregation into state and merge steps only for
functions whose state function exists in FUNCTION_REGISTRY; prom_min and
prom_max lived in the promql crate, so min/max stopped being pushed down to
datanodes. Move them to common-function and register them before the state
functions are derived.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update sqlness results for PromQL float semantics

Plans and column names now show prom_min/prom_max, the NaN guards of
comparisons and sqrt, and the NaN sort key of topk. round() results follow
Prometheus' formula. Distributed min/max keep the state/merge split.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep column names and NULLs in IEEE comparisons and sqrt

Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed
TQL output column names and turned NULL operands into true for `!=` and
false for the bool forms. Implement comparisons and sqrt as PromQL UDFs
that propagate NULL and report the column name of the expression they
replace; prom_min/prom_max report min/max column names the same way.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: restore column names in sqlness results for PromQL float semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): share the prom_min/prom_max groups accumulator with tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep distinct names for prom_min/prom_max

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:10:18 +00:00
dennis zhuang 845fc0af9c fix(promql): default a stepless subquery to a one-minute step (#9481)
* fix(promql): default a stepless subquery to a one-minute step

`x[5m:]` evaluated the inner expression at the query step, which is 1s
for instant queries. Prometheus uses the evaluation interval, one minute
by default.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover the default subquery step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover the default subquery step in a range query

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:10:07 +00:00
discord9 6577b9bb6c fix(promql): attach metric-name identity to instant range-selector samples
#9477 introduced a root path for instant queries whose root is a range
selector, but it bypassed the marked identity column the response layer
reads. Save the table name right after matcher preprocessing (before
setup_context resets it for nonexistent metrics) and attach the
metric-name column on both return paths, mirroring the vector and
matrix selector roots. Without this, GET /v1/query?query=x[5m]
silently dropped __name__ from the response.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:57:21 +08:00
Weny Xu e6f3ca10a5 fix(deps): bump OpenDAL to 0.59.4 for OSS RRSA credentials (#9520)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-12 03:32:46 +00:00
discord9 a9d9c490b3 test(sqlness): refresh metric-name plans after syncing main
Regenerate the standalone and distributed regressions on main d640ab8bc5. Update the distributed TSID binary scan mode to legacy and record the conservative dropped-name uniqueness check in the histogram plan. Visible samples and tabular result schemas are unchanged.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:14:05 +08:00
discord9 981e84caef feat(promql): reserve __name__ as the metric name at the PromQL boundary
A table whose own schema carries a column spelled __name__ cannot be
interpreted as PromQL input: the semantic metric name is the table's
identity, so a second, ordinary business column of that name would be
ambiguous. setup_context() now refuses such tables with
ReservedMetricNameColumn (InvalidArguments) at the single selector entry,
before any context is filled; legitimately rewritten names from
label_join/label_replace/count_values are produced above the selector and
never pass through it.

With the reserved-name contract, an unmarked physical __name__ column can
no longer reach the Prometheus HTTP serializer, so the exported_name
auto-rename branch in ColumnLayout::infer is removed and tag names are
emitted verbatim.

Also narrow column_reference_is_constant to an explicit whitelist of
value-preserving plan nodes instead of any single-input passthrough, so a
future extension cannot silently waive the dropped-name uniqueness check.

Tests that constructed a physical __name__ tag as legal input now assert
the entry refusal; the internal __promql_metric_name collision tests are
unaffected.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:14:05 +08:00
discord9 42994f9bea fix(promql): check dropped-name uniqueness by identity constancy
Replace the UnionDistinctOn shape check in drop_metric_name with a
provable constancy analysis of the identity column. A name that is
rewritten by label_join/label_replace, generated by count_values, or
copied across a binary/set operation is not constant, so dropping it
asserts a unique label set per evaluation timestamp; a selector-attached
literal name is constant and adds no check. The analysis only follows a
column through Projection/Alias/Column and single-holding-inner-join
inputs and conservatively assumes non-constant everywhere else, so it can
over-check but never misses a collision such as
abs(label_join(label_join(m,"__name__","","k"),"k","")).

count_values with a semantic __name__ destination is now a controlled
rename instead of a silent ordinary label: it takes over or attaches the
marked identity with the metric-name role, groups by the formatted sample
value unless a without() modifier excludes it (in which case the label is
neither generated nor grouped), and keeps ordinary destinations ordinary.
The marker-relocation helper is shared between the label functions and
the aggregation path so an ordinary destination that occupies the marked
column's physical name moves the identity aside before grouping keys are
resolved.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 1817d0bb76 fix(promql): close metric-name identity privilege and boundary gaps
- count_values never forges the identity: the label it generates is always
  an ordinary column named as asked, with no metric-name role metadata and
  no reuse of the marked column's physical name. A plain aggregate argument
  can no longer inject a sample value into the privileged identity column
  (which the HTTP layer would otherwise emit as __name__ and TQL would
  silently strip). With no marker forged, dropping the name of a
  count_values result cannot collide, so the union-subtree check remains
  the only duplicate-label-set source.
- binary_match_label_mask no longer lets an ordinary on()/ignoring() label
  hit the marked identity by its physical name; only the semantic __name__
  resolves to it, and a real user column sharing the name still matches.
- label_join/label_replace move the marked identity to a free internal
  name before writing an ordinary destination that occupies its physical
  name, so writing a label can no longer silently replace the identity.
- sort_by_label reads a missing or NULL string label as the empty value,
  so a series that lost its name sorts first ascending like Prometheus,
  instead of NULLS LAST.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 6106a6ea08 refactor(promql): unify semantic label resolution and label rewrites
Converge the ordinary-label and __name__ paths that had forked during the
rebase:

- resolve_label_col() is the single label-to-column entry: __name__ reads
  the marked identity column, ordinary labels resolve like before but can
  no longer hit the marked column by its physical name, and a missing
  label is absent for every caller (label functions, aggregation, sort).
- label_replace keeps one algorithm for every destination: read source
  (missing as ""), match anchored regex, expand captures (empty -> NULL),
  keep the old destination value when unmatched. Ordinary destinations may
  overwrite existing labels again (#9385 semantics); collapsing series is
  refused by the existing runtime label-set check instead of a plan-time
  InvalidDestinationLabelName.
- write_destination_label() is the single destination-column entry for
  label_replace and label_join, so writing an empty label has one
  representation and a zero-source label_join on __name__ removes the
  identity instead of projecting an unmarked physical __name__ column.
- drop_metric_name() now asserts a unique label set per evaluation
  timestamp when dropping the identity of a union that may ignore it
  (abs(a or on(__name__) b)), reusing assert_unique_match_group; TQL
  strip_metric_name_column() stays free of PromQL series checks.
- sort_by_label(..., "__name__") on input without a name sorts by a
  missing label instead of failing with field_not_found.

Production code is a net reduction (-38 lines); the removed ordinary-label
regex branch, the overwrite tri-state and the duplicated destination
construction are replaced by the three shared entries.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 e86a22733c test(promql): align planner expectations with rebased main
Refresh the planner test expectations after rebasing onto upstream main
(78a7b93292, #9385): ordinary label_join of a missing label now joins as
the empty string and keeps its separator, label_replace of an existing
destination is refused at plan time, and the internal metric-name column
appears in the affected function/subquery plans. Also resolve two
rebase-introduced planner nits: rewrite a clippy filter_map_bool_then and
reformat a regex error construction.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 e27e944e3a test(sqlness): split TSID join plans by environment
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 65e7e245b7 test(sqlness): cover PromQL metric name output boundary
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 cb18653c9f fix(promql): report dropped modulo histogram samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 d1ace8ff5b fix(promql): report dropped mixed histogram arithmetic samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 1c46e08806 test(promql): update plans for metric name identity
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 b316be8f44 test(promql): assert TSID arithmetic drops metric name
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 68133528aa test(promql): execute TSID joins across logical metrics
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 35bd458994 feat(promql): resolve metric name in label functions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 a660c892e9 feat(promql): hide metric identity in TQL tabular results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 466c641c4f feat(promql): use marked metric names exclusively in responses
Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 5037f8b6c4 feat(promql): apply vector and set metric-name lifecycle
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 4fe3a09adf feat(promql): apply scalar binary metric-name lifecycle
Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 a2057b04c6 feat(promql): apply aggregate metric-name lifecycle
Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 9b6f557d5f feat(promql): apply function metric-name lifecycle
Preserve identity for sorting and last_over_time, drop it after other function results and histogram folding, and resolve semantic name sort keys exclusively through metadata. Seven focused tests pass on Rust 1.96.1. Remaining aggregation, binary/set and output-boundary activation are separate work.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 850890fc6c fix(query): retain dependencies of nested aggregate projections
Collect retained projection expression dependencies before pruning instant inputs. Three unmarked regressions failed with missing aux before the seven-line fix and pass afterward, alongside three selector identity tests on Rust 1.96.1.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 3b6bcbc0ff feat(promql): checkpoint internal selector metric identity
Attach field-marked name columns at selector lowering and remove identity on unary minus. Raw selector execution and collision tests passed; unary fixture lookup corrected but rerun pending. Intermediate stack checkpoint: remaining operator lifecycle and tabular output boundaries must be completed before activation is merge-ready.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 1bb956c6da feat(promql): read metric identity from result field metadata
Add an optional metadata-marked metric-name column to response conversion while retaining caller-provided names for unmarked schemas. Preserve existing sample selection and add focused identity regressions.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:10:49 +08:00
discord9 28ec3ff5aa fix(table): require complete primary keys for hash partitioning (#9507)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
v1.4.0-nightly-20261012
2026-10-10 12:28:47 +00:00
discord9 32e8467cda fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): use current DataFusion plan downcasts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): match null rendering in join regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh union plan for unknown scan partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh shared plans after distribution rule removal

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover semi joins with remote dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen partitioned merge scan regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove unused merge scan test helpers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): measure merge scan hash repartition cost

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt merge scan deserialization to two-arg constructor

Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): adapt nested merge scan integration constructor

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 11:18:46 +00:00
sun ad83232ad1 feat: update dashboard to v0.14.0 (#9510) 2026-10-10 09:21:56 +00:00
fys 9a57e1195d feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com>
2026-10-10 08:31:18 +00:00
dennis zhuang b913be2af7 fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries

An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): keep encode_substrait on a valid range query

A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): plan wrapped range-vector roots and reject a zero subquery step

TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover wrapped range-vector roots and a zero subquery step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): default subquery step and reject range vectors in range queries

A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover root subquery default step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): align root subqueries with subquery_child_window

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): return millisecond timestamps for range-vector results

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): keep the time index timezone and name its table in errors

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 07:04:02 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
discord9 178a0021b8 fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds

Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.

The metasrv-side validate_flow_options remains as a defensive check.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): use DISTINCT output columns as sink key for streaming flows

When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.

Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): preserve extension options and complete DISTINCT sink identity

Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): align meta unknown option status assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): use valid aliases in DISTINCT sink regression SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): record complete DISTINCT sink SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:45:11 +00:00
dennis zhuang a8e293fd79 fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch

Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update flow_tql error for a negative TQL end time

A negative end now parses and fails the end >= start check instead of
the timestamp conversion.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): reject evaluation times outside the millisecond range

The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop an unused import

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round query times to milliseconds like Prometheus

Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round the float fraction before converting to nanoseconds

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: skip the overflow case where SystemTime cannot hold it

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 05:26:21 +00:00
shuiyisong d9bdded741 chore: open middleware (#9502)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-10 04:33:16 +00:00