Commit Graph
789 Commits
Author SHA1 Message Date
discord9 1817d0bb76 fix(promql): close metric-name identity privilege and boundary gaps
- count_values never forges the identity: the label it generates is always
  an ordinary column named as asked, with no metric-name role metadata and
  no reuse of the marked column's physical name. A plain aggregate argument
  can no longer inject a sample value into the privileged identity column
  (which the HTTP layer would otherwise emit as __name__ and TQL would
  silently strip). With no marker forged, dropping the name of a
  count_values result cannot collide, so the union-subtree check remains
  the only duplicate-label-set source.
- binary_match_label_mask no longer lets an ordinary on()/ignoring() label
  hit the marked identity by its physical name; only the semantic __name__
  resolves to it, and a real user column sharing the name still matches.
- label_join/label_replace move the marked identity to a free internal
  name before writing an ordinary destination that occupies its physical
  name, so writing a label can no longer silently replace the identity.
- sort_by_label reads a missing or NULL string label as the empty value,
  so a series that lost its name sorts first ascending like Prometheus,
  instead of NULLS LAST.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 6106a6ea08 refactor(promql): unify semantic label resolution and label rewrites
Converge the ordinary-label and __name__ paths that had forked during the
rebase:

- resolve_label_col() is the single label-to-column entry: __name__ reads
  the marked identity column, ordinary labels resolve like before but can
  no longer hit the marked column by its physical name, and a missing
  label is absent for every caller (label functions, aggregation, sort).
- label_replace keeps one algorithm for every destination: read source
  (missing as ""), match anchored regex, expand captures (empty -> NULL),
  keep the old destination value when unmatched. Ordinary destinations may
  overwrite existing labels again (#9385 semantics); collapsing series is
  refused by the existing runtime label-set check instead of a plan-time
  InvalidDestinationLabelName.
- write_destination_label() is the single destination-column entry for
  label_replace and label_join, so writing an empty label has one
  representation and a zero-source label_join on __name__ removes the
  identity instead of projecting an unmarked physical __name__ column.
- drop_metric_name() now asserts a unique label set per evaluation
  timestamp when dropping the identity of a union that may ignore it
  (abs(a or on(__name__) b)), reusing assert_unique_match_group; TQL
  strip_metric_name_column() stays free of PromQL series checks.
- sort_by_label(..., "__name__") on input without a name sorts by a
  missing label instead of failing with field_not_found.

Production code is a net reduction (-38 lines); the removed ordinary-label
regex branch, the overwrite tri-state and the duplicated destination
construction are replaced by the three shared entries.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 e86a22733c test(promql): align planner expectations with rebased main
Refresh the planner test expectations after rebasing onto upstream main
(78a7b93292, #9385): ordinary label_join of a missing label now joins as
the empty string and keeps its separator, label_replace of an existing
destination is refused at plan time, and the internal metric-name column
appears in the affected function/subquery plans. Also resolve two
rebase-introduced planner nits: rewrite a clippy filter_map_bool_then and
reformat a regex error construction.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:13:07 +08:00
discord9 cb18653c9f fix(promql): report dropped modulo histogram samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 d1ace8ff5b fix(promql): report dropped mixed histogram arithmetic samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 1c46e08806 test(promql): update plans for metric name identity
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 b316be8f44 test(promql): assert TSID arithmetic drops metric name
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 68133528aa test(promql): execute TSID joins across logical metrics
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 35bd458994 feat(promql): resolve metric name in label functions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 a660c892e9 feat(promql): hide metric identity in TQL tabular results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:24 +08:00
discord9 5037f8b6c4 feat(promql): apply vector and set metric-name lifecycle
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 4fe3a09adf feat(promql): apply scalar binary metric-name lifecycle
Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 a2057b04c6 feat(promql): apply aggregate metric-name lifecycle
Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 9b6f557d5f feat(promql): apply function metric-name lifecycle
Preserve identity for sorting and last_over_time, drop it after other function results and histogram folding, and resolve semantic name sort keys exclusively through metadata. Seven focused tests pass on Rust 1.96.1. Remaining aggregation, binary/set and output-boundary activation are separate work.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 850890fc6c fix(query): retain dependencies of nested aggregate projections
Collect retained projection expression dependencies before pruning instant inputs. Three unmarked regressions failed with missing aux before the seven-line fix and pass afterward, alongside three selector identity tests on Rust 1.96.1.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 3b6bcbc0ff feat(promql): checkpoint internal selector metric identity
Attach field-marked name columns at selector lowering and remove identity on unary minus. Raw selector execution and collision tests passed; unary fixture lookup corrected but rerun pending. Intermediate stack checkpoint: remaining operator lifecycle and tabular output boundaries must be completed before activation is merge-ready.

Signed-off-by: Greptime Agent <agent@localhost>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 11:12:23 +08:00
discord9 32e8467cda fix(query): stop reporting region scans as hash partitioned (#9491)
* fix(query): stop reporting region scans as hash partitioned

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): use current DataFusion plan downcasts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): match null rendering in join regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh union plan for unknown scan partitioning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh shared plans after distribution rule removal

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): cover semi joins with remote dynamic filtering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen partitioned merge scan regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove unused merge scan test helpers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): measure merge scan hash repartition cost

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt merge scan deserialization to two-arg constructor

Adapt the production and test serializer callers added by #9206 and remove the obsolete partition_cols assertion.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): adapt nested merge scan integration constructor

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 11:18:46 +00:00
fys 9a57e1195d feat(json2): json_get respect type hints (#9349)
Signed-off-by: fys <fengys1996@gmail.com>
2026-10-10 08:31:18 +00:00
dennis zhuang b913be2af7 fix(promql): return range-vector results for instant queries (#9477)
* fix(promql): return range-vector results for instant queries

An instant query of a range selector or subquery (e.g. `up[5m]`) returned
the internal range arrays, which panicked when the frontend decoded them as
dictionaries. Plan such queries as the selected samples with their own
timestamps, and reject range-vector expressions in range queries like
Prometheus does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): keep encode_substrait on a valid range query

A range query of a range-vector expression now fails like in Prometheus.
Wrap the subquery in max_over_time so the case still encodes both
RangeManipulate nodes through substrait.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): plan wrapped range-vector roots and reject a zero subquery step

TQL EXPLAIN, ANALYZE and AS alias wrap the root in an extension, which the
range-vector path did not unwrap. A zero subquery step divided by zero.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover wrapped range-vector roots and a zero subquery step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): default subquery step and reject range vectors in range queries

A root subquery without a step uses the one-minute default instead of
the query step. The HTTP range query endpoint rejects range-vector
expressions even when start equals end.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover root subquery default step

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): align root subqueries with subquery_child_window

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): return millisecond timestamps for range-vector results

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): keep the time index timezone and name its table in errors

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 07:04:02 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
dennis zhuang a8e293fd79 fix(promql): accept evaluation times before the Unix epoch (#9480)
* fix(promql): accept evaluation times before the Unix epoch

Prometheus accepts negative timestamps in query parameters; GreptimeDB
rejected them while parsing and would panic converting them in the planner.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update flow_tql error for a negative TQL end time

A negative end now parses and fails the end >= start check instead of
the timestamp conversion.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): reject evaluation times outside the millisecond range

The signed millisecond conversion cast u128 to i64 unchecked, so a far-past
time wrapped into the future. Reuse the checked conversion from
label_values and return an error instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop an unused import

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round query times to milliseconds like Prometheus

Numeric times round the fraction of a second to milliseconds, and
pre-epoch times floor sub-millisecond parts, matching Prometheus'
parseTime and timestamp.FromTime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round the float fraction before converting to nanoseconds

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: skip the overflow case where SystemTime cannot hold it

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-10 05:26:21 +00:00
dennis zhuang 6da5352c53 fix(promql): keep vector labels when a computed scalar is on the right (#9478)
* fix(promql): keep vector labels when a computed scalar is on the right

`metric + time()` or `metric * scalar(other)` planned the join with the
scalar side's context, which has no tags, so every series collapsed into
one unlabeled series. Use the vector side's context when the right operand
is scalar-typed.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): update results for labels kept with a computed scalar on the right

v * pi() and v + time() now keep the vector's labels and time index name,
as in Prometheus.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): qualify columns of an unnamed operand in vector joins

A comparison filter's projection has no table name, so after the join its
time index resolved unqualified and was ambiguous when the other operand had
the same column. Also keep the empty-metric branch unchanged for scalar
left operands.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover comparison filters joined with a scalar

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-09 07:16:42 +00:00
discord9 0d2cb1aa7e fix(promql): align subqueries to the absolute evaluation grid (#9400)
* fix(promql): align subqueries to the absolute evaluation grid

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(promql): streamline subquery comments

Remove redundant arithmetic narration and restated test comments. Keep grid, fallback, validation and overflow contracts, and preserve all regression tests and assertions. Regenerate SQLness output only for comment echoes.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-09 03:50:54 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
dennis zhuang 526dd48bda fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: verify compound filter semantics and last-row selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: consolidate label filter regression coverage

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: separate filter evaluation and timestamp adaptation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: inline predicate and timestamp tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: keep direct IN predicates outside simple filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: coerce partition bounds before simple filtering

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover partition bound overflow after timestamp widening

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 07:43:25 +00:00
dennis zhuang 78a7b93292 fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics

- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): align label_replace and label_join edge cases with Prometheus

- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): join absent labels as empty strings in label_join

concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): skip rows without a sample in timestamp()

timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 01:42:11 +00:00
dennis zhuang 218000e21b fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns

col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): resolve remote read matcher labels as unqualified columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover same-name columns differing in case and without()

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-29 08:13:28 +00:00
DeviousCardiandClaude Opus 5.5 d6474b961c fix(promql): apply offset to subquery evaluation window (#9364)
* fix(promql): apply offset to subquery evaluation window

`prom_subquery_expr_to_plan` destructured `SubqueryExpr` without reading
`offset`, so `<subquery>[range:step] offset <d>` planned exactly the same
window as the un-offset form and silently returned data for the wrong time
range. The plain vector/matrix-selector paths already threaded the offset
through `selector_to_series_normalize_plan` and `RangeManipulate`.

Shift the inner evaluation window back by the offset and pass the offset to
the subquery's `RangeManipulate`, which maps the inner samples forward onto
the evaluation timeline before bucketing them into ranges. This matches
Prometheus, whose `evaluator.subqueryTimeRange` evaluates the inner
expression over `(start - offset - range, end - offset]` and whose
`evalSubquery` then hands the samples to the outer range-vector function as
a `MatrixSelector` that still carries the subquery offset. An offset on the
inner selector composes additively, as `subqueryTimes` documents.

`RangeManipulate`'s protobuf message has no offset field and recovers it on
decode from an immediately underlying `SeriesNormalize`. Since
`RangeManipulate` is commutative in `dist_plan` and can be pushed below a
`MergeScan`, insert that carrier node so the offset survives distributed
planning instead of decoding as zero.

Known divergence, unchanged by this commit: Prometheus anchors subquery step
points on absolute epoch multiples of the step, while GreptimeDB anchors them
on the evaluation start. The two agree whenever the offset is a multiple of
the subquery step; the added sqlness cases stay within that range.

Closes #9330

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* docs(promql): correct the subquery-offset rationale and pin the histogram shift

Follow-up to e132c267, which stated two things inaccurately and left one
path untested. No behaviour change beyond a conditional clone.

The `SeriesNormalize` carrier was described as existing "so the offset
survives distributed planning", implying standalone is unaffected. That is
wrong. `local_offset` is read on the substrait decode path
(`range_manipulate.rs`, `instant_manipulate.rs`), and GreptimeDB routes
PromQL plans through `MergeScan`/substrait in standalone too -- the
standalone results `promql/encode_substrait.result` and `precisions.result`
both show `MergeScan [is_placeholder=false, remote_input=[...]]`. Deleting
the carrier therefore empties subquery-with-offset results in standalone as
well, while `count_over_time_subquery_with_offset` keeps passing because it
asserts the pre-serialization plan. Say so at the node, so the next reader
does not remove it believing standalone is safe.

The carrier is also not semantically inert: `SeriesNormalizeStream::normalize`
shifts native histogram `start_timestamp` payloads by `offset`. On the
subquery path that runs on a computed inner result, on top of any offset the
inner selector already applied, and nothing pinned it. It is consistent --
`RangeManipulate` moves the millisecond time index forward by the same
`offset_ms` -- so the distance from a histogram's start timestamp to the
sample carrying it, which reset/rate detection reads, is invariant. Add
`subquery_offset_shifts_histogram_start_and_time_index_together`, which
executes the subquery's node stack over native histogram samples for a
zero and a non-zero inner offset. It cannot be a sqlness case: native
histograms are a struct column with no SQL type or literal
(`sql_data_type_to_concrete_data_type` rejects structs) and the sqlness
runner speaks only MySQL/Postgres, so such rows only arrive over
gRPC/remote-write v2.

The divergence note blamed sub-step offsets. Measured, that condition is too
narrow: an unaligned evaluation timestamp alone diverges, with no offset at
all -- `sum_over_time(fine[20s:10s])` at t=57 samples 47s and 57s here
against Prometheus's 40s and 50s. The real condition is that
`(start - offset)` is not a multiple of the subquery step, and it is
pre-existing: the committed `tql eval (359, 359, '1s')
sum_over_time(metric_total[60s:10s])` case is already an instance of it
(`359 - 60 + 10 = 309`). Reword the comment and the sqlness header
accordingly. A sub-step offset stays accepted: `foo[20s:10s] offset 5s` is
valid PromQL, erroring on it would be a regression and would not close the
gap anyway. This repo has no PromQL-compatibility documentation page, so
there is nowhere else to record it.

Also stop cloning the series key columns when there is no offset, and stop
the `offset 0s` sqlness comment implying its parser error is evidence about
this fix -- the same message comes from a `u32` overflow in
`promql-parser`'s shared duration check (`offset 9999999999d`).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* refactor(promql): trim subquery-offset comments and tests per review

Shorten the planner and sqlness comments to the semantic contract, build
`SeriesDivide` with a clone instead of an `Option`, and drop the
normalize.rs histogram test and the `offset 0s` parser case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>

* fix(promql): record the subquery offset as the range fold offset

The subquery's `RangeManipulate` now shifts the payload timestamps by the
offset, so `predict_linear` over an offset subquery must recover the
evaluation time with that offset instead of 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>

---------

Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-29 03:38:43 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
discord9 28e415a103 feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors

The planner previously dropped the `at` field of selectors (`at: _`),
so `some_metric @ 300` silently returned step-following values instead
of the samples anchored at the fixed timestamp.

Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch`
semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply
`offset` to the anchor, rewrite the selector offset to
`eval_start - anchor`, scan only the anchored window, then report the
same window at every evaluation step via a grid-wide-replay
`InstantManipulate`. `@ start()` / `@ end()` resolve against the
statement's evaluation range (`stmt_start` / `stmt_end`), which subquery
planning does not rewrite.

Timestamps before the Unix epoch are accepted; unrepresentable ones are
rejected with `AtModifierTimestampOutOfRange` instead of wrapping.
Selectors without `@` are planned exactly as before.

Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): center predict_linear on the evaluation timestamp

predict_linear_impl used the window's last sample time as the evaluation
timestamp, and the UDF took only (ts_range, value_range, t) with no channel
for the current instant. After a range selector is folded for @ (or with an
offset) the same window is replayed at every step, so the prediction stayed
constant at the anchor's answer; even a plain window ended before the step
produced a stale value.

Give prom_predict_linear a 4th argument carrying the step's evaluation
instant (ms timestamp), derived in the planner from the row's time index
plus the offset the window was folded with (at_offset for @, offset_ms
otherwise, recorded on PromPlannerContext). The regression is centered on
that instant, matching Prometheus' use of enh.Ts. The mixed
float/native-histogram path forwards the same argument.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): tighten @ modifier planner and predict_linear

- range_fold_offset is always a concrete millisecond offset, not an
  Option; the single-step replay helper no longer wraps an infallible
  plan in Result.
- predict_linear's eval timestamp is always cast to Timestamp(ms) at the
  call site, so the UDF drops its dead Int64 branch and the extra
  func_name argument.
- Drop two redundant comparison cases from the at_modifier sqlness test
  and fix the lookback window comment to the half-open (0s, 300s].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): accept parse-time rejection of @ on Windows

`@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime
holds it and the planner rejects the anchor it cannot represent, but a
Windows SystemTime tops out near 1.8e12 seconds, so the parser's
checked_add fails first and the same literal is rejected while parsing.
Accept either rejection path so the test passes on both platforms.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): avoid replaying label_join over rewritten series keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): narrow anchored range call promotion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): address @ modifier review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): satisfy super import format check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): address at modifier review follow-ups

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 09:38:12 +00:00
discord9 f956f4da30 refactor(promql): split planner into focused modules (#9354)
* refactor(promql): move planner tests into separate module

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): move function-specific planner methods into module

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): move OR planner method into module

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): move binary island planner into module

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): keep binary result labels in planner root

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): limit planner helper visibility to parent module

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): group set operator planning and localize imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): use crate-rooted planner imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 05:30:33 +00:00
discord9 678aa81cae fix(client): complete transport lane isolation (#9030)
* fix(client): complete transport lane isolation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(client): deprecate legacy single-manager constructors

Per review: mark the legacy single-manager constructors and helper as
deprecated so callers move to the isolated query/control manager pair.
Tests intentionally exercising the legacy path are annotated with
`#[allow(deprecated)]`.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(client): update deprecated constructor callers for CI

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 04:39:20 +00:00
discord9 4ffd93d8dc fix(query): preserve global limits with DataFusion optimizer fixes (#9071)
* fix(query): preserve global fetch through physical-optimizer repartitioning

Aggregate soft-limit pushdown inserts a global CoalescePartitionsExec(fetch)
below an AggregateExec whose input is HashPartitioned/KeyPartitioned. The
DataFusion physical optimizer's EnforceDistribution/EnforceSorting passes then
dropped or rewrote that fetched coalesce, losing the global limit. Root-caused
and fixed in the DataFusion fork (GreptimeTeam/datafusion PR #35); this repo
pins to that fix commit and adds regression coverage.

- pin datafusion to GreptimeTeam/datafusion 48510c7f5 (fetch preservation fix)
- global_limit.rs: adapt to new DF API (input_distribution_requirements,
  child_distribution, replace_children+Recompute); accept HashPartitioned and
  KeyPartitioned as partitioning to restore; add regression unit tests
- sqlness: extend standalone common ssts and refresh distributed ssts_limit
  result for the preserved fetch plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: drop dead HashPartitioned compat arms and test wrapper

- remove HashPartitioned compatibility matches (pinned DF only emits
  KeyPartitioned); drop their #[expect(deprecated)] attributes
- remove single-use agg_with_limit helper whose seed limit is always
  overwritten by the soft-limit transform
- restore Cargo.lock dependency edges unrelated to the DataFusion pin
  bump (cargo update --precise had downgraded 26 unrelated packages)

Addresses oracle ablation review.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(deps): pin DataFusion to merged fetch preservation fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 03:35:36 +00:00
Ning Sun 20619273ee chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates

Move the workspace from the pinned nightly-2026-03-21 to stable
1.96.1 and drop all 23 '#![feature]' gates across 13 crates,
rewriting the still-unstable API usages with stable equivalents:

- try_blocks: closures / an async block (table, query, common-function, servers)
- duration_constructors: Duration::from_secs(n * 86400) / (n * 60)
- iterator_try_collect: collect::<Result<Vec<_>, _>>()
- box_patterns: as_deref() + matches! chains (sql)
- error_iter: error_chain_root() source-chain walker (common-error);
  sources() includes the error itself, so the walker never panics
- int_roundings: div_floor -> div_euclid (equal for positive divisors)
- iter_partition_in_place: stable sort_by_key partition helper (index)
- hash_set_entry: HashSet::insert bool / contains+insert
- trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin)
- string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned()
- never_type: Infallible (common-recordbatch)
- debug_closure_helpers: closure-backed DebugFmt newtype (mito2)
- binary_heap_pop_if: peek().is_some_and() + pop()
- exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds
- stmt_expr_attributes: stale gate, no usages

Also fix release-dev-builder-images.yaml, which parsed
rust-toolchain.toml with a date-only regex and would produce empty
image versions with a stable channel; it now extracts the full
channel token. Dev-builder images verified against stable 1.96.1
(image build, default-toolchain behavior, binstall/nextest, riscv64
and android targets, in-image cargo check).

Validated on 1.96.1: cargo check --workspace --all-targets, clippy
--workspace --all-targets --all-features -D warnings, cargo fmt
--check, and nextest on all 13 affected crates (4586 passed).

Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging
first.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: update flake checksum

* chore: use wild for linker in flake

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-24 08:37:43 +00:00
discord9 abedeb2bea fix(query): keep count_values generated label in enclosing expressions (#9223)
* fix(query): keep count_values generated label in enclosing expressions

`count_values("v", m)` projects the generated label as a real output
column, but did not register it in `ctx.tag_columns`. Enclosing
expressions (abs/round/+1/topk/label_replace/vector join) rebuild their
projection from `ctx.tag_columns` and silently drop the label.

Register the generated label in `ctx.tag_columns` after the projection,
and give it the same qualifier as other tag columns so qualified
references resolve. PromQL overwrites an input label with the same name,
so drop any existing tag with that name first to avoid duplicate column
ambiguity.

Fixes https://github.com/GreptimeTeam/greptimedb/issues/9181
Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-1

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): use total_cmp in count_values test helper

Silence clippy::needless_borrow on partial_cmp(&right.1); f64 sorting
uses total_cmp, matching the other planner test helpers.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): keep count_values generated label as sink table primary key

FindGroupByFinalName::f_up only renamed a group key when the projection
aliased the key column directly. count_values now projects its generated
label as a unary UDF over the sampled column
(prom_float_to_string(value) AS label) while the aggregate still groups
by the raw column, so the name match failed and the sink table lost the
label from its PRIMARY KEY, demoting it to a DOUBLE value column.

Allow a projection above the aggregate to rename a group key by deriving
its output from a single group-key column (unary scalar function or cast
over that column only). Multi-column expressions, case, aggregates,
windows, subqueries and literals are still rejected so a computed column
cannot be mistaken for the group key.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): group count_values by the formatted sample value

The numeric count_values branch grouped by the raw sample column and only
formatted the value into the generated label in the post-aggregate
projection. Two distinct raw values that collapse to one label text
(e.g. BIGINT 9007199254740992 and 9007199254740993, both 9007199254740992
in Float64) were split into two groups, each emitting the same label set
at one timestamp, violating Prometheus' unique-label-set-per-timestamp
invariant.

Return the formatted value expression (prom_float_to_string, with a
CAST to Float64 for non-Float64 inputs) as previous_field_expressions so
the aggregate groups by the same expression that produces the label,
mirroring the existing mixed float/native-histogram precedent.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): drop needless borrow in count_values test helper

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): don't replace group key with derived unary expr

FindGroupByFinalName treated any unary expression of a single group
column (scalar fn, Cast, TryCast) as a rename of that group key and
swapped it in as the sink primary key. A derived expression such as
lower(host) is many-to-one, so distinct groups (HOST_A vs host_a) would
collapse to the same primary key and be silently deduplicated.

Narrow the matching to direct name-matched aliases of the actual group
expression only, and drop the derived-unary path (is_unary_expr_of_column)
and the allow_derived flag. count_values already groups by the formatted
expression name, so it is unaffected.

Add a regression asserting host survives and host_lc does not replace it
under both optimizer settings.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-24 08:18:41 +00:00
dennis zhuang def0c2ec5c fix: only push down aggregates grouped by the partition columns themselves (#9337)
* fix: only push down aggregates grouped by the partition columns themselves

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: keep grouping sets on the frontend for partitioned tables

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 06:49:52 +00:00
dennis zhuang b9f991502c refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 05:38:54 +00:00
dennis zhuang 88197f4019 fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings

A vector-vector binary operation projected one operand's whole tag set and
inner-joined without any cardinality check, so `on()`/`ignoring()` did not
reduce the result labels, `group_left`/`group_right` changed nothing, and a
non-unique match group produced a cross product that PromQL cannot represent.

Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the
matching labels, `ignoring(...)` drops them, and a group modifier keeps the
many side's labels plus the `group_x(...)` labels taken from the one side.
A label the one side does not carry is deleted from the result. The reduced
label set no longer identifies the operand series, so `__tsid` is dropped
from the context on this path.

Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)`
window and a scalar UDF that fails the query on a repeated group: on the one
side before the join, and on the result labels after it, matching where
Prometheus raises each of its three errors. Series are unique by their whole
tag set, so the window is only planted when the match keys drop a tag; plain
arithmetic and `on(<all tags>)` plan exactly as before.

Closes #9207, closes #9208, closes #9209.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): keep __tsid when the result labels are an operand's whole tag set

Deriving the result labels dropped `__tsid` from the context unconditionally,
so an enclosing operation fell back to joining on the tag columns even where
the column still identified the result series.

Keep it when every result label comes from one operand and covers that
operand's whole tag set: no other operand value reaches the labels, and the
matching gives each of its rows a single partner, so its `__tsid` is still one
per result series. That is the common `on(<all tags>)` and bare `group_left`
shape; a matching that actually drops a tag still clears it.

The column is re-qualified as the result's own, which is how the enclosing
expression and the context look it up.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the match group count column unambiguous

The cardinality check aliased its row count to a fixed `__promql_match_group_count`.
An operand carrying a label of that name made the window output two fields with
the same name, and planning failed with "Schema contains qualified field name
collide_right.__promql_match_group_count and unqualified field name
__promql_match_group_count which would be ambiguous".

Pick a name the operand does not already have, the way the `or` operator
allocates its match key columns.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover a match group spread over several regions

The cardinality check runs above the merge of the region scans, so it counts a
match group globally. Nothing pinned that: every table in these cases holds a
single region, and a check evaluated per region would pass them all.

Partition the operand on a column outside the match keys, which puts the two
series of one match group in different regions, and assert both the pre-join
and the post-join check still reject it. The case runs in the distributed
environment too, where the regions sit on different datanodes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): group an outer aggregate on the labels the operand kept

`by`/`without` planning topped up a missing grouping column by walking down to
the table scan and re-projecting it. That is right for a column the scan pruned
for efficiency, but the labels a matching modifier deletes are also absent from
the operand's output, and they were restored the same way:

  sum without(host) (a / on(host) b)

`on(host)` leaves the operand with `host` alone, so the sum covers everything
and Prometheus answers `{} 10`. Instead `device` came back from the scan under
`a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for
`sum by(device)` of that operand, which has no `device` to group on at all.

Take the grouping labels from the operand's own label set rather than from the
row keys of the scan beneath it. A label pruned from the plan is still in that
set and still gets restored; a label the operand dropped is not.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): drop the unreachable aggregation tag top-up

`by`/`without` planning could restore a grouping column that the plan no longer
carried by rewriting the scan underneath it. Once the grouping labels come from
the operand's own label set, there is nothing left for it to restore: a scan
projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds
matcher columns to that set), so a label in the set is always in the schema.

Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the
standalone and the distributed environment.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): drop cases that guarded the removed tag top-up

Three plain selector aggregates were there to show that restoring a pruned
grouping column still worked. With the restore gone they only repeat what the
aggregate cases already cover. Also fix a comment that still said the metric
engine scan prunes tag columns: it projects every label of the operand.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): note the duplicate a propagated matcher hides

A matcher copied onto the one-side operand removes groups without a partner
before the cardinality check sees them, so a duplicate in such a group is not
reported. Prometheus checks every group of the one side and fails the query.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 11:08:18 +00:00
Weny Xu a35f9d5c78 fix: address Windows test failures and run full Windows CI (#9305)
* fix: use relative object keys for Windows filesystem access

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: cover Windows path and time limits in full test CI

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: use relative keys in metadata snapshot tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-23 05:16:41 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
dennis zhuang 25bca4609f perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 11:25:50 +00:00
Lei, HUANG b5199bc59a feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: gate Jev filtering behind an opt-in Cargo feature

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: verify Jev registration with default features

Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: clarify Jev concurrency scope and stabilization work

Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: enable renamed ai-functions feature by default

Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: remove extra AI feature-off checks

Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: rename AI feature to ai_functions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 03:03:23 +00:00
discord9 233e23b352 fix(promql): keep value-field grouping labels out of matching filter propagation (#9242)
* fix(promql): keep value-field grouping labels out of matching filter propagation

#9202 propagates matching-label matchers to the scanned selector using the
planned operands' tag columns. For an aggregated operand those are its
grouping labels, and agg_modifier_to_col resolves by(...) names against the
input schema only, so a value field named in by(...) is reported as a tag.
A value field varies between the samples of one series, so lowering a
matcher on it below sample selection (PromInstantManipulate) can drop the
newest sample, promote a stale one from the lookback window, and fabricate
a match the un-rewritten query does not produce.

Track the by(...) labels that name value fields of the aggregated operand's
input in PromPlannerContext::aggregation_field_labels, and refuse to
propagate matchers on them while still requiring a matcher to name a
grouping label to cross an aggregation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: sync matching_filter result fixture comment with the PR number

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 07:09:31 +00:00
discord9 9e9a8cac20 fix(query): insert MergeScan into nested scalar subqueries (#9261)
* fix(query): insert MergeScan into nested scalar subqueries

DataFusion 55 keeps uncorrelated scalar subqueries as expression
subqueries (enable_physical_uncorrelated_scalar_subquery, default true)
instead of decorrelating them into joins, and executes them via the new
physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked
the plan with a plain TreeNode transform that does not descend into
expression subqueries, so MergeScan was only inserted for depth-1
subqueries. A scalar subquery nested inside another scalar subquery kept
a bare frontend DistTable TableScan and failed at execution with
"Unsupported operation: get stream from a distributed table".

Use the subquery-aware transform so handle_subquery (PlanRewriter /
MergeScan insertion) runs for subquery plans at every nesting depth.

Fixes #9260.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen nested scalar subquery regression coverage

Address review on #9261:
- Replace the ineffective 'no bare TableScan' string check with a real
  subquery-aware plan walk (apply_with_subqueries); MergeScan hides its
  remote input from traversal, so any TableScan the walk reaches was
  genuinely left unwrapped.
- Add a distributed regression case on a range-partitioned table so the
  nested inner aggregate must merge partial results across regions
  (global AVG feeding an outer SUM filter).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 03:58:38 +00:00
fys e82cb0af74 feat(json2): respect type hints during JSON2 type concretization (#9222)
* fix: prefer JSON2 type hints for uncast read pushdown

* feat(query): materialize JSON get result types before planning

* fix(query): apply JSON2 type hints to parsed paths

* fix(query): apply JSON2 type hints before distributed planning

* chore: remove f

* refactor: the code style of json_get_type_hint

* chore: add sqlness case

* fix: add json expr planner, remove json get type hint

* chore: add sqlness test

* fix(query): respect JSON2 type hints in query planning

* chore: add more sqlness cases

* fix: cargo clippy

* fix: cargo clippy

* chore: update sqlness test result
2026-09-20 07:31:04 +00:00
dennis zhuang 007836ba7a fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query

`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.

Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.

Only metric engine tables are covered; other engines share no column space
to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): batch-resolve metric names by table id

Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.

The catalog walk stays, but only to find the physical tables to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): read an absent label as the empty string

A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.

Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.

This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): don't panic on pre-epoch label value bounds

`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.

Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): drop applicable_matchers, share the distinct scan

With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.

Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 03:11:06 +00:00
dennis zhuang b1106a9c5a perf(promql): propagate matching-label filters between binary operands (#9202)
* perf(promql): propagate matching-label filters between binary operands

A one-to-one arithmetic binary expression inner-joins its operands on the
matching labels, so every row that survives the join already satisfies the
other operand's equality matchers on those labels. Copy those matchers to the
other operand so both scans drop non-joining series before execution instead of
feeding them to the join.

Both operands are planned before the rewrite: a selector matcher can also
constrain a value field, and only the planned contexts tell tags and fields
apart. A matcher is copied only when its name is a tag column on both sides, its
value is non-empty, and it is one of the matching labels. Operands are limited
to vector selectors, parentheses, and label-preserving range functions applied
directly to a matrix selector; `ignoring(...)`, group modifiers, fill values,
set and comparison operators, regex matchers and `or` matcher groups are left
alone.

Selector matchers are now deduplicated in place instead of through a `HashSet`,
so the generated scan filter keeps a stable order once a selector carries more
than one matcher.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): propagate ignoring, non-equality and aggregated matchers

Widens the matcher propagation added in the previous commit to the shapes it
was leaving on the table.

The join compares matching labels with plain column equality
(`normalized_match_key_expr` and its coalescing are confined to `or`), so any
predicate on a matching label is already enforced on both sides for every
surviving pair: it originates on one operand, and the join carries it to the
pairs it forms. Copying it to the other operand can only drop rows that had no
surviving partner. That argument does not depend on the matcher kind, so regular
expressions, negations and empty values now propagate too. `=~".*"` stays out:
it lowers to no filter at all, so copying it would only force a re-plan.

`ignoring(...)` is no longer rejected. A label is a join key exactly when it is a
tag on both sides and not named in `ignoring`, which is what
`binary_join_key_columns` computes and what the caller can now answer from the
two planned contexts.

Operands may now be aggregations that partition by their grouping labels, which
is the shape most real queries use. `agg_modifier_to_col` rewrites
`ctx.tag_columns` to the grouping labels, so a label found in an aggregated
operand's tags is a group key, and filtering the aggregate's input by it drops
exactly the corresponding output groups. `topk`, `bottomk` and `limitk` are
excluded because they select across a group and carry input labels through --
`prom_topk_bottomk_to_plan` leaves `ctx.tag_columns` alone, so the tag check
cannot catch them. `count_values` is excluded because it adds an output label
that does not exist in its input.

The rollup whitelist now covers every label-preserving range function the
planner implements, and finds the matrix argument by position so multi-argument
rollups such as `quantile_over_time` and `predict_linear` qualify.
`absent_over_time` stays out: it synthesizes a series from the matchers when its
input has none.

Re-running the sqlness case against an unmodified planner produces a
byte-identical `.result` for all 27 queries.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): pin that excluded matching labels stay on their own operand

`ignoring(device)` and `on(host)` both leave `device` out of the join keys, so a
`device` matcher must not reach the other operand. Neither case was covered end
to end: a regression there silently drops the left operand's `eth1` series
instead of returning them, which the two added queries now catch.

Also corrects the comment at the rewrite site, which still described re-planning
as touching a leaf selector. Aggregated operands are re-planned as a whole; what
holds is that the rewrite only ever adds matchers to a selector, so the enclosing
operand's table reference, time index and field columns are unchanged.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): log why matching-filter propagation was skipped

Splits the guard chain into `try_propagate`, whose `Err` names the reason the
rewrite does not apply, and logs it once in `propagate`. Debugging a query that
did not get the filter no longer means stepping through the guards.

The reasons also replace the comments that used to explain the same conditions,
and the remaining comments lose the parts that restated the code or repeated
each other.

Two of them were wrong rather than verbose. Saying `topk`'s input "must not be
filtered" reads as a claim about PromQL: `topk(1, m{host="x"})` is perfectly
legal, and what matters is that filtering before `topk` changes the candidate
set it ranks. Saying the subset-matching case "keeps its many-to-many result"
read as an endorsement of behaviour Prometheus rejects outright; it now records
that the query is a cross product here and points at #9209.

The propagation cannot change whether such a check would fire: series in one
match group agree on every matching label, so a matcher over one of those labels
keeps all of them or drops all of them.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep copied matchers out of the operand's selector metadata

Re-planning an operand with a copied matcher also rebuilt its
`PromPlannerContext::selector_matcher`, and that context is what an enclosing
expression reads. `create_absent_plan` turns the equality matchers found there
into the labels `absent()` reports, so

    absent(counter_metric{host="missing"} / on(host, device) gauge_metric)

gained a `host="missing"` label that `main` does not produce. The inner
expression was empty either way; only the reported label set changed.

The copied matcher belongs to the scan, not to the operand's identity, so both
re-planned contexts now keep the matchers their operand was written with.
`selector_matcher` has one other reader, `create_table_scan_plan`, which
consumes it while the operand is being planned and is unaffected.

Reported by @discord9.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-18 07:40:33 +00:00
jeremyhi dca01654bc feat: prepare and execute database Metric exports (#9180)
* feat: prepare and authorize captured database exports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: bound database export jobs and drain failures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: validate database export identity and restore equivalence

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate database export directory URLs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve Windows export directory paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject local database export filename aliases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: consolidate database export planning policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: restore escaped database export filenames

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align database restore assertions with shared policies

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: clarify database export boundaries and names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-17 12:02:39 +00:00
LFC aacf04cf6e feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-17 10:08:45 +00:00
discord9 39e34c0b74 refactor: migrate partition_statistics to statistics_from_inputs (#9175)
DataFusion 55 deprecated ExecutionPlan::partition_statistics in favor of
statistics_from_inputs with StatisticsContext::compute. Migrate the two
remaining GreptimeDB overrides, RegionScanExec and MergeScanExec, and
update the RegionScanExec test to the new API, dropping its
allow(deprecated).

Behavior is unchanged: RegionScanExec keeps the append-mode-only,
exact-source-rows statistics gate from #9154, and MergeScanExec keeps
reporting unknown statistics for per-partition requests.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 08:23:16 +00:00