Commit Graph
1737 Commits
Author SHA1 Message Date
discord9 cbc2a729a5 feat(common-meta): add raw-series preflight and execution surface to Datanode trait (#9659)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-18 15:32:45 +08:00
discord9 3ef71b5702 feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-18 11:33:20 +08:00
discord9 88a1e684a0 feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors (#9656)
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).

## Refer to a related PR or issue link (optional)

Independent PR. No dependencies.

## What's changed and what's your intention?

Recognize `SeriesScanExec` in query metrics and preserve typed errors
through the record batch adapter:

- `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped
stats attribution works.
- `RecordBatchStreamAdapter`: preserves the original typed error via
`BoxedError` instead of wrapping into a generic execution error. This
lets downstream recover the specific status code (e.g.,
`InvalidArguments` instead of `EngineExecuteQuery`).

**2 files, +203/-9 production + 1 sqlness expected result.**

### Error code change

The typed error preservation changes the error code for the
`semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)`
to `1004(InvalidArguments)`. The error message text is unchanged. The
new code is correct: the original error was an argument validation
error, not a generic execution failure. The sqlness expected result has
been updated.

### Verification

- `cargo check --all-targets` ✅, `cargo nextest run -p
common-recordbatch` → **65/65 passed** ✅
- sqlness expected result updated for the typed error code change

## PR Checklist

- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
2026-10-18 09:55:54 +08:00
discord9 5d93b8177c feat(common-query): adapt ordered record batches to raw series (#9652)
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).

## Refer to a related PR or issue link (optional)

No dependencies on other open PRs. Builds on shared series types from
#9598/#9599.

## What's changed and what's your intention?

Add `record_batches_to_raw_series` in `common-query`: converts an
ordinary DataFusion `RecordBatch` stream into contiguous raw series
chunks.

The adapter validates column types at construction (time must be
Timestamp(Millisecond, None), tags must be string/dictionary, fields
must be Float64 or native histogram), encodes series identity via Arrow
RowConverter, and produces `RawSeriesChunk`s with explicit labels and
contiguous ordering. Float64 fields share source buffers (zero-copy);
histograms reuse the existing `histogram_samples_from_arrow` bridge.

Chunking merges equal-key runs within and across batches; a key change
or end-of-input sets `is_last`. Timestamps must be strictly increasing
within a series. Empty batches are skipped with a bounded poll loop.

**3 files, +1666/-0 lines, including 20 tests.** No production callers
yet — consumed by follow-up scan wiring PRs.

### Verification

- `cargo check --locked -p common-query --all-targets` ✅
- `cargo nextest run --locked -p common-query --retries 0` ✅
- `cargo clippy --locked -p common-query --all-targets -- -D warnings` ✅

## PR Checklist

- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
2026-10-18 09:55:47 +08:00
discord9 3caba1b7c2 feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 11:03:51 +08:00
discord9 997cbf87f7 feat(common-query): declare raw series frame ordering in schema metadata
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 11:03:29 +08:00
discord9 1106128fe5 feat(common-query): adapt ordered record batches to raw series
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 10:44:01 +08:00
discord9 eddd595b82 perf(query): batch raw series across stream chunks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-16 16:38:09 +08:00
discord9 a7ca3cce4e fix(query): preserve raw time axis lifecycle proof across contract update
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 21:19:20 +08:00
discord9 0428b4217d fix(query): verify logical frame field bindings
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 20:45:46 +08:00
discord9 b774c32075 fix(query): bound raw frame validation history and empty polling
Keep only adjacent sequence state instead of retaining every completed key. The producer remains responsible for contiguous key runs; ordered streams still reject key regressions.

Yield after 32 validated empty batches per unpack poll without losing pending rows or terminal errors. Preserve whole-batch atomicity and document the producer contract.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 19:18:07 +08:00
discord9 cfb66f5f2e chore: synchronize raw frame stream stack base
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:23:37 +08:00
discord9 1164877b5a chore: synchronize merged series samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:18:18 +08:00
discord9 70a6c4551f refactor(query): share native histogram bucket assembly
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 18:14:47 +08:00
discord9 bdc12034e5 fix(datatypes): address series foundation review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 17:56:57 +08:00
discord9 696259a1ce feat(query): add incremental raw series frame streams
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 17:16:16 +08:00
discord9 9a4d53ea64 feat(query): normalize stored histograms into shared samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 13:08:24 +08:00
discord9 d9d8dfff68 feat(query): estimate quantiles from borrowed histogram views
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 12:41:56 +08:00
dennis zhuang 5c4d9eb4f8 fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE

The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.

Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop unused lazy_static from index and avoid typo-flagged test tokens

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: name probe extraction with verbs and document CharClass

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: rebuild legacy fulltext bloom indexes and probe lower(col)

The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.

Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover legacy bloom index rewrite by compaction in the compat case

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: log outdated fulltext bloom indexes in the consistency check

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: explain the unrecorded fulltext bloom version

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-14 08:07:24 +00:00
Ning Sun f41e86f43a feat(meta-client): forward actor and event context for region follower RPCs (#9568)
The manage-region-follower chain submitted its procedures with an empty
executor context: `ProcedureServiceHandler::manage_region_follower`
accepted no query context, the operator built an
`ExecutorContext::default()`, and `MetaClient` dropped the context before
delegating to `RegionFollowerClient`. The submitting user (actor) and the
procedure event metadata therefore never reached the submitted
procedures.

- `ProcedureServiceHandler::manage_region_follower` now takes a
  `QueryContextRef`, consistent with `migrate_region`, `gc_regions` and
  `gc_table`.
- `ProcedureServiceOperator` derives the `ExecutorContext` (actor and
  `TriggerReason::Manual` event input) from the query context.
- The `RegionFollowerClient` trait methods now accept `&ExecutorContext`
  and `MetaClient` forwards it, so client implementations can transport
  the actor and event context on the wire.
- `procedure_event_context`/`procedure_actor` are now public so client
  implementations can reuse the trusted protocol derivation.
2026-10-14 04:11:53 +00:00
Weny Xu f690bbb204 refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): update new WAL replay tests after index removal

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 15:31:04 +00:00
Weny Xu 863eebdecb feat(frontend): retry transient distributed write failures (#9543)
* refactor(operator): route shared requests through NodeManager

Remove the concrete NodeClients requester API and use the existing node manager for owned and shared requests. Preserve the original write hint while the legacy error adapter remains in place.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): preserve retry policy through error hints

Propagate client errors through External and let DDL read retry hints while preserving cleanup and terminal-child policies. Region transport failures retain the legacy Retryable classification, which now also participates in bounded write retries.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): restore sparse partition tags when rebuilding writes

Decode retained primary keys when refreshed partition expressions need different tags. Keep the partitioner and multi-region sparse write regression tests with the fix.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): stop detached write retries when callers cancel

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(frontend): configure retries across distributed write paths

Wire the shared dispatcher through frontend and logical batching, preserving partition columns and request headers. Keep standalone retries disabled and include configuration, batching, and storage regression coverage.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 14:51:35 +00:00
Weny Xu d86741d0ad feat(operator): retry failed writes with refreshed partitions (#9542)
* feat(operator): retry failed writes with refreshed partitions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): simplify retry test type references

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): clarify write request rebuilding names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): encapsulate write retry attempts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(operator): group inserter configuration methods

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): share routed peers and separate retry decisions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(operator): avoid temporary write routing allocations

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 06:02:43 +00:00
Weny Xu 684c8ff291 feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate route refresh retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: preserve local error variant imports in retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): distinguish staging rejection from write backpressure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): preserve legacy retry hints in instruction replies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(client): inline retry hint compatibility cases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(datanode): align batch-open retry hint assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: use imported types in region request tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 02:29:54 +00:00
discord9 f50f42140a build: reduce checkout-specific inputs for cache reuse (#9198)
* test(mito2): cover regex inverted index pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* build: keep build inputs checkout-relative for cross-worktree cache reuse

Two small changes so compilations are byte-identical across worktrees
of the same commit and can share a build cache (mbx, sccache):

- src/common/version/build.rs: derive the workspace root relative to
  CARGO_MANIFEST_DIR instead of the CARGO_WORKSPACE_DIR injected by
  .cargo/config.toml (an absolute path). The env var remains as an
  explicit override for the submodule case; the [env] injection in
  .cargo/config.toml is removed so the relative derivation is used by
  default.
- Replace three env!("CARGO_MANIFEST_DIR") uses in test/bench code
  (servers prom_decode bench + decode test, common-meta test_util)
  with a runtime std::env::var lookup, so the compiled artifact no
  longer embeds the checkout path.

No behavior change: same files are read, same version info emitted.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* build: address cross-worktree cache review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-12 08:08:21 +00:00
jeremyhi 18726c7602 fix: bound metric snapshot memory and reduce export overhead (#9505)
* fix(mito): bound retained scan memory

Compact unused tag dictionary values before merge retention and avoid splitting
a single scan source into redundant batches. Include regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(snapshot): bound writes and reduce logical export overhead

Bound metadata writes to 8 MiB and abort failed writers. Preserve compatible
string dictionaries, bound conversion work, combine Parquet row groups and
filter single-table exports before routing. Include resource-boundary and
packed-restore regression coverage.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: clarify metric snapshot acceptance criteria

Specify populated scale fixtures, controlled S3 request delay, native macOS
validation, paging exclusions and alternating comparison rounds.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): split batches only for merged readers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(snapshot): cover supported types in both layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: consolidate metric snapshot experiment conclusions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito): honor series statistics when splitting L0 scans

Apply the existing series-density heuristic to L0 files with known statistics and include them in the batch-size estimate. Preserve the fallback for older files without statistics.

Cover short-series overlapping L0 files with an ordered latest-value regression.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito): explain memory risk of splitting short L0 series

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address snapshot export review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(cli): accept source errors in lazy snapshot contexts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-12 06:17:03 +00:00
dennis zhuang f3a0841f4f fix(promql): follow IEEE 754 float semantics in comparisons and aggregations (#9479)
* fix(promql): follow IEEE 754 float semantics in operators and aggregations

- Comparisons: Arrow orders floats totally (NaN == NaN, NaN > +Inf); PromQL
  comparisons with NaN are false except !=.
- min/max: skip NaN unless the whole group is NaN. DataFusion's grouped
  min/max let NaN replace and be replaced by neighbours, so the result
  depended on input order.
- topk/bottomk rank NaN last.
- round() rounds ties up and returns NaN for a zero step, like Prometheus.
- sqrt() of a negative number is NaN instead of an error.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): round() without a step rounds to integers

The planner passed a zero step for single-argument round(), which now
yields NaN like Prometheus; pass 1 instead. Found by the compatibility harness.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): apply the sqrt guard to builtin functions and register min/max for plan decoding

sqrt resolves through the session's builtin functions, not the UDF branch.
Distributed plans are decoded with the serializer's session, which did not
know prom_min/prom_max.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover NaN and rounding semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the nested count rewrite for PromQL min/max

count(min(x) by (l)) was rewritten by count_nest_aggr only for the builtin
min/max names. prom_min/prom_max return NULL for the same groups, so the
rewrite stays valid. Update plan snapshots for the IEEE comparison, sqrt,
topk and min/max changes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): register prom_min/prom_max in the function registry

Distributed plans split an aggregation into state and merge steps only for
functions whose state function exists in FUNCTION_REGISTRY; prom_min and
prom_max lived in the promql crate, so min/max stopped being pushed down to
datanodes. Move them to common-function and register them before the state
functions are derived.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: update sqlness results for PromQL float semantics

Plans and column names now show prom_min/prom_max, the NaN guards of
comparisons and sqrt, and the NaN sort key of topk. round() results follow
Prometheus' formula. Distributed min/max keep the state/merge split.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep column names and NULLs in IEEE comparisons and sqrt

Rewriting comparisons as `a op b AND NOT (isnan(a) OR isnan(b))` changed
TQL output column names and turned NULL operands into true for `!=` and
false for the bool forms. Implement comparisons and sqrt as PromQL UDFs
that propagate NULL and report the column name of the expression they
replace; prom_min/prom_max report min/max column names the same way.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: restore column names in sqlness results for PromQL float semantics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): share the prom_min/prom_max groups accumulator with tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep distinct names for prom_min/prom_max

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-12 05:10:18 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
discord9 178a0021b8 fix: unify unknown flow option rejection across OSS and enterprise builds (#9319)
* fix(operator): unify unknown flow option rejection across OSS and enterprise builds

Remove the enterprise-gated passthrough that let unknown flow options
through SQL-layer validation. Both OSS and enterprise binaries now reject
unknown flow options with a single 1004(InvalidArguments) error at the
SQL layer, producing identical sqlness output.

The metasrv-side validate_flow_options remains as a defensive check.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): use DISTINCT output columns as sink key for streaming flows

When a streaming flow's source table has no primary key, sink key
inference produced an empty key. For SELECT DISTINCT v, the DISTINCT
establishes v as the output key, but the inference only looked at
source primary keys. Combined with the zero-valued placeholder
timestamp, all output rows shared the same storage identity and were
deduplicated into a single row.

Use all non-time-index output columns as the sink key when the plan
is a DISTINCT and no source primary key was inherited.

Signed-off-by: discord9 <discord9@163.com>

* fix(flow): preserve extension options and complete DISTINCT sink identity

Keep enterprise CREATE FLOW options available to the existing handler and report unknown ordinary options as InvalidArguments with the OSS SQL error text. Infer DISTINCT sink keys from every selected non-time-index output column, including when source key columns survive. Wait for asynchronous mirrors in show_create_flow and exercise 1000 distinct sink rows in the SQL regression.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): align meta unknown option status assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): use valid aliases in DISTINCT sink regression SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): record complete DISTINCT sink SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:45:11 +00:00
shuiyisong a907cbfbe4 perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues for tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: bound asynchronous trace auxiliary writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: bound trace auxiliary cache by estimated memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: size trace auxiliary cache from host or pod memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add metrics for cache hit and miss

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: unify frontend cache metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: merge and reduce

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add init for the metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-10 02:45:43 +00:00
fengyue-xve f1a27ab599 refactor(function): rename uadf_impl to udf_impl in aggregate functions (#9482)
refactor: rename uadf_impl to udf_impl in aggregate functions

Signed-off-by: fengyue-xve <1759519137@qq.com>
2026-10-09 09:58:41 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
dennis zhuang 526dd48bda fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: verify compound filter semantics and last-row selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: consolidate label filter regression coverage

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: separate filter evaluation and timestamp adaptation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: inline predicate and timestamp tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: keep direct IN predicates outside simple filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: coerce partition bounds before simple filtering

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover partition bound overflow after timestamp widening

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 07:43:25 +00:00
jeremyhi 3c0e2a8d55 feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject empty packed export time ranges

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover cancellation during packed export I/O

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 15:22:02 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
jeremyhi 03823a9a01 feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL

Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog
thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the
object store WAL config, validated by the datanode and the store.

In the `enqueued` mode `append_batch` returns on admission with the
entry ids assigned and the object is created in the background. At a
backlog threshold the next append is held back until an upload
completes. A transient create failure is repeated under the same
sequence with the same bytes; any conflicting object poisons the
store. `stop` uploads the backlog, or returns the error that dropped
it once stop began. `obsolete` clamps the watermark to the durable
entry id, and an id the store handed out needs no sequence floor.

Add `LogStore::wait_durable` with a default that returns at once. The
object store WAL answers it once the region is durable and indexed
through the entry id, and fails it after a backlog was lost.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): poison an enqueued store on permanent create failures

Repeat a failed create in the enqueued mode only when the storage error
is retryable; any other storage error poisons the store. A conflicting
object poisons an enqueued store without reading its epoch, so a failed
header read cannot turn the conflict into a retry.

A durability wait for an id above the highest id the store handed out
now waits for the handed-out ids of the region below it instead of
returning at once.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): poison on permanent create failures after stop begins

A create that fails with a storage error that is not retryable poisons
an enqueued store even after stop began; only a transient failure drops
the backlog without poisoning. Durability waiters whose callers stopped
waiting are pruned before a new waiter is queued. The backlog age test
no longer depends on a follow-up append finishing within the threshold.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): answer a durability wait once no earlier entry is pending

A durability wait now returns once the region holds no entry at or
below the target that is handed out but not durable, instead of waiting
for the largest id handed out to the region. A later object of the
region that is still being created no longer holds back a wait whose
target it does not cover.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): order the pending durability wait check after the actor

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(store-api): state that the default wait_durable keeps each store's guarantee

The default `LogStore::wait_durable` returns at once, which keeps each
log store's own acknowledgement guarantee; Raft Engine with
`sync_write = false` acknowledges before its periodic sync, so the
documentation no longer claims that every entry id a caller holds is
durable. The object store WAL configuration test now also serializes
the new options and reads them back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): limit the acknowledgement guarantees to the durable mode

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): repeat enqueued creates that a retry layer marks persistent

An object store wrapped in the OpenDAL retry layer reports a temporary
error that outlasted its retries as persistent rather than temporary.
The enqueued mode now repeats a create after any storage error that is
not permanent, so a transient outage behind the retry layer no longer
poisons the store and drops the acknowledged backlog; after stop began
such a failure still drops the backlog without poisoning.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover a persistent create failure after stop begins

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 09:17:45 +00:00
DeviousCardiandClaude Opus 5.5 d5a6a9ac05 fix(auth): follow symlink chains in watch_file_user_provider (#9365)
* fix(auth): follow symlink chains in watch_file_user_provider

The shared file watcher only watched the immediate parent directory of the
configured path, so swapping a directory symlink elsewhere in the chain
(e.g. `ln -sfn secrets.2 secrets`, as sops-nix does) never produced an
event and the new users file was not loaded until restart.

The watcher now resolves each path component by component and also watches
the directory holding every symlink it meets, plus the directory of the
final file. After each relevant event the chain is re-resolved and the
watches are re-armed, so later in-place edits of the new target are seen.
Only the final file's directory is required at startup; a link directory
that cannot be watched (e.g. mode 0711) is logged and retried later.

Events are now filtered to the chain's links and final files, so that
watching e.g. /tmp does not reload on unrelated files. Names are compared
case-insensitively on macOS and Windows. TLS cert reload uses the same
helper.

Close #9310

Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL

* fix(common-config): warn when the file watcher watches the root dir

On macOS /tmp, /var and /etc are symlinks under /, so following a symlink
chain there adds / to the watched directories, and the FSEvents backend
then sees every event on the volume. Log a warning when that happens.
Also shorten the comment on skipped link directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL
Signed-off-by: Aarav <aaravsjadav@gmail.com>

---------

Signed-off-by: Aarav <aaravsjadav@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 04:33:46 +00:00
jeremyhi bf67246645 perf: bound concurrent Metric export writers (#9296)
* fix: preserve objects after conditional Metric export collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: share Metric export writer and payload budgets

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover bounded Metric export conversion and mixed roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve export output after ambiguous close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: align HTTP export timeout with managed cancellation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: trim redundant Metric export roundtrips

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: bound pending Metric export writer handles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: protect export files on failed writes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve ordinary export cleanup and large rows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: report Metric export collisions as invalid arguments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed conditional export closes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support view arrays in managed ordinary export

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: expect successful cleanup after failed close

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up unsynced overwrite after close failure

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: scope export JSON expansion estimates to JSON columns

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: rechunk ordinary export batches within the write budget

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 03:10:21 +00:00
Weny Xu 5ffd01a70a fix: preserve primary key order when syncing columns (#9189)
* fix: preserve primary key order when syncing columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject invalid SyncColumns metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-27 02:15:47 +00:00
Yingwen 3c2aac0a55 feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(storage): route series index build requests

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(admin): add BUILD_SERIES_INDEX

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: specify compaction type in series index fixtures

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: correct series index SQL fixtures and error assertions

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(compat): preserve legacy index rebuild across upgrades

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(sql): cover series index admin validation

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): bound series index maintenance queue

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: remove series index how-to guide

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: remove index build upgrade compatibility case

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: address series index reconciliation review feedback

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: bound manual series index reconciliation admission

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update greptime-proto to merged index build options

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-24 09:39:21 +00:00
dennis zhuang 357c935804 fix: make vector aggregates work with GROUP BY and partial aggregation (#9338)
* fix: make vector aggregates work with GROUP BY and partial aggregation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: make partitioned vec_avg case distinguish weighted averages

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 08:45:40 +00:00
Ning Sun 20619273ee chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates

Move the workspace from the pinned nightly-2026-03-21 to stable
1.96.1 and drop all 23 '#![feature]' gates across 13 crates,
rewriting the still-unstable API usages with stable equivalents:

- try_blocks: closures / an async block (table, query, common-function, servers)
- duration_constructors: Duration::from_secs(n * 86400) / (n * 60)
- iterator_try_collect: collect::<Result<Vec<_>, _>>()
- box_patterns: as_deref() + matches! chains (sql)
- error_iter: error_chain_root() source-chain walker (common-error);
  sources() includes the error itself, so the walker never panics
- int_roundings: div_floor -> div_euclid (equal for positive divisors)
- iter_partition_in_place: stable sort_by_key partition helper (index)
- hash_set_entry: HashSet::insert bool / contains+insert
- trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin)
- string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned()
- never_type: Infallible (common-recordbatch)
- debug_closure_helpers: closure-backed DebugFmt newtype (mito2)
- binary_heap_pop_if: peek().is_some_and() + pop()
- exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds
- stmt_expr_attributes: stale gate, no usages

Also fix release-dev-builder-images.yaml, which parsed
rust-toolchain.toml with a date-only regex and would produce empty
image versions with a stable channel; it now extracts the full
channel token. Dev-builder images verified against stable 1.96.1
(image build, default-toolchain behavior, binstall/nextest, riscv64
and android targets, in-image cargo check).

Validated on 1.96.1: cargo check --workspace --all-targets, clippy
--workspace --all-targets --all-features -D warnings, cargo fmt
--check, and nextest on all 13 affected crates (4586 passed).

Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging
first.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: update flake checksum

* chore: use wild for linker in flake

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-24 08:37:43 +00:00
dennis zhuang affc0a1b1d fix: align ordered aggregate state type with the accumulator output (#9340)
* fix: align ordered aggregate state type with the accumulator output

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: reject mismatched aggregate states and keep hard-ordered aggregates unsplit

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: keep WITHIN GROUP aggregates splittable and relabel state fields by position

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 07:02:14 +00:00
dennis zhuang cc1be82858 fix: merge every state row in geo_path and json_encode_path (#9339)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 02:12:11 +00:00
shuiyisong 2bc18c483d feat: add schema metadata stream to SchemaManager (#9325)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-09-23 12:25:56 +00:00
Weny Xu a35f9d5c78 fix: address Windows test failures and run full Windows CI (#9305)
* fix: use relative object keys for Windows filesystem access

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: cover Windows path and time limits in full test CI

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: use relative keys in metadata snapshot tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-23 05:16:41 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
dennis zhuang f3eb8e6c72 test: cut integration test time and make the storage matrix meaningful (#9308)
* test: cut integration test time and make the storage matrix meaningful

tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.

- Point the PR CI object-store matrix at the MinIO already started by
  tests-integration/fixtures. Three GT_S3_* consumers did not read
  GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
  they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
  release integration-test container. The release previously ran every
  remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
  /tmp/greptimedb_cache, which the datanode wipes on startup, so a
  starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
  non-trivial SST representation: JSON/JSON2 columns, native histograms,
  metric-engine logical tables, and tables carrying fulltext or skipping
  indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
  shorten it in the event tests, which otherwise wait a 5s window per DDL
  they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
  test_grpc_message_size_ok and is now rewritten to assert the negotiated
  grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
  prefixes of their copy_from siblings; two standalone/distributed event
  test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
  in a parallelism-dependent order, so its last digits are unstable; the
  test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
  ttl=7d with 2025 timestamps, so its rows were only readable as long as
  they stayed in the memtable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: address review — wire nightly real-S3 job into check-status, keep the short event interval

The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.

In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 01:52:52 +00:00
Lei, HUANG d0f8f4b80c feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 14:12:24 +00:00
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-22 12:56:26 +00:00