Commit Graph
6245 Commits
Author SHA1 Message Date
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-09-22 12:56:26 +00:00
dennis zhuang aa36f74feb fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 14:38:22 +00:00
dennis zhuang 25bca4609f perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 11:25:50 +00:00
Ning Sun 2388257c35 ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-09-22 08:37:21 +00:00
houyuwushang 614b46e5f5 fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
2026-09-22 08:03:40 +00:00
Dhruv Vaishnav f1e9a74f00 fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
2026-09-22 07:35:07 +00:00
jeremyhi 9dfe057199 test: make export chunk deletion failure deterministic (#9291)
* test: make export chunk deletion failure deterministic

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify export chunk deletion failure fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: guide deterministic storage failure tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 04:58:58 +00:00
Weny Xu 67497f14e5 refactor: isolate logical table preparation and reuse Flow notifications (#9212)
refactor: isolate logical table preparation and share Flow notifications

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-22 04:09:59 +00:00
dennis zhuang e65f10c8ad fix(query): stop encoding oversized dynamic filters (#9267)
* fix(query): stop encoding oversized dynamic filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(query): stop bounded encoding partway through an IN list

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-22 04:02:40 +00:00
jeremyhi 2a63ea2660 feat(log-store): add object store WAL reads and obsolete watermarks (#9248)
* feat(log-store): add object store WAL reads and obsolete watermarks

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): reuse decoded WAL payload allocations

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): clarify WAL region mismatch error

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 04:01:31 +00:00
Lei, HUANG b5199bc59a feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: gate Jev filtering behind an opt-in Cargo feature

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: verify Jev registration with default features

Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: clarify Jev concurrency scope and stabilization work

Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: enable renamed ai-functions feature by default

Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: remove extra AI feature-off checks

Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: rename AI feature to ai_functions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 03:03:23 +00:00
LFC 16978cf6c2 feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2

Signed-off-by: luofucong <luofc@foxmail.com>

* fix: remove unused annotation context import

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-22 02:39:36 +00:00
jeremyhi 4a980f350f docs: make packed snapshots part of the Metric export/import RFC (#9247)
* docs: revise Metric snapshots around packed Parquet objects

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: summarize experiment conclusions and bound multipart fallback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: specify packed snapshot persistence contract

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 00:13:13 +00:00
jeremyhi 19c127d9c3 feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: restore packed metric snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: stream large packed parquet entries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: allow default S3 endpoint in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: skip unconfigured S3 in packed restore test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use portable file URLs in packed restore fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate packed snapshot structure before restore

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve strict manifest decoding and verify fixture

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject packed layout on both database export paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: implement file size in coordinator test storage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: use expect_err for rejected export layouts

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 00:07:04 +00:00
Lei, HUANG edbcb8224e fix: fail startup on duplicate region engine configs (#9281)
* fix: reject duplicate region engine configurations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: reuse canonical engine names in config validation

Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: clarify zero-based duplicate engine config indices

State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 09:19:07 +00:00
jeremyhi 263e229103 feat: add experimental Metric export to V2 snapshots (#9233)
* feat: add experimental Metric export to V2 snapshots

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: validate the complete Metric export capability response

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: construct portable file URLs for Metric export fixtures

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: address Metric export review nits

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-21 09:10:01 +00:00
Weny Xu e27167139f refactor: isolate logical batch scheduling and flushing (#9211)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-21 07:56:28 +00:00
Lei, HUANG 6a084d9a4f fix: release completed SST write buffers during compaction (#9243)
* fix: upgrade OpenDAL and honor SST write buffer size

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: track removal of the OpenDAL bridge fork

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): honor SST write buffers across upload paths

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:47:23 +00:00
shuiyisongandLei, HUANG 66d38e8e1c feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add `InsertLimitInterceptor` hook to `Inserter`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: attribute insert limit checks to the target table's database

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: unify write admission through metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce write admission for pending row batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: distinguish internal requests for ingestion metering

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: admit split ingestion requests once per database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: OpenTSDB throws error reason

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use meter crate main rev

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude database ingest rate limit from table options

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: reserve channel 255 for internal requests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-21 07:27:22 +00:00
dennis zhuang b332bb67d6 perf(telemetry): remove the shared lock from TraceLayer (#9269)
* perf(telemetry): remove the shared lock from TraceLayer

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(telemetry): stop collecting span data while tracing is disabled

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-21 07:19:20 +00:00
discord9 233e23b352 fix(promql): keep value-field grouping labels out of matching filter propagation (#9242)
* fix(promql): keep value-field grouping labels out of matching filter propagation

#9202 propagates matching-label matchers to the scanned selector using the
planned operands' tag columns. For an aggregated operand those are its
grouping labels, and agg_modifier_to_col resolves by(...) names against the
input schema only, so a value field named in by(...) is reported as a tag.
A value field varies between the samples of one series, so lowering a
matcher on it below sample selection (PromInstantManipulate) can drop the
newest sample, promote a stale one from the lookback window, and fabricate
a match the un-rewritten query does not produce.

Track the by(...) labels that name value fields of the aggregated operand's
input in PromPlannerContext::aggregation_field_labels, and refuse to
propagate matchers on them while still requiring a matcher to name a
grouping label to cross an aggregation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: sync matching_filter result fixture comment with the PR number

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 07:09:31 +00:00
discord9 95f693c396 perf(servers): stream Prometheus HTTP responses incrementally (#9229)
* perf(servers): stream Prometheus HTTP responses incrementally

Replace try_collect + full-buffer conversion in the Stream branch of
from_query_result with incremental per-batch merging via the shared
merge_batch path, so record batches are dropped as soon as they are
consumed instead of all coexisting in memory.

- Add ColumnLayout to dedupe schema inference shared by both paths
- Add SeriesKeyLookup (Equivalent-based borrow lookup) so series keys
  are looked up by borrowed label slices without materializing owned
  keys per lookup
- Empty streams short-circuit to empty data before schema inference
- Add stream-side mixed histogram + Vector dual-path equivalence test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(servers): address review comments on stream response tests

- Inline single-caller tags_capacity helper
- Drop timestamp column from empty-stream test so it pins the
  short-circuit happens before schema inference
- Fix cross-batch merge test data so a series actually spans the
  batch boundary (second batch carries earlier timestamps)

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(servers): add lazy stream release assertion and pinned cross-batch content

- LazyDropCheckingStream creates one batch per poll and asserts via
  Weak that the previous batch was released before the next poll,
  protecting the core property that batches are dropped eagerly
- Pin series "a" merged samples to exact timestamps and values in
  stream_result_matches_record_batches_result, so a regression in
  shared merge_batch logic cannot make both paths agree on wrong output

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 07:08:39 +00:00
Ning Sun 4df557bb9e test: exclude testing feature completely (#9072)
* test: exclude testing feature completely

* chore: fmt
2026-09-21 06:45:04 +00:00
Weny Xu d375851abd fix(ci): stabilize long-range benchmark execution and artifact collection (#9241)
* fix(ci): store long-range benchmark data on disk

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): update vmbench runtime and stage benchmark artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): reduce vmbench generator batches and parallelism

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): use runtime with unified benchmark report metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-21 05:33:42 +00:00
discord9 6647dbc437 feat!: upgrade DataFusion fork to 55.1.0 (#9177)
Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0.
DataFusion 55.1.0 is a patch release on branch-55 containing eleven
cherry-picked fixes (schema-adaptation struct filters, cast/projection
metadata propagation, nested-nullability aggregation adaptation,
UnnestExec batch_size, RightMark join ordering panic, empty-struct
ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches
rebase onto it with no textual or semantic overlap; none of them is
absorbed upstream, so all are retained.

Fork pin moves to discord9/datafusion branch greptimedb-55.1.0,
commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 +
20 patches).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 04:14:22 +00:00
discord9 9e9a8cac20 fix(query): insert MergeScan into nested scalar subqueries (#9261)
* fix(query): insert MergeScan into nested scalar subqueries

DataFusion 55 keeps uncorrelated scalar subqueries as expression
subqueries (enable_physical_uncorrelated_scalar_subquery, default true)
instead of decorrelating them into joins, and executes them via the new
physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked
the plan with a plain TreeNode transform that does not descend into
expression subqueries, so MergeScan was only inserted for depth-1
subqueries. A scalar subquery nested inside another scalar subquery kept
a bare frontend DistTable TableScan and failed at execution with
"Unsupported operation: get stream from a distributed table".

Use the subquery-aware transform so handle_subquery (PlanRewriter /
MergeScan insertion) runs for subquery plans at every nesting depth.

Fixes #9260.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): strengthen nested scalar subquery regression coverage

Address review on #9261:
- Replace the ineffective 'no bare TableScan' string check with a real
  subquery-aware plan walk (apply_with_subqueries); MergeScan hides its
  remote input from traversal, so any TableScan the walk reaches was
  genuinely left unwrapped.
- Add a distributed regression case on a range-partitioned table so the
  nested inner aggregate must merge partial results across regions
  (global AVG feeding an outer SUM filter).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-21 03:58:38 +00:00
github-actions[bot]andgreptimedb-ci 9bad25e701 ci: update dev-builder image tag (#9270)
Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com>
2026-09-21 03:43:54 +00:00
Weny Xu 941193e9c1 refactor: reorganize logical table batching and isolate encoding (#9210)
* refactor: relocate the logical table batcher

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: isolate logical batch conversion and region writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-21 03:37:47 +00:00
Ning Sun 055bbbc93d feat: update pgwire to 0.41 (#9025)
* feat: update pgwire to 0.41

* feat: update datafusion-pg-catalog and arrow-pg
2026-09-21 02:35:05 +00:00
Ning Sun 04f07a91b9 ci: use mold in dev-builder (#9262) 2026-09-21 02:26:49 +00:00
Lei, HUANG e9dd79d1d2 fix(mito2): compare primary key ranges across schema versions (#9205)
* fix(mito2): compare primary key ranges across schema versions

Bind FileHandle ranges to the pinned region schema and append cached constant defaults to historical Dense keys. Preserve raw SST statistics, reject inexact bounds, and avoid invalidating views for unrelated metadata changes.

Cover schema evolution, default changes, and tombstone retention through real compaction and reopen regressions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): report invalid primary key ranges

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): scope primary key ranges to comparisons

Keep raw PK bounds only in FileHandleInner and move schema-aware mapping and caching into task-local comparison contexts.

Use explicit contexts for compaction overlap checks, window aggregation, and series scans. Preserve pinned-schema isolation, late statistics, and shared file lifecycle state without rebinding every handle.

Cover cache isolation across region owners and adapt range fixtures to real Dense encodings. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): cache aligned primary key ranges per file

Replace task-local range maps with a single-slot cache in FileHandleInner, keyed by the target schema version. Preserve raw bounds for realignment across snapshots and default changes.

Share schema mappers across comparison paths and use copy-on-write SST lists for metadata updates. Simplify range mapping to accept encoded bounds and assert the same-table contract at the file accessor.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify primary key mapper schema snapshot

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): align primary key range fixtures with table contract

Remove obsolete cross-table fallback expectations after region validation became a caller contract. Give compaction fixtures matching table identities, including the active-window L1 scenario.

Clarify the mapper precondition and format the simplified alignment call. All 1530 mito2 tests and Clippy for all targets with the testing feature pass.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): enable filesystem GC in release unit tests

Let unit tests use the filesystem-backed object-store GC path regardless of optimization profile. Keep the production release GC selection unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito-codec): skip release value decoding in PK prefix counts

Validate field values only in debug builds and unit tests while keeping boundary, truncation, and trailing-byte checks in every build.

Cover the linked library in debug and release integration tests, and verify that release unit tests still perform value validation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito-codec): remove redundant prefix integration tests

Retain the codec unit tests and cross-schema compaction regressions while dropping the standalone build-profile test file and its release-only invalid-value expectation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: wait for MySQL to accept authenticated TCP queries

Add a healthcheck using the configured test account and database. Docker Compose --wait previously only observed container startup because the fixture image had no healthcheck, allowing metasrv to connect before MySQL initialization completed.

Verify readiness with SELECT 1 over TCP rather than the initialization socket.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-20 14:30:12 +00:00
LFC fcb165cf77 feat(trace): support Jaeger queries for Trace V2 and optimize writes (follow-up to #9192) (#9257)
feat(trace): support Jaeger queries for v2 and optimize fixed-column writes

Signed-off-by: luofucong <luofc@foxmail.com>
2026-09-20 08:53:00 +00:00
dennis zhuang 462172b9bb perf(servers): reduce temporary memory in protocol handling (#9249)
* perf(prom): release the remote write v1 decode buffer before writing

remote_write_v1 kept the decoded builder alive until the handler returned,
so the decompressed request payload stayed resident across the downstream
write or pipeline await. With 8 concurrent large requests that is one extra
copy of every payload held for the whole write.

Rows and pipeline values own their data, so the builder can be dropped as
soon as the conversion is done.

Sustained-write A/B, 8 runs per side, 50M samples each: jemalloc allocated
median drops 7.8% (155.0-162.7 MiB -> 141.4-158.6 MiB); samples per CPU
second is unchanged (-0.6%, fully overlapping ranges).

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(servers): move row values into SQL JSON responses

The format=json renderer cloned every serde_json::Value and kept the whole
row set alive while building the response. Move the values out instead, so
each row is released as soon as it is converted.

Slicing the row to the schema width keeps the panic on rows narrower than
the schema; a plain zip would silently truncate them. Duplicate column names
still resolve to the last value and extra row values are still ignored.

Isolated conversion measurements: live peak drops 33% on a 4096-row 16 KiB
string fixture and 36% on a nested-JSON fixture, with no measured slowdown.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(prom): release the compressed remote write body after decompression

Both the v1 and v2 decoders held the compressed Bytes until they returned,
which spans the whole protobuf decode and row conversion. Decompression
copies the payload into an independent buffer, so the body can go as soon
as it succeeds.

The compression fallback, decode errors and request counting are unchanged.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* revert(prom): keep the remote write v1 decode buffer until the write finishes

This reverts commit 8232c5b3bb.

The v1 decoder fabricates `&'static [u8]` pointing into its own decode buffer
(prom_remote_write/types.rs), so the compiler checks nothing about that
buffer's lifetime. Holding the builder until the handler returns is what keeps
the decoder safe by construction; releasing it early made that safety depend on
every consumer copying out of the buffer, which holds today but nothing
enforces.

Document the requirement at the binding instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(prom): release the remote write v1 decode buffer before writing

This reverts commit ca3af722d0, restoring
8232c5b3bb.

The decoder borrows the decompressed buffer while parsing, but copies
everything out when it builds rows: tag values through
`PromValidationMode::decode_string`, column names through `to_owned`, and the
only live borrows (`TableBuilder::col_indexes`) are dropped inside
`as_insert_requests`. The resulting `ContextReq` holds prost types with no
lifetime parameters, so it cannot reference the buffer.

Record that at the binding so the next reader does not have to re-derive it
from three files.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 07:46:40 +00:00
Lei, HUANG f40d20110c feat(mito2): split SWCS output files by size threshold (#9259)
* feat(mito2): split SWCS output files by size threshold

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): use resolved SWCS output size options

Read the SWCS output file size threshold from the resolved region options so database-level compaction settings are honored. Align the existing picker test with this configuration source.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-20 07:45:37 +00:00
fys e82cb0af74 feat(json2): respect type hints during JSON2 type concretization (#9222)
* fix: prefer JSON2 type hints for uncast read pushdown

* feat(query): materialize JSON get result types before planning

* fix(query): apply JSON2 type hints to parsed paths

* fix(query): apply JSON2 type hints before distributed planning

* chore: remove f

* refactor: the code style of json_get_type_hint

* chore: add sqlness case

* fix: add json expr planner, remove json get type hint

* chore: add sqlness test

* fix(query): respect JSON2 type hints in query planning

* chore: add more sqlness cases

* fix: cargo clippy

* fix: cargo clippy

* chore: update sqlness test result
2026-09-20 07:31:04 +00:00
dennis zhuang bbc4c4bf1d perf(otlp): share trace resource and scope attributes (#9253)
* perf(otlp): share trace resource and scope attributes

Parsing an OTLP trace request copied the resource attributes and the
scope attributes once per span. A request with wide resource attributes
kept one full copy per span alive until the rows were built.

Spans and their group now share one `Arc` per resource and per scope.
Empty attributes stay `None`, so a resource or scope without attributes
costs no allocation and no refcounting. v0 takes ownership back when it
encodes, v1 clones attributes item by item instead of rebuilding a Vec,
and v2 borrows the span instead of cloning it for every row.

Column values, attribute order, the `service.name` lookup and the
auxiliary table writes are unchanged.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(otlp): stop copying v1 attribute keys

The v1 row writer rebuilds every column name as `{prefix}.{key}`, so the
key string it cloned from the shared resource and scope attributes was
dropped unused. `resource_attributes.service.name` was also cloned in
full before being skipped for the top level service name column.

Shared attributes are now read in place and only values that reach a row
are copied. Span attributes keep moving their values as before.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 06:18:24 +00:00
dennis zhuang 43ce07fa69 fix(mito2): cancel cache construction for incomplete scans (#9254)
* fix(mito2): cancel cache construction for incomplete scans

CacheBatchBuffer spawned the background concat task and dropped its join
handle. A scan that was cancelled or failed therefore left the task alive:
it kept compacting already queued batches, and while waiting for a range
result memory permit it held them, even though without a finish command
the result can never be put into the cache.

Keep the handle and abort it when the buffer is dropped. The handle is
cleared once the task owns the finish command, so a completed scan still
populates the cache after its stream is dropped. Abort does not preempt a
concat that is already running; it takes effect the next time the task is
polled.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(mito2): wait for the permit park before cancelling the buffer

An empty buffered_batches only proves the batches were enqueued, so the
cancellation test could abort a concat task that had never been polled.
Count acquisitions that find too few permits, a test-only signal, and drop
the buffer once the task has reached that wait. Nothing awaits between the
check and the parking, so an observed increment means the caller is about
to wait for permits the test holds for the rest of the case.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 04:58:47 +00:00
jeremyhi 33cfb72a43 fix: make database export assertions portable on Windows (#9256)
fix: compare database export paths portably on Windows

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-20 04:23:37 +00:00
Yingwen 798adb64e7 feat(mito2): make range index reads and builds opt-in (#9219)
* feat: add config flag to control range index reads

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: disable range index builds when configured and default to off

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-20 03:13:52 +00:00
dennis zhuang e99f75f9b3 fix(mito2): hold the bulk compact permit for the whole blocking task (#9252)
CompactDispatcher acquired a borrowed permit inside the async wrapper and
dropped it as soon as the blocking task was submitted, so the semaphore
never bounded the compactions that actually ran. Acquire an owned permit
and move it into the blocking closure so it covers the queue wait and the
merge itself.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 03:12:51 +00:00
dennis zhuang 007836ba7a fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query

`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.

Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.

Only metric engine tables are covered; other engines share no column space
to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): batch-resolve metric names by table id

Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.

The catalog walk stays, but only to find the physical tables to scan.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): read an absent label as the empty string

A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.

Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.

This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): don't panic on pre-epoch label value bounds

`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.

Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(prometheus): drop applicable_matchers, share the distinct scan

With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.

Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 03:11:06 +00:00
dennis zhuang f29a7d8ff0 docs(agents): clarify the documentation-update checklist item (#9255)
Agents routinely check "This PR requires documentation updates" for
changes users never see. Say what the item means: the docs site repo,
not rustdoc or in-repo comments.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-20 01:59:49 +00:00
Mohd Quamar Tyagianddennis zhuang 59eb63c663 fix(servers): merge duplicate timeseries across RecordBatches in Prometheus remote read (#9032)
* fix(servers): merge timeseries across RecordBatches in Prometheus remote read

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>

* fix(servers): exercise recordbatches_to_timeseries and sort samples with sort_unstable_by_key

---------

Signed-off-by: Tyagiquamar <mohdquamartyagi@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-09-19 22:32:09 +00:00
Dhruv VaishnavandWeny Xu 2ab8475b84 feat: add status label to datanode failed-insert metric (#9191)
* feat: label datanode failed inserts by status

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix: rename failed-insert metric label to code

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-19 09:27:10 +00:00
dennis zhuang 0e5bcaee54 ci: pin crate-ci/typos to v1.50.2 (#9246)
crate-ci/typos renamed its default branch from master to main on
2026-09-18 (0a3d75e) and the master branch is gone, so every workflow
run since then fails at job setup with:

  Unable to resolve action `crate-ci/typos@master`, unable to find version `master`

Pin to the latest release tag instead of tracking a branch. This matches
how the other actions in these two workflows are referenced and keeps the
check reproducible.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-19 02:46:12 +00:00
Sainath Singineedi fbbc017be4 feat: expose region min/max timestamp in region_statistics (#9060)
* feat: expose region min/max timestamp in region_statistics

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>

* test: cover region_statistic time range assembly and projected values

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>

---------

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>
2026-09-19 02:26:31 +00:00
jeremyhi 5ccc29ce77 feat(log-store): add object store WAL store construction and recovery (#9238)
* feat(log-store): add object store WAL store construction and recovery

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): derive WAL store prefix from node and generation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe unsupported operation boundary

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-19 02:12:59 +00:00
discord9anddiscord9 983738101a feat(flow): admit mergeable average states in incremental plans (#9235)
Incremental batching flows already merge sink state for scalar
aggregates and the HLL/UDDSketch/stddev state families. AVG now exposes
a mergeable Binary state (avg_state / avg_merge with
__avg_state_delta_merge), so admit those aggregates too instead of
forcing a full snapshot for flows whose only state column is an
average.

merge_op_for_aggregate_expr takes the aggregate input schema so the
avg_merge arm can require a Binary state argument; the state form is
the aggregate result persisted by the sink, so no extra coercion is
needed. Other input types keep being rejected.

Also cover avg_state, avg_merge and duplicate AVG projections in the
incremental plan analysis tests, extend the mixed state-family rewrite
test with an average column, and extend the standalone partitioned
state-merge SQLness case with avg_state/avg_merge compared against a
direct avg over the source.

Signed-off-by: discord9 <discord9@outlook.com>
Co-authored-by: discord9 <discord9@outlook.com>
2026-09-18 11:34:32 +00:00
Lei, HUANG cb6c7a5c6d perf(mito-codec): streamline sparse label extraction and buffer sizing (#9217)
* perf(mito-codec): validate sparse labels once

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito-codec): reserve complete sparse label chunks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-18 09:51:01 +00:00
discord9 27090496d1 test(flow): stabilize FLUSH_FLOW assertions after async source-table mirrors (#9230)
* test(flow): wait for async source-table mirrors before FLUSH_FLOW

Streaming flows execute inline in the flownode insert handler since
#8976, and source-table inserts are mirrored from the frontend as
detached tasks. Tests that INSERT then ADMIN FLUSH_FLOW then SELECT
intermittently miss rows on slow CI runners because the flush no longer
implies the mirror reached the flownode (flow_no_aggr lost row 'l',
flow_advance_ttl lost row '23' after the post-restart reinsert).

Add SQLNESS SLEEP 3s between those mirror inserts and the flush, the
established pattern in the flow suite, so the assertions are stable.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): also guard first-section flushes in flow_advance_ttl

Local reproduction without sleeps failed in a window the previous
commit did not cover: the first INSERT (20,20,22) of each section is
followed immediately by ADMIN FLUSH_FLOW, and the distributed run
observed an empty sink on that SELECT (~1 in 8 iterations). Add the
same SLEEP 3s guard before those two flushes.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-18 08:03:07 +00:00