rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-11-14 06:46:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,239 Commits 403 Branches 285 Tags
narthex
Commit Graph
7239 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
discord9 9018736d37 perf(mito-codec): index sparse label columns by id once at bind time (#10419) 
The sparse-label audit walked the physical primary key once per label:
column() and position() were linear scans over the described columns,
so decoding a key with N labels over M primary-key columns cost O(N·M)
per chunk. Build an id→encoding-position index once in bind() and use
it for both lookups, so the cost drops to O(N). The full structural
audit still runs before any value is read, and error text and
error-selection order are unchanged.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-13 19:10:41 +08:00
dennis zhuang 7b72fb6987 refactor(promql): drop the Go math port and use the standard library (#10392) 
* refactor(promql): drop the Go math port and use the standard library

The pointwise kernels called a port of Go's `math` functions as they behave
on amd64. Results are compared within a tolerance, and the port reproduced
amd64 assembly quirks that Go on arm64 does not share.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): stop claiming map_arithmetic follows Go's math functions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 23:02:04 -07:00
dennis zhuang bdb2e6777b refactor(promql): derive native output columns and run-time labels with one walk (#10394) 
The planner's `native_output_tags` was a copy of `fused_output_tags` in
`promql::native::plan` read over tag columns instead of run-time label
sets. The planner calls `fused_output_tags`, which takes over the one rule
the copy had on its own: `or` reports the metric name of each row.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 22:13:15 -07:00
dennis zhuang f02d6d682a refactor(promql): share NativeNode child traversal across crates (#10391) 
Move `node_children` and `with_node_children` from the query planner to
`NativeExpr::children` and `NativeExpr::with_children`, add
`NativeNode::{any, inputs, is_data_free}`, and replace the hand-written
structural walks over `NativeExpr` with them.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 21:51:23 -07:00
dennis zhuang 21e2255f4b fix(promql): info() ACL targets, per-series routing, reset collisions (#10385) 
* fix(sql): include the table PromQL info() reads in extracted tables

`info(v)` and `info(v, {label="x"})` read `target_info`, which is not written
as a selector in the expression, so table-level permission targets missed it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep per-series provenance behind CooperativeExec

`EnsureCooperative` wraps the `SeriesScanExec` leaf and rebuilds
`RawSeriesSourceExec` over the wrapper, which cleared `per_series_scan` and
left `SeriesFrameOutput` on the merge-inputs router.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): drop histogram pairs whose counter reset hints contradict

Prometheus returns the collision as the error of `+`/`-` and skips the pair
before counting its match. The native path kept the sample and counted it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(frontend): keep info() label selectors out of metric discovery

The table extractor now claims `target_info` only for the selector shape the
planner reads it with; other shapes stay unresolved. `MetricDiscoveryVisitor`
skips the label selector of `info`, which the planner never discovers with,
so an unresolved one is no longer answered by the caller's discoverable tables.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): expect the dropped pair on a histogram reset hint collision

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 21:51:00 -07:00
dennis zhuang f576134b91 refactor(promql): read binary operators' kernel operations from one table (#10393) 
* refactor(promql): drop the Go math port and use the standard library

The pointwise kernels called a port of Go's `math` functions as they behave
on amd64. Results are compared within a tolerance, and the port reproduced
amd64 assembly quirks that Go on arm64 does not share.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): read every binary operator's kernel operation from one table

`BinaryOp::kind` maps an operator to the arithmetic or comparison the map
kernels implement. The vector-vector kernel, the scalar broadcast and the
compiler's constant fold use it instead of their own tables and their own
copies of the arithmetic.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): use BinaryOp::is_comparison for the hand-written comparison checks

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 21:50:13 -07:00
discord9 edfed9a1e9 Merge main into narthex (pre-merge rehearsal) 
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).

## Refer to a related PR or issue link (optional)

Pre-merge rehearsal for keeping `narthex` current with `main`. The proto
pin below points at the branch of GreptimeTeam/greptime-proto#355.

## What's changed and what's your intention?

**This is a merge rehearsal, opened as a draft on purpose.** It merges
`main` (`62dad907d0f`) into `narthex` (`115d85b6699`), resolves all 31
conflicts, and lands on a tree that builds and passes the suites below —
so that the next real back-merge is mechanical instead of a week of
archaeology. Three known blockers are listed at the end; **do not merge
before they are cleared**.

The branch is `main` + narthex's own work: 293 files, +241,450/−2,699
relative to `main`.

| Check | Result |
|---|---|
| `cargo check --workspace` | 0 errors |
| `cargo check --workspace --all-targets` | 0 errors |
| `cargo nextest run -p promql` | 1215/1215 passed |
| `cargo nextest run -p mito2 -E 'test(/series_index/)'` | 185/185
passed |
| `cargo nextest run -p mito2 -E 'test(/cache/)'` | 257/257 passed |

Not run: full workspace test suite, sqlness (`tql-explain-analyze` needs
re-recording), `tests-integration`, `tests/compatibility`.

### The conflicts that actually matter

Six places where "take both sides" compiles but is wrong. Each is marked
`TODO(narthex-merge)` in the tree.

#### 1. `query.experimental_native_promql` is rejected by main's
query-option snapshot — the native route silently disables

main (#9494) added `QueryContext::query_option_snapshot`, which runs
every `query.*` extension through `canonical_query_option_name` and
fails on unknown keys. narthex's native gate reads
`query.experimental_native_promql` from that same extension map, so
after a naive merge **every native request is rejected before reaching
`plan_pql`**. Adding the key to `RESERVED_EXTENSION_KEYS` instead is
also wrong: that list makes the HTTP/gRPC hint layer *drop* the key,
which silently falls back to the legacy planner.

Resolution: register it in
`src/session/src/query_options.rs::canonical_query_option_name`, so the
snapshot accepts it and keeps it on the context.

Review question: should this be a publicly settable option at all, or
should the gate read a typed extension written by `apply_query_options`
(which would also let `SET` work consistently)?

#### 2. `cache.rs`: two independent redesigns of the same
decoded-metadata cache

- main: `SstDecodedMetaKey::new(file_id, PageIndexPolicy)` →
`SstMetaLoadResult`, async `get_decoded_sst_meta`,
`CacheSizeRegistration`, and it **deleted** the `PageRangeCache` /
`PageFragmentKey` / `PageFragmentGroupKey` family in favour of
`ParquetPageCache` + `RangeResultCache`.
- narthex: `DecodedKey`/`DecodedValue` over `SstMetaKey`,
`Cache::builder().weigher(shared_weight)`, and
`CacheManager::page_cache: Option<Arc<PageRangeCache>>` consumed by
`cache/columnar.rs`.

Resolution: main's architecture is the baseline; narthex's columnar
metadata gets its own `CacheManager::columnar_meta_cache` tier, funded
by a quarter of `sst_meta_cache_size` with the decoded tier reduced from
half to a quarter, so the total budget stays exactly what the user
configured.

The budget split is **gated on the format**:
`enable_columnar_meta_cache` is only set when
`experimental_series_index_format == Columnar`, so the default
deployment keeps main's exact `1/2 + 1/2` split and never constructs the
extra tier. A first cut of this merge reserved the quarter
unconditionally, which would have halved main's decoded tier for
everyone; that is fixed in the second commit of this PR.

**Deferred cost: narthex's native payload byte cache is now disabled**
(`columnar_payload()` returns `None`). main's `ParquetPageCache` only
serves uniform-size pages through a `RangeReader` and cannot hold these
16 KiB payload blocks, and reviving a second page cache would
double-count `greptime_mito_cache_bytes`. Two tests that assert
warm-payload reuse are `#[ignore]`d. Unlike the budget split this one
*does* change behaviour when the columnar format is on, and needs an
explicit decision.

#### 3. `series_index`: main's memtable-pin fix vs narthex's format
selection

main (#9588) rewrote `reconcile_series_indexes` around a memtable-free
`ReconcileSnapshot` so that a `VersionRef` is never held across `.await`
(holding it pins flushed memtables). narthex needs
`version.options.ttl`, `version.metadata`,
`series_index_version_control.formats` and `backend.reader(..).await` to
pick a series-index format.

Resolution: keep the snapshot invariant; clone `ttl` / `metadata` /
`formats` out of it before any await, and introduce
`build_series_index(store, region, metadata, bucket, entry, backend,
purger)` — a signature neither side had. `build_samples` /
`build_sample_windows` now take `&RegionMetadataRef` instead of
`&VersionRef`.

#### 4. `ScanInput` no longer carries a scan memory pool

main (#9101) moved scan memory accounting to the request-scoped
`QueryScanContext::memory_pool` and deleted
`ScanRegion::scan_memory_pool`. narthex's columnar reader registers four
`MemoryConsumer`s against the deleted field.

Resolution: `SeriesScanExec::execute` now threads
`context.memory_pool()` into its `QueryScanContext`, but the columnar
path inside `read/series_scan/prepared.rs` still uses an unbounded pool,
so **`scan_memory_limit` does not cover native columnar reads**. Note
the trap: main's default is now 25% of system memory where the old pool
was unlimited, so a naive "just reuse the old pool" fix would silently
tighten it ~4x.

#### 5. Sparse primary-key validation vs narthex's fixtures

main enforces (in `fix!: enforce sparse primary key schema`) that a
sparse primary key starts with `(__table_id, __tsid)` **and** that every
PK column id exists in the schema. Five `promql::source::tests` fixtures
satisfied neither.

Resolution: the test helper registers `__table_id` / `__tsid` as columns
and prefixes the key with them. narthex's production code never
constructs a sparse PK, so this is fixture adaptation — but it does
confirm that both branches extended the same persisted sparse key
encoding independently, which is worth a second look in review.

#### 6. `.github`: duplicated cherry-picks, plus two silent-coverage
traps

- The `metric-snapshot-s3-bench` family (6 files) is a **byte-identical
cherry-pick of #9573 that missed main's #10157**; `scope-rules.toml`,
`scope-cases.toml`, `github-script-tests/action.yml`, `heavy-ci.md` are
a squashed backport (#10053) that is a strict subset of main. All
resolved to main. `heavy-ci-policy.test.cjs` resolves to narthex (it
adds 13 lines of passing cases).
- `integration.yml` resolved **whole-file to main**. main renamed the
env to `FUZZ_VERSION`, narthex to `CARGO_FUZZ_VERSION`, which is
referenced by four cache-key strings — mixing hunks leaves it undefined
and silently changes every fuzz cache key.
- Two traps worth a reviewer's attention.
`tests-integration/tests/http.rs` registers tests through a **hard-coded
name list** inside `macro_rules! http_tests`; a hunk-level resolution
silently un-registers tests without a compile error. Verified in this
branch: all 11 narthex names and 6 main names are present, 89 total, no
duplicates. And `.github/ci/scope-rules.toml`'s fuzz gate matches
nothing for `.github/scripts/prepare-fuzz-corpus.sh`, because the
`src_tree = "src/"` fail-open net does not cover that path — a PR
touching only that script would skip fuzz CI silently. Resolved to
main's ruleset, which lists it.

### The other 25 conflicts

- **Cargo**: `Cargo.toml` had exactly one real conflict
(`greptime-proto` rev); everything else is a union (main's
`src/common/cache`, `src/relational-engine`, `datafusion-execution`,
`sha2`, pprof pin, `sqlx 0.9`, `simd-json 0.19` + narthex's `postcard`,
`stacker`, `twox-hash`). `Cargo.lock` was regenerated, not hand-merged.
`src/datatypes/Cargo.toml` keeps main's `enterprise = []` feature and
drops `regex` (main removed its last use).
- **mito2**: `metrics.rs` union (narthex's `CACHE_LOAD` duplicates
main's `cache_loading_metrics()` — TODO), `read/series_scan.rs` import
union, `series_index/tests.rs` follows the new signature.
- **query**: `dist_plan/merge_scan.rs` conflicted only in a doc comment
(function body auto-merged); the new comment states the two invariants —
`greptime.series.*` schemas are compared for exact identity, everything
else keeps main's lenient nullability rule, and the exact branch runs
first. `optimizer.rs` / `query_engine/state.rs` are adjacent insertions,
both kept.
- **datatypes / store-api / session**: re-export, trait-item and
`QueryContext` literal unions.
- **compatibility case**: `fulltext_bloom_tokens/case.toml` — both sides
fixed the same thing (`from_range`), so exactly one may survive; kept
main's `<v1.3.0` (concatenating both yields a duplicate TOML key).

### Default-configuration differences from `main` (audit)

An audit of "does a default deployment ever execute narthex code" found
that the native path itself is properly gated (engine option `false`
**and** per-request extension absent; `plan_pql` is the only entry
point; SQL has no native branch; `metric-engine`'s diff is test-only; no
table-level override exists). But **three behaviour changes still apply
on the default path**, and they need to be either gated or consciously
accepted:

1. **PromQL instant output cap removed** — `InstantManipulate` no longer
enforces the previous output-point limit. With `memory_pool_size` set to
a finite value this can turn a previously rejected instant query into a
`ResourcesExhausted` failure somewhere else, or into a much larger
allocation.
2. **topk/bottomk/limitk response `metric_name`** —
`servers/src/http/prometheus.rs:1527-1545` rewrites the metric name in
the HTTP response with no switch.
3. **ACL-scoped metric discovery** — `frontend/src/instance.rs:1374`
reads the per-request extension to derive discovery authorisation
targets; with an ACL checker installed this changes which targets are
submitted.

There are also per-query overheads that do not change results: an eager
`time_pruned_files` build, an unconditional `series_disjoint`
computation, and two whole-tree traversals
(`native_raw_plan_bounds_region_reads`,
`rewrite_hybrid_binary_operands`). Each is cheap but runs for every
default query.

These are tracked for a follow-up PR whose acceptance criterion is: **a
default-configured deployment produces byte-identical plans to `main`**.
That PR will also gate the three items above, short-circuit the
traversals, downgrade `pub mod native_compile`, and decide whether
`query.experimental_native_promql` should be a public option at all.

### Blockers before this leaves draft

1. **proto pin** — currently points at the fork branch for
GreptimeTeam/greptime-proto#355 (`b9159cd6`). Replace with the upstream
rev once #355 lands.
2. **Native payload byte cache** — restore it with a dedicated
page-cache budget (and un-ignore the two samples tests), or accept the
loss explicitly.
3. **`query.experimental_native_promql`** — decide public option vs
typed extension.

### Compatibility

- No wire/persisted format change here beyond the proto bump carried by
#355; both sides left `proto/` untouched.
- `samples.json` / `GTSERIES` / `GTSAMPLE` artifacts are unchanged by
the merge. Note the pre-existing (not introduced here) behaviour:
narthex's `load_version_control` prefers `samples.json` and discards the
legacy catalogs when it exists.

## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.

- [ ] I have written the necessary rustdoc comments.
- [ ] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).

## Merge timing

- [ ] Merge during the nightly window <!-- nightly-merge -->

<!--
Left unchecked on purpose: this is a rehearsal and still has three open
blockers.
-->
 2026-11-12 19:20:33 +08:00
discord9 fc28fae13d fix(mito-codec): register reserved columns in the sparse label fixtures 
main enforces (`fix!: enforce sparse primary key schema`) that a sparse primary
key starts with `(__table_id, __tsid)` and that every primary-key column id
exists in the schema. The label-decoding fixtures built a sparse key that
satisfied neither, so `RegionMetadataBuilder::build()` panicked and the
workspace `test` job failed.

Same adaptation as the promql source fixtures.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-12 17:52:16 +08:00
discord9 622cb196ec fix(mito2): drop a needless borrow in series-index reconciliation 
`files` is already a reference after the merge, so `&files` tripped
clippy::needless_borrow and failed CI's `-D warnings`.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-12 17:01:02 +08:00
discord9 ac900d1ad5 fix(mito2): keep main's cache split unless the columnar format is enabled 
The merge reserved a quarter of `sst_meta_cache_size` for the native-reader
metadata tier unconditionally, which silently halved main's decoded tier
(1/2 -> 1/4) on every deployment, including the ones that never enable the
columnar series-index format.

Gate that reservation on `experimental_series_index_format == Columnar`, so the
default split stays exactly what main ships and `columnar_meta_cache` is not
even constructed. Tests that exercise the columnar tier now opt in explicitly.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-12 16:38:00 +08:00
discord9 5194e06122 Merge remote-tracking branch 'upstream/main' into narthex-merge-main 2026-11-12 16:00:40 +08:00
discord9 ee9ad87fa1 Merge upstream/main into narthex (pre-merge rehearsal, not for push) 
Rehearsal merge of main (f8d8fe1071) into narthex (115d85b669) on a throwaway
local branch. 31 semantic conflicts resolved; workspace and all test targets
compile; promql 1215/1215 and mito2 series_index 185/185 pass.

Pre-merge pins and follow-ups are marked with TODO(narthex-merge):
- greptime-proto is temporarily pinned to the fork branch carrying
  GreptimeTeam/greptime-proto#355; replace with the upstream rev once it lands.
- native payload byte cache is disabled (main removed PageRangeCache); two
  samples tests are #[ignore]d until it gets a dedicated page-cache budget.
- columnar metadata now has its own CacheManager tier (a quarter of
  sst_meta_cache_size); the unified decoded tier was not kept.
- query.experimental_native_promql is registered as a query option, otherwise
  the native route is silently rejected by the option snapshot.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-12 15:45:04 +08:00
dennis zhuang 62dad907d0 fix: redact credential table options in user-visible output (#10346) 
information_schema.tables.create_options (and SHOW TABLE STATUS) printed
object store credentials of external tables verbatim, and SHOW CREATE TABLE
masked only access_key_id and secret_access_key. Mask all four credential
keys in both paths, and sort extra options in create_options by key.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-12 05:49:19 +00:00
discord9 115d85b669 fix(mito2): identify series by their exact sparse key, not by tsid (#10330) 
* fix(mito2): identify series by their exact sparse key, not by tsid

`(table_id, tsid)` is a hash, so distinct series can share it: 2000 label
values already produce 20 colliding pairs and a 36k-series SST keeps only 27856
distinct tsids. The columnar series writer silently merged such pairs (first
label set kept, row counts added) and the samples writer either reported
time_regressed or, with increasing timestamps, attached one series' samples to
another's labels. Every one of those decisions used the hash as identity.

The SST already stores the exact sparse primary key, so nothing on the storage
side changes: `tsid`, the hash function and the key layout are untouched. What
changes is that the artifacts, their readers and the prepared read path now
decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is
only the coarse, order-preserving prefix used for ordering, pruning and
bucketing.

- `ExactSeriesKey` is the new identity type: equality, ordering and hashing are
  defined on the key bytes alone, `id()` is documented as a derived prefix that
  is never compared. `from_sparse_key` validates the entire key, not just the
  prefix, and bounds it at 8 KiB.
- Series format v2 keeps one row per exact key; the descriptor layout and the
  existing per-row tags already express this once the writer aggregates by key.
  Coarse page/block/part bounds may now touch, so readers prune conservatively.
- Samples format v2 records the owning key per block, appends a `__primary_key`
  column, and lets a key change restart timestamps and ordinals while the same
  key keeps the monotonic rules. Exact selection travels in
  `SamplesReadRequest.keys` next to the coarse `ids`.
- The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from
  the artifact's own tags, associates samples through `__primary_key`, and uses
  the exact key for the merge key and the direct-label path.
- Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts
  without it are structurally valid and perfectly ordered yet may contain an
  already merged row, so they are rejected and rebuilt from SSTs; maintenance
  rotates them out instead of reusing them, and the local cache version drops
  the rebuildable store. There is no dual reader and no migration.

Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times,
identical); `cargo nextest run -p mito-codec` 57 passed; clippy
`-D warnings` and rustfmt clean. Tests pin the real captured pair
(`host-020078` / `host-029838`, both retained with their own labels), the
silent-loss variant with globally increasing timestamps, sixteen distinct keys
under one forced tsid in both PK encodings, candidate isolation end to end, and
rejection of a frozen v1 fixture.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): close the exact-identity review findings

- The two-phase series scan assumed candidate discovery had already enforced
  tag predicates through an "exact assigned-series set". An assignment is a
  `(table_id, tsid)` hash, so a colliding partner shares it without matching
  the predicate, and skipping tag filters returned its rows. Apply the
  predicates the precise filter already holds instead of assuming them.
- `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but
  not when a writer built one, so the writer could publish a key the reader
  refused; a single 8 KiB label already encodes past the cap. Length is
  bounded by the containers that store a key (one key per block, descriptor,
  page and column limits), and long labels are real data, so drop the cap
  instead of enforcing it on both sides.
- The `__primary_key` column copies one key per output row, but the read
  reservation budgeted a fixed 32 bytes per column and metadata pages did not
  count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB
  against a ~1 MiB estimate. Budget the real bytes and count the heap.
- `SamplesReadRequest` carried both `keys` and `ids`, which the only caller
  derived from the same candidates; selection is key-only now and the reader
  derives its coarse pruning set, so the two selectors cannot disagree.
- Restore the unknown-field-codec projection coverage that the frozen v1
  fixture used to carry, against a generated v2 artifact, and assert v1
  rejection through the compatibility property rather than a field name.
- Drop the redundant columnar increasing-timestamp test (that writer never
  reads timestamps) and correct the samples format doc: the five internal
  columns and `__primary_key` are non-null, value fields stay nullable.

Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean. Each new test was run with its fix reverted
first: the two-phase test returned the colliding partner's rows, the long-key
tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read
8192 rows of 1 KiB keys while allocating ~8 MiB.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget the exact key column exactly and drop stale skip-tag claims

The read reservation charged the exact-key column its logical bytes, but the
column was built with `BinaryArray::from_iter_values`, whose value buffer grows
geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for
2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was
oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and
reserves the key bytes once (1.003x on the same fixture), which is what the
reservation already assumes. The new test asserts the pool covers what the
yielded batch retains, that the column keeps no growth slack, and that the
reservation is released on drop; 2135 rows sits just past a doubling step so a
growing builder fails it.

Also make the documentation match the mechanism and the encoder:

- `read/pruner.rs` still told readers the data phase skips tag filtering because
  candidate discovery had applied the predicates exactly once. Assignments are
  `(table_id, tsid)` hashes, so a colliding partner shares the identifier
  without matching the predicate, and the data phase does apply tag predicates.
- The initial-window fallback test anchored on `reads[5]`, which is the second
  payload window; it now identifies the initial window by its payload-header
  offset and asserts the window spans 128 blocks, so a traversal change cannot
  quietly point it at another read.
- The samples format descriptor sizes are recomputed from the encoder
  (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix
  `45,248`, each `+ key_length`; the page-fit sentence now states the enforced
  constraint (descriptor and leaf prefix inside one page) instead of claiming
  the widest descriptor always fits.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget each projected sample array's fixed overhead

The read reservation charged per-row bytes plus the payload sizes, but not
the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the
array struct plus its buffer capacities, so a one-row, key-only projection
reserved `key_length + 72` for the 1344 bytes it actually retained. Add
`ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per
projected column, as their own term, and cover the one-row key-only case in
`exact_key_output_reservation_covers_the_retained_array`: the pool must cover
the batch's reported memory size while it is alive and be empty after drop.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* chore(mito2): pin the series-index cache token to 1

The branch inherits `2` from an earlier layout change. This branch is
experimental and has no deployments, so it pins the token at `1`: whatever a
different build wrote into the rebuildable local store is discarded on startup
and rebuilt from the SSTs.

The token is not the compatibility boundary on its own. Artifacts are rejected
on open when they lack the mandatory `identity_encoding` discriminator, so a
hash-only artifact is never served even when the reset has not run yet (a read
that precedes it, or an artifact outside this cache).

Signed-off-by: discord9 <discord9@163.com>

* feat(mito2): keep exact series keys out of the samples descriptor page

A block descriptor used to embed the whole exact key, and a descriptor shares a
65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on
key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with
1,024 fields. Valid SST primary keys can be longer, so their samples builds failed
with "sample descriptor exceeds leaf limit" even though the key itself was
accepted.

The key now lives in a per-part key area and the descriptor carries a reference:

  part    = [header 16][column payloads][key area][leaf pages / root][trailer 120]
  trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 |
            key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8
  block   = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 |
            max i64 | ordinal u64 | rows u32 | columns u32  (56 bytes, fixed)

Descriptor size no longer depends on key length (232 bytes for one present field,
45,244 for 1,024), so the ceilings are gone. Keys are written once per series,
back to back in block order and without padding, so an equal reference means
"same exact series" and a key may exceed any page limit; a reference is validated
against `key_end` with a checked add, an overlap between consecutive new series is
rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor
is refused before any row is emitted.

Reads stay lazy: only blocks that survive coarse selection fetch a key, at most
once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the
part's aligned cache when one is configured. Measured on the 260-series fixture:
260 per-series key reads before, 1 merged read after, with the fetched byte count
asserted to be exactly the key bytes.

The read contract also stops materializing one key per row: `SamplesBatch
{ batch, key }` carries the block's key once, the exact-key column is refused in
projections, and `prepared` attributes a whole batch with a single binary search.
For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to
1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one
owned copy of the key instead of two, and `retained_rows` is gone in favour of the
`prepare_retained_rows` metric.

No version bump: the layout changes in place. Artifacts written by an earlier
shape of this branch (112-byte trailer, or keys inside descriptors) are
unreadable and must be rebuilt, which the cache token and the required
`identity_encoding` discriminator already bring about.

One user-visible cost: keys retained by a scan are charged to its memory pool, so
a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture
that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment.

Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared
series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs;
clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by
reverting the relevant hunk:
- restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with
  "sample descriptor exceeds leaf limit";
- the 260-series coalescing assertion fails against per-series reads.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): assert the duplicate-drop on the preparation's own rows

`overlapping_buckets_retain_only_new_label_rows` compared the deltas of two
process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every
preparation in the process increments both, so under parallel test execution the
two deltas do not pair up and the test failed intermittently - it always passed in
isolation.

Assert the property the pairing was standing in for, on this preparation's own
rows: read every series of the prepared source and require that no series keeps
two rows at the same timestamp. A surviving duplicate from the overlapping bucket
would repeat a timestamp; the counters can no longer make the result depend on
what else the test binary happens to be running.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): drop the now-unused preparation counter helper

The duplicate-drop assertion no longer reads the process-global counters, so
`prepared_count` has no callers and `-D warnings` rejects the dead function in
`cargo check` and `cargo clippy` CI. Remove it.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): scope key references per part, bound key reads, checksum keys

Review of the key-area change found three defects that ordinary data can hit and
three smaller ones. All six are fixed with a test that fails beforehand.

Key references are file-local offsets, and the ordering state kept them across
parts: a series continuing into another part looked like an overlapping
reference (or a new series with a non-zero ordinal), and two different series in
structurally identical parts could be mistaken for one because their offsets and
lengths coincided. Reference ordering is now scoped to a part; continuation
across a boundary is decided on the exact key bytes plus time/ordinal, and the
loaded keys must be nondecreasing, a repeated key must continue its series.

A key prefetch fetched the whole window and only then grew the reservation, and
the per-key fallback charged after its read, so a small pool could read and
assemble a page of long keys before reporting exhaustion. The peak (fetched
entry + owned key + alignment/assembly) is now reserved before the read; on
refusal the reader drops alignment first, then halves the window, and a returned
key is cloned only after the block reserved for it. A test asserts that an
unpayable prefetch issues no key read at all.

The key area had no integrity check, so flipping one byte of a label - leaving
framing and `(table_id, tsid)` intact - could silently change which series a
block belongs to. Every stored key now carries its XXH64, excluded from the
reference length and verified when fetched.

The manifest discriminator now names the layout
(`exact-sparse-key-area` for samples; the series index keeps
`exact-sparse-key`), so an artifact with inline keys is refused at open instead
of being marked reusable by maintenance and failing later at a part read. No
version number changes anywhere. A key copy is also counted correctly when a
series spans a metadata rollover: the "first copy of this part" decision is
taken after the rollover, not before.

Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): charge the key peak on every path and chain the order check

Four follow-ups from the confirmation review of the key-area change.

The per-key fallback reserved only the retained and stored bytes and then
fetched through the aligned payload cache, whose aligned/assembly peak it never
charged, so a leftover key could read a whole aligned block the pool had not
paid for. It now reserves the same peak as the page prefetch: aligned budget
first, and on refusal an explicitly cache-less reader with the unaligned peak,
before any read.

The key carried across a part boundary was a deep copy taken while the per-part
reservation was being released, so it lived on uncharged until the next part.
It is now held by its own pool consumer, charged before the copy and released
when it is replaced or when the boundary falls behind the scan.

The content-order check was chained over the last *traversed* block, which an
unloaded block overwrote, so a block pruned by selection could hide reversed or
restarted keys - and could drop the boundary key entirely when a part ended on a
pruned block. It is now chained over the last *loaded* key, with adjacency taken
from "the last traversed block is the last loaded one", so a gap still requires
increasing order and a strictly greater ordinal but not exactly +1.

Finally, the key-area read counter in the tests only saw reads fully inside the
key area, missing aligned reads that start before it; it now counts reads that
intersect, and the cached-read test is documented as a cache-behaviour guard
rather than a failing-before proof for the budget work.

Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.
Failing-before evidence: a 16384-byte key-area read issued with only 15080
reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden
behind a skipped block; and a pruned block hiding the boundary key.

Signed-off-by: discord9 <discord9@163.com>

* docs(mito2): qualify the ordinal rule to one exact key

The samples format said a loaded key's ordinal must still grow, which reads as if
ordinals had to increase across different series too. They do not: only an equal
exact key continues, an adjacent block of that key requires +1, and a different
key starts at zero.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-12 12:10:51 +08:00
fengyue-xve f8d8fe1071 fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk (#9452) 
* fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk

The Elasticsearch _bulk handlers read only pipeline_name and msg_field and
hardcoded the pipeline version to None, so the x-greptime-pipeline-version
header and the version URL parameter were silently ignored and the latest
version was always used.

Resolve the version from the version query parameter, falling back to the
x-greptime-pipeline-version header, and pass it to
PipelineDefinition::from_name; an unparsable version now fails the request
instead of being ignored. Without either source the latest version is used,
as before.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(elasticsearch): reject invalid pipeline version headers and use HTTP item status

Signed-off-by: fengyue-xve <1759519137@qq.com>

---------

Signed-off-by: fengyue-xve <1759519137@qq.com>
 2026-11-12 01:11:21 +00:00
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518) 
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
 2026-11-12 01:09:42 +00:00
Weny Xu 15fd97e369 fix(meta-srv): stop failing over regions that are still opening (#10290) 
* chore: update proto for opening region heartbeats

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reset failure detectors during background region initialization

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve detector history for opening regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update opening region proto revision

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update opening region proto to merged main

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-11 12:57:10 +00:00
discord9 90d0c00a82 fix(promql): resolve the OR right operand's time-index column from its schema (#10335) 
`or` inferred the qualifier of an operand plan from its first field
(`schema.qualified_field(0)`). A binary operation with an explicit matching
modifier derives its result labels from the left operand when the match
reduces the label set, so its projected plan carries two qualifiers: the
labels are `lhs.<label>` while the time index keeps the context table's
qualifier (`rhs.<greptime_timestamp>`). Prefixing the time index with the
first field's qualifier then referenced a column that does not exist, and
planning failed with

  Internal error during building DataFusion plan: No field named
  lhs.greptime_timestamp. Did you mean 'rhs.greptime_timestamp'?

for e.g. `sum by (l) (f(a) / ignoring(x) group_left() f(b))` used as the
right operand of `or on (l)`. `f(a) / ignoring(x) f(b)` triggers it as
well, so this is not specific to a group modifier. Without a matching
modifier every projected column takes the right operand's qualifier, which
is why plain `a / b` never hit this.

Resolve the time index against the actual operand schema: keep the
preferred qualifier when it really holds the column (uniform schemas, so no
plan changes), otherwise take the qualifier of the column found by name and
let DataFusion report ambiguity instead of guessing.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-11 10:06:12 +00:00
discord9 91258beae3 refactor: remove dead native analysis and duplicate columnar filtering (#10314) 
* refactor(promql): drop unused native build sharing flag

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep retained-label windows unified

Signed-off-by: discord9 <discord9@163.com>

* perf(mito2): skip exact columnar residual filters

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin nested window placement

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-11 15:29:28 +08:00
jeremyhi d1e5a2df8f fix(mito2): check the lease before manifest commits and reload regions after conflicts (#10311) 
* fix(mito2): check the lease before manifest commits and reload regions after conflicts

In conditional manifest mode, refuse a commit whose Leader deadline has
passed before its PUT, and fence a region whose commit hits a manifest
version conflict, then reload it in the background and swap the new
instance in.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): release the flush status when the WAL release after a flush fails

handle_flush_finished failed the flush's waiters and returned when the WAL
release after a committed flush failed, but left the flush registered in the
flush scheduler, so every later flush of the region, and the reload of a
fenced region, waited behind it forever. Handle the failure as a failed
flush, which removes the registration and answers the pending requests.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover staging saves after the deadline and a fence during catchup

Add a listener hook before a catchup reopens the region, and test that a
fence landing there makes the catchup answer RegionBusy without replacing
the instance. Also test that entering staging after the Leader deadline
saves no staging manifest.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that set_region_role does not check the reload mark

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-11-11 06:44:06 +00:00
Lei, HUANG c8e0d29df2 feat(mito2): publish table compaction results conditionally (#10310) 
* feat(mito2): publish table compaction results conditionally

Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): write table compaction outputs to level 0

Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): release verified footers before manifest publication

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): compare typed table compaction options

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover table compaction output validation failures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify series count verification limits

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject index declarations without output index objects

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document table compaction outputs as L0

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): map the table compaction publication module

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-11-11 05:42:51 +00:00
dennis zhuang ddc16bafe8 fix(datatypes): render jsonb stored as JSON text without panics or wrong values (#10197) 
* fix(datatypes): decode jsonb before rendering it as text

jsonb_to_string serialized the bytes through RawJsonb's Serialize impl,
which panics with an out-of-range slice index on JSON text such as
"hello world" or "        null" (stored by COPY FROM and MySQL text
parameters) and on truncated jsonb. Its text fallback was also stricter than
the extended parser reads used before.

Decode with jsonb::from_slice, as reads did before #10112, then write the
value out with serde_json escaping for keys and strings.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(datatypes): keep streaming jsonb serialization in jsonb_to_string

Decoding into a jsonb::Value tree allocates per array, object and key on
every JSON cell returned over PostgreSQL and MySQL. Go back to serializing
the raw bytes with RawJsonb's Serialize impl, and bump jsonb to a rev whose
Serialize returns an error instead of panicking on out-of-range payloads
(GreptimeTeam/jsonb#2). Bytes that fail to serialize are parsed as JSON
text with the extended parser, as jsonb::from_slice does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(datatypes): pin text values that used to read back as false

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: bump jsonb to reject scalar headers with non-zero length bits

JSON text over ~2 MB could still pass the payload bounds check and read back as a wrong value (GreptimeTeam/jsonb#2).

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(datatypes): report the text parse error when JSON bytes are neither jsonb nor text

Also pin jsonb_to_serde_json on JSON text scalars that used to decode as false.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: pin jsonb to the merged commit on main

GreptimeTeam/jsonb#2 reached main through GreptimeTeam/jsonb#3.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-11 03:46:56 +00:00
discord9 e49bb09480 feat(promql): support metric-name rewrites in native labels (#10176) 
Signed-off-by: discord9 <discord9@163.com>
 2026-11-11 11:40:40 +08:00
discord9 97c8136c3c chore(deps): bump datafusion to ae04160b (#10320) 
* chore(deps): bump datafusion to ae04160b

Move all 16 `[patch.crates-io]` DataFusion pins from `cc5534492` to
`ae04160b`, the merge commit of GreptimeTeam/datafusion#42.

That fork commit backports six upstream apache/datafusion fixes:

- #26452 timestamp-to-time casts no longer drive statistics pruning
- #24675 memory-limited NLJ fallback refuses left-emission joins with a
  multi-partition probe side instead of emitting wrong unmatched rows
- #24891 RepartitionExec spill-pool deadlock with two open files
- #25076 null-aware anti join no longer emits rows after a sibling probe
  partition saw NULL
- #25339 correlated NOT IN with a non-equality correlation
- #26578 null-aware anti join drops build rows with a dictionary NULL key

The first three are faithful backports. The last three are re-implementations:
upstream restructured `hash_join/stream.rs` before writing them, so the patches
do not apply to this fork's older copy of that file. `#25339` covers the
single-equi-key plus residual-filter shape; multi-key correlated `NOT IN` still
fails at planning as before.

`Cargo.lock` is regenerated (35 `datafusion*` packages).

No runtime acceptance has been run against this pin; see the PR description for
the three checks worth doing before shipping.

Signed-off-by: discord9 <discord9@163.com>

* fix(common-query): pass the pruning bitmap to HashTableLookupExpr::new

#39 added a fifth parameter to `HashTableLookupExpr::new`; the dynamic-filter
payload round-trip test still constructed it with four arguments, which breaks
`--all-targets` builds even though the library itself compiles.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): bump datafusion to 70f84c5b

Move the pins from `ae04160b` to `70f84c5b`, the head of
`greptimedb-55.1-thin-fork` after GreptimeTeam/datafusion#37 was merged on top of
#39/#41/#42.

`ae04160b` did not contain #37, so the reverse-cast constraint propagation
guard was lost and `tests/cases/distributed/optimizer/partition_pruning_cast`
failed: the filter `CAST(k AS BIGINT) = 1` made the planner treat the float
column `k` as a singleton and drop the requested `ORDER BY k DESC`.

The new pin carries both the reverse-cast guard and the six release backports,
so the regression is covered again.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 18:17:24 +00:00
Weny Xu d061e08717 feat: flush regions under object store WAL retention pressure (#10309) 
* feat(store-api): expose advisory WAL flush candidates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(log-store): select regions blocking WAL object reclamation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): schedule region flushes under WAL retention pressure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover WAL pressure reclamation and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): prioritize oldest WAL reclamation blockers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): deterministically exhaust range cache admission budget

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 16:06:17 +00:00
Weny Xu 54b6bd5b44 feat(mito2): support extension index providers and input reuse (#10297) 
* feat(mito2): register extension index providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito2): support index input reuse and extension budgets

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 14:02:48 +00:00
discord9 8004f3e88d perf(promql): reduce repeated work in the native query path (#10212) 
* perf(promql): elide redundant tracking after name-dropping windows

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): fold aggregate samples without staging buffers

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): reuse clean labels when dropping metric names

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover literal arithmetic over histogram windows

Signed-off-by: discord9 <discord9@163.com>

* test(promql): resolve all-target clippy lints

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 20:24:51 +08:00
jeremyhi 77cb39b5d1 feat(mito2): record pending WAL tails in region manifests (#10308) 
* feat(mito2): record pending WAL tails in the region manifest

Adds `pending_tails` and `handoff_floors` to the region manifest and its
checkpoint, and an optional `pending_tails` / `handoff_floor` pair to
`RegionEdit`. An edit without them leaves the records unchanged. A floor is
dropped once the flushed entry id reaches it; a tail stays until a commit
removes it. Neither is written when empty, so manifests without them encode
as before.

Opening a region replays its WAL after this node's tails and floor, and WAL
releases at open and after a flush stay below this node's tails. Nothing
writes these records yet, so every region behaves as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): hold back reads and region changes while WAL tails are pending

A region with pending WAL tails is missing data that only those tails hold.
Until they are recovered, scans and memtable and SST stats fail with the
retryable `RegionPendingTails` error. Truncate, entering staging, a
repartition flush, a close-time flush on request, and ALTERs other than
adding nullable columns without a default or changing options are rejected
before they change any region state. Compaction pauses. A full-listing GC
pass skips the region while it has pending tails, or while a newer manifest
may exist or the manifest listing fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): name the WAL releases that keep pending tails

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that reopening restores the pending-tail read gate

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin what the pending-tail fixtures and rejections promise

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the replay start in the open log and share the pending-tail check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the per-node order of pending tails and who keeps later writes out of them

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending

When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the pending compaction task state when WAL tails skip it

Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-11-10 12:21:17 +00:00
discord9 e0213b5296 test(promql): qualify native corpus partition parallelism (#10181) 
Use disjoint flushed ranges in the corpus and real-query fixtures, then assert every real SeriesScanExec reads on all four partitions. Assert every region-side fused prefix and partial aggregate publishes on all four partitions; require width-four frontend-raw nodes to do the same while leaving final combine nodes reported only.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 19:08:53 +08:00
Weny Xu e2e980fe39 feat(mito2): add column sketch options and index metadata (#10295) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 09:55:20 +00:00
jeremyhi 9eb027f738 test: disable the event recorder in tests that count WAL writes (#10316) 
The object store WAL restart test and the skip-WAL tests built on
MockInstanceImpl assume only the tables they create write to the WAL.
The event recorder also writes greptime_private.events into the same
WAL, so when it flushes during the test (the first tick of its interval
lands after the first DDL event on Windows), the extra region keeps WAL
objects from being collected or shows up in the WAL snapshot.

Record no events in these tests.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-11-10 09:29:35 +00:00
discord9 0c4b7dc982 feat(promql): let count and group fold native histograms 
`count` and `group` fold native histogram samples in the native aggregate: neither reads a value, so a histogram is one present sample. `min`, `max`, `stddev` and `stdvar` read a float and now skip histogram samples the way Prometheus does, reporting what they skipped; `sum` and `avg` keep the existing plan until #10199. A group holding only histograms answers nothing under a skipping aggregate, and `quantile` keeps the existing plan because this runtime has no kernel for it.
 2026-11-10 17:13:18 +08:00
discord9 ffa0815285 refactor(series): hold the frame layout marker at one value 
The series frame is the native PromQL path's internal transport, and that path lives on `narthex`, an experimental branch: it may break its own formats between revisions, so a frame revision number has no compatibility promise to protect. The marker becomes movable once the path reaches a release branch, where a producer and a consumer can legitimately be different revisions.

The reader still refuses a frame that declares any other value, which turns a mismatched producer into an error instead of values decoded under the wrong columns. The two wire fixtures are renamed to `v1_metadata.json`, and the drift cases now use the value that is not current.
 2026-11-10 16:40:03 +08:00
discord9 405451ac7f test(promql): cover discovery under an aggregate on the standalone route (#10255) 
Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 16:39:25 +08:00
discord9 7c183e0c7f feat(query): run an embedded TQL statement on the native PromQL route (#10288) 
* refactor(query): enforce column requirements per node, not per plan root

Signed-off-by: discord9 <discord9@163.com>

* feat(query): run an embedded TQL statement on the native route

Signed-off-by: discord9 <discord9@163.com>

* fix(query): publish one CTE schema on both TQL routes

Signed-off-by: discord9 <discord9@163.com>

* fix(query): publish the CTE schema durably, including its metadata

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 16:38:36 +08:00
shuiyisong ee1fc7782f feat(prometheus): query persisted metric exemplars (#9924) 
* feat(prometheus): add exemplar queries and lifecycle coverage

Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows.

Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): preserve query labels with source identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): floor bounds and add audit hooks

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* • fix(exemplar): include aggregation parameter selectors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): avoid v1 exemplar quota stalls

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align exemplar query timestamps with Prometheus

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove unnecessory doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-11-10 08:11:49 +00:00
discord9 196a86de98 feat(promql): compile absent_over_time on the native route (#10198) 
`absent_over_time(<selector>[<range>])` had no native kernel, so the
statement declined to the existing planner; there a nameless selector has
no metric name to look up, and the query failed instead of answering the
label set Prometheus reports.

Bind it to the whole-vector operation `absent` already uses, over a window
read of the selector's range. The fold keeps the metric name: a presence
fold that drops it would fold two metrics which differ only by name into
one series and refuse the second value, while `absent` only asks whether
the window held anything at all. Labels come from the same derivation
`absent` uses, now also accepting a matrix selector.

A statement holding a nameless selector that discovery expands into more
than one metric table keeps declining: keeping the name would declare
`__name__` without a bound metric name, and dropping it would merge the
metrics. The typed empty input of a selector that matched no metric table
is excluded from that decline, so a zero-match query plans natively and
answers from it.

The legacy planner is untouched.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 15:53:22 +08:00
shuiyisong 09c0291575 fix: restrict JSON retries to invalid UTF-8 errors (#10302) 
* fix: restrict JSON retries to invalid UTF-8 errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: isolate UTF-8 retry counter assertions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-11-10 07:42:27 +00:00
Weny Xu 65dc999180 feat: support cancelling compaction tasks by task ID (#10229) 
* feat: add ADMIN compaction task cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject metadata region compaction cancellation before routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged cancellation commit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 07:38:08 +00:00
Weny Xu 77a7d998bb perf(puffin): reuse reader metadata and scoped cache keys (#10294) 
perf(puffin): reuse reader metadata and blob lookup

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 07:20:24 +00:00
Weny Xu d2be2980c2 refactor(mito2): unify builtin index planning and execution (#10293) 
* refactor(mito2): separate index target resolution from creation

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit da65987533)

* refactor(mito2): unify builtin index planning and execution

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-10 07:01:07 +00:00
Lei, HUANG f5c1020f5f feat(mito2): add shared SST storage and reclamation foundations (#10256) 
* feat(mito2): add shared SST storage and reclamation foundations

Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): isolate table compaction manifest semantics

Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): remove manifest-backed compaction task state

Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): guard staging manifests and reclaim shared range indexes

Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): simplify GC deletion and lingering rules

Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): keep regions open after manifest validation rejection

Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject ambiguous shared SST reclassification

Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): define truncate semantics for shared SST references

Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): define shared copy results and reject metadata imports

Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): unify shared SST level references and bounds docs

Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): move the shared SST level to a max sentinel

Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): keep dropped-region tombstones while files are deferred

Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject adopting an existing L1 SST as shared

Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-11-10 06:39:15 +00:00
discord9 451f802f5b fix(promql): replace the residual hybrid binaries and drop the 64-level depth cap (#10254) 
Remove the last hybrid composition: when a native statement is refused as a
whole but each side is independently admissible, the sides are now compiled
through the same native frame preparation, compiler and placement path the
whole-native route uses and meet at a shallow native Binary, instead of the
native operands being composed by the legacy planner's Join.

Admission is decided by the composition itself - equal grids, the binder's
matching/cardinality/fill support, the output column's kind, and the runtime
execution state - rather than by a predicate that copied what the removed
hybrid route used to accept.

The artificial 64-level native depth cap is gone on both the construction and
the decoding side, with no replacement cap, no depth-based splitting and no
product stack configuration; the recursive walks that actually need it grow
their stack on demand (validate, runtime build, runtime pull).

Histogram binary results are preserved and their diagnostics aligned with
Prometheus: an unsupported pairing synthesized by a fill now reports the same
Info, and an incompatible bucket layout reports only the schema Warning
instead of also reporting a reset-hint contradiction. The legacy histogram
UDFs follow the same order so the two routes stay in step.

Signed-off-by: discord9 <discord9@163.com>
 2026-11-10 14:33:53 +08:00
Ning Sun 7cd2a3050e fix: pgwire update 0.42.2 (#10300) 2026-11-10 02:57:50 +00:00
dennis zhuang 16123d806b perf(query): push column projections through MergeScan (#10221) 
* perf(query): push column projections through MergeScan

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(query): retain empty projections and refresh sqlness plans

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(query): update TSID join plans after projection pushdown

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-11-09 22:14:09 +00:00
jeremyhi 8da533c35f fix(mito2): keep rows written during staging in the WAL (#10253) 
* fix(mito2): keep rows written during staging in the WAL

A flush during staging committed its SSTs to the staging manifest, then
dropped the memtable and released the WAL. If staging did not end
successfully those rows were gone. A flush that committed after staging
had ended lost its rows even though staging succeeded.

Staging regions no longer flush. They are skipped by memory-pressure,
periodic and region-limit flushes, and their writes are rejected instead
of queued at twice the region limit or when the node's staging memtables
reach half of the engine's flush limit. A manual or repartition flush on
a staging region with data gets a retryable error; the migration
pre-flush and the WAL prune flush reply success without flushing. Closing
a staging region leaves staging first, so skip-WAL rows are flushed to
the normal manifest.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep accounting and closing staging regions during staged edits

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): track staged edits without relying on the partition info

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): synchronize staging close tests with worker admission

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): observe the manifest lock wait when closing staging regions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): detect staged edits through the staging partition info

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): record that staging regions must not flush

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): do not restore staging after a close abandons the edit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject region edits during a close-time flush

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style(mito2): format the staging tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject staging edits once staging is abandoned

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): bind apply-staging edits to their partition expression

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that staging regions pin their topic's prune point

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-11-09 16:38:59 +00:00
sunanddennis zhuang 4c4c8a58fd feat: update dashboard to v0.14.5 (#10270) 
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2026-11-09 16:36:54 +00:00
Arpit JainandClaude 402e4a97fc fix(time): avoid i32 overflow when converting an out-of-range date (#10299) 
Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-09 16:34:48 +00:00
Weny Xu 73698114de feat(log-store): add a read-only object store WAL reader (#10269) 
* refactor(log-store): extract shared WAL reading helpers

Move object indexing, chain traversal, history checks, segment prefetching and object I/O out of the writable store. Preserve local recovery and read policies.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(log-store): separate read-only WAL object access

Introduce WalObjectRead as the read capability shared by indexing and prefetching. Keep writes and startup listing on WalObjectIo, and adapt writable-store test doubles without changing their behavior.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(log-store): add a read-only object store WAL reader

Validate source identity with NodeWalPrefix and share readers by source. Read known-last-entry or listed ranges using fresh listings and bounded footer concurrency, without holding the cache lock across I/O.

Evict indexes outside the current listing, keep release extraction in startup recovery, and cover bounds, history, cache lifetime, prefix validation and concurrent reads.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): coalesce remote WAL footer loads with shared cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): report missing source WAL history separately

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): cover remote WAL chain and listing boundaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): wait for asynchronous WAL index release

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): align WAL recovery assertions with shared history messages

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-11-09 16:14:20 +00:00
discord9 7b04d85399 fix(promql): plan a zero-match selector beside another input (#10243) 
* fix(promql): plan a zero-match selector beside another input

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover a zero-match selector beside a real one in distributed

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): bind a zero-match selector to its set-operand layout

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): propagate empty set layouts through nested operands

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover empty operands beside arithmetic and non-bare sets

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): record set layouts through subqueries

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-11-09 23:40:03 +08:00