Commit Graph
6642 Commits
Author SHA1 Message Date
jeremyhi dd941a8e33 fix(mito2): wait for a stopped region's checkpoint before reopening it (#10376)
* fix(mito2): wait for a stopped region's checkpoint before reopening it

A region's manifest checkpoint runs as a background task that writes the
checkpoint, then `_last_checkpoint`, then deletes older manifest files:
the deltas up to the previous checkpoint, or every older delta at the
region's first checkpoint. Stopping the manifest manager does not wait
for it, so a new instance of the region could read the manifest while the
old task still wrote it. On a store that writes files in place the open
could read an empty `_last_checkpoint`. On any store, an open that found
no checkpoint could have deltas it had just listed deleted by the first
checkpoint's cleanup.

Wait for the stopped instance's checkpoint at engine stop, after the
worker loop ends, and before a fenced region is reloaded. The wait does
not hold the manifest lock, which the worker may take meanwhile: the
checkpoint task owns the sender of a watch channel, which closes however
the task ends.

Fixes the flaky test_change_region_compaction_window (#7135).

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover the checkpoint wait through cleanup, panic and abort

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): align checkpoint wait names and log the wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover concurrent and late waits on the checkpoint

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): keep two checkpoint waiters through the end of the task

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a wait after the checkpoint ended does not start waiting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): address review on the stopped checkpoint wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the stop contract and assert it before the checkpoint wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the checkpoint completion future and keep its task state together

Rename `pending_checkpoint_done` to `pending_checkpoint_completion` on
`Checkpointer` and `RegionManifestManager`, and state how it differs from
`wait_for_pending_checkpoint`. Keep the task handle and its completion
receiver in one `CheckpointTask`, so reaping the handle also drops the
receiver. Use plain backticks in the `stop` doc, which is public and
cannot link to a crate-private method.

Narrow the comment in `RegionWorkerLoop::clean`: a datanode or standalone
shutdown closes every region before it stops the engines, so there the
wait only covers regions whose close failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 10:52:27 +00:00
Weny Xu 89e14ab936 fix(mito2): cancel compaction merges at batch boundaries (#10411)
* fix(mito2): cancel compaction merges at batch boundaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve upload errors and expose merge cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): normalize encoder cancellation at SST writer boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(mito2): use imports for SST write types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): share cooperative upload cancellation with tokens

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): classify nested upload cancellation errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 08:32:11 +00:00
21ada9de09 chore: remove stray merge-main.log (#10445)
merge-main.log at the repository root holds the output of a `git merge`
that stopped on conflicts in src/flow, followed by an `exit=1` line. It
looks like it was committed by accident in 6fa375021 (#9171), and
nothing in the repository references it.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-11-16 04:32:29 +00:00
Ning SunandLei, HUANG b30e002cef feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>])
ingests one Prometheus Remote Write request body over the postgres wire
protocol — a GreptimeDB-specific, binary COPY format (wire format code
1). The payload is byte-for-byte the body a remote-write client POSTs
to /v1/prometheus/write, compression included: buffered until CopyDone
and completed through the exact ingestion core that endpoint runs.

To share that core, the HTTP handlers' ingestion bodies are extracted
into axum-free functions (ingest_remote_write_v1/v2 plus the context
extension stamping), and the HTTP routes become thin wrappers — their
behavior is unchanged, and the copy layer runs the identical decode,
validation, conversion, batched/prepared write paths, per-batch
write-progress accounting and context extensions, under the
connection's regular postgres context.

The WITH options carry what the HTTP endpoint takes as its
content-type/-encoding and query parameters; session-provided values
(database, user) are not repeated:
- version 1|2 (default 1), matching the content-type's proto
- zstd, matching Content-Encoding: zstd (snappy by default)
- physical_table, matching the ?physical_table= parameter
- pipeline_name/pipeline_version, matching the pipeline headers
  (v1 only; for v2 they are accepted and ignored, exactly like the
  HTTP route ignores them)

Table-less like the other ingestion formats; one COPY statement is one
bounded request (shared 64 MiB limit); the row count reports written
rows (v1) or written samples plus histograms (v2). The flush enforces
the frontend's shared request-memory quota (max_in_flight_write_bytes)
— the same limiter the HTTP ingestion protocols use, so COPY cannot
bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write
carries the handler, metric-engine flag, validation mode and shared
memory limiter.

e2e: v1 body chunked across CopyData boundaries; v2 samples with
readback; physical_table option routing; garbage-body decode failure
with connection reuse; a tiny-quota regression where a compressed body
whose decompressed size exceeds the budget fails like the HTTP route.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(servers): propagate COPY body reservations and test write accounting

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): share Prometheus logical batching with COPY ingestion

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): narrow Prometheus COPY scope and share write outcomes

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
v1.4.0-nightly-20261116
2026-11-16 02:45:34 +00:00
Jingbo ChaiandClaude Opus 5.5 7e177b06fe fix(telemetry): avoid spurious log directory error on first start (#10446)
* fix(telemetry): avoid spurious log directory error on first start

Since 0.2.4, tracing-appender prunes old log files when an appender
with max_log_files is built, and it lists the log directory before
creating it. When the directory does not exist yet, it prints "Error
reading the log directory/files" to stderr. GreptimeDB sets
max_log_files by default, so every start with a missing log directory
printed this line.

Create the log directory in build_file_appender before building the
appender. Creation errors are left to the appender build, which
reports them as before. Add a test that checks the stderr of a child
process.

Works around https://github.com/tokio-rs/tracing/issues/3612.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(telemetry): drop the first-start log directory test

Review feedback: the test is not needed for such a minor change.
Restore the unix-only test imports that it had merged.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 01:38:28 +00:00
dayan02devanddennis zhuang dbd080b840 fix(recordbatch): treat -0.0 and +0.0 as equal in SimpleFilterEvaluator (#10443)
* test(recordbatch): compare float zero predicates with DataFusion

`SimpleFilterEvaluator` filters SST rows exactly, so it has to agree
with the SQL predicate that DataFusion evaluates above the scan.
DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons
against both zeros, NaN and NULL in Float64 and Float32 columns, and
check the selected rows against DataFusion's filter.

On main this fails: `v = Float64(0)` selects only the +0.0 row, while
DataFusion selects both zeros.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters

DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison
kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator
called the kernels directly, and for floats they compare bit patterns:
-0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as
an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)`
dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding
+0.0. Rows still in a memtable matched.

Normalize the literal when the predicate is built and the input before
it is evaluated, with the helpers DataFusion uses for the same purpose.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* test(sqlness): cover flushed float rows holding -0.0

Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`,
`>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of
them, as they do while the rows are still in the memtable.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

---------

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-11-16 00:43:04 +00:00
Yingwen 1dbc0cd80f fix(mito2): avoid unrelated sparse tag expansion during compaction (#10414)
* fix(mito2): avoid unrelated sparse tag expansion during compaction

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): cover sparse compaction filter and compatibility boundaries

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): handle projected fields in sparse compaction mapper

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): remove redundant compaction projection mapper

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-15 02:34:57 +00:00
Yingwen 1e39eab06b fix(ci): bound nightly kind deployment and restore inotify limits (#10423)
* fix(ci): bound nightly kind deployment and restore inotify limits

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(ci): use shell timeouts in composite actions

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-14 22:16:28 +00:00
dennis zhuang 176bc07b53 docs(skills): focus release notes on user impact (#10442)
* docs(skills): focus release notes on user impact

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(skills): simplify compatibility guidance

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(skills): clarify release scope and highlight criteria

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(skills): align release targets and generated change groups

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-14 02:29:06 +00:00
LFCandgreptimedb-ci 7f7c003ff2 chore(ci): update compatibility versions (#10439)
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
2026-11-14 00:17:49 +00:00
Chenglong Zheng 01d286dba8 docs(rfc): add object-store authority and leader election RFC (#10306)
* docs(rfc): add object-store authority and leader election RFC

Signed-off-by: hahaha <a310608318@gmail.com>

* docs(rfc): fix object-store authority and leader election RFC

Signed-off-by: hahaha <a310608318@gmail.com>

* docs(rfc): fix object-store authority and leader election RFC

Signed-off-by: hahaha <a310608318@gmail.com>

* docs(rfc): fix object-store authority and leader election RFC

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-11-13 18:17:11 +00:00
Weny Xu beb1f88696 feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): release safe WAL history when closing regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover bounded WAL release across close and restart

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): clamp bounded releases to durable entries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): observe close release failures and cover recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: describe bounded WAL release lifecycle

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): make bounded release fence sealing deterministic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): bound queued releases and ignore shutdown hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): cover close release persistence and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): wait for failed close flush cleanup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify bounded WAL release backpressure and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(mito2): clarify staging test retry invariant

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): back off durable release retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): drive release retry after backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): share and verify WAL retry backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify retained WAL release retry semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 12:44:46 +00:00
Weny Xu 68145fc727 fix(mito2): coordinate region edits with repartition lifecycle requests (#10354)
* fix(mito2): coordinate region edits with repartition requests

Extend the pending ALTER queue to defer lifecycle requests blocked by region edits. Preserve scheduler handoffs and drain accepted edits before entering staging, rejecting new unbound imports at that boundary.

Keep ApplyStagingManifest's asynchronous execution and close-time flush behavior. Cover queue bounds, failures, staging transitions, close/reload races, and flush concurrency.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): settle region waiters before completion returns

Reject pending lifecycle requests when retiring a region. Give each completion one dispatch owner while preserving stalled-write ordering, edit-before-compaction dispatch, and fenced-region reload handling.

Cover edit retirement with queued requests; the regression times out when removal only rejects edits.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve queued close flush requirements

Defer later closes behind a pending close even when they do not request a flush themselves. Reuse the existing close-time flush and waiter handoff.

Cover ordinary WAL writes with mixed flush_on_close values and verify committed SSTs before reopening.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito2): clarify region request scheduling names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): retain lifecycle integration after rebase

Restore staging-session checks and bound test edits from the prior merge resolution. Keep StagingEnded dispatch and classify upstream tail replay notifications without adding lifecycle ownership.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 08:39:06 +00:00
Weny Xu 643bdd72db fix(metric-engine): reuse existing column ids when adding columns is retried (#10372)
The metric engine derives the columns to add from the cached physical
schema, and `DataRegion::assemble_alter_request` assigned every given
column `max(non-reserved column id) + 1` without checking the existing
names. That cache can lag behind: Mito reads its in-memory metadata
without going through the region worker, and a caller future dropped on a
client timeout does not withdraw the request that is already queued. A
retry then treats an installed column as new and Mito rejects it, because
the name exists with a different id.

Recheck the real metadata in `assemble_alter_request`, validate an
existing column with `validate_shared_column` and drop it from the
payload, assign ids only to genuinely new columns, and skip the alter when
nothing is new. Serialize the existence check, the id allocation and the
physical alter per physical data region: the mutex lives in
`PhysicalRegionState` and is held by the metric engine orchestration, so
concurrent requests on logical regions sharing the region cannot allocate
one id to different columns. `DataRegion` stays a stateless adapter.

Build the logical mapping and the cached physical columns from the
metadata read after the alter, and report a missing column as an error
instead of panicking.

Signed-off-by: Wenyu Xu <wenymedia@gmail.com>
2026-11-13 08:30:11 +00:00
dennis zhuang 6e025eef61 fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): validate MySQL handshake scope and SQL USE

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): authorize PostgreSQL schema selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): defer PostgreSQL database lookup until authorized

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(servers): propagate catalog lookup errors in protocol tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(postgres): skip unusable search path candidates

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): log database authorization and selection outcomes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): preserve search path responses and schema boundaries

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): clarify database selection diagnostics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): format handshake username in structured logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): secure database-less sessions and failed startup

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): retain authorized default database selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(auth): separate database selection from request dispatch

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(mysql): align schema validation with database-less login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(auth): explain database-less session restrictions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 08:07:57 +00:00
discord9 f88af153ec feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership

Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): send lookup membership as a key list

Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level.

Signed-off-by: discord9 <discord9@163.com>

* docs: tighten narrow membership contract

Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): retain safe bounds on membership fallback

When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): skip unrepresentable dynamic filter snapshots

Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable.

Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): clarify 642-key payload budget comment

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): tighten dynamic membership encoding

Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): reset stale dynamic filters on unsupported updates

Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): scope no-true rule to the encoder

Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): correct reset wording

Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-13 07:51:06 +00:00
Weny Xu 7eb766c9e4 feat(common-meta): make RDS metadata backends closable (#10417)
* feat(common-meta): make RDS metadata backends closable

Standalone closes the metadata backend on shutdown, and
`MetadataKvBackendCreator` now returns a `ClosableKvBackendRef`. Only the
raft-engine backend implemented `ClosableKvBackend`, so external metadata
backends could not be wired into the standalone creator.

Implement `ClosableKvBackend` for `RdsStore` and return
`ClosableKvBackendRef` from `with_mysql_pool`/`with_pg_pool` and the
metasrv RDS builders. The stores only hold reference-counted connection
pools that are released with the last backend handle, so `close()` has
nothing to release explicitly.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): close RDS connection pools on shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): preserve closable RDS constructors and test shutdown semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 07:18:56 +00:00
Weny Xu 180dfc68f7 perf(mito): reduce SST metadata memory overhead (#10356)
* perf(mito): reduce SST metadata memory overhead

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(cache): add a sharded weak interner

Share equal live allocations without exposing lock guards or retaining payloads. Sweep candidate buckets incrementally and cover concurrent publication and weak-reference churn.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): share SST metadata through the sharded interner

Require shared metadata wiring for manifests, canonicalize checkpoint entries without cloning their payloads, and centralize runtime registration. Keep live content variants distinct and verify runtime and recovered allocation identity.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: compile SST metadata consumers across feature sets

Wrap fixture manifest metadata in Arc explicitly and gate the test-only HashMap import with its consuming feature. Validate default mito tests and all cmd targets and features with Clippy warnings denied.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): reclaim retired metadata while idle

Run bounded interner maintenance on the shared blocking runtime without retaining its owner between passes. Shrink sparse indexes and release empty backing storage, preserving live readers and reclaiming their metadata after delayed release.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): use intern-arc for shared SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): expose interner counts and decouple manifest setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(partition): support hashing partition expressions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): hash intern SST metadata without ordering values

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 06:57:23 +00:00
jeremyhi 6faa9ec862 feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store

`LogStore::read_node` streams one region's entries in `(after, upper]` from
the WAL of another node that shares the log store's storage, without writing
to it. The default returns `None`.

The object store WAL implements it with a registry of read-only readers over
its own object store. The node prefix comes from the configured root and the
store's own generation, and the returned stream holds the reader, so reads of
the same node share its cached object indexes. `WalReadUpperBound` moves to
store-api so the trait can name it; log-store re-exports it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): replay pending WAL tails in the background

A leader that is writable, altering or editing and still has pending tails
replays them in one background task per region and publishes the result in
one manifest edit, which opens the read gate and resumes compaction once no
tail remains.

- The worker starts the task when a lease, a catchup or a role change makes
  the region writable, and cancels it on close, drop, downgrade and when it
  replaces the instance. The task always reports its end, so the worker can
  start another replay or reload a region a conflict fenced.
- Each tail is read from this node's WAL, or through `LogStore::read_node`
  for another node, and replayed with the per-entry code of open-time replay
  into memtables outside the region's version. Chunks of about 64 MiB are
  written with the SST half of the flush task, never across two tails, since
  an SST gives its rows its largest sequence.
- The edit adds the SSTs, clears the tails, and advances the flushed entry id
  to the last entry read and the flushed sequence to the largest chunk
  sequence. It commits outside the edit queue, under the manifest lock, only
  while the region is a writable leader recording exactly the replayed
  tails. A busy region or an unknown result retries the same bytes; a
  conflict keeps the chunks; a certain failure deletes them.
- Another node's listed tail is only a bound. After a catchup made the
  region writable, the edit keeps its unread part until the region receives
  a Leader lease, then a second edit clears it.
- After the commit the task releases the WAL up to the new replay start,
  kept below this node's remaining tails, and the worker raises the region's
  last entry id to the new flushed entry id, so a later truncate cannot pull
  it back.
- WAL pressure no longer flushes a region whose WAL release one of this
  node's tails caps.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): harden tail replay across leases, cancellation and uncertain commits

- A Leader lease counts as received after a catchup only if its heartbeat
  was sent after the catchup promotion: the writable fast path does not
  check the start of a downgrade, so a lease granted before an earlier
  downgrade could otherwise clear the mark. The mark is the promotion time,
  so the fast path still reads one atomic.
- A direct promotion counts as a lease.
- The tail commit also raises the committed sequence, so later writes of the
  region take sequences above the replayed rows.
- Registering a writable region starts a replay, which covers a create that
  reopens an existing region.
- Once an attempt may have persisted the edit, later certain failures retry
  the same content instead of building a new edit.
- WAL reads stop when the task is cancelled, and a cancelled task does not
  start another commit attempt.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that closing a region cancels a stalled tail read

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): hold tail commit attempts by number instead of sleeping

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): cancel tail replays when the worker stops

The worker's cleanup closes its remaining regions without the close handler,
so it now cancels every running tail replay first; otherwise a replay stalled
in a WAL read or waiting for a lease outlives the engine. The stalled-read
test covers an engine stop as well as a close. The docs no longer claim that
every SST gives all its rows its largest sequence.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): shorten two tail replay comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): stop tail reads at their upper bound

A tail read over this node's WAL kept reading past the tail's upto until the
log ended, so an unreadable later entry failed a replay that already had every
entry it needed. Stop the read once it reaches upto.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): inline the tail replay notify helper

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that writes during a tail replay win over replayed rows

Whatever records pending tails must raise the region's committed sequence to
the tails' largest sequence, so writes accepted while the replay runs get
larger sequences than the replayed rows. Record this in AGENTS.md and test it
with a write that lands between two commit attempts of the replay.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): bound reads of this node's listed tails

A listed bound may name no entry, so a read of this node's WAL never saw an
entry equal to it and kept polling later objects; one that failed to read
failed the tail. Read every listed tail through `read_node`, which stops at
the bound, and keep reported tails on the node's own WAL.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that rows around a tail replay survive a reopen

The tail commit raises the flushed entry id to the largest entry read, which
can be another node's. Whatever records tails keeps this node's later entry
ids above every tail and leaves none of its unflushed entries below one
outside its own tails. Record this in AGENTS.md, and make the replay test
meet it and reopen the region without a flush afterwards.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject graceful staging entry of a region with pending tails

Entering staging through `set_region_role_state_gracefully` skipped the
pending-tail check that `EnterStaging` runs, so a region with pending tails
could enter staging, end its replay, and return to Leader with nothing
replaying its tails. Run the same check before entering staging there.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): replay tails of a region altered right after promotion

A promotion checked whether it made the region writable after setting the role. If the
worker started an allowed ALTER or edit in between, the check saw Altering or Editing
and asked for no replay, and nothing asked for one later. Check for a leader that a
replay runs on instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover the WAL release and locked checks of tail commits

Check that a tail commit releases the replayed WAL, that it is refused under the manifest
lock once the region was demoted or its tails changed, that a commit retried while
altering keeps its chunks, and that a replay waiting for a lease does not end and start
again.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): note that read_node also reads this node's WAL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep no region version during a tail replay round

A replay round held the region's version until it ended, and with it the memtables the
region had when the round started. A flush removed them from newer versions, but their
write buffer memory was only freed once the round dropped them, so a slow read, a commit
retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails
and the flushed entry id instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): count a direct Leader role set as a lease only when it promotes

A direct role set to Leader counted as a Leader lease even when the region already was a
leader. On a region a catchup promoted, it cleared the mark that keeps another node's
unread listed tail until a lease arrives, so the next commit could drop that tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): move the tail replay leader check onto MitoRegion

The engine and the replay task both use it, next to the other role checks. Its doc now
names the Writable, Altering and Editing states and why staging and downgrading
regions replay nothing.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay waits for a Leader lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin the pressure skip, staging rejection and remote read error

The WAL pressure test gets a case whose own tail starts above the flushed entry id and
still flushes; the graceful staging case checks the rejection; and a remote tail on a log
store that reads no other node's WAL fails as unsupported and keeps its tails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): list every tail replay trigger and the missing admission limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): rename tail replay helpers and notifications

The names now say what each one checks, records or announces: the
replay predicate is a capability rather than a running replay, the
lease hook records a lease, the writable notice is an event like its
siblings, and the commit notice pairs with the end notice.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): ask whether a Leader lease came after a catchup

`is_writable_by_catchup` was negated at both of its call sites. Its
inverse, `leader_lease_received`, is what the replay reads.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): plan the read of a pending tail in the tail replay

`Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task
decides whether a tail comes from this node's own WAL or up to its bound
through `Wal::read_node`, which now takes plain arguments.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay result goes stale

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): send the writable notice through notify_worker

The tail replay already has one helper that sends a background notice
to a region's worker; the writable notice now uses it too, and so logs
a worker that has stopped.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that tail replays skip staging regions and index like flushes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): say why each tail gets its own SSTs and what chunks are for

Ordering between tails needs a tail per SST: an SST can give all its
rows its largest sequence, so an earlier tail sharing an SST with a later
one would take the later tail's sequence and outrank rows that a leader
between the two tails flushed. Within a tail, the 64 MiB chunks only
bound memory: the sequence floor already puts writes accepted during the
replay above every tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the tail replay end guard like other drop guards

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): skip a tail replay edit that a sync already superseded

A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay stops before publishing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover lock waits, busy edits and repeated triggers in tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log every exit that drops a tail replay

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): tell the event listener when a tail replay checks for a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log each tail replay exit once, where it happens

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): make the replacement helper a TestRegion method and fix a lease comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-13 06:32:03 +00:00
Lei, HUANG ed62f8ff54 feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions (#10361)
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions

`iceberg_hash` partition rules could not be evaluated from SQL, so users had no
way to measure a hash distribution or check which bucket a value falls into.
Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)`
which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)`
as text for copy-paste into `PARTITION ON COLUMNS`.

Both functions are immutable and read-only, and no persisted format changes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): handle untyped NULL and small integer counts

`iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray
that the shared array form had no arm for, so it failed with an unsupported-type
error instead of hashing to NULL. Add a DataType::Null arm in the shared
implementation, so both the SQL function and any future rule evaluation keep the
advertised NULL-to-NULL behavior.

`iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the
value was a valid bucket count, because Signature::any leaves the argument width
unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64
arms, and cover them plus the untyped NULL case in unit and SQL tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_breakpoints SQL function

Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution:
a handful of hot keys can still land in one bucket, and a uniform split keeps
hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes
any number of constant values in any order (one shared hashable type, exactly
the values a partition rule compares the hashed column against), hashes them
through the shared implementation, and prints them sorted and deduplicated:

    SELECT iceberg_hash_breakpoints('iceberg', 34);
    -- 1210000089, 2017239379

The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS`
rule, so buckets can be placed around known key values instead of equal widths.
Columns and NULLs are rejected with explicit errors: a column cannot be hashed
at planning time, and a NULL hash never matches a comparison bound, so a NULL
value cannot be a breakpoint (NULL rows always fall in the lowest bucket).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_ranges_sql SQL function

The helpers so far print bare breakpoint numbers, so a user still had to expand
them into comparison rules by hand: n buckets mean n rules with up to
2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders
the whole list from the same equal-width breakpoints:

    SELECT iceberg_hash_ranges_sql('host', 2);
    -- iceberg_hash(host) < 0,
    -- iceberg_hash(host) >= 0

The output drops straight into the expression list of `PARTITION ON COLUMNS`
(rule per line). It only renders text; it does not create or change partitions.
The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the
rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket
count propagates as NULL; columns, non-string names, empty names, out-of-range
counts and wrong arity are rejected with explicit errors.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): include function identity in ScalarUdf equality

The ScalarUdf wrapper compared and hashed only the signature, so any two
functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash
and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared
equal. DataFusion compares expressions structurally, and its SQL planner
rewrites SELECT expressions against GROUP BY keys with base_exprs.contains
(rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x,
iceberg_hash(x) fail planning, because isnull(x) matched the grouped
iceberg_hash(x) and was rewritten into an aggregate column that does not
exist.

Compare and hash the function name alongside the signature, so functions are
distinguishable while two instances of the same function still compare (and
hash) equal.

Add regressions: same-signature functions stay distinct ScalarUDFs and stay
separable in hash-based containers, and grouping by one while selecting the
other plans and executes in both directions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): quote rendered column names and reject mixed hash domains

`iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a
legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses
as a subtraction; names with spaces, dots, quotes or upper case could not be used
either. Quote the name as a SQL identifier and double embedded quotes, so the
rendered rules stay one identifier and survive parsing. Cover the round trip by
feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain,
dashed, upper-case, quoted and spaced names.

`iceberg_hash_breakpoints` accepted values of different types, but strings and
integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so
mixing them yields breakpoints that describe no real column. Require one hash
domain (all strings or all integers, integer widths may mix) and name the
offending argument otherwise. The existing SQL case that mixed
`('iceberg', 34)` now uses two strings.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(function): register the raw hash UDF and tidy the helpers

Registering `iceberg_hash` through a `Function` wrapper built a different
`ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared
the SQL expression and the rule expression as unequal; the wrapper also rebuilt
a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`,
so register `iceberg_hash_udf()` directly and add a test that the registered
function and `IcebergHash::to_logical_expr` produce equal expressions.

Also: collapse the arity check and slice destructuring into one `let ... else`;
make the bucket-count helpers associated functions instead of constructing a
`default()` instance; pass the caller name through so an error raised from
`iceberg_hash_ranges_sql` names that function rather than
`iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY`
instead of rebuilding and wrapping each one.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): accept every string column name and cover the rules end to end

`iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its
column-name argument, so a valid constant string that arrives as
`Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as
non-string. `Signature::any` leaves argument types unchanged, so the function
has to accept all string representations itself; use `ScalarValue::try_as_str`
and keep the NULL and empty-name checks. A unit case and a SQL-level case pin
that `Utf8View` renders the same rules as a plain literal.

Add a sqlness case that renders the four buckets for the quoted column
`"host-name"`, creates the table from that exact text, inserts one value per
bucket plus NULL, and checks the row counts implied by the rendered predicates
against the rows each region stored. This closes the gap between rendering the
rules and routing rows with them.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(partition): hash constants through IcebergHash::evaluate_array

`iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for
line. Call the existing method from `into_hashed_scalar` and drop the extra
public entry point, so the SQL helpers still hash through the code a partition
rule evaluates without a second identical API.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-13 06:10:13 +00:00
dennis zhuang ca1b8c7c98 fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): redact credentials at their owning fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): cover skipped SAS tokens and export logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask SAS tokens before overlapping account names

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): redact SAS tokens before overlapping account fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask containing credentials before shorter values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 05:35:39 +00:00
dennis zhuang 7b455f2b81 docs: record authorization invariants in servers and frontend guides (#10384)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 04:07:39 +00:00
dennis zhuang e7f75a73dd fix(frontend): classify physical entity graph sources (#10383)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 04:07:36 +00:00
dennis zhuang 56cb03f4fb fix(function): handle unaligned binary vector literals (#10380)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 04:07:31 +00:00
Weny Xu 7d0186457b fix(mito2): notify flush waiters after failure cleanup (#10398)
* fix(mito2): notify flush waiters after failure cleanup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): simplify flush failure replies and test notification ordering

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 03:57:28 +00:00
dennis zhuang 4bda315fde fix(frontend): check table permissions for COPY FROM STDIN (#10374)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 02:17:23 +00:00
dennis zhuang 5d88de5290 fix(servers): authorize HTTP database overrides (#10378)
* fix(servers): authorize HTTP database overrides

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): update the HTTP output benchmark handler call

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): authorize Prometheus database overrides

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): retain authorized label lookup scope

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 16:42:09 +00:00
dennis zhuang 87af50bfdd fix(build): restrict dashboard tokens to the GitHub origin (#10379)
* fix(build): restrict dashboard tokens to the GitHub origin

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: remove standalone dashboard download test

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 16:35:53 +00:00
sun 2e1ce9808a feat: update dashboard to v0.14.6 (#10365) 2026-11-12 16:24:42 +00:00
jeremyhi 9f6c957e2b fix(mito2): commit staging edits once when staging ends (#10322)
* fix(mito2): commit staging edits once when staging ends

Staging edits (the new partition expression, the imported files) were saved
as deltas in the staging directory, each taking a manifest version. The end of
staging listed that directory and merged every delta it found, including
deltas a stale writer wrote late.

Staged actions now stay in memory and take no version. The end of staging
commits them once, as the next version, in a new EndingStaging state that
stalls writes and holds role changes back until the commit returns. If the
commit may have been written and the region stays staging, the encoded commit
is kept: a retried request stages nothing new and commits the same content,
and other staged edits are refused with a retryable error meanwhile.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle an uncertain end of staging before later commits

Under overwrite publication a save that may have landed must not be
overwritten, so every commit takes its version before the save and a failed
save leaves that version unused. For the end of staging, the version is now
taken only once the result is known: a failed save is read back at once, and
if the read-back cannot tell, the commit stays unsettled. Every later commit
first reads it back again, applies it to the manifest and the region version if
it landed, and is refused with a retryable error while it still cannot tell.
A follower sync past that version forgets it.

Committed staging actions are applied to the region version through the
manifest context, so the end of staging and a later commit that settles it
share one path.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle before checking a commit and keep the lock into the end of staging

- Settle an unknown end of staging when taking the manifest lock, before the
  write is checked against the manifest, and fire the hook of a landed one
  with the lock released, so a failing commit cannot drop it. Entering staging
  settles too; a staging region settles only when staging ends.
- Pass the held manifest lock from the graceful Leader path into the end of
  staging, so a queued close cannot abort staging in between.
- Wait for the worker to drain the stalled writes before clearing the staging
  directory.
- Let an empty import retry a frozen end commit.
- Count the size of an end delta found by reading it back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep settling out of staging and before a new staging

- Do not settle while the region is staging or a staged edit is running; only
  the end of staging settles there, as it also leaves staging.
- Settle before entering staging through a graceful role change, and fire the
  hook of a landed result even if entering staging then fails.
- List the end-of-staging outcomes in the order they are decided.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): settle before the metadata backfill and fence the late staging clear

- Settle an unknown end of staging before a graceful Leader on a writable
  region checks whether to backfill metadata; a landed result is merged into
  the backfill notification.
- Skip the staging directory clear after the end of staging if the region was
  closed meanwhile, as it may be open and staging again.
- Clear the staging directory after an end of staging with nothing to commit.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): take the overwrite version before the end of staging is saved

Under overwrite publication the end of staging takes its version before the
save again, as before this change, so a later commit never overwrites a save
that may have landed. After an error that may have written it, the region goes
back to staging with its staged actions and encoded bytes kept, and the retry
writes the same bytes at the next version; a reopen replays both as one commit.

This removes the immediate read-back of a failed end commit, the unsettled mark
and the settling before later commits, the ManifestUnsettled error, and the
region version handle on the manifest context. The end of staging applies the
committed actions to the region version itself again.

A graceful Leader from staging takes the manifest lock again after the end of
staging and runs the metadata backfill, firing one notification for both, as
before this change.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): check the lease and fence conflicts in the end of staging commit

The end of staging commits its staged actions on its own path, so it
skipped the checks other manifest commits make: in conditional
publication it saved after the Leader deadline had passed, and a version
conflict left the region unfenced, so the worker never reloaded it. Run
both commits through one core that makes those checks.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): let the worker pass the drain to the end of staging

The end of staging built and sent the worker's StagingEnded notification
itself, so the region module depended on the worker's dispatch types.
The worker now passes the drain as a future that the end of staging
awaits after it drops the manifest lock.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the write order across the end of staging

The end of staging stores the region's next state under the manifest
lock, and the worker handles the writes it stalled meanwhile later, on
StagingEnded. A write that the worker took in between was applied
before the earlier stalled ones. The worker now stalls a region's later
writes behind those until StagingEnded drains them.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): read and encode the staged actions without copying them

The end of staging copied the staged actions to read them, to encode
them and to commit them. Read and encode them in place, and copy them
for the region hook only when one is registered. The commit keeps its
own copy, so the staged actions survive a failed commit for the retry.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover a flush while staging ends and the staging directory clear

Check that a migration flush gets a retryable error while the region
ends staging and succeeds after it. Plant a delta from an older binary
in the staging directory before applying the staging manifest, so the
check that the directory is cleared looks at the target region and
fails if the end of staging leaves it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): tidy the end of staging wording and a test helper

Nothing is merged at the end of staging any more, so its log, error
reasons and bindings say staged or committed. Fix the region hook doc
sentence and table cells for actions staged in memory, and move the
import helper of the conditional manifest tests next to the other
helpers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the staged files of a region that left staging

An end of staging that leaves for Follower or Downgrading kept the staged
actions in the manager, and the SST listing and all_manifest_files merged
them into the files of a region that never committed them. The end of
staging now drops them when it leaves staging, and both readers merge the
staging manifest only while the region is staging.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): check a resent import against the staged files in linear time

The check that a resent import only adds files that are already staged
walked every staged edit for every file, quadratic in the file count. It
now collects the staged file ids into a set once.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(mito2): apply the committed staged actions without another copy

The end of staging cloned the staged actions to split them for the region
version before the commit, on top of the copy the commit applies. The
commit now hands the staged list back once it succeeds, and the end of
staging checks the staged Change by reference before the commit and splits
the returned list after it. One copy stays, because a successful commit
uses the actions twice: a copy builds the new manifest, and the returned
list updates the region version. The region hook gets its copy only when
one is registered, as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): return the normal version from all_manifest_files and drop a dead fallback

all_manifest_files preferred the staging manifest's version. It now returns
the normal version, as its doc says. A test helper's unreachable
`unwrap_or(path)` after `rsplit` becomes `unwrap_or_default()`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the staging helpers after what they do

- Rename is_staging_or_staged_edit to in_staging_session. The Staging and
  EndingStaging pair lives in RegionRoleState::in_staging_session, which the
  compaction scheduler, region_role, should_abort_index and the region-level
  helper use.
- Rename finish_ending_staging to leave_ending_staging, since it can also
  land on Follower, Downgrading or Staging, and tie end_staging,
  EndingStaging and StagingEnded together in the EndingStaging doc.
- Rename contains_import to only_adds_staged_files and the staged list
  field to actions.
- Move the staging paragraph to the public update(), state on end_staging
  that the drain is never awaited under the manifest lock, say when
  all_manifest_files merges staged files, say that the listings do not rely
  on the clear at the end of staging, and count the end of staging in the
  flush rule of the mito2 AGENTS.md, naming the predicates the flush paths
  test.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): keep the end-of-staging drain next to the stalled requests

Move drain_ended_staging into worker.rs, and the set of regions whose writes
wait for it into StalledRequests, which adds, checks and removes them, so
the stall bookkeeping and its release live together.

Rejecting all stalled writes now also forgets these regions. Their stalled
writes are rejected with them, so a later write has nothing to keep order
behind and is handled under the region's current state.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): keep the staged view with the staged actions

The manifest with the staged actions applied was a separate field of the
manager, set and cleared next to the actions by hand. It is now a field of
StagedActions, so clearing the staged actions clears it as well, and its
accessor says what it returns.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): split EndStagingPending out of RegionBusy

A staged append that arrives while a possibly written end commit waits for
its retry was refused with RegionBusy, which therefore had to be in the
not-written arm of may_have_persisted_manifest_update for every producer.
It now gets EndStagingPending (status RegionBusy, retryable), which alone
takes that place, and RegionBusy keeps its earlier classification.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover regions that are ending staging

Check that a region ending staging reports the StagingLeader role and skips
compaction scheduling, and that its bulk writes stall, as do later ones
behind them after staging ended, until the worker drains them. Share the
stall wait and the StagingEnded drain between the tests.

The tests wait for these stalls through a listener on their own engine, so
the end-of-staging stalls notify the test listener as the write buffer
stalls do, and CountingStallListener moves to the shared test listeners.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-12 13:47:01 +00:00
Weny Xu 6420082685 fix(meta): skip metric engine identity columns when merging partition columns (#10368)
CreateLogicalTables copies every physical partition column into the logical
schema as a nullable String tag. After a physical metric table is repartitioned
by the internal `__table_id` column, that column reached the region create
request and the metric engine rejected it with "Internal column __table_id is
reserved", so no logical table could be created.

Filter engine-owned identity columns while merging, reusing
`is_metric_engine_internal_column`. Filtering at the merge point also protects
procedures that persisted an unfiltered `physical_partition_columns` list.
User partition columns keep the previous behavior.

Write routing for a physical table partitioned by `__table_id` is still
tracked by #10352; this change only unblocks logical table creation.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-12 12:03:33 +00:00
discord9 c937e3997b perf(mito2): prefilter single-column dynamic filters (#10340)
* perf(mito2): prefilter dynamic join membership

Read the narrow dynamic-membership column before the full projection and use a stable per-row-group mask to select rows. The live filter above the scan stays authoritative; bounds-only and unsupported shapes keep all rows.

Signed-off-by: discord9 <discord9@163.com>

* test: cover dynamic prefilter cache and generations

Prove the prefilter-result cache keeps the static mask but never reuses a dynamic mask across producer generations. Document why a superseded dynamic mask is discarded rather than assumed monotone.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 10:05:51 +00:00
Weny Xu b5cdcba82a fix(datanode): keep flush task error status code and retry hint (#10360)
The synchronous FlushRegions handler wrapped the error returned by the
flush task into Unexpected before building the reply. InstructionError
then only copied the Unexpected status code and the default NonRetryable
hint, so metasrv could not tell a transient flush failure from a
non-retryable one.

Return the task error unchanged so its status code and retry hint reach
FlushRegionReply, and rebuild the reply message with the same debug chain
the previous wrapper produced to keep the diagnostics unchanged.

Add handler-level tests covering RegionBusy, RegionNotFound and Unexpected
task errors under both FailFast and TryAll.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-12 08:56:31 +00:00
jeremyhi 6a202adbe7 fix(log-store): skip objects collected below the boundary in WAL reads (#10343)
* fix(log-store): skip objects collected below the boundary in WAL reads

The read-only WAL reader lists the source prefix and then reads the
header and footer of every listed object. While the store writing the
prefix is still collecting objects, a listed object can be deleted
before its footer is read. The NotFound failed the whole read, even when
the object lay below the reclaim boundary and was off the chain the read
replays, so reads of a large prefix under steady writes failed often.

Apply the rule startup recovery already uses. An object gone below the
reclaim boundary at which the chain of the read stops is skipped. Any
other vanished object, on the chain or off it, still fails the read: it
may be the object that completes a later chain or carries the largest
epoch. Startup recovery and the reader now share one NotFound split and
one boundary check.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover unreadable objects in listed WAL reads

A NotFound below the boundary is skipped, but any other error reading a
header or footer must still fail the read. Cover it with a corrupted
header below the boundary in both modes.

Describe the boundary in `ensure_vanished_below` and `read` in terms
that also fit the `LastEntry` mode, where the chain can stop at the
object of `after` instead of a reclaim boundary.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe where WAL reads stop following links

In the `LastEntry` mode a later object can carry a reclaim boundary
above the object of the reported entry. The read then keeps that object
as its tip below the boundary and follows no link from it. Say that the
boundary stops the links to predecessors, and that the tip is the one
object below it a read can need.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): pin the outcomes of vanished-object reader tests

The unreadable-object test now asserts the InvalidWalObject error for
object 1 inside the index cache's wrapper, instead of any error other
than NotFound. The vanished chain-object test states the outcome it
guards against: without object 2 a listed read could select the chain
ending at object 1 and succeed with no entries.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-12 08:02:29 +00:00
dennis zhuang 62dad907d0 fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed
object store credentials of external tables verbatim, and SHOW CREATE TABLE
masked only access_key_id and secret_access_key. Mask all four credential
keys in both paths, and sort extra options in create_options by key.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 05:49:19 +00:00
fengyue-xve f8d8fe1071 fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk (#9452)
* fix: honor the pipeline version header and URL parameter in Elasticsearch _bulk

The Elasticsearch _bulk handlers read only pipeline_name and msg_field and
hardcoded the pipeline version to None, so the x-greptime-pipeline-version
header and the version URL parameter were silently ignored and the latest
version was always used.

Resolve the version from the version query parameter, falling back to the
x-greptime-pipeline-version header, and pass it to
PipelineDefinition::from_name; an unparsable version now fails the request
instead of being ignored. Without either source the latest version is used,
as before.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(elasticsearch): reject invalid pipeline version headers and use HTTP item status

Signed-off-by: fengyue-xve <1759519137@qq.com>

---------

Signed-off-by: fengyue-xve <1759519137@qq.com>
2026-11-12 01:11:21 +00:00
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
2026-11-12 01:09:42 +00:00
Weny Xu 15fd97e369 fix(meta-srv): stop failing over regions that are still opening (#10290)
* chore: update proto for opening region heartbeats

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reset failure detectors during background region initialization

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve detector history for opening regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update opening region proto revision

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update opening region proto to merged main

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-11 12:57:10 +00:00
discord9 90d0c00a82 fix(promql): resolve the OR right operand's time-index column from its schema (#10335)
`or` inferred the qualifier of an operand plan from its first field
(`schema.qualified_field(0)`). A binary operation with an explicit matching
modifier derives its result labels from the left operand when the match
reduces the label set, so its projected plan carries two qualifiers: the
labels are `lhs.<label>` while the time index keeps the context table's
qualifier (`rhs.<greptime_timestamp>`). Prefixing the time index with the
first field's qualifier then referenced a column that does not exist, and
planning failed with

  Internal error during building DataFusion plan: No field named
  lhs.greptime_timestamp. Did you mean 'rhs.greptime_timestamp'?

for e.g. `sum by (l) (f(a) / ignoring(x) group_left() f(b))` used as the
right operand of `or on (l)`. `f(a) / ignoring(x) f(b)` triggers it as
well, so this is not specific to a group modifier. Without a matching
modifier every projected column takes the right operand's qualifier, which
is why plain `a / b` never hit this.

Resolve the time index against the actual operand schema: keep the
preferred qualifier when it really holds the column (uniform schemas, so no
plan changes), otherwise take the qualifier of the column found by name and
let DataFusion report ambiguity instead of guessing.

Signed-off-by: discord9 <discord9@163.com>
2026-11-11 10:06:12 +00:00
jeremyhi d1e5a2df8f fix(mito2): check the lease before manifest commits and reload regions after conflicts (#10311)
* fix(mito2): check the lease before manifest commits and reload regions after conflicts

In conditional manifest mode, refuse a commit whose Leader deadline has
passed before its PUT, and fence a region whose commit hits a manifest
version conflict, then reload it in the background and swap the new
instance in.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): release the flush status when the WAL release after a flush fails

handle_flush_finished failed the flush's waiters and returned when the WAL
release after a committed flush failed, but left the flush registered in the
flush scheduler, so every later flush of the region, and the reload of a
fenced region, waited behind it forever. Handle the failure as a failed
flush, which removes the registration and answers the pending requests.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover staging saves after the deadline and a fence during catchup

Add a listener hook before a catchup reopens the region, and test that a
fence landing there makes the catchup answer RegionBusy without replacing
the instance. Also test that entering staging after the Leader deadline
saves no staging manifest.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that set_region_role does not check the reload mark

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-11 06:44:06 +00:00
Lei, HUANG c8e0d29df2 feat(mito2): publish table compaction results conditionally (#10310)
* feat(mito2): publish table compaction results conditionally

Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): write table compaction outputs to level 0

Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): release verified footers before manifest publication

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): compare typed table compaction options

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover table compaction output validation failures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify series count verification limits

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject index declarations without output index objects

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document table compaction outputs as L0

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): map the table compaction publication module

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-11 05:42:51 +00:00
dennis zhuang ddc16bafe8 fix(datatypes): render jsonb stored as JSON text without panics or wrong values (#10197)
* fix(datatypes): decode jsonb before rendering it as text

jsonb_to_string serialized the bytes through RawJsonb's Serialize impl,
which panics with an out-of-range slice index on JSON text such as
"hello world" or "        null" (stored by COPY FROM and MySQL text
parameters) and on truncated jsonb. Its text fallback was also stricter than
the extended parser reads used before.

Decode with jsonb::from_slice, as reads did before #10112, then write the
value out with serde_json escaping for keys and strings.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(datatypes): keep streaming jsonb serialization in jsonb_to_string

Decoding into a jsonb::Value tree allocates per array, object and key on
every JSON cell returned over PostgreSQL and MySQL. Go back to serializing
the raw bytes with RawJsonb's Serialize impl, and bump jsonb to a rev whose
Serialize returns an error instead of panicking on out-of-range payloads
(GreptimeTeam/jsonb#2). Bytes that fail to serialize are parsed as JSON
text with the extended parser, as jsonb::from_slice does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(datatypes): pin text values that used to read back as false

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: bump jsonb to reject scalar headers with non-zero length bits

JSON text over ~2 MB could still pass the payload bounds check and read back as a wrong value (GreptimeTeam/jsonb#2).

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(datatypes): report the text parse error when JSON bytes are neither jsonb nor text

Also pin jsonb_to_serde_json on JSON text scalars that used to decode as false.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: pin jsonb to the merged commit on main

GreptimeTeam/jsonb#2 reached main through GreptimeTeam/jsonb#3.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-11 03:46:56 +00:00
discord9 97c8136c3c chore(deps): bump datafusion to ae04160b (#10320)
* chore(deps): bump datafusion to ae04160b

Move all 16 `[patch.crates-io]` DataFusion pins from `cc5534492` to
`ae04160b`, the merge commit of GreptimeTeam/datafusion#42.

That fork commit backports six upstream apache/datafusion fixes:

- #26452 timestamp-to-time casts no longer drive statistics pruning
- #24675 memory-limited NLJ fallback refuses left-emission joins with a
  multi-partition probe side instead of emitting wrong unmatched rows
- #24891 RepartitionExec spill-pool deadlock with two open files
- #25076 null-aware anti join no longer emits rows after a sibling probe
  partition saw NULL
- #25339 correlated NOT IN with a non-equality correlation
- #26578 null-aware anti join drops build rows with a dictionary NULL key

The first three are faithful backports. The last three are re-implementations:
upstream restructured `hash_join/stream.rs` before writing them, so the patches
do not apply to this fork's older copy of that file. `#25339` covers the
single-equi-key plus residual-filter shape; multi-key correlated `NOT IN` still
fails at planning as before.

`Cargo.lock` is regenerated (35 `datafusion*` packages).

No runtime acceptance has been run against this pin; see the PR description for
the three checks worth doing before shipping.

Signed-off-by: discord9 <discord9@163.com>

* fix(common-query): pass the pruning bitmap to HashTableLookupExpr::new

#39 added a fifth parameter to `HashTableLookupExpr::new`; the dynamic-filter
payload round-trip test still constructed it with four arguments, which breaks
`--all-targets` builds even though the library itself compiles.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): bump datafusion to 70f84c5b

Move the pins from `ae04160b` to `70f84c5b`, the head of
`greptimedb-55.1-thin-fork` after GreptimeTeam/datafusion#37 was merged on top of
#39/#41/#42.

`ae04160b` did not contain #37, so the reverse-cast constraint propagation
guard was lost and `tests/cases/distributed/optimizer/partition_pruning_cast`
failed: the filter `CAST(k AS BIGINT) = 1` made the planner treat the float
column `k` as a singleton and drop the requested `ORDER BY k DESC`.

The new pin carries both the reverse-cast guard and the six release backports,
so the regression is covered again.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-10 18:17:24 +00:00
Weny Xu d061e08717 feat: flush regions under object store WAL retention pressure (#10309)
* feat(store-api): expose advisory WAL flush candidates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(log-store): select regions blocking WAL object reclamation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): schedule region flushes under WAL retention pressure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover WAL pressure reclamation and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): prioritize oldest WAL reclamation blockers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): deterministically exhaust range cache admission budget

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 16:06:17 +00:00
Weny Xu 54b6bd5b44 feat(mito2): support extension index providers and input reuse (#10297)
* feat(mito2): register extension index providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito2): support index input reuse and extension budgets

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 14:02:48 +00:00
jeremyhi 77cb39b5d1 feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest

Adds `pending_tails` and `handoff_floors` to the region manifest and its
checkpoint, and an optional `pending_tails` / `handoff_floor` pair to
`RegionEdit`. An edit without them leaves the records unchanged. A floor is
dropped once the flushed entry id reaches it; a tail stays until a commit
removes it. Neither is written when empty, so manifests without them encode
as before.

Opening a region replays its WAL after this node's tails and floor, and WAL
releases at open and after a flush stay below this node's tails. Nothing
writes these records yet, so every region behaves as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): hold back reads and region changes while WAL tails are pending

A region with pending WAL tails is missing data that only those tails hold.
Until they are recovered, scans and memtable and SST stats fail with the
retryable `RegionPendingTails` error. Truncate, entering staging, a
repartition flush, a close-time flush on request, and ALTERs other than
adding nullable columns without a default or changing options are rejected
before they change any region state. Compaction pauses. A full-listing GC
pass skips the region while it has pending tails, or while a newer manifest
may exist or the manifest listing fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): name the WAL releases that keep pending tails

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that reopening restores the pending-tail read gate

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin what the pending-tail fixtures and rejections promise

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the replay start in the open log and share the pending-tail check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the per-node order of pending tails and who keeps later writes out of them

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending

When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the pending compaction task state when WAL tails skip it

Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-10 12:21:17 +00:00
Weny Xu e2e980fe39 feat(mito2): add column sketch options and index metadata (#10295)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 09:55:20 +00:00
jeremyhi 9eb027f738 test: disable the event recorder in tests that count WAL writes (#10316)
The object store WAL restart test and the skip-WAL tests built on
MockInstanceImpl assume only the tables they create write to the WAL.
The event recorder also writes greptime_private.events into the same
WAL, so when it flushes during the test (the first tick of its interval
lands after the first DDL event on Windows), the extra region keeps WAL
objects from being collected or shows up in the WAL snapshot.

Record no events in these tests.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-10 09:29:35 +00:00
shuiyisong ee1fc7782f feat(prometheus): query persisted metric exemplars (#9924)
* feat(prometheus): add exemplar queries and lifecycle coverage

Expose /v1/prometheus/api/v1/query_exemplars with exact time bounds and series selectors. Resolve and authorize source selectors before scanning exemplar storage, enforce query limits and cancellation, preserve exact observation identity, and return warnings for malformed stored rows.

Includes HTTP and authorization tests, pre-created-table and concurrent-DDL coverage, OTLP Arrow ingestion checks, and cross-protocol lifecycle tests across memory, WAL, SST, compaction and reopen. The query-dependent ingestion tests live here so every lower layer builds independently. Add metric-exemplar usage and retention guidance.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): preserve query labels with source identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): floor bounds and add audit hooks

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* • fix(exemplar): include aggregation parameter selectors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): avoid v1 exemplar quota stalls

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align exemplar query timestamps with Prometheus

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove unnecessory doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-10 08:11:49 +00:00
shuiyisong 09c0291575 fix: restrict JSON retries to invalid UTF-8 errors (#10302)
* fix: restrict JSON retries to invalid UTF-8 errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: isolate UTF-8 retry counter assertions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-10 07:42:27 +00:00