Commit Graph
6050 Commits
Author SHA1 Message Date
discord9 48bb6beadd feat(promql): name native result columns from expressions (#10231)
Use the statement expression for native value column names, resolve legacy value columns by type rather than by name, and reject label-function destination collisions instead of dropping a value. Keep matrix roots named after the whole statement, preserve an Int64 __tsid value, and propagate schema-collision errors from canonical boundaries.

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 12:53:55 +08:00
discord9 3bdde3d339 test(query): prove a default engine option keeps native requests on the legacy route (#10459)
The engine-wide `experimental_native_promql` option is the outer gate. #10371
pinned its default, but nothing showed what that default does to a request that
explicitly asks for the native route — which is the case that actually matters
when the branch is merged back: a client can always send the per-request
extension, and the engine option is the only thing left deciding.

`plan_native_promql_with_query_options` always sets
`query.experimental_native_promql=true` on the request, so passing
`QueryOptions::default()` covers exactly that combination. The assertion looks
at the rendered plan for both `FusedSeries` and `RawSeriesSource`, so a future
change that lets the request alone admit the route fails here.

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 12:01:04 +08:00
discord9 6796dfeaee test(promql): cover unequal-width owner routing and scalar broadcasts (#10039)
Signed-off-by: discord9 <discord9@163.com>
2026-11-17 11:58:52 +08:00
discord9 089121c613 Merge main into narthex
Conflicts:
- `src/common/query/Cargo.toml`: `futures-util` stays non-optional because the
  series-frame API exposes `futures_util::Stream`; `testing` only pulls
  `datafusion-datasource`.
- `src/frontend/src/instance.rs`: import union (`ServerMemoryLimiter` from main,
  `MetricDiscoveryScope` from narthex).
- `Cargo.toml` / `Cargo.lock`: follow main's newer greptime-proto rev
  (`c46f7ee9c`), which contains the `RawSeriesSource` this branch needs.
- `tests-integration/src/standalone.rs`: keep both `query_options` and
  `auto_create_physical_table_timestamp_unit`.

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 10:13:41 +08:00
fys 453c35e9d3 feat: support trigger cache invalidation through frontend plugins (#10460)
Signed-off-by: fys <fengys1996@gmail.com>
2026-11-16 23:14:07 +00:00
jeremyhi c6962802a9 feat(meta-srv): bind candidate region leases to the migration's registration (#10440)
* feat(meta-srv): bind candidate region leases to the migration's registration

A region migration now gives its candidate region registration a random non-zero
candidate lease id. Leases granted because of that registration carry it, but only
while the table route exists, and the upgrade instruction carries the same id. The
datanode passes the id of each lease to the engine, and Mito records the lease of
the latest heartbeat per region outside the region instance, so it survives a
reopen during catchup.

The migration also drops the candidate registration before it restores the route
on rollback, and before the procedure leaves the migration tracker when it ends
without a rollback.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a lease only while the region stays open

Record a granted lease under the same lock that removes it after the region closes, so a lease recorded during a close does not outlive the region. The metric engine records the data region's lease when the metadata region ignores it. Remove the roles-only extraction of operating regions, and cover a present candidate lease id in the upgrade JSON.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* build: point greptime-proto at the candidate lease id change

Use the greptime-proto commit that adds GrantedRegion.candidate_lease_id (greptime-proto#357), which also brings in the RawSeriesSource Substrait payload message. Rely on the declaration order of RegionMigrationProcedure to drop the candidate registration before the tracker entry, and document it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop lease records when workers stop

Remove the lease records of the regions a stopping worker closes. Check under the removal lock deterministically in tests, cover a lease that arrives after a region is removed, assert the candidate lease id on the migration's real registration and its retry, and match ids per region in a batched upgrade.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): drop a redundant helper comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(store-api): pin the candidate lease id wire bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: document candidate registration and lease record lifecycles

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep candidate lease ids through customized lease renewers

The heartbeat handler now sets the candidate lease id of each lease a
customized renewer returns from the renewed region with the same id.

The metric engine records a lease for the data region only when the
metadata region ignored it, since the data region records its own
leases. The countdown reset command carries a named struct instead of
a tuple.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): cover a lease the data region ignores

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tighten lease id lookups and lease test assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: align lease naming and share the txn-observing kv backend

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: tidy lease naming, docs and shared test helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop field comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop method comments that restate their names

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: say which heartbeat a lease's send time refers to

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: drop the reset deadline field comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: state when an operating region's role and id reach a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: pin greptime-proto to the merged candidate lease id change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: add the candidate lease id argument to the direct demotion test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: order new keeper helpers after public items and test equal send times

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 18:37:17 +00:00
Ning Sun c679ca5fdc feat(ingestion): configurable time index unit for auto-created physical metric tables (#10189)
* feat(ingestion): configurable time index unit for auto-created physical metric tables

Auto-created physical metric tables hardcoded a millisecond time index.
Add an `auto_create_physical_table_timestamp_unit` option (frontend and
standalone configs, default "Millisecond") that selects the time index
unit for physical metric tables created by auto table creation. New
logical tables created under such a physical table inherit its unit, so
Prometheus remote write, OTLP metrics, and other metric-engine ingestion
paths write timestamps in the configured unit.

The option threads through all auto-create sites: the operator inserter's
`create_physical_table_on_demand`, the frontend instance's batched
logical-table flush path (`create_metric_physical_table_if_missing`),
the logical-table batcher's unit alignment fallback, and the OTLP metric
preflight validation unit. Millisecond remains the default, so existing
behavior is unchanged unless configured.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(otlp): match default physical schema to configured timestamp unit

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-16 17:00:51 +00:00
Ning Sun 7e76d28908 fix(postgres): account for buffered InfluxDB and OTLP COPY bodies (#10450)
Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-16 15:59:31 +00:00
Weny Xu 610794e447 perf: batch logical table cleanup when dropping databases (#10461)
* perf: batch logical table cleanup when dropping databases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refine drop database batching and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify drop database traversal and batch sizing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: tighten drop database batch size estimate

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: narrow logical table metadata cleanup API

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 14:17:35 +00:00
jeremyhi 0df6e25f3e feat(mito2): bound tail replay resources and recover from failed rounds (#10444)
* feat(mito2): bound tail replay resources and recover from failed rounds

Tail replays now write SSTs through a MemtableSstWriter split out of
RegionFlushTask, with their own WriteType and metrics, so replay chunks
no longer count as flushes. A node runs at most 16 replays at once,
gives their SST writes their own permits, and bounds their chunk
memory by a budget of 1/8 of the global write buffer: each replay keeps
a floor of budget/32 and the chunks share a pool of budget/2 above it.

After a failed round, a writable region whose storage holds a newer
manifest is fenced and reloaded once the replay ends, so tails that
another writer already committed stop being retried.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): make the replay update check cancellable and keep flush logs

The manifest check after a failed replay round listed storage under the
manifest read lock and could not be cancelled, so closing a region whose
listing stalled waited on it. The listing now runs under the replay's
cancellation, and only the fence takes the write lock.

Flush completion logs keep their reason, replay metrics record memtable
compaction, and tests cover a stalled check on close, flushes running
while replay writes are saturated, and the no-index case of async index
builds.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the wait for the replay fence lock cancellable

The replay now waits for the manifest write lock under its cancellation
and only rechecks and fences once it holds the lock, so a demoted,
closed or replaced region ends a replay queued behind another manifest
operation. It logs when it waits for a replay permit, and the replay
metrics are crate-private.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): wait for the replay to reach the fence before demoting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): address review of the tail replay limits

The update check after a failed round lists the manifest directory
without holding the manifest lock; the write-lock recheck catches any
version installed meanwhile. A replay snapshots its tails once it holds
its permit, logs only when it waits for one, and logs when it fences or
declines to. The writer takes the region hook instead of the manifest
context, and names, docs and tests follow the review.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): cancel the tail replay on a direct demotion

A direct role change that leaves a region unable to replay its tails,
such as a demotion when its lease expires, now asks the worker to
cancel the replay, as a graceful downgrade already does. A replay
waiting on storage I/O then ends, and a later promotion starts another.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): cancel the tail replay when a lease demotes the region

A lease that grants a non-writable role now cancels the region's tail
replay like a direct demotion does; both share one helper, which sends
nothing when the region has no pending tails. Docs, logs, metric
observation and tests follow the second review.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): ignore demotion notices of replaced region instances

A demotion notice now carries the region instance it applied to, and the worker cancels the replay only while that instance is still registered, so a late notice of a replaced instance leaves the new instance's replay running.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): narrow the tail replay byte counter doc

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): align tail replay names and docs, test reservation bookkeeping

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): derive the replay log reason from its write type

The SST writer now logs the flush reason for flushes and the write type
otherwise, so the replay no longer passes a bare string. Drops an engine
test that also passed without the replay memory pool; the reservation
unit tests cover the pool's accounting.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): close the demotion and manifest version races of tail replay

A role change now cancels the tail replay whenever it leaves the region
unable to replay, instead of comparing with a role read before the
change, which a concurrent promotion could make stale. The manifest
check after a failed round lists from, and rechecks, the installed
manifest version rather than the last version number, which a failed
commit leaves ahead of it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin the demotion notice to the state after the role change

Adds a case where the region is already a follower when the engine sets its role, and words the unwritable notice docs by the state after the change.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): rewrap the unwritable notice doc

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): notify only when a role change makes the region unable to replay

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): assert every set_role return value

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): drop transition headings that repeat the code

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): restrict memtable writer construction and simplify write metrics

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): tie the flush reason to the writer kind and share replay test helpers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): correct the replay histogram range and listener doc

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): keep tail replay doc links within the documented items' visibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 13:40:00 +00:00
shuiyisong 0a93810c23 fix: improve exemplar handling and batch memory admission (#10363)
* chore: update README

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): reserve memory for table batches

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): prefix the labels column

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): hide exemplar table from metadata

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): reconcile source tags before writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): prune tags and retain source IDs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): record slow queries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): skip exemplars when source writes fail

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: account for held memory in batch admission

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: config comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: simplify exemplar and memory tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: deduplicate exemplar setup

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-16 11:55:20 +00:00
Weny Xu fcc1891ba6 feat(mito2): support index plugins in region creation and rebuilds (#10298)
* feat(mito2): validate index plugin options before region creation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito2): support index plugins in rebuilds

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): skip unnecessary SST reads during index rebuilds

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): adapt index rebuilds to shared file metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): keep async flush indexes independent of sketch providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): finish flush setup before seeding historical indexes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve index rebuild scope and published extensions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito2): remove speculative flush index guard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve existing indexes on incomplete rebuilds

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): separate background indexing from sketch maintenance

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 11:35:52 +00:00
jeremyhi dd941a8e33 fix(mito2): wait for a stopped region's checkpoint before reopening it (#10376)
* fix(mito2): wait for a stopped region's checkpoint before reopening it

A region's manifest checkpoint runs as a background task that writes the
checkpoint, then `_last_checkpoint`, then deletes older manifest files:
the deltas up to the previous checkpoint, or every older delta at the
region's first checkpoint. Stopping the manifest manager does not wait
for it, so a new instance of the region could read the manifest while the
old task still wrote it. On a store that writes files in place the open
could read an empty `_last_checkpoint`. On any store, an open that found
no checkpoint could have deltas it had just listed deleted by the first
checkpoint's cleanup.

Wait for the stopped instance's checkpoint at engine stop, after the
worker loop ends, and before a fenced region is reloaded. The wait does
not hold the manifest lock, which the worker may take meanwhile: the
checkpoint task owns the sender of a watch channel, which closes however
the task ends.

Fixes the flaky test_change_region_compaction_window (#7135).

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover the checkpoint wait through cleanup, panic and abort

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): align checkpoint wait names and log the wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover concurrent and late waits on the checkpoint

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): keep two checkpoint waiters through the end of the task

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a wait after the checkpoint ended does not start waiting

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): address review on the stopped checkpoint wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the stop contract and assert it before the checkpoint wait

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the checkpoint completion future and keep its task state together

Rename `pending_checkpoint_done` to `pending_checkpoint_completion` on
`Checkpointer` and `RegionManifestManager`, and state how it differs from
`wait_for_pending_checkpoint`. Keep the task handle and its completion
receiver in one `CheckpointTask`, so reaping the handle also drops the
receiver. Use plain backticks in the `stop` doc, which is public and
cannot link to a crate-private method.

Narrow the comment in `RegionWorkerLoop::clean`: a datanode or standalone
shutdown closes every region before it stops the engines, so there the
wait only covers regions whose close failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-16 10:52:27 +00:00
discord9 996b7e9e8d docs(promql): trim design-narration comments in native planner and runtime (#10421)
Comment-only cleanup on top of the segmentation move (#10418) and the topology it carries (#10321).
2026-11-16 17:40:22 +08:00
discord9 9948a163c1 refactor(mito2): drop the unreachable samples builder and the unused prepared-series contract (#10430)
* refactor(mito2): drop the unreachable per-SST samples builder

Production only ever constructs the columnar samples format: SeriesFormats::from_config is
reached from with_format with experimental_series_index_format, where Legacy yields no
samples backend and Columnar yields ColumnarSamplesFormat. The other SamplesFormat impl is
cfg(test). So the per-SST branch of the maintenance task could never run, and build_samples
had no caller but its own test.

The daily columnar body is now unconditional, and the former branch position holds an
explicit UnsupportedOperation error so an unknown backend fails loudly instead of building
nothing. The two fake-format lifecycle tests were duplicates of
real_samples_reconcile_reopen_and_retire, which drives the same build/reuse/reopen/
catalog-failure/purge scenario against the real format; the abort test moved to
build_sample_windows, and a new test pins the refusal.

The compatibility fixtures are untouched and still pass.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): cover retirement after a successful samples rebuild

format_skeleton_build_reopen_and_retire was the only test of the retirement that follows a
successful rebuild; real_samples_reconcile_reopen_and_retire covered the failed-publish path
only. Removing the samples loop's mark_deleted() from publish_index_version left every other
series_index and series_scan test green, which is what made the gap invisible.

The real-format test now also rebuilds a day by deleting one of its objects, checks that the
superseded generation is no longer in the published snapshot, that nothing is purged while the
old snapshot and its handle are alive, and that its objects are purged once they are dropped.
With mark_deleted() commented out, all four cases fail on that last assertion.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): pin the purge to the last reader handle, not the snapshot

Dropping the published snapshot was allowed to release a superseded generation as long as
one of its handles was still held elsewhere, which no test covered: the retirement assertions
only checked that nothing was purged while the snapshot lived, then dropped every handle at
once.

The test now holds a reader handle on every generation the superseded snapshot published
before releasing it, so that release must purge nothing, and asserts that between the two
drops. A rebuild replaces several generations of a source group at once, so holding only the
one the test tracks would let its siblings be released legitimately and the assertion would
fail for a reason unrelated to the lifecycle under test.

A mutation that purges on snapshot drop -- ignoring live reader handles -- fails this new
assertion in all four cases and none of the later ones.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 17:38:58 +08:00
discord9 a1627ee1c4 refactor(promql): share the extrapolation factor between float and histogram windows (#10425)
* refactor(promql): share the extrapolation factor between float and histogram windows

A native histogram counter can carry a start timestamp inside its window, which the
histogram range path turned into a synthetic zero and extrapolated from with its own
inline millisecond-domain formula. The float path already computed the same factor in
extrapolation_factor, so the two routes could drift apart: the histogram branch counted
the synthetic zero as an extra sample interval and extrapolated towards zero from a
negative first count.

The factor now takes an ExtrapolationSpan with two anchors -- sampled_from, where the
increase was sampled, and average_from, where the average sample distance is measured.
A float window sets both to its first sample; a histogram window starts the sampled
interval at the synthetic zero and keeps the average on the real samples. A synthetic
zero is the counter's zero point, so it is expressed as zero_point = Some(0.0), which
reproduces the removed duration_to_start = 0.0 exactly.

The sampled_interval == 0 divergence is deliberate and documented: a float window keeps
its increase with a factor of 1, while the histogram window answers no value, since its
buckets are all zero and an Inf or NaN rate would only poison its consumer.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): make the two-anchor extrapolation tests able to fail

Each of these could only pass: the route-equality table gave both fillings the same anchors,
and the synthetic-zero geometry kept its ends under the threshold with the average anchor
taken either way, so an anchor swap answered the same 1.375.

The table now also carries a window whose anchors differ, pinned to 1.5 for the histogram
filling and 1.75 for the float one, so collapsing the sampled anchor onto the average fails.
The synthetic-zero test uses zero 1000, samples 2000/3000, eval 4500, range 4000: correct is
1.25, and measuring the average from the zero gives 1.75. Both numbers are asserted, and a
temporary anchor swap was confirmed to fail the test before it was reverted.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): give the synthetic-zero range test anchors it can tell apart

The 1.375 geometry answered the same with the average anchor measured from the synthetic
zero, so the end-to-end test agreed with itself either way. It now uses zero 1000, samples
2000/3000, eval 4500, range 4000: the count is (4 - 2) + 2 = 4, the factor 1.25, so an
increase answers 5.0 and a rate 1.25, while a swapped anchor answers 7.0 and 1.75.

The swap was applied to the production call site temporarily and failed the test with
"Increase over two samples: 7 vs 5" before it was reverted.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): give the two-anchor window geometry that can fail both ways

The previous geometry had its end inside both thresholds, so an average interval read
from the sampled anchor answered the same 1.5 as the correct one; only a sampled interval
read from the average anchor changed the answer.

With zero point 2000, average 2500 and last 4500 the end is past the correct threshold
(2.2s) and inside the wrong one (2.75s), so it truncates one way and not the other: the
correct answer is 1.4, an average anchor read from sampled gives 2.0, and a sampled anchor
read from average gives 1.5. Both mutants are pinned as assertions, and both were applied
to the production formula to confirm this test fails on them.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 17:38:48 +08:00
discord9 2e4141e77d refactor(promql): move the segmentation engine into the promql crate (#10418)
Lands the segmentation-engine move together with the uniform native-PromQL topology it builds on (#10321), which this branch carries as its base.
2026-11-16 17:36:28 +08:00
discord9 94ec1a5a17 chore(promql): drop production-dead state and sample helpers (#10408)
- native/state.rs: remove ensure_bound_state and its only call site, plus
  the now-dead state_kind/state_components helpers (about 60 lines).
- instant_eval.rs: remove the strictly-increasing re-check in
  select_instants; RawTimestamps::try_new already enforces the order.
- native/split.rs: the Window arm declines on input_grid.is_none(); the
  declared-vs-input grid equality is already enforced when the window is
  bound (native/plan.rs).
- datatypes/series/samples.rs: remove Samples::into_dense and its
  arrow_result helper; only datatypes tests called it. The eight
  densification tests existed only to exercise it and go with it.

Verified: promql 1209/1209, datatypes 402/402, clippy -D warnings clean.

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 17:36:15 +08:00
discord9 b136da871f test(promql): stop deriving budget expectations from the estimator (#10409)
About fourteen accounting/reservation tests computed their expected figure,
pool limit, or 'exactly fits' budget from operand_chunk_bytes /
evaluated_bytes or a copied row-size constant, so a wrong formula stayed
green. They now assert an independent behavioural property instead: a
charge appears and is released to zero, an oversized operand is refused by
the pool, a refused chunk leaves nothing reserved, or a bound stated in
the bytes the fixture's own data holds.

Ablation proof: temporarily doubling operand_chunk_bytes now fails 9
tests instead of 3, so these tests can actually catch a wrong formula.
The ablation was reverted; promql 1209/1209 green, clippy clean.

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 17:17:50 +08:00
Weny Xu 89e14ab936 fix(mito2): cancel compaction merges at batch boundaries (#10411)
* fix(mito2): cancel compaction merges at batch boundaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve upload errors and expose merge cancellation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): normalize encoder cancellation at SST writer boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(mito2): use imports for SST write types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): share cooperative upload cancellation with tokens

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): classify nested upload cancellation errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-16 08:32:11 +00:00
discord9 5517c4b76f refactor(promql): let the legacy histogram path read the native kernel (#10420)
* refactor(promql): read the legacy histogram estimators from the native kernel

`prom_native_histogram_quantile` and `prom_native_histogram_fraction` estimated from
`common_query`'s `NativeHistogram` bucket walk: exponential bucket bounds computed as `2^x`, and a
forward walk from the lowest bucket only. The native kernel in `histogram_eval` walks the same
histogram the way Prometheus does -- the bound table in `bounds`, and a walk from the highest bucket
down once a finite-sum quantile reaches the median -- so one histogram had two answers that differ
in their last places.

Read the `NativeHistogram` through the borrowed `HistogramView` both representations share and
return the kernel's answer, mapping its outcome back onto the NaN-observation notes the legacy UDFs
record. Their signatures and the notes are unchanged; only the arithmetic behind them moves.

The `2^x` estimator in `common_query::native_histogram` stays for that crate's public API and is
annotated as such: no PromQL path reaches it any more.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): delegate the legacy histogram arithmetic to the native kernel

`prom_native_histogram_add`, `_sub`, `_mul_scalar`, `_scalar_mul`, `_div_scalar`, `_neg`, the
sum/avg accumulators, `delta`/`idelta`/`rate`/`increase`, and the weighted mean that merges partial
states all scaled and combined `NativeHistogram`s with that type's own reconciliation: its own
zero-bucket widening, schema reduction and custom-bound intersection, over `2^x` bounds.
`FloatHistogram::add`/`sub`/`mul`/`div` already implement Prometheus's rules (`addBuckets`,
`reconcileZeroBuckets`, `CopyToSchema`, `intersectCustomBucketBounds`) and the native kernel's
golden tests cover them, so this keeps one implementation of the semantics.

`AddOutcome` folds into the existing semantics rather than being dropped: `custom_bounds_reconciled`
is the intersection the kernel keeps in `custom_values`, which the callers already announce from the
operands, and `counter_reset_collision` is the contradiction the kernel answers with the unknown
hint the legacy arithmetic derived from two hints that disagree; both are asserted in debug builds.
The public `NativeHistogram` signatures are unchanged, the sums are compacted as before so
`promql_eq` and the serialized spans keep their answers, and unary minus goes through the kernel's
multiplication like the other scalings.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the legacy delegation's observable answers

Comparing the bridge against the kernel it calls only shows they agree today; it cannot show
what either answer means. These tests pin the answers themselves.

- quantile with a NaN sum over several buckets answers 32.0 with NaNSkew, because the NaN walk
  leaves the last bucket for the interpolation; below the rank it answers NaN with NaNResult.
- an exact tie answers 1.0 through the legacy walk and 2.0 through the kernel, on a fixture with
  an empty bucket between the two, and neither records a note.
- assert_same_payload now compares every field, including the start timestamp policy, custom
  values, the complete span segmentation and the bucket bits.
- round-tripping preserves reset hints, custom values, zero-length spans and gaps.
- add/sub reconcile the reset hint and the custom bounds, compact cancelled buckets, and keep
  promql_eq's expanded-index equality.
- scale/divide_by keep the field-by-field fallback for an unreadable layout -- mismatched counts,
  a backwards span, an overflowing index, a negative length -- and preserve its start timestamp;
  add/sub answer None for the same layouts. Zero, negative and NaN factors are pinned.
- rate/increase/delta/idelta/irate clear the start timestamp of a derived histogram, while a
  single-sample sum aggregate keeps its stored one.

A layout whose later span steps backwards is refused before the kernel sees it: the legacy
bucket_indices rejected it, while side_from_view would have accepted repeated indices.

Signed-off-by: discord9 <discord9@163.com>

* test(http): pin the legacy histogram route's rate, NaN quantile and derived windows

The delegation changes what three kinds of well-formed data answer, so assert it where a user
sees it. Every request runs without the native hint and asserts !executes_fused_series on the
observed plan, which proves the legacy route rather than assuming it from the missing hint.

- rate and increase over a raw sample with a start timestamp inside the window still extrapolate
  from the synthetic zero: count 8/3 and sum 10/3 for a rate, 8 and 10 for an increase, with no
  infos or warnings.
- histogram_quantile(0.5) over schema 0, buckets [8, 1], count 10 and a NaN sum answers 32.0 and
  reports the NaN-observation info, matching Prometheus's walk that leaves the last bucket for
  the interpolation. The previous estimator answered 2^-0.375.
- rate over a raw window answers one series while the same window over h*1 answers none: the
  derived sample carries no start timestamp, so it has no synthetic zero to extrapolate from.
  Both phases are covered, memtable and flushed SST.

The native-histogram test now installs the route interceptor the way its sibling tests do, so the
plan can be observed; all of its original assertions are unchanged.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 16:26:52 +08:00
discord9 50b6b5bf82 feat(promql): answer max_of and min_of over two scalars natively (#10431)
* feat(promql): answer max_of and min_of over two scalars natively

Prometheus defines both as two scalars in, one scalar out (experimental), and the pinned
promql-parser carries the same signature, so an instant-vector argument is already a parse
error. The ScalarFunction kernel implementing Go's math.Max/math.Min -- an infinity of the
function's own sign beats NaN, NaN beats any other value, +0 beats -0 -- existed with its
golden cases but was never reachable from a query.

The call now binds through the existing parameterized-map tower: compile_call ->
scalar_function_call -> ParameterizedOp::MaxOf/MinOf -> ParameterizedMath -> PointMath::Scalar
-> ScalarFunction::apply, executed by map_math_with_parameters. Only MaxOf/MinOf take the
scalar-in/scalar-out shape; round and clamp* keep their vector contract, and clamp's
max < min emptiness rule moved into PointMath verbatim. A constant first argument is folded
and materialized as the one chunk it stands for, so max_of(1, 2) answers 2 without reading
input, and two scalars on different grids are refused rather than answered wrongly.

The 64-case golden now runs through NativeExecutor for both functions instead of the kernel
alone, and the by-position pairing test was un-ignored and pointed at the parameterized path.
map_scalar_function, whose only caller was that test, is removed.

The legacy planner is untouched: neither name was implemented there.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): keep the call dispatch's comment beside the check it describes

Signed-off-by: discord9 <discord9@163.com>

* chore(compat): move the from-versions window to v1.3.1

v1.3.1 has published release assets, so the compatibility check asks for it and fails on
every branch that still names v1.3.0.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 14:58:05 +08:00
discord9 9b627fd861 refactor(promql): drop four kernel entries the native runtime replaced (#10429)
* refactor(promql): drop four kernel entries the native runtime replaced

Each of these had no caller left outside tests, and each has a live replacement the
runtime already uses. The golden cases move to that replacement rather than being deleted
with the entry, so the inputs they covered are still covered.

FloatMerger::merge/merge_budgeted/merge_series -> merge_row, which reads a StateRow
instead of a PartialAggregate. The merge tests now encode partials into rows and drive
merge_row, keeping the fold order, the shuffled partition orders, the NaN and signed-zero
cases and the admission-before-insert assertions. One test goes away: the spare capacity of
an input partial's nested Vec has no counterpart in a row, whose memory is charged as Arrow
buffers by the caller and whose shared parent buffer the router charges separately.

map_negate -> -1 * x, which is how the compiler already lowers unary minus. NaN bit
patterns differ between negation and multiplication (the sign and quiet bits are not
specified by IEEE), so the assertion is that the answer is still a NaN carrying its payload;
staleness is decided at instant evaluation and a Map's holes are validity bits, so neither
reads that payload. Prometheus's neg cases pass unchanged.

map_compare -> map_compare_literal/map_compare_series, which is what production already
called. The histogram cases now assert that kernel's existing drop-and-report behaviour
instead of the dead entry's whole-query error; production behaviour is unchanged.

scalar_arithmetic/scalar_compare -> the compile-time constant folding the compiler
performs, asserted through constant_series.

ScalarFunction and map_scalar_function stay: removing them depends on the parameterized
math variants another change adds.

Signed-off-by: discord9 <discord9@163.com>

* chore(compat): move the from-versions window to v1.3.1

v1.3.1 has published release assets, so the compatibility check asks for it and fails on
every branch that still names v1.3.0.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): spell the state kind out in the merge-row encoder

The encoder stands in for the executor that hands merge_row its input, and it was calling
native::state::state_kind to name the kind. #10408 deletes that helper along with the other
production-dead items in its scope, so depending on it makes these two changes fail to
compile when they land together. The mapping is eight lines and belongs to the encoder
anyway: it is the one place that has to agree with what the runtime would derive.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 14:57:54 +08:00
discord9 44973dba08 refactor: drop two helpers whose production callers are gone (#10427)
* refactor(promql): drop the uncalled merge_exact_series helper

The helper carried #[allow(dead_code)] and a comment saying the fused series stack would
call it once that caller landed on this branch. No caller exists: no references in src,
tests-integration, tests-fuzz or tests, and no re-export. Its only dependency,
merge_same_labelset, is a shared grid_series helper with other users and stays.

Signed-off-by: discord9 <discord9@163.com>

* refactor(datatypes): drop StateFrameRow::to_state_series

Only three promql tests called it, and each asserted frame content -- labels, grid length,
per-point state values and presence -- after the conversion rather than the conversion
itself. They now read the row's fields() and time axis directly, so the unpack and
declaration-validation path is still exercised and no assertion was dropped.

The private state_float helper went with it; StateSeries itself stays, still produced by
pack_aggregate_state and the repartition fixtures.

Signed-off-by: discord9 <discord9@163.com>

* chore(compat): move the from-versions window to v1.3.1

v1.3.1 has published release assets, so the compatibility check asks for it and fails on
every branch that still names v1.3.0.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 14:57:43 +08:00
discord9 a1be266c3c fix(promql): ignore histogram samples in scalar() and share one kernel (#10426)
* fix(promql): ignore histogram samples in scalar() and share one kernel

Prometheus' funcScalar counts only float samples: histogram samples are ignored silently,
so they neither supply a value nor make a point hold more than one element. The select
kernel rejected them instead, while the native ScalarOf runtime kept its own inline loop
that ignored them, so scalar(histogram_metric) answered differently depending on the
route.

The kernel now ignores histogram samples and the runtime calls it, deleting the inline
copy. A point with one float answers that float, with only histograms or with several
floats answers the Go NaN.

The kernel builds its answer as one dense run over the grid: a Vec<f64> seeded with the
Go NaN, where the first float at a point writes its value and a second writes NaN back,
plus the counts the caller already reserves. That buffer becomes the result's Float64Array
by move, so the per-point peak is the 8 bytes of values plus 1 of counts the reservation
already counts. Staging through Vec<Option<f64>> and GridSeriesBuilder would have held
about 33 bytes per point against a 9-byte reservation.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): resolve the evaluation grid once in scalar()

grid_points already resolves and validates the grid, so asking for the schema's grid again
right after could only disagree with what was just checked.

Signed-off-by: discord9 <discord9@163.com>

* chore(compat): move the from-versions window to v1.3.1

v1.3.1 has published release assets, so the compatibility check asks for it and fails on
every branch that still names v1.3.0.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 14:57:32 +08:00
discord9 aa9c616fe5 perf(promql): route proven per-series chains disjointly and forward explicit child frames (#10413)
perf(promql): keep per-series provenance and let identity-preserving windows route disjointly

RawSeriesSourceExec re-derived per_series_scan by downcasting the child
directly, so EnsureCooperative's wrapper silently dropped the proof and
the root router fell back to MergeInputs. Peel CooperativeExec before
identifying SeriesScanExec, and check the child's bound tags cover the
primary key the same way a bare scan is checked.

FusedSeriesExec now accepts a RawSeriesSourceExec input, and a Window
whose function keeps the metric name and whose child preserves
partition identity, so last_over_time/first_over_time over a per-series
scan can publish each identity as soon as its chain closes.

When the child already publishes this node's frame, RawSeriesSourceExec
forwards it 1:1 instead of unpack→relabel→pack.

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 13:20:59 +08:00
discord9 3a032fbbc6 test(query): pin the native PromQL option default to off (#10371)
The native route is gated by two halves: the engine-wide
`experimental_native_promql` option and the per-request
`query.experimental_native_promql` extension. The extension half is already
covered by the `native_promql_requested_*` tests in
`query::promql::planner::native`, and the plan-shape no-op is covered by
`SeriesFrameOutput`'s `a_plan_without_a_frame_declaration_is_untouched`.

What was not pinned is the default of the engine option itself. Flipping that
`false` to `true` would expose the native route to every query that sets the
extension, and no test would fail. Assert it directly so the opt-in property
cannot regress silently.

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 13:09:44 +08:00
discord9 d0324193fe perf(promql): avoid two copies on the native binary path (#10338)
* perf(promql): write single-field binary results in one pass

A merge of one field has nothing to merge: the field filed every label
set's rows already sorted by point, and already refused two rows at one
point. Writing them straight into the builder drops the per-label-set
map, the triples table, its sort and the row buffer that the multi-field
merge needs, and bills the same columns that merge bills.

Profile shares of one 20s workload on a six-thousand-series shape put
merge_fields at 6.1% of samples before and 3.2% after; the query result
is byte-identical. Wall-clock comparison on that host could not resolve
the difference: session drift there is larger than the effect.

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): move a chunk into its only runtime input

One source feeding one runtime input is the common shape, and the chunk
was cloned for it and then dropped. Moving it removes that copy. Several
inputs still clone per input, and a producer state batch stays refused
either way: it carries the charge for its whole payload and belongs to a
different handover.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 12:53:40 +08:00
discord9 b49f542b25 fix(promql): declare the per-row name column on every native rows answer
A native rows result now always declares `__name__`, a nullable `Utf8` column appended after the
labels and marked as the metric name, and each row carries the name of the series it was read from.
A row whose value is computed rather than read (a math kernel, a folding aggregate, a range
function) carries null there.

The name is part of what the statement answers, not of how a response is displayed: the walk that
derives a node's tags reports the name of a discovered source and of an `or` unconditionally, so
`sum by (__name__)` groups on the identity of the series rather than collapsing every series into
one group.

`with_row_name_column` appends the column when the root is a samples root. It moves an existing name
column to the end rather than adding a second one, because neither a set operation nor a discovered
source leaves an existing name last: `or` keeps the order of the left operand and appends only the
tags of the right side it has not seen, while a discovered source sorts its tags, which puts the
name wherever its bytes fall. Every rows layout then answers one order - the time index, the value
fields, the labels, then the name. A `label_replace(..., "__name__", ...)` result is therefore not
declared twice, and a scalar root keeps its two columns. The collision check runs after the column
is appended, so a name that would clash with a label still declines the layout, and a split's
terminal schema declares the column while the frame producers keep the tags they were compiled with.

A CTE answered natively publishes the column too, after the existing planner's columns and only when
the answer declares one: `publish_legacy_schema` keeps the metadata-carrying name column out of its
type-based pairing, so a label cannot claim it, and validates the published schema column by column.

`abs(a or b)` now reports no `__name__` at all instead of the statement's name on every row: the
operator drops the name, so the declared column's values are null. Shapes that keep the name (plain
selection, filtering comparison, `or`, `topk`/`sort`, `first/last_over_time`, `sum by (__name__)`)
carry it per row.

Signed-off-by: discord9 <discord9@163.com>
2026-11-16 12:05:59 +08:00
Ning SunandLei, HUANG b30e002cef feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN (#10092)
* feat(postgres): Prometheus Remote Write protobuf ingest via COPY FROM STDIN

COPY FROM STDIN WITH (FORMAT promwrite [, version 2] [, <options>])
ingests one Prometheus Remote Write request body over the postgres wire
protocol — a GreptimeDB-specific, binary COPY format (wire format code
1). The payload is byte-for-byte the body a remote-write client POSTs
to /v1/prometheus/write, compression included: buffered until CopyDone
and completed through the exact ingestion core that endpoint runs.

To share that core, the HTTP handlers' ingestion bodies are extracted
into axum-free functions (ingest_remote_write_v1/v2 plus the context
extension stamping), and the HTTP routes become thin wrappers — their
behavior is unchanged, and the copy layer runs the identical decode,
validation, conversion, batched/prepared write paths, per-batch
write-progress accounting and context extensions, under the
connection's regular postgres context.

The WITH options carry what the HTTP endpoint takes as its
content-type/-encoding and query parameters; session-provided values
(database, user) are not repeated:
- version 1|2 (default 1), matching the content-type's proto
- zstd, matching Content-Encoding: zstd (snappy by default)
- physical_table, matching the ?physical_table= parameter
- pipeline_name/pipeline_version, matching the pipeline headers
  (v1 only; for v2 they are accepted and ignored, exactly like the
  HTTP route ignores them)

Table-less like the other ingestion formats; one COPY statement is one
bounded request (shared 64 MiB limit); the row count reports written
rows (v1) or written samples plus histograms (v2). The flush enforces
the frontend's shared request-memory quota (max_in_flight_write_bytes)
— the same limiter the HTTP ingestion protocols use, so COPY cannot
bypass the aggregate budget. Wiring: PostgresServer::with_prom_remote_write
carries the handler, metric-engine flag, validation mode and shared
memory limiter.

e2e: v1 body chunked across CopyData boundaries; v2 samples with
readback; physical_table option routing; garbage-body decode failure
with connection reuse; a tiny-quota regression where a compressed body
whose decompressed size exceeds the budget fails like the HTTP route.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(servers): propagate COPY body reservations and test write accounting

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(postgres): share Prometheus logical batching with COPY ingestion

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(postgres): narrow Prometheus COPY scope and share write outcomes

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2026-11-16 02:45:34 +00:00
Jingbo ChaiandClaude Opus 5.5 7e177b06fe fix(telemetry): avoid spurious log directory error on first start (#10446)
* fix(telemetry): avoid spurious log directory error on first start

Since 0.2.4, tracing-appender prunes old log files when an appender
with max_log_files is built, and it lists the log directory before
creating it. When the directory does not exist yet, it prints "Error
reading the log directory/files" to stderr. GreptimeDB sets
max_log_files by default, so every start with a missing log directory
printed this line.

Create the log directory in build_file_appender before building the
appender. Creation errors are left to the appender build, which
reports them as before. Add a test that checks the stderr of a child
process.

Works around https://github.com/tokio-rs/tracing/issues/3612.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(telemetry): drop the first-start log directory test

Review feedback: the test is not needed for such a minor change.
Restore the unix-only test imports that it had merged.

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Signed-off-by: shuxiachai <shuxiachai@163.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 01:38:28 +00:00
dayan02devanddennis zhuang dbd080b840 fix(recordbatch): treat -0.0 and +0.0 as equal in SimpleFilterEvaluator (#10443)
* test(recordbatch): compare float zero predicates with DataFusion

`SimpleFilterEvaluator` filters SST rows exactly, so it has to agree
with the SQL predicate that DataFusion evaluates above the scan.
DataFusion compares -0.0 and +0.0 as equal. Evaluate comparisons
against both zeros, NaN and NULL in Float64 and Float32 columns, and
check the selected rows against DataFusion's filter.

On main this fails: `v = Float64(0)` selects only the +0.0 row, while
DataFusion selects both zeros.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* fix(recordbatch): compare -0.0 and +0.0 as equal in simple filters

DataFusion 55 normalizes -0.0 to +0.0 before Arrow's comparison
kernels, so SQL compares the two zeros as equal. SimpleFilterEvaluator
called the kernels directly, and for floats they compare bit patterns:
-0.0 != +0.0 and -0.0 < +0.0. Mito applies the evaluator to SST rows as
an exact filter, so after a flush `v = 0.0`, `v >= 0` and `v IN (0.0)`
dropped the rows holding -0.0, and `v = -0.0` dropped the rows holding
+0.0. Rows still in a memtable matched.

Normalize the literal when the predicate is built and the input before
it is evaluated, with the helpers DataFusion uses for the same purpose.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

* test(sqlness): cover flushed float rows holding -0.0

Flush rows holding -0.0, 0.0 and 1.0, then filter with `= 0.0`,
`>= 0`, `= -0.0` and `IN (0.0)`. Both zeros have to match every one of
them, as they do while the rows are still in the memtable.

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>

---------

Signed-off-by: dayan02dev <pawarudayan92@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
2026-11-16 00:43:04 +00:00
Yingwen 1dbc0cd80f fix(mito2): avoid unrelated sparse tag expansion during compaction (#10414)
* fix(mito2): avoid unrelated sparse tag expansion during compaction

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): cover sparse compaction filter and compatibility boundaries

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): handle projected fields in sparse compaction mapper

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): remove redundant compaction projection mapper

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-15 02:34:57 +00:00
Weny Xu beb1f88696 feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): release safe WAL history when closing regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover bounded WAL release across close and restart

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): clamp bounded releases to durable entries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): observe close release failures and cover recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: describe bounded WAL release lifecycle

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): make bounded release fence sealing deterministic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): bound queued releases and ignore shutdown hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): cover close release persistence and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): wait for failed close flush cleanup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify bounded WAL release backpressure and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(mito2): clarify staging test retry invariant

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): back off durable release retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): drive release retry after backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): share and verify WAL retry backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify retained WAL release retry semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 12:44:46 +00:00
discord9 9018736d37 perf(mito-codec): index sparse label columns by id once at bind time (#10419)
The sparse-label audit walked the physical primary key once per label:
column() and position() were linear scans over the described columns,
so decoding a key with N labels over M primary-key columns cost O(N·M)
per chunk. Build an id→encoding-position index once in bind() and use
it for both lookups, so the cost drops to O(N). The full structural
audit still runs before any value is read, and error text and
error-selection order are unchanged.

Signed-off-by: discord9 <discord9@163.com>
2026-11-13 19:10:41 +08:00
Weny Xu 68145fc727 fix(mito2): coordinate region edits with repartition lifecycle requests (#10354)
* fix(mito2): coordinate region edits with repartition requests

Extend the pending ALTER queue to defer lifecycle requests blocked by region edits. Preserve scheduler handoffs and drain accepted edits before entering staging, rejecting new unbound imports at that boundary.

Keep ApplyStagingManifest's asynchronous execution and close-time flush behavior. Cover queue bounds, failures, staging transitions, close/reload races, and flush concurrency.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): settle region waiters before completion returns

Reject pending lifecycle requests when retiring a region. Give each completion one dispatch owner while preserving stalled-write ordering, edit-before-compaction dispatch, and fenced-region reload handling.

Cover edit retirement with queued requests; the regression times out when removal only rejects edits.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): preserve queued close flush requirements

Defer later closes behind a pending close even when they do not request a flush themselves. Reuse the existing close-time flush and waiter handoff.

Cover ordinary WAL writes with mixed flush_on_close values and verify committed SSTs before reopening.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito2): clarify region request scheduling names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): retain lifecycle integration after rebase

Restore staging-session checks and bound test edits from the prior merge resolution. Keep StagingEnded dispatch and classify upstream tail replay notifications without adding lifecycle ownership.

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 08:39:06 +00:00
Weny Xu 643bdd72db fix(metric-engine): reuse existing column ids when adding columns is retried (#10372)
The metric engine derives the columns to add from the cached physical
schema, and `DataRegion::assemble_alter_request` assigned every given
column `max(non-reserved column id) + 1` without checking the existing
names. That cache can lag behind: Mito reads its in-memory metadata
without going through the region worker, and a caller future dropped on a
client timeout does not withdraw the request that is already queued. A
retry then treats an installed column as new and Mito rejects it, because
the name exists with a different id.

Recheck the real metadata in `assemble_alter_request`, validate an
existing column with `validate_shared_column` and drop it from the
payload, assign ids only to genuinely new columns, and skip the alter when
nothing is new. Serialize the existence check, the id allocation and the
physical alter per physical data region: the mutex lives in
`PhysicalRegionState` and is held by the metric engine orchestration, so
concurrent requests on logical regions sharing the region cannot allocate
one id to different columns. `DataRegion` stays a stateless adapter.

Build the logical mapping and the cached physical columns from the
metadata read after the alter, and report a missing column as an error
instead of panicking.

Signed-off-by: Wenyu Xu <wenymedia@gmail.com>
2026-11-13 08:30:11 +00:00
dennis zhuang 6e025eef61 fix(auth): secure MySQL and PostgreSQL database selection (#10382)
* fix(mysql): authorize the default database at login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): validate MySQL handshake scope and SQL USE

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): authorize PostgreSQL schema selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): defer PostgreSQL database lookup until authorized

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(servers): propagate catalog lookup errors in protocol tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(postgres): skip unusable search path candidates

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): log database authorization and selection outcomes

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): preserve search path responses and schema boundaries

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): clarify database selection diagnostics

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): format handshake username in structured logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): secure database-less sessions and failed startup

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mysql): retain authorized default database selection

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(auth): separate database selection from request dispatch

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(mysql): align schema validation with database-less login

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(auth): explain database-less session restrictions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 08:07:57 +00:00
discord9 f88af153ec feat(query): serialize remote dynamic membership (#10341)
* feat(query): serialize remote dynamic membership

Recover single-column UTF-8 build-key membership from InList and HashTableLookupExpr producers and send it to remote datanodes using the existing PhysicalInListNode. Non-monotone or unsupported subtrees widen to true; the receiver keeps the live filter authoritative.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): send lookup membership as a key list

Serialize a HashTableLookupExpr as the existing PhysicalInListNode and preserve the rest of the expression tree. Remove per-subtree monotone and true rewrites; only an unrepresentable lookup or an over-budget payload falls back at the whole-payload level.

Signed-off-by: discord9 <discord9@163.com>

* docs: tighten narrow membership contract

Clarify that the key-list transport is a positive pruning hint, not universal Boolean equivalence, and test that a NULL probe survives the IS NULL wrapper.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): retain safe bounds on membership fallback

When a key-list predicate cannot be shipped, keep top-level conjuncts that contain no membership and re-encode their conjunction. Drop membership-bearing conjuncts whole; fall back to true only when no safe bounds remain.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): skip unrepresentable dynamic filter snapshots

Do not synthesize a true predicate when a runtime hash-table lookup has no wire representation. Capture and fanout skip the affected snapshot, keep the query running, and report PushedDown::No so the parent filter is retained. Safe membership-free conjuncts still travel, and a producer-supplied true remains encodable.

Cover restrictive-to-true updates, unsupported initial registrations with later supported updates, final unsupported cleanup, and NULL build keys.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): clarify 642-key payload budget comment

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): tighten dynamic membership encoding

Preserve static IN conjuncts when only a runtime lookup is unrepresentable, apply the IN-list exclusion only on the over-budget path, and derive the membership count guard from the byte budget before materializing literals. Document that decoded dynamic filters are pruning-only, fix the NOT/CASE contract wording, deduplicate test helpers, and justify the schema-less InList proto encoding.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): reset stale dynamic filters on unsupported updates

Publish an explicit true reset when a producer generation cannot be carried, instead of leaving an older narrower predicate that could prune rows after widening. Keep the unsupported expression unrewritten; the reset is a transport-built invalidation. Type over-budget build-key memberships as DynFilterPayloadTooLarge, accept scalar-subquery IN lists under the conjunct-retention contract, and clarify the no-initial-predicate wording.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): scope no-true rule to the encoder

Clarify that the encoder never substitutes true for a producer node, while the long-lived fanout path publishes an explicit true reset to invalidate a stale region predicate. Point at the fanout caller instead of claiming producers always skip refused generations.

Signed-off-by: discord9 <discord9@163.com>

* docs(query): correct reset wording

Fix the inverted conjunct-retention sentence and describe initial capture versus ongoing fanout resets consistently in the error docs.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-13 07:51:06 +00:00
Weny Xu 7eb766c9e4 feat(common-meta): make RDS metadata backends closable (#10417)
* feat(common-meta): make RDS metadata backends closable

Standalone closes the metadata backend on shutdown, and
`MetadataKvBackendCreator` now returns a `ClosableKvBackendRef`. Only the
raft-engine backend implemented `ClosableKvBackend`, so external metadata
backends could not be wired into the standalone creator.

Implement `ClosableKvBackend` for `RdsStore` and return
`ClosableKvBackendRef` from `with_mysql_pool`/`with_pg_pool` and the
metasrv RDS builders. The stores only hold reference-counted connection
pools that are released with the last backend handle, so `close()` has
nothing to release explicitly.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): close RDS connection pools on shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(common-meta): preserve closable RDS constructors and test shutdown semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 07:18:56 +00:00
Weny Xu 180dfc68f7 perf(mito): reduce SST metadata memory overhead (#10356)
* perf(mito): reduce SST metadata memory overhead

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(cache): add a sharded weak interner

Share equal live allocations without exposing lock guards or retaining payloads. Sweep candidate buckets incrementally and cover concurrent publication and weak-reference churn.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): share SST metadata through the sharded interner

Require shared metadata wiring for manifests, canonicalize checkpoint entries without cloning their payloads, and centralize runtime registration. Keep live content variants distinct and verify runtime and recovered allocation identity.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: compile SST metadata consumers across feature sets

Wrap fixture manifest metadata in Arc explicitly and gate the test-only HashMap import with its consuming feature. Validate default mito tests and all cmd targets and features with Clippy warnings denied.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): reclaim retired metadata while idle

Run bounded interner maintenance on the shared blocking runtime without retaining its owner between passes. Shrink sparse indexes and release empty backing storage, preserving live readers and reclaiming their metadata after delayed release.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(mito): use intern-arc for shared SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): expose interner counts and decouple manifest setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(partition): support hashing partition expressions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): hash intern SST metadata without ordering values

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 06:57:23 +00:00
jeremyhi 6faa9ec862 feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store

`LogStore::read_node` streams one region's entries in `(after, upper]` from
the WAL of another node that shares the log store's storage, without writing
to it. The default returns `None`.

The object store WAL implements it with a registry of read-only readers over
its own object store. The node prefix comes from the configured root and the
store's own generation, and the returned stream holds the reader, so reads of
the same node share its cached object indexes. `WalReadUpperBound` moves to
store-api so the trait can name it; log-store re-exports it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): replay pending WAL tails in the background

A leader that is writable, altering or editing and still has pending tails
replays them in one background task per region and publishes the result in
one manifest edit, which opens the read gate and resumes compaction once no
tail remains.

- The worker starts the task when a lease, a catchup or a role change makes
  the region writable, and cancels it on close, drop, downgrade and when it
  replaces the instance. The task always reports its end, so the worker can
  start another replay or reload a region a conflict fenced.
- Each tail is read from this node's WAL, or through `LogStore::read_node`
  for another node, and replayed with the per-entry code of open-time replay
  into memtables outside the region's version. Chunks of about 64 MiB are
  written with the SST half of the flush task, never across two tails, since
  an SST gives its rows its largest sequence.
- The edit adds the SSTs, clears the tails, and advances the flushed entry id
  to the last entry read and the flushed sequence to the largest chunk
  sequence. It commits outside the edit queue, under the manifest lock, only
  while the region is a writable leader recording exactly the replayed
  tails. A busy region or an unknown result retries the same bytes; a
  conflict keeps the chunks; a certain failure deletes them.
- Another node's listed tail is only a bound. After a catchup made the
  region writable, the edit keeps its unread part until the region receives
  a Leader lease, then a second edit clears it.
- After the commit the task releases the WAL up to the new replay start,
  kept below this node's remaining tails, and the worker raises the region's
  last entry id to the new flushed entry id, so a later truncate cannot pull
  it back.
- WAL pressure no longer flushes a region whose WAL release one of this
  node's tails caps.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): harden tail replay across leases, cancellation and uncertain commits

- A Leader lease counts as received after a catchup only if its heartbeat
  was sent after the catchup promotion: the writable fast path does not
  check the start of a downgrade, so a lease granted before an earlier
  downgrade could otherwise clear the mark. The mark is the promotion time,
  so the fast path still reads one atomic.
- A direct promotion counts as a lease.
- The tail commit also raises the committed sequence, so later writes of the
  region take sequences above the replayed rows.
- Registering a writable region starts a replay, which covers a create that
  reopens an existing region.
- Once an attempt may have persisted the edit, later certain failures retry
  the same content instead of building a new edit.
- WAL reads stop when the task is cancelled, and a cancelled task does not
  start another commit attempt.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that closing a region cancels a stalled tail read

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): hold tail commit attempts by number instead of sleeping

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): cancel tail replays when the worker stops

The worker's cleanup closes its remaining regions without the close handler,
so it now cancels every running tail replay first; otherwise a replay stalled
in a WAL read or waiting for a lease outlives the engine. The stalled-read
test covers an engine stop as well as a close. The docs no longer claim that
every SST gives all its rows its largest sequence.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): shorten two tail replay comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): stop tail reads at their upper bound

A tail read over this node's WAL kept reading past the tail's upto until the
log ended, so an unreadable later entry failed a replay that already had every
entry it needed. Stop the read once it reaches upto.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): inline the tail replay notify helper

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that writes during a tail replay win over replayed rows

Whatever records pending tails must raise the region's committed sequence to
the tails' largest sequence, so writes accepted while the replay runs get
larger sequences than the replayed rows. Record this in AGENTS.md and test it
with a write that lands between two commit attempts of the replay.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): bound reads of this node's listed tails

A listed bound may name no entry, so a read of this node's WAL never saw an
entry equal to it and kept polling later objects; one that failed to read
failed the tail. Read every listed tail through `read_node`, which stops at
the bound, and keep reported tails on the node's own WAL.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that rows around a tail replay survive a reopen

The tail commit raises the flushed entry id to the largest entry read, which
can be another node's. Whatever records tails keeps this node's later entry
ids above every tail and leaves none of its unflushed entries below one
outside its own tails. Record this in AGENTS.md, and make the replay test
meet it and reopen the region without a flush afterwards.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject graceful staging entry of a region with pending tails

Entering staging through `set_region_role_state_gracefully` skipped the
pending-tail check that `EnterStaging` runs, so a region with pending tails
could enter staging, end its replay, and return to Leader with nothing
replaying its tails. Run the same check before entering staging there.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): replay tails of a region altered right after promotion

A promotion checked whether it made the region writable after setting the role. If the
worker started an allowed ALTER or edit in between, the check saw Altering or Editing
and asked for no replay, and nothing asked for one later. Check for a leader that a
replay runs on instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover the WAL release and locked checks of tail commits

Check that a tail commit releases the replayed WAL, that it is refused under the manifest
lock once the region was demoted or its tails changed, that a commit retried while
altering keeps its chunks, and that a replay waiting for a lease does not end and start
again.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): note that read_node also reads this node's WAL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep no region version during a tail replay round

A replay round held the region's version until it ended, and with it the memtables the
region had when the round started. A flush removed them from newer versions, but their
write buffer memory was only freed once the round dropped them, so a slow read, a commit
retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails
and the flushed entry id instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): count a direct Leader role set as a lease only when it promotes

A direct role set to Leader counted as a Leader lease even when the region already was a
leader. On a region a catchup promoted, it cleared the mark that keeps another node's
unread listed tail until a lease arrives, so the next commit could drop that tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): move the tail replay leader check onto MitoRegion

The engine and the replay task both use it, next to the other role checks. Its doc now
names the Writable, Altering and Editing states and why staging and downgrading
regions replay nothing.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay waits for a Leader lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin the pressure skip, staging rejection and remote read error

The WAL pressure test gets a case whose own tail starts above the flushed entry id and
still flushes; the graceful staging case checks the rejection; and a remote tail on a log
store that reads no other node's WAL fails as unsupported and keeps its tails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): list every tail replay trigger and the missing admission limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): rename tail replay helpers and notifications

The names now say what each one checks, records or announces: the
replay predicate is a capability rather than a running replay, the
lease hook records a lease, the writable notice is an event like its
siblings, and the commit notice pairs with the end notice.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): ask whether a Leader lease came after a catchup

`is_writable_by_catchup` was negated at both of its call sites. Its
inverse, `leader_lease_received`, is what the replay reads.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): plan the read of a pending tail in the tail replay

`Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task
decides whether a tail comes from this node's own WAL or up to its bound
through `Wal::read_node`, which now takes plain arguments.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay result goes stale

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): send the writable notice through notify_worker

The tail replay already has one helper that sends a background notice
to a region's worker; the writable notice now uses it too, and so logs
a worker that has stopped.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that tail replays skip staging regions and index like flushes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): say why each tail gets its own SSTs and what chunks are for

Ordering between tails needs a tail per SST: an SST can give all its
rows its largest sequence, so an earlier tail sharing an SST with a later
one would take the later tail's sequence and outrank rows that a leader
between the two tails flushed. Within a tail, the 64 MiB chunks only
bound memory: the sequence floor already puts writes accepted during the
replay above every tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the tail replay end guard like other drop guards

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): skip a tail replay edit that a sync already superseded

A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay stops before publishing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover lock waits, busy edits and repeated triggers in tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log every exit that drops a tail replay

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): tell the event listener when a tail replay checks for a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log each tail replay exit once, where it happens

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): make the replacement helper a TestRegion method and fix a lease comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-13 06:32:03 +00:00
Lei, HUANG ed62f8ff54 feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions (#10361)
* feat(function): expose iceberg_hash and iceberg_hash_bounds SQL functions

`iceberg_hash` partition rules could not be evaluated from SQL, so users had no
way to measure a hash distribution or check which bucket a value falls into.
Expose the same UDF that partition rules plan, plus `iceberg_hash_bounds(n)`
which prints the equal-width breakpoints `bi = i32::MIN + floor(i * 2^32 / n)`
as text for copy-paste into `PARTITION ON COLUMNS`.

Both functions are immutable and read-only, and no persisted format changes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): handle untyped NULL and small integer counts

`iceberg_hash(NULL)` arrived as ScalarValue::Null, which becomes a NullArray
that the shared array form had no arm for, so it failed with an unsupported-type
error instead of hashing to NULL. Add a DataType::Null arm in the shared
implementation, so both the SQL function and any future rule evaluation keep the
advertised NULL-to-NULL behavior.

`iceberg_hash_bounds` rejected Int8/Int16/UInt8/UInt16 constants even when the
value was a valid bucket count, because Signature::any leaves the argument width
unchanged. Accept every integer width like the existing Int32/Int64/UInt32/UInt64
arms, and cover them plus the untyped NULL case in unit and SQL tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_breakpoints SQL function

Equal-width `iceberg_hash_bounds(n)` buckets cannot follow a key distribution:
a handful of hot keys can still land in one bucket, and a uniform split keeps
hashing them together. Add `iceberg_hash_breakpoints(v1, v2, ...)`, which takes
any number of constant values in any order (one shared hashable type, exactly
the values a partition rule compares the hashed column against), hashes them
through the shared implementation, and prints them sorted and deduplicated:

    SELECT iceberg_hash_breakpoints('iceberg', 34);
    -- 1210000089, 2017239379

The output drops straight into the breakpoint list of a `PARTITION ON COLUMNS`
rule, so buckets can be placed around known key values instead of equal widths.
Columns and NULLs are rejected with explicit errors: a column cannot be hashed
at planning time, and a NULL hash never matches a comparison bound, so a NULL
value cannot be a breakpoint (NULL rows always fall in the lowest bucket).

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(function): add iceberg_hash_ranges_sql SQL function

The helpers so far print bare breakpoint numbers, so a user still had to expand
them into comparison rules by hand: n buckets mean n rules with up to
2n - 1 `iceberg_hash` call sites. `iceberg_hash_ranges_sql(column, n)` renders
the whole list from the same equal-width breakpoints:

    SELECT iceberg_hash_ranges_sql('host', 2);
    -- iceberg_hash(host) < 0,
    -- iceberg_hash(host) >= 0

The output drops straight into the expression list of `PARTITION ON COLUMNS`
(rule per line). It only renders text; it does not create or change partitions.
The breakpoints come from the shared `IcebergHashBoundsFunction` logic, so the
rendered rules and `iceberg_hash_bounds(n)` always agree. NULL name or bucket
count propagates as NULL; columns, non-string names, empty names, out-of-range
counts and wrong arity are rejected with explicit errors.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): include function identity in ScalarUdf equality

The ScalarUdf wrapper compared and hashed only the signature, so any two
functions sharing a signature were the same ScalarUDF: isnull, iceberg_hash
and iceberg_hash_bounds all have Signature::any(1, Immutable) and compared
equal. DataFusion compares expressions structurally, and its SQL planner
rewrites SELECT expressions against GROUP BY keys with base_exprs.contains
(rebase_expr); the collision made SELECT isnull(x) FROM t GROUP BY x,
iceberg_hash(x) fail planning, because isnull(x) matched the grouped
iceberg_hash(x) and was rewritten into an aggregate column that does not
exist.

Compare and hash the function name alongside the signature, so functions are
distinguishable while two instances of the same function still compare (and
hash) equal.

Add regressions: same-signature functions stay distinct ScalarUDFs and stay
separable in hash-based containers, and grouping by one while selecting the
other plans and executes in both directions.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): quote rendered column names and reject mixed hash domains

`iceberg_hash_ranges_sql` interpolated the column argument verbatim, so a
legal name such as `host-name` produced `iceberg_hash(host-name)`, which parses
as a subtraction; names with spaces, dots, quotes or upper case could not be used
either. Quote the name as a SQL identifier and double embedded quotes, so the
rendered rules stay one identifier and survive parsing. Cover the round trip by
feeding the output back into `CREATE TABLE ... PARTITION ON COLUMNS` for plain,
dashed, upper-case, quoted and spaced names.

`iceberg_hash_breakpoints` accepted values of different types, but strings and
integers hash through different encodings (UTF-8 bytes versus Iceberg longs), so
mixing them yields breakpoints that describe no real column. Require one hash
domain (all strings or all integers, integer widths may mix) and name the
offending argument otherwise. The existing SQL case that mixed
`('iceberg', 34)` now uses two strings.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(function): register the raw hash UDF and tidy the helpers

Registering `iceberg_hash` through a `Function` wrapper built a different
`ScalarUDFImpl` type from the one partition rules plan, so DataFusion compared
the SQL expression and the rule expression as unequal; the wrapper also rebuilt
a `ScalarUDF` per batch. `ScalarFunctionFactory` already has `From<ScalarUDF>`,
so register `iceberg_hash_udf()` directly and add a test that the registered
function and `IcebergHash::to_logical_expr` produce equal expressions.

Also: collapse the arity check and slice destructuring into one `let ... else`;
make the bucket-count helpers associated functions instead of constructing a
`default()` instance; pass the caller name through so an error raised from
`iceberg_hash_ranges_sql` names that function rather than
`iceberg_hash_bounds`; and fetch the SQL test's UDFs from `FUNCTION_REGISTRY`
instead of rebuilding and wrapping each one.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): accept every string column name and cover the rules end to end

`iceberg_hash_ranges_sql` matched only `Utf8` and `LargeUtf8` for its
column-name argument, so a valid constant string that arrives as
`Utf8View` (for example `arrow_cast('host', 'Utf8View')`) was rejected as
non-string. `Signature::any` leaves argument types unchanged, so the function
has to accept all string representations itself; use `ScalarValue::try_as_str`
and keep the NULL and empty-name checks. A unit case and a SQL-level case pin
that `Utf8View` renders the same rules as a plain literal.

Add a sqlness case that renders the four buckets for the quoted column
`"host-name"`, creates the table from that exact text, inserts one value per
bucket plus NULL, and checks the row counts implied by the rendered predicates
against the rows each region stored. This closes the gap between rendering the
rules and routing rows with them.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(partition): hash constants through IcebergHash::evaluate_array

`iceberg_hash_values` duplicated `IcebergHash::evaluate_array` line for
line. Call the existing method from `into_hashed_scalar` and drop the extra
public entry point, so the SQL helpers still hash through the code a partition
rule evaluates without a second identical API.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-13 06:10:13 +00:00
dennis zhuang 7b72fb6987 refactor(promql): drop the Go math port and use the standard library (#10392)
* refactor(promql): drop the Go math port and use the standard library

The pointwise kernels called a port of Go's `math` functions as they behave
on amd64. Results are compared within a tolerance, and the port reproduced
amd64 assembly quirks that Go on arm64 does not share.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): stop claiming map_arithmetic follows Go's math functions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 23:02:04 -07:00
dennis zhuang ca1b8c7c98 fix(config)!: redact credentials from diagnostic TOML (#10381)
* fix(config): redact credentials from diagnostic TOML

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): redact credentials at their owning fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(config): cover skipped SAS tokens and export logs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask SAS tokens before overlapping account names

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): redact SAS tokens before overlapping account fields

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cli): mask containing credentials before shorter values

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 05:35:39 +00:00
dennis zhuang bdb2e6777b refactor(promql): derive native output columns and run-time labels with one walk (#10394)
The planner's `native_output_tags` was a copy of `fused_output_tags` in
`promql::native::plan` read over tag columns instead of run-time label
sets. The planner calls `fused_output_tags`, which takes over the one rule
the copy had on its own: `or` reports the metric name of each row.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 22:13:15 -07:00
dennis zhuang f02d6d682a refactor(promql): share NativeNode child traversal across crates (#10391)
Move `node_children` and `with_node_children` from the query planner to
`NativeExpr::children` and `NativeExpr::with_children`, add
`NativeNode::{any, inputs, is_data_free}`, and replace the hand-written
structural walks over `NativeExpr` with them.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 21:51:23 -07:00
dennis zhuang 21e2255f4b fix(promql): info() ACL targets, per-series routing, reset collisions (#10385)
* fix(sql): include the table PromQL info() reads in extracted tables

`info(v)` and `info(v, {label="x"})` read `target_info`, which is not written
as a selector in the expression, so table-level permission targets missed it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep per-series provenance behind CooperativeExec

`EnsureCooperative` wraps the `SeriesScanExec` leaf and rebuilds
`RawSeriesSourceExec` over the wrapper, which cleared `per_series_scan` and
left `SeriesFrameOutput` on the merge-inputs router.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): drop histogram pairs whose counter reset hints contradict

Prometheus returns the collision as the error of `+`/`-` and skips the pair
before counting its match. The native path kept the sample and counted it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(frontend): keep info() label selectors out of metric discovery

The table extractor now claims `target_info` only for the selector shape the
planner reads it with; other shapes stay unresolved. `MetricDiscoveryVisitor`
skips the label selector of `info`, which the planner never discovers with,
so an unresolved one is no longer answered by the caller's discoverable tables.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): expect the dropped pair on a histogram reset hint collision

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 21:51:00 -07:00
dennis zhuang f576134b91 refactor(promql): read binary operators' kernel operations from one table (#10393)
* refactor(promql): drop the Go math port and use the standard library

The pointwise kernels called a port of Go's `math` functions as they behave
on amd64. Results are compared within a tolerance, and the port reproduced
amd64 assembly quirks that Go on arm64 does not share.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): read every binary operator's kernel operation from one table

`BinaryOp::kind` maps an operator to the arithmetic or comparison the map
kernels implement. The vector-vector kernel, the scalar broadcast and the
compiler's constant fold use it instead of their own tables and their own
copies of the arithmetic.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): use BinaryOp::is_comparison for the hand-written comparison checks

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 21:50:13 -07:00
dennis zhuang 7b455f2b81 docs: record authorization invariants in servers and frontend guides (#10384)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-13 04:07:39 +00:00