Commit Graph
1768 Commits
Author SHA1 Message Date
Weny Xu 4e5b60b869 refactor: rename flushed_parquet_bytes to ingested_parquet_bytes (#10258)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 04:25:42 +00:00
Weny Xu 8bdb7d5509 feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 03:17:48 +00:00
fys 3051cb41db feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths

Signed-off-by: fys <fengys1996@gmail.com>

* chore: use rebased proto for JSON path fulltext

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: reuse index target memory estimate for fulltext cache

Signed-off-by: fys <fengys1996@gmail.com>

* fix: filter disabled fulltext options for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* docs: clarify fulltext intermediate path uniqueness

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: name index targets consistently

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): apply JSON fulltext indexes per compatible target

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): skip invalid JSON fulltext index targets

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): require enabled fulltext indexes for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: simplify JSON fulltext index construction and pruning

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): unify fulltext target compatibility and creator APIs

Signed-off-by: fys <fengys1996@gmail.com>

* perf(function): preserve string layouts when evaluating matches

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): centralize fulltext predicate extraction errors

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: remove enable flag from JSON fulltext proto options

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: clarify JSON fulltext index helpers and imports

Signed-off-by: fys <fengys1996@gmail.com>

* fix: allow rebuilt fulltext indexes on ordinary columns

Signed-off-by: fys <fengys1996@gmail.com>

* test: spell out false positive rate in fulltext tests

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto to upstream merged revision

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): initialize fulltext option in multi-region tests

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-06 10:33:10 +00:00
Weny Xu 4a85e6fcf3 feat(meta): expose datanode drain and scheduling HTTP APIs (#10083)
* feat(meta): expose idempotent datanode drain HTTP APIs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify drain admission and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): reject concurrent datanode drains

Align the drain tracker with region migration and return DrainDatanodeRunning for tracked nodes. Restore the original recover hook and rebuild reservations in the loader without procedure IDs.

Retain admission coordination and recovery readiness to protect scheduling restoration. Update concurrent submission and persisted recovery tests for rejection semantics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): allow procedure recovery retry after startup failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): classify active drain conflicts as illegal state

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 08:34:13 +00:00
Weny Xu d750d25965 perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: push down SST manifest scalar aggregates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: remove redundant SST manifest inspection guide

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: generalize SST manifest aggregate pushdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: retain frontend aggregation for unsupported inspection states

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: expose remote SST aggregate plans in EXPLAIN

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: normalize SST analyzer in TQL explain snapshots

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: guard SST predicate types before remote inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:37:00 +00:00
Ning Sun b21b1cdb69 fix(function): require ADMIN statement privileges to execute admin functions (#10120)
* fix(function): require ADMIN statement privileges to execute admin functions

Read-only accounts could execute the admin UDFs (flush_table,
compact_table, flush_flow, gc_regions, migrate_region, reconcile_*,
build_index, ...) through ordinary SELECT statements: SELECT is a
read-only statement so it passes the statement-level privilege check,
and the functions were only guarded by the greptime-catalog scope
check inside the UDF. That let read-only users trigger WAL flushes,
manifest version bumps, compactions, flow-sink writes and, in
distributed mode, region GC/migration and SST deletion.

Wrap the privilege check into the admin_fn macro itself, next to the
existing greptime-catalog check: before executing, every #[admin_fn]
function now requires the same privileges as its ADMIN statement. The
check synthesizes the PermissionReq::SqlStatement request an
ADMIN <fn>(...) statement produces and evaluates it with the same
PermissionCheckerRef plugin the frontend installs, so query invocation
(SELECT <fn>(...), EXPLAIN, views, flows) and the ADMIN statement path
are gated identically — in particular read-only accounts are denied
with PermissionDenied (7006), while write-privileged users keep both
invocation styles. With auth disabled (no permission checker), the
functions stay invocable by everyone, exactly like ADMIN statements.
The denial is wrapped as External(BoxedError) so the 7006 status code
survives both the query execution path and the admin statement
executor.

Regression tests: unit tests for the check (no-checker allows,
read-only denied 7006, read-write allowed), an end-to-end test over
the macro-generated flush_table UDF, and an integration test where a
read-only account is denied 7006 on all 13 functions from SELECT while
a read-write account keeps SELECT and ADMIN FLUSH_TABLE working.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(function): expose admin privilege check to downstream macros

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-11-04 18:18:33 +00:00
discord9 65a89945dd feat(flow): support caller-owned fenced incremental deltas (#9996)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
2026-11-04 18:18:23 +00:00
shuiyisong fd74b9239e feat(otlp): store validated metric exemplars (#9923)
* feat(otlp): persist metric exemplars

OTLP metric conversion now emits exemplars through the shared storage contract. It validates timestamps, IDs, values and label budgets, and associates histogram exemplars only with emitted buckets that contain their values. Invalid observations are dropped and counted while valid points and siblings remain ingestible.

Frontend ingestion authorizes the emitted destinations and retains confirmed completions across metric and exemplar writes, including logical-table batching. Includes conversion and schema-failure isolation tests; tests that also query the new Prometheus endpoint land in the final layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(otlp): bind exemplars to source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): bound exemplar memory and preserve series

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars without metric engine

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): skip exemplars for ordinary tables

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(exemplar)!: prefix the shared table name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): avoid waiting with body permits held

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:09:57 +00:00
Minh Vu 25003cea16 fix(time): parse zero-hour timezone offsets correctly (#10091)
fix(time): preserve zero-hour timezone offset signs

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
2026-11-04 06:49:45 +00:00
fys b6cf3384dc fix: register trigger procedure loaders through DdlManager (#10123)
Add a loader registration hook to TriggerDdlManager and invoke it
  before procedure recovery. Test that registration reaches the
  configured trigger DDL manager.

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-04 04:31:23 +00:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
Weny Xu 7339a56abd feat: share streaming range results across scans (#9794)
* feat: share streaming range results across scans

Share post-merge range batches across concurrent readers while preserving stream boundaries, cancellation and retry behavior. Encapsulate range ownership, compaction and publication in RangeResultCache, backed by the common SyncCache primitive.

Retain SST retirement fencing and memory accounting. Track live shared-buffer bytes and worker events alongside existing cache metrics.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito2): clean up shared range registrations on release

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-03 11:33:34 +00:00
Weny Xu 6491700fe6 fix(meta): forward flushed Parquet bytes in region statistics (#10100)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 11:42:56 +00:00
Weny Xu 295bed98ec feat: share Parquet byte page loads (#9793)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 10:52:31 +00:00
shuiyisong a57ab18eaa feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries

Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers.

Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(servers): coalesce flow notifications

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(ingest): reduce write progress overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): drain confirmed writes on cancellation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(batcher): reduce row owner allocations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(batcher): restore per-flush metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* revert: scope retry-hint fix to storage writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): restore flush metrics and benchmark

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(ingest): propagate sync batch write errors

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: preserve peer batching for confirmed writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: avoid unnecessary row owner copies

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: reuse IPC buffers in tracked retries

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* style: order write dispatcher imports

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-02 03:09:13 +00:00
Weny Xu c035670ec0 feat: share Puffin metadata and index page loads (#9792)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-30 07:25:55 +00:00
zhaiAohan 8a38b90135 feat: add relational region engine skeleton and registration (#9953)
* feat: add relational region engine skeleton and registration

Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered.

Signed-off-by: tian1220A <1573612521@qq.com>

* feat: add relational connection and idempotent region lifecycle

Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder.

Signed-off-by: tian1220A <1573612521@qq.com>

* refactor: address review comments on relational region engine

- remove connection.rs (only used by its own tests); it will be
  reintroduced when connection handling is wired up
- drop RelationalOptions::version (not validated or used to select a
  deserialization format)
- validate an idempotent create against the existing region definition
  (region id / schema / primary key / options) and return Unexpected on
  mismatch, with a reason field on RegionExists
- use serde.workspace and remove the empty [features] section
- cargo fmt

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: adapt relational engine to latest main

- implement RegionEngine::set_region_role_with_lease (no-op lease, the
  engine takes no writes so no lease is needed)
- pass node_id to build_object_store_manager in the registration test
- drop stale url dependency from the relational-engine lock entry

Signed-off-by: tian1220A <1573612521@qq.com>

* ci: register relational-engine in scope rules

Signed-off-by: tian1220A <1573612521@qq.com>

* fix: remove table self-dependency from Cargo.lock

Signed-off-by: tian1220A <1573612521@qq.com>

---------

Signed-off-by: tian1220A <1573612521@qq.com>
2026-10-30 04:06:01 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00
shuiyisong efed847ca6 fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce sparse primary key schema

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: fix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-28 17:09:33 +00:00
Weny Xu b83ceb5e7b feat: add reusable paged byte cache (#9791)
* feat: add reusable paged byte cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): include common cache in scope rules

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 17:09:23 +00:00
Weny Xu df33737e9d feat: add shared asynchronous cache primitives (#9790)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 08:45:10 +00:00
discord9 bd77b599cc refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-28 07:27:04 +00:00
Weny Xu cdc0d3a055 feat(procedure): support detached subprocedure recovery (#10009)
Select Discard or Detach when submitting children via Status::Suspended.
Persist the policy from the child's first normal or rollback checkpoint.
Recover detached children as independent roots with their original id and
context, using existing execution and cleanup without rebuilding the graph.

Keep fresh root admission serialized while allowing accepted procedures to
recover above the submission limit. Cover first persistence, repeated
recovery, rollback, cleanup, defaults, and concurrent admission.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-28 07:26:22 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
Chenglong Zheng 2117f8b059 feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID

Signed-off-by: hahaha <a310608318@gmail.com>

* feat: Fix and add tests according to review comments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: the zero-argument signature in ADMIN swallows explicit arguments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: fix the CI failures caused by SQL version issues

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: apply the changes based on the review

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-27 07:27:55 +00:00
shuiyisong 5e9bb3c689 feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align native histogram ingestion with remote read validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-27 03:09:13 +00:00
jeremyhi 6f5c064f54 fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process

Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.

The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.

Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache load and pusher updates coherent

A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.

A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.

The epoch check moves into the heartbeat service.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache coherent with writes while it is disabled

A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): pin the epoch 0 exemption and pusher order, and update docs

The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): register a datanode pusher only after its first request passes

A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.

Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): check a datanode epoch again before replying

A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): handle a datanode's requests one at a time

A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.

The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:02:40 +00:00
Weny Xu c96cdd0026 perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): stream and prune manifest SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(datanode): stream inspection queries through reusable providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(catalog): push manifest filters through inspection queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: align manifest column constants with module conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: reduce manifest inspection allocations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: narrow manifest inspection API visibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: retain residual filters for manifest inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 08:54:02 +00:00
Weny Xu 5523cf3874 feat(meta): track per-node scheduling state and fence region placement (#9884)
feat: track per-node scheduling state and fence region placement

Add a cluster-wide, persistent "can this node receive new regions" flag and
make every metasrv placement path respect it. This is the fence half of
`ADMIN drain_node`; the drain procedure, gRPC entry and region evacuation land
separately.

State layout
- New namespace `__node_scheduling/{datanode,flownode}/{node_id}` holding
  `{"schedulable": false}`. A missing key means schedulable, so undrain is a
  delete and no stale `schedulable: true` records are left behind.
- `NodeSchedulingKey`/`NodeSchedulingValue` mirror the existing KV conventions
  (`NodeInfoKey`'s `{role, node_id}` shape, `LeaseValue`'s `*Value` suffix).
  The role is part of the prefix, so extending to flownodes needs no key change.
- `NodeSchedulingManager` writes with `put_if_not_exists`/`compare_and_put`, so
  concurrent marks change the state exactly once, and reads a single prefix
  range per placement decision (DDL/failover only, not per query).

Placement
- `NodeInfo` gains `#[serde(skip)] schedulable: Option<bool>` plus
  `is_schedulable()`; `None` (unresolved) counts as schedulable, and the field
  never reaches heartbeat bytes, so the wire/in-memory format is unchanged.
- `SchedulingAwarePeerDiscovery` stamps the flag on `active_datanodes`. It is
  additive (never hides nodes) because failover still needs the full active set
  to compute its exclusion list.
- All three built-in selectors now fetch candidates through
  `selector::common::active_schedulable_datanodes`; `RegionSupervisor` adds
  unschedulable datanodes to `exclude_peer_ids` for both selector variants.

Enforcing the single fence
- `MetaPeerClient` no longer implements `PeerDiscovery`; its discovery methods
  are inherent, and `DatabaseOperator` (frontends only) takes the concrete
  client. `SchedulingAwarePeerDiscovery`, built once in `MetasrvBuilder`, is
  therefore the only way to obtain a `PeerDiscoveryRef`, so bypassing the fence
  is a compile error rather than a silent misplacement.

Compatibility
- New key namespace only; `NodeInfo` serialization is byte-identical, so old
  metasrv nodes and persisted heartbeats keep working. The flag itself is stored
  in the durable KV, so a follower metasrv sees marks written by the leader.

Verification
- `cargo nextest run -p meta-srv node_scheduling` (13), `selector` (12),
  `supervisor` (14), `-p common-meta cluster` (9), `-p meta-client` (36),
  `-p standalone` (compiles, no tests).
- `cargo clippy -p meta-srv -p common-meta -p meta-client -p standalone
  --all-targets -- -D warnings`, `cargo fmt --all -- --check`.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-23 06:46:56 +00:00
NightG1owZandNightG1owZ b886ae525d feat(config): support overriding region engine options from env (#9700)
* feat(config): support overriding region engine options from env

`region_engine` is a `Vec` of externally tagged enums, so plain dotted
environment keys cannot address it: `config` walks the path, replaces the
default sequence with a table and loading fails with `invalid type: map,
expected a sequence`.

Address the entries by tag name instead
(`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the
default entries in a dedicated source, so the engines and fields that are not
mentioned keep their configured value. An entry the default sequence does not
have is appended; a numeric index, an override of the whole sequence, a field
below a non-table default and two variables that set a path and a field below
it are reported with a dedicated error instead of a serde type error.

A configuration file that configures the sequence as a whole still replaces
the merged value, which the example configs document.

Closes #8620

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* docs(config): clarify that the metric engine is always enabled

`region_engine` defaults to [mito, file], but the datanode builds and
registers the metric engine unconditionally. Reword the example configs so
`mito` and `file` read as the default configuration entries and naming
`metric` only supplies its options, then regenerate config.md.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* docs(config): drop the env-var explanation from the config reference

Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* test(config): cover tagged sequence environment collection

Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

---------

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
2026-10-23 01:46:55 +00:00
shuiyisong 2eff52c757 feat(otlp)!: support raw delta exponential histograms (#9754)
* feat(otlp): support raw delta exponential histograms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): guard point accounting and benchmark metrics ingestion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove otlp_metric_compat logic

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): cache physical schema lookups during metric preflight

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: minor rename

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): batch metric rows and reuse schema validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: replace super in import path

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-22 13:36:30 +00:00
fys f35057d9c2 feat(json2): support inverted index (#9324)
feat: build inverted indexes for JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-22 11:42:42 +00:00
Ning Sun af52dfafd8 chore(deps): upgrade sqlx to 0.9 to pick up webpki-roots 1.0.10 (#9862)
sqlx 0.8 pins the unmaintained webpki-roots 0.26 line for its rustls
trust store, which still contains the recently compromised e-Tugra CA
root. Upgrading sqlx to 0.9 switches its dependency to webpki-roots
1.x, and the lockfile now resolves to 1.0.10 where the distrusted
root has been removed from the compiled-in Mozilla trust store.

sqlx 0.9 requires dynamic SQL strings to be explicitly audited via
the new SqlSafeStr trait: only &'static str literals are accepted
as-is. All affected call sites (MySQL KV backend, MySQL election,
integration tests and fuzz targets) now wrap their internally
generated SQL templates in AssertSqlSafe; the wrapped SQL is built
from operator config or test-local identifiers only, and dynamic
data keeps flowing through bind parameters.

The mysql-rsa feature is enabled alongside mysql to preserve 0.8
authentication behavior: sqlx 0.9 gates non-TLS RSA password auth
(caching_sha2_password / sha256_password) behind it.

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-10-22 07:27:19 +00:00
Weny Xu b4aae2f73b fix(meta): release all PostgreSQL election advisory locks on step-down (#9864)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-22 06:20:23 +00:00
LFC 67bd279586 fix(query): preserve Arrow types when decoding dictionary output (#9858)
Signed-off-by: luofucong <luofc@foxmail.com>
2026-10-22 04:24:04 +00:00
Weny Xu 9e9346d2ff fix: preserve datanode write batches when retrying (#9720)
* fix: preserve datanode write batches when retrying

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: isolate invalid entries in metric write batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: adapt batch writes to the unified metric API

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: explicitly accept partial results for batched retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: assert terminal retry hint instead of masked error text

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: preserve application status in region failure fixtures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: refresh routes after region close

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-22 03:36:02 +00:00
Chenglong Zheng 9f9b35c270 feat: support wait and timeout options for TRUNCATE TABLE (#9753)
* feat: support wait and timeout options for TRUNCATE TABLE

* fix: address review feedback for truncate DDL options

* fix: address review feedback for truncate DDL options
2026-10-21 07:57:16 +00:00
Weny Xu f866ef2bd9 feat: return per-entry region write failures (#9728)
* feat: return per-entry region write failures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints for single-entry region writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve region write error compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify region partial result handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align partial failure summaries with input order

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: separate region write response handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: fix partial result expectations and region map guard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: limit lockfile changes to greptime proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: make partial region results an explicit client opt-in

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: pin greptime-proto to merged partial-result changes

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-21 05:51:15 +00:00
Lei, HUANG a1b2362e58 fix: avoid duplicate keys in update flow metadata txn (#9816)
CREATE OR REPLACE FLOW deleted and then put the same flow route,
flownode flow and table flow keys in a single txn when the replacement
kept the partitions, flownodes or source tables, which etcd rejects
with "duplicate key given in txn request".

Only delete the old keys that are not put again. Closes #9810.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-21 03:52:34 +00:00
shuiyisong 41ec5e0763 fix: decouple slow-query logging from the global log level (#9765)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-20 07:40:54 +00:00
dennis zhuang da88ff8653 fix(partition): route NULLs like row inserts and accept signed-minimum bounds (#9697)
* fix(partition): route NULLs like row inserts and accept signed-minimum bounds

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep equality two-valued for NULL rows and accept unsigned -0

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): treat NULL masks as false in batch routing and keep comparisons bare

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(compat): update partition filter in analyze verbose expected output

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-19 15:22:13 +00:00
fengyue-xve 1e29e8d4b6 fix(meta): validate persisted partition bounds before altering regions (#9487)
* fix(meta): validate persisted partition bounds before altering regions

A partition bound is persisted with the type of the column at the time
it was written. When an ALTER TABLE changes the type of a column
referenced by a partition expression (widening the time index unit is
the common case), each region rewrites the bound literal to the new
type while building its scan predicate. A millisecond bound like year
3000 fits milliseconds but overflows nanoseconds, so every scan of the
region fails once the table schema is updated -- and some regions may
already be altered when the failure surfaces.

Validate every region's persisted expression on prepare, before any
region is modified: parse the persisted JSON (mirrored types, since
common-meta cannot depend on the partition crate which depends on it),
replay the same value -> ScalarValue conversion and checked cast the
region performs, and reject the ALTER naming the offending region when
a bound cannot be represented with the new column type. Incompatible
changes are rejected atomically; compatible widening proceeds unchanged.

Adds meta-level tests (multi-region rejection naming the second region,
compatible widening completing) and an end-to-end sqlness case.

Fixes #9454

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix: access ColumnSchema fields directly and use plain format args

ColumnSchema exposes name and data_type as public fields, not methods,
and the telemetry macros only accept a format string after the error.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: access ColumnSchema data_type field in alter table tests

Same fix as the previous commit: data_type is a public field on
ColumnSchema, not a method.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop borrows now that data_type is read as a field

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop useless format! in nested expression test

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: cover the partition tail in change_col_type sqlness case

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: drop spurious blank line in change_col_type result

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(meta): compare partition bounds with the scan comparison coercion

The preflight validator cast the persisted bound directly to the altered
column type, which rejected ALTERs the scan itself accepts: the region
coerces both the column and the bound to the common comparison type
(ExprSimplifier::coerce in PredicateGroup::new), so narrowing a field
column with an out-of-range bound still scans. Compare through
comparison_coercion(target_type, bound_type) instead and only require the
bound to be castable to the common type.

Also add a round-trip test in the partition crate that serializes real
PartitionExprs and runs them through the validator, so drift between the
Persisted* mirror and the partition types fails CI, and fix the error
message to match the comparison semantics.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* test: update the sqlness fixture for the new incompatibility message

The error message now reflects the comparison semantics: the persisted
bound can no longer be compared with the new column type.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* build(deps): refresh Cargo.lock for the common-meta datafusion-expr dependency

common-meta gained a datafusion-expr dependency in ff9c3f4 but the lock
file was not refreshed, so cargo check --locked refused to run on CI.

Signed-off-by: fengyue-xve <1759519137@qq.com>

* fix(meta): reject every non-widening type change of a partitioned column

The region routes rows on write through `MultiDimPartitionRule::find_region`,
which compares the persisted bound against row values directly: `Value::cmp`
panics on mismatched variants and equality misroutes silently. After a column
partitioned at `v < 3000000000` is altered from BIGINT to INT, every INSERT
compares `Value::Int32` rows against the persisted `Value::Int64` bound and
panics in the frontend, even though the scan-side coercion this check mirrors
still works.

Reject any type change of a column referenced by a persisted partition
expression unless it is a timestamp unit widening, which keeps the bound and
the rows comparable because `Timestamp` comparisons convert units first; the
widening keeps the scan cast check unchanged.

Signed-off-by: fengyue-xve <1759519137@qq.com>

---------

Signed-off-by: fengyue-xve <1759519137@qq.com>
2026-10-19 14:49:14 +00:00
Yingwen b3dc5674e9 feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: unify scan memory limits with configurable accounting modes

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): reuse query runtime memory pool for series merges

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): wait for cancelled scan reservations before exhaustion check

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs(mito2): explain parallel scan cancellation cleanup

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): deprecate waiting for scan memory quota

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(recordbatch): remove scan memory wait handling

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(table): initialize memory pool in scan test fixtures

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-19 10:09:30 +00:00
dennis zhuang 672e94f61d feat: add column encoding option and use it for trace span ids (#9629)
* feat: add column encoding option and set it for trace span ids

COMPRESSION WITH accepts encoding = 'plain' | 'delta_binary_packed'.
Either disables the dictionary for the column; delta_binary_packed is
limited to integer and timestamp columns.

OTLP trace v1/v2 tables now create span_id and parent_span_id with plain
encoding and 64KiB pages, and timestamp_end with delta_binary_packed.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: add compatibility cases for column encoding options

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: move column encoding downgrade checks into downgrade_compatibility

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-16 09:40:54 +00:00
Weny Xu 4859e57e0b feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): share repartition affinity until group allocation

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 14:29:10 +00:00
jeremyhi deae75ffee fix(log-store): refuse to open object store WAL regions whose history is missing (#9561)
* fix(log-store): refuse to open object store WAL regions whose history is missing

A region persists only its WAL prefix, not the object store that holds it.
After `[wal].storage_provider`, or the bucket or root of the provider it
names, changes, or after the objects under the prefix are deleted, every
region reopened against a prefix without its history, replayed nothing and
silently lost its acknowledged writes that were not flushed.

Add `LogStore::check_region_history` and `LogStore::start_region_history`,
whose defaults accept and do nothing, so Raft Engine and Kafka are
unchanged. The object store WAL reports the history of a region as missing
when recovery found no object under the prefix, or when the region's
watermark names an object above the newest object holding entries that
recovery indexed, unless the catalog holds the region's entries through the
watermark or the region's history starts in the store. Both rest on
invariants of the prefix: an opened prefix is never empty, and a persisted
watermark names an object at or below the newest object holding entries,
which collection never deletes. The segment of release records counts as
entries, since an object holding only release records can be the one
collection keeps; start objects hold no segment and do not move the bound.

When a store that found its prefix empty refuses a region, the actor gives
up its start object, so the next start refuses the region again. It agrees
only while nothing was handed an id, no batch, append, release or create
waits, no failed release is kept, no other object was listed, the catalog
holds only the start object and no region's history started in the store,
deciding under the lock region registration takes. It then records a
terminal error, deletes the object as one of its tracked deletes, which
`stop` waits for, and answers every refusal once that delete settled.

Mito runs the check before it replays an existing region on open, on
catch-up and when a create finds the region present, with the larger of
the flushed and truncated entry ids, and records the history of a region
it creates; regions that skip the WAL and noop regions are not checked. A
missing history fails the open with an error naming the region, the
prefix, the configured storage and how to move the WAL by copying its
objects. The new `[wal].accept_missing_history` option, off by default,
opens such a region without its unflushed writes instead, with a warning
and the counter
`greptime_logstore_object_store_wal_missing_history_accepted_total`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the object store WAL history check

Describe the open-time history check, the prefix invariants it relies on,
how release records bound the history, the start object withdrawal of a
refused open and the window that remains, the accept_missing_history
option and how to move a WAL by copying its objects, and update the
metrics, Mito integration, evidence, compatibility and cluster alignment
sections.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): unindex the start object a refused open withdrew

A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): separate the withdrawal windows of start objects and entries

An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep a successor's chain complete across a late withdrawal

A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a created region's WAL history before its manifest

A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the start boundary of a chain without segments and the create order

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-15 12:38:01 +00:00
discord9 c3469fad51 fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures

Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths.

Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities.

Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve upstream dependencies after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): fail fast on definitive tracked-write open rejection

A tracked write treats every application-level error from the frontend as
an ambiguous transport failure and retries the same pinned frontend
forever. During a rolling upgrade an older frontend does not recognize the
tracked-write command extension and processes Open as an ordinary empty
SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal
endpoint rejects the extension outright. Both are definitive rejections
that no same-frontend retry can ever resolve, so the flow stalls.

Classify the Open-phase errors by status code: a definitive rejection
(Unsupported / InvalidArguments / PermissionDenied) fails fast with a
terminal pre-write error, while genuinely ambiguous transport failures
(Unavailable, timeout, connection reset, ...) keep the same-capability
retry. Write/Close rejections are deliberately NOT failed fast: once a
write may have been admitted, a same-code rejection (e.g. WriterLost, or
UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome
ambiguous, and failing fast would let the caller replay and duplicate
rows.

Add a regression test that a legacy empty-SQL Open rejection fails fast
with a single attempt and surfaces as Error::External carrying a
TerminalError.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preflight tracked writes without submitting data

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): retain tracked command extensions while decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-15 10:49:09 +00:00
Weny Xu cbfbad469f feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover load-based affinity fallback queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): borrow affinity requirements during selection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove unused affinity error variant

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 09:41:31 +00:00
dennis zhuang f7a3a7033c fix(query): keep state column names unique when splitting aggregates (#9594)
* fix(query): keep state column names unique when splitting aggregates

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(query): probe allocated names when renaming colliding state columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-15 07:51:17 +00:00
dennis zhuang 5c4d9eb4f8 fix: align fulltext bloom tokens with matches_term and use them for LIKE (#9470)
* fix: align fulltext bloom tokens with matches_term and use them for LIKE

The bloom fulltext index stored tokens that did not follow
matches_term's word boundaries (`_`, ASCII next to Han, Han substrings),
so pruning dropped rows matches_term accepts.

Tokens and query probes now share the character classes of
matches_term, Han runs are indexed as bigrams, and LIKE patterns use the
same probes. The bloom blob is written under a new key; blobs from older
versions are no longer probed. Tantivy no longer serves matches_term.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: drop unused lazy_static from index and avoid typo-flagged test tokens

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: link like_probes to tokenize instead of the downstream ScriptTokenizer

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: name probe extraction with verbs and document CharClass

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: rebuild legacy fulltext bloom indexes and probe lower(col)

The manifest records the fulltext bloom version per column, so ADMIN
BUILD_INDEX now rebuilds SSTs whose bloom index predates the current
tokens, and ssts_index_meta shows the version.

Case-insensitive bloom indexes also store the tokens of the whole
lowercased text for rows where it can differ (U+0130, U+212A, sigma),
so matches_term(lower(col), ..) and lower(col) LIKE .. use the index.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: cover legacy bloom index rewrite by compaction in the compat case

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: log outdated fulltext bloom indexes in the consistency check

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: explain the unrecorded fulltext bloom version

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-14 08:07:24 +00:00