Commit Graph
646 Commits
Author SHA1 Message Date
discord9 ee9ad87fa1 Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main (f8d8fe1071) into narthex (115d85b669) on a throwaway
local branch. 31 semantic conflicts resolved; workspace and all test targets
compile; promql 1215/1215 and mito2 series_index 185/185 pass.

Pre-merge pins and follow-ups are marked with TODO(narthex-merge):
- greptime-proto is temporarily pinned to the fork branch carrying
  GreptimeTeam/greptime-proto#355; replace with the upstream rev once it lands.
- native payload byte cache is disabled (main removed PageRangeCache); two
  samples tests are #[ignore]d until it gets a dedicated page-cache budget.
- columnar metadata now has its own CacheManager tier (a quarter of
  sst_meta_cache_size); the unified decoded tier was not kept.
- query.experimental_native_promql is registered as a query option, otherwise
  the native route is silently rejected by the option snapshot.

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 15:45:04 +08:00
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
2026-11-12 01:09:42 +00:00
Weny Xu 54b6bd5b44 feat(mito2): support extension index providers and input reuse (#10297)
* feat(mito2): register extension index providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito2): support index input reuse and extension budgets

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 14:02:48 +00:00
jeremyhi 77cb39b5d1 feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest

Adds `pending_tails` and `handoff_floors` to the region manifest and its
checkpoint, and an optional `pending_tails` / `handoff_floor` pair to
`RegionEdit`. An edit without them leaves the records unchanged. A floor is
dropped once the flushed entry id reaches it; a tail stays until a commit
removes it. Neither is written when empty, so manifests without them encode
as before.

Opening a region replays its WAL after this node's tails and floor, and WAL
releases at open and after a flush stay below this node's tails. Nothing
writes these records yet, so every region behaves as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): hold back reads and region changes while WAL tails are pending

A region with pending WAL tails is missing data that only those tails hold.
Until they are recovered, scans and memtable and SST stats fail with the
retryable `RegionPendingTails` error. Truncate, entering staging, a
repartition flush, a close-time flush on request, and ALTERs other than
adding nullable columns without a default or changing options are rejected
before they change any region state. Compaction pauses. A full-listing GC
pass skips the region while it has pending tails, or while a newer manifest
may exist or the manifest listing fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): name the WAL releases that keep pending tails

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that reopening restores the pending-tail read gate

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin what the pending-tail fixtures and rejections promise

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the replay start in the open log and share the pending-tail check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the per-node order of pending tails and who keeps later writes out of them

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending

When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the pending compaction task state when WAL tails skip it

Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-10 12:21:17 +00:00
Lei, HUANG 557843dd68 fix(catalog): use advertised peer address as process identity (#10153)
* fix(catalog): use advertised peer address as process identity

The process manager identified the local frontend by its public gRPC
address, while metasrv discovers peers by the address advertised in the
heartbeat (the internal gRPC address when configured). With separate
public and internal listeners, list_all_processes failed to exclude the
local node, queried its own management service as a remote frontend, and
appended the local registry again, so every local query was listed twice.
The same mismatch broke routing of KILL to other frontends.

Identify a frontend by the address it advertises to metasrv:
- add FrontendOptions::peer_addr, shared by the heartbeat and the process
  manager, and FrontendOptions::create_process_manager to wire it once;
- use the advertised address as the process identity, for self-exclusion
  in list_all_processes, and for local/remote routing in kill_process.

Process ids now carry the address other frontends use to reach the node.
When no internal gRPC listener is configured this is the public gRPC
address, so the previous behavior is unchanged. The internal listener is
the only one serving the frontend management service, so process ids stay
routable.

The new distributed regression tests exercise separate public and internal
addresses: the local query must be listed exactly once while remote
frontends are still listed, and KILL must route to a remote frontend.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: keep local peer address reserved in kill-routing test

The test reserved the local frontend address with a temporary listener that
was dropped immediately, then reserved the remote address separately. The
kernel could assign the same port to the remote frontend, so the remote kill
request would take the local branch and fail to find the remote process.

Bind the local address once and keep the listener alive for the whole test so
the local and remote frontend identities cannot collide.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-04 18:18:38 +00:00
shuiyisong 4633c37dbb fix: test (#10149)
fix: fixture

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 10:21:42 +00:00
shuiyisong adf29740b2 fix(auth): redact inline user-provider credentials (#10115)
* fix(auth): redact inline user-provider credentials

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(auth): close user-provider redaction gaps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-04 03:39:51 +00:00
Weny Xu 295bed98ec feat: share Parquet byte page loads (#9793)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-02 10:52:31 +00:00
Lei, HUANG 045cf74c3f perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task

Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.

The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.

One output of 4.8M rows took 1.28s instead of 1.65s.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): parallelize decoding and encoding of compaction outputs

Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.

The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.

The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): cancel compaction producers before consumer polling

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): support per-table compaction output parallelism

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoding tasks before returning failures

Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.

Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoders on SST write errors

Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.

Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain compaction outputs on cancellation

Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.

Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-28 17:09:26 +00:00
Lei, HUANG 17ed988f49 fix(mito): preserve SST lifetime across index replacement (#9950)
* fix(mito): preserve SST lifetime across index replacement

Share retirement and final purge across immutable index metadata views. Keep version-specific GC references until readers release them and match Parquet liveness by file identity.

Fixes #9944

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): list-based SST purge and single FilePurger::remove_file

The final purge of a retired SST no longer depends on FileMeta: index
files of every version are discovered by listing
({region_dir}/index/{file_id}. prefix), so even when the last live view
is a stale one, the purge still removes all newer index files.

- SstLifetime is slimmed down to { deleted, file_purger }; the purge now
  runs in the last FileHandleInner::drop (Arc::strong_count == 1, which
  is race-free here since only inners of the same file hold the lifetime).
- FilePurger is merged back into a single remove_file(file_meta,
  is_delete) method; is_delete means the file was retired AND this is the
  last live view.
- LocalFilePurger deletes the SST and all its index files via the new
  delete_file_and_all_indexes helper (list before delete, so a listing
  failure leaves the file untouched).
- Remove the dead index_outdated flag and its accessors, which had no
  callers in the entire repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): elect final SST view release atomically

Track metadata views with an atomic counter instead of inspecting the lifetime Arc strong count. Concurrent destructors now elect exactly one final releaser while preserving the single remove_file interface.

Add a deterministic concurrent-drop regression covering retired and live files.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): restore Arc-owned SST cleanup and clarify reference release

Restore the lifetime design from 835227dcbc, using the final Arc drop and latest metadata instead of manual view counting and per-SST index listing.

Rename FilePurger::remove_file to release_file_ref while retaining purge_file for final cleanup. Preserve and adapt the concurrent view-release regression test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): track SST references by file lifetime

Register one reference guard per logical SST lifetime and release it on final drop, including normal region close. Remove per-view release_file_ref and index_outdated handling.

Protect all index generations while their SST is referenced. Preserve the existing GC snapshot format by expanding the recorded index generation upper bound for older version-aware workers.

Update coverage for independent references, normal close, cross-region ownership, concurrent final cleanup and both GC listing modes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito): release SST references directly on lifetime drop

Remove the redundant FileReferenceGuard and let SstLifetime register and unregister its file reference directly. Preserve unconditional reference release on normal close and retirement-only purge.

Expose the reference manager through FilePurger and adapt reference manager tests without changing GC or snapshot behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito): verify SST cleanup with real storage and isolate GC policy cases

Replace the recording-only concurrent purge test with filesystem assertions using LocalFilePurger and shared fixtures. Keep intermediate-view retention and retired versus live file cleanup coverage without claiming exhaustive thread interleavings.

Separate GC reference-policy cases from reader-release ordering and reuse their setup. Production code remains unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito): format SST lifetime changes

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito): reclaim obsolete indexes by reader version

Track exact index-version references per metadata view and restore version-specific Puffin liveness in both GC paths. Keep file-level Parquet protection and shared SST lifetime cleanup, without expanding snapshots into unreferenced index generations.

Include old readers of imported files even when the source region is not open locally. Add real-storage GC coverage for reader release and lingering while retaining Parquet and the current index.

Validated workspace checks with default and all features, all-target all-feature Clippy with warnings denied, and all 1853 mito2 tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-27 10:20:45 +00:00
Yingwen 371c4e9004 feat: support query concurrency limits in standalone (#9958)
Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-26 07:19:00 +00:00
discord9 1ced0f7fdb chore(query): share the range assignment with every caller
Signed-off-by: discord9 <discord9@163.com>
2026-10-23 19:57:38 +08:00
discord9 da27d115ca feat(query): let a region input travel as a plan when regions are in process
A plan holding a node with no Substrait serializer was always cut at that node,
because the analyzer asks whether a plan encodes before it asks whether a node
commutes with the region split. That is right for a region reached over the wire:
the plan is encoded into the request and has to decode on the other side.

Standalone does not use that path. Its region server is the one built in the same
process, its query handler passes the request straight to `RegionServer::handle_read`,
and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is
work nobody reads, and refusing to place a node because it cannot be encoded
refuses a plan that would have travelled intact.

So the engine now knows whether its regions are reached in process, and the
encodability gate applies only when they are not. The capability is set where it
can be stated truthfully - the standalone bootstrap, which creates both the
frontend and the region server it reads - and it defaults to false everywhere
else, so an ordinary build and every other deployment keeps today's behaviour
byte for byte.

Skipping the gate skips one early return and adds no reason to expand:
`Categorizer::check_plan` still runs and still decides, and a plan that does not
encode for any other reason still expands at the node that caused it. Two tests
hold those two halves apart.

Signed-off-by: discord9 <discord9@163.com>
2026-10-23 19:57:38 +08:00
NightG1owZandNightG1owZ b886ae525d feat(config): support overriding region engine options from env (#9700)
* feat(config): support overriding region engine options from env

`region_engine` is a `Vec` of externally tagged enums, so plain dotted
environment keys cannot address it: `config` walks the path, replaces the
default sequence with a table and loading fails with `invalid type: map,
expected a sequence`.

Address the entries by tag name instead
(`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the
default entries in a dedicated source, so the engines and fields that are not
mentioned keep their configured value. An entry the default sequence does not
have is appended; a numeric index, an override of the whole sequence, a field
below a non-table default and two variables that set a path and a field below
it are reported with a dedicated error instead of a serde type error.

A configuration file that configures the sequence as a whole still replaces
the merged value, which the example configs document.

Closes #8620

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* docs(config): clarify that the metric engine is always enabled

`region_engine` defaults to [mito, file], but the datanode builds and
registers the metric engine unconditionally. Reword the example configs so
`mito` and `file` read as the default configuration entries and naming
`metric` only supplies its options, then regenerate config.md.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* docs(config): drop the env-var explanation from the config reference

Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

* test(config): cover tagged sequence environment collection

Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment.

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>

---------

Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com>
2026-10-23 01:46:55 +00:00
Yingwen b3dc5674e9 feat(mito2)!: account SST decoding against shared scan memory limits (#9101)
* feat(mito2): add shared estimated SST scan memory budget

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: unify scan memory limits with configurable accounting modes

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): reuse query runtime memory pool for series merges

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): wait for cancelled scan reservations before exhaustion check

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs(mito2): explain parallel scan cancellation cleanup

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): deprecate waiting for scan memory quota

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(recordbatch): remove scan memory wait handling

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(table): initialize memory pool in scan test fixtures

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-19 10:09:30 +00:00
Weny Xu 4859e57e0b feat(meta): apply affinity to failover and repartition (#9604)
* feat(meta): apply affinity to failover and repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): share repartition affinity until group allocation

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 14:29:10 +00:00
Weny Xu cbfbad469f feat(meta): add affinity hooks (#9596)
* feat(meta): add affinity hooks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover load-based affinity fallback queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): borrow affinity requirements during selection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove unused affinity error variant

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-15 09:41:31 +00:00
Weny Xu f690bbb204 refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): update new WAL replay tests after index removal

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 15:31:04 +00:00
liyang 7c32c38ba3 feat(cmd): add --http-api-server-addr CLI flags (#9512)
* feat(cmd): add --enable-api-server and --api-server-addr CLI flags

Signed-off-by: liyang <daviderli614@gmail.com>

* apply suggestion

Signed-off-by: liyang <daviderli614@gmail.com>

---------

Signed-off-by: liyang <daviderli614@gmail.com>
2026-10-12 10:17:15 +00:00
discord9 28697c9e55 feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode

Give a datanode the ability to decode and execute a plan whose MergeScan
carries a nested plan (e.g. an inner MergeScan over a build table), so a
distributed sub-plan pushed to a datanode can itself fan out to other
datanodes. This is the execution/codec capability only; it is not wired
to any user-facing switch and produces no plan by itself.

Codec (src/query/src/query_engine/default_serializer.rs):
- MergeScanAwareSerializer keeps the request's SessionState and the
  engine catalog manager, and decodes a MergeScan payload's tables by
  (catalog, schema, table) name through the engine catalog rather than
  the region-aware request catalog list (which binds every table to the
  request's region and breaks on mismatched columns).

Datanode (src/datanode):
- New DatanodeRegionQueryHandler (region_query.rs) serves region queries
  for the MergeScan nodes of a received plan, resolving region leaders
  via PartitionRuleManager and dispatching to peer datanodes.
- Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into
  the datanode's catalog/cache registry (with_dist_planner stays false:
  a datanode never rewrites its own plan).

Bug fixes uncovered while wiring this up:
- cache registry: register the table/table_info/table_name/table_route/
  partition_info caches the datanode's catalog manager and partition
  rule manager require (previously missing -> panic on lookup).
- default_serializer: decode_sub_plan bridge no longer drives a future
  on a current-thread runtime from within another executor (spawn a
  dedicated thread + runtime when not on a multi-thread runtime).

Tested by query_engine::default_serializer unit tests: nested MergeScan
round-trip and engine-catalog resolution with mismatched column names.
No user-facing behavior change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep GreptimeDB function bindings after rebuilding decode state

SessionStateBuilder::new_from_existing collects the function registry
(HashMap) into a Vec and build() re-registers each entry; register_udf
registers aliases before the function's own name and the Vec order is
unstable. Rebuilding the session state to install MergeScanAwareSerializer
after the GreptimeDB functions were registered could therefore rebind a
name/alias (e.g. date_format, an alias of DataFusion's to_char) back to
the built-in function. This path runs for every decoded plan, not only
nested MergeScan ones.

Extract the GreptimeDB function registration into register_greptime_functions
and re-apply it after the final build of both the top-level decode state and
the MergeScan payload state. Add regression tests asserting date_format stays
bound to the GreptimeDB implementation on both paths.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): address review on nested MergeScan capability

Three follow-ups from review of the nested MergeScan execution capability:

1. Layer the datanode caches (P2b). table_cache derives from
   table_info_cache + table_name_cache, and partition_info_cache derives
   from table_route_cache, but they were all registered in a single flat
   CacheRegistry whose invalidate() runs concurrently with no ordering.
   A derived cache could be invalidated and refilled from a base cache
   that had not been invalidated yet, leaving stale metadata. Build the
   base caches and the derived caches as separate layers of a
   LayeredCacheRegistry (reusing the existing layering mechanism) so base
   caches finish invalidating before derived ones. Both the production
   path (cmd) and the test scaffolding now build the same layered
   registry; the flat build_datanode_cache_registry is removed.

2. Don't let a nested MergeScan's internal region query compete for the
   concurrency permit (P2a). An outer query holds its permit until its
   stream is fully consumed, and a nested MergeScan makes the datanode
   issue region queries to peer datanodes as execution stages of that
   same query. If such an inner stage also took a permit, the outer query
   waiting on it would deadlock against its own held permit (or time out
   when max_concurrent_queries > 0). Mark inner DN->DN region queries as
   internal stages: QueryRequest gains an  flag carried over
   the wire in the request header's query_context extensions (reserved
   key, not injectable via hints), and the region server skips acquiring
   a concurrency permit for internal-stage requests on both acquire
   sites. Outer queries still acquire normally.

3. Add an end-to-end capability test
   (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build
   Join(<local probe region>, MergeScan(build regions on both datanodes))
   with mismatched column names and duplicate build keys, ship it to a
   datanode over a real TCP Flight endpoint, consume the stream fully and
   compare the row multiset against the frontend's reference join result,
   assert a real cross-datanode DoGet via an RPC counter, and assert an
   inner region failure fails the whole query. Also migrate the
   with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt date_format regression test to the updated DataFusion fork

ScalarUDFImpl's trait bounds changed so its impl can no longer be
downcast via as_any(); assert the function equals the GreptimeDB
implementation instead, which to_char can never satisfy.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover the internal stage under the concurrency limit

Add an end-to-end regression test for the nested MergeScan capability
under max_concurrent_queries = 1: both datanodes' only permit is held by
the outer queries' streams while their inner MergeScans fetch the build
regions from the same datanodes over real Flight connections. The result
consumption is bounded by a timeout, so a broken link anywhere in the
chain (marking the inner request internal, carrying the marker in the
request header, honoring it at the receiving acquire sites) fails the
test instead of deadlocking.

Also add a minimal with_datanode_options_override hook to the cluster
builder, and fix the test module doc: the outer request is handed to
RegionServer::handle_remote_read in process; only the inner region
queries go over the wire.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(datanode): cover cold route loads under unrelated invalidations

The table route cache uses InitStrategy::VersionChecked, whose version
counter belongs to the whole CacheContainer rather than to a single
table: any TableId invalidation bumps the shared version, so a cold
load of one table's route retries when an unrelated table is
invalidated concurrently. Document this as a known limitation on the
region query handler, and pin the behavior with an integration test:
run the nested plans while a background task invalidates an unrelated
table id on every datanode, assert the results still match the expected
multiset (retries only add work, they never return a stale route), and
report the observed load amplification via the cache miss counter
(2 loads without invalidations, 8 over three runs with 24 unrelated
invalidations in this run). Also expose the datanode cache registries
on the test cluster so a test can drive the local invalidation path
that the heartbeat handler drives after a metasrv broadcast.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): trim nested plan decoder commentary

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: remove redundant nested scan support plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): decode nested MergeScan payloads asynchronously

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): avoid unnecessary extension name allocation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain extension decoding for manual session states

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(datanode): keep nested scan query admission unchanged

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): honor query handler factories and trim nested scan fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(datanode): retain required cache error import

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): reuse registered state for nested scan decoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): verify payload information schema table functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): retain nested scan runtime and cold-route guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): address nested scan review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(session): preserve read preference across region query hops

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(session): verify strict context rejection at Flow ingress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): remove redundant serializer error case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-10 06:49:38 +00:00
shuiyisong a907cbfbe4 perf: cache OTLP trace service and operation writes (#9432)
* perf: cache OTLP trace service and operation writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix CR issues for tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: bound asynchronous trace auxiliary writes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf: bound trace auxiliary cache by estimated memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: size trace auxiliary cache from host or pod memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add metrics for cache hit and miss

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: unify frontend cache metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: merge and reduce

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add init for the metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-10 02:45:43 +00:00
dennis zhuang e2fd8c5695 docs(config): align example configs and config.md with actual defaults (#9460)
* docs(config): align example configs and config.md with actual defaults

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(cmd): compare example defaults inside optional sections

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-08 07:23:22 +00:00
jeremyhi f9ab3c296b feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region

A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.

The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cmd): cover the best-effort standalone stop

Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.

Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 06:33:40 +00:00
Dragon RoarandDennis Zhuang d7f5331876 feat: add CORS support for gRPC-Web on the frontend gRPC server (#9374)
* feat: add CORS support for gRPC-Web on the frontend gRPC server

A browser-based gRPC-Web client cannot read a cross-origin response. The
request carries a non-simple content type, so the browser sends an
`OPTIONS` preflight first, and it drops the call when the response has no
`Access-Control-Allow-Origin`.

Add `enable_cors` and `cors_allowed_origins` to `[grpc]`, threaded from
`GrpcOptions` to `GrpcServerConfig` the same way `tls` is.

The CORS layer sits outside `tonic_web::GrpcWebLayer`, so it answers the
preflight before the request reaches the gRPC routes. It exposes
`grpc-status`, `grpc-message` and `grpc-status-details-bin`, which a
browser cannot read otherwise. An empty `cors_allowed_origins` allows any
origin.

`enable_cors` defaults to false, unlike `[http]`. `GrpcOptions` is shared
by the public `[grpc]` section and the internal `[internal_grpc]` one, and
serde cannot tell them apart, so a true default would also turn CORS on
for the internal gRPC listeners: the frontend internal gRPC server, and
the datanode and flownode gRPC servers. None of them authenticate callers,
and a browser on the host or in the cluster network can reach all of them.
Set `enable_cors = true` to turn it on.

Tests start a real server on an ephemeral port and cover the preflight, a
custom origin list, and the disabled case.

Update the example TOMLs and regenerate config/config.md.

Signed-off-by: lczllx <2181719471@qq.com>

* fix: drop the redundant OPTIONS from the gRPC CORS allow_methods

Signed-off-by: lczllx <2181719471@qq.com>

* test: assert the allow-headers and allow-methods headers in the gRPC CORS preflight

The preflight answers with `access-control-allow-headers: *` from
`AllowHeaders::any()`, and with `access-control-allow-methods: post` from the
POST-only `allow_methods` list. Pin both, the way the HTTP CORS test pins its
own headers: a missing allow-headers header would let the preflight pass the
origin check and still have the browser block every gRPC-Web call, which a
non-browser client would never notice.

Signed-off-by: lczllx <2181719471@qq.com>

* docs(config): stop the example configs from setting the new gRPC CORS keys

Signed-off-by: lczllx <2181719471@qq.com>

* test(servers): cover the exposed gRPC CORS headers and pin the option plumbing

Signed-off-by: lczllx <2181719471@qq.com>

* docs(config): document the gRPC CORS origin example with `#+`

Signed-off-by: lczllx <2181719471@qq.com>

* fix(frontend): keep CORS off the internal gRPC server

Signed-off-by: lczllx <2181719471@qq.com>

* docs(grpc): document that the internal gRPC server never serves CORS

Signed-off-by: lczllx <2181719471@qq.com>

* Update src/servers/src/grpc.rs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: trim redundant comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: share CORS origin parsing and simplify gRPC-Web CORS tests

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor: enable gRPC CORS only on the frontend public server

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(config): clarify the gRPC CORS options

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 10:40:17 +00:00
jeremyhi a99cb56d71 feat(datanode): wire the object store WAL into the datanode and standalone (#9386)
* feat(datanode): wire the object store WAL into the datanode and standalone

A datanode configured with `provider = "experimental_object_store"` now
builds an `ObjectStoreLogStore` instead of failing with the placeholder
"not supported yet" error. The builder resolves `storage_provider`
against the object store manager (empty selects the default store, an
unknown name is rejected as an invalid `storage_provider`), opens the
store under the node id and the standalone generation, stops it if a
later build step fails, and hands it to `Datanode`, whose shutdown stops
it after the region server. Datanode shutdown is now best-effort: every
step runs and the first error is returned. A datanode with a metasrv
client still rejects the provider.

The standalone bootstrap builds `WalProvider::ObjectStore` with the same
derived node prefix before the metasrv WAL conversion, so the WAL
options it allocates match the prefix the store runs under.

`log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's
test utilities can build an engine on the object store WAL over an
in-memory object store, and new engine tests cover create, reopen,
replay isolation between regions, batch open, drop and offline cleanup,
the latest entry id after replay, and the rejection of object store WAL
options on a Raft Engine log store.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(datanode): stop every region engine before the object store WAL

The region server returned at the first engine whose stop failed, so the
engines after it in map order, Mito included, were never stopped before
the datanode stopped the object store WAL. It now attempts every engine,
returns the first error and logs the later ones.

`ObjectStoreLogStore` is also exported from the `log-store` crate root,
and its callers import it from there.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-29 13:17:42 +00:00
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-28 12:35:47 +00:00
75bd8e9ce6 feat: add HDFS object storage backend (#8701)
* feat: add HDFS object storage backend

Signed-off-by: Minghan2005 <cambrianocean@gmail.com>

* fix: make HDFS storage operations durable

Gate the native HDFS backend behind an explicit feature. Publish writes through same-directory temporary files and atomic HDFS Rename2 replacement, and provide streaming copy fallback for COPY_REGION. Add regression coverage for interrupted writes and the region-copy path.

Signed-off-by: Minghan2005 <cambrianocean@gmail.com>

* ci: run HDFS object store tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: note HDFS temporary file cleanup follow-up

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: enable HDFS object storage by default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: remove redundant HDFS build feature notes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: Minghan2005 <cambrianocean@gmail.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Co-authored-by: Minghan2005 <cambrianocean@gmail.com>
Co-authored-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 08:39:48 +00:00
discord9 678aa81cae fix(client): complete transport lane isolation (#9030)
* fix(client): complete transport lane isolation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(client): deprecate legacy single-manager constructors

Per review: mark the legacy single-manager constructors and helper as
deprecated so callers move to the isolated query/control manager pair.
Tests intentionally exercising the legacy path are annotated with
`#[allow(deprecated)]`.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(client): update deprecated constructor callers for CI

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 04:39:20 +00:00
dennis zhuang b9f991502c refactor: remove experimental vector index (#9345)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-24 05:38:54 +00:00
Yingwen c7fa48ef95 feat(mito): limit approximate series index disk usage (#9313)
* feat(mito): limit series index disk usage

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): enforce series index quota during reconciliation

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): remove series index disk budget layer

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): simplify series index limit to estimated usage

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): trust index catalogs when loading snapshots

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito): minimize series index disk limit changes

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): keep series index cleanup running at capacity

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): avoid no-op index clones and stabilize capacity tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: update config API expectation and stabilize index build tests

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-23 14:18:42 +00:00
shuiyisong 20dde2601f feat: enable native histogram ingestion by default (#9301)
* feat: enable native histogram ingestion by default

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-09-23 13:03:51 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
Weny Xu 723da69b21 feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-22 14:38:17 +00:00
Lei, HUANG b5199bc59a feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: gate Jev filtering behind an opt-in Cargo feature

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: verify Jev registration with default features

Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: clarify Jev concurrency scope and stabilization work

Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: enable renamed ai-functions feature by default

Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: remove extra AI feature-off checks

Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: rename AI feature to ai_functions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-22 03:03:23 +00:00
discord9 7c7132ea65 refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): execute streaming flows with DataFusion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): remove legacy streaming runtime

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): avoid retrying stateless sink inserts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): align stateless writes with sink schema

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): reject stale stateless source schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): validate stateless flow routing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* Revert "test(mito2): cover regex inverted index pruning"

This reverts commit 79e96ac745.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): preserve source timestamps in stateless flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address stateless review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): recover stateless flows after schema changes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): serialize schema rebuilds and validate retained sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): verify streaming recovery and schema changes through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): cool down failed schema rebuilds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): explain legacy aggregate recreation requirements

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): adapt stateless provider downcast to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 06:44:01 +00:00
jeremyhi 7fb0d9350c feat(wal): add the object store WAL provider identity and configuration (#9203)
* feat(wal): add the object store WAL provider identity and configuration

Add the identity and configuration of the experimental object store WAL
without the log store implementation:

- store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a
  remote WAL.
- common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`)
  with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and
  `on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as
  `object_store` with the key `wal.object_store.prefix`. The metasrv config
  conversion rejects the new provider.
- common-meta, meta-srv, mito2: handle the new variants, map the region WAL
  options to the provider and reject them on a Raft Engine or Kafka log store.
- datanode: validate the configuration and fail with a "not supported yet"
  error until the log store lands.
- Example configs and the generated config docs.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(wal): accept object store WAL options when re-enabling WAL

Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-17 03:51:36 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
discord9 ba0f7acd93 feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): correct float encoding checks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): cover float SST encoding

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): compile float encoding tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): release parquet test writer

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): register float test primary key

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): verify BSS write lifecycles

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(metric-engine): verify BSS physical SST

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): verify bulk BSS lifecycle

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): compile bulk BSS test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): narrow bulk encoding constructors

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): accept generated float upgrade output

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): accept generated float downgrade output

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): narrow bulk encoding builder

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): add default versus BSS storage comparison

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): align BSS reader benchmarks with prior study

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): parse current read benchmark averages

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): retain default float encoding in direct SST fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): isolate BSS user SSTs and benchmark every file

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): record measured BSS storage and reader tradeoffs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): expose warm scan variability and evidence limits

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): clarify BSS baseline and storage measurement scope

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): model bounded mixed integer and fractional metric series

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): report bounded mixed BSS measurements and query regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(perf): qualify timings affected by concurrent host builds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): include float BSS comparison in default regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): omit unsupported float encoding option from baseline setup

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 12:05:42 +00:00
Lei, HUANG 21aed0371f feat(pprof): switch CPU profiler to framehop unwinder (#9125)
* feat(pprof): switch CPU profiler to framehop unwinder

Replace the default libgcc-based unwinder in pprof-rs with the
framehop unwinder, which is designed to be async-signal-safe:

- framehop performs no heap allocation during unwinding
  (MustNotAllocateDuringUnwind)
- It handles prologue/epilogue interruption correctly
- It falls back to frame-pointer unwinding when CFI is unavailable
- It does not depend on libgcc's unwind implementation, which is
  documented as not signal-safe (see tikv/pprof-rs#36)

Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof,
cmd, servers) to unify on a single version. pprof 0.15 also replaces
parking_lot with spin-rs to avoid a potential profiler deadlock (#268).

This addresses the libgcc_s.so.1 #GP crash observed in production
when CPU profiling is active, by eliminating the libgcc unwinder
from the signal handler path entirely.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(pprof): gate framehop-unwinder to supported targets

framehop-unwinder is only available on x86_64/aarch64 Linux/macOS.
Enabling it unconditionally for all Unix targets breaks the build on
riscv64 and other platforms: pprof disables its backtrace-rs fallback
when framehop-unwinder is set, but the framehop module is not compiled
on unsupported targets, leaving no TraceImpl implementation.

Split the pprof dependency: the base target.'cfg(unix)' block carries
the common features (flamegraph, prost-codec, protobuf), and a separate
target block adds framehop-unwinder only on supported targets.
Cargo unions features from both blocks on matching targets, so x86_64
and aarch64 Linux/macOS get the full feature set while other Unix
targets fall back to the default backtrace-rs implementation.

Addresses review comment discussion_r3987313632.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-11 09:39:13 +00:00
Lei, HUANG ab0b1f5cce fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix (#9103)
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix

Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from
0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle
tcache init failures gracefully').

On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that
adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before
enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1).
Without the ordering fix, a reentrant allocation during TSD bootstrap
(e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true
is active) can observe an enabled-but-uninitialized tcache, corrupting
per-thread tcache metadata and crashing the process in arena_stats_merge,
calloc, or the libgcc unwinder.

The patch is pinned by rev and should be removed once tikv/jemallocator
ships a jemalloc snapshot that includes 54f22c83.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1

GreptimeTeam/jemalloc#1 has been merged; point the patch at the
jemallocator commit referencing the merge commit on release-5.3.1.
Jemalloc source content is unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main

GreptimeTeam/jemallocator#1 has been merged; reference the merge
commit e1846d8c on main instead of the PR head branch. Jemalloc
source content is unchanged.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(servers): bump tikv-jemallocator dev-dependency to 0.7

main added a target.'cfg(not(windows))'.dev-dependencies entry on
tikv-jemallocator 0.6 for servers after this branch diverged. On the
merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which
conflicts with the patched 0.7 (links = "jemalloc" may only appear
once in the dependency graph), failing version selection in CI.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-10 13:06:43 +00:00
discord9 da5cb1a190 perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: guard instant last row correctness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update instant query explain results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: apply last row after source deduplication

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: scope post-merge last row selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover instant PromQL last row

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): sort generated SST rows before writing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): cover instant last row selection in sqlness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): avoid last row hints for lossy timestamp casts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): preserve stale marker semantics across flushes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): decode dictionary labels in stale regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): avoid reserved column name in stale fixture

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): exercise LastRow hints and filtered results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: keep after-merge mode in LastRow selector

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: reject instant LastRow across residual filters

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: expect after-merge selector in instant vector guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: explain instant LastRow filter eligibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restrict instant LastRow to safe selector nodes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: show LastRow merge mode directly in diagnostics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh LastRow display in explain expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-09 12:38:52 +00:00
Yingwen 9940de4917 feat: add scanbench query suites and structured results (#9050)
* perf: add scanbench partition diagnostics

Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes.

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support scanbench query suites

Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb)
Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: align scanbench port with existing engine metrics

Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option.

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-09-07 08:28:35 +00:00
XuanwoandWenyXu 4c12ea1aba chore(deps): bump opendal to 0.58.1 (#8742)
* chore(deps): bump opendal to 0.58.1

Upgrade direct opendal dependency and workspace object_store_opendal pin
from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal
0.58.0). Adapt to OpenDAL 0.58 composition API:

- Operator::new returns a finished operator; drop .finish() call sites
- Replace HttpClientLayer / raw::HttpClient with OperationContext +
  HttpTransporter (ReqwestTransport)
- Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to
  Service + Layer::apply_service
- Rewrite SecureFs reader/writer/lister for sync factories and StreamRead
- Use OperatorInfo::capability() instead of removed native_capability()

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: retrigger CI after udeps runner segfault

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(object-store): restore suffix read simulation for secure fs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adapt remaining callers to OpenDAL 0.58

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
2026-09-07 08:16:47 +00:00
discord9 27a7047f31 feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): cover preserve row sequence table alter

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): clear preserve_row_sequence marker on copy_region_from

copy_region_from copies source FileMeta into the target region, which has an
independent sequence domain. The physical per-row sequences in the copied
file belong to the source region only; trusting them in the target would let
an exact sequence-range request replay source-domain rows as if they were
target sequences. Clear the preserve_row_sequence marker on copied files so
the target fails closed with SequenceRangeUnsupported until the scan provably
cannot intersect the copied rows.

Add a regression test: copying from a preserve-enabled source into a
preserve-enabled target clears the marker, and an exact (2, 7] request on the
target returns SequenceRangeUnsupported instead of replaying source rows.

Fixes #8865

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: remove redundant doc comments for exact sequence range options

Approved comment-cleanup-only changes for #8865: drop outdated doc
summaries duplicated on the exact_sequence_range wrapper and the
preserve_row_sequence field, drop pure-restatement doc comments on the
SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and
remove the four structural SQL comments from the alter_preserve_row_sequence
case. No behavior changes; .result regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): fail closed exact reads on copied files and extension ranges

Address review feedback on #8865:

- copy_region_from: clear the source-domain FileMeta::sequence along with
  the preserve_row_sequence marker. An unmarked file retaining a stale
  source-domain max sequence could be silently skipped by
  files_allow_exact_sequence_range() as 'proven disjoint' in the target's
  independent sequence domain, dropping rows on exact (C, H] reads. With
  sequence=None the capability check fails closed (SequenceRangeUnsupported)
  until the copied rows are provably disjoint.
- Engine/reader: reject exact sequence-range reads whenever a follower
  region has an extension range provider attached. Extension streams are
  returned without a row-level sequence filter, so exactness cannot be
  proven; treat the capability as missing (fail closed) instead of emitting
  out-of-range rows. The reader also fails closed as defense in depth.

Tests: extend copy_region_from regression to assert the copied file's
sequence hint is cleared; mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): use doc comments for test function descriptions

Elevate the block comments describing test functions (in scan_test and
copy_region_from_test) to /// doc comments, matching the convention used
elsewhere in the exact sequence range change. No logic change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): extract helpers and trim comment noise in exact sequence reads

PR finalization for #8865 (zero behavior change, full mito2 suite green):

- engine: extract validate_sequence_fences and
  sequence_range_unsupported_reason, keeping error variants, check order
  and reason strings identical; OSS binds the extension blocker to false.
- handle_copy_region: extract remap_copied_file_meta and
  file_descriptors_for_meta; rename file_ids -> source_file_ids and
  files_to_copy -> new_file_metas.
- compactor: rename max_input_sequence -> known_max_input_sequence,
  document the None semantics (empty input vs unknown sequence).
- Remove restating/outdated comments (ScanInput::sequence_range doc
  first line, outdated file-pruning note, options test restatements),
  compress verbatim comments while keeping why/invariants/contracts.

Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt,
git diff --check, mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): reject foreign-region SSTs in exact sequence reads

Reading an SST whose FileMeta.region_id differs from the scanned region
means the region's sequence domain is broken (manifest corruption or a
repartition/copy path that leaked a source-domain file). Treat this as
an explicit RegionSequenceDomainBroken error instead of silently
ignoring the file's sequence or falling back to a full scan: the region
is unusable for exact sequence-range reads until the foreign lineage is
compacted away or repaired.

- files_allow_exact_sequence_range / exact_sequence_range now return
  Result and propagate the error through engine fence validation and
  scan construction (StatusCode::Internal, distinct from the
  fallback-capable SequenceRangeUnsupported).
- Row-level flat-batch sequence filtering rejects foreign-region files
  as defense in depth.
- Engine test asserts the broken-domain error rather than
  Unsupported/fallback.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping

An unmarked file's FileMeta.sequence may be synthesized by the
region-edit or repartition paths (committed+1 import barrier), not a
physical max of its rows. Treating it as a whole-file disjoint proof
could permanently skip rows that were never incrementally consumed
once the flow checkpoint passes that value.

Exact sequence-range capability now requires every SST in the region to
carry the preserve_row_sequence marker; any unmarked file disables
exactness (fallback), and the (C, H] file-selection skip also only
applies to marked files. Foreign-region files still raise
RegionSequenceDomainBroken as before.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): scope exact sequence-range capability to the time-selected read set

The exact capability check used to walk the entire SstVersion, so a
single unmarked or foreign-region SST anywhere in the region disabled
exact reads or raised RegionSequenceDomainBroken even when the
request's time range could never touch that file.

Both the engine fence and the scan builder now derive the read set with
shared time-pruning + exact-min/sst-min selection and validate
capability only over the files actually selected: a time-pruned file
cannot contribute a row to (C, H], so it cannot affect exactness. The
existing fail-loud semantics are unchanged for every selected file
(foreign region id -> RegionSequenceDomainBroken; unmarked -> exact
unavailable).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): wash untrusted sequences in compaction and restore barrier skipping

Compaction with any non-preserved input now writes a sequence-less
output: the physical __sequence column is zeroed (the flat format
requires the internal columns) and FileMeta.sequence records the
region-local admission barrier committed_sequence + 1 (falling back to
the flushed frontier). preserve_row_sequence stays false.

Exact sequence-range scans interpret an unmarked file's sequence as an
admission barrier: barrier <= C means flow has already consumed the
whole file, so it is skipped at file level; a missing or newer barrier
fails closed. Foreign-region files stay in the selected read set so the
capability fence still raises RegionSequenceDomainBroken.

This closes the recovery loop: after a region repartition, one
time-scoped fallback consumes the migrated rows, then compaction washes
the untrusted per-row sequences away and exact incremental reads resume
via file-level barrier skipping.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): drop restating comments in known_max_input_sequence tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): trim SQLness result EOF whitespace

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): reuse exact scan file selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): strengthen sequence scan coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): trim ALTER option comments

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): remove no-op bulk compaction check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve trusted row sequences when reading SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve target sequence domain for imported SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): add trailing blank line to SQLness result EOF

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): trim exact sequence scan plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): fold exact SST selection checks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make legacy compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make PK compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-03 06:20:09 +00:00
discord9andRuihang Xia 43c30d1446 feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics

Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to dynamic-config revision

Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): runtime-adjustable workload scheduler parameters

Expose dynamic adjustment of the experimental workload scheduler at
runtime:

- common-runtime: set_workload_scheduler_weights and
  set_workload_scheduler_max_concurrent_polls, which forward to the
  catio scheduler's set_weight/set_max_concurrent_polls when the
  scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
  /debug/workload_scheduler/max_concurrent_polls POST handlers, so
  operators can rebalance query/write shares or admission concurrency
  without restarting the datanode.

Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): add GET /debug/workload_scheduler status endpoint

Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to time-accounting revision

Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to lock-free sampling revision

Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio to scheduler PR head

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): add scheduler bypass control

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: advance catio scheduler fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: regenerate config docs for workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio scheduler test fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(http): satisfy scheduler lifecycle clippy

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: add distributed scheduler toggle coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: finalize workload scheduler runtime controls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio atomic weights

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: preserve unrelated lockfile resolution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(runtime): downsample scheduler time accounting

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(runtime): verify cross-runtime scheduler progress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): configure scheduler poll sampling

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): clarify scheduler activation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): explain scheduler use case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2026-09-02 07:02:39 +00:00
Lei, HUANG f5212d3631 feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-09-01 08:30:05 +00:00
discord9 00d43b29ad feat(query): add experimental DataFusion spill-to-disk controls (#8884)
* feat(query): add experimental DataFusion spill-to-disk controls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(config): regenerate configuration reference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update config API for spill defaults

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): address spill configuration review

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): preserve spill settings with runtime plugins

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-01 07:40:55 +00:00
discord9 51b94bb73f fix(cmd): gate daemon integration test on Unix (#8987)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-08-31 07:02:44 +00:00
zhaiAohan de6a903cb1 feat: Add built-in daemon mode for standalone service (#8960)
* feat: Add built-in daemon mode for standalone service

Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`.

- Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built.
- stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender.
- On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground).

Closes #7314

Signed-off-by: tian1220A <1573612521@qq.com>

* test(cmd): add integration test for standalone daemon mode

Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking.

Signed-off-by: tian1220A <1573612521@qq.com>

* fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt

Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional.

Also fix the rustfmt formatting issues that caused fmt-check to fail.

Signed-off-by: tian1220A <1573612521@qq.com>

* chore(cmd): declare daemonize dependency directly in cmd

daemonize is only used by the cmd crate. Move it out of the workspace
dependencies and declare it directly in the unix-only dependencies of
src/cmd/Cargo.toml.

Signed-off-by: tian1220A <1573612521@qq.com>

---------

Signed-off-by: tian1220A <1573612521@qq.com>
2026-08-30 05:26:00 +00:00
Lei, HUANG c0b8612c41 fix(cmd): configure meta client in frontend plugin test (#8977)
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-08-28 13:45:06 +00:00