rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-14 07:52:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,065 Commits 266 Branches 278 Tags
6b7377293cbee346d6e32f519587dcc56c4cdc0e
Commit Graph
607 Commits
Author SHA1 Message Date
discord9 1572325f55 refactor: keep after-merge mode in LastRow selector 
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-09 17:25:39 +08:00
discord9 1d6bd1aa5c fix(perf): sort generated SST rows before writing 
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-09 17:25:38 +08:00
Yingwen 9940de4917 feat: add scanbench query suites and structured results (#9050) 
* perf: add scanbench partition diagnostics

Port benchmark source and documentation from c5dd5e6356a8903f1dcf493eb46dc318612dc953. Exclude the Mito metrics changes.

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support scanbench query suites

Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit def9c22069b0d29c611695a68ad1b31110c845cb)
Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: align scanbench port with existing engine metrics

Remove documentation and fixture references to unported Mito metrics. Enable dev-tools in the build example and remove the obsolete force-flat-format option.

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-09-07 08:28:35 +00:00
XuanwoandWenyXu 4c12ea1aba chore(deps): bump opendal to 0.58.1 (#8742) 
* chore(deps): bump opendal to 0.58.1

Upgrade direct opendal dependency and workspace object_store_opendal pin
from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal
0.58.0). Adapt to OpenDAL 0.58 composition API:

- Operator::new returns a finished operator; drop .finish() call sites
- Replace HttpClientLayer / raw::HttpClient with OperationContext +
  HttpTransporter (ReqwestTransport)
- Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to
  Service + Layer::apply_service
- Rewrite SecureFs reader/writer/lister for sync factories and StreamRead
- Use OperatorInfo::capability() instead of removed native_capability()

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: retrigger CI after udeps runner segfault

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(object-store): restore suffix read simulation for secure fs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adapt remaining callers to OpenDAL 0.58

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2026-09-07 08:16:47 +00:00
discord9 27a7047f31 feat: preserve row sequences and support exact sequence-range reads (#8865) 
* feat(mito2): support exact sequence range reads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): cover preserve row sequence table alter

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): clear preserve_row_sequence marker on copy_region_from

copy_region_from copies source FileMeta into the target region, which has an
independent sequence domain. The physical per-row sequences in the copied
file belong to the source region only; trusting them in the target would let
an exact sequence-range request replay source-domain rows as if they were
target sequences. Clear the preserve_row_sequence marker on copied files so
the target fails closed with SequenceRangeUnsupported until the scan provably
cannot intersect the copied rows.

Add a regression test: copying from a preserve-enabled source into a
preserve-enabled target clears the marker, and an exact (2, 7] request on the
target returns SequenceRangeUnsupported instead of replaying source rows.

Fixes #8865

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: remove redundant doc comments for exact sequence range options

Approved comment-cleanup-only changes for #8865: drop outdated doc
summaries duplicated on the exact_sequence_range wrapper and the
preserve_row_sequence field, drop pure-restatement doc comments on the
SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and
remove the four structural SQL comments from the alter_preserve_row_sequence
case. No behavior changes; .result regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): fail closed exact reads on copied files and extension ranges

Address review feedback on #8865:

- copy_region_from: clear the source-domain FileMeta::sequence along with
  the preserve_row_sequence marker. An unmarked file retaining a stale
  source-domain max sequence could be silently skipped by
  files_allow_exact_sequence_range() as 'proven disjoint' in the target's
  independent sequence domain, dropping rows on exact (C, H] reads. With
  sequence=None the capability check fails closed (SequenceRangeUnsupported)
  until the copied rows are provably disjoint.
- Engine/reader: reject exact sequence-range reads whenever a follower
  region has an extension range provider attached. Extension streams are
  returned without a row-level sequence filter, so exactness cannot be
  proven; treat the capability as missing (fail closed) instead of emitting
  out-of-range rows. The reader also fails closed as defense in depth.

Tests: extend copy_region_from regression to assert the copied file's
sequence hint is cleared; mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): use doc comments for test function descriptions

Elevate the block comments describing test functions (in scan_test and
copy_region_from_test) to /// doc comments, matching the convention used
elsewhere in the exact sequence range change. No logic change.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): extract helpers and trim comment noise in exact sequence reads

PR finalization for #8865 (zero behavior change, full mito2 suite green):

- engine: extract validate_sequence_fences and
  sequence_range_unsupported_reason, keeping error variants, check order
  and reason strings identical; OSS binds the extension blocker to false.
- handle_copy_region: extract remap_copied_file_meta and
  file_descriptors_for_meta; rename file_ids -> source_file_ids and
  files_to_copy -> new_file_metas.
- compactor: rename max_input_sequence -> known_max_input_sequence,
  document the None semantics (empty input vs unknown sequence).
- Remove restating/outdated comments (ScanInput::sequence_range doc
  first line, outdated file-pruning note, options test restatements),
  compress verbatim comments while keeping why/invariants/contracts.

Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt,
git diff --check, mito2 suite 1148/1148 passing.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): reject foreign-region SSTs in exact sequence reads

Reading an SST whose FileMeta.region_id differs from the scanned region
means the region's sequence domain is broken (manifest corruption or a
repartition/copy path that leaked a source-domain file). Treat this as
an explicit RegionSequenceDomainBroken error instead of silently
ignoring the file's sequence or falling back to a full scan: the region
is unusable for exact sequence-range reads until the foreign lineage is
compacted away or repaired.

- files_allow_exact_sequence_range / exact_sequence_range now return
  Result and propagate the error through engine fence validation and
  scan construction (StatusCode::Internal, distinct from the
  fallback-capable SequenceRangeUnsupported).
- Row-level flat-batch sequence filtering rejects foreign-region files
  as defense in depth.
- Engine test asserts the broken-domain error rather than
  Unsupported/fallback.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping

An unmarked file's FileMeta.sequence may be synthesized by the
region-edit or repartition paths (committed+1 import barrier), not a
physical max of its rows. Treating it as a whole-file disjoint proof
could permanently skip rows that were never incrementally consumed
once the flow checkpoint passes that value.

Exact sequence-range capability now requires every SST in the region to
carry the preserve_row_sequence marker; any unmarked file disables
exactness (fallback), and the (C, H] file-selection skip also only
applies to marked files. Foreign-region files still raise
RegionSequenceDomainBroken as before.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): scope exact sequence-range capability to the time-selected read set

The exact capability check used to walk the entire SstVersion, so a
single unmarked or foreign-region SST anywhere in the region disabled
exact reads or raised RegionSequenceDomainBroken even when the
request's time range could never touch that file.

Both the engine fence and the scan builder now derive the read set with
shared time-pruning + exact-min/sst-min selection and validate
capability only over the files actually selected: a time-pruned file
cannot contribute a row to (C, H], so it cannot affect exactness. The
existing fail-loud semantics are unchanged for every selected file
(foreign region id -> RegionSequenceDomainBroken; unmarked -> exact
unavailable).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): wash untrusted sequences in compaction and restore barrier skipping

Compaction with any non-preserved input now writes a sequence-less
output: the physical __sequence column is zeroed (the flat format
requires the internal columns) and FileMeta.sequence records the
region-local admission barrier committed_sequence + 1 (falling back to
the flushed frontier). preserve_row_sequence stays false.

Exact sequence-range scans interpret an unmarked file's sequence as an
admission barrier: barrier <= C means flow has already consumed the
whole file, so it is skipped at file level; a missing or newer barrier
fails closed. Foreign-region files stay in the selected read set so the
capability fence still raises RegionSequenceDomainBroken.

This closes the recovery loop: after a region repartition, one
time-scoped fallback consumes the migrated rows, then compaction washes
the untrusted per-row sequences away and exact incremental reads resume
via file-level barrier skipping.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): drop restating comments in known_max_input_sequence tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): trim SQLness result EOF whitespace

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): reuse exact scan file selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): strengthen sequence scan coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style(mito2): trim ALTER option comments

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): remove no-op bulk compaction check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve trusted row sequences when reading SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(mito2): preserve target sequence domain for imported SSTs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): add trailing blank line to SQLness result EOF

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore(mito2): trim exact sequence scan plumbing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(mito2): fold exact SST selection checks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make legacy compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(mito2): make PK compaction rewrite deterministic

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-03 06:20:09 +00:00
discord9andRuihang Xia 43c30d1446 feat(runtime): add weighted workload scheduler (#8736) 
* feat(runtime): add weighted workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics

Use the GreptimeTeam/catio fork (pinned c20eafc) which adds
ClassStats::total_admission_wait and ClassStats::admitted, recorded
at each QUEUED -> ADMITTED transition. This exposes the scheduler's
own admission delay (excluding Tokio queueing and poll execution),
enabling admission-wait based fairness gates.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to dynamic-config revision

Bump the catio scheduler fork to 9f4b028 which adds
Scheduler::set_weight and Scheduler::set_max_concurrent_polls for
runtime configuration.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): runtime-adjustable workload scheduler parameters

Expose dynamic adjustment of the experimental workload scheduler at
runtime:

- common-runtime: set_workload_scheduler_weights and
  set_workload_scheduler_max_concurrent_polls, which forward to the
  catio scheduler's set_weight/set_max_concurrent_polls when the
  scheduler is enabled and reject zero values.
- servers: /debug/workload_scheduler/weights and
  /debug/workload_scheduler/max_concurrent_polls POST handlers, so
  operators can rebalance query/write shares or admission concurrency
  without restarting the datanode.

Both endpoints return 400 with a clear reason when the scheduler is
disabled or the requested value is invalid.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(perf): add GET /debug/workload_scheduler status endpoint

Returns the current weights (per class), max_concurrent_polls,
active_polls and per-class counters (queued, tasks, wakes, polls,
completed, cancelled, admitted, total_admission_wait) as JSON. When the
scheduler is disabled, returns enabled=false with the other fields
omitted, so operators can distinguish 'disabled' from an error.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to time-accounting revision

Bump the catio scheduler fork to 257ba56 which replaces
admission-count accounting with real execution-time accounting
(pass += exec_time / (weight * concurrency)), so CPU share follows the
configured weights regardless of poll length.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: bump catio to lock-free sampling revision

Bump the catio scheduler fork to efdc0a4 which adds an optional
downsampled clock sampling mode (SchedulerBuilder::sample_every_polls,
default off) with a lock-free per-class atomic counter, so the
downsampled path costs one fetch_add per poll instead of a global
mutex.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio to scheduler PR head

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): add scheduler bypass control

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: advance catio scheduler fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: regenerate config docs for workload scheduler

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin catio scheduler test fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(http): satisfy scheduler lifecycle clippy

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: add distributed scheduler toggle coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: finalize workload scheduler runtime controls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: pin merged catio atomic weights

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: preserve unrelated lockfile resolution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(runtime): downsample scheduler time accounting

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(runtime): verify cross-runtime scheduler progress

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(runtime): configure scheduler poll sampling

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): clarify scheduler activation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(runtime): explain scheduler use case

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2026-09-02 07:02:39 +00:00
Lei, HUANG f5212d3631 feat(mito2): add write cache upload hook (#8992) 
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-09-01 08:30:05 +00:00
discord9 00d43b29ad feat(query): add experimental DataFusion spill-to-disk controls (#8884) 
* feat(query): add experimental DataFusion spill-to-disk controls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(config): regenerate configuration reference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update config API for spill defaults

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): address spill configuration review

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): preserve spill settings with runtime plugins

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-01 07:40:55 +00:00
discord9 51b94bb73f fix(cmd): gate daemon integration test on Unix (#8987) 
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-08-31 07:02:44 +00:00
zhaiAohan de6a903cb1 feat: Add built-in daemon mode for standalone service (#8960) 
* feat: Add built-in daemon mode for standalone service

Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`.

- Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built.
- stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender.
- On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground).

Closes #7314

Signed-off-by: tian1220A <1573612521@qq.com>

* test(cmd): add integration test for standalone daemon mode

Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking.

Signed-off-by: tian1220A <1573612521@qq.com>

* fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt

Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional.

Also fix the rustfmt formatting issues that caused fmt-check to fail.

Signed-off-by: tian1220A <1573612521@qq.com>

* chore(cmd): declare daemonize dependency directly in cmd

daemonize is only used by the cmd crate. Move it out of the workspace
dependencies and declare it directly in the unix-only dependencies of
src/cmd/Cargo.toml.

Signed-off-by: tian1220A <1573612521@qq.com>

---------

Signed-off-by: tian1220A <1573612521@qq.com>
 2026-08-30 05:26:00 +00:00
Lei, HUANG c0b8612c41 fix(cmd): configure meta client in frontend plugin test (#8977) 
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-28 13:45:06 +00:00
Yingwen ba3c5a939e chore(mito2): reduce default auto flush interval (#8971) 
Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-08-28 05:56:44 +00:00
Lei, HUANG a2f39ecf7b feat: harden frontend heartbeat extensions (#8803) 
Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`.

Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-27 17:27:52 +00:00
discord9 beded6e232 fix(flow): avoid insert select HTTP/2 stalls (#8962) 
* fix(flow): avoid insert select HTTP/2 stalls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): share record batch forwarding for DML

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-08-27 04:51:39 +00:00
Yingwen 046862f067 feat(cmd): improve parquet rewrite fidelity and scanbench output (#8947) 
* feat(cmd): preserve parquet rewrite metadata settings

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(cmd): print scanner metrics in verbose scanbench

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(cmd): preserve SST truncation settings in property dumps

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-08-25 09:22:53 +00:00
discord9 3493d2d0fb perf(servers)!: speed up Prometheus JSON response building with ryu and per-series entry reuse (#8815) 
* fix(perf): align direct-SST CREATE TABLE with baked index metadata

The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.

CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse

PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).

- Replace f64::to_string() with ryu::Buffer::format_finite for finite
  values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
  previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
  is clustered by series, so consecutive rows reuse the same IndexMap
  entry via get_index_mut instead of rebuilding and hashing the label
  vector (worst case adds one Vec comparison per series transition).

Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.

Perf (aligned base d90cca4b75, 256 series x 481 points):
- prom_range_2h (JSON response path): 29.31ms -> 21.46ms (-26.8%)
- tql_range_2h_control (non-JSON path): +1.87% (noise)

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(servers): keep Prometheus wire format for integral floats with ryu

ryu::Buffer::format_finite prints integral values as "1.0", but the
Prometheus JSON wire format (matching std f64::to_string) expects "1".
Strip the trailing ".0" that ryu only emits for integral values; extreme
values keep ryu scientific notation, and NaN/Inf keep std output. Adds
wire-format tests covering 1.0, 0.0, -0.0, 1.5, 0.1, 1e21, 1e30, 1e-7,
f64::MAX, NaN, ±Inf.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(servers): address Prometheus response review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(servers)!: use ryu for Prometheus sample values

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(cmd): skip Prometheus execution time extraction

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-08-25 05:16:50 +00:00
Yingwen 0cc83c4570 feat(cmd): add parquet development tools (#8939) 
* feat(cmd): add parquet metadata development tool

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(cmd): add parquet rewrite development tool

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(cmd): add SST replacement development tool

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(cmd): support local parquet files in parquetbench

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(cmd): clean up parquet development tools

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(cmd): share datanode tool utilities

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(cmd): satisfy parquet tool lints

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: document parquet development tools

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(cmd): rename SST replacement module

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(cmd): validate parquet rewrite options

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-08-24 13:34:30 +00:00
shuiyisong 1c5eabcbbf feat(otlp): support cumulative exponential histograms (#8900) 
* feat: implement exponential histogram

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove duplicate tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): enforce exponential histogram ingestion safety

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update rfc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): remove protocol-coupled histogram checks

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(otlp): reuse native histogram schema across data points

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: merge repeated OTLP histogram fragments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(otlp): build rejection messages lazily

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: add doc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-08-24 12:59:44 +00:00
Lei, HUANG 2182dccd9b fix: cap default runtime sizes to a minimum of 2 threads (#8908) 
* fix: cap default runtime sizes to a minimum of 2 threads

RuntimeOptions derived its default sizes directly from num_cpus. On
single-core machines every runtime (global, compact, query, ingest)
ended up with one worker thread, which can easily deadlock async code
(e.g. block_on combined with spawn).

Clamp all CPU-derived runtime sizes to at least 2 threads.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: init logging before runtimes so runtime options are logged

The global runtimes were initialized before the global logging
subscriber, so the "Creating runtime ..." info logs that carry the
runtime sizes were silently dropped. Initialize logging first in all
node start paths; common-telemetry has no dependency on
common-runtime, so the reorder is safe.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-19 10:04:50 +00:00
fys 6538db6d61 refactor(json2): push down json2 type hints to parquet reads (#8833) 
* refactor(mito): push down json2 type hints to parquet reads

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share json2 target types with arc

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): derive json2 output schema from target types

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): simplify read columns construction

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cargo check

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito): reject JSON hints for non-JSON2 read columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix: do not pushdown json type hint of non-json2-col

Signed-off-by: fys <fengys1996@gmail.com>

* fix: unit test

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(query): simplify JSON type hint application

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: clean up JSON2 type hint handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): keep JSON2 hints with flat read format

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito): use raw parquet projection for output schema

Signed-off-by: fys <fengys1996@gmail.com>

* fix(query): note JSON2 hint scope limitation

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): store JSON target types as native types

Signed-off-by: fys <fengys1996@gmail.com>

* test(json2): cover join hint qualifier limitation

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): remove JSON2 fallback from compat cast

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito): document ReadColumns ordering contract

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cargo clippy

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
 2026-08-14 06:21:17 +00:00
Weny Xu 943eee852f feat(event): record admin function executions (#8835) 
* feat(event): record admin function executions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): handle admin function recording edge cases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): record actor for admin functions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve admin function event values

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve non-finite admin results

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-11 12:50:50 +00:00
shuiyisongandfys d4af650ec0 perf: reduce cold workspace compile time (#8801) 
* refactor: remove datanode and meta-srv dep from frontend

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: use on-device protoc if possible

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: remove unused dep

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: version and docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update logs

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
 2026-08-10 07:18:16 +00:00
discord9 1693b2727c refactor: port query regression runner to Rust (#8651) 
* refactor: port query regression runner to Rust

Signed-off-by: discord9 <discord9@163.com>

* ci: remove optional OTLP report plotter

Signed-off-by: discord9 <discord9@163.com>

* refactor: split query regression runner into modules

Signed-off-by: discord9 <discord9@163.com>

* style: use crate-qualified imports in query regression runner

Signed-off-by: discord9 <discord9@163.com>

* refactor: simplify query regression runner internals

Signed-off-by: discord9 <discord9@163.com>

* feat: abstract inspect-footer storage access behind object store destination

Add an optional --destination <TOML> to inspect-footer (and
--base-destination/--candidate-destination to finalize-remote) so the
storage inspection reads DB data files through the opendal-backed
object_store abstraction instead of bare std::fs. Local paths keep
working unchanged via the --root shortcut (File backend); remote
backends (S3/GCS/...) are described by a DestinationConfig TOML
reusing the object-store crate's ObjectStoreConfig serde shape.

- inspect_footer: list via ObjectStore::list + ObjectMeta filtering
  (parquet keys, non-zero size, metadata/ segment), read footers
  async via ParquetObjectReader + ParquetMetaDataReader with known
  file size (no extra HEAD); output JSON schema unchanged
- finalize-remote: --base-data-home/--candidate-data-home become
  optional, mutually exclusive with the new --*-destination args
- cmd deps: add object_store_opendal + datafusion_object_store
- tests: fs-backend list+footer integration tests (metadata filtering,
  destination TOML mode, root/destination exclusivity)

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: drop needless borrow in inspect footer test

Fix clippy::needless_borrows_for_generic_args in the inspect-footer test
(fs::create_dir_all(table.join("metadata"))). Missed by the earlier
focused clippy run because it only covered --bin targets.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-08-06 13:01:32 +00:00
jeremyhi b30d17f89c docs: align wal.sync_period documented default with actual fallback (5s) (#8753) 
The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.

Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.

- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-08-05 08:56:55 +00:00
jeremyhi 9f724aa5f6 feat: make frontend heartbeat extensible and lifecycle-safe (#8726) 
* feat: make frontend heartbeat extensible and lifecycle-safe

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: isolate heartbeat extension response handlers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: cancel in-flight heartbeat response handling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover heartbeat wire compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed heartbeat startup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address frontend heartbeat review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-08-05 04:38:54 +00:00
Lei, HUANG c55f297dec chore!: gate soft-drop table behind the enterprise feature (#8747) 
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-05 04:28:30 +00:00
Lei, HUANG ff7e7f13b8 fix(mito2): limit compaction picker threads (#8704) 
* fix(mito2): limit compaction picker threads

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): extract TWCS input picking

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): make compaction picker async

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): remove redundant build_output test helper

After making the compaction picker async and extracting TWCS input
picking, the test-only build_output helper is just a thin wrapper around
build_output_with_time_range. Drop it and call the full method directly
from tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: correct compact runtime config wording

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(runtime): harden compact blocking limit check

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-31 15:15:55 +00:00
jeremyhi 448f973593 fix: sandbox SQL local filesystem access (#8708) 
* fix: sandbox SQL local filesystem access

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file sandbox review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: support Windows local copy paths

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve sandbox path errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: simplify local path error context

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf: stream secure filesystem listings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* style: derive local file access default

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: improve local file access errors

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address local file access review findings

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: simplify local file access coverage

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: harden sandboxed local file backends

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject directory copy targets before creation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid implicit string clone in file table listing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-07-31 13:23:15 +00:00
dennis zhuang f817690b82 fix(mito2): make async index publication conditional (#8676) 
* fix(mito2): make async index publication conditional

Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale.

Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(mito2): centralize manifest update finalization

Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): retain index build leases across reopen

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): avoid retiring scheduler on index failure

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): handle cross-region index publication

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): use physical region for index paths

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(cmd): update noop index builder

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): avoid reusing published index versions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore(mito2): log untracked index build stops

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(mito2): set compaction time range in index test

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-07-30 08:50:01 +00:00
Yingwen 7da4f46532 fix: configure datanode client gRPC message limits (#8642) 
Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-07-28 08:35:42 +00:00
Weny Xu 7344d47756 feat(event-recorder): configure lifecycle event recording (#8648) 
* refactor(event-recorder): centralize event table helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(procedure): wire lifecycle event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event-recorder): filter events by type

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): derive event type filter default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): decouple frontend filtering

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove docs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): complete configuration support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event-recorder): centralize filter ownership

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(config): update event recorder snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(frontend): decouple slow query event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-27 13:05:56 +00:00
fys 3e67c67607 refactor: simplify scan projection to root column indices (#8629) 
* refactor: simplify scan projection to root column indices

Signed-off-by: fys <fengys1996@gmail.com>

* fix: sqlness test

Signed-off-by: fys <fengys1996@gmail.com>

* chore: code style adjust

Signed-off-by: fys <fengys1996@gmail.com>

* fix: cr

Signed-off-by: fys <fengys1996@gmail.com>

* add comment and test

Signed-off-by: fys <fengys1996@gmail.com>

* minor change

Signed-off-by: fys <fengys1996@gmail.com>

* fix: unit test

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
 2026-07-27 12:04:19 +00:00
shuiyisong f5f5d468bb ci: add OTLP trace ingestion regression testing (#8631) 
* chore: update CI config

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add CI

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update CI config

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* ci: report otelgen runner diagnostics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add script to draw result diagram

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-07-26 15:08:27 +00:00
Yingwen 9b579c7618 feat: make parquet row group size configurable (#8446) 
* feat: make parquet row group size configurable via max_row_group_row_count region option

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: split inverted index segments across row group boundaries

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: verify max_row_group_row_count region option

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update row group size sqlness result

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): use parquet row group size for index rebuild

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: update query perf fixture indexer builder

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): clamp index selection to parquet rows

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support altering parquet row group size

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): handle idempotent append mode in mixed alters

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): stage mixed region option alters atomically

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): ignore inverted indexes with mismatched row counts

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-07-23 03:56:10 +00:00
fys 598a412fbb feat: add mysql object store backend (#8560) 
* feat: add mysql object store backend

Signed-off-by: fys <fengys1996@gmail.com>

* fix: code review

Signed-off-by: fys <fengys1996@gmail.com>

* add feature guard

---------

Signed-off-by: fys <fengys1996@gmail.com>
 2026-07-21 13:53:27 +00:00
shuiyisong 5065dfcf2f feat: grant creators access to newly created databases (#8566) 
* feat: implement auto database acl after create database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: timeout conversion:

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove register loaders parameter in ddl_manager initialization

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-07-21 11:29:49 +00:00
discord9 56addd0623 fix: stream remote analyze metrics while pending (#8405) 
* fix: stream remote analyze metrics while pending

Signed-off-by: discord9 <discord9@163.com>

* test: verify flight metrics preserve pending batch

Signed-off-by: discord9 <discord9@163.com>

* fix: preserve direct SST perf queries in plans

Signed-off-by: discord9 <discord9@163.com>

* fix: bind flight metrics capability to query

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-07-21 04:17:50 +00:00
Ning Sun e3d2864947 test: gate pluginoption test for oss repo only (#8569) 2026-07-20 15:27:05 +00:00
Ning Sun 63e63078b2 feat: allow unknown PluginOptions with a warning message (#8550) 
* feat: allow unknown PluginOptions with a warning message

* refactor(plugins): deserialize plugin option from borrowed Value

Address review feedback on #8550: in filter_known_plugin_options,
deserialize each entry from a borrowed serde_json::Value reference
instead of cloning it. This removes the per-entry Value clone on the
happy path while keeping the original value available for the warning
log when an entry is unrecognized.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: address review comments

* fix: verify against downstream project

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2026-07-20 11:16:20 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554) 
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-20 08:28:35 +00:00
Lei, HUANG 4873fc4f18 feat(meta): add retention GC for soft-dropped tables (#8526) 
* feat(meta): persist soft-drop retention metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): complete soft-drop retention coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): finalize soft-drop metadata after prepare

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): purge expired soft-dropped tables

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep soft-drop GC responsive

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): deduplicate soft-drop purge tasks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): release purge reservations safely

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(meta): simplify soft-drop GC wiring

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): persist soft-drop retention deadlines

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): retain soft-drop config for recovery

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete standalone DDL test context

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): recheck retention before automatic purge

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep experimental soft drop disabled

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): rotate soft-drop purge candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): address soft-drop retention review

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): protect retained table tombstones

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve post-cleanup purge state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): fence purge by drop generation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-16 08:47:03 +00:00
dennis zhuang 67683cef2e feat: support SCRAM auth for Postgres (#8304) 
* feat: support SCRAM auth for Postgres

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: add pg_scram_sha256 format to hash-password command

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: harden Postgres SCRAM auth

- Verify the client-final nonce matches the server-issued nonce, per RFC 5802
  transcript validation, instead of only checking the channel-binding field.
- Replace the per-connection PBKDF2 over a random password for unknown users
  with a deterministic mock verifier keyed by the username and a process-wide
  secret. This avoids a CPU-exhaustion DoS on unknown usernames and removes a
  username-enumeration oracle: the SCRAM server-first salt and iteration count
  are now stable per username and indistinguishable from a real user, with no
  PBKDF2 cost and random keys that never accept a proof.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* style: format PG_SCRAM_MOCK_SECRET declaration

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: precompute stable SCRAM verifier for plaintext users

Plaintext-backed credentials derived a Postgres SCRAM verifier on the fly
on every connection, using a fresh random salt and running PBKDF2 each
time. That made a known plaintext user distinguishable from stored-hash
and unknown (mock) users through both the unstable server-first salt and
the per-connection timing, enabling username enumeration.

Precompute the SCRAM verifier once at load time (stable salt, default
iteration count) and reuse it, matching the mock verifier handed to
unknown users. Document that non-default iteration counts remain
observable in the SCRAM handshake and weaken enumeration resistance.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: normalize passwords for Postgres SCRAM

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: docs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-07-15 11:49:43 +00:00
Ning SunCopilot Autofix powered by AIdennis zhuang
 		f8aff344e9 feat!: remove configuration of sparse_primary_key_encoding (#8470) 
* feat: remove configuration of sparse_primary_key_encoding, always use true

* Apply suggestions from code review

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix: merge

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-07-15 09:28:24 +00:00
Yingwen 0a002f4d65 feat: support per-region write buffer limits (#8473) 
* feat(mito): add per-region write buffer limit

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat(mito): add default region write buffer size

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: expand configuration change checklist

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: cover table write buffer size option

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): refine region write buffer checks

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): keep region-stalled writes queued

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): allow zero region write buffer size

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): validate region state before write stall

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): reject writes beyond region hard limit

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update example

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-07-15 07:31:25 +00:00
discord9 02283b6ba0 test(perf): add remote write storage inspection (#8444) 
* feat: add remote write value distributions

Signed-off-by: discord9 <discord9@163.com>

* test: extend remote write value distributions

Signed-off-by: discord9 <discord9@163.com>

* test: inspect remote write parquet storage

Signed-off-by: discord9 <discord9@163.com>

* test: normalize remote write perf fixtures

Signed-off-by: discord9 <discord9@163.com>

* test: add heavy remote write perf case

Signed-off-by: discord9 <discord9@163.com>

* test: use head greptime for read bench

Signed-off-by: discord9 <discord9@163.com>

* test: keep heavy remote write case local

Signed-off-by: discord9 <discord9@163.com>

* test: tune remote write perf smoke case

Signed-off-by: discord9 <discord9@163.com>

* test: fix query fixture import style

Signed-off-by: discord9 <discord9@163.com>

* test: cover remote write value distributions

Signed-off-by: discord9 <discord9@163.com>

* test: add integer counter perf case

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-07-10 12:27:27 +00:00
raphaelroshan 74bdb1f2a8 fix: compare all LoggingOptions fields in PartialEq (#8449) 
* fix: compare all LoggingOptions fields in PartialEq

Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>

* test: set otlp_export_protocol in load_config_test expected configs

The example configs set otlp_export_protocol = "http", but the expected
LoggingOptions blocks for datanode/frontend/metasrv/standalone left it at
the default None. Now that PartialEq compares all fields, add the field to
match the parsed configs (as the flownode case already does).

Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>

---------

Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
 2026-07-10 05:46:20 +00:00
discord9 12f83828b1 feat: add Prom remote-write query regression scenario (#8413) 
* feat: add Prom remote-write query regression scenario

Signed-off-by: discord9 <discord9@163.com>

* test: add high-cardinality remote-write query case

Signed-off-by: discord9 <discord9@163.com>

* feat: chunk remote-write query regression loads

Signed-off-by: discord9 <discord9@163.com>

* test: use multi-day remote-write regression case

Signed-off-by: discord9 <discord9@163.com>

* fix: address query regression review comments

Signed-off-by: discord9 <discord9@163.com>

* ci: allow large query regression comments

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-07-07 07:58:56 +00:00
Lei, HUANG 3758911a2e feat: add soft-drop table recovery procedures (#8061) 
* feat: add close-first soft-drop table flow
Soft-drop now tombstones table metadata and closes datanode regions instead of
issuing physical drop requests, while preserving hard-drop cleanup semantics and
blocking conflicting drops of recreated table names.
Files:
- `src/common/meta/src/ddl.rs`
- `src/common/meta/src/ddl/drop_table.rs`
- `src/common/meta/src/ddl/drop_table/executor.rs`
- `src/common/meta/src/error.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/meta-srv/src/metasrv/builder.rs`
- `src/cmd/src/standalone.rs`
- `src/common/meta/src/test_util.rs`
- `src/meta-srv/src/procedure/utils.rs`
- `tests-integration/src/standalone.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: add undrop and purge table procedures

Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf.

Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/common/meta/src/ddl.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl_manager.rs\`
- \`src/common/meta/src/rpc/ddl.rs\`
- \`src/common/meta/src/key.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`
- \`src/mito2/src/engine/open_test.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: reopen soft-dropped regions before purge

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: disable soft-drop operations for metric logical tables

Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases.

Files:
- `src/common/meta/src/ddl/drop_table/metadata.rs`
- `src/common/meta/src/ddl/purge_dropped_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/utils.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: sync failure detectors during soft-drop lifecycle

Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions.

Files:

- \`src/common/meta/src/ddl.rs\`

- \`src/common/meta/src/ddl/drop_table.rs\`

- \`src/common/meta/src/ddl/undrop_table.rs\`

- \`src/common/meta/src/ddl/purge_dropped_table.rs\`

- \`src/common/meta/src/ddl/tests/drop_table.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: simplify soft-drop table tests

Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused.

Files:

- `src/common/meta/src/ddl/tests/drop_table.rs`

- `src/common/meta/src/key.rs`

- `src/mito2/src/engine/open_test.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: reopen follower regions during undrop

Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed.

Files:

- `src/common/meta/src/ddl/undrop_table.rs`

- `src/common/meta/src/ddl/tests/drop_table.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: use dropped table ddl expr protos

Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API.

Files:

- `Cargo.toml`

- `Cargo.lock`

- `src/api/src/helper.rs`

- `src/common/meta/src/ddl/drop_table/executor.rs`

- `src/common/meta/src/rpc/ddl.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* revert: use inline dropped table task protos

Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code.

Files:

- `Cargo.toml`

- `Cargo.lock`

- `src/api/src/helper.rs`

- `src/common/meta/src/rpc/ddl.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: various soft-drop improvements

- Bump `greptime-proto` dependency revision
  (`Cargo.toml`, `Cargo.lock`)
- Pass `region_wal_options` directly without pre-serialization in undrop flow
  (`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`)
- Remove unused `RegionNumber` import
  (`src/common/meta/src/ddl/utils.rs`)
- Add `reset_failure_detectors` to test mock
  (`src/common/meta/src/ddl/tests/drop_table.rs`)
- Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask`
  (`src/common/meta/src/rpc/ddl.rs`)

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: validate table name match in undrop procedure

Add a guard in `UndropTableProcedure::on_prepare()` to check that the
dropped table name matches the undrop task name, returning `TableNotFound`
on mismatch.  This prevents undropping a table by a different name when
only the table ID is known.

- `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation
- `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: simplify UndropTableTask to use table_id only

Remove catalog, schema, and table name fields from `UndropTableTask`
since the table name can be derived from the dropped table metadata
in the procedure itself. This eliminates redundant fields and the
associated name-validation test.

Simplify locking in `UndropTableProcedure` to only use `TableLock`.

Update `greptime-proto` dependency revision.

- `Cargo.toml`, `Cargo.lock`
- `src/common/meta/src/rpc/ddl.rs`
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: detect table name conflict during tombstone restore in undrop

- Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore

- Added `TombstoneTargetAlreadyExists` error variant

- Map tombstone conflict to `TableAlreadyExists` in undrop procedure

- Added test for undrop failing when live name created after prepare

Files:

- `src/common/meta/src/ddl/tests/drop_table.rs`

- `src/common/meta/src/ddl/undrop_table.rs`

- `src/common/meta/src/error.rs`

- `src/common/meta/src/key/tombstone.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: make PurgeDroppedTableTask table_id-only

Streamline the purge-dropped-table flow by requiring a table_id
instead of allowing name-based fallback.

- Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs`
- Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs`
- Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs`
- Bumped `greptime-proto` dependency

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(tombstone): chunk values by per-key txn ops instead of fixed divisor

Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware
constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`)
to correctly account for per-key transaction operations. Added
`TxnOpLimitKvBackend` test helper and two new tests
(`test_restore_chunks_by_total_txn_ops_limit`,
`test_create_chunks_by_total_txn_ops_limit`) verifying chunking under
tight txn op limits.

Affected file:
- `src/common/meta/src/key/tombstone.rs` — chunk size fix,
  `TxnOpLimitKvBackend` helper, two new tests

Signed-off-by: Lei, HUANG &lt;ratuthomm@gmail.com&gt;
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge

- `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before
  transitioning to DeleteTombstone state
- `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into
  `open_regions_inner` with an `ignore_region_not_found` flag; expose
  `open_regions_ignore_region_not_found` for purge replayer
- `src/common/meta/src/ddl/purge_dropped_table.rs`: use
  `open_regions_ignore_region_not_found` in replayed purge procedures
- `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop
  idempotency and purge replay tolerance of dropped regions

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: fix clippy

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(soft-drop): open regions before restoring undrop metadata

Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(tombstone): fail fast on invalid txn op budget

Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump greptime-proto to main branch commit

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG &lt;ratuthomm@gmail.com&gt;
 2026-07-05 09:02:48 +00:00
discord9 c44f8da646 feat: add query regression perf harness (#8406) 
* feat: add query regression perf harness

Signed-off-by: discord9 <discord9@163.com>

* feat: extend query regression cases

Signed-off-by: discord9 <discord9@163.com>

* ci: harden query regression workflows

Signed-off-by: discord9 <discord9@163.com>

* fix: address query regression review comments

Signed-off-by: discord9 <discord9@163.com>

* ci: limit query regression PR triggers

Signed-off-by: discord9 <discord9@163.com>

* ci: run full query regression case set

Signed-off-by: discord9 <discord9@163.com>

* refactor: model query regression scenarios

Signed-off-by: discord9 <discord9@163.com>

* fix: avoid unenforced query regression thresholds

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-07-03 09:09:01 +00:00
Lei, HUANG a3461caf9d feat: expose region read load metrics (#8316) 
* feat: expose region read load through Prometheus metrics and heartbeat

Introduce region-level query load tracking (CPU time and scanned bytes)
collected by `RegionScanExec`, exposed via Prometheus metrics and optionally
reported through heartbeat region stats.

- **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add
  `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`,
  and `greptime_mito_region_written_bytes_since_open` gauge metrics.
- **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`,
  `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with
  `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle
  management (init, reset, remove-on-drop).
- **RegionStatistic** (`src/store-api/src/region_engine.rs`,
  `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and
  `query_scanned_bytes` fields.
- **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from
  metadata and data regions.
- **Heartbeat** (`src/datanode/src/heartbeat.rs`,
  `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat
  `RegionStat`; add test.
- **Query engine** (`src/query/src/options.rs`,
  `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`,
  `src/query/src/dist_plan/merge_scan.rs`,
  `src/query/src/dist_plan/analyzer.rs`,
  `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config;
  wire `RegionScanExec` to accumulate CPU time and scanned bytes.
- **Table scan** (`src/table/src/table/scan.rs`,
  `src/table/src/table/metrics.rs`): Wire table scan metrics.
- **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`,
  `config/frontend.example.toml`, `config/config.md`): Add example config and
  documentation for `enable_region_query_load_report`.
- **Tests** (`src/mito2/src/engine/basic_test.rs`,
  `src/mito2/src/engine/close_test.rs`,
  `src/cmd/tests/load_config_test.rs`,
  `src/flow/src/adapter.rs`): Add unit tests for region query load reporting
  and metric cleanup on region close; set default config values.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move region read load report config from query layer to mito engine

Move the `enable_region_query_load_report` setting from query-level config
(`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config
(`MitoConfig`), and expose it through the `RegionScanner` trait instead
of `ScanRequest`/`PrepareRequest`.

- Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs`
- Scan region plumbing: `src/mito2/src/read/scan_region.rs`
- RegionScanner trait: `src/store-api/src/region_engine.rs`
- Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs`
- RegionScanExec: `src/table/src/table/scan.rs`
- Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs`
- Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs`
- Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move region query load report config from MitoConfig to LoggingOptions

Relocate the `enable_region_query_load_report` setting from
`MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`),
and thread it into `MitoEngineBuilder` instead of reading from
the engine config directly. This makes the region read-load
reporting a per-node logging/observability concern rather than
a per-engine storage setting.

- `config/config.md`
- `config/datanode.example.toml`
- `config/standalone.example.toml`
- `src/common/telemetry/src/logging.rs`
- `src/datanode/src/datanode.rs`
- `src/mito2/src/config.rs`
- `src/mito2/src/engine.rs`
- `src/mito2/src/region.rs`

Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: report region query load on stream drop instead of stream end

Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()`
to `Drop::drop()` so that region query load is reported even when the
stream is dropped prematurely (not just when fully consumed).

Affected files:
- `src/table/src/table/scan.rs`

Signed-off-by: Lei, Huang <huanglei@qiyi.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: make region query load reporting configurable

Introduce `enable_region_query_load_report` flag to optionally report
per-region `query_cpu_time` and `query_scanned_bytes` metrics instead
of always creating them. When disabled, the Prometheus gauges are not
created (`None`), avoiding metric churn for workloads that do not
need query-level load tracking.

- `src/common/meta/src/datanode.rs` — Placeholder fields for query load
- `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset
- `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener`
- `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop`
- `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open
- `src/mito2/src/worker/handle_create.rs` — Pass flag on region create
- `src/mito2/src/worker/handle_open.rs` — Pass flag on region open
- `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder`
- `src/mito2/src/test_util.rs` — Test helpers for both modes
- `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases
- `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove elapsed_compute metric from scan stream

The elapsed_compute metric conflated poll-wait time with actual CPU
computation, making it misleading. Removed the metric and its
recording path from StreamMetrics and StreamWithMetricWrapper.

Added a test asserting that poll duration is not reported as
elapsed_compute.

- `src/table/src/table/metrics.rs` — removed elapsed_compute field,
  builder, and record_elapsed_compute method
- `src/table/src/table/scan.rs` — removed record_elapsed_compute
  call; added SlowRecordBatchStream test helper and
  wrapper_poll_time_is_not_elapsed_compute test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: disable region query load report for compaction scans

Compaction scans are internal operations initiated by the engine,
not user queries. Disable region query load reporting when the
scan input is marked as compaction to avoid misleading load metrics.

- `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report`
  to `false` when compaction is enabled; add unit test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* test: add `enable_per_region_metrics` config to HTTP integration test

- Enable per-region metrics config in HTTP test setup

\`tests-integration/tests/http.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove region query load reporting tests and helpers

Remove the region query load reporting feature from the codebase,
including tests, test utilities, and helper infrastructure that were
part of this now-deprecated functionality.

Specifically:

- Remove region query load reporting tests from
  `src/mito2/src/engine/basic_test.rs` and
  `src/table/src/table/scan.rs`, and the region close metrics test
  from `src/mito2/src/engine/close_test.rs`

- Remove region query load report test utilities and simplify engine
  construction helpers in `src/mito2/src/test_util.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* perf: avoid disabled region query load timing

Summary:
- Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled.
- Keep region query-load CPU accounting active only when reporting is enabled.

Files:
- `src/table/src/table/scan.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: move per-region query load reporting from storage to query engine

Move `enable_per_region_metrics` from datanode to frontend config and
migrate query load tracking (CPU time, scanned bytes) from mito2
storage engine to the query engine's distributed scan planner. The
storage-level metrics plumbing and `enable_region_query_load_report`
flag are removed from mito2, `ScanInput`, `ScanRegion`, and
`RegionScanner`. Query-level metrics are now collected in
`merge_scan.rs` via `scan_region_load`.

- `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes`
  metrics, `enable_region_query_load_report` plumbing from engine,
  region, opener, scanner types, workers
- `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes`
  from `RegionStatistic`
- `src/metric-engine/` -- Remove query load fields from
  `get_region_statistic`
- `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`;
  wire through planner, optimizer, merge scan with `scan_region_load`
  metrics
- `src/frontend/` -- Pass `enable_per_region_metrics` into
  `QueryOptions`
- `src/common/meta/` -- Remove TODO for query load fields
- `config/` -- Move `enable_per_region_metrics` from datanode to
  frontend and standalone example configs
- `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode
  config test
- `src/flow/` -- Add `enable_per_region_metrics` default to flownode
  options
- `src/table/` -- Remove unused query load fields from scan
- `src/datanode/` -- Remove
  `with_enable_region_query_load_report` calls

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: remove obsolete mito write load metric

Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path.

Related files:
- \`src/mito2/src/metrics.rs\`
- \`src/mito2/src/region.rs\`
- \`src/mito2/src/region/opener.rs\`
- \`src/mito2/src/region_write_ctx.rs\`
- \`src/mito2/src/engine/basic_test.rs\`
- \`src/mito2/src/worker.rs\`
- \`src/mito2/src/config.rs\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: change region query load metrics from gauge to counter

Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from
`IntGaugeVec` to `IntCounterVec` since these values are monotonically
increasing and do not need gauge semantics. Update corresponding `add`
calls to `inc_by` in merge scan reporting.

Files:
- `src/store-api/src/metrics.rs` — metric type and label changes
- `src/query/src/dist_plan/merge_scan.rs` — caller adaptation

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: pass ReadItem directly to report_region_query_load

Move `region_scan_load` call to the caller, so `report_region_query_load`
accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`.

- `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call,
  remove stale test

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* feat: ensure region query load is reported on MergeScanExec drop

Remove the `enable_per_region_metrics` parameter from `report_region_query_load`
so region load metrics are always emitted. Add a `Drop` impl for
`MergeScanExec` that reports sub-stage metrics when the executor is
dropped, covering edge cases where per-region metric emission was
missed. Add a unit test verifying CPU time and scanned bytes are
recorded on drop.

Affected file: `src/query/src/dist_plan/merge_scan.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: gate region query load reporting

Guard drop-time region query load reporting with the configured per-region metrics flag.

Related files:
- \`src/query/src/dist_plan/merge_scan.rs\`

Symbols:
- \`MergeScanExec::drop\`
- \`enable_per_region_metrics\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: clean region query load metrics on drop

Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry.

Related files:
- \`src/mito2/src/region.rs\`

Symbols:
- \`MitoRegion::drop\`
- \`REGION_QUERY_CPU_TIME\`
- \`REGION_QUERY_SCANNED_BYTES\`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
 2026-06-17 16:03:09 +00:00