mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-14 07:52:34 +00:00
0f87a2e89446b1b72c2c5e7541ddd5e7ca951be1
604
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
0f87a2e894 |
test(perf): add default versus BSS storage comparison
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 discord9
|
27a7047f31 |
feat: preserve row sequences and support exact sequence-range reads (#8865)
* feat(mito2): support exact sequence range reads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover preserve row sequence table alter Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): clear preserve_row_sequence marker on copy_region_from copy_region_from copies source FileMeta into the target region, which has an independent sequence domain. The physical per-row sequences in the copied file belong to the source region only; trusting them in the target would let an exact sequence-range request replay source-domain rows as if they were target sequences. Clear the preserve_row_sequence marker on copied files so the target fails closed with SequenceRangeUnsupported until the scan provably cannot intersect the copied rows. Add a regression test: copying from a preserve-enabled source into a preserve-enabled target clears the marker, and an exact (2, 7] request on the target returns SequenceRangeUnsupported instead of replaying source rows. Fixes #8865 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: remove redundant doc comments for exact sequence range options Approved comment-cleanup-only changes for #8865: drop outdated doc summaries duplicated on the exact_sequence_range wrapper and the preserve_row_sequence field, drop pure-restatement doc comments on the SetRegionOption/UnsetRegionOption PreserveRowSequence variants, and remove the four structural SQL comments from the alter_preserve_row_sequence case. No behavior changes; .result regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): fail closed exact reads on copied files and extension ranges Address review feedback on #8865: - copy_region_from: clear the source-domain FileMeta::sequence along with the preserve_row_sequence marker. An unmarked file retaining a stale source-domain max sequence could be silently skipped by files_allow_exact_sequence_range() as 'proven disjoint' in the target's independent sequence domain, dropping rows on exact (C, H] reads. With sequence=None the capability check fails closed (SequenceRangeUnsupported) until the copied rows are provably disjoint. - Engine/reader: reject exact sequence-range reads whenever a follower region has an extension range provider attached. Extension streams are returned without a row-level sequence filter, so exactness cannot be proven; treat the capability as missing (fail closed) instead of emitting out-of-range rows. The reader also fails closed as defense in depth. Tests: extend copy_region_from regression to assert the copied file's sequence hint is cleared; mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): use doc comments for test function descriptions Elevate the block comments describing test functions (in scan_test and copy_region_from_test) to /// doc comments, matching the convention used elsewhere in the exact sequence range change. No logic change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): extract helpers and trim comment noise in exact sequence reads PR finalization for #8865 (zero behavior change, full mito2 suite green): - engine: extract validate_sequence_fences and sequence_range_unsupported_reason, keeping error variants, check order and reason strings identical; OSS binds the extension blocker to false. - handle_copy_region: extract remap_copied_file_meta and file_descriptors_for_meta; rename file_ids -> source_file_ids and files_to_copy -> new_file_metas. - compactor: rename max_input_sequence -> known_max_input_sequence, document the None semantics (empty input vs unknown sequence). - Remove restating/outdated comments (ScanInput::sequence_range doc first line, outdated file-pruning note, options test restatements), compress verbatim comments while keeping why/invariants/contracts. Verified: cargo check -p mito2 (+ --features enterprise), cargo fmt, git diff --check, mito2 suite 1148/1148 passing. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): reject foreign-region SSTs in exact sequence reads Reading an SST whose FileMeta.region_id differs from the scanned region means the region's sequence domain is broken (manifest corruption or a repartition/copy path that leaked a source-domain file). Treat this as an explicit RegionSequenceDomainBroken error instead of silently ignoring the file's sequence or falling back to a full scan: the region is unusable for exact sequence-range reads until the foreign lineage is compacted away or repaired. - files_allow_exact_sequence_range / exact_sequence_range now return Result and propagate the error through engine fence validation and scan construction (StatusCode::Internal, distinct from the fallback-capable SequenceRangeUnsupported). - Row-level flat-batch sequence filtering rejects foreign-region files as defense in depth. - Engine test asserts the broken-domain error rather than Unsupported/fallback. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): never trust unmarked SSTs for exact-range disjoint skipping An unmarked file's FileMeta.sequence may be synthesized by the region-edit or repartition paths (committed+1 import barrier), not a physical max of its rows. Treating it as a whole-file disjoint proof could permanently skip rows that were never incrementally consumed once the flow checkpoint passes that value. Exact sequence-range capability now requires every SST in the region to carry the preserve_row_sequence marker; any unmarked file disables exactness (fallback), and the (C, H] file-selection skip also only applies to marked files. Foreign-region files still raise RegionSequenceDomainBroken as before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): scope exact sequence-range capability to the time-selected read set The exact capability check used to walk the entire SstVersion, so a single unmarked or foreign-region SST anywhere in the region disabled exact reads or raised RegionSequenceDomainBroken even when the request's time range could never touch that file. Both the engine fence and the scan builder now derive the read set with shared time-pruning + exact-min/sst-min selection and validate capability only over the files actually selected: a time-pruned file cannot contribute a row to (C, H], so it cannot affect exactness. The existing fail-loud semantics are unchanged for every selected file (foreign region id -> RegionSequenceDomainBroken; unmarked -> exact unavailable). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): wash untrusted sequences in compaction and restore barrier skipping Compaction with any non-preserved input now writes a sequence-less output: the physical __sequence column is zeroed (the flat format requires the internal columns) and FileMeta.sequence records the region-local admission barrier committed_sequence + 1 (falling back to the flushed frontier). preserve_row_sequence stays false. Exact sequence-range scans interpret an unmarked file's sequence as an admission barrier: barrier <= C means flow has already consumed the whole file, so it is skipped at file level; a missing or newer barrier fails closed. Foreign-region files stay in the selected read set so the capability fence still raises RegionSequenceDomainBroken. This closes the recovery loop: after a region repartition, one time-scoped fallback consumes the migrated rows, then compaction washes the untrusted per-row sequences away and exact incremental reads resume via file-level barrier skipping. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): drop restating comments in known_max_input_sequence tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): trim SQLness result EOF whitespace Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): reuse exact scan file selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): strengthen sequence scan coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(mito2): trim ALTER option comments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): remove no-op bulk compaction check Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve trusted row sequences when reading SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): preserve target sequence domain for imported SSTs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): add trailing blank line to SQLness result EOF Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): trim exact sequence scan plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): fold exact SST selection checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make legacy compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): make PK compaction rewrite deterministic Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 discord9andRuihang Xia
|
43c30d1446 |
feat(runtime): add weighted workload scheduler (#8736)
* feat(runtime): add weighted workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): switch catio to GreptimeTeam fork with admission-wait metrics Use the GreptimeTeam/catio fork (pinned c20eafc) which adds ClassStats::total_admission_wait and ClassStats::admitted, recorded at each QUEUED -> ADMITTED transition. This exposes the scheduler's own admission delay (excluding Tokio queueing and poll execution), enabling admission-wait based fairness gates. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to dynamic-config revision Bump the catio scheduler fork to 9f4b028 which adds Scheduler::set_weight and Scheduler::set_max_concurrent_polls for runtime configuration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): runtime-adjustable workload scheduler parameters Expose dynamic adjustment of the experimental workload scheduler at runtime: - common-runtime: set_workload_scheduler_weights and set_workload_scheduler_max_concurrent_polls, which forward to the catio scheduler's set_weight/set_max_concurrent_polls when the scheduler is enabled and reject zero values. - servers: /debug/workload_scheduler/weights and /debug/workload_scheduler/max_concurrent_polls POST handlers, so operators can rebalance query/write shares or admission concurrency without restarting the datanode. Both endpoints return 400 with a clear reason when the scheduler is disabled or the requested value is invalid. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(perf): add GET /debug/workload_scheduler status endpoint Returns the current weights (per class), max_concurrent_polls, active_polls and per-class counters (queued, tasks, wakes, polls, completed, cancelled, admitted, total_admission_wait) as JSON. When the scheduler is disabled, returns enabled=false with the other fields omitted, so operators can distinguish 'disabled' from an error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to time-accounting revision Bump the catio scheduler fork to 257ba56 which replaces admission-count accounting with real execution-time accounting (pass += exec_time / (weight * concurrency)), so CPU share follows the configured weights regardless of poll length. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: bump catio to lock-free sampling revision Bump the catio scheduler fork to efdc0a4 which adds an optional downsampled clock sampling mode (SchedulerBuilder::sample_every_polls, default off) with a lock-free per-class atomic counter, so the downsampled path costs one fetch_add per poll instead of a global mutex. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio to scheduler PR head Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): add scheduler bypass control Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: advance catio scheduler fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: regenerate config docs for workload scheduler Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin catio scheduler test fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(http): satisfy scheduler lifecycle clippy Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: add distributed scheduler toggle coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: finalize workload scheduler runtime controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: pin merged catio atomic weights Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: preserve unrelated lockfile resolution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(runtime): downsample scheduler time accounting Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(runtime): verify cross-runtime scheduler progress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(runtime): configure scheduler poll sampling Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): clarify scheduler activation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(runtime): explain scheduler use case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
 Lei, HUANG
|
f5212d3631 |
feat(mito2): add write cache upload hook (#8992)
Add `WriteCacheUploadStoreWrapper` in `src/mito2/src/cache/write_cache.rs` and wire it through `src/mito2/src/worker.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
00d43b29ad |
feat(query): add experimental DataFusion spill-to-disk controls (#8884)
* feat(query): add experimental DataFusion spill-to-disk controls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(config): regenerate configuration reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update config API for spill defaults Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): address spill configuration review Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): preserve spill settings with runtime plugins Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
51b94bb73f |
fix(cmd): gate daemon integration test on Unix (#8987)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 zhaiAohan
|
de6a903cb1 |
feat: Add built-in daemon mode for standalone service (#8960)
* feat: Add built-in daemon mode for standalone service Add an optional --daemon/-d flag to `greptimedb standalone start` to run the server as a background daemon detached from the shell session, similar to Redis `daemonize yes`. - Use the daemonize crate (Unix only) to fork/setsid and detach from the controlling terminal before the Tokio runtime is built. - stdin/stdout/stderr are redirected to /dev/null; logs continue to go to data_home/logs/ via the existing tracing appender. - On non-Unix platforms, --daemon is a no-op with a warning (runs in foreground). Closes #7314 Signed-off-by: tian1220A <1573612521@qq.com> * test(cmd): add integration test for standalone daemon mode Verify that standalone start --daemon detaches from the shell and brings up the HTTP listener, guarding against regressions where the daemon blocks in the foreground or crashes after forking. Signed-off-by: tian1220A <1573612521@qq.com> * fix(cmd): gate --daemon flag with #[cfg(unix)] and fix rustfmt Address review feedback to not expose --daemon/-d on non-Unix platforms. The daemon field is now gated with #[cfg(unix)], and a cross-platform is_daemon() accessor returns false on non-Unix so maybe_daemonize() stays unconditional. Also fix the rustfmt formatting issues that caused fmt-check to fail. Signed-off-by: tian1220A <1573612521@qq.com> * chore(cmd): declare daemonize dependency directly in cmd daemonize is only used by the cmd crate. Move it out of the workspace dependencies and declare it directly in the unix-only dependencies of src/cmd/Cargo.toml. Signed-off-by: tian1220A <1573612521@qq.com> --------- Signed-off-by: tian1220A <1573612521@qq.com> |
||
|
Lei, HUANG
|
c0b8612c41 |
fix(cmd): configure meta client in frontend plugin test (#8977)
Provide valid distributed frontend options in `src/cmd/src/frontend.rs` so enterprise plugin setup can preserve prefilled heartbeat extensions without weakening production validation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 Yingwen
|
ba3c5a939e |
chore(mito2): reduce default auto flush interval (#8971)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
a2f39ecf7b |
feat: harden frontend heartbeat extensions (#8803)
Add a typed `Command::build_with_heartbeat_extensions` seam in `src/cmd/src/frontend.rs`. Freeze and harden `FrontendHeartbeatExtensions` in `src/frontend/src/heartbeat.rs`, with lifecycle and race coverage in `src/frontend/src/heartbeat/tests.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
beded6e232 |
fix(flow): avoid insert select HTTP/2 stalls (#8962)
* fix(flow): avoid insert select HTTP/2 stalls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): share record batch forwarding for DML Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
046862f067 |
feat(cmd): improve parquet rewrite fidelity and scanbench output (#8947)
* feat(cmd): preserve parquet rewrite metadata settings Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): print scanner metrics in verbose scanbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): preserve SST truncation settings in property dumps Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
3493d2d0fb |
perf(servers)!: speed up Prometheus JSON response building with ryu and per-series entry reuse (#8815)
* fix(perf): align direct-SST CREATE TABLE with baked index metadata
The offline fixture generator (query_perf_fixture::direct_sst::
build_region_metadata) bakes greptime:inverted_index /
greptime:skipping_index field metadata into the region manifest for
tag/field columns, but create_table_sql emitted a bare CREATE TABLE
without those declarations. MergeScan's remote-schema validation then
failed on any tag/field projection (HTTP 500 'advertised remote stream
schema field mismatch'), breaking direct_readable_sst perf cases.
CREATE TABLE now declares the matching SKIPPING INDEX WITH
(granularity='1') / INVERTED INDEX column options. A round-trip test
proves the emitted SQL is parser-valid and yields the exact catalog
metadata.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): speed up Prometheus JSON response building with ryu and per-series entry reuse
PrometheusJsonResponse::record_batches_to_data spends ~47% of its CPU
in f64::to_string() per sample and ~32% in IndexMap::entry() per row
(60s profile of concurrent query_range workloads, ~800k series).
- Replace f64::to_string() with ryu::Buffer::format_finite for finite
values (shortest round-trip, 3-5x faster); NaN/+Inf/-Inf keep the
previous std formatting so wire output is unchanged.
- Remember the previous row's label vector and entry index; query output
is clustered by series, so consecutive rows reuse the same IndexMap
entry via get_index_mut instead of rebuilding and hashing the label
vector (worst case adds one Vec comparison per series transition).
Also adds a query-regression case (prom_json_response) that measures the
real Prometheus HTTP range API path (/v1/prometheus/api/v1/query_range),
which is the only frontend path that builds the Prometheus JSON response
(TQL ANALYZE formats the SQL JSON shape instead), plus a prom_http query
kind in the regression runner.
Perf (aligned base
|
||
|
Yingwen
|
0cc83c4570 |
feat(cmd): add parquet development tools (#8939)
* feat(cmd): add parquet metadata development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): add parquet rewrite development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): add SST replacement development tool Signed-off-by: evenyag <realevenyag@gmail.com> * feat(cmd): support local parquet files in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): clean up parquet development tools Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): share datanode tool utilities Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): satisfy parquet tool lints Signed-off-by: evenyag <realevenyag@gmail.com> * docs: document parquet development tools Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): rename SST replacement module Signed-off-by: evenyag <realevenyag@gmail.com> * fix(cmd): validate parquet rewrite options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 shuiyisong
|
1c5eabcbbf |
feat(otlp): support cumulative exponential histograms (#8900)
* feat: implement exponential histogram Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove duplicate tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): enforce exponential histogram ingestion safety Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update rfc Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): remove protocol-coupled histogram checks Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(otlp): reuse native histogram schema across data points Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: merge repeated OTLP histogram fragments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(otlp): build rejection messages lazily Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: add doc Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
2182dccd9b |
fix: cap default runtime sizes to a minimum of 2 threads (#8908)
* fix: cap default runtime sizes to a minimum of 2 threads RuntimeOptions derived its default sizes directly from num_cpus. On single-core machines every runtime (global, compact, query, ingest) ended up with one worker thread, which can easily deadlock async code (e.g. block_on combined with spawn). Clamp all CPU-derived runtime sizes to at least 2 threads. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: init logging before runtimes so runtime options are logged The global runtimes were initialized before the global logging subscriber, so the "Creating runtime ..." info logs that carry the runtime sizes were silently dropped. Initialize logging first in all node start paths; common-telemetry has no dependency on common-runtime, so the reorder is safe. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 fys
|
6538db6d61 |
refactor(json2): push down json2 type hints to parquet reads (#8833)
* refactor(mito): push down json2 type hints to parquet reads Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share json2 target types with arc Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): derive json2 output schema from target types Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): simplify read columns construction Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo check Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): reject JSON hints for non-JSON2 read columns Signed-off-by: fys <fengys1996@gmail.com> * fix: do not pushdown json type hint of non-json2-col Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> * refactor(query): simplify JSON type hint application Signed-off-by: fys <fengys1996@gmail.com> * refactor: clean up JSON2 type hint handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): keep JSON2 hints with flat read format Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): use raw parquet projection for output schema Signed-off-by: fys <fengys1996@gmail.com> * fix(query): note JSON2 hint scope limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): store JSON target types as native types Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover join hint qualifier limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): remove JSON2 fallback from compat cast Signed-off-by: fys <fengys1996@gmail.com> * docs(mito): document ReadColumns ordering contract Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo clippy Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 Weny Xu
|
943eee852f |
feat(event): record admin function executions (#8835)
* feat(event): record admin function executions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): handle admin function recording edge cases Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event): record actor for admin functions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve admin function event values Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event): preserve non-finite admin results Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
shuiyisongandfys
|
d4af650ec0 |
perf: reduce cold workspace compile time (#8801)
* refactor: remove datanode and meta-srv dep from frontend Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: use on-device protoc if possible Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: remove unused dep Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: version and docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update logs Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> |
||
|
discord9
|
1693b2727c |
refactor: port query regression runner to Rust (#8651)
* refactor: port query regression runner to Rust Signed-off-by: discord9 <discord9@163.com> * ci: remove optional OTLP report plotter Signed-off-by: discord9 <discord9@163.com> * refactor: split query regression runner into modules Signed-off-by: discord9 <discord9@163.com> * style: use crate-qualified imports in query regression runner Signed-off-by: discord9 <discord9@163.com> * refactor: simplify query regression runner internals Signed-off-by: discord9 <discord9@163.com> * feat: abstract inspect-footer storage access behind object store destination Add an optional --destination <TOML> to inspect-footer (and --base-destination/--candidate-destination to finalize-remote) so the storage inspection reads DB data files through the opendal-backed object_store abstraction instead of bare std::fs. Local paths keep working unchanged via the --root shortcut (File backend); remote backends (S3/GCS/...) are described by a DestinationConfig TOML reusing the object-store crate's ObjectStoreConfig serde shape. - inspect_footer: list via ObjectStore::list + ObjectMeta filtering (parquet keys, non-zero size, metadata/ segment), read footers async via ParquetObjectReader + ParquetMetaDataReader with known file size (no extra HEAD); output JSON schema unchanged - finalize-remote: --base-data-home/--candidate-data-home become optional, mutually exclusive with the new --*-destination args - cmd deps: add object_store_opendal + datafusion_object_store - tests: fs-backend list+footer integration tests (metadata filtering, destination TOML mode, root/destination exclusivity) Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: drop needless borrow in inspect footer test Fix clippy::needless_borrows_for_generic_args in the inspect-footer test (fs::create_dir_all(table.join("metadata"))). Missed by the earlier focused clippy run because it only covered --bin targets. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 jeremyhi
|
b30d17f89c |
docs: align wal.sync_period documented default with actual fallback (5s) (#8753)
The example TOMLs and generated config.md documented the default of wal.sync_period as "10s", but since #5677 moved the WAL sync task to a background RepeatedTask, an unset sync_period falls back to 5s in RaftEngineLogStore. The two paths therefore had different fsync periods: deployments based on the example configs used 10s while bare configs used 5s. Align the documentation with the actual code behavior (5s) instead of changing the code fallback to 10s, so that no existing deployment silently gets a larger data-loss window on host power loss. - config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s - config/config.md: regenerated via make config-docs - src/cmd/tests/load_config_test.rs: update assertions accordingly Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
9f724aa5f6 |
feat: make frontend heartbeat extensible and lifecycle-safe (#8726)
* feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Lei, HUANG
|
c55f297dec |
chore!: gate soft-drop table behind the enterprise feature (#8747)
* chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
ff7e7f13b8 |
fix(mito2): limit compaction picker threads (#8704)
* fix(mito2): limit compaction picker threads Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): extract TWCS input picking Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): make compaction picker async Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant build_output test helper After making the compaction picker async and extracting TWCS input picking, the test-only build_output helper is just a thin wrapper around build_output_with_time_range. Drop it and call the full method directly from tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: correct compact runtime config wording Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(runtime): harden compact blocking limit check Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 dennis zhuang
|
f817690b82 |
fix(mito2): make async index publication conditional (#8676)
* fix(mito2): make async index publication conditional Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale. Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): centralize manifest update finalization Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): retain index build leases across reopen Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): avoid retiring scheduler on index failure Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): handle cross-region index publication Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): use physical region for index paths Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cmd): update noop index builder Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): avoid reusing published index versions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): log untracked index build stops Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): set compaction time range in index test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
7da4f46532 |
fix: configure datanode client gRPC message limits (#8642)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
7344d47756 |
feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(procedure): wire lifecycle event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event-recorder): filter events by type Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): derive event type filter default Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): decouple frontend filtering Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove docs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): complete configuration support Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event-recorder): centralize filter ownership Signed-off-by: WenyXu <wenymedia@gmail.com> * test(config): update event recorder snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(frontend): decouple slow query event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
fys
|
3e67c67607 |
refactor: simplify scan projection to root column indices (#8629)
* refactor: simplify scan projection to root column indices Signed-off-by: fys <fengys1996@gmail.com> * fix: sqlness test Signed-off-by: fys <fengys1996@gmail.com> * chore: code style adjust Signed-off-by: fys <fengys1996@gmail.com> * fix: cr Signed-off-by: fys <fengys1996@gmail.com> * add comment and test Signed-off-by: fys <fengys1996@gmail.com> * minor change Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
shuiyisong
|
f5f5d468bb |
ci: add OTLP trace ingestion regression testing (#8631)
* chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add CI Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * ci: report otelgen runner diagnostics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add script to draw result diagram Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
9b579c7618 |
feat: make parquet row group size configurable (#8446)
* feat: make parquet row group size configurable via max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * feat: split inverted index segments across row group boundaries Signed-off-by: evenyag <realevenyag@gmail.com> * test: verify max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * test: update row group size sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): use parquet row group size for index rebuild Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update query perf fixture indexer builder Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clamp index selection to parquet rows Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support altering parquet row group size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle idempotent append mode in mixed alters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): stage mixed region option alters atomically Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): ignore inverted indexes with mismatched row counts Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
fys
|
598a412fbb |
feat: add mysql object store backend (#8560)
* feat: add mysql object store backend Signed-off-by: fys <fengys1996@gmail.com> * fix: code review Signed-off-by: fys <fengys1996@gmail.com> * add feature guard --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
shuiyisong
|
5065dfcf2f |
feat: grant creators access to newly created databases (#8566)
* feat: implement auto database acl after create database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: timeout conversion: Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove register loaders parameter in ddl_manager initialization Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ning Sun
|
e3d2864947 | test: gate pluginoption test for oss repo only (#8569) | ||
|
Ning Sun
|
63e63078b2 |
feat: allow unknown PluginOptions with a warning message (#8550)
* feat: allow unknown PluginOptions with a warning message * refactor(plugins): deserialize plugin option from borrowed Value Address review feedback on #8550: in filter_known_plugin_options, deserialize each entry from a borrowed serde_json::Value reference instead of cloning it. This removes the per-entry Value clone on the happy path while keeping the original value available for the warning log when an entry is unrecognized. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: address review comments * fix: verify against downstream project --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Lei, HUANG
|
6a53bb053c |
feat: enable soft-drop table lifecycle (#8554)
* feat: add purge_table admin function Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: restrict purge_table to admin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep purge tombstone lookup consistent Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: verify soft-drop table lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: run soft-drop lifecycle sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: read purge tombstones authoritatively Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): invalidate soft-drop marker caches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): enable configurable table soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(ci): configure gc for soft-drop sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): close regions before soft-drop metadata Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): avoid rollback after soft-drop tombstone Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): validate admin single-row calls Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): log purge dropped table target Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): mark soft-drop config experimental Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: sqlness test base Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
4873fc4f18 |
feat(meta): add retention GC for soft-dropped tables (#8526)
* feat(meta): persist soft-drop retention metadata Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): complete soft-drop retention coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): finalize soft-drop metadata after prepare Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): purge expired soft-dropped tables Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): keep soft-drop GC responsive Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): deduplicate soft-drop purge tasks Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): release purge reservations safely Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(meta): simplify soft-drop GC wiring Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): persist soft-drop retention deadlines Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): retain soft-drop config for recovery Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete standalone DDL test context Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): recheck retention before automatic purge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): keep experimental soft drop disabled Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): rotate soft-drop purge candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): address soft-drop retention review Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): protect retained table tombstones Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve post-cleanup purge state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): fence purge by drop generation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
67683cef2e |
feat: support SCRAM auth for Postgres (#8304)
* feat: support SCRAM auth for Postgres Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add pg_scram_sha256 format to hash-password command Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: harden Postgres SCRAM auth - Verify the client-final nonce matches the server-issued nonce, per RFC 5802 transcript validation, instead of only checking the channel-binding field. - Replace the per-connection PBKDF2 over a random password for unknown users with a deterministic mock verifier keyed by the username and a process-wide secret. This avoids a CPU-exhaustion DoS on unknown usernames and removes a username-enumeration oracle: the SCRAM server-first salt and iteration count are now stable per username and indistinguishable from a real user, with no PBKDF2 cost and random keys that never accept a proof. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style: format PG_SCRAM_MOCK_SECRET declaration Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: precompute stable SCRAM verifier for plaintext users Plaintext-backed credentials derived a Postgres SCRAM verifier on the fly on every connection, using a fresh random salt and running PBKDF2 each time. That made a known plaintext user distinguishable from stored-hash and unknown (mock) users through both the unstable server-first salt and the per-connection timing, enabling username enumeration. Precompute the SCRAM verifier once at load time (stable salt, default iteration count) and reuse it, matching the mock verifier handed to unknown users. Document that non-default iteration counts remain observable in the SCRAM handshake and weaken enumeration resistance. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: normalize passwords for Postgres SCRAM Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: docs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
f8aff344e9 |
feat!: remove configuration of sparse_primary_key_encoding (#8470)
* feat: remove configuration of sparse_primary_key_encoding, always use true * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix: merge --------- Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Yingwen
|
0a002f4d65 |
feat: support per-region write buffer limits (#8473)
* feat(mito): add per-region write buffer limit Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): add default region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * docs: expand configuration change checklist Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover table write buffer size option Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): refine region write buffer checks Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): keep region-stalled writes queued Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): allow zero region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate region state before write stall Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): reject writes beyond region hard limit Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update example Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
02283b6ba0 |
test(perf): add remote write storage inspection (#8444)
* feat: add remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: extend remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: inspect remote write parquet storage Signed-off-by: discord9 <discord9@163.com> * test: normalize remote write perf fixtures Signed-off-by: discord9 <discord9@163.com> * test: add heavy remote write perf case Signed-off-by: discord9 <discord9@163.com> * test: use head greptime for read bench Signed-off-by: discord9 <discord9@163.com> * test: keep heavy remote write case local Signed-off-by: discord9 <discord9@163.com> * test: tune remote write perf smoke case Signed-off-by: discord9 <discord9@163.com> * test: fix query fixture import style Signed-off-by: discord9 <discord9@163.com> * test: cover remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: add integer counter perf case Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 raphaelroshan
|
74bdb1f2a8 |
fix: compare all LoggingOptions fields in PartialEq (#8449)
* fix: compare all LoggingOptions fields in PartialEq Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> * test: set otlp_export_protocol in load_config_test expected configs The example configs set otlp_export_protocol = "http", but the expected LoggingOptions blocks for datanode/frontend/metasrv/standalone left it at the default None. Now that PartialEq compares all fields, add the field to match the parsed configs (as the flownode case already does). Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> --------- Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> |
||
|
discord9
|
12f83828b1 |
feat: add Prom remote-write query regression scenario (#8413)
* feat: add Prom remote-write query regression scenario Signed-off-by: discord9 <discord9@163.com> * test: add high-cardinality remote-write query case Signed-off-by: discord9 <discord9@163.com> * feat: chunk remote-write query regression loads Signed-off-by: discord9 <discord9@163.com> * test: use multi-day remote-write regression case Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: allow large query regression comments Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
3758911a2e |
feat: add soft-drop table recovery procedures (#8061)
* feat: add close-first soft-drop table flow Soft-drop now tombstones table metadata and closes datanode regions instead of issuing physical drop requests, while preserving hard-drop cleanup semantics and blocking conflicting drops of recreated table names. Files: - `src/common/meta/src/ddl.rs` - `src/common/meta/src/ddl/drop_table.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/ddl_manager.rs` - `src/meta-srv/src/metasrv/builder.rs` - `src/cmd/src/standalone.rs` - `src/common/meta/src/test_util.rs` - `src/meta-srv/src/procedure/utils.rs` - `tests-integration/src/standalone.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add undrop and purge table procedures Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl_manager.rs\` - \`src/common/meta/src/rpc/ddl.rs\` - \`src/common/meta/src/key.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` - \`src/mito2/src/engine/open_test.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen soft-dropped regions before purge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable soft-drop operations for metric logical tables Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases. Files: - `src/common/meta/src/ddl/drop_table/metadata.rs` - `src/common/meta/src/ddl/purge_dropped_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/utils.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sync failure detectors during soft-drop lifecycle Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions. Files: - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/drop_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify soft-drop table tests Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused. Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/key.rs` - `src/mito2/src/engine/open_test.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen follower regions during undrop Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: use dropped table ddl expr protos Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: use inline dropped table task protos Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: various soft-drop improvements - Bump `greptime-proto` dependency revision (`Cargo.toml`, `Cargo.lock`) - Pass `region_wal_options` directly without pre-serialization in undrop flow (`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`) - Remove unused `RegionNumber` import (`src/common/meta/src/ddl/utils.rs`) - Add `reset_failure_detectors` to test mock (`src/common/meta/src/ddl/tests/drop_table.rs`) - Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask` (`src/common/meta/src/rpc/ddl.rs`) Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: validate table name match in undrop procedure Add a guard in `UndropTableProcedure::on_prepare()` to check that the dropped table name matches the undrop task name, returning `TableNotFound` on mismatch. This prevents undropping a table by a different name when only the table ID is known. - `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation - `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: simplify UndropTableTask to use table_id only Remove catalog, schema, and table name fields from `UndropTableTask` since the table name can be derived from the dropped table metadata in the procedure itself. This eliminates redundant fields and the associated name-validation test. Simplify locking in `UndropTableProcedure` to only use `TableLock`. Update `greptime-proto` dependency revision. - `Cargo.toml`, `Cargo.lock` - `src/common/meta/src/rpc/ddl.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: detect table name conflict during tombstone restore in undrop - Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore - Added `TombstoneTargetAlreadyExists` error variant - Map tombstone conflict to `TableAlreadyExists` in undrop procedure - Added test for undrop failing when live name created after prepare Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/key/tombstone.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: make PurgeDroppedTableTask table_id-only Streamline the purge-dropped-table flow by requiring a table_id instead of allowing name-based fallback. - Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs` - Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs` - Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs` - Bumped `greptime-proto` dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): chunk values by per-key txn ops instead of fixed divisor Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`) to correctly account for per-key transaction operations. Added `TxnOpLimitKvBackend` test helper and two new tests (`test_restore_chunks_by_total_txn_ops_limit`, `test_create_chunks_by_total_txn_ops_limit`) verifying chunking under tight txn op limits. Affected file: - `src/common/meta/src/key/tombstone.rs` — chunk size fix, `TxnOpLimitKvBackend` helper, two new tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge - `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before transitioning to DeleteTombstone state - `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into `open_regions_inner` with an `ignore_region_not_found` flag; expose `open_regions_ignore_region_not_found` for purge replayer - `src/common/meta/src/ddl/purge_dropped_table.rs`: use `open_regions_ignore_region_not_found` in replayed purge procedures - `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop idempotency and purge replay tolerance of dropped regions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: fix clippy Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(soft-drop): open regions before restoring undrop metadata Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): fail fast on invalid txn op budget Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump greptime-proto to main branch commit Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
c44f8da646 |
feat: add query regression perf harness (#8406)
* feat: add query regression perf harness Signed-off-by: discord9 <discord9@163.com> * feat: extend query regression cases Signed-off-by: discord9 <discord9@163.com> * ci: harden query regression workflows Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: limit query regression PR triggers Signed-off-by: discord9 <discord9@163.com> * ci: run full query regression case set Signed-off-by: discord9 <discord9@163.com> * refactor: model query regression scenarios Signed-off-by: discord9 <discord9@163.com> * fix: avoid unenforced query regression thresholds Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
dennis zhuangandYingwen
|
c9062fe042 |
feat: add password hash generation command (#8288)
* feat: add password hash generation command Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject empty password in hash-password command A blank plaintext password is rejected by the user provider before verifier comparison, so a verifier generated from an empty password is unusable. Fail fast on EOF or an empty line from --password-stdin (and on an empty --password) instead of printing a dead verifier. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
e74a73638d |
feat: separate datanode query and ingestion runtimes (#8246)
* feat: add datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode runtime handles Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: wire datanode runtimes into region server Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode ingestion to ingestion runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode query runtime stream bridge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode reads to query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode global runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: use common datanode runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: run mito scan tasks on query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: split datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: share global fallback for datanode runtimes Use the global runtime as the fallback for datanode query and ingestion runtimes when datanode-specific pools are not initialized. This avoids creating unused datanode worker pools in non-datanode services. Files: - `src/common/runtime/src/global.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: forward query runtime stream metrics Forward inner stream metrics through the datanode query runtime bridge so `EXPLAIN ANALYZE` can report plan metrics after stream polling moves to the query runtime. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: route metric batch puts to ingest runtime Run the optimized metric batch put path on the datanode ingest runtime so metric ingestion does not bypass runtime isolation. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: abort query producer on stream drop Abort the datanode query runtime producer when the returned read stream is dropped so cancelled clients do not leave query work running in the background. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify query stream bridge setup Create the inner read stream before spawning the datanode query runtime producer so setup does not use an extra task and initialization channel. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Update Datanode Runtime Options and Region Server Logic - **`global.rs`**: Adjusted `datanode_ingest_rt_size` to utilize all available CPUs for improved performance. - **`region_server.rs`**: Simplified the collection of `put_requests` and optimized the `put_regions_batch` call for better efficiency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Remove Redundant Checks and Simplify Code - **`global.rs`**: Removed the assertion check for already initialized global runtimes to streamline the initialization process. - **`region_server.rs`**: Simplified the extraction of `Put` requests by removing unnecessary cloning and restructuring the iterator logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: remove redundant spawn_datanode_query in RegionServer::handle_read The outer `spawn_datanode_query` wrapped `handle_read_inner` on the same runtime, creating a nested spawn that consumed query runtime threads unnecessarily under concurrent read load. The gRPC handler already provides runtime isolation, so the inner call is sufficient. - `src/datanode/src/region_server.rs` — inline `handle_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resolve test mismatch and redundant spawn in handle_remote_read - `src/common/runtime/src/global.rs` — update test assertion to match default `datanode_ingest_rt_size` of `cpus` instead of `1` - `src/datanode/src/region_server.rs` — inline `handle_remote_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: rename datanode runtimes Summary: - Rename datanode runtime APIs from `datanode_query` and `datanode_ingest` to `query` and `ingest`. - Rename runtime config keys from `datanode_query_rt_size` and `datanode_ingest_rt_size` to `query_rt_size` and `ingest_rt_size`. - Update config docs, example config, and config-loading coverage. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/tests/load_config_test.rs` - `src/datanode/src/region_server.rs` - `src/mito2/src/read/pruner.rs` - `src/mito2/src/read/range_cache.rs` - `src/mito2/src/read/scan_region.rs` - `src/mito2/src/read/series_scan.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: consolidate runtime options Summary: - Embed datanode runtime sizes in shared `RuntimeOptions` and remove the extra `GreptimeOptions` runtime type parameter. - Use the unified `RuntimeOptions` for datanode global and datanode-specific runtime initialization. - Update datanode runtime config coverage and ingest runtime default documentation. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/src/options.rs` - `src/cmd/src/datanode.rs` - `src/cmd/src/datanode/builder.rs` - `src/cmd/tests/load_config_test.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: guard against double initialization of datanode runtimes Add an assertion in `init_datanode_runtimes` to panic when global runtimes are already initialized, preventing silent overwrites. - `src/common/runtime/src/global.rs` — assert guard in `init_datanode_runtimes` and test `test_set_datanode_runtimes_panics_after_global_runtimes_initialized` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
1451a51ea0 |
feat: add tools for development (#8260)
* feat(cmd): add dev-tools binary with parquetbench command Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add feature gate dev-tools Signed-off-by: evenyag <realevenyag@gmail.com> * feat(dev-tools): print output column count in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * feat(dev-tools): print output schema in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): move parquetbench into datanode CLI, drop dev-tools binary Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): validate pprof-after-warmup requires >=2 iterations Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): use absolute path instead of super:: import in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): base parquetbench throughput on scanned row-group bytes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |