mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-15 00:12:29 +00:00
efe75d408b77312facd46adf411497bdfb375690
584
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 fys
|
66a13cb3b7 |
refactor(json2): push down json2 type hints to parquet reads (#8833)
* refactor(mito): push down json2 type hints to parquet reads Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share json2 target types with arc Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): derive json2 output schema from target types Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): simplify read columns construction Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo check Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): reject JSON hints for non-JSON2 read columns Signed-off-by: fys <fengys1996@gmail.com> * fix: do not pushdown json type hint of non-json2-col Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> * refactor(query): simplify JSON type hint application Signed-off-by: fys <fengys1996@gmail.com> * refactor: clean up JSON2 type hint handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): keep JSON2 hints with flat read format Signed-off-by: fys <fengys1996@gmail.com> * fix(mito): use raw parquet projection for output schema Signed-off-by: fys <fengys1996@gmail.com> * fix(query): note JSON2 hint scope limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): store JSON target types as native types Signed-off-by: fys <fengys1996@gmail.com> * test(json2): cover join hint qualifier limitation Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): remove JSON2 fallback from compat cast Signed-off-by: fys <fengys1996@gmail.com> * docs(mito): document ReadColumns ordering contract Signed-off-by: fys <fengys1996@gmail.com> * fix: cargo clippy Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> (cherry picked from commit a4f2408141839518608df8a0b144da8383f2d022) (cherry picked from commit 4a50fba3a2ed0be9a610ed18b295c2c3645d8cc7) |
||
 jeremyhi
|
45309752df |
feat: make frontend heartbeat extensible and lifecycle-safe (#8726)
* feat: make frontend heartbeat extensible and lifecycle-safe Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: isolate heartbeat extension response handlers Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: cancel in-flight heartbeat response handling Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover heartbeat wire compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed heartbeat startup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address frontend heartbeat review feedback Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> (cherry picked from commit 6ffa27bc8e78b5e11b2db3d2a898438446b298be) |
||
 Lei, HUANG
|
8eeff8b441 |
fix(mito2): limit compaction picker threads (#8704)
* fix(mito2): limit compaction picker threads Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): extract TWCS input picking Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): make compaction picker async Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove redundant build_output test helper After making the compaction picker async and extracting TWCS input picking, the test-only build_output helper is just a thin wrapper around build_output_with_time_range. Drop it and call the full method directly from tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: correct compact runtime config wording Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(runtime): harden compact blocking limit check Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
7a2479496f |
chore!: gate soft-drop table behind the enterprise feature (#8747)
* chore: gate soft-drop table behind the enterprise feature Soft-drop table becomes an enterprise-only feature: - metasrv rejects gc.experimental_soft_drop.enable=true at startup in non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled without the enterprise feature as a second line of defense - the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table() registration, and information_schema.recycle_bin registration are compiled out unless the enterprise feature is enabled - common-meta procedures, tombstone keys, and DdlTask serde stay unconditional for persisted-procedure recovery and wire compatibility - the [gc.experimental_soft_drop] section is removed from the OSS example config and generated docs (moving to the enterprise repo) - the soft-drop sqlness cases and their CI job are removed from OSS (moving to the enterprise repo); affected information_schema .result files are regenerated Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: limit unused_variables allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch accidental unused variables in register_admin_only. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: include the config key in the soft-drop enterprise gate error Addresses review comment: name gc.experimental_soft_drop.enable in the startup validation error so users can locate the setting quickly when it is set via env vars or layered config. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: limit unused_mut allow to non-enterprise builds Addresses review comment: apply the allow via cfg_attr so enterprise builds still catch unused mut in the table_ddl_event test setup. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: reject soft-drop DDL submissions in non-enterprise builds Addresses review comment: clients could bypass the SQL-level gates by submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly to the procedure service. Reject fresh submissions at the DdlManager boundary in non-enterprise builds while keeping the procedure loaders registered for crash recovery and wire compatibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: stop --enable-gc from enabling soft drop in the sqlness template Addresses review comment: the metasrv test template rendered [gc.experimental_soft_drop] enable = true under the generic --enable-gc flag, which non-enterprise metasrv now rejects at startup, making the documented --enable-gc mode unusable in OSS. Keep the flag scoped to plain GC; enterprise soft-drop coverage moves to the enterprise repo. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate fresh soft-drop procedures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: gate soft-drop fallback coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: gate soft-drop procedure implementation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate drop table soft-drop behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: gate expired soft-drop gc behavior Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: test enterprise table ddl lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark purge_table as enterprise licensed The purge_table module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark recycle_bin as enterprise licensed The recycle_bin module is compiled only with the enterprise feature, so apply the Enterprise License header and register it with both license header configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: mark soft-drop procedure sources as enterprise licensed The purge and undrop procedure implementations plus the recycle-bin test module compile only with the enterprise feature. Apply the Enterprise License header and register them with both license configurations. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 dennis zhuang
|
f817690b82 |
fix(mito2): make async index publication conditional (#8676)
* fix(mito2): make async index publication conditional Check the captured SST generation and commit index metadata under the same manifest write lock. Revalidate the committed metadata before applying it to the in-memory version, and clean exact-version artifacts when either publication stage becomes stale. Add deterministic compaction and overlapping-index tests covering reopen consistency, duplicate rows, cache cleanup, and both file purgers. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(mito2): centralize manifest update finalization Share the locked update, lock release, follower check, and hook firing path between regular manifest updates and conditional index publication. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): retain index build leases across reopen Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): avoid retiring scheduler on index failure Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): handle cross-region index publication Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): use physical region for index paths Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(cmd): update noop index builder Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): avoid reusing published index versions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore(mito2): log untracked index build stops Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mito2): set compaction time range in index test Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Yingwen
|
7da4f46532 |
fix: configure datanode client gRPC message limits (#8642)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Weny Xu
|
7344d47756 |
feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(procedure): wire lifecycle event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(event-recorder): filter events by type Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): derive event type filter default Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): decouple frontend filtering Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove docs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(event-recorder): complete configuration support Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(event-recorder): centralize filter ownership Signed-off-by: WenyXu <wenymedia@gmail.com> * test(config): update event recorder snapshot Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(frontend): decouple slow query event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
fys
|
3e67c67607 |
refactor: simplify scan projection to root column indices (#8629)
* refactor: simplify scan projection to root column indices Signed-off-by: fys <fengys1996@gmail.com> * fix: sqlness test Signed-off-by: fys <fengys1996@gmail.com> * chore: code style adjust Signed-off-by: fys <fengys1996@gmail.com> * fix: cr Signed-off-by: fys <fengys1996@gmail.com> * add comment and test Signed-off-by: fys <fengys1996@gmail.com> * minor change Signed-off-by: fys <fengys1996@gmail.com> * fix: unit test Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
 shuiyisong
|
f5f5d468bb |
ci: add OTLP trace ingestion regression testing (#8631)
* chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add CI Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update CI config Signed-off-by: shuiyisong <xixing.sys@gmail.com> * ci: report otelgen runner diagnostics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add script to draw result diagram Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
9b579c7618 |
feat: make parquet row group size configurable (#8446)
* feat: make parquet row group size configurable via max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * feat: split inverted index segments across row group boundaries Signed-off-by: evenyag <realevenyag@gmail.com> * test: verify max_row_group_row_count region option Signed-off-by: evenyag <realevenyag@gmail.com> * test: update row group size sqlness result Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): use parquet row group size for index rebuild Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update query perf fixture indexer builder Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): clamp index selection to parquet rows Signed-off-by: evenyag <realevenyag@gmail.com> * feat: support altering parquet row group size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): handle idempotent append mode in mixed alters Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): stage mixed region option alters atomically Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): ignore inverted indexes with mismatched row counts Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
fys
|
598a412fbb |
feat: add mysql object store backend (#8560)
* feat: add mysql object store backend Signed-off-by: fys <fengys1996@gmail.com> * fix: code review Signed-off-by: fys <fengys1996@gmail.com> * add feature guard --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
shuiyisong
|
5065dfcf2f |
feat: grant creators access to newly created databases (#8566)
* feat: implement auto database acl after create database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: timeout conversion: Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove register loaders parameter in ddl_manager initialization Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ning Sun
|
e3d2864947 | test: gate pluginoption test for oss repo only (#8569) | ||
|
Ning Sun
|
63e63078b2 |
feat: allow unknown PluginOptions with a warning message (#8550)
* feat: allow unknown PluginOptions with a warning message * refactor(plugins): deserialize plugin option from borrowed Value Address review feedback on #8550: in filter_known_plugin_options, deserialize each entry from a borrowed serde_json::Value reference instead of cloning it. This removes the per-entry Value clone on the happy path while keeping the original value available for the warning log when an entry is unrecognized. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: address review comments * fix: verify against downstream project --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
Lei, HUANG
|
6a53bb053c |
feat: enable soft-drop table lifecycle (#8554)
* feat: add purge_table admin function Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: restrict purge_table to admin Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep purge tombstone lookup consistent Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test: verify soft-drop table lifecycle Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: run soft-drop lifecycle sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: read purge tombstones authoritatively Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): invalidate soft-drop marker caches Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): enable configurable table soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(ci): configure gc for soft-drop sqlness Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): close regions before soft-drop metadata Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): avoid rollback after soft-drop tombstone Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(function): validate admin single-row calls Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): log purge dropped table target Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): mark soft-drop config experimental Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: sqlness test base Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
4873fc4f18 |
feat(meta): add retention GC for soft-dropped tables (#8526)
* feat(meta): persist soft-drop retention metadata Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): complete soft-drop retention coverage Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): finalize soft-drop metadata after prepare Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): purge expired soft-dropped tables Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): keep soft-drop GC responsive Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): deduplicate soft-drop purge tasks Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): release purge reservations safely Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(meta): simplify soft-drop GC wiring Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(meta): persist soft-drop retention deadlines Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): retain soft-drop config for recovery Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete standalone DDL test context Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): recheck retention before automatic purge Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): keep experimental soft drop disabled Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): rotate soft-drop purge candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): address soft-drop retention review Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): protect retained table tombstones Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve post-cleanup purge state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): fence purge by drop generation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
67683cef2e |
feat: support SCRAM auth for Postgres (#8304)
* feat: support SCRAM auth for Postgres Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: add pg_scram_sha256 format to hash-password command Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: harden Postgres SCRAM auth - Verify the client-final nonce matches the server-issued nonce, per RFC 5802 transcript validation, instead of only checking the channel-binding field. - Replace the per-connection PBKDF2 over a random password for unknown users with a deterministic mock verifier keyed by the username and a process-wide secret. This avoids a CPU-exhaustion DoS on unknown usernames and removes a username-enumeration oracle: the SCRAM server-first salt and iteration count are now stable per username and indistinguishable from a real user, with no PBKDF2 cost and random keys that never accept a proof. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * style: format PG_SCRAM_MOCK_SECRET declaration Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: precompute stable SCRAM verifier for plaintext users Plaintext-backed credentials derived a Postgres SCRAM verifier on the fly on every connection, using a fresh random salt and running PBKDF2 each time. That made a known plaintext user distinguishable from stored-hash and unknown (mock) users through both the unstable server-first salt and the per-connection timing, enabling username enumeration. Precompute the SCRAM verifier once at load time (stable salt, default iteration count) and reuse it, matching the mock verifier handed to unknown users. Document that non-default iteration counts remain observable in the SCRAM handshake and weaken enumeration resistance. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: normalize passwords for Postgres SCRAM Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: docs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
f8aff344e9 |
feat!: remove configuration of sparse_primary_key_encoding (#8470)
* feat: remove configuration of sparse_primary_key_encoding, always use true * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix: merge --------- Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Yingwen
|
0a002f4d65 |
feat: support per-region write buffer limits (#8473)
* feat(mito): add per-region write buffer limit Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): add default region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * docs: expand configuration change checklist Signed-off-by: evenyag <realevenyag@gmail.com> * test: cover table write buffer size option Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): refine region write buffer checks Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): keep region-stalled writes queued Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): allow zero region write buffer size Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate region state before write stall Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): reject writes beyond region hard limit Signed-off-by: evenyag <realevenyag@gmail.com> * docs: update example Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
02283b6ba0 |
test(perf): add remote write storage inspection (#8444)
* feat: add remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: extend remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: inspect remote write parquet storage Signed-off-by: discord9 <discord9@163.com> * test: normalize remote write perf fixtures Signed-off-by: discord9 <discord9@163.com> * test: add heavy remote write perf case Signed-off-by: discord9 <discord9@163.com> * test: use head greptime for read bench Signed-off-by: discord9 <discord9@163.com> * test: keep heavy remote write case local Signed-off-by: discord9 <discord9@163.com> * test: tune remote write perf smoke case Signed-off-by: discord9 <discord9@163.com> * test: fix query fixture import style Signed-off-by: discord9 <discord9@163.com> * test: cover remote write value distributions Signed-off-by: discord9 <discord9@163.com> * test: add integer counter perf case Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 raphaelroshan
|
74bdb1f2a8 |
fix: compare all LoggingOptions fields in PartialEq (#8449)
* fix: compare all LoggingOptions fields in PartialEq Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> * test: set otlp_export_protocol in load_config_test expected configs The example configs set otlp_export_protocol = "http", but the expected LoggingOptions blocks for datanode/frontend/metasrv/standalone left it at the default None. Now that PartialEq compares all fields, add the field to match the parsed configs (as the flownode case already does). Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> --------- Signed-off-by: raphaelroshan <raphaelroshan@gmail.com> |
||
|
discord9
|
12f83828b1 |
feat: add Prom remote-write query regression scenario (#8413)
* feat: add Prom remote-write query regression scenario Signed-off-by: discord9 <discord9@163.com> * test: add high-cardinality remote-write query case Signed-off-by: discord9 <discord9@163.com> * feat: chunk remote-write query regression loads Signed-off-by: discord9 <discord9@163.com> * test: use multi-day remote-write regression case Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: allow large query regression comments Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
3758911a2e |
feat: add soft-drop table recovery procedures (#8061)
* feat: add close-first soft-drop table flow Soft-drop now tombstones table metadata and closes datanode regions instead of issuing physical drop requests, while preserving hard-drop cleanup semantics and blocking conflicting drops of recreated table names. Files: - `src/common/meta/src/ddl.rs` - `src/common/meta/src/ddl/drop_table.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/ddl_manager.rs` - `src/meta-srv/src/metasrv/builder.rs` - `src/cmd/src/standalone.rs` - `src/common/meta/src/test_util.rs` - `src/meta-srv/src/procedure/utils.rs` - `tests-integration/src/standalone.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add undrop and purge table procedures Add soft-drop recovery and cleanup procedures, wire their DDL task handling, and update \`greptime-proto\` so the new tasks can round-trip through protobuf. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl_manager.rs\` - \`src/common/meta/src/rpc/ddl.rs\` - \`src/common/meta/src/key.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` - \`src/mito2/src/engine/open_test.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen soft-dropped regions before purge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable soft-drop operations for metric logical tables Prevent soft-dropping, undropping, and purging of metric engine logical tables by explicitly returning unsupported errors. This introduces `is_metric_engine_logical_table` to identify metric logical tables and adds corresponding test cases. Files: - `src/common/meta/src/ddl/drop_table/metadata.rs` - `src/common/meta/src/ddl/purge_dropped_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/utils.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sync failure detectors during soft-drop lifecycle Keep region failure detector state aligned as soft-dropped tables close, reopen, and purge regions so stale detectors do not trigger failover for unavailable or deleted regions. Files: - \`src/common/meta/src/ddl.rs\` - \`src/common/meta/src/ddl/drop_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify soft-drop table tests Consolidate redundant soft-drop lifecycle assertions into existing end-to-end tests and share dropped-table metadata setup to keep the branch coverage focused. Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/key.rs` - `src/mito2/src/engine/open_test.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: reopen follower regions during undrop Reopen all replicas when restoring dropped physical tables so recovered replicated tables do not leave follower regions closed. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: use dropped table ddl expr protos Update greptime-proto and adapt dropped table DDL task conversions to the shared expression wrappers required by the proto API. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/ddl/drop_table/executor.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: use inline dropped table task protos Point GreptimeDB at the proto revision that restores direct dropped table task fields and remove wrapper-expression conversion code. Files: - `Cargo.toml` - `Cargo.lock` - `src/api/src/helper.rs` - `src/common/meta/src/rpc/ddl.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: various soft-drop improvements - Bump `greptime-proto` dependency revision (`Cargo.toml`, `Cargo.lock`) - Pass `region_wal_options` directly without pre-serialization in undrop flow (`src/common/meta/src/ddl/undrop_table.rs`, `src/common/meta/src/key.rs`) - Remove unused `RegionNumber` import (`src/common/meta/src/ddl/utils.rs`) - Add `reset_failure_detectors` to test mock (`src/common/meta/src/ddl/tests/drop_table.rs`) - Add JSON roundtrip tests for `UndropTableTask` and `PurgeDroppedTableTask` (`src/common/meta/src/rpc/ddl.rs`) Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: validate table name match in undrop procedure Add a guard in `UndropTableProcedure::on_prepare()` to check that the dropped table name matches the undrop task name, returning `TableNotFound` on mismatch. This prevents undropping a table by a different name when only the table ID is known. - `src/common/meta/src/ddl/undrop_table.rs` — add table-name validation - `src/common/meta/src/ddl/tests/drop_table.rs` — add test for name mismatch Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: simplify UndropTableTask to use table_id only Remove catalog, schema, and table name fields from `UndropTableTask` since the table name can be derived from the dropped table metadata in the procedure itself. This eliminates redundant fields and the associated name-validation test. Simplify locking in `UndropTableProcedure` to only use `TableLock`. Update `greptime-proto` dependency revision. - `Cargo.toml`, `Cargo.lock` - `src/common/meta/src/rpc/ddl.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: detect table name conflict during tombstone restore in undrop - Added `require_dest_not_exists` parameter to tombstone `move_values` to check destination key existence during restore - Added `TombstoneTargetAlreadyExists` error variant - Map tombstone conflict to `TableAlreadyExists` in undrop procedure - Added test for undrop failing when live name created after prepare Files: - `src/common/meta/src/ddl/tests/drop_table.rs` - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/error.rs` - `src/common/meta/src/key/tombstone.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: make PurgeDroppedTableTask table_id-only Streamline the purge-dropped-table flow by requiring a table_id instead of allowing name-based fallback. - Refactored `PurgeDroppedTableTask` to hold only `table_id` in `src/common/meta/src/rpc/ddl.rs` - Simplified purge procedure in `src/common/meta/src/ddl/purge_dropped_table.rs` - Adapted tests in `src/common/meta/src/ddl/tests/drop_table.rs` - Bumped `greptime-proto` dependency Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): chunk values by per-key txn ops instead of fixed divisor Replaced the fixed `max_txn_ops() / 2` chunk size with operation-aware constants (`MOVE_VALUE_TXN_OPS_PER_KEY=4`, `RESTORE_VALUE_TXN_OPS_PER_KEY=6`) to correctly account for per-key transaction operations. Added `TxnOpLimitKvBackend` test helper and two new tests (`test_restore_chunks_by_total_txn_ops_limit`, `test_create_chunks_by_total_txn_ops_limit`) verifying chunking under tight txn op limits. Affected file: - `src/common/meta/src/key/tombstone.rs` — chunk size fix, `TxnOpLimitKvBackend` helper, two new tests Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(soft-drop): deregister failure detectors and handle replayed open-regions during purge - `src/common/meta/src/ddl/drop_table.rs`: deregister failure detectors before transitioning to DeleteTombstone state - `src/common/meta/src/ddl/undrop_table.rs`: refactor `open_regions` into `open_regions_inner` with an `ignore_region_not_found` flag; expose `open_regions_ignore_region_not_found` for purge replayer - `src/common/meta/src/ddl/purge_dropped_table.rs`: use `open_regions_ignore_region_not_found` in replayed purge procedures - `src/common/meta/src/ddl/tests/drop_table.rs`: add tests for undrop idempotency and purge replay tolerance of dropped regions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: fix clippy Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(soft-drop): open regions before restoring undrop metadata Restore undropped table metadata only after physical regions have been reopened, keeping the table hidden while regions are still closed. Preserve the live-name conflict check before opening regions and cover the ordering with a regression test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(tombstone): fail fast on invalid txn op budget Fail before issuing `TombstoneManager` transactions when the configured `max_txn_ops` cannot fit one key. Add coverage for undersized restore budgets in `src/common/meta/src/key/tombstone.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump greptime-proto to main branch commit Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
c44f8da646 |
feat: add query regression perf harness (#8406)
* feat: add query regression perf harness Signed-off-by: discord9 <discord9@163.com> * feat: extend query regression cases Signed-off-by: discord9 <discord9@163.com> * ci: harden query regression workflows Signed-off-by: discord9 <discord9@163.com> * fix: address query regression review comments Signed-off-by: discord9 <discord9@163.com> * ci: limit query regression PR triggers Signed-off-by: discord9 <discord9@163.com> * ci: run full query regression case set Signed-off-by: discord9 <discord9@163.com> * refactor: model query regression scenarios Signed-off-by: discord9 <discord9@163.com> * fix: avoid unenforced query regression thresholds Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
dennis zhuangandYingwen
|
c9062fe042 |
feat: add password hash generation command (#8288)
* feat: add password hash generation command Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject empty password in hash-password command A blank plaintext password is rejected by the user provider before verifier comparison, so a verifier generated from an empty password is unusable. Fail fast on EOF or an empty line from --password-stdin (and on an empty --password) instead of printing a dead verifier. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
e74a73638d |
feat: separate datanode query and ingestion runtimes (#8246)
* feat: add datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode runtime handles Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: wire datanode runtimes into region server Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode ingestion to ingestion runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode query runtime stream bridge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode reads to query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode global runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: use common datanode runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: run mito scan tasks on query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: split datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: share global fallback for datanode runtimes Use the global runtime as the fallback for datanode query and ingestion runtimes when datanode-specific pools are not initialized. This avoids creating unused datanode worker pools in non-datanode services. Files: - `src/common/runtime/src/global.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: forward query runtime stream metrics Forward inner stream metrics through the datanode query runtime bridge so `EXPLAIN ANALYZE` can report plan metrics after stream polling moves to the query runtime. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: route metric batch puts to ingest runtime Run the optimized metric batch put path on the datanode ingest runtime so metric ingestion does not bypass runtime isolation. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: abort query producer on stream drop Abort the datanode query runtime producer when the returned read stream is dropped so cancelled clients do not leave query work running in the background. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify query stream bridge setup Create the inner read stream before spawning the datanode query runtime producer so setup does not use an extra task and initialization channel. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Update Datanode Runtime Options and Region Server Logic - **`global.rs`**: Adjusted `datanode_ingest_rt_size` to utilize all available CPUs for improved performance. - **`region_server.rs`**: Simplified the collection of `put_requests` and optimized the `put_regions_batch` call for better efficiency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Remove Redundant Checks and Simplify Code - **`global.rs`**: Removed the assertion check for already initialized global runtimes to streamline the initialization process. - **`region_server.rs`**: Simplified the extraction of `Put` requests by removing unnecessary cloning and restructuring the iterator logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: remove redundant spawn_datanode_query in RegionServer::handle_read The outer `spawn_datanode_query` wrapped `handle_read_inner` on the same runtime, creating a nested spawn that consumed query runtime threads unnecessarily under concurrent read load. The gRPC handler already provides runtime isolation, so the inner call is sufficient. - `src/datanode/src/region_server.rs` — inline `handle_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resolve test mismatch and redundant spawn in handle_remote_read - `src/common/runtime/src/global.rs` — update test assertion to match default `datanode_ingest_rt_size` of `cpus` instead of `1` - `src/datanode/src/region_server.rs` — inline `handle_remote_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: rename datanode runtimes Summary: - Rename datanode runtime APIs from `datanode_query` and `datanode_ingest` to `query` and `ingest`. - Rename runtime config keys from `datanode_query_rt_size` and `datanode_ingest_rt_size` to `query_rt_size` and `ingest_rt_size`. - Update config docs, example config, and config-loading coverage. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/tests/load_config_test.rs` - `src/datanode/src/region_server.rs` - `src/mito2/src/read/pruner.rs` - `src/mito2/src/read/range_cache.rs` - `src/mito2/src/read/scan_region.rs` - `src/mito2/src/read/series_scan.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: consolidate runtime options Summary: - Embed datanode runtime sizes in shared `RuntimeOptions` and remove the extra `GreptimeOptions` runtime type parameter. - Use the unified `RuntimeOptions` for datanode global and datanode-specific runtime initialization. - Update datanode runtime config coverage and ingest runtime default documentation. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/src/options.rs` - `src/cmd/src/datanode.rs` - `src/cmd/src/datanode/builder.rs` - `src/cmd/tests/load_config_test.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: guard against double initialization of datanode runtimes Add an assertion in `init_datanode_runtimes` to panic when global runtimes are already initialized, preventing silent overwrites. - `src/common/runtime/src/global.rs` — assert guard in `init_datanode_runtimes` and test `test_set_datanode_runtimes_panics_after_global_runtimes_initialized` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
1451a51ea0 |
feat: add tools for development (#8260)
* feat(cmd): add dev-tools binary with parquetbench command Signed-off-by: evenyag <realevenyag@gmail.com> * chore: add feature gate dev-tools Signed-off-by: evenyag <realevenyag@gmail.com> * feat(dev-tools): print output column count in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * feat(dev-tools): print output schema in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(cmd): move parquetbench into datanode CLI, drop dev-tools binary Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): validate pprof-after-warmup requires >=2 iterations Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): use absolute path instead of super:: import in parquetbench Signed-off-by: evenyag <realevenyag@gmail.com> * fix(dev-tools): base parquetbench throughput on scanned row-group bytes Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
jeremyhi
|
1e53b1a157 |
fix(config): align scan memory limit default with code (#8228)
* fix(config): align scan memory limit default with code Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by AI comments Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
shuiyisong
|
5fd5b91b29 |
chore: remove auth in flownode (#8244)
* chore: remove auth in flownode Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add flow startup check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: clippy Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Ning Sun
|
fd64ced4da |
feat: introduce plugin setup functions with richer context (#8256)
feat: enrich plugin setup context |
||
|
Weny Xu
|
935ef9a361 |
feat: check open region requirements (#8194)
* feat: check open region capabilities Signed-off-by: WenyXu <wenymedia@gmail.com> * test: allow file region migration tests Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refine open region requirements Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use fs scheme constant Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover open region requirement predicate Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: check file engine open requirements Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
c1b0418377 |
refactor: give instance to start plugin functions (#8208)
* refactor: give instance to start plugin functions * refactor: remove start_standalone_plugins because it's never used * chore: make sure code is nicely formatted |
||
|
dennis zhuang
|
ed9312f8e3 |
feat: global switch for creating tables automatically (#8203)
* feat: global switch for creating table automatically Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: make auto_create_table as comment by default Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: respect gloabl switch for metric engine Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
17815830ed |
chore: add LeaderServicesContext control to standalone (#8164)
* chore: add refresh hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: merge start_with_context and start Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: place reset in recover Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: revert stop changes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
e1e75b3ffe |
feat: implement a cache for the prefilter (#8102)
* feat: cache parquet prefilter results Signed-off-by: evenyag <realevenyag@gmail.com> * chore: set result cache size Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: rename is_stable to is_immutable and reject ScalarVariable Signed-off-by: evenyag <realevenyag@gmail.com> * chore: typo Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: use capacity() for prefilter key memory accounting Signed-off-by: evenyag <realevenyag@gmail.com> * feat: per filter cache Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: support other variants in MaybeFilter Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: split compute_projection_mask Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: build_prefilter_masks takes PrefilterEntry Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
28bed396e2 |
chore: introduce user cache invalidation api (#8129)
* chore: introduce user cache invalidation api Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update using plugins hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
2fdbe6c8c3 |
feat: expose node info for placement selectors (#8095)
* feat: expose node info for placement selectors Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`. Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation. Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: skip datanode node info without stats Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: avoid unnecessary workload clones Skip workload cloning for inactive nodes and for active node-info lookups without workload filters. Files: `src/meta-srv/src/discovery/utils.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: require frontend origin address Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts. Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: reuse resolved frontend address Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins. Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: preserve datanode lease liveness Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars. Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Remove stale datanode lease helper - `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs` - `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/env-based-region-selector-oss: simplify lease discovery - `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 LFC
|
abf4623440 |
refactor: store the schema of flat source (#8091)
* refactor: store the schema of flat source Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
shuiyisong
|
7279e48e22 |
chore: wrap standalone runtime with trait (#8083)
* chore: introduce standalone start service trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add region server to the trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 QuakeWang
|
e203ff9e1f |
fix: revoke meta kv writes outside metasrv leader (#8060)
* fix: revoke meta kv writes outside metasrv leader Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: address meta kv write guard review comments Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: change visibility of new_writable method to super Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: gate direct meta store writes to tests Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: keep meta store server path unchanged Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
 Han
|
b5997c6797 |
test: cover standalone user provider config (#8067)
* test: cover standalone user provider config Signed-off-by: Detachm <42765252+Detachm@users.noreply.github.com> * test: cover config-driven http auth Signed-off-by: Detachm <42765252+Detachm@users.noreply.github.com> --------- Signed-off-by: Detachm <42765252+Detachm@users.noreply.github.com> |
||
|
Lei, HUANG
|
42aa58aa27 |
feat: support env vars in heartbeat (#8064)
* feat: support reporting env vars in heartbeat messages to metasrv Add `heartbeat_env_vars` config option for datanode and frontend. When configured, the specified environment variable values are read at startup and sent to metasrv in every heartbeat via the `extensions` map. Metasrv extracts and stores them in `NodeInfo` for use in routing decisions (e.g. AZ-aware region placement). - Add `EnvVars` helper in `common/meta/src/datanode.rs` following the existing `GcStat` extension pattern with `into_extensions`/`from_extensions` - Add `env_vars: HashMap<String, String>` field to `NodeInfo` in `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat - Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`, `FrontendOptions`, and `StandaloneOptions` - Inject env vars into heartbeat `extensions` in both datanode and frontend heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`) - Extract env vars from `req.extensions` in all three metasrv `CollectXxxClusterInfoHandler`s - Update `NodeInfo` construction sites in `meta-client`, `discovery/lease.rs`, and `standalone/information_extension.rs` - Update expected TOML output in `tests-integration/tests/http.rs` - Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: address heartbeat env review feedback Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: log error on deserialization failure Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: send heartbeat env vars once Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resend heartbeat env vars after reconnect Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: keep env vars in every heartbeat Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
shuiyisong
|
c59ed1b291 |
test: use standalone flag in test (#8068)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
9faa012fa1 |
chore: add trait for creating meta kvbackend in standalone (#8063)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisongandevenyag
|
0effc30778 |
chore: update the opendal to 0.56 rc2 (#8003)
* chore: update opendal version Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update opendal version Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: grpc init Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: dep versions Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove aws-lc-rs in reqwest Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rebase main and fix compile Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: remove unused deps Signed-off-by: shuiyisong <xixing.sys@gmail.com> * Revert "fix: remove aws-lc-rs in reqwest" This reverts commit 90bfafca06f877befb36f3e54bb72fcfc4c56778. * chore: remove aws-lc-sys from blacklist Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add tls deps Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: idemptent install in rds Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: test Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use aws-lc-sys as possible Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: lint Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: address comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: address CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> * fix: sync opendal compat adapter with upstream Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address compat clippy warnings Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: evenyag <realevenyag@gmail.com> |
||
|
QuakeWang
|
8825ea3fdf |
fix!: align gRPC CLI option names with config naming (#8021)
* fix: align gRPC CLI option names with config naming Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: warn on deprecated metasrv grpc config Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
Lei, HUANG
|
f6851cf8d7 |
feat(mito2): add PK-range-aware TWCS overlap handling (#7954)
* feat(mito2): extract and cache primary key range for SST files Extracts primary key ranges from SST files during flush and compaction, and caches them in FileHandle for future use (e.g., overlapping checks). Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/compaction-overlapping-check: - **Enhance Primary Key Range Logic**: Updated the `primary_key_ranges_overlap` function in `run.rs` to use `chunk()` for comparing byte ranges, improving accuracy in overlap detection. - **Refactor Run Assignment Logic**: Simplified the logic for assigning items to runs in `run.rs` by removing redundant match statements and using `is_empty()` and `iter().any()` for cleaner checks. - **Add Test for Transitivity Break**: Introduced a new test `test_find_sorted_runs_handles_2d_transitivity_break` in `run.rs` to ensure correct handling of transitivity breaks in sorted runs. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/compaction-overlapping-check: - **Remove PK-disjoint detection logic**: Simplified the compaction logic in `twcs.rs` by removing the `has_time_overlapping_pairs` function and related logic for PK-disjoint detection. This includes the removal of the `append_mode_force_compact` condition and associated tests. - **Update compaction trigger settings**: Modified `append_mode_test.rs` to set `compaction.twcs.trigger_file_num` to "2" and adjusted the expected number of files in the scanner assertion from 1 to 2. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: rebase main Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/compaction-overlapping-check: ### Remove Unused Import in `compactor.rs` - Removed the unused import `compact_request` from `compactor.rs` to clean up the codebase. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: tighten `mito2` file lifecycle handling Refine compaction, flush, and SST/version bookkeeping across `src/mito2/src/compaction/*`, `src/mito2/src/flush.rs`, `src/mito2/src/region/*`, `src/mito2/src/sst/*`, and related tests/utilities. * fix: reuse primary key range merge in twcs compaction Centralize primary key range merging so can call the shared helper from . Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): simplify `FileHandle` initialization and internalize primary key range extraction - Updated `FileHandle::new` to automatically compute the primary key range directly from `FileMeta`. - Restricted `FileHandle::new_with_primary_key_range` to be test-only by adding the `#[cfg(test)]` attribute. - Simplified `SstVersion::add_files` by adopting the updated `FileHandle::new` instead of manually providing the primary key range. Modified files: - `src/mito2/src/sst/file.rs` - `src/mito2/src/sst/version.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/compaction-overlapping-check: ### Improve File Management and Documentation - **`twcs.rs`**: Added a comment to clarify the merging of small files when there are no overlapping files. - **`version.rs` (in `region` and `sst`)**: Enhanced documentation for `add_files` method, explaining its functionality and panic conditions. Simplified the file handle creation logic in `sst/version.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/compaction-overlapping-check: ### Enhance Primary Key Range Handling in `opener.rs` - Updated logic in `opener.rs` to set the primary key range for file handles when it is not already defined. This change ensures that the primary key range is extracted and set using `extract_primary_key_range` when necessary. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * docs: add docs for the necessity of checking pk and timesmaps while find overlapping files. * chore: address review comments Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |