rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-14 07:52:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,272 Commits 266 Branches 278 Tags
ff40f4c2afbcd28e65d172c345bcbd3bd3b0641a
Commit Graph
673 Commits
Author SHA1 Message Date
dennis zhuang 079ffec116 chore: remove dead code left behind by removed features (#9377) 
* chore: remove dead code left behind by removed features

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: remove unused RouteInfoCorrupted error

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-09-28 12:35:47 +00:00
Weny Xu 5ffd01a70a fix: preserve primary key order when syncing columns (#9189) 
* fix: preserve primary key order when syncing columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: reject invalid SyncColumns metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-09-27 02:15:47 +00:00
shuiyisong 2bc18c483d feat: add schema metadata stream to SchemaManager (#9325) 
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-09-23 12:25:56 +00:00
Weny Xu a35f9d5c78 fix: address Windows test failures and run full Windows CI (#9305) 
* fix: use relative object keys for Windows filesystem access

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: cover Windows path and time limits in full test CI

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: use relative keys in metadata snapshot tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-09-23 05:16:41 +00:00
Ning SunDhruv Vaishnavhouyuwushangdennis zhuangfysLei, HUANGWeny Xu
 		1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294) 
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2026-09-23 04:56:44 +00:00
fys 045441e3cc feat(json2): support altering JSON2 settings (#9029) 
* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>
 2026-09-22 12:56:26 +00:00
Dhruv Vaishnav f1e9a74f00 fix(meta): populate physical metric table column ids (#9286) 
* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-09-22 07:35:07 +00:00
Sainath Singineedi fbbc017be4 feat: expose region min/max timestamp in region_statistics (#9060) 
* feat: expose region min/max timestamp in region_statistics

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>

* test: cover region_statistic time range assembly and projected values

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>

---------

Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com>
 2026-09-19 02:26:31 +00:00
discord9 6fa375021f feat(flow): expose extension-owned batching execution hooks (#9171) 
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: use exact sequence ranges for capable incremental flow sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: format exact sequence recovery tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: merge aggregate states in incremental flows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: exercise dispatched exact delta failure recovery

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: record flushed exact sequence flow results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to state merge execution tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat(flow): expose extension-owned batching execution hooks

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): correct extension matcher borrowing and regression assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: prove executed empty exact deltas keep incremental mode

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to empty exact delta regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover aggregate state merge through the full flow path

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: pass query engine to flow batching task regression calls

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-18 04:35:29 +00:00
jeremyhi 7fb0d9350c feat(wal): add the object store WAL provider identity and configuration (#9203) 
* feat(wal): add the object store WAL provider identity and configuration

Add the identity and configuration of the experimental object store WAL
without the log store implementation:

- store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a
  remote WAL.
- common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`)
  with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and
  `on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as
  `object_store` with the key `wal.object_store.prefix`. The metasrv config
  conversion rejects the new provider.
- common-meta, meta-srv, mito2: handle the new variants, map the region WAL
  options to the provider and reject them on a Raft Engine or Kafka log store.
- datanode: validate the configuration and fail with a "not supported yet"
  error until the log store lands.
- Example configs and the generated config docs.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(wal): accept object store WAL options when re-enabling WAL

Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-09-17 03:51:36 +00:00
discord9 d185a8bf76 feat: use exact sequence ranges for incremental Flow reads (#9165) 
* fix: preserve structured query errors through distributed execution

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: use exact sequence ranges for capable incremental flow sources

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update SQL expectations for preserved query error codes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: format exact sequence recovery tests

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: exercise dispatched exact delta failure recovery

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: record flushed exact sequence flow results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-16 08:26:58 +00:00
Dhruv Vaishnav 7e2a75f771 feat: allow re-enabling WAL after disabling (#9130) 
* feat: allow re-enabling WAL after disabling

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test: regenerate skip_wal sqlness result

The previous commit changed tests/cases/standalone/common/skip_wal.sql without
regenerating the matching .result, so every Sqlness suite failed on the
mismatch. tests/cases/distributed/common is a symlink to standalone/common, so
the single stale file accounted for all five failing variants.

Regenerated from a real run. The recorded output now covers the cases the .sql
added:

* A table created with skip_wal = 'true' has no real WAL provider, so enabling
  WAL is refused with Unsupported rather than the previous InvalidArguments.
* A RaftEngine-backed table accepts the true -> false transition, and repeating
  it is a no-op.
* SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including
  after a restart.
* A row written while WAL was skipped is absent after restart, while a row
  written after WAL was restored survives.

Verified locally with nothing else running on the machine: sqlness skip_wal
passes in both the standalone and distributed environments, and the regenerated
file is byte-identical to the output of an independent earlier run.

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test: strengthen skip_wal provider coverage

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* Fix WAL provider handling during re-enable

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* Refactor WAL provider check

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-09-16 07:52:59 +00:00
discord9 13c69cdaee perf(gc): pack file reference exchange (#9009) 
* perf(gc): pack file reference exchange

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): address packed reference review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(gc): stop without retry when maintenance is enabled

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(meta): avoid logging malformed mailbox payloads

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-14 07:30:39 +00:00
Dhruv Vaishnav 83ff0d8138 feat(meta): record catalog and database reconciliation events (#8896) 
* feat(meta): record catalog and database reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): join catalog and database reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-09-11 07:00:44 +00:00
Weny Xu adda50e03f feat: add repartition partition count hint (#9080) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-09-10 05:04:21 +00:00
Lei, HUANG 6fa1023b7f feat(mito2): introduce TWCS active window compaction (#9011) 
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows

Split the single TWCS trigger_file_num into per-window-state thresholds:
the active window keeps the existing trigger (default 4, legacy
compaction.twcs.trigger_file_num stays a compatible alias), while
inactive windows use a new trigger (default 2). Inactive windows
additionally fall back from balanced L0-only/L1-only candidates to a
progress-making unbalanced mixed candidate so historical windows can
converge; the active window retains the row/byte balance guards.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): bound inactive TWCS window convergence by rewrite budget

Inactive windows that cannot compact within one level previously either
stayed stuck (a threshold-qualified but unbalanced level returned no
candidate without trying any fallback) or fell back to a mixed merge
with no balance checks at all, which could rewrite a huge compacted file
to absorb tiny fresh files.

Inactive windows now converge progressively: threshold-qualified
balanced picks, sub-threshold balanced single-level picks, a mixed merge
whose total rewrite must fit in the output file budget, and finally an
L0-only merge without balance checks. Windows that qualify for none of
these are left uncompacted, bounding write amplification.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover TWCS window trigger options in alter_table_options sqlness case

Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and
compaction.twcs.inactive_window.trigger_file_num end to end, including
that setting the canonical active key removes the legacy
compaction.twcs.trigger_file_num alias from the table options.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): derive TWCS active window from the max-sequence file

The active window was determined by the max event-time window among
level-0 files. Between an L0 compaction removing its inputs and the next
flush landing, level 0 is empty, so the active window transiently became
None and every window fell back to the inactive rules - triggering
full-window convergence merges during ongoing ingestion whose outputs
are then superseded by new data.

Flush and compaction outputs both inherit the max input sequence, so the
file with the highest sequence across all levels always tracks the most
recent write. Use its window as the active window, falling back to the
previous L0-based rule when no file carries a sequence (legacy files).

The new helper deliberately computes window keys with the
assign_to_windows convention (truncate to seconds, then align up),
because the result is compared against window keys produced there; the
older ceil-based helper is kept unchanged for the legacy fallback path.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): add active-window L1 compaction safety trigger

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(compat): cover TWCS active window options

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): align TWCS window trigger validation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): resolve database TWCS trigger aliases

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): prioritize newer compaction windows

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): repick serial compaction outputs

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): configure inactive-window L1 trigger

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): prioritize TWCS compaction candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve TWCS trigger downgrade compatibility

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(sql): distinguish invalid database option values

Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(meta): explain TWCS legacy key compatibility

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify active window trigger field

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(options): normalize TWCS trigger aliases

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): ignore ineligible files for active window

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): mark explicit TWCS options as overrides

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): normalize zero compaction output size

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(options): validate database TWCS trigger values

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(store-api): collapse TWCS alias conflict condition

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-09-09 13:02:26 +00:00
Dhruv Vaishnav 35f5485974 feat(meta): record logical-table reconciliation events (#8941) 
* feat(meta): add logical table reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): preserve logical reconciliation progress

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): preserve logical region retry progress

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): simplify logical reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): assert logical event values

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-09-07 08:29:26 +00:00
XuanwoandWenyXu 4c12ea1aba chore(deps): bump opendal to 0.58.1 (#8742) 
* chore(deps): bump opendal to 0.58.1

Upgrade direct opendal dependency and workspace object_store_opendal pin
from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal
0.58.0). Adapt to OpenDAL 0.58 composition API:

- Operator::new returns a finished operator; drop .finish() call sites
- Replace HttpClientLayer / raw::HttpClient with OperationContext +
  HttpTransporter (ReqwestTransport)
- Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to
  Service + Layer::apply_service
- Rewrite SecureFs reader/writer/lister for sync factories and StreamRead
- Use OperatorInfo::capability() instead of removed native_capability()

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: retrigger CI after udeps runner segfault

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(object-store): restore suffix read simulation for secure fs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adapt remaining callers to OpenDAL 0.58

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2026-09-07 08:16:47 +00:00
discord9 ad7b0ace64 feat(flow): support eval schedule offsets (#8878) 
* feat(flow): support eval schedule offsets

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): remove redundant schedule assertion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(flow): trim eval offset compatibility scope

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(flow): trim eval offset edge coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(flow): trim eval offset comment noise

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address eval offset review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(compat): cover Flow eval offset persistence

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-09-04 02:39:59 +00:00
Weny Xu c01de4afdc fix(operator): whitelist private system table auto create (#8930) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-28 08:49:49 +00:00
Dhruv Vaishnav 9198462869 feat(meta): record physical table reconciliation events (#8935) 
* feat(meta): record physical table reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* docs(config): add reconciliation table event

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): address reconciliation event review feedback

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): keep reconciliation event summary volatile

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* refactor(meta): remove unused table state downcasting

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-08-28 06:17:55 +00:00
Ning Sun b31f05eb59 fix: update tokio-postgres and correct explain/fetch cursor output schema (#8955) 
* chore(deps): update tokio-postgres

* fix: describing fetch cursor and analyze
 2026-08-27 03:37:59 +00:00
dennis zhuang 09c0b23a23 feat: manage semantic table options via ALTER TABLE SET/UNSET (#8880) 
* fix(meta): actually acquire logical table locks in alter-logical-tables procedure

The procedure listed its logical table locks from table_info_values,
which is only filled during Prepare, while procedure lock keys are
fixed at submission — so the logical locks were never acquired. Today
every writer of a logical table's info is serialized by the physical
table lock, which hides the problem; a metadata-only alter procedure
targeting a single logical table would race it.

Resolve the logical table ids at submission, persist them in the
procedure state (serde(default): state dumped by older versions keeps
the previous behavior), lock physical + logical tables, and re-check
the resolved ids against the locked set at Prepare so a table dropped
and recreated after submission cannot be mutated without a lock.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: manage semantic table options via ALTER TABLE SET/UNSET

CREATE TABLE accepts greptime.semantic.* options, but ALTER TABLE SET
routed every option through SetRegionOption, whose closed match
rejects them — tables auto-created by ingestion could never receive
semantic declarations after the fact.

Semantic options are pure metadata markers no region consumes, so
they now take a metadata-only alter, following the repartition-hint
precedent:

- New AlterKind::SetAnnotations/UnsetAnnotations carrying an
  AnnotationFamily (currently only Semantic), so future marker-style
  option families reuse the same machinery. The converter classifies
  a SET/UNSET batch by key prefix and rejects batches that mix
  annotation keys with regular options.
- The procedure reuses the MetadataOnly flow: no region dispatch,
  table-info update plus cache invalidation only.
- Validation lives in the table-meta mutation layer, so it runs at
  frontend verification and again in the procedure's prepare step
  under the table lock: SET is strict (known key, value domain,
  entity columns exist and render as strings); UNSET is lenient
  inside the namespace so stale keys can be cleaned up.
  ModifyColumnTypes re-checks columns referenced by entity
  declarations at the same layer, closing a verify-then-execute race.
- Logical metric tables are supported: an annotation alter submits a
  regular alter-table task locking only the logical table, and the
  DDL manager's physical-route guard admits it.
- create_table_info re-checks semantic value domains for gRPC-built
  expressions that bypass the SQL parser.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(table): centralize annotation option classification and validation

Address review feedback on the AnnotationFamily abstraction: with only
one variant that every consumer immediately destructured, the
generality was fake. Make it real and exhaustive instead:

- AnnotationFamily gains RepartitionHint: repartition.column.hint is
  the same kind of marker option (pure metadata, no region consumes
  it) and previously had a hand-rolled special case in the converter,
  the metadata-only classifier, and a dedicated AlterKind pair — all
  deleted, one classification API remains. Per-family logical-table
  eligibility (allows_logical_tables) replaces the hard-coded
  Semantic check in the DDL manager guard.
- One validation core in the table crate (check_annotation) serves
  both DDL entry points. CREATE and ALTER previously duplicated the
  rules; each keeps its existing error variants, status codes and
  messages via thin adapters over a typed error (ALTER missing column
  stays 4002 TableColumnNotFound, CREATE stays InvalidArguments).
- The batch classifier returns Result instead of swallowing the
  mixed-batch error: a mixed SET on a logical table now reports the
  actual problem instead of UnexpectedLogicalRouteTable, and the flow
  classifiers propagate instead of guessing. The converter also moves
  its owned payloads instead of cloning them.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(meta): cover logical-table annotation alter routing

The route-guard branch admitting metadata-only annotation alters on
logical tables was only exercised end to end by sqlness. Pin it at the
DDL manager level: a semantic SET on a logical table succeeds, updates
only the logical table's metadata and dispatches nothing to datanodes;
a mixed batch reports its own error instead of the route guard's; the
repartition hint stays rejected on logical routes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep entity guard on ADD COLUMN and report missing columns first

Review follow-ups: the old verify_alter loop scanned the post-alter
schema, so it also caught DROP COLUMN followed by re-adding the
declared column with a non-string type — the mutation-layer move only
kept the MODIFY path. Guard add_columns the same way (this also covers
ingestion auto-alter). And run the MODIFY drift check after the
existence lookup, so altering a dropped-but-still-declared column
reports ColumnNotExists (4002) like every other MODIFY on a missing
column.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* style(grpc-expr): drop a test comment restating the classifier doc

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(table): rename annotation validation helpers per review

check_annotation* validated and normalized; align the names with the
validate_and_normalize_* convention nearby, and spell out
AnnotationContext (Cx is not used in this repo).

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-08-17 09:53:59 +00:00
Weny Xu ed4271af40 feat(procedure): record event actor (#8849) 
* feat(event): record procedure actor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: handle streamed region migration output

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover procedure actors across SQL protocols

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-14 09:36:01 +00:00
Weny Xu 1af4c33524 refactor(event): separate procedure submission context (#8856) 
* refactor(event): separate procedure submission context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): map extensions and forward GC context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(gc): initialize integration test context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): pass procedure context to DDL helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify procedure submission contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): separate procedure and query contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): clarify procedure context propagation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve procedure submission context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): move DDL context by value

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): retain manual GC event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): tighten procedure context API

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update greptime-proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-13 07:29:00 +00:00
Weny Xu 72f6cf09bf refactor(procedure): centralize event context handling (#8834) 
* refactor(procedure): centralize event context handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify migration trigger reason handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): avoid cloning event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-11 09:05:26 +00:00
Weny Xu 32e215cad0 fix(event): preserve procedure lifecycle locators (#8787) 
* fix(event): preserve procedure lifecycle locators

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve dropped table lifecycle locators

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): cover lifecycle locators

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix lifecycle context expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-11 07:12:45 +00:00
Weny Xu 60c7bbcaf3 refactor: share query channel definition (#8825) 
refactor: share query channel protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-10 10:47:50 +00:00
jeremyhi 584d66e3c3 fix(meta): validate leaders before enabling skip_wal (#8823) 
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-08-10 09:49:35 +00:00
dennis zhuang 5d4699db1c docs: refine coding agent maps (#8790) 
* docs: refine coding agent maps

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: trim license header guidance

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: update README links and project status

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-08-10 02:45:37 +00:00
Weny Xu 1f1c9270a8 feat(event): add event context to procedure events (#8734) 
* feat(event): add trigger context to procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix trigger context event contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve trigger context origins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve lifecycle trigger contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate enterprise trigger imports

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): assert trigger contexts exactly

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): restore migration test literals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): add trigger context to non-table ddl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): simplify trigger context encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve auto alter trigger reason

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): require explicit trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): derive trigger context at ddl boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): propagate DDL trigger reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve alter table rebase conflict

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fix alter table trigger context setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): pass trigger context to region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix unknown trigger context assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): centralize trigger context protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration trigger context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(procedure): rename event runtime context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): rename trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration event context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-07 12:37:58 +00:00
discord9 70470bafbe fix(flow): fix flow stats aggregation and df_plan_to_sql quoting (#8729) 
* fix(flow): fix flow stats aggregation and df_plan_to_sql quoting

1. Distributed-mode flow stats last-writer-wins overwrite:
   Each flownode heartbeat put its local flow state map into the single
   global __flow/state key, so reports from different nodes overwrote
   each other. Store per-flownode reports under
   __flow/state/node/{node_id} in the in-memory KV and aggregate on each
   heartbeat (last_exec_time_map/state_size/start_time_map take the max
   across nodes) into the global key. FlowStateHandler derives node
   identity from header.member_id (fallback peer.id) and ignores
   identity-less reports. Per-node keys clear automatically on leader
   change KV reset. Adapts to FlowStateValue.start_time_map added in
   #8392.

2. df_plan_to_sql unquoted special characters break flush/scheduled
   execution: ForceQuoteIdentifiers only quoted uppercase identifiers,
   so Prometheus-style table names with ':' (e.g. cpu_cores:sum) were
   left unquoted, producing invalid SQL ('keyword: :'). Quote any
   identifier with non-[a-z0-9_] chars using double quotes
   (dialect-neutral).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(flow): address review comments on quoting and logging

- df_plan_to_sql: also quote digit-leading identifiers (e.g. 123metrics)
  which would produce invalid SQL when re-parsed. SQL keywords are
  intentionally not checked (ALL_KEYWORDS would over-quote common column
  names like number; the unparse failure path has an InsertIntoPlan
  fallback).
- flow_state_handler: downgrade identity-less report log from warn! to
  debug! to avoid an anomalous sender spamming warn every heartbeat.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
 2026-08-07 09:49:55 +00:00
jeremyhi c7c1a61d56 feat: support enabling skip_wal with ALTER TABLE (#8730) 
* feat: support enabling skip_wal with alter table

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover skip wal on metric physical region

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: retry skip wal alter on route changes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address skip wal review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: preserve skip wal create options

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: prefer typed skip wal option

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address skip wal review comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: canonicalize legacy skip wal option

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: keep typed skip wal canonical

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: simplify skip wal option tracking

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-08-06 08:29:46 +00:00
+4
Palak Jhadennis zhuangLei, HUANGshuiyisongdiscord9Weny XuNing SunWhis LiaoLanqing YangsunNing SunjeremyhiYingwenLFC
 		cabc2f6cc6 feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (#7987) (#8392) 
* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(flow): add information_schema.flow_statistics and SHOW FLOW STATUS (fixes #7987)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): add sqlness golden result for flow_status

Signed-off-by: Palak Jha <palakjha916@gmail.com>

* fix(catalog): remove unused OptionExt import in flow_statistics

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs(flow): fix stale 'recent errors' comment on QueryFlowExecStats

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test: regenerate golden results for flow_statistics table

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* style: apply rustfmt to flow_statistics changes

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(catalog): hoist current_time_millis out of flow loop and clamp uptime

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed fmt_check.log

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* Update flow_status.result

del eof trailing blank line as per review

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(flow): restore runner-generated trailing blank line for sqlness

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* postgres: include SHOW FLOW STATUS in extended-query describe (return flow_statistics fields)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: address reviewer feedback on flow_statistics PR

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): resolve merge conflicts with main

Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* sqlness check post gen (information_schema.result)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): record start_time after req/snapshot_seqs built, before dispatch

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(sql): handle ShowFlowStatus in match statement at util.rs

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: review patch implementation

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: remove accidentally committed local tool output files and fix fmt

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): apply rustfmt to get_full_flow_stat return type

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix worker.rs return type formatting

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(flow): re-apply rustfmt to get_full_flow_stat return type after merge

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: merge conflicts

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line (#8652)

* fix(auth): warn when credential load disables Postgres SCRAM or drops a line

Static and watch user providers degraded silently in two ways:

- A single non-SCRAM verifier (mysql_native_password, or a legacy
  pbkdf2_sha256 hash that predates SCRAM) disables Postgres SCRAM for
  every user and falls back to cleartext, with no signal to the operator.
- A malformed credential line (commonly a plaintext password containing
  '=', which splits into more than two parts) was dropped without a trace.

Emit a warning at each credential load for both cases so operators don't
unknowingly serve cleartext passwords over Postgres or lose a user. This
is logging only; authentication behavior is unchanged. The SCRAM check
never logs secrets, and the malformed-line warning logs the line number
and file, never the line content.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(auth): warn on credential file read error before truncating

A read error from lines() (I/O failure or invalid UTF-8) ends the
iterator via map_while, silently dropping every remaining credential.
Warn with the line number and file before truncating, matching the
malformed-line handling, so the drop is observable.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* test(object-store): fix racy SecureFs abort test (#8720)

test_writer_abort_is_unsupported_without_atomic_write asserted the file
content immediately after abort() returned Unsupported. SecureFsWriter
writes through tokio::fs::File, whose write_all() only enqueues a blocking
write task (tokio's poll_write returns Ready before the write completes),
so the data may not be visible yet when the test reads the file. Drop the
race-prone content assertion and only verify the Unsupported contract.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(query): plan native histogram functions (#8705)

* feat(query): plan native histogram functions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue & add tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* perf(promql): avoid repeated scans in sliding range evaluation (#8646)

* perf(promql): use two pointers for sliding range boundaries

Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.

Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
  ~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
  shorter windows stayed within run-order noise.

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): specialize changes/resets with adaptive edge counting

The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.

Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.

Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
  and 12.0-20.9% server latency improvement; controls stayed within drift.

Signed-off-by: discord9 <discord9@163.com>

* ci(query-regression): include PromQL range boundary case in defaults

An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): address sliding range review nits

Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.

Signed-off-by: discord9 <discord9@163.com>

* style(promql): apply pinned rustfmt

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover sparse range results

Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* ci: optimize fuzz and split workflows (#8710)

* ci: batch fuzz targets in GitHub Actions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: improve fuzz test observability

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): preserve fuzz setup failure artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(ci): keep fuzz mock output in logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: optimize fuzz worker cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: warm fuzz target binaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: isolate fuzz workflow

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: centralize fuzz target preparation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: split general workflows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline docs required checks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: transfer fuzz targets as artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve fuzz binary permissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: streamline fuzz workers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: cache PR build dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: retain main build cache policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: address fuzz review feedback

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: add public constructor for compactor (#8724)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(logging): add enable_file_logging option to disable file logging (#8721)

Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* avoid cloning final Prometheus remote write row (#8733)

perf: avoid cloning final Prometheus remote write row

Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(function): add json_object_keys scalar function (#8722)

Expose JSON object key listing for outermost objects, with sqlness coverage.

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor(mito2): revise compaction trigger behavior (#8706)

* refactor(mito2): revise compaction trigger behavior

Distinguish automatic and manual triggers, coalesce explicit automatic follow-ups, and reject concurrent manual compactions.

Remove implicit post-execution continuation and transient idle statuses so scheduler entries always represent an active lifecycle.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): track automatic compaction follow-ups

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): fix compaction transition rustdoc

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): mark manual compaction conflict retryable

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor(mito2): drop unused RequestCancelResult::NotRunning variant

request_cancel is only called in tests where the region is guaranteed to be
running, so the NotRunning case was dead code. Simplify to unwrap() and
remove the variant.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): gate test-only cancellation import

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): prioritize DDL after compaction planning

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: update dashboard to v0.13.11 (#8737)

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(object-store): skip removed-entry lister test on Windows (#8735)

DirEntry on Windows is a snapshot from FindFirstFileW: file_type() and
metadata() keep returning cached data after the file is removed, so
read_list_entry() cannot observe the deletion. The test asserts the
Unix behavior (lstat returns ENOENT) and fails deterministically on
Windows nightly CI (4/4 tries). Gate it with #[cfg(not(windows))].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): preserve remote dynamic filter target (#8615)

* fix(query): preserve remote dynamic filter target

Signed-off-by: discord9 <discord9@163.com>

* fix(query): check RDF subscriber registration

Signed-off-by: discord9 <discord9@163.com>

* fix(query): refresh initial dyn filter snapshot before dispatch and handle RDF unregister

The remote dynamic filter dispatch ordering regression: freeze the target,
pre-register subscribers, refresh the initial snapshot, then dispatch.
Also implement handle_remote_dyn_filter_unregister to keep unregister
targets consistent with do_get/update.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): update test-only RegionQueryHandler impl to new trait signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: rework README release badges, drop star history, fix grpc flag (#8743)

* docs: show stable, latest and nightly version badges in README

The single release badge rendered whatever GitHub considered newest, so a
pre-release such as v1.2.0-beta.1 looked like the recommended version.

Split it into three self-updating badges using the shields.io `filter`
parameter, keyed off the existing tag naming:

- stable: `!*-*` matches tags without a hyphen (v1.1.4)
- latest: `!*-*-*` excludes nightly and dev builds (v1.2.0-beta.1)
- nightly: `*-nightly-*` matches the weekly build (v1.2.0-nightly-20260706)

No workflow changes are needed; the badges track new releases on their own.
A one-line caption below them says which channel to pick. The release-date
badge is dropped as the three version badges already carry that signal.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: remove star history chart from README

The chart carried a sealed_token in three URLs and added a large
third-party image to the Project Status section without saying anything
the badges and case studies do not already cover.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs: use --grpc-bind-addr in README quickstart

--rpc-bind-addr is now only a hidden alias of --grpc-bind-addr and no
longer shows up in --help.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* Update README.md

Co-authored-by: Ning Sun <classicning@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore: check enterprise-gated files are listed in both license configs (#8750)

* chore: check enterprise-gated files are listed in both license configs

A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is
governed by the GreptimeDB Enterprise License, so it must appear in the
`includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml.
hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and
passes the default check precisely because it was never excluded from it.

scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations,
resolves them to files (submodules included) and diffs that set against both
configs, also reporting stale entries. It runs in the license job in CI and as
`make check-enterprise-license`.

Documents the split it cannot decide for you — whole enterprise features get
their own file, a gated match arm stays inline — in
.agents/architecture-invariants.md.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: tighten enterprise license checks

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(operator): invalidate local cache after dropping view (#8748)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* chore!: gate soft-drop table behind the enterprise feature (#8747)

* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: make frontend heartbeat extensible and lifecycle-safe (#8726)

* feat: make frontend heartbeat extensible and lifecycle-safe

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: isolate heartbeat extension response handlers

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: cancel in-flight heartbeat response handling

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test: cover heartbeat wire compatibility

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: clean up failed heartbeat startup

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: address frontend heartbeat review feedback

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(meta): release region guards after drop rollback (#8751)

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor!: move native histogram config and `prom_validation_mode` to prom_store (#8744)

* chore: adjust the position of experimental_enable_prometheus_native_histogram

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: move prom_validation_mode as well

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add health-aware gRPC client routing (#8684)

* feat: add gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: harden gRPC client health routing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: defer gRPC client health checks until first use

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(mito2): discard unflushed region data safely (#8600)

* feat: support discarding unflushed region data

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): wake stalled writers after discard

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): drop redundant manifest check for discarding unflushed data

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* docs: align wal.sync_period documented default with actual fallback (5s) (#8753)

The example TOMLs and generated config.md documented the default of
wal.sync_period as "10s", but since #5677 moved the WAL sync task to a
background RepeatedTask, an unset sync_period falls back to 5s in
RaftEngineLogStore. The two paths therefore had different fsync
periods: deployments based on the example configs used 10s while bare
configs used 5s.

Align the documentation with the actual code behavior (5s) instead of
changing the code fallback to 10s, so that no existing deployment
silently gets a larger data-loss window on host power loss.

- config/datanode.example.toml, config/standalone.example.toml: 10s -> 5s
- config/config.md: regenerated via make config-docs
- src/cmd/tests/load_config_test.rs: update assertions accordingly

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: support Utf8View labels in Prometheus response (#8754)

Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix(query): validate merge scan remote schema (#8579)

* fix(query): validate merge scan remote schema

Signed-off-by: discord9 <discord9@163.com>

* fix(query): treat JSON columns as schema-compatible across wire/decode forms

CI (Sqlness json2_limit standalone + distributed) failed on the new
remote-schema validation: a JSON column is Binary + extension metadata
(ARROW:extension:name=greptime.json, greptime:type=Json) on the wire but
decodes to Struct(...) with the extension metadata — validate_remote_schema
compared raw arrow data_type and rejected it as a mismatch.

Adds json_fields_compatible(): JSON fields are equal when name and
nullability match, greptime:type matches, and the JSON2 settings
(ARROW:extension:metadata type hints) match, ignoring the physical arrow
type. Only JSON fields may bypass the raw-type comparison; non-JSON
validation stays strict.

Adds 4 regression tests mirroring the CI failure (wire-binary vs
decoded-struct accepted both directions; different JSON2 settings
rejected; JSON vs plain Binary rejected).

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat(grafana): add events dashboard (#8725)

* feat(grafana): add events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): tolerate evolving event schemas

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): address events dashboard review

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): restore events dashboard panels

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): bound events dashboard queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): include historical event catalogs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): preserve events drill-down context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): correct events lifecycle outcomes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty event type ranges

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): scope event catalogs to submissions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): handle empty events dashboard

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(grafana): refresh event schema variables

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* refactor: separate a json2 extension type (#8745)

Signed-off-by: luofucong <luofc@foxmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* feat: add admin function registrar (#8762)

* feat: add admin function registrar

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: reject admin function name collisions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix typo in admin function test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: onepizzateam <palakjha916@gmail.com>

* fix: information_schema.rs table initialization issue

Signed-off-by: onepizzateam <palakjha916@gmail.com>

* rustfmt fix

Signed-off-by: onepizzateam <palakjha916@gmail.com>

---------

Signed-off-by: onepizzateam <palakjha916@gmail.com>
Signed-off-by: Palak Jha <palakjha916@gmail.com>
Signed-off-by: polar <palakjha916@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: xhwhis <hi@whis.me>
Signed-off-by: lyang24 <lanqingy93@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: Whis Liao <xhwhis@gmail.com>
Co-authored-by: Lanqing Yang <lanqingy93@gmail.com>
Co-authored-by: sun <sunchang_long@163.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
 2026-08-06 07:24:59 +00:00
Weny Xu b70daafc77 fix(meta): release region guards after drop rollback (#8751) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-08-05 06:15:00 +00:00
Lei, HUANG c55f297dec chore!: gate soft-drop table behind the enterprise feature (#8747) 
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-05 04:28:30 +00:00
Weny Xu ad2c1bfc59 fix(meta): preserve legacy WAL options compatibility (#8707) 
* fix: preserve legacy region WAL options format

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add downgrade compatibility coverage

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: deduplicate compat restart

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: harden downgrade compatibility check

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-31 13:40:27 +00:00
Weny Xu 1aa35716cb refactor(event): store procedure trigger as JSONB (#8700) 
* refactor(event): store procedure trigger as JSONB

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix JSONB procedure trigger assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): remove procedure trigger display

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): update batch GC trigger assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-31 13:22:26 +00:00
Weny Xu 47ca5c362e feat: add table DDL procedure events (#8627) 
* feat(meta): emit table DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): honor table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align table DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): bound table DDL event payloads

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): consolidate table DDL event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(meta): use crate visibility in event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: stabilize table DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): exclude repartition from alter table events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve table event rebase conflicts

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-30 03:24:47 +00:00
dennis zhuang cecb2d93a2 fix(repartition): enforce GC across lifecycle (#8678) 
* fix(repartition): enforce GC across lifecycle

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(repartition): narrow GC recovery enforcement

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(metasrv): avoid repeated GC preflight during bootstrap

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(metasrv): clarify legacy GC reconciliation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2026-07-29 12:04:16 +00:00
Weny Xu 8ca6132b84 feat: add events for create and drop view (#8626) 
* feat(procedure): add view ddl events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(procedure): satisfy view event clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): add view DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): group view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): centralize view event schema

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): use singular view event module

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): align view event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): share DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: document view event recorder types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): simplify view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-29 03:13:44 +00:00
Weny Xu 775a9af3b8 feat: add procedure events for Flow DDL (#8632) 
* feat(meta): record Flow DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove query schema from Flow events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(tests): fix Flow DDL event test

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 09:02:11 +00:00
Weny Xu 09e1d24365 feat: add database DDL procedure events (#8623) 
* feat(meta): add database DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(sqlness): disable event recording

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: poll database DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(config): list database DDL event types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): satisfy clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 05:03:06 +00:00
jeremyhi 8ab5e68d48 fix: fail closed on malformed password assignments (#8622) 
* fix: fail closed on malformed password assignments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: close mixed password redaction bypass

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-07-24 07:51:12 +00:00
raphaelroshan e19df8960e fix: prevent credential leaks in sanitize_connection_string (#8539) 
Parse PostgreSQL DSNs (URL or libpq keyword) with tokio_postgres::Config —
the backend's own parser — and log its Debug, which redacts the password.
This matches the backend grammar exactly (multi-host URIs, backslash
escapes, any Unicode whitespace, percent-encoded query keys, and '&'/';'/
'://' inside values) rather than approximating it by hand. Other URLs are
redacted via the url crate; a best-effort keyword fallback covers inputs
neither parser accepts.

Signed-off-by: raphaelroshan <raphaelroshan@gmail.com>
 2026-07-23 07:05:34 +00:00
shuiyisong 5065dfcf2f feat: grant creators access to newly created databases (#8566) 
* feat: implement auto database acl after create database

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: timeout conversion:

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove register loaders parameter in ddl_manager initialization

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-07-21 11:29:49 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554) 
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-20 08:28:35 +00:00
Lei, HUANG bbf989ac36 feat: support soft-drop recycle bin and UNDROP TABLE (#8546) 
* feat: support full WAL retirement

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete close request migration

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: guard Kafka provider setup behind index collector check

Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.

Affected file:
- `src/log-store/src/kafka/log_store.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: avoid to_vec

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: clean up soft-dropped regions offline

Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.

Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): reject file-engine soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: preserve soft-drop cleanup split state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: remove obsolete CleanUp match arm from RegionRequest

The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): clean every soft-dropped region replica

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): order soft-drop replica cleanup

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): order soft-drop replica cleanup"

This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): clean every soft-dropped region replica"

This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(catalog): expose soft drops in recycle bin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(sql): add UNDROP TABLE

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(sql): cover UNDROP TABLE execution

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(sql): keep successful UNDROP result

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: reject stale undrop by name-based lookup after tombstone consumed

When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.

Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.

- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
  rejection case

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(log-store): keep Kafka obsolete_all as no-op

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(catalog): hide purging tables from recycle bin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(catalog): scope recycle bin scans by catalog

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(sqlness): update recycle bin expectations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: avoid redundant recycle bin allocations

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-17 08:15:21 +00:00
Lei, HUANG 4873fc4f18 feat(meta): add retention GC for soft-dropped tables (#8526) 
* feat(meta): persist soft-drop retention metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): complete soft-drop retention coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): finalize soft-drop metadata after prepare

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): purge expired soft-dropped tables

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep soft-drop GC responsive

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): deduplicate soft-drop purge tasks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): release purge reservations safely

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(meta): simplify soft-drop GC wiring

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): persist soft-drop retention deadlines

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): retain soft-drop config for recovery

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete standalone DDL test context

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): recheck retention before automatic purge

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep experimental soft drop disabled

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): rotate soft-drop purge candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): address soft-drop retention review

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): protect retained table tombstones

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve post-cleanup purge state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): fence purge by drop generation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-16 08:47:03 +00:00
Lei, HUANG 81d8beee3a feat: clean up soft-dropped regions offline (#8458) 
* fix(meta): skip reopening dropped tables during purge

Purge soft-dropped tables by dropping stored routes directly instead of
reopening tombstoned regions first. Treat legacy
\`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only
transition to \`DropRegions\`.

Files:
- \`src/common/meta/src/ddl/purge_dropped_table.rs\`
- \`src/common/meta/src/ddl/undrop_table.rs\`
- \`src/common/meta/src/ddl/tests/drop_table.rs\`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): guard undrop restore race

Serialize `UNDROP TABLE` with same-name creates by seeding the
original table name before procedure submission, and clean up reopened
regions when metadata restore fails.

Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/common/meta/src/ddl_manager.rs`
- `src/common/meta/src/ddl/tests/drop_table.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: clean up soft-dropped regions offline

Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.

Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(datanode): reject cleanup for registered regions

Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping.

Add coverage for `OfflineCleanup` engine selection and registered-region rejection.

Files:
- `src/datanode/src/region_server.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): require tombstone before undrop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): reject file-engine soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: harden soft-drop cleanup paths

Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop.

Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup.

Files:
- `src/common/meta/src/ddl/undrop_table.rs`
- `src/mito2/src/engine/open_test.rs`
- `src/mito2/src/worker/handle_open.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump cleanup proto dependency

Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload.

Files:
- \`Cargo.toml\`
- \`Cargo.lock\`
- \`src/store-api/src/region_request.rs\`
- \`src/common/meta/src/ddl/drop_table/executor.rs\`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: share region cleanup helpers

Share common region cleanup helpers between normal drop and offline cleanup
while keeping their preconditions separate.

- Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`.
- Share runtime artifact and manifest cache cleanup after full deletion paths.
- Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC.

Files:
- `src/mito2/src/worker/handle_drop.rs`
- `src/mito2/src/worker/handle_open.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: preserve soft-drop cleanup split state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: remove obsolete CleanUp match arm from RegionRequest

The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): clean every soft-dropped region replica

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): order soft-drop replica cleanup

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): order soft-drop replica cleanup"

This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* Revert "fix(meta): clean every soft-dropped region replica"

This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-14 08:39:36 +00:00