Commit Graph
15 Commits
Author SHA1 Message Date
Weny Xu 1af4c33524 refactor(event): separate procedure submission context (#8856)
* refactor(event): separate procedure submission context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): map extensions and forward GC context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(gc): initialize integration test context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): pass procedure context to DDL helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify procedure submission contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): separate procedure and query contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): clarify procedure context propagation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve procedure submission context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): move DDL context by value

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): retain manual GC event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): tighten procedure context API

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update greptime-proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-13 07:29:00 +00:00
Weny Xu 72f6cf09bf refactor(procedure): centralize event context handling (#8834)
* refactor(procedure): centralize event context handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify migration trigger reason handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): avoid cloning event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-11 09:05:26 +00:00
Weny Xu 1f1c9270a8 feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix trigger context event contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve trigger context origins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve lifecycle trigger contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate enterprise trigger imports

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): assert trigger contexts exactly

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): restore migration test literals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): add trigger context to non-table ddl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): simplify trigger context encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve auto alter trigger reason

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): require explicit trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): derive trigger context at ddl boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): propagate DDL trigger reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve alter table rebase conflict

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fix alter table trigger context setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): pass trigger context to region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix unknown trigger context assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): centralize trigger context protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration trigger context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(procedure): rename event runtime context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): rename trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration event context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-08-07 12:37:58 +00:00
Weny Xu 8f11629e34 feat(metasrv): add batch GC lifecycle events (#8673)
* feat(metasrv): add batch GC lifecycle events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): reduce batch GC event fanout

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): fix batch GC event import

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): refine batch GC events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): preserve batch GC reports on failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): retain batch GC reports on retry

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): harden batch GC report merging

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: scope repartition SST assertions to target table

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-31 07:13:56 +00:00
Weny Xu 7039e54835 fix: require metasrv GC for repartition (#8497)
* fix: require GC for repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: enable GC for repartition integration tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-14 12:04:03 +00:00
discord9 149b687a92 fix: preserve repartitioned file refs during GC (#8445)
* fix: preserve repartitioned file refs during GC

Signed-off-by: discord9 <discord9@163.com>

* refactor: clarify file refs target region

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-08 08:33:02 +00:00
Weny Xu 61f802cdf2 feat: support repartition with target partition columns (#8278)
* feat(sql): support repartition on columns syntax

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): support repartition target partition columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(operator): wire repartition target columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): validate repartition target columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover repartition on columns integration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(operator): validate duplicate target partition columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: upgrade proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-12 07:36:53 +00:00
Weny Xu 91ac84019b feat(meta-srv): support repartition for unpartitioned tables (#8186)
* feat(meta-srv): update repartition partition metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta-srv): connect repartition source metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta-srv): cover unpartitioned repartition rollback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: connect unpartitioned repartition SQL path

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-05-28 03:31:55 +00:00
Yingwen 7840aa1bb4 refactor(mito2)!: remove PartitionTreeMemtable (#8080)
* feat: switch partition tree to bulk

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: keep partition tree memtable for migration test

Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: drop partition tree memtable from runtime

Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.

Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): move timestamp_array_to_i64_slice into read module

Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests

The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(mito2): delete PartitionTreeMemtable implementation

The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.

MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor(mito-codec): drop skip_partition_column parameter

PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore(mito2): remove unused MemtableConfig enum

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt code

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove unused variant

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update test_config_api

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: remove unused memtable test helpers

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address review comment

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: support bulk memtable options

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: sanitize config

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: remove partition tree options from region options

Move primary_key_encoding to the top level

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: make ssts test datetime replaced text stable

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update sqlness result

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: validate_enum_options consider bulk memtable

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: pass region id when parsing region options

Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: align sst_format with bulk memtable on parse and open

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-15 11:49:27 +00:00
discord9 56ee8baa3f feat: admin gc table/regions (#7619)
* feat: gc table

Signed-off-by: discord9 <discord9@163.com>

* test: admin gc

Signed-off-by: discord9 <discord9@163.com>

* chore: after rebase fix

Signed-off-by: discord9 <discord9@163.com>

* refactor: GcStats

Signed-off-by: discord9 <discord9@163.com>

* refactor: use gc ticker for admin gc

Signed-off-by: discord9 <discord9@163.com>

* fix: region routes override

Signed-off-by: discord9 <discord9@163.com>

* test: non happy path

Signed-off-by: discord9 <discord9@163.com>

* refactor: gc job report enum

Signed-off-by: discord9 <discord9@163.com>

* test: process 0 regions

Signed-off-by: discord9 <discord9@163.com>

* after rebase

Signed-off-by: discord9 <discord9@163.com>

* feat: allow manual gc to return error

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* chore: timeout and update proto

Signed-off-by: discord9 <discord9@163.com>

* chore: udpate proto

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-03-06 08:25:44 +00:00
Weny Xu 8026b23834 feat: partition rule version validation for writes and staging (#7628)
* feat: verify partition rule

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add partition version cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: header check

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fmt toml

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: header

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fix clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-02-06 12:16:34 +00:00
Ruihang Xia 38b5df574c refactor: SQL and gRPC server handlers (#7637)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-30 07:19:45 +00:00
Weny Xu 5bfc728d32 fix(repartition): improve physical region allocation and compaction read path correctness (#7621)
* fix: fix metadata region

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adjust repartition flow and compaction read compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename compaction mapper and pk projection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename `CompactionProjectionMapper`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: clarify compaction projection naming

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: allow create physical table with internal columns

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix template logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit test

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-28 04:04:05 +00:00
Ruihang Xia c83868c4eb feat: partition rule simplifier (#7622)
* basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reuse collider

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify range helpers

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* notes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update unit test resule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-27 14:31:20 +00:00
Weny Xuanddiscord9 4fb61047cb test: add integration tests for repartition (#7560)
* test: add integration tests for mito repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update test result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add integration tests for metric repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: correct results

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: enable tests for object store

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add compaction and gc

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit test

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* more cases

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: file ref also in repart mapping

Signed-off-by: discord9 <discord9@163.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: set a longer timeout for mock metasrv channel manager

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: discord9 <discord9@163.com>
2026-01-22 10:14:40 +00:00