Commit Graph
22 Commits
Author SHA1 Message Date
discord9 64b5096683 fix(pipeline): restore cache TTL configuration for v1.2
Complete the configuration scope of upstream #9022, including frontend and standalone propagation, examples, generated docs, and config serialization coverage.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-08 14:37:11 +08:00
Weny Xu 5d059c84bd refactor(procedure): centralize event context handling (#8834)
* refactor(procedure): centralize event context handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): simplify migration trigger reason handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): avoid cloning event context

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 72f6cf09bf)
2026-08-21 11:51:11 +08:00
Weny Xu e9ad585975 feat(event): add event context to procedure events (#8734)
* feat(event): add trigger context to procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix trigger context event contracts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve trigger context origins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve lifecycle trigger contexts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): gate enterprise trigger imports

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): assert trigger contexts exactly

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): restore migration test literals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event): add trigger context to non-table ddl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): simplify trigger context encoding

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event): preserve auto alter trigger reason

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): require explicit trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): derive trigger context at ddl boundary

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): propagate DDL trigger reasons

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve alter table rebase conflict

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): fix alter table trigger context setup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): pass trigger context to region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix unknown trigger context assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): centralize trigger context protocol mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration trigger context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(procedure): rename event runtime context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event): rename trigger context

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(event): fix migration event context assertion

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit 1f1c9270a8)
2026-08-21 11:51:11 +08:00
dennis zhuang cecb2d93a2 fix(repartition): enforce GC across lifecycle (#8678)
* fix(repartition): enforce GC across lifecycle

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(repartition): narrow GC recovery enforcement

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(metasrv): avoid repeated GC preflight during bootstrap

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(metasrv): clarify legacy GC reconciliation

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-07-29 12:04:16 +00:00
Weny Xu 7344d47756 feat(event-recorder): configure lifecycle event recording (#8648)
* refactor(event-recorder): centralize event table helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(procedure): wire lifecycle event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event-recorder): filter events by type

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): derive event type filter default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): decouple frontend filtering

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove docs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): complete configuration support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event-recorder): centralize filter ownership

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(config): update event recorder snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(frontend): decouple slow query event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-27 13:05:56 +00:00
Weny Xu 2dc6addb66 feat: report region query stats in heartbeat (#8401)
* feat: report region read load in heartbeat

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: expose region query stats in information schema

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update sqlness result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: record region query stats on stream drop

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: keep region query cpu stats in nanoseconds

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-03 06:11:57 +00:00
dennis zhuang ed9312f8e3 feat: global switch for creating tables automatically (#8203)
* feat: global switch for creating table automatically

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: make auto_create_table as comment by default

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: respect gloabl switch for metric engine

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-05-31 23:51:14 +00:00
Weny Xu 91ac84019b feat(meta-srv): support repartition for unpartitioned tables (#8186)
* feat(meta-srv): update repartition partition metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta-srv): connect repartition source metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta-srv): cover unpartitioned repartition rollback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: connect unpartitioned repartition SQL path

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-05-28 03:31:55 +00:00
Lei, HUANG 42aa58aa27 feat: support env vars in heartbeat (#8064)
* feat: support reporting env vars in heartbeat messages to metasrv

Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).

- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
  existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
  `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
  `FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
  heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
  `CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
  `discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat

Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: address heartbeat env review feedback

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: log error on deserialization failure

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: send heartbeat env vars once

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: resend heartbeat env vars after reconnect

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* revert: keep env vars in every heartbeat

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
2026-05-08 07:37:53 +00:00
Ruihang Xia 77013d9085 feat: report flow stats from streaming and batching engines (#7701)
* fix: report flow stats from streaming and batching engines

* handle restart report handler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-11 07:38:09 +00:00
Weny Xu 653740b42a feat(meta): add repartition timeouts and metrics (#7587)
* feat(meta): add repartition timeouts and metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add logs and remove default value

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-19 09:37:35 +00:00
Weny Xu 4e35028e28 test(cli): fix snapshot test paths on Windows (#7574)
* test(cli): fix snapshot test paths on Windows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix bugs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: styling

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-16 06:54:24 +00:00
Weny Xu ed171d9264 feat(cli): support RaftEngine store backend for metadata snapshot operations (#7467)
* feat(cli): support RaftEngine store backend for metadata snapshot operations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: allow empty aksk

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-14 02:24:36 +00:00
Weny Xu 567d3e66e9 feat: integrate repartition procedure into DdlManager (#7548)
* feat: add repartition procedure factory support to DdlManager

- Introduce RepartitionProcedureFactory trait for creating and registering
  repartition procedures
- Implement DefaultRepartitionProcedureFactory for metasrv with full support
- Implement StandaloneRepartitionProcedureFactory for standalone (unsupported)
- Add procedure loader registration for RepartitionProcedure and
  RepartitionGroupProcedure
- Add helper methods to TableMetadataAllocator for allocator access
- Add error types for repartition procedure operations
- Update DdlManager to accept and use RepartitionProcedureFactoryRef

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: integrate repartition procedure into DdlManager

- Add submit_repartition_task() to handle repartition from alter table
- Route Repartition operations in submit_alter_table_task() to repartition factory
- Refactor: rename submit_procedure() to execute_procedure_and_wait()
- Make all DDL operations wait for completion by default
- Add submit_procedure() for fire-and-forget submissions
- Add CreateRepartitionProcedure error type
- Add placeholder Repartition handling in grpc-expr (unsupported)
- Update greptime-proto dependency

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: implement ALTER TABLE REPARTITION procedure submission

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(repartition): handle central region in apply staging manifest

- Introduce ApplyStagingManifestInstructions struct to organize instructions
- Add special handling for central region when applying staging manifests
- Transition state from UpdateMetadata to RepartitionEnd after applying staging manifests
- Remove next_state() method in RepartitionStart and inline state transitions
- Improve logging and expression serialization in DDL statement executor
- Move repartition tests from standalone to distributed test suite

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-01-09 08:37:21 +00:00
jeremyhi 6a6b34c709 feat!: memory limiter unification write path (#7437)
* feat: remove option max_in_flight_write_bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: replace RequestMemoryLimiter

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: add integration test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by AI comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor: global permit pool on writing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by ai comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2025-12-23 02:18:49 +00:00
WaterWhisperer de9ae6066f refactor: remove export_metrics and related configuration (#7236)
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>
2025-11-17 02:32:22 +00:00
shuiyisong a20ac4f9e5 feat: prefix option for timestamp index and value column (#7125)
* refactor: use GREPTIME_TIMESTAMP const

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat: add config for default ts col name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: replace GREPTIME_TIMESTAMP with function get

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update config doc

* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: remove opts on flownode and metasrv

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add validation for ts column name

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: use get_or_init to avoid test error

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fmt

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: using empty string to disable prefix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update comment

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: address CR issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2025-10-27 08:00:03 +00:00
zyy17andYingwen a0e6bcbeb3 feat: add cpu_usage_millicores and memory_usage_bytes in information_schema.cluster_info table. (#7051)
* refactor: add `hostname` in cluster_info table

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update information schema result

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* feat: enable zstd for bulk memtable encoded parts (#7045)

feat: enable zstd in bulk memtable

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: add `get_total_cpu_millicores()` / `get_total_cpu_cores()` / `get_total_memory_bytes()` / `get_total_memory_readable()` in common-stat

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* feat: add `cpu_usage_millicores` and `memory_usage_bytes` in `information_schema.cluster_info` table

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: compile warning and integration test failed

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: integration test failed

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `ResourceStat`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: apply code review comments

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update greptime-proto

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2025-10-24 03:12:45 +00:00
zyy17 cf1b8392af refactor!: unify the API of getting total cpu and memory (#7049)
* refactor: add `get_total_cpu_millicores()` / `get_total_cpu_cores()` / `get_total_memory_bytes()` / `get_total_memory_readable()` in common-stat

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* tests: update sqlness test cases

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2025-10-16 12:41:34 +00:00
zyy17 6ab4672866 refactor: add peer_hostname field in information_schema.cluster_info table (#7050)
* refactor: add `hostname` in cluster_info table

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update information schema result

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: apply code review comments

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: update greptime-proto

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add the compatibility for old proto

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2025-10-16 06:02:47 +00:00
zyy17 0717773f62 refactor!: add enable_read_cache config to support disable read cache explicitly (#6834)
* refactor: add `enable_read_cache` config to support disable read cache explicitly

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: if `cache_path` is empty and `enable_read_cache` is true, set the default cache dir

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: remove the unessary Option type for `ObjectStorageCacheConfig`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: sanitize cache config in `DatanodeOptions` and `StandaloneOptions`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: code review comment

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: apply code review comments

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2025-09-26 09:44:12 +00:00
Weny Xu b3d413258d feat: extract standalone functionality and introduce plugin-based router configuration (#7002)
* feat: extract standalone functionality and introduce plugin-based router configuration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: ensure dump file does not exist

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: introduce `External` error

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-09-22 11:21:04 +00:00