rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-09-13 00:42:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,060 Commits 197 Branches 272 Tags
stack/insert-skip-wal-feature
Commit Graph
109 Commits
Author SHA1 Message Date
Weny Xu b46de8c828 feat(telemetry): add log directory size retention (#8997) 
* feat(telemetry): add log directory size retention

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: update config API logging fixture

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(telemetry): recover log retention state

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(telemetry): handle log retention cleanup errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(telemetry): cover log count retention on rotation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(telemetry): cover log directory retention

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(telemetry): avoid log filename allocation

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-09-08 07:31:10 +00:00
Dhruv Vaishnav 35f5485974 feat(meta): record logical-table reconciliation events (#8941) 
* feat(meta): add logical table reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): preserve logical reconciliation progress

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): preserve logical region retry progress

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): simplify logical reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): assert logical event values

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-09-07 08:29:26 +00:00
Weny Xu a932433d21 fix(wal): bound Kafka requests and extend latency buckets (#9026) 
* fix(wal): bound Kafka requests and extend latency buckets

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore(wal): update rskafka request timeout revision

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(config): document Kafka WAL timeouts in MetaSrv

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: sort common-wal dev dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-09-04 09:30:02 +00:00
Dhruv Vaishnav 9198462869 feat(meta): record physical table reconciliation events (#8935) 
* feat(meta): record physical table reconciliation events

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* docs(config): add reconciliation table event

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): address reconciliation event review feedback

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(meta): keep reconciliation event summary volatile

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* refactor(meta): remove unused table state downcasting

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
 2026-08-28 06:17:55 +00:00
Lei, HUANG 2182dccd9b fix: cap default runtime sizes to a minimum of 2 threads (#8908) 
* fix: cap default runtime sizes to a minimum of 2 threads

RuntimeOptions derived its default sizes directly from num_cpus. On
single-core machines every runtime (global, compact, query, ingest)
ended up with one worker thread, which can easily deadlock async code
(e.g. block_on combined with spawn).

Clamp all CPU-derived runtime sizes to at least 2 threads.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: init logging before runtimes so runtime options are logged

The global runtimes were initialized before the global logging
subscriber, so the "Creating runtime ..." info logs that carry the
runtime sizes were silently dropped. Initialize logging first in all
node start paths; common-telemetry has no dependency on
common-runtime, so the reorder is safe.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-19 10:04:50 +00:00
Lei, HUANG c55f297dec chore!: gate soft-drop table behind the enterprise feature (#8747) 
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-05 04:28:30 +00:00
Whis Liao e58f21ed6d feat(logging): add enable_file_logging option to disable file logging (#8721) 
Signed-off-by: xhwhis <hi@whis.me>
 2026-08-03 12:01:08 +00:00
Lei, HUANG ff7e7f13b8 fix(mito2): limit compaction picker threads (#8704) 
* fix(mito2): limit compaction picker threads

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): extract TWCS input picking

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): make compaction picker async

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): remove redundant build_output test helper

After making the compaction picker async and extracting TWCS input
picking, the test-only build_output helper is just a thin wrapper around
build_output_with_time_range. Drop it and call the full method directly
from tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: correct compact runtime config wording

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(runtime): harden compact blocking limit check

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-31 15:15:55 +00:00
Weny Xu 8f11629e34 feat(metasrv): add batch GC lifecycle events (#8673) 
* feat(metasrv): add batch GC lifecycle events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): reduce batch GC event fanout

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): fix batch GC event import

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): refine batch GC events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): preserve batch GC reports on failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): retain batch GC reports on retry

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): harden batch GC report merging

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: scope repartition SST assertions to target table

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-31 07:13:56 +00:00
Weny Xu 979a22b38a feat(metasrv): record WAL prune procedure events (#8677) 
* feat(metasrv): record WAL prune procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(metasrv): expand WAL prune procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix toml fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): clarify WAL prune event semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-30 09:09:27 +00:00
Weny Xu 47ca5c362e feat: add table DDL procedure events (#8627) 
* feat(meta): emit table DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): honor table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align table DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): bound table DDL event payloads

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): consolidate table DDL event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(meta): use crate visibility in event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: stabilize table DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): exclude repartition from alter table events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve table event rebase conflicts

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-30 03:24:47 +00:00
Weny Xu 31f9a9a6fd feat(metasrv): add repartition lifecycle events (#8665) 
* feat(metasrv): add repartition lifecycle events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): simplify event module names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(procedure): emit submitted events for child procedures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): flatten repartition event payload

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): defer repartition topology rows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(procedure): avoid events on failed child spawn

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-29 13:15:43 +00:00
Weny Xu 8ca6132b84 feat: add events for create and drop view (#8626) 
* feat(procedure): add view ddl events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(procedure): satisfy view event clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): add view DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): group view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): centralize view event schema

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): use singular view event module

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): align view event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): share DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: document view event recorder types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): simplify view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-29 03:13:44 +00:00
Weny Xu 775a9af3b8 feat: add procedure events for Flow DDL (#8632) 
* feat(meta): record Flow DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove query schema from Flow events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(tests): fix Flow DDL event test

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 09:02:11 +00:00
Yingwen 7da4f46532 fix: configure datanode client gRPC message limits (#8642) 
Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-07-28 08:35:42 +00:00
Weny Xu 09e1d24365 feat: add database DDL procedure events (#8623) 
* feat(meta): add database DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(sqlness): disable event recording

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: poll database DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(config): list database DDL event types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): satisfy clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 05:03:06 +00:00
Weny Xu 7344d47756 feat(event-recorder): configure lifecycle event recording (#8648) 
* refactor(event-recorder): centralize event table helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(procedure): wire lifecycle event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event-recorder): filter events by type

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): derive event type filter default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): decouple frontend filtering

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove docs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): complete configuration support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event-recorder): centralize filter ownership

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(config): update event recorder snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(frontend): decouple slow query event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-27 13:05:56 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554) 
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-20 08:28:35 +00:00
Lei, HUANG 4873fc4f18 feat(meta): add retention GC for soft-dropped tables (#8526) 
* feat(meta): persist soft-drop retention metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): complete soft-drop retention coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): finalize soft-drop metadata after prepare

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): purge expired soft-dropped tables

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep soft-drop GC responsive

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): deduplicate soft-drop purge tasks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): release purge reservations safely

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(meta): simplify soft-drop GC wiring

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): persist soft-drop retention deadlines

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): retain soft-drop config for recovery

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete standalone DDL test context

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): recheck retention before automatic purge

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep experimental soft drop disabled

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): rotate soft-drop purge candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): address soft-drop retention review

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): protect retained table tombstones

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve post-cleanup purge state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): fence purge by drop generation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-16 08:47:03 +00:00
Weny Xu 05c4588f90 feat: support remote WAL logical pruning (#8259) 
* feat: support logical deletion for remote WAL pruning

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: trigger remote WAL flush for lagging prunable regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: use from_mins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-10 04:40:51 +00:00
June dc5fab93a8 fix: remove unparsed [heartbeat] sections from node example configs (#8092) 
Fixes #8087

FrontendOptions, DatanodeOptions, and FlownodeOptions do not define
a heartbeat field, so the [heartbeat] sections in their example TOML
files were never parsed.

Heartbeat intervals are actually negotiated from metasrv during the
heartbeat handshake:
- Datanode/Flownode: interval = metasrv.heartbeat_interval
- Frontend: interval = metasrv.heartbeat_interval * 6

This mislead operators into thinking they could tune heartbeat timing
locally. Remove the sections to eliminate confusion.

Metasrv's heartbeat_interval remains documented and functional.
 2026-05-11 08:34:51 +00:00
QuakeWang 8825ea3fdf fix!: align gRPC CLI option names with config naming (#8021) 
* fix: align gRPC CLI option names with config naming

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: warn on deprecated metasrv grpc config

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
 2026-04-24 09:51:01 +00:00
discord9 00f568ed28 fix: gc update repart map properly (#7606) 
* feat: update repart map

Signed-off-by: discord9 <discord9@163.com>

* fix: table id write lock

Signed-off-by: discord9 <discord9@163.com>

* chore: default value

Signed-off-by: discord9 <discord9@163.com>

* chore: config

Signed-off-by: discord9 <discord9@163.com>

* test: update repartition map

Signed-off-by: discord9 <discord9@163.com>

* fix: empty file ref set

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: properly log error

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-01-28 04:31:19 +00:00
Lanqing Yang 8d9afc83e3 feat: allow auto schema creation for pg (#7459) 
Signed-off-by: lyang24 <lanqingy93@gmail.com>
 2025-12-23 08:55:24 +00:00
LFC 625fdd09ea refactor!: remove not working metasrv cli option (#7446) 
Signed-off-by: luofucong <luofc@foxmail.com>
 2025-12-23 06:55:17 +00:00
Weny Xu 4d66bd96b8 feat: make distributed time constants and client timeouts configurable (#7433) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-19 02:23:20 +00:00
fys c9a7b1fd68 docs(config): clarify store_addrs format (#7279) 2025-11-21 22:26:52 +00:00
WaterWhisperer de9ae6066f refactor: remove export_metrics and related configuration (#7236) 
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>
 2025-11-17 02:32:22 +00:00
Ruihang Xia f65dcd12cc feat: refine failure detector (#7005) 
* feat: refine failure detector

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert back default value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert change of test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-09-24 01:43:22 +00:00
Weny Xu dbb76483e8 chore: disable stats persistence by default (#6900) 
* chore: disable stats persistence by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-09-03 12:39:43 +00:00
LogicandWeny Xu cbcfdf9d65 feat: add optional schema for Postgres metadata tables (#6764) 
* feat(meta): add optional schema for Postgres metadata tables

- Add `schema` option to specify a custom schema for metadata tables
- Update `PgStore` and `PgElection` to support optional schema
- Modify SQL templates to use schema when provided
- Add tests for schema support in Postgres backend

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(meta): remove unused `create_schema_statement` and simplify `PgSqlTemplateFactory`

- Remove `create_schema_statement` from `PgSqlTemplateSet` struct
- Simplify `PgSqlTemplateFactory` by removing `new` method and merging it with `with_schema`
- Update related tests to reflect these changes

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(meta-srv): remove unused imports

- Remove unused import of BoxedError from common_error::ext- Remove unused import of TlsOption from servers::tls

Signed-off-by: Logic <zqr10159@dromara.org>

* build(meta): update Postgres version and add error handling imports

- Update Postgres version to 17 in docker-compose.yml
- Add BoxedError import for error handling in meta-srv

Signed-off-by: Logic <zqr10159@dromara.org>

* feat(postgres): add support for optional schema in PgElection and related components

Signed-off-by: Logic <zqr10159@dromara.org>

* feat(postgres): add support for optional schema in PgElection and related components

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(develop): update Postgres schema commands to specify host

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(postgres): simplify plugin options handling and update SQL examples

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(postgres): simplify plugin options handling and update SQL examples

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(postgres): update meta_election_lock_id description for optional schema support

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(postgres): add health check and fallback wait for Postgres in CI setup

* fix(postgres): update Docker setup for Postgres and add support for Postgres 15

* fix(postgres): remove redundant Postgres setup step in CI configuration

* Update tests-integration/fixtures/postgres/init.sql

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update .github/workflows/develop.yml

* Update tests-integration/fixtures/docker-compose.yml

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: Refactor PostgreSQL backend to support optional schema in PgStore and related SQL templates

* feat: Update PostgreSQL configuration and add PG15 specific integration tests

* feat: Update PostgreSQL configuration and add PG15 specific integration tests

* refactor(postgres): update test schemas from 'greptime_schema' to 'test_schema'

* Update .github/workflows/develop.yml

* refactor: minor factor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit test

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Logic <zqr10159@dromara.org>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2025-08-28 09:24:14 +00:00
Weny Xu f441598247 fix: correct config doc (#6836) 
fix: fix typo

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-28 06:26:25 +00:00
Yan Tingwang 32a3ef36f9 feat(metasrv): support tls for etcd client (#6818) 
* add TLS support for etcd client connections~

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* locate correct certs

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated certs

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated CI

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated CI

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Update docker-compose.yml

* tests for TLS client creation

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* modify tests

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

---------

Signed-off-by: codephage2020 <tingwangyan2020@163.com>
 2025-08-27 07:41:05 +00:00
Weny Xu 566a647ec7 feat: add replay checkpoint to reduce overhead for remote WAL (#6816) 
* feat: introduce `TopicRegionValue`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: persist region replay checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: udpate config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: send open region instructions with reply checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: use usize

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add topic name pattern

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: enable wal prune by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-27 07:24:33 +00:00
Ning Sun cdc168e753 feat: support for custom headers in otel exporter (#6773) 
* feat: support for custom headers in otel exporter

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove wrapping option

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2025-08-21 13:10:13 +00:00
Weny XuandYingwen 896d72191e feat: introduce PersistStatsHandler (#6777) 
* feat: add `Inserter` trait and impl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: import items

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce `PersistStatsHandler`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: disable persisting stats in sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* reset channel manager

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: avoid to collect

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove insert options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: use `write_bytes` instead of `write_bytes_per_sec`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: compute write bytes delta

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* Update src/meta-srv/src/handler/persist_stats_handler.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2025-08-21 09:34:58 +00:00
Ning Sun 5a4036cc66 feat: update opentelemetry family (#6762) 
* feat: update opentelemetry family

Signed-off-by: Ning Sun <sunning@greptime.com>

* doc: update doc samples

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: toml format

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: update default otel enpoint

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2025-08-19 09:23:50 +00:00
Weny Xu 021ad09c21 refactor: simplify WAL pruning procedure and introduce region flush trigger (#6741) 
* chore: add logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: update wal config for metasrv

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce region flush trigger

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: debug assert

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: log level

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify wal prune procedure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: upgrade rskafka

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: always flush inactive regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: refactor flush trigger

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused code

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: typo

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: update unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-14 14:15:30 +00:00
zyy17 777da35b0d refactor: unify the event recorder (#6689) 
* refactor: unify the event recorder

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `table_name()` in `Event` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `slow_query_options` in `Instance`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `EventHandlerOptions` and `options()` in `EventHandler` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `aggregate_events_by_type()` and support log mode of slow query

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: support to set ttl by using extension of query context

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine the configs fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: sqlness test errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use `Duration` type instead of `String` for ttl fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use pre-allocation for building RowInsertRequests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: polish code for `group_events_by_type()` and `build_row_inserts_request()`, also add the unit tests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine comments

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-12 18:26:12 +00:00
zyy17 cc35bab5e4 feat: record the migration events in metasrv (#6579) 
* feat: collect procedure event

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* feat: collect region migration events

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* test: add integration test

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix docs error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration test error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: change status code for errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `event()` in Procedure

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve trait design

1. Add `user_metadata()` in `Procedure` trait;

2. Add `Eventable` trait;

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-06 03:30:33 +00:00
Ruihang Xia 572e29b158 feat: support tls for pg backend (#6611) 
* load tls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl tls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* pass options

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement require mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* default to prefer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update example config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust example config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle client cert and key properly

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement verify_ca and verify_full

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update integration test for config api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change config name and default mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-04 00:41:08 +00:00
Yingwen 9527e0df2f feat: HTTP API to activate/deactive heap prof (activate by default) (#6593) 
* feat: add HTTP API to activate/deactivate heap profiling

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add HTTP API to get profiling status

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: enable heap prof by default

Signed-off-by: evenyag <realevenyag@gmail.com>

* build: add "prof:true,prof_active:false" as default env to dockerfiles

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: activate heap profiling after log initialization

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add memory options to control whether to activate profiling

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update docs

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt toml

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: fix config test

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: usage of new api

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: log profile after version

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update how to docs

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: fix how to docs

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-01 03:24:56 +00:00
Weny Xu 5fcb97724f chore: correct typo in configuration (#6411) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-06-27 07:55:13 +00:00
Weny Xu 24019334ee feat: implement automatic region failure detector registrations (#6370) 
* feat: implement automatic region failure detector registrations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add more tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add `region_failure_detector_initialization_delay` option

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-06-24 06:12:12 +00:00
zyy17andCopilot a49edc6ca6 refactor: add otlp_export_protocol config to support export trace data through gRPC and HTTP protocol (#6357) 
* refactor: support http traces

* refactor: add `otlp_export_protocol` config to support export trace data through gRPC and HTTP protocol

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* Update src/common/telemetry/src/logging.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-06-23 02:56:12 +00:00
jeremyhi 12648f388a feat: refactor grpc options of metasrv (#6275) 
* feat: refactor grpc options of metasrv

* fix: unit test

* feat: config update
 2025-06-10 01:35:57 +00:00
dennis zhuang 0311db3089 fix: export metrics settings in sample config (#6170) 
* fix: export metrics settings in sample config

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: forgot update docs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: test loading example configs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2025-06-09 10:53:01 +00:00
zyy17 fdf32a8f46 refactor: respect data_home as root data home directory (#6050) 
* refactor: initialize logging dir by using data_home

* chore: remove tail '/' for dir name
 2025-06-09 02:31:21 +00:00
jeremyhi 3104d49434 chore: example of http config in metasrv (#6218) 
* chore: example of http config in metasrv

* docs: make config-docs
 2025-05-30 03:27:54 +00:00
zyy17 14b655ea57 refactor: add SlowQueryRecorder to record slow query in system table and refactor slow query options (#6008) 
* refactor: add common-slow-query crate

* refactor: refine the naming

* chore: fix clippy

* chore: fix typo

* chore: sperate SlowQueryOptions From Logging

* chore: fix clippy

* chore: fix ci

* chore: refine the code

* chore: update config example

* refactor: use drop() to end the slow query timer

* refactor: move common-slow-query to frontend crate

* chore: polish some code

* refactor: code review

* refactor: add promql_range/promql_step/promql_start/promql_end fields in slow_queries

* refactor: add build_slow_query_logger()

* refactor: turn on slow query on frontend by default
 2025-05-15 04:18:48 +00:00