rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-15 16:32:34 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,929 Commits 274 Branches 279 Tags
fix/http-sql-explain-analyze-timeout
Commit Graph
104 Commits
Author SHA1 Message Date
Lei, HUANG c55f297dec chore!: gate soft-drop table behind the enterprise feature (#8747) 
* chore: gate soft-drop table behind the enterprise feature

Soft-drop table becomes an enterprise-only feature:

- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
  non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
  without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
  registration, and information_schema.recycle_bin registration are
  compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
  unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
  example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
  (moving to the enterprise repo); affected information_schema .result
  files are regenerated

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: limit unused_variables allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: include the config key in the soft-drop enterprise gate error

Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: limit unused_mut allow to non-enterprise builds

Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: reject soft-drop DDL submissions in non-enterprise builds

Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: stop --enable-gc from enabling soft drop in the sqlness template

Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate fresh soft-drop procedures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: gate soft-drop fallback coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: gate soft-drop procedure implementation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate drop table soft-drop behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: gate expired soft-drop gc behavior

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: test enterprise table ddl lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark purge_table as enterprise licensed

The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark recycle_bin as enterprise licensed

The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: mark soft-drop procedure sources as enterprise licensed

The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-08-05 04:28:30 +00:00
Whis Liao e58f21ed6d feat(logging): add enable_file_logging option to disable file logging (#8721) 
Signed-off-by: xhwhis <hi@whis.me>
 2026-08-03 12:01:08 +00:00
Lei, HUANG ff7e7f13b8 fix(mito2): limit compaction picker threads (#8704) 
* fix(mito2): limit compaction picker threads

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): extract TWCS input picking

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): make compaction picker async

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): remove redundant build_output test helper

After making the compaction picker async and extracting TWCS input
picking, the test-only build_output helper is just a thin wrapper around
build_output_with_time_range. Drop it and call the full method directly
from tests.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: correct compact runtime config wording

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(runtime): harden compact blocking limit check

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-31 15:15:55 +00:00
Weny Xu 8f11629e34 feat(metasrv): add batch GC lifecycle events (#8673) 
* feat(metasrv): add batch GC lifecycle events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): reduce batch GC event fanout

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): fix batch GC event import

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): refine batch GC events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): preserve batch GC reports on failure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): retain batch GC reports on retry

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): harden batch GC report merging

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: scope repartition SST assertions to target table

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-31 07:13:56 +00:00
Weny Xu 979a22b38a feat(metasrv): record WAL prune procedure events (#8677) 
* feat(metasrv): record WAL prune procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(metasrv): expand WAL prune procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix toml fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(metasrv): clarify WAL prune event semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-30 09:09:27 +00:00
Weny Xu 47ca5c362e feat: add table DDL procedure events (#8627) 
* feat(meta): emit table DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): honor table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): cover table DDL event filters

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align table DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): bound table DDL event payloads

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): consolidate table DDL event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style(meta): use crate visibility in event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: stabilize table DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): exclude repartition from alter table events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): resolve table event rebase conflicts

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-30 03:24:47 +00:00
Weny Xu 31f9a9a6fd feat(metasrv): add repartition lifecycle events (#8665) 
* feat(metasrv): add repartition lifecycle events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): simplify event module names

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(procedure): emit submitted events for child procedures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): flatten repartition event payload

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(metasrv): defer repartition topology rows

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(procedure): avoid events on failed child spawn

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-29 13:15:43 +00:00
Weny Xu 8ca6132b84 feat: add events for create and drop view (#8626) 
* feat(procedure): add view ddl events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(procedure): satisfy view event clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): add view DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): group view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): centralize view event schema

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): use singular view event module

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): align view event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(integration): share DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: document view event recorder types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): align view DDL event conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(meta): simplify view event tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-29 03:13:44 +00:00
Weny Xu 775a9af3b8 feat: add procedure events for Flow DDL (#8632) 
* feat(meta): record Flow DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): remove query schema from Flow events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(tests): fix Flow DDL event test

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 09:02:11 +00:00
Yingwen 7da4f46532 fix: configure datanode client gRPC message limits (#8642) 
Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-07-28 08:35:42 +00:00
Weny Xu 09e1d24365 feat: add database DDL procedure events (#8623) 
* feat(meta): add database DDL procedure events

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(sqlness): disable event recording

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: poll database DDL event assertions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(config): list database DDL event types

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(test): satisfy clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-28 05:03:06 +00:00
Weny Xu 7344d47756 feat(event-recorder): configure lifecycle event recording (#8648) 
* refactor(event-recorder): centralize event table helpers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(procedure): wire lifecycle event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(event-recorder): filter events by type

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): derive event type filter default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): decouple frontend filtering

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove docs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(event-recorder): complete configuration support

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(event-recorder): centralize filter ownership

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(config): update event recorder snapshot

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(frontend): decouple slow query event recorder

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-07-27 13:05:56 +00:00
Lei, HUANG 6a53bb053c feat: enable soft-drop table lifecycle (#8554) 
* feat: add purge_table admin function

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: restrict purge_table to admin

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep purge tombstone lookup consistent

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test: verify soft-drop table lifecycle

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* ci: run soft-drop lifecycle sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: read purge tombstones authoritatively

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): invalidate soft-drop marker caches

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): enable configurable table soft drop

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(ci): configure gc for soft-drop sqlness

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): close regions before soft-drop metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): avoid rollback after soft-drop tombstone

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(function): validate admin single-row calls

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): log purge dropped table target

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): mark soft-drop config experimental

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: sqlness test base

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-20 08:28:35 +00:00
Lei, HUANG 4873fc4f18 feat(meta): add retention GC for soft-dropped tables (#8526) 
* feat(meta): persist soft-drop retention metadata

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): complete soft-drop retention coverage

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): finalize soft-drop metadata after prepare

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): purge expired soft-dropped tables

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep soft-drop GC responsive

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): deduplicate soft-drop purge tasks

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): release purge reservations safely

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(meta): simplify soft-drop GC wiring

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(meta): persist soft-drop retention deadlines

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): retain soft-drop config for recovery

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete standalone DDL test context

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): recheck retention before automatic purge

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): keep experimental soft drop disabled

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): rotate soft-drop purge candidates

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): address soft-drop retention review

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): protect retained table tombstones

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): preserve post-cleanup purge state

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(meta): fence purge by drop generation

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
 2026-07-16 08:47:03 +00:00
Weny Xu 05c4588f90 feat: support remote WAL logical pruning (#8259) 
* feat: support logical deletion for remote WAL pruning

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: trigger remote WAL flush for lagging prunable regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: use from_mins

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-10 04:40:51 +00:00
June dc5fab93a8 fix: remove unparsed [heartbeat] sections from node example configs (#8092) 
Fixes #8087

FrontendOptions, DatanodeOptions, and FlownodeOptions do not define
a heartbeat field, so the [heartbeat] sections in their example TOML
files were never parsed.

Heartbeat intervals are actually negotiated from metasrv during the
heartbeat handshake:
- Datanode/Flownode: interval = metasrv.heartbeat_interval
- Frontend: interval = metasrv.heartbeat_interval * 6

This mislead operators into thinking they could tune heartbeat timing
locally. Remove the sections to eliminate confusion.

Metasrv's heartbeat_interval remains documented and functional.
 2026-05-11 08:34:51 +00:00
QuakeWang 8825ea3fdf fix!: align gRPC CLI option names with config naming (#8021) 
* fix: align gRPC CLI option names with config naming

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: warn on deprecated metasrv grpc config

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
 2026-04-24 09:51:01 +00:00
discord9 00f568ed28 fix: gc update repart map properly (#7606) 
* feat: update repart map

Signed-off-by: discord9 <discord9@163.com>

* fix: table id write lock

Signed-off-by: discord9 <discord9@163.com>

* chore: default value

Signed-off-by: discord9 <discord9@163.com>

* chore: config

Signed-off-by: discord9 <discord9@163.com>

* test: update repartition map

Signed-off-by: discord9 <discord9@163.com>

* fix: empty file ref set

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: properly log error

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-01-28 04:31:19 +00:00
Lanqing Yang 8d9afc83e3 feat: allow auto schema creation for pg (#7459) 
Signed-off-by: lyang24 <lanqingy93@gmail.com>
 2025-12-23 08:55:24 +00:00
LFC 625fdd09ea refactor!: remove not working metasrv cli option (#7446) 
Signed-off-by: luofucong <luofc@foxmail.com>
 2025-12-23 06:55:17 +00:00
Weny Xu 4d66bd96b8 feat: make distributed time constants and client timeouts configurable (#7433) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-19 02:23:20 +00:00
fys c9a7b1fd68 docs(config): clarify store_addrs format (#7279) 2025-11-21 22:26:52 +00:00
WaterWhisperer de9ae6066f refactor: remove export_metrics and related configuration (#7236) 
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>
 2025-11-17 02:32:22 +00:00
Ruihang Xia f65dcd12cc feat: refine failure detector (#7005) 
* feat: refine failure detector

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert back default value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert change of test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-09-24 01:43:22 +00:00
Weny Xu dbb76483e8 chore: disable stats persistence by default (#6900) 
* chore: disable stats persistence by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-09-03 12:39:43 +00:00
LogicandWeny Xu cbcfdf9d65 feat: add optional schema for Postgres metadata tables (#6764) 
* feat(meta): add optional schema for Postgres metadata tables

- Add `schema` option to specify a custom schema for metadata tables
- Update `PgStore` and `PgElection` to support optional schema
- Modify SQL templates to use schema when provided
- Add tests for schema support in Postgres backend

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(meta): remove unused `create_schema_statement` and simplify `PgSqlTemplateFactory`

- Remove `create_schema_statement` from `PgSqlTemplateSet` struct
- Simplify `PgSqlTemplateFactory` by removing `new` method and merging it with `with_schema`
- Update related tests to reflect these changes

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(meta-srv): remove unused imports

- Remove unused import of BoxedError from common_error::ext- Remove unused import of TlsOption from servers::tls

Signed-off-by: Logic <zqr10159@dromara.org>

* build(meta): update Postgres version and add error handling imports

- Update Postgres version to 17 in docker-compose.yml
- Add BoxedError import for error handling in meta-srv

Signed-off-by: Logic <zqr10159@dromara.org>

* feat(postgres): add support for optional schema in PgElection and related components

Signed-off-by: Logic <zqr10159@dromara.org>

* feat(postgres): add support for optional schema in PgElection and related components

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(develop): update Postgres schema commands to specify host

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(postgres): simplify plugin options handling and update SQL examples

Signed-off-by: Logic <zqr10159@dromara.org>

* refactor(postgres): simplify plugin options handling and update SQL examples

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(postgres): update meta_election_lock_id description for optional schema support

Signed-off-by: Logic <zqr10159@dromara.org>

* fix(postgres): add health check and fallback wait for Postgres in CI setup

* fix(postgres): update Docker setup for Postgres and add support for Postgres 15

* fix(postgres): remove redundant Postgres setup step in CI configuration

* Update tests-integration/fixtures/postgres/init.sql

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update .github/workflows/develop.yml

* Update tests-integration/fixtures/docker-compose.yml

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/common/meta/src/kv_backend/rds/postgres.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: Refactor PostgreSQL backend to support optional schema in PgStore and related SQL templates

* feat: Update PostgreSQL configuration and add PG15 specific integration tests

* feat: Update PostgreSQL configuration and add PG15 specific integration tests

* refactor(postgres): update test schemas from 'greptime_schema' to 'test_schema'

* Update .github/workflows/develop.yml

* refactor: minor factor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit test

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Logic <zqr10159@dromara.org>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2025-08-28 09:24:14 +00:00
Weny Xu f441598247 fix: correct config doc (#6836) 
fix: fix typo

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-28 06:26:25 +00:00
Yan Tingwang 32a3ef36f9 feat(metasrv): support tls for etcd client (#6818) 
* add TLS support for etcd client connections~

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* locate correct certs

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated certs

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated CI

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Updated CI

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* Update docker-compose.yml

* tests for TLS client creation

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

* modify tests

Signed-off-by: codephage2020 <tingwangyan2020@163.com>

---------

Signed-off-by: codephage2020 <tingwangyan2020@163.com>
 2025-08-27 07:41:05 +00:00
Weny Xu 566a647ec7 feat: add replay checkpoint to reduce overhead for remote WAL (#6816) 
* feat: introduce `TopicRegionValue`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: persist region replay checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: udpate config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: send open region instructions with reply checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: use usize

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add topic name pattern

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: enable wal prune by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-27 07:24:33 +00:00
Ning Sun cdc168e753 feat: support for custom headers in otel exporter (#6773) 
* feat: support for custom headers in otel exporter

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: remove wrapping option

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2025-08-21 13:10:13 +00:00
Weny XuandYingwen 896d72191e feat: introduce PersistStatsHandler (#6777) 
* feat: add `Inserter` trait and impl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: import items

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce `PersistStatsHandler`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: disable persisting stats in sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* reset channel manager

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: avoid to collect

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove insert options

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: use `write_bytes` instead of `write_bytes_per_sec`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: compute write bytes delta

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* Update src/meta-srv/src/handler/persist_stats_handler.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2025-08-21 09:34:58 +00:00
Ning Sun 5a4036cc66 feat: update opentelemetry family (#6762) 
* feat: update opentelemetry family

Signed-off-by: Ning Sun <sunning@greptime.com>

* doc: update doc samples

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: toml format

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: update default otel enpoint

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2025-08-19 09:23:50 +00:00
Weny Xu 021ad09c21 refactor: simplify WAL pruning procedure and introduce region flush trigger (#6741) 
* chore: add logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: update wal config for metasrv

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce region flush trigger

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: debug assert

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: log level

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: simplify wal prune procedure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: upgrade rskafka

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: always flush inactive regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: refactor flush trigger

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused code

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: typo

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: update unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-14 14:15:30 +00:00
zyy17 777da35b0d refactor: unify the event recorder (#6689) 
* refactor: unify the event recorder

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `table_name()` in `Event` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `slow_query_options` in `Instance`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `EventHandlerOptions` and `options()` in `EventHandler` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `aggregate_events_by_type()` and support log mode of slow query

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: support to set ttl by using extension of query context

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine the configs fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: sqlness test errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use `Duration` type instead of `String` for ttl fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use pre-allocation for building RowInsertRequests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: polish code for `group_events_by_type()` and `build_row_inserts_request()`, also add the unit tests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine comments

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-12 18:26:12 +00:00
zyy17 cc35bab5e4 feat: record the migration events in metasrv (#6579) 
* feat: collect procedure event

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* feat: collect region migration events

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* test: add integration test

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix docs error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration test error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: change status code for errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `event()` in Procedure

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve trait design

1. Add `user_metadata()` in `Procedure` trait;

2. Add `Eventable` trait;

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-06 03:30:33 +00:00
Ruihang Xia 572e29b158 feat: support tls for pg backend (#6611) 
* load tls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl tls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* pass options

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement require mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* default to prefer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update example config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust example config

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle client cert and key properly

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement verify_ca and verify_full

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update integration test for config api

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change config name and default mode

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-04 00:41:08 +00:00
Yingwen 9527e0df2f feat: HTTP API to activate/deactive heap prof (activate by default) (#6593) 
* feat: add HTTP API to activate/deactivate heap profiling

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add HTTP API to get profiling status

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: enable heap prof by default

Signed-off-by: evenyag <realevenyag@gmail.com>

* build: add "prof:true,prof_active:false" as default env to dockerfiles

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: activate heap profiling after log initialization

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add memory options to control whether to activate profiling

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update docs

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt toml

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: fix config test

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: usage of new api

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: log profile after version

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update how to docs

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: fix how to docs

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-01 03:24:56 +00:00
Weny Xu 5fcb97724f chore: correct typo in configuration (#6411) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-06-27 07:55:13 +00:00
Weny Xu 24019334ee feat: implement automatic region failure detector registrations (#6370) 
* feat: implement automatic region failure detector registrations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add more tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add `region_failure_detector_initialization_delay` option

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-06-24 06:12:12 +00:00
zyy17andCopilot a49edc6ca6 refactor: add otlp_export_protocol config to support export trace data through gRPC and HTTP protocol (#6357) 
* refactor: support http traces

* refactor: add `otlp_export_protocol` config to support export trace data through gRPC and HTTP protocol

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* Update src/common/telemetry/src/logging.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-06-23 02:56:12 +00:00
jeremyhi 12648f388a feat: refactor grpc options of metasrv (#6275) 
* feat: refactor grpc options of metasrv

* fix: unit test

* feat: config update
 2025-06-10 01:35:57 +00:00
dennis zhuang 0311db3089 fix: export metrics settings in sample config (#6170) 
* fix: export metrics settings in sample config

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: forgot update docs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: test loading example configs

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
 2025-06-09 10:53:01 +00:00
zyy17 fdf32a8f46 refactor: respect data_home as root data home directory (#6050) 
* refactor: initialize logging dir by using data_home

* chore: remove tail '/' for dir name
 2025-06-09 02:31:21 +00:00
jeremyhi 3104d49434 chore: example of http config in metasrv (#6218) 
* chore: example of http config in metasrv

* docs: make config-docs
 2025-05-30 03:27:54 +00:00
zyy17 14b655ea57 refactor: add SlowQueryRecorder to record slow query in system table and refactor slow query options (#6008) 
* refactor: add common-slow-query crate

* refactor: refine the naming

* chore: fix clippy

* chore: fix typo

* chore: sperate SlowQueryOptions From Logging

* chore: fix clippy

* chore: fix ci

* chore: refine the code

* chore: update config example

* refactor: use drop() to end the slow query timer

* refactor: move common-slow-query to frontend crate

* chore: polish some code

* refactor: code review

* refactor: add promql_range/promql_step/promql_start/promql_end fields in slow_queries

* refactor: add build_slow_query_logger()

* refactor: turn on slow query on frontend by default
 2025-05-15 04:18:48 +00:00
Weny Xu e30753fc31 feat: allow forced region failover for local WAL (#5972) 
* feat: allow forced region failover for local WAL

* chore: upgrade config.md

* chore: apply suggestions from CR
 2025-04-24 08:11:45 +00:00
Yuhan Wang e817a65d75 feat: enable submitting wal prune procedure periodically (#5867) 
* feat: enable submitting wal prune procedure periodically

* chore: fix and add options

* test: add unit test

* test: fix unit test

* test: enable active_wal_pruning in test

* test: update default config

* chore: update config name

* refactor: use semaphore to control the number of prune process

* refactor: use split client for wal prune manager and topic creator

* chore: add configs

* chore: apply review comments

* fix: use tracker properly

* fix: use guard to track semaphore

* test: update unit tests

* chore: update config name

* chore: use prunable_entry_id

* refactor: semaphore to only limit the process of submitting

* chore: remove legacy sort

* chore: better configs

* fix: update config.md

* chore: respect fmt

* test: update unit tests

* chore: use interval_at

* fix: fix unit test

* test: fix unit test

* test: fix unit test

* chore: apply review comments

* docs: update config docs
 2025-04-18 16:02:33 +00:00
Weny Xu ac0f9ab575 refactor: remove backoff config (#5808) 
* refactor: remove backoff config

* chore: update config.md

* fix: correct backoff config

* chore: change deadline to 120s
 2025-04-07 07:22:22 +00:00
Weny Xu d3a60d8821 feat: add limit for the number of running procedures (#5793) 
* refactor: remove unused `messages`

* feat: introduce running procedure num limit

* feat: update config

* chore: apply suggestions from CR

* feat: impl `status_code` for `log-store` crate
 2025-03-31 06:14:21 +00:00
Ning Sun dc76571166 feat: move default data path from /tmp to current directory (#5719) 2025-03-16 09:57:46 +00:00