Yingwen
b8e0c49cb4
feat: add an flag to enable the experimental flat format ( #6976 )
...
* feat: add enable_experimental_flat_format flag to enable flat format
Signed-off-by: evenyag <realevenyag@gmail.com >
* refactor: extract build_scan_input for CompactionSstReaderBuilder
Signed-off-by: evenyag <realevenyag@gmail.com >
* chore: add compact memtable cost to flush metrics
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: Sets compact dispatcher for bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: Cast dictionary to target type in FlatProjectionMapper
Signed-off-by: evenyag <realevenyag@gmail.com >
* fix: add time index to FlatProjectionMapper::batch_schema
Signed-off-by: evenyag <realevenyag@gmail.com >
* chore: update config toml
Signed-off-by: evenyag <realevenyag@gmail.com >
* fix: pass flat_format to ProjectionMapper in CompactionSstReaderBuilder
Signed-off-by: evenyag <realevenyag@gmail.com >
---------
Signed-off-by: evenyag <realevenyag@gmail.com >
2025-09-16 09:33:12 +00:00
Weny Xu
dbb76483e8
chore: disable stats persistence by default ( #6900 )
...
* chore: disable stats persistence by default
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: fix clippy
Signed-off-by: WenyXu <wenymedia@gmail.com >
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-09-03 12:39:43 +00:00
Yingwen
ab96703d8f
chore: enlarge max file limit to 384 ( #6868 )
...
chore: enlarge max concurrent files to 384
Signed-off-by: evenyag <realevenyag@gmail.com >
2025-09-01 09:26:20 +00:00
Logic
cbcfdf9d65
feat: add optional schema for Postgres metadata tables ( #6764 )
...
* feat(meta): add optional schema for Postgres metadata tables
- Add `schema` option to specify a custom schema for metadata tables
- Update `PgStore` and `PgElection` to support optional schema
- Modify SQL templates to use schema when provided
- Add tests for schema support in Postgres backend
Signed-off-by: Logic <zqr10159@dromara.org >
* refactor(meta): remove unused `create_schema_statement` and simplify `PgSqlTemplateFactory`
- Remove `create_schema_statement` from `PgSqlTemplateSet` struct
- Simplify `PgSqlTemplateFactory` by removing `new` method and merging it with `with_schema`
- Update related tests to reflect these changes
Signed-off-by: Logic <zqr10159@dromara.org >
* refactor(meta-srv): remove unused imports
- Remove unused import of BoxedError from common_error::ext- Remove unused import of TlsOption from servers::tls
Signed-off-by: Logic <zqr10159@dromara.org >
* build(meta): update Postgres version and add error handling imports
- Update Postgres version to 17 in docker-compose.yml
- Add BoxedError import for error handling in meta-srv
Signed-off-by: Logic <zqr10159@dromara.org >
* feat(postgres): add support for optional schema in PgElection and related components
Signed-off-by: Logic <zqr10159@dromara.org >
* feat(postgres): add support for optional schema in PgElection and related components
Signed-off-by: Logic <zqr10159@dromara.org >
* fix(develop): update Postgres schema commands to specify host
Signed-off-by: Logic <zqr10159@dromara.org >
* refactor(postgres): simplify plugin options handling and update SQL examples
Signed-off-by: Logic <zqr10159@dromara.org >
* refactor(postgres): simplify plugin options handling and update SQL examples
Signed-off-by: Logic <zqr10159@dromara.org >
* fix(postgres): update meta_election_lock_id description for optional schema support
Signed-off-by: Logic <zqr10159@dromara.org >
* fix(postgres): add health check and fallback wait for Postgres in CI setup
* fix(postgres): update Docker setup for Postgres and add support for Postgres 15
* fix(postgres): remove redundant Postgres setup step in CI configuration
* Update tests-integration/fixtures/postgres/init.sql
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update .github/workflows/develop.yml
* Update tests-integration/fixtures/docker-compose.yml
* Update src/common/meta/src/kv_backend/rds/postgres.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/common/meta/src/kv_backend/rds/postgres.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/common/meta/src/kv_backend/rds/postgres.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* Update src/common/meta/src/kv_backend/rds/postgres.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fix: Refactor PostgreSQL backend to support optional schema in PgStore and related SQL templates
* feat: Update PostgreSQL configuration and add PG15 specific integration tests
* feat: Update PostgreSQL configuration and add PG15 specific integration tests
* refactor(postgres): update test schemas from 'greptime_schema' to 'test_schema'
* Update .github/workflows/develop.yml
* refactor: minor factor
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: fix unit test
Signed-off-by: WenyXu <wenymedia@gmail.com >
---------
Signed-off-by: Logic <zqr10159@dromara.org >
Signed-off-by: WenyXu <wenymedia@gmail.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2025-08-28 09:24:14 +00:00
discord9
bacd9c7d15
feat: add event ts to region manifest ( #6751 )
...
* feat: add event ts to region manifest
Signed-off-by: discord9 <discord9@163.com >
delete files
Signed-off-by: discord9 <discord9@163.com >
chore: clippy
Signed-off-by: discord9 <discord9@163.com >
lower concurrency
Signed-off-by: discord9 <discord9@163.com >
feat: gc use delta manifest to get expel time
Signed-off-by: discord9 <discord9@163.com >
docs: terminalogy
Signed-off-by: discord9 <discord9@163.com >
refactor: some advices from review
Signed-off-by: discord9 <discord9@163.com >
feat: manifest add removed files field
Signed-off-by: discord9 <discord9@163.com >
feat(WIP): add remove time in manifest
Signed-off-by: discord9 <discord9@163.com >
wip: more config
Signed-off-by: discord9 <discord9@163.com >
feat: manifest uodate removed files
Signed-off-by: discord9 <discord9@163.com >
test: add remove file opts field
Signed-off-by: discord9 <discord9@163.com >
test: fix test
Signed-off-by: discord9 <discord9@163.com >
chore: delete gc.rs
Signed-off-by: discord9 <discord9@163.com >
* feat: proper option name
Signed-off-by: discord9 <discord9@163.com >
* refactor: per review
Signed-off-by: discord9 <discord9@163.com >
* test: update manifest size
Signed-off-by: discord9 <discord9@163.com >
* test: fix eq
Signed-off-by: discord9 <discord9@163.com >
* refactor: some per review
Signed-off-by: discord9 <discord9@163.com >
* refactor: per review
Signed-off-by: discord9 <discord9@163.com >
* refactor: more per review
Signed-off-by: discord9 <discord9@163.com >
* test: update manifest size
Signed-off-by: discord9 <discord9@163.com >
* test: update config
Signed-off-by: discord9 <discord9@163.com >
* feat: keep count 0 means only use ttl
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-08-28 09:08:18 +00:00
Weny Xu
f441598247
fix: correct config doc ( #6836 )
...
fix: fix typo
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-08-28 06:26:25 +00:00
discord9
5ef4dd1743
docs: add internal grpc ports ( #6815 )
...
* docs: add internal grpc ports
Signed-off-by: discord9 <discord9@163.com >
* fix: update example toml
Signed-off-by: discord9 <discord9@163.com >
* fix: grpc option use default for missing field
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-08-27 08:20:27 +00:00
Yan Tingwang
32a3ef36f9
feat(metasrv): support tls for etcd client ( #6818 )
...
* add TLS support for etcd client connections~
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* locate correct certs
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* Updated certs
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* Updated CI
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* Updated CI
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* Update docker-compose.yml
* tests for TLS client creation
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
* modify tests
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
---------
Signed-off-by: codephage2020 <tingwangyan2020@163.com >
2025-08-27 07:41:05 +00:00
Weny Xu
566a647ec7
feat: add replay checkpoint to reduce overhead for remote WAL ( #6816 )
...
* feat: introduce `TopicRegionValue`
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: persist region replay checkpoint
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: introduce checkpoint
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: udpate config.md
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: minor refactor
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: send open region instructions with reply checkpoint
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: use usize
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: fix unit tests
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: fix unit tests
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: add topic name pattern
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: enable wal prune by default
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-08-27 07:24:33 +00:00
Ning Sun
cdc168e753
feat: support for custom headers in otel exporter ( #6773 )
...
* feat: support for custom headers in otel exporter
Signed-off-by: Ning Sun <sunning@greptime.com >
* chore: remove wrapping option
Signed-off-by: Ning Sun <sunning@greptime.com >
---------
Signed-off-by: Ning Sun <sunning@greptime.com >
2025-08-21 13:10:13 +00:00
Weny Xu
896d72191e
feat: introduce PersistStatsHandler ( #6777 )
...
* feat: add `Inserter` trait and impl
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: import items
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: introduce `PersistStatsHandler`
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: disable persisting stats in sqlness
Signed-off-by: WenyXu <wenymedia@gmail.com >
* reset channel manager
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: avoid to collect
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: remove insert options
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: use `write_bytes` instead of `write_bytes_per_sec`
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: compute write bytes delta
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* test: add unit tests
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* Update src/meta-srv/src/handler/persist_stats_handler.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2025-08-21 09:34:58 +00:00
Ning Sun
5a4036cc66
feat: update opentelemetry family ( #6762 )
...
* feat: update opentelemetry family
Signed-off-by: Ning Sun <sunning@greptime.com >
* doc: update doc samples
Signed-off-by: Ning Sun <sunning@greptime.com >
* chore: toml format
Signed-off-by: Ning Sun <sunning@greptime.com >
* chore: update default otel enpoint
Signed-off-by: Ning Sun <sunning@greptime.com >
---------
Signed-off-by: Ning Sun <sunning@greptime.com >
2025-08-19 09:23:50 +00:00
discord9
dfc29eb3b3
feat: flownode grpc client to frontend tls option ( #6750 )
...
* feat: flownode grpc client to frontend tls option
Signed-off-by: discord9 <discord9@163.com >
* refactor: client tls option
Signed-off-by: discord9 <discord9@163.com >
* refactor: client_tls to frontend_tls
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-08-15 10:44:27 +00:00
Weny Xu
021ad09c21
refactor: simplify WAL pruning procedure and introduce region flush trigger ( #6741 )
...
* chore: add logs
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: update wal config for metasrv
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: introduce region flush trigger
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: debug assert
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: log level
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: simplify wal prune procedure
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: upgrade rskafka
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: always flush inactive regions
Signed-off-by: WenyXu <wenymedia@gmail.com >
* refactor: refactor flush trigger
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: remove unused code
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: typo
Signed-off-by: WenyXu <wenymedia@gmail.com >
* fix: update unit tests
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: add metrics
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: rename
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com >
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-08-14 14:15:30 +00:00
sunheyi
d03f85287e
feat: mysql add prepared_stmt_cache_capacity ( #6639 )
...
* feat: your clear and concise commit message
Signed-off-by: sunheyi <1061867552@qq.com >
* fix error
Signed-off-by: sunheyi <1061867552@qq.com >
* add param
Signed-off-by: sunheyi <1061867552@qq.com >
* fix
Signed-off-by: sunheyi <1061867552@qq.com >
* fix doc error
Signed-off-by: sunheyi <1061867552@qq.com >
---------
Signed-off-by: sunheyi <1061867552@qq.com >
2025-08-14 08:19:10 +00:00
zyy17
777da35b0d
refactor: unify the event recorder ( #6689 )
...
* refactor: unify the event recorder
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: add `table_name()` in `Event` trait
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: add `slow_query_options` in `Instance`
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: add `EventHandlerOptions` and `options()` in `EventHandler` trait
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: add `aggregate_events_by_type()` and support log mode of slow query
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: polish the code
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* fix: clippy errors
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: support to set ttl by using extension of query context
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: refine the configs fields
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* fix: sqlness test errors
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: use `Duration` type instead of `String` for ttl fields
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: use pre-allocation for building RowInsertRequests
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: fix clippy errors
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: code review
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: fix integration errors
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: polish code for `group_events_by_type()` and `build_row_inserts_request()`, also add the unit tests
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: refine comments
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com >
2025-08-12 18:26:12 +00:00
LFC
253d89b5cc
feat: able to set read preference to flownode ( #6696 )
...
fix: correctly compare the opened follower regions in startup
Signed-off-by: luofucong <luofc@foxmail.com >
2025-08-08 09:08:09 +00:00
zyy17
cc35bab5e4
feat: record the migration events in metasrv ( #6579 )
...
* feat: collect procedure event
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* feat: collect region migration events
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* test: add integration test
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: fix docs error
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: fix integration test error
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: change status code for errors
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: add `event()` in Procedure
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* refactor: improve trait design
1. Add `user_metadata()` in `Procedure` trait;
2. Add `Eventable` trait;
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* chore: polish the code
Signed-off-by: zyy17 <zyylsxm@gmail.com >
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com >
2025-08-06 03:30:33 +00:00
Yingwen
280024d7f8
feat: Add option to limit the files reading simultaneously ( #6635 )
...
* feat: limits the max number of files to scan at the same time
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: make max_concurrent_scan_files configurable
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: reduce concurrent scan files to 128
Signed-off-by: evenyag <realevenyag@gmail.com >
* docs: update config example
Signed-off-by: evenyag <realevenyag@gmail.com >
* test: add test for max_concurrent_scan_files
Signed-off-by: evenyag <realevenyag@gmail.com >
* style: fix clippy
Signed-off-by: evenyag <realevenyag@gmail.com >
* test: update config test
Signed-off-by: evenyag <realevenyag@gmail.com >
---------
Signed-off-by: evenyag <realevenyag@gmail.com >
2025-08-04 07:18:58 +00:00
Ruihang Xia
572e29b158
feat: support tls for pg backend ( #6611 )
...
* load tls
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* impl tls
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* pass options
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement require mode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* clean up
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* default to prefer
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update example config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* adjust example config
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* handle client cert and key properly
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* implement verify_ca and verify_full
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update integration test for config api
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change config name and default mode
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-08-04 00:41:08 +00:00
Yingwen
9527e0df2f
feat: HTTP API to activate/deactive heap prof (activate by default) ( #6593 )
...
* feat: add HTTP API to activate/deactivate heap profiling
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: add HTTP API to get profiling status
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: enable heap prof by default
Signed-off-by: evenyag <realevenyag@gmail.com >
* build: add "prof:true,prof_active:false" as default env to dockerfiles
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: activate heap profiling after log initialization
Signed-off-by: evenyag <realevenyag@gmail.com >
* feat: add memory options to control whether to activate profiling
Signed-off-by: evenyag <realevenyag@gmail.com >
* docs: update docs
Signed-off-by: evenyag <realevenyag@gmail.com >
* chore: fmt toml
Signed-off-by: evenyag <realevenyag@gmail.com >
* test: fix config test
Signed-off-by: evenyag <realevenyag@gmail.com >
* docs: usage of new api
Signed-off-by: evenyag <realevenyag@gmail.com >
* chore: log profile after version
Signed-off-by: evenyag <realevenyag@gmail.com >
* docs: update how to docs
Signed-off-by: evenyag <realevenyag@gmail.com >
* docs: fix how to docs
Signed-off-by: evenyag <realevenyag@gmail.com >
---------
Signed-off-by: evenyag <realevenyag@gmail.com >
2025-08-01 03:24:56 +00:00
discord9
8fef177575
feat: fallback when failed to push down using DistPlanner ( #6574 )
...
* test: fix fallback testcase
Signed-off-by: discord9 <discord9@163.com >
* add metric
Signed-off-by: discord9 <discord9@163.com >
* feat: fallback add to config variable
Signed-off-by: discord9 <discord9@163.com >
* feat: set in var&set in hint
Signed-off-by: discord9 <discord9@163.com >
* chore: update test
Signed-off-by: discord9 <discord9@163.com >
* feat: also in toml config
Signed-off-by: discord9 <discord9@163.com >
* fix test
Signed-off-by: discord9 <discord9@163.com >
* docs: comment about setting from different source
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-07-29 04:37:00 +00:00
discord9
bbc9f3ea1e
refactor: expose flow batching mode constants to config ( #6442 )
...
* refactor: make flow batching mode constant to configs
Signed-off-by: discord9 <discord9@163.com >
* docs: config docs
Signed-off-by: discord9 <discord9@163.com >
* docs: update code comment
Signed-off-by: discord9 <discord9@163.com >
* test: fix test_config_api
Signed-off-by: discord9 <discord9@163.com >
* feat: more batch opts
Signed-off-by: discord9 <discord9@163.com >
* fix after rebase
Signed-off-by: discord9 <discord9@163.com >
* chore: per review
Signed-off-by: discord9 <discord9@163.com >
* per review experimental options
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-07-16 08:05:20 +00:00
discord9
616e76941a
feat: flow query parallel=1&query faster with many windows&min one time window ( #6324 )
...
* feat: flow query parallel=1&query faster when
windows too many&min one time window
Signed-off-by: discord9 <discord9@163.com >
* chore: default flow query parallelism=1
Signed-off-by: discord9 <discord9@163.com >
* refactor: use query options in flownode per review
Signed-off-by: discord9 <discord9@163.com >
* docs: update comment
Signed-off-by: discord9 <discord9@163.com >
* chore: fix test
Signed-off-by: discord9 <discord9@163.com >
* chore: per review
Signed-off-by: discord9 <discord9@163.com >
* chore: make config docs
Signed-off-by: discord9 <discord9@163.com >
---------
Signed-off-by: discord9 <discord9@163.com >
2025-06-30 02:17:01 +00:00
Weny Xu
5fcb97724f
chore: correct typo in configuration ( #6411 )
...
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-06-27 07:55:13 +00:00
Weny Xu
24019334ee
feat: implement automatic region failure detector registrations ( #6370 )
...
* feat: implement automatic region failure detector registrations
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: remove unused error
Signed-off-by: WenyXu <wenymedia@gmail.com >
* test: add more tests
Signed-off-by: WenyXu <wenymedia@gmail.com >
* feat: add `region_failure_detector_initialization_delay` option
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: update config.md
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com >
* chore: update config.md
Signed-off-by: WenyXu <wenymedia@gmail.com >
---------
Signed-off-by: WenyXu <wenymedia@gmail.com >
2025-06-24 06:12:12 +00:00
zyy17
a49edc6ca6
refactor: add otlp_export_protocol config to support export trace data through gRPC and HTTP protocol ( #6357 )
...
* refactor: support http traces
* refactor: add `otlp_export_protocol` config to support export trace data through gRPC and HTTP protocol
Signed-off-by: zyy17 <zyylsxm@gmail.com >
* Update src/common/telemetry/src/logging.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-06-23 02:56:12 +00:00
rgidda
e026f766d2
feat(storage): Add skip_ssl_validation option for object storage HTTP client ( #6358 )
...
* feat(storage): Add skip_ssl_validation option for object storage HTTP client
Signed-off-by: rgidda <rgidda@hitachivantara.com >
* fix(test): Broken test case for - Add skip_ssl_validation option for object storage HTTP client
Signed-off-by: rgidda <rgidda@hitachivantara.com >
* fix: test
* fix: test
---------
Signed-off-by: rgidda <rgidda@hitachivantara.com >
Co-authored-by: rgidda <rgidda@hitachivantara.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2025-06-20 08:08:19 +00:00
Lei, HUANG
f9f905ae14
fix: config docs ( #6294 )
...
fix/config-docs:
Update `config.md` to specify default compression mode
- Added default value `none` for `grpc.flight_compression` in both frontend and datanode sections of `config/config.md`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com >
2025-06-11 07:31:42 +00:00
Lei, HUANG
1d53dd26ae
chore: add option for arrow flight compression mode ( #6283 )
...
* chore/enable-flight-encoder:
### Add Flight Compression Support
- **Configuration Updates**:
- Added `grpc.flight_compression` option to `config/config.md`, `config/datanode.example.toml`, and `config/frontend.example.toml` to specify compression modes for Arrow IPC service.
- **Code Enhancements**:
- Updated `FlightEncoder` in `src/common/grpc/src/flight.rs` to support compression modes.
- Modified `RegionServer` and `DatanodeBuilder` in `src/datanode/src/datanode.rs` and `src/datanode/src/region_server.rs` to handle `FlightCompression`.
- Integrated `FlightCompression` in `src/servers/src/grpc.rs` and `src/servers/src/grpc/flight.rs` to manage compression settings.
- **Testing and Integration**:
- Updated test utilities and integration tests in `tests-integration/src/grpc/flight.rs` and `tests-integration/src/test_util.rs` to include `FlightCompression`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com >
* chore/enable-flight-encoder:
### Enable Compression in FlightClient
- **`client.rs`**: Updated `make_flight_client` to accept `send_compression` and `accept_compression` parameters, enabling Zstd compression for sending and receiving messages.
- **`client_manager.rs`**: Modified `datanode` method to pass compression settings from `ChannelConfig` to `RegionRequester`.
- **`database.rs`**: Adjusted calls to `make_flight_client` to include compression parameters.
- **`region.rs`**: Updated `RegionRequester` to store and utilize compression settings.
- **`frontend.rs`**: Configured `ChannelConfig` to enable compression based on options.
- **`channel_manager.rs`**: Added `send_compression` and `accept_compression` fields to `ChannelConfig` with default values and updated tests accordingly.
Signed-off-by: Lei, HUANG <lhuang@greptime.com >
* chore/enable-flight-encoder:
### Update Compression Defaults and Documentation
- **Configuration Files**: Updated `datanode.example.toml` and `frontend.example.toml` to include a default setting comment for `flight_compression`, specifying it defaults to `none`.
- **gRPC Server Code**: Modified `grpc.rs` to set `None` as the default for `FlightCompression` instead of `ArrowIpc`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com >
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com >
Signed-off-by: Lei, HUANG <lhuang@greptime.com >
2025-06-11 06:54:10 +00:00
jeremyhi
12648f388a
feat: refactor grpc options of metasrv ( #6275 )
...
* feat: refactor grpc options of metasrv
* fix: unit test
* feat: config update
2025-06-10 01:35:57 +00:00
dennis zhuang
0311db3089
fix: export metrics settings in sample config ( #6170 )
...
* fix: export metrics settings in sample config
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* chore: forgot update docs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
* fix: test loading example configs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com >
2025-06-09 10:53:01 +00:00
zyy17
fdf32a8f46
refactor: respect data_home as root data home directory ( #6050 )
...
* refactor: initialize logging dir by using data_home
* chore: remove tail '/' for dir name
2025-06-09 02:31:21 +00:00
jeremyhi
3104d49434
chore: example of http config in metasrv ( #6218 )
...
* chore: example of http config in metasrv
* docs: make config-docs
2025-05-30 03:27:54 +00:00
Lei, HUANG
183e1dc031
feat(http): lossy string validation in prom remote write ( #6213 )
...
* feat/lossy-string-validation-in-prom-remote-write:
### Commit Message
#### Refactor Prometheus Validation Mode
- **Replace `is_strict_mode` with `PromValidationMode` Enum:**
- Updated `HttpOptions` and related structures to use `PromValidationMode` enum instead of the boolean `is_strict_mode`.
- Modified functions and tests to accommodate the new enum, ensuring flexible validation modes (`Strict`, `Lossy`, `Unchecked`).
- Affected files: `server.rs`, `prom_decode.rs`, `http.rs`, `prom_store.rs`, `prom_row_builder.rs`, `proto.rs`, `prom_store_test.rs`, `test_util.rs`, `http.rs`.
- **Enhance UTF-8 String Decoding:**
- Introduced `decode_string` function to handle UTF-8 string decoding based on the selected `PromValidationMode`.
- Affected files: `proto.rs`, `prom_row_builder.rs`.
This refactor improves the flexibility and clarity of Prometheus request handling by allowing different validation strategies.
* feat/lossy-string-validation-in-prom-remote-write:
- **Add Prometheus Validation Mode Configuration:**
- Updated `config/config.md`, `config/frontend.example.toml`, and `config/standalone.example.toml` to include `http.prom_validation_mode` setting for Prometheus remote write requests.
- **Enhance Benchmarking for Prometheus Requests:**
- Modified `src/servers/benches/prom_decode.rs` to benchmark different Prometheus validation modes (`Strict`, `Lossy`, `Unchecked`).
- **Implement and Test String Decoding:**
- Added `decode_string` function and comprehensive tests in `src/servers/src/proto.rs` to handle string decoding with different validation modes.
* feat/lossy-string-validation-in-prom-remote-write:
### Add Histogram Buckets to Metrics
- **Files Modified**: `src/servers/src/metrics.rs`
- **Key Changes**:
- Added specific histogram buckets to `METRIC_MYSQL_QUERY_TIMER`, `METRIC_POSTGRES_QUERY_TIMER`, and `METRIC_SERVER_GRPC_PROM_REQUEST_TIMER` to enhance granularity in query elapsed time metrics.
* feat/lossy-string-validation-in-prom-remote-write:
### Update Prometheus Validation Mode Default
- **Config Documentation**: Updated the default description for `http.prom_validation_mode` to indicate that "strict" is the default option in `config.md`, `frontend.example.toml`, and `standalone.example.toml`.
- **HTTP Server Implementation**: Changed the default `prom_validation_mode` to `PromValidationMode::Strict` in `src/servers/src/http.rs`.
* feat/lossy-string-validation-in-prom-remote-write:
**Commit Message:**
Update Prometheus Validation Mode to Strict
- Changed `http.prom_validation_mode` from `unchecked` to `strict` in `config.md`, `frontend.example.toml`, and
`standalone.example.toml` to enforce strict validation of Prometheus remote write requests.
2025-05-29 11:08:57 +00:00
Zhenchi
400229c384
feat: introduce index result cache ( #6110 )
...
* feat: introduce index result cache
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/mito2/src/sst/index/inverted_index/applier/builder.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* optimize selector_len
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2025-05-20 01:45:42 +00:00
zyy17
01076069a3
chore: modify default slow_query.threshold from 5s to 30s ( #6107 )
...
chore: modify slow_query.threshold from 5s to 30s
2025-05-15 20:16:13 +00:00
zyy17
14b655ea57
refactor: add SlowQueryRecorder to record slow query in system table and refactor slow query options ( #6008 )
...
* refactor: add common-slow-query crate
* refactor: refine the naming
* chore: fix clippy
* chore: fix typo
* chore: sperate SlowQueryOptions From Logging
* chore: fix clippy
* chore: fix ci
* chore: refine the code
* chore: update config example
* refactor: use drop() to end the slow query timer
* refactor: move common-slow-query to frontend crate
* chore: polish some code
* refactor: code review
* refactor: add promql_range/promql_step/promql_start/promql_end fields in slow_queries
* refactor: add build_slow_query_logger()
* refactor: turn on slow query on frontend by default
2025-05-15 04:18:48 +00:00
Weny Xu
e30753fc31
feat: allow forced region failover for local WAL ( #5972 )
...
* feat: allow forced region failover for local WAL
* chore: upgrade config.md
* chore: apply suggestions from CR
2025-04-24 08:11:45 +00:00
Yuhan Wang
e817a65d75
feat: enable submitting wal prune procedure periodically ( #5867 )
...
* feat: enable submitting wal prune procedure periodically
* chore: fix and add options
* test: add unit test
* test: fix unit test
* test: enable active_wal_pruning in test
* test: update default config
* chore: update config name
* refactor: use semaphore to control the number of prune process
* refactor: use split client for wal prune manager and topic creator
* chore: add configs
* chore: apply review comments
* fix: use tracker properly
* fix: use guard to track semaphore
* test: update unit tests
* chore: update config name
* chore: use prunable_entry_id
* refactor: semaphore to only limit the process of submitting
* chore: remove legacy sort
* chore: better configs
* fix: update config.md
* chore: respect fmt
* test: update unit tests
* chore: use interval_at
* fix: fix unit test
* test: fix unit test
* test: fix unit test
* chore: apply review comments
* docs: update config docs
2025-04-18 16:02:33 +00:00
Ruihang Xia
747b71bf74
feat: add query engine options ( #5895 )
...
* feat: add query engine options
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update example
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2025-04-14 13:12:37 +00:00
Weny Xu
ac0f9ab575
refactor: remove backoff config ( #5808 )
...
* refactor: remove backoff config
* chore: update config.md
* fix: correct backoff config
* chore: change deadline to 120s
2025-04-07 07:22:22 +00:00
fys
33c9fb737c
refactor: remove mode option in configuration files ( #5809 )
...
* refactor: remove mode option in configuration files
* chore: remove mode in configuration file
* remvoe mode field in FlownodeOptions
* add comment for test
* update config.md
* remove mode field in standalone options
* fix: ci
2025-04-01 07:14:10 +00:00
Weny Xu
d3a60d8821
feat: add limit for the number of running procedures ( #5793 )
...
* refactor: remove unused `messages`
* feat: introduce running procedure num limit
* feat: update config
* chore: apply suggestions from CR
* feat: impl `status_code` for `log-store` crate
2025-03-31 06:14:21 +00:00
Ning Sun
9e63018198
feat: disable http timeout ( #5721 )
...
* feat: update to disable http timeout by default
* feat: make http timeout default to 0
* test: correct test case
* chore: generate new config doc
* test: correct tests
2025-03-18 01:18:56 +00:00
Ning Sun
dc76571166
feat: move default data path from /tmp to current directory ( #5719 )
2025-03-16 09:57:46 +00:00
Yingwen
25645a3303
feat: expose virtual_host_style config for s3 storage ( #5696 )
...
* feat: expose enable_virtual_host_style for s3 storage
* docs: update examples
* test: fix config test
2025-03-12 13:46:56 +00:00
Lei, HUANG
765d1277ee
fix(metasrv): clean expired nodes in memory ( #5592 )
...
* fix/frontend-node-state: Refactor NodeInfoKey and Context Handling in Meta Server
• Removed unused cluster_id from NodeInfoKey struct.
• Updated HeartbeatHandlerGroup to return Context alongside HeartbeatResponse.
• Added current_node_info to Context for tracking node information.
• Implemented on_node_disconnect in Context to handle node disconnection events, specifically for Frontend roles.
• Adjusted register_pusher function to return PusherId directly.
• Updated tests to accommodate changes in Context structure.
* fix/frontend-node-state: Refactor Heartbeat Handler Context Management
Refactored the HeartbeatHandlerGroup::handle method to use a mutable reference for Context instead of passing it by value. This change simplifies the
context management by eliminating the need to return the context with the response. Updated the Metasrv implementation to align with this new context
handling approach, improving code clarity and reducing unnecessary context cloning.
* revert: clean cluster info on disconnect
* fix/frontend-node-state: Add Frontend Expiry Listener and Update NodeInfoKey Conversion
• Introduced FrontendExpiryListener to manage the expiration of frontend nodes, including its integration with leadership change notifications.
• Modified NodeInfoKey conversion to use references, enhancing efficiency and consistency across the codebase.
• Updated collect_cluster_info_handler and metasrv to incorporate the new listener and conversion changes.
• Added frontend_expiry module to the project structure for better organization and maintainability.
* chore: add config for node expiry
* add some doc
* fix: clippy
* fix/frontend-node-state:
### Refactor Node Expiry Handling
- **Configuration Update**: Removed `node_expiry_tick` from `metasrv.example.toml` and `MetasrvOptions` in `metasrv.rs`.
- **Module Renaming**: Renamed `frontend_expiry.rs` to `node_expiry_listener.rs` and updated references in `lib.rs`.
- **Code Refactoring**: Replaced `FrontendExpiryListener` with `NodeExpiryListener` in `node_expiry_listener.rs` and `metasrv.rs`, removing the tick interval and adjusting logic to use a fixed 60-second interval for node expiry checks.
* fix/frontend-node-state:
Improve logging in `node_expiry_listener.rs`
- Enhanced warning message to include peer information when an unrecognized node info key is encountered in `node_expiry_listener.rs`.
* docs: update config docs
* fix/frontend-node-state:
**Refactor Context Handling in Heartbeat Services**
- Updated `HeartbeatHandlerGroup` in `handler.rs` to pass `Context` by value instead of by mutable reference, allowing for more flexible context
management.
- Modified `Metasrv` implementation in `heartbeat.rs` to clone `Context` when passing to `handle` method, ensuring thread safety and consistency in
asynchronous operations.
2025-02-27 06:16:36 +00:00
Zhenchi
421e38c481
feat: allow purging a given puffin file in staging area ( #5558 )
...
* feat: purge a given puffin file in staging area
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* polish log
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* ttl set to 2d
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: expose staging_ttl to index config
* fix test
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* use `invalidate_entries_if` instead of maintaining map
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* run_pending_tasks after purging
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: evenyag <realevenyag@gmail.com >
2025-02-19 08:58:30 +00:00
zyy17
954310f917
feat: implement Jaeger query APIs ( #5452 )
...
* feat: implement jaeger query api
* test: add some unit tests
* test: add integration tests for jaeger query APIs
* refactor: parse tags from url parameters
* refactor: support to query traces by tags
* refactor: add limit parameter
* refactor: add jaeger query api metrics
* chore: add some comment docs and default limit value
* test: add more unit tests
* docs: add jaeger options in config docs
* refactor: code review
* wip
* refactor: use datafusion's dataframe APIs to query traces
* refactor: code review
* chore: format test cases
* refactor: add check_schema()
* chore: fix clippy errors and rename function name
* refactor: throw error when covert start_time and duration error
* chore: modify incorrect request type name
* chore: remove unecessary serde rename
* refactor: add some important comments
* refactor: add SPAN_KIND_PREFIX
* refactor: code review
2025-02-12 23:36:38 +00:00
