rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-25 01:10:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,395 Commits 131 Branches 253 Tags
c049ce6ab1ae26b61bb7ef0a029f119a6d2d63d6
Commit Graph

383 Commits

Author SHA1 Message Date
Lei, HUANG
a617e0dbef feat: use cache kv manager for SchemaMetadataManager (#5053) 
* feat: add cache for schema options

* fix/use-cache-kv-manager: Add cache invalidation handling to Datanode's heartbeat task

 • Implement InvalidateSchemaCacheHandler in heartbeat.rs to handle cache invalidation instructions.
 • Update HeartbeatTask constructor to accept cached_kv_backend and pass it to InvalidateSchemaCacheHandler.
 • Modify DatanodeBuilder to clone cached_kv_backend when creating schema_metadata_manager.
 • Refactor MetasrvCacheInvalidator in cache_invalidator.rs to reuse MailboxMessage for broadcasting to different channels.

* fix: only remove schema related cache entries

* chore: add more tests

* fix/use-cache-kv-manager: Moved InvalidateSchemaCacheHandler to a separate module

 • Extracted InvalidateSchemaCacheHandler and associated tests into a new file cache_invalidator.rs
 • Removed async_trait and CacheInvalidator related code from heartbeat.rs
 • Added cache_invalidator module declaration in handler.rs

* fix: unit tests

* fix/use-cache-kv-manager:
 Standardize TODO comment format in CachedKvBackend txn method

* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs

* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs

* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs

---------

Co-authored-by: jeremyhi <jiachun_feng@proton.me>
 2024-11-26 12:24:47 +00:00
Weny Xu
027284ed1b chore(cli): set default timeout for cli commands (#5021) 
* chore(cli): set default timeout for cli commands

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: update comments

* fix: treats `None` as `0s` to disable server-side default timeout

* chore: update comments
 2024-11-20 07:36:17 +00:00
discord9
3633f25d0c feat: also shutdown gracefully on sigterm on unix (#5023) 
* feat: support SIGTERM on unix

* chore: log

* fix: Result type
 2024-11-19 15:20:33 +00:00
shuiyisong
c199604ece feat: Loki remote write (#4941) 
* chore: add debug loki remote write url

* chore: add decode snappy

* chore: format output

* feature: impl loki remote write

* fix: special labels deserialize

* chore: move result to folder

* chore: finish todo in loki write

* test: loki write

* chore: fix cr issue

* chore: fix cr issue

* chore: fix cr issue

* chore: update pre-commit config

* chore: fix cr issue

Co-authored-by: dennis zhuang <killme2008@gmail.com>

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-11-18 08:39:17 +00:00
Weny Xu
7c135c0ef9 feat: introduce DynamicTimeoutLayer (#5006) 
* feat: introduce `DynamicTimeoutLayer`

* test: add unit test

* chore: apply suggestions from CR

* feat: add timeout option for cli
 2024-11-18 07:10:40 +00:00
Lin Yihai
0b6d78a527 refactor: consolidate DatanodeClientOptions (#4966) 
refactor!: consolidate `DatanodeClientOptions`
 2024-11-12 09:57:41 +00:00
dennis zhuang
dcc08f6b3e feat: adds the number of rows and index files size to region_statistics table (#4909) 
* feat: adds index size to region statistics

* feat: adds the number of rows for region statistics

* test: adds sqlness test for region_statistics

* fix: test
 2024-10-30 11:12:58 +00:00
LFC
fd8eba36a8 refactor: make use of the "pre_execute" in sql execution interceptor (#4875) 
* feat: dynamic definition of plugin options

* rebase

* revert

* fix ci
 2024-10-30 09:16:46 +00:00
Lei, HUANG
e328c7067c chore: udapte Rust toolchain to 2024-10-19 (#4857) 
* update rust toolchain

* change toolchain to 2024-10-17

* fix: clippy

* fix: ut

* bump shadow-rs

* fix: use nightly-2024-10-19

* fix: clippy

* chore/udapte-toolchain-2024-10-17: Update DEV_BUILDER_IMAGE_TAG to 2024-10-19-a5c00e85-20241024184445 in Makefile
 2024-10-25 00:23:32 +00:00
Weny Xu
8b307e4548 feat: introduce the PluginOptions (#4835) 
* feat: introduce the `PluginOptions`

* chore: apply suggestions from CR
 2024-10-24 12:02:10 +00:00
Weny Xu
5d42e63ab0 fix!: replace timeout_millis and connect_timeout_millis with Duration in DatanodeClientOptions (#4867) 
* fix: correct options struct

* fix: fix unit test
 2024-10-23 08:20:34 +00:00
Weny Xu
3197b8b535 feat: introduce default customizers (#4831) 
* feat: introduce `DefaultHeartbeatHandlerGroupBuilderCustomizer` and `DefaultLeadershipChangeNotifierCustomizer`

* chore: code styling
 2024-10-15 09:48:13 +00:00
Weny Xu
b5233e500b feat: defer HeartbeatHandlerGroup construction and enhance LeadershipChangeNotifier (#4826) 
* feat: enhance `HeartbeatHandlerGroup`

* chore: apply suggestions from CR

* chore: minor refactoring

* chore: code styling

* chore: apply suggestions from CR
 2024-10-15 03:35:31 +00:00
Yingwen
3b2ce31a19 feat: enable prof features by default (#4815) 
* feat: enable prof by default

* docs: don't need to build with features

* feat: add common-pprof as optional dep for pprof feature

* build: remove optional

* feat: use dump_text
 2024-10-14 03:32:47 +00:00
jeremyhi
4b34f610aa feat: information extension (#4811) 
* feat: information extension

* Update manager.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: by comment

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-10-11 03:13:49 +00:00
Weny Xu
6e776d5f98 feat: support to reject write after flushing (#4759) 
* refactor: use `RegionRoleState` instead of `RegionState`

* feat: introducing `RegionLeaderState::Downgrading`

* refactor: introduce `set_region_role_state_gracefully`

* refactor: use `set_region_role` instead of `set_writable`

* feat: support to reject write after flushing

* fix: fix unit tests

* test: add unit test for `should_reject_write`

* chore: add comments

* chore: refine comments

* fix: fix unit test

* test: enable fuzz tests for Local WAL

* chore: add logs

* chore: rename `RegionStatus` to `RegionState`

* feat: introduce `DowngradingLeader`

* chore: rename

* refactor: refactor `set_role_state` tests

* test: ensure downgrading region will reject write

* chore: enhance logs

* chore: refine name

* chore: refine comment

* test: add tests for `set_role_role_state`

* fix: fix unit tests

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-09-30 08:28:51 +00:00
zyy17
e39a9e6feb feat: add StatementStatistics for slow query logging implementation (#4719) 
* feat: log slow query

* feat: log slow query for sql

* refactor: add slow query logging options

* ci: fix errors

* feat: add StatementStatistics

* chore: revert modification of servers crate

* docs: update config docs

* fix: clippy errors
 2024-09-30 03:26:50 +00:00
taobo
0c9b8eb0d2 feat: improve observability for procedure (#4675) 
* feat: improve observability for procedure

* fix: test error

* test: add sqlness test for information_schema.procedure_info

* fix: sqlness test error

* fix: cr comment

* chore: update proto version

* fix: apply cr comment

* update version

* fix: cr comment

* optimize procedure type output format

* upgrade dep version

* fix: clippy error

* fix: `procedure` borrowed error

* fix: optimize code
 2024-09-20 06:07:53 +00:00
Ruihang Xia
f5cf25b0db refactor: remove DfPlan wrapper (#4733) 
* refactor: remove DfPlan wrapper

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test assertion

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-09-19 12:29:33 +00:00
discord9
d1e0602c76 fix: opensrv Use After Free update (#4732) 
* chore: version skew

* fix: even more version skew

* feat: use `ring` instead of `aws-lc` for remove nasm assembler on windows

* feat: use `ring` for pgwire

* feat: change to use `aws-lc-sys` on windows instead

* feat: change back to use `ring`

* chore: provide CryptoProvider

* feat: use upstream repo

* feat: install ring crypto lib in main

* chore: use same fn to install in tests

* feat: make pgwire use `ring`
 2024-09-19 04:12:13 +00:00
zyy17
0847ff36ce fix: config test failed and use similar_asserts::assert_eq to replace assert_eq for long string compare (#4731) 
* fix: config test failed and use 'similar_asserts::assert_eq' to replace 'assert_eq' for long string compare

* Update Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/cmd/tests/load_config_test.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-09-18 07:41:25 +00:00
zyy17
aa03d3b11c docs: use docs comment prefix and bump toml2docs version (#4711) 2024-09-11 07:49:23 +00:00
Lei, HUANG
208afe402b feat(wal): increase recovery parallelism (#4689) 
* Refactor RaftEngineLogStore to use references for config

 - Updated `RaftEngineLogStore::try_new` to accept a reference to `RaftEngineConfig` instead of taking ownership.
 - Replaced direct usage of `config` with individual fields (`sync_write`, `sync_period`, `read_batch_size`).
 - Adjusted test cases to pass references to `RaftEngineConfig`.

* Add parallelism configuration for WAL recovery

 - Introduced `recovery_parallelism` setting in `datanode.example.toml` and `standalone.example.toml` for configuring parallelism during WAL recovery.
 - Updated `Cargo.lock` and `Cargo.toml` to include `num_cpus` dependency.
 - Modified `RaftEngineConfig` to include `recovery_parallelism` with a default value set to the number of CP

* feat/wal-recovery-parallelism:
 Add `wal.recovery_parallelism` configuration option

 - Introduced `wal.recovery_parallelism` to config.md for specifying parallelism during WAL recovery.
 - Updated `RaftEngineLogStore` to include `recovery_threads` from the new configuration.

* fix: ut
 2024-09-09 04:25:24 +00:00
Ruihang Xia
93f202694c refactor: remove unused error variants (#4666) 
* add python script

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix all negative cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* setup CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-09-03 13:19:38 +00:00
jeremyhi
8d6cd8ae16 feat: export import database (#4654) 
* feat: export database create sql

* feat: import create database

* Update src/cmd/src/cli/export.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/cmd/src/cli/import.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/cmd/src/error.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: make show create fail fast

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-09-03 04:45:25 +00:00
Weny Xu
306bd25c64 fix: expose missing options for initializing regions (#4660) 
* fix: expose `init_regions_in_background` and `init_regions_parallelism` opts

* fix: ci
 2024-09-02 03:11:18 +00:00
jeremyhi
f641c562c2 feat: show create database (#4642) 
* feat: show create database

* feat: add sqlness test

* chore: reorder mod and use

* feat: show create schema

* Update src/frontend/src/instance.rs
 2024-08-30 03:58:11 +00:00
jeremyhi
5e4bac2633 feat: import cli tool (#4639) 
* feat: import create tables

* feat: import databasse

* fix: export view schema
 2024-08-29 09:32:21 +00:00
Weny Xu
79f40a762b fix: set selector_result_cache_size in unit test again (#4641) 2024-08-29 07:14:40 +00:00
dennis zhuang
841e66c810 fix: config api and export metrics default database (#4633) 2024-08-28 14:28:49 +00:00
Weny Xu
744946957e fix: set selector_result_cache_size in unit test (#4631) 2024-08-28 07:24:17 +00:00
zyy17
eacc7bc471 refactor: add app in greptime_app_version metric (#4626) 
refactor: add app in greptime_app_version metric
 2024-08-27 11:19:01 +00:00
Weny Xu
25cd61b310 chore: upgrade toolchain to nightly-2024-08-07 (#4549) 
* chore: upgrade toolchain to `nightly-2024-08-07`

* chore(ci): upgrade toolchain

* fix: fix unit test
 2024-08-22 11:02:18 +00:00
Yingwen
a12a905578 chore: disable ttl for write cache by default (#4595) 
* chore: remove default write cache ttl

* docs: update example config

* chore: fix ci
 2024-08-21 08:38:38 +00:00
fys
9d885fa0c2 chore: bump tikv-jemalloc* to "0.6" (#4590) 
chore: bump tikv-jemalloc* ti "0.6"
 2024-08-20 09:08:21 +00:00
Ran Joe
2a73e0937f fix(common_version): short_version with empty branch (#4572) 2024-08-19 03:14:49 +00:00
Lanqing Yang
93be81c041 feat: implement postgres kvbackend (#4421) 2024-08-14 22:49:32 +00:00
Ning Sun
9532ffb954 fix: configuration example for selector (#4532) 
* fix: configuration example for selector

* docs: update config docs

* test: update unit tests for configuration in meta
 2024-08-09 09:51:05 +00:00
discord9
8f3293d4fb fix: larger stack size in debug mode (#4521) 
* fix: larger stack size in debug mode

* chore: typo

* chore: clippy

* chore: per review

* chore: rename thread

* chore: per review

* refactor: better looking cfg

* chore: async main entry
 2024-08-09 07:01:20 +00:00
Weny Xu
a25d9f736f chore: set default otlp_endpoint (#4508) 
* chore: set default `otlp_endpoint`

* fix: fix ci
 2024-08-06 06:48:14 +00:00
discord9
e6cc4df8c8 feat: flow recreate on reboot (#4509) 
* feat: flow reboot clean

* refactor: per review

* refactor: per review

* test: sqlness flow reboot
 2024-08-05 13:57:48 +00:00
Jeremyhi
d67314789c feat: export all schemas and data at once in export tool (#4478) 
* feat: export all schemas and data at onece

* feat: introduce export all to export schemas and data at once

* feat: default value for target

* feat: refactor export target

* chore: fix unit test
 2024-08-01 09:14:44 +00:00
Jeremyhi
1ea43da9ea feat: default export catalog name (#4464) 
* feat: default export catalog name

* chore: default catalog name
 2024-07-31 03:39:39 +00:00
Ruihang Xia
7daf24c47f feat: remove dedicated runtime for grpc, mysql and pg protocols (#4436) 
* feat: remove dedicated runtime for grpc, mysql and pg protocols

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove other runtimes

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* spawn compact task into compact_runtime

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refine naming

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/servers/tests/mysql/mysql_server_test.rs

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* turnoff fuzz test matrix fail fast option

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: update rt config for ci tests

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-30 06:17:58 +00:00
zyy17
1a38f36d2d refactor!: Remove Mode from FrontendOptions (#4401) 
refactor: remove `Mode` from `FrontendOptions`

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2024-07-29 06:57:01 +00:00
discord9
021ec7b6ac feat(flow): flush_flow function (#4416) 
* refactor: df err variant

* WIP

* chore: update proto version

* chore: revert mistaken rust-toolchain

* feat(WIP): added FlowService to QueryEngine

* refactor: move flow service to operator

* refactor: flush use flow name not id

* refactor: use full path in macro

* feat: flush flow

* feat: impl flush flow

* chore: remove unused

* chore: meaninful response

* chore: remove unused

* chore: clippy

* fix: flush_flow with proper blocking

* test: sqlness tests added back for flow

* test: better predicate for flush_flow

* refactor: rwlock

* fix: flush lock

* fix: flush lock write then drop

* test: add a new flow sqlness test

* fix: sqlness testcase

* chore: style

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-26 23:04:13 +00:00
discord9
024dac8171 chore: add a compile cfg for python in cmd package (#4406) 
* chore: add a compile cfg for python

* fix: feature gate additive turn off default features in workspace&add cfg in place

* chore: remove unused in different cfg
 2024-07-24 20:03:53 +00:00
Ran Miller
e935bf7574 refactor: Remove PhysicalOptimizer and LogicalOptimizer trait (#4426) 
* refactor(query): Remove LogicalOptimizer trait

* refactor(query): Remove PhysicalOptimizer trait
 2024-07-24 13:01:44 +00:00
Jeremyhi
b90267dd80 feat: export database data (#4382) 
* feat: export database data

* feat: export data with time range

* feat: refactor the data dir

* feat: by comment
 2024-07-19 09:29:45 +00:00
zyy17
ce900e850a fix: user provider can't be configured by config file or environment variables (#4398) 2024-07-19 08:41:29 +00:00