rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-09 23:02:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,645 Commits 104 Branches 227 Tags
f16ce3ca2749f2135cd225d2cefc49e92209daec
Commit Graph

300 Commits

Author SHA1 Message Date
Weny Xu
e15294db41 feat: introduce TableRouteCache to PartitionRuleManager (#3922) 
* chore: add `CompositeTableRouteCacheRef` to `PartitionRuleManager`

* chore: update comments

* fix: add metrics for `get`

* chore: apply suggestions from CR

* chore: correct cache name

* feat: implement `LayeredCacheRegistry`

* fix: invalidate logical tables by physical table id

* refactor: replace `CacheRegistry` with `LayeredCacheRegistry`

* chore: update comments

* chore: apply suggestions from CR

* chore: fix fmt

* refactor: use `TableRouteCache` instead

* chore: apply suggestions from CR

* chore: fix clippy
 2024-05-13 13:26:43 +00:00
Jeremyhi
5d8084a32f chore: store-addr to store-addrs (#3925) 2024-05-13 08:30:25 +00:00
zyy17
60eb5de3f1 refactor: add tracing options in xOptions (#3919) 
* refactor: add tracing options in {DatanodeOptions, FrontendOptions, MetasrvOptions, StandaloneOptions}

* ci: fix integration error

* refactor: minor modification for initialization of tracing options
 2024-05-13 05:16:50 +00:00
Jeremyhi
6ab3aeb142 refactor: rename metasrv_addr to metasrv_addrs (#3921) 2024-05-12 03:44:21 +00:00
Weny Xu
89da42dbc1 refactor: refactor frontend cache (#3912) 
* feat: implement the `TableCache`

* refactor: use `TableCache`

* refactor: replace `TableFlowManager` with `TableFlownodeSetCache`

* refactor: introduce cache crate

* chore: add comments

* chore: update comments

* chore: apply suggestions from CR

* chore: rename `cache_invalidator` to `local_cache_invalidator`

* chore(fuzz): set `acquire_timeout` to 30s
 2024-05-11 09:58:18 +00:00
zyy17
b1ef327bac refactor: remove MixOptions and use StandaloneOptions only (#3893) 
* refactor: remove MixOptions and use StandaloneOptions only

* refactor: refactor code by code review comments

1. Use '&self' in frontend_options() and datanode_options();

2. Remove unused 'clone()';

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* ci: fix integration error

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2024-05-10 16:08:34 +00:00
Ruihang Xia
115c74791d build(deps): bump snafu to 0.8 (#3911) 
* change Cargo.toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias in script engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-10 13:36:25 +00:00
zyy17
a68072cb21 refactor: add the main cli entry point struct Command{} and simplify main() (#3850) 
refactor: add the main cli entrypoint struct `Command{}` to simplify `main()`
 2024-05-07 09:08:35 +00:00
Kelvin Wu
154f561da1 refactor: passing QueryContext to RegionServer (#3829) 
* refactor: passing QueryContext to RegionServer

* refactor: change the return type of build() in QueryContextBuilder

* fix: update greptime-proto reference

* chore: apply suggestion

* chore: revert the last commit

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-05-07 15:10:59 +08:00
Ruihang Xia
530353785c refactor: remove re-export from logging (#3865) 
* refactor: remove re-export from logging

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge problem

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* run formatter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-06 13:26:01 +00:00
shuiyisong
6e9e8fad26 refactor!: remove opentsdb tcp server (#3828) 
* refactor: remove opentsdb tcp server

* refactor: remove config and add test

* refactor: update docs and remove unused code
 2024-05-06 06:42:05 +00:00
dennis zhuang
65d47bab56 feat: adds information_schema cluster_info table (#3832) 
* feat: adds server running mode to KvBackendCatalogManager

* feat: adds MetaClient to KvBackendCatalogManager

* feat: impl information_schema.cluster_info table

* fix: forgot files

* test: update information_schema result

* feat: adds start_time and uptime to cluster_info

* chore: tweak cargo and comment

* feat: rename greptime_region_peers to region_peers

* fix: cluster_info result

* chore: simplify sqlness commands

* chore: set peer_id to -1 for frontends

* fix: move cluster_info to greptime catalog

* chore: use official proto

* feat: adds active_time

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: STANDALONE for runtime_metrics

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
Co-authored-by: tison <wander4096@gmail.com>
 2024-05-02 02:49:46 +00:00
dimbtp
3b89b9ddd8 refactor: move database client to test (#3820) 
* remove `Database` in `cli export`

* move `Database` to tests-integration

* fix clippy

* move `DatabaseClient` along with `Database`

* `cli export` now use http api

* add TODO for `tests-integration` dependencies

* cleanup code

* 'cli export' test use http api

* remove unsed dependencies in cmd crate

* apply review comments and clean code

* remove unused methods

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-05-01 05:55:13 +00:00
Weny Xu
81f3007f6f refactor: remove TableMetaKey trait (#3837) 
* refactor: use MetaKey trait instead of TableMetaKey

* test: add tests

* test: add more tests

* refactor: use `&str` instead of `String`
 2024-04-30 06:31:45 +00:00
Weny Xu
701aba9cdb refactor: rename flow task to flow (#3833) 
* refactor: rename to `MIN_USER_FLOW_ID`

* refactor: rename to `FLOW_ID_SEQ`

* refactor: rename to `flow_id_sequence`

* refactor: rename to `FlowMetadataManager`

* refactor: rename flow_task.rs to flow.rs

* refactor: rename to FlowInfoManager

* refactor: rename to FlowName

* refactor: rename to FlownodeFlow

* refactor: rename to TableFlow

* refactor: remove TASK

* refactor: rename to __flow

* refactor: rename to flow_id

* refactor: rename to flow_name

* refactor: update comments

* refactor: rename to flow_metadata_manager

* refactor: rename to flow_metadata_allocator

* refactor: rename to FlowMetadataAllocator

* refactor: rename task suffix

* refactor: rename FlowTask to FlowInfo

* refactor: rename FlowTaskScoped to FlowScoped

* refactor: rename FlowTaskId to FlowId

* chore: bump proto to b5412f7

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-04-29 14:02:52 +00:00
Weny Xu
b493ea1b38 feat: implement the CreateFlowProcedure (#3810) 
* feat: implement `FlowTaskMetadataAllocator`

* feat: add `FlowTaskMetadataManagerRef` and `FlowTaskMetadataAllocatorRef`

* chore: fix clippy

* feat: add `FlowTaskNameLock`

* feat: implement the `CreateFlowTaskProcedure`

* chore: rename to `CreateFlowProcedure`

* chore: apply suggestions from CR

* feat: invoke create flow procedure

* chore: apply suggestions from CR

* refactor: rename TYPE_NAME

* feat: register the procedure

* chore: apply suggestions from CR

* feat: acquire the lock of sink table name
 2024-04-29 12:34:11 +00:00
tison
c387687262 refactor: use secrecy SerectString to hold secrets option (#3804) 
* build: centralize secrecy dependency

Signed-off-by: tison <wander4096@gmail.com>

* add secrecy to sql crate

Signed-off-by: tison <wander4096@gmail.com>

* try impl

Signed-off-by: tison <wander4096@gmail.com>

* update test

Signed-off-by: tison <wander4096@gmail.com>

* make linters happy

Signed-off-by: tison <wander4096@gmail.com>

* bundle secrecy

Signed-off-by: tison <wander4096@gmail.com>

* bundle secrecy

Signed-off-by: tison <wander4096@gmail.com>

* replace secrecy

Signed-off-by: tison <wander4096@gmail.com>

* tidy clones

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* updated

Signed-off-by: tison <wander4096@gmail.com>

* Apply suggestions from code review

Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>

* use BTreeMap

Signed-off-by: tison <wander4096@gmail.com>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
 2024-04-29 02:18:18 +00:00
Weny Xu
3112ced9c0 chore: rename all datanode_manager to node_manager (#3813) 2024-04-28 02:34:06 +00:00
Weny Xu
eb3d2ca759 refactor: refactor DatanodeManager to NodeManager (#3811) 
* chore: bump greptime-proto to 2c14c6e

* refactor: refactor `DatanodeManager` to `NodeManager`
 2024-04-26 11:31:03 +00:00
tison
bba3108e0d refactor!: unify sql options into OptionMap (#3792) 
* unify sql options into OptionMap

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* Update src/sql/src/util.rs

* drop legacy regions option

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-04-25 04:06:05 +00:00
Weny Xu
20a933e395 refactor: simplify the PaginationStream (#3787) 
* refactor: simplify the `PaginationStream`

* refactor: refactor decode fn

* fix: fix clippy
 2024-04-24 04:29:41 +00:00
LFC
86a989517e refactor: move the version string to common (#3783) 2024-04-23 14:21:34 +00:00
Ruihang Xia
0c5f4801b7 build: update toolchain to nightly-2024-04-18 (#3740) 
* chore: update toolchain to nightly-2024-04-17

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update fuzz test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update to nightly-2024-04-18

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* avoid unnecessary allow clippy attrs

Signed-off-by: tison <wander4096@gmail.com>

* help the compiler find the clone is unnecessary and make clippy happy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-04-19 05:42:34 +00:00
Weny Xu
d12379106e feat(drop_table): support to rollback table metadata (#3692) 
* feat: support to rollback table metadata

* refactor: store table route value instead of physical table route

* feat(drop_table): support to rollback table metadata

* test: add rollback tests for drop table

* fix: do not set region to readonly

* test: add sqlness tests

* feat: implement TombstoneManager

* test: add tests for TombstoneManager

* refactor: using TombstoneManager

* chore: remove unused code

* fix: fix typo

* refactor: using `on_restore_metadata`

* refactor: add `executor` to `DropTableProcedure`

* refactor: simplify the `TombstoneManager`

* refactor: refactor `Key`

* refactor: carry more info

* feat: add `destroy_table_metadata`

* refactor: remove redundant table_route_value

* feat: ensure the key is empty

* feat: introcude `table_metadata_keys`

* chore: carry more info

* chore: remove clone

* chore: apply suggestions from CR

* feat: delete metadata tombstone
 2024-04-16 09:22:41 +00:00
dimbtp
02f806fba9 fix: cli export "create table" with quoted names (#3684) 
* fix: cli export `create table` with quoted names

* add test

* apply review comments

* fix to pass check

* remove eprintln for clippy check

* use prebuilt binary to avoid compile

* ci run coverage after build

* drop dirty hack test

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-04-11 06:56:14 +00:00
Weny Xu
fda1523ced refactor: refactor alter table procedure (#3678) 
* refactor: refactor alter table procedure

* chore: apply suggestions from CR

* chore: remove `alter_expr` and `alter_kind`
 2024-04-09 10:35:51 +00:00
JeremyHi
1629435888 chore: unify name metasrv (#3671) 
chore: unify name
 2024-04-09 03:03:26 +00:00
Weny Xu
a0f243c128 feat(procedure): enable auto split large value (#3628) 
* chore: add comments

* chore: remove `pub`

* chore: rename to `merge_multiple_values`

* chore: fix typo

* feat(procedure): enable auto split large value

* chore: apply suggestions from CR

* chore: rename to `max_metadata_value_size`

* chore: remove the NoneAsEmptyString

* chore: set default max_metadata_value_size to 1500KiB
 2024-04-02 12:13:59 +00:00
JeremyHi
f49cd0ca18 refactor: cache invalidator (#3611) 
* chore: remove some alias

* refactor: cache invalidator
 2024-03-29 07:33:51 +00:00
niebayes
d5ba2fcf9d test: add more integration test for kafka wal (#3190) 
* test: add integration tests for kafka wal

* chore: rebase main

* chore: unify naming convention for wal config

* chore: add register loaders switch

* chore: alter tables by adding a new column

* chore: move rand to dev-dependencies

* chore: update Cargo.lock
 2024-03-28 06:55:18 +00:00
JeremyHi
9428cb8e7c feat: remove support for logical tables in the create table procedure (#3592) 
* feat: Remove support for logical tables in the create table procedure

* chore: remove the redandent table ids alloc

* chore: minor fix
 2024-03-27 10:03:42 +00:00
Weny Xu
623c930736 refactor: refactor drop table executor (#3589) 
* refactor: refactor drop table executor

* chore: apply suggestions from CR
 2024-03-27 06:29:54 +00:00
JeremyHi
58c7858cd4 feat: update physical table schema on alter logical tables (#3585) 
* feat: update physical table schema on alter

* feat: alter logical table in sql path

* feat: invalidate cache step1

* feat: invalidate cache step2

* feat: invalidate cache step3

* feat: invalidate cache step4

* fix: failed ut

* fix: standalone cache invalidator

* feat: log the count of already finished

* feat: re-invalidate cache

* chore: by comment

* chore: Update src/common/meta/src/ddl/create_logical_tables.rs

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-03-26 14:29:53 +00:00
dimbtp
dd18d8c97b build(deps): remove some unused dependencies (#3582) 
* build(deps): remove some unused dependencies

* add `arc-swap` dependency back
 2024-03-26 12:48:28 +00:00
Ruihang Xia
8f9676aad2 fix: incorrect version info in (#3586) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-03-26 09:31:01 +00:00
shuiyisong
0bb949787c refactor: introduce new Output with OutputMeta (#3466) 
* refactor: introduce new output struct

* chore: add helper function

* chore: update comment

* chore: update commit

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: rename according to cr

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-03-11 02:24:09 +00:00
JeremyHi
352bd7b6fd feat: max-txn-ops option (#3458) 
* feat: max-txn-ops limit

* chore: by comment
 2024-03-08 02:34:40 +00:00
LFC
2035e7bf4c refactor: set the actual bound port in server handler (#3353) 
* refactor: set the actual bound port so we can use port 0 in testing

* Update src/servers/src/server.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fmt

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-02-23 02:49:11 +00:00
dennis zhuang
564fe3beca feat: impl migrate_region and procedure_state SQL function (#3325) 
* fix: logical region can't find region routes

* feat: fetch partitions info in batch

* refactor: rename batch functions

* refactor: rename DdlTaskExecutor to ProcedureExecutor

* feat: impl migrate_region and query_procedure_state for ProcedureExecutor

* feat: adds SQL function procedure_state and finish migrate_region impl

* fix: constant vector

* feat: unit tests for migrate_region and procedure_state

* test: test region migration by SQL

* fix: compile error after rebeasing

* fix: clippy warnings

* feat: ensure procedure_state and migrate_region can be only called under greptime catalog

* fix: license header
 2024-02-22 02:37:11 +00:00
SteveLauC
e9a2b0a9ee chore: use workspace-wide lints (#3352) 
* chore: use workspace-wide lints

* respond to review
 2024-02-22 01:01:10 +00:00
Weny Xu
41656c8635 refactor: allocate table id in the procedure (#3271) 
* refactor: replace TableMetadataManager with TableNameManager

* refactor: allocate table id in the procedure

* refactor: refactor client logical of handling retries

* feat(test_util): add TestCreateTableExprBuilder

* feat(test_util): add MockDatanodeManager

* feat(test_util): add new_ddl_context

* feat(test_util): add build_raw_table_info_from_expr

* feat(test_util): add MockDatanodeManager::new

* feat(procedure): add downcast_output_ref to Status

* test(create_table): add tests for CreateTableProcedure on_prepare

* refactor(ddl): rename handle_operate_region_error to add_peer_context_if_need

* test(create_table): add tests for CreateTableProcedure on_datanode_create_regions

* test(create_table): add tests for CreateTableProcedure on_create_metadata

* refactor(meta): use CreateTableExprBuilder

* feat(create_table): ensure number of partitions is greater than 0

* refactor: rename to add_peer_context_if_needed

* feat: add context for panic

* refactor: simplify the should_retry

* refactor: use Option<&T> instead of &Option<T>

* refactor: move downcast_output_ref under cfg(test)

* chore: fmt toml
 2024-02-21 04:38:46 +00:00
shuiyisong
bf5e1905cd refactor: bring metrics to http output (#3247) 
* refactor: bring metrics to http output

* chore: remove unwrap

* chore: make walk plan accumulate

* chore: change field name and comment

* chore: add metrics to http resp header

* chore: move PrometheusJsonResponse to a separate file and impl IntoResponse

* chore: put metrics in prometheus resp header too
 2024-02-20 03:25:18 +00:00
dennis zhuang
8b73067815 feat: impl partitions and region_peers information schema (#3278) 
* feat: impl partitions table

* fix: typo

* feat: impl region_peers information schema

* chore: rename region_peers to greptime_region_peers

* chore: rename statuses to upper case

* fix: comments

* chore: update partition result

* chore: remove redundant checking

* refactor: replace 42 with constant

* feat: fetch region routes in batch
 2024-02-19 06:47:14 +00:00
tison
4e04a4e48f build: support build without git (#3309) 
* build: support build without git

Signed-off-by: tison <wander4096@gmail.com>

* chore

Signed-off-by: tison <wander4096@gmail.com>

* address comment

Signed-off-by: tison <wander4096@gmail.com>

* fix syntax

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-02-18 10:30:01 +00:00
fys
74bfb09195 feat: support cache for batch_get in CachedMetaKvBackend (#3277) 
* feat: support cache for batch_get in CachedMetaKvBackend.

* add doc of CachedMetaKvBackend

* fix: cr

* fix: correct some words

* fix cr
 2024-02-06 03:15:03 +00:00
shuiyisong
4cbdf64d52 chore: start plugins during standalone startup & comply with current catalog while changing database (#3282) 
* chore: start plugins in standalone

* chore: respect current catalog in use statement for mysql

* chore: reduce unnecessory convert to string

* chore: reduce duplicate code
 2024-02-06 02:41:37 +00:00
shuiyisong
6ab3a88042 fix: use fe_opts after setup_frontend_plugins in standalone (#3275) 
* chore: modify standalone startup opts

* chore: move frontend and datanode options
 2024-02-02 10:36:08 +00:00
tison
60e760b168 fix: cli export database default value (#3259) 
Signed-off-by: tison <wander4096@gmail.com>
 2024-01-30 09:56:05 +00:00
LFC
f81e37f508 refactor: make http server built flexibly (#3225) 
* refactor: make http server built flexibly

* Apply suggestions from code review

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* fix: resolve PR comments

* Fix CI.

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2024-01-24 03:45:08 +00:00
dennis zhuang
6a12c27e78 feat: make query be aware of timezone setting (#3175) 
* feat: let TypeConversionRule aware query context timezone setting

* chore: don't optimize explain command

* feat: parse string into timestamp with timezone

* fix: compile error

* chore: check the scalar value type in predicate

* chore: remove mut for engine context

* chore: return none if the scalar value is utf8 in time range predicate

* fix: some fixme

* feat: let Date and DateTime parsing from string value be aware of timezone

* chore: tweak

* test: add datetime from_str test with timezone

* feat: construct function context from query context

* test: add timezone test for to_unixtime and date_format function

* fix: typo

* chore: apply suggestion

* test: adds string with timezone

* chore: apply CR suggestion

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* chore: apply suggestion

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-01-22 14:14:03 +00:00