Weny Xu
701aba9cdb
refactor: rename flow task to flow ( #3833 )
...
* refactor: rename to `MIN_USER_FLOW_ID`
* refactor: rename to `FLOW_ID_SEQ`
* refactor: rename to `flow_id_sequence`
* refactor: rename to `FlowMetadataManager`
* refactor: rename flow_task.rs to flow.rs
* refactor: rename to FlowInfoManager
* refactor: rename to FlowName
* refactor: rename to FlownodeFlow
* refactor: rename to TableFlow
* refactor: remove TASK
* refactor: rename to __flow
* refactor: rename to flow_id
* refactor: rename to flow_name
* refactor: update comments
* refactor: rename to flow_metadata_manager
* refactor: rename to flow_metadata_allocator
* refactor: rename to FlowMetadataAllocator
* refactor: rename task suffix
* refactor: rename FlowTask to FlowInfo
* refactor: rename FlowTaskScoped to FlowScoped
* refactor: rename FlowTaskId to FlowId
* chore: bump proto to b5412f7
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-04-29 14:02:52 +00:00
Weny Xu
b493ea1b38
feat: implement the CreateFlowProcedure ( #3810 )
...
* feat: implement `FlowTaskMetadataAllocator`
* feat: add `FlowTaskMetadataManagerRef` and `FlowTaskMetadataAllocatorRef`
* chore: fix clippy
* feat: add `FlowTaskNameLock`
* feat: implement the `CreateFlowTaskProcedure`
* chore: rename to `CreateFlowProcedure`
* chore: apply suggestions from CR
* feat: invoke create flow procedure
* chore: apply suggestions from CR
* refactor: rename TYPE_NAME
* feat: register the procedure
* chore: apply suggestions from CR
* feat: acquire the lock of sink table name
2024-04-29 12:34:11 +00:00
Ruihang Xia
7ef18c0915
feat: impl parser and operator for CREATE FLOW ( #3806 )
...
* feat: impl parser for CREATE TASK
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finish parser
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* wip expr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* finish expr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename output to sink
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix parser
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove debug code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* upload lock file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename symbol
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Apply suggestions from code review
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
* remove other task word
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* task name to flow name
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* one more comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Jeremyhi <jiachun_feng@proton.me >
2024-04-28 12:26:49 +00:00
Weny Xu
3112ced9c0
chore: rename all datanode_manager to node_manager ( #3813 )
2024-04-28 02:34:06 +00:00
Weny Xu
eb3d2ca759
refactor: refactor DatanodeManager to NodeManager ( #3811 )
...
* chore: bump greptime-proto to 2c14c6e
* refactor: refactor `DatanodeManager` to `NodeManager`
2024-04-26 11:31:03 +00:00
Weny Xu
1ec595134d
feat: define CreateFlowTask and DropFlowTask ( #3801 )
...
* chore: bump proto to cefc73f
* feat: add `CreateFlowTask` and `DropFlowTask`
* chore: bump to 87f2b38
* chore: bump to 783682f
2024-04-25 12:30:31 +00:00
Weny Xu
9206f60b28
feat: implement FlowTaskMetadataManager ( #3766 )
...
* feat: implement `FlowMetadataManager`
* chore: remove dead code
* refactor: change `sink_tables` to `sink_table`
* refactor: add `PartitionId`
* feat: implement FlowTaskNameManager
* refactor: update doc of keys
* fix: return partition id in `tasks`
* refactor: rename to `FlowTaskId`
* chore: add comments
* chore: add `task_id` in `TaskAlreadyExists`
* chore: add comments
* fix: fmt
* refactor: simplify the docoder
* chore: update comments
* feat: implement `FlowTaskScoped` and `CatalogScoped`
* refactor: refactor flow task keys
* refactor: remove metadata mod
* refactor: rename to `FlowTaskInfo`
* chore: add comments
* refactor: rename to `FlowTaskMetadataManager`
* chore: remove dead code
* Apply suggestions from code review
* chore: change to `pub(crate)`
* chore: apply suggestions from CR
* fix: fix fmt
* chore: fmt doc
2024-04-25 11:59:24 +00:00
Weny Xu
20a933e395
refactor: simplify the PaginationStream ( #3787 )
...
* refactor: simplify the `PaginationStream`
* refactor: refactor decode fn
* fix: fix clippy
2024-04-24 04:29:41 +00:00
Kould
b619950c70
feat: add ChangeColumnType for AlterKind ( #3757 )
...
* feat: add `ModifyColumn` for `AlterKind`
* chore: additional code comments for `AlterKind::ModifyColumns`
* fix: add nullable check on `ModifyColumn`
* style: codefmt
* style: fix the code based on review suggestions
* style: fix the code based on review suggestions
* style: rename `ModifyColumn` -> `ChangeColumnType`
* style: code fmt
* style: `change_columns_type` -> `change_column_types`
2024-04-24 04:27:23 +00:00
LFC
19a9035f4b
fix: operating region guards should be dropped when procedure is done ( #3775 )
2024-04-23 06:21:53 +00:00
LFC
96c01a3bf0
fix: the dropping_regions guards should be dropped on procedure done ( #3771 )
...
* fix: the `dropping_regions` guards should be dropped on procedure done
* fix ci
2024-04-23 02:44:12 +00:00
Ruihang Xia
bf21527f18
fix: set is_time_index properly on updating physical table's schema ( #3770 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-04-22 12:37:25 +00:00
Ruihang Xia
0c5f4801b7
build: update toolchain to nightly-2024-04-18 ( #3740 )
...
* chore: update toolchain to nightly-2024-04-17
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix ut
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update fuzz test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to nightly-2024-04-18
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add document
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update CI
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* avoid unnecessary allow clippy attrs
Signed-off-by: tison <wander4096@gmail.com >
* help the compiler find the clone is unnecessary and make clippy happy
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-19 05:42:34 +00:00
Weny Xu
510782261d
refactor: avoid unnecessary cloning ( #3734 )
...
refactor: using `TxnOpGetResponseSet`
2024-04-18 09:02:28 +00:00
Weny Xu
2a2a44883f
refactor(meta): Ensure all moving values remain unchanged between two transactions ( #3727 )
...
* feat: implement `move_values`
* refactor: using `move_values`
* refactor: refactor executor
* chore: fix clippy
* refactor: remove `CasKeyChanged` error
* refactor: refactor `move_values`
* chore: update comments
* refactor: do not compare `dest_key`
* chore: update comments
* chore: apply suggestions from CR
* chore: remove `#[inline]`
* chore: check length of keys and dest_key
2024-04-18 05:35:54 +00:00
maco
4248dfcf36
feat: support invalidate schema name key cache ( #3725 )
...
* feat: support invalidate schema name key cache
* fix: remove pub for invalidate_schema_cache
* refactor: add DropMetadataBroadcast State Op
* fix: delete files
2024-04-18 04:02:06 +00:00
tison
50ae4dc174
refactor: merge RegionHandleResult into RegionHandleResponse ( #3721 )
...
* refactor: merge RegionHandleResult into RegionHandleResponse
Signed-off-by: tison <wander4096@gmail.com >
* RegionResponse to api::region
Signed-off-by: tison <wander4096@gmail.com >
* order
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-04-17 10:03:20 +00:00
Weny Xu
d12379106e
feat(drop_table): support to rollback table metadata ( #3692 )
...
* feat: support to rollback table metadata
* refactor: store table route value instead of physical table route
* feat(drop_table): support to rollback table metadata
* test: add rollback tests for drop table
* fix: do not set region to readonly
* test: add sqlness tests
* feat: implement TombstoneManager
* test: add tests for TombstoneManager
* refactor: using TombstoneManager
* chore: remove unused code
* fix: fix typo
* refactor: using `on_restore_metadata`
* refactor: add `executor` to `DropTableProcedure`
* refactor: simplify the `TombstoneManager`
* refactor: refactor `Key`
* refactor: carry more info
* feat: add `destroy_table_metadata`
* refactor: remove redundant table_route_value
* feat: ensure the key is empty
* feat: introcude `table_metadata_keys`
* chore: carry more info
* chore: remove clone
* chore: apply suggestions from CR
* feat: delete metadata tombstone
2024-04-16 09:22:41 +00:00
Weny Xu
64941d848e
fix(alter_table): ignore request outdated error ( #3715 )
2024-04-16 08:18:38 +00:00
maco
d5a948a0a6
test: Add tests for KvBackend trait implement ( #3700 )
...
* test: add etcd
* optimize code
* test: add etcd tests
* fix: typos
* fix: taplo error and clippy
* avoid print
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-15 10:51:59 +00:00
Jeremyhi
87795248dd
feat: get metasrv clusterinfo ( #3696 )
...
* feat: add doc for MetasrvOptions
* feat: register candidate before election
* feat: get all peers metasrv
* chore: simply code
* chore: proto rev
* Update src/common/meta/src/cluster.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* Update src/meta-client/src/client.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fmt
Signed-off-by: tison <wander4096@gmail.com >
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* impl<T: AsRef<[u8]>> From<T> for LeaderValue
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-04-15 08:10:48 +00:00
Weny Xu
c1e005b148
refactor: drop table procedure ( #3688 )
...
* refactor: refactor drop table procedure
* refactor: refactor test utils
2024-04-10 12:22:10 +00:00
Yohan Wal
34f935df66
chore: create database api change in protobuf ( #3682 )
2024-04-09 12:11:38 +00:00
Weny Xu
fda1523ced
refactor: refactor alter table procedure ( #3678 )
...
* refactor: refactor alter table procedure
* chore: apply suggestions from CR
* chore: remove `alter_expr` and `alter_kind`
2024-04-09 10:35:51 +00:00
JeremyHi
1629435888
chore: unify name metasrv ( #3671 )
...
chore: unify name
2024-04-09 03:03:26 +00:00
Weny Xu
b51089fa61
fix: DeserializedValueWithBytes::from_inner misusing ( #3676 )
...
* fix: fix `DeserializedValueWithBytes::from_inner` misusing
* Update src/common/meta/src/key.rs
---------
Co-authored-by: tison <wander4096@gmail.com >
2024-04-09 01:48:35 +00:00
Weny Xu
32b9639d7c
feat(procedure): support to rollback ( #3625 )
...
* feat: add rollback method
* refactor: simplify the state control
* feat(procedure): support to rollback
* test: add tests for rollback
* feat: persist rollback procedure state
* feat: rollback procedure after restarting
* feat: add `CommitRollback`, `RollingBack` to ProcedureStateResponse
* chore: apply suggestions from CR
* feat: persist rollback error
* feat: add `is_support_rollback`
* chore: apply suggestions from CR
* chore: update greptime-proto
* chore: rename to `rollback_supported`
* chore: rename to `RollbackProcedureRecovered`
2024-04-08 11:23:23 +00:00
JeremyHi
12286f07ac
feat: cluster information ( #3631 )
...
* chore: keep the same method order in KvBackend
* feat: make meta client can get all node info of cluster
* feat: cluster info data model
* feat: frontend and datanode info
* feat: list node info
* chore: remove the method: is_started
* fix: scan key prefix
* chore: impl From for NodeInfoKey
* chore: doc for trait and struct
* chore: reuse the error
* chore: refactor two collec cluster info handlers
* chore: remove inline
* chore: refactor two collec cluster info handlers
2024-04-08 07:48:36 +00:00
Yohan Wal
c4798d1913
refactor: move create database to procedure ( #3626 )
...
* refactor: move create database to procedure
* feat: enable database creation of rpc
* chore: update the commit hash of greptime-proto
2024-04-08 07:05:55 +00:00
Weny Xu
a0f243c128
feat(procedure): enable auto split large value ( #3628 )
...
* chore: add comments
* chore: remove `pub`
* chore: rename to `merge_multiple_values`
* chore: fix typo
* feat(procedure): enable auto split large value
* chore: apply suggestions from CR
* chore: rename to `max_metadata_value_size`
* chore: remove the NoneAsEmptyString
* chore: set default max_metadata_value_size to 1500KiB
2024-04-02 12:13:59 +00:00
JeremyHi
a61fb98e4a
refactor: alter logical tables ( #3618 )
...
* refactor: on prepare
* refactor: on create regions
* refactor: update metadata
2024-04-02 06:21:34 +00:00
Weny Xu
6c316d268f
feat(procedure): auto split large value to multiple values ( #3605 )
...
* feat: implement MultipleValuesStream
* refactor: move KeySet to common-procedure
* refactor: move MultipleValuesStream to common-procedure
* refactor: refactor String to KeySet
* fix: fix dropping `collecting` unexpectedly
* fix: fix typo
* refactor: add the fast path of put
* refactor: remove `single_value_collector`
* refactor: use `extend` instead of `push`
* test: add more tests for `KvStateStore`
* test(etcd_store): add more tests for `KvStateStore`
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: refactor with async_stream
* Update src/common/procedure/src/store/util.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-04-01 12:04:29 +00:00
JeremyHi
0eb023bb23
feat: group requests by peer ( #3619 )
2024-04-01 03:10:22 +00:00
JeremyHi
93da45f678
feat: let alter table procedure can only alter physical table ( #3613 )
...
* feat: let alter table procedure can only alter physicale table
* chore: rm unnecessary todo
2024-03-29 09:50:33 +00:00
JeremyHi
f49cd0ca18
refactor: cache invalidator ( #3611 )
...
* chore: remove some alias
* refactor: cache invalidator
2024-03-29 07:33:51 +00:00
niebayes
d5ba2fcf9d
test: add more integration test for kafka wal ( #3190 )
...
* test: add integration tests for kafka wal
* chore: rebase main
* chore: unify naming convention for wal config
* chore: add register loaders switch
* chore: alter tables by adding a new column
* chore: move rand to dev-dependencies
* chore: update Cargo.lock
2024-03-28 06:55:18 +00:00
JeremyHi
92a8e863de
chore: do not reply for broadcast msg ( #3595 )
2024-03-27 11:39:23 +00:00
JeremyHi
9428cb8e7c
feat: remove support for logical tables in the create table procedure ( #3592 )
...
* feat: Remove support for logical tables in the create table procedure
* chore: remove the redandent table ids alloc
* chore: minor fix
2024-03-27 10:03:42 +00:00
Weny Xu
5addb7d75a
test: add tests for drop databases ( #3594 )
...
* refactor: minimize visibility of drop database steps
* feat: implement as_any
* refactor: move common functions to test_util
* test: add tests for drop databases
* fix: fix deteling physical table route unexpectedly
2024-03-27 09:18:37 +00:00
Weny Xu
623c930736
refactor: refactor drop table executor ( #3589 )
...
* refactor: refactor drop table executor
* chore: apply suggestions from CR
2024-03-27 06:29:54 +00:00
JeremyHi
5fa01e7a96
feat: create regions persist true ( #3590 )
...
* feat: change open-region-step's status persist as true
* feat: avoid cloning
* fix: fix unit test
2024-03-27 06:26:58 +00:00
JeremyHi
58c7858cd4
feat: update physical table schema on alter logical tables ( #3585 )
...
* feat: update physical table schema on alter
* feat: alter logical table in sql path
* feat: invalidate cache step1
* feat: invalidate cache step2
* feat: invalidate cache step3
* feat: invalidate cache step4
* fix: failed ut
* fix: standalone cache invalidator
* feat: log the count of already finished
* feat: re-invalidate cache
* chore: by comment
* chore: Update src/common/meta/src/ddl/create_logical_tables.rs
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-03-26 14:29:53 +00:00
dimbtp
dd18d8c97b
build(deps): remove some unused dependencies ( #3582 )
...
* build(deps): remove some unused dependencies
* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
JeremyHi
c2dd1136fe
feat: batch alter logical tables ( #3569 )
...
* feat: add unit test for alter logical tables
* Update src/common/meta/src/ddl/alter_table.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* feat: add some comments
* chore: add debug_assert_eq
* chore: fix some nits
* chore: remove the method batch_get_table_routes
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-03-26 07:07:23 +00:00
Weny Xu
bf14d33962
feat: implement the drop database procedure ( #3541 )
...
* refactor: remove Sync trait of Procedure
* refactor: remove unnecessary async
* feat: implement the drop database procedure
* refactor: refactor DdlManager register_loaders
* feat: register the DropDatabaseProcedureLoader
* chore: fmt toml
* feat: support to submit DropDatabaseTask
* feat: support drop database stmt
* fix: empty the tables stream
* fix: ensure the factory always exists
* test: update sqlness results
* chore: correct comments
* test: update sqlness results
* test: update sqlness results
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-03-25 06:12:47 +00:00
Ruihang Xia
992c7ec71b
feat: update physical table's schema on creating logical table ( #3570 )
...
* feat: update physical table's schema on creating logical table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove debug code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* tweak ut const
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* invalid physical table cache
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-03-25 03:19:30 +00:00
Ruihang Xia
2b2fd80bf4
feat: return new added columns in region server's extension response ( #3533 )
...
* feat: adapt the new proto response
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update interfaces
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* write columns to extension
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* use physical column's schema
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* sort logical columns by name
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* return physical table's column
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Update src/common/meta/src/datanode_manager.rs
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
* implement sort column logic
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* proxy create table procedure to create logical table
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add unit test for sort_columns
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-03-23 09:31:16 +00:00
Weny Xu
856a4e1e4f
refactor: refactor CacheInvalidator ( #3550 )
...
* refactor: refactor InvalidateCache Instruction
* refactor: refactor CacheInvalidator
2024-03-20 10:18:28 +00:00
Weny Xu
fa0f3555d4
refactor: introduce the DropTableExecutor ( #3534 )
...
* refactor: introduce the DropTableExecutor
* fix: register the dropping regions
* test: add tests for on_prepare
* chroe: add TODO comment
2024-03-19 08:29:12 +00:00
Weny Xu
343525dab8
refactor: remove removed-prefixed keys ( #3535 )
2024-03-18 11:07:30 +00:00
