Commit Graph

273 Commits

Author SHA1 Message Date
Weny Xu
a0f243c128 feat(procedure): enable auto split large value (#3628)
* chore: add comments

* chore: remove `pub`

* chore: rename to `merge_multiple_values`

* chore: fix typo

* feat(procedure): enable auto split large value

* chore: apply suggestions from CR

* chore: rename to `max_metadata_value_size`

* chore: remove the NoneAsEmptyString

* chore: set default max_metadata_value_size to 1500KiB
2024-04-02 12:13:59 +00:00
JeremyHi
f49cd0ca18 refactor: cache invalidator (#3611)
* chore: remove some alias

* refactor: cache invalidator
2024-03-29 07:33:51 +00:00
niebayes
d5ba2fcf9d test: add more integration test for kafka wal (#3190)
* test: add integration tests for kafka wal

* chore: rebase main

* chore: unify naming convention for wal config

* chore: add register loaders switch

* chore: alter tables by adding a new column

* chore: move rand to dev-dependencies

* chore: update Cargo.lock
2024-03-28 06:55:18 +00:00
JeremyHi
9428cb8e7c feat: remove support for logical tables in the create table procedure (#3592)
* feat: Remove support for logical tables in the create table procedure

* chore: remove the redandent table ids alloc

* chore: minor fix
2024-03-27 10:03:42 +00:00
Weny Xu
623c930736 refactor: refactor drop table executor (#3589)
* refactor: refactor drop table executor

* chore: apply suggestions from CR
2024-03-27 06:29:54 +00:00
JeremyHi
58c7858cd4 feat: update physical table schema on alter logical tables (#3585)
* feat: update physical table schema on alter

* feat: alter logical table in sql path

* feat: invalidate cache step1

* feat: invalidate cache step2

* feat: invalidate cache step3

* feat: invalidate cache step4

* fix: failed ut

* fix: standalone cache invalidator

* feat: log the count of already finished

* feat: re-invalidate cache

* chore: by comment

* chore: Update src/common/meta/src/ddl/create_logical_tables.rs

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-03-26 14:29:53 +00:00
dimbtp
dd18d8c97b build(deps): remove some unused dependencies (#3582)
* build(deps): remove some unused dependencies

* add `arc-swap` dependency back
2024-03-26 12:48:28 +00:00
Ruihang Xia
8f9676aad2 fix: incorrect version info in (#3586)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2024-03-26 09:31:01 +00:00
shuiyisong
0bb949787c refactor: introduce new Output with OutputMeta (#3466)
* refactor: introduce new output struct

* chore: add helper function

* chore: update comment

* chore: update commit

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: rename according to cr

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2024-03-11 02:24:09 +00:00
JeremyHi
352bd7b6fd feat: max-txn-ops option (#3458)
* feat: max-txn-ops limit

* chore: by comment
2024-03-08 02:34:40 +00:00
LFC
2035e7bf4c refactor: set the actual bound port in server handler (#3353)
* refactor: set the actual bound port so we can use port 0 in testing

* Update src/servers/src/server.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fmt

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
2024-02-23 02:49:11 +00:00
dennis zhuang
564fe3beca feat: impl migrate_region and procedure_state SQL function (#3325)
* fix: logical region can't find region routes

* feat: fetch partitions info in batch

* refactor: rename batch functions

* refactor: rename DdlTaskExecutor to ProcedureExecutor

* feat: impl migrate_region and query_procedure_state for ProcedureExecutor

* feat: adds SQL function procedure_state and finish migrate_region impl

* fix: constant vector

* feat: unit tests for migrate_region and procedure_state

* test: test region migration by SQL

* fix: compile error after rebeasing

* fix: clippy warnings

* feat: ensure procedure_state and migrate_region can be only called under greptime catalog

* fix: license header
2024-02-22 02:37:11 +00:00
SteveLauC
e9a2b0a9ee chore: use workspace-wide lints (#3352)
* chore: use workspace-wide lints

* respond to review
2024-02-22 01:01:10 +00:00
Weny Xu
41656c8635 refactor: allocate table id in the procedure (#3271)
* refactor: replace TableMetadataManager with TableNameManager

* refactor: allocate table id in the procedure

* refactor: refactor client logical of handling retries

* feat(test_util): add TestCreateTableExprBuilder

* feat(test_util): add MockDatanodeManager

* feat(test_util): add new_ddl_context

* feat(test_util): add build_raw_table_info_from_expr

* feat(test_util): add MockDatanodeManager::new

* feat(procedure): add downcast_output_ref to Status

* test(create_table): add tests for CreateTableProcedure on_prepare

* refactor(ddl): rename handle_operate_region_error to add_peer_context_if_need

* test(create_table): add tests for CreateTableProcedure on_datanode_create_regions

* test(create_table): add tests for CreateTableProcedure on_create_metadata

* refactor(meta): use CreateTableExprBuilder

* feat(create_table): ensure number of partitions is greater than 0

* refactor: rename to add_peer_context_if_needed

* feat: add context for panic

* refactor: simplify the should_retry

* refactor: use Option<&T> instead of &Option<T>

* refactor: move downcast_output_ref under cfg(test)

* chore: fmt toml
2024-02-21 04:38:46 +00:00
shuiyisong
bf5e1905cd refactor: bring metrics to http output (#3247)
* refactor: bring metrics to http output

* chore: remove unwrap

* chore: make walk plan accumulate

* chore: change field name and comment

* chore: add metrics to http resp header

* chore: move PrometheusJsonResponse to a separate file and impl IntoResponse

* chore: put metrics in prometheus resp header too
2024-02-20 03:25:18 +00:00
dennis zhuang
8b73067815 feat: impl partitions and region_peers information schema (#3278)
* feat: impl partitions table

* fix: typo

* feat: impl region_peers information schema

* chore: rename region_peers to greptime_region_peers

* chore: rename statuses to upper case

* fix: comments

* chore: update partition result

* chore: remove redundant checking

* refactor: replace 42 with constant

* feat: fetch region routes in batch
2024-02-19 06:47:14 +00:00
tison
4e04a4e48f build: support build without git (#3309)
* build: support build without git

Signed-off-by: tison <wander4096@gmail.com>

* chore

Signed-off-by: tison <wander4096@gmail.com>

* address comment

Signed-off-by: tison <wander4096@gmail.com>

* fix syntax

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2024-02-18 10:30:01 +00:00
fys
74bfb09195 feat: support cache for batch_get in CachedMetaKvBackend (#3277)
* feat: support cache for batch_get in CachedMetaKvBackend.

* add doc of CachedMetaKvBackend

* fix: cr

* fix: correct some words

* fix cr
2024-02-06 03:15:03 +00:00
shuiyisong
4cbdf64d52 chore: start plugins during standalone startup & comply with current catalog while changing database (#3282)
* chore: start plugins in standalone

* chore: respect current catalog in use statement for mysql

* chore: reduce unnecessory convert to string

* chore: reduce duplicate code
2024-02-06 02:41:37 +00:00
shuiyisong
6ab3a88042 fix: use fe_opts after setup_frontend_plugins in standalone (#3275)
* chore: modify standalone startup opts

* chore: move frontend and datanode options
2024-02-02 10:36:08 +00:00
tison
60e760b168 fix: cli export database default value (#3259)
Signed-off-by: tison <wander4096@gmail.com>
2024-01-30 09:56:05 +00:00
LFC
f81e37f508 refactor: make http server built flexibly (#3225)
* refactor: make http server built flexibly

* Apply suggestions from code review

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* fix: resolve PR comments

* Fix CI.

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
2024-01-24 03:45:08 +00:00
dennis zhuang
6a12c27e78 feat: make query be aware of timezone setting (#3175)
* feat: let TypeConversionRule aware query context timezone setting

* chore: don't optimize explain command

* feat: parse string into timestamp with timezone

* fix: compile error

* chore: check the scalar value type in predicate

* chore: remove mut for engine context

* chore: return none if the scalar value is utf8 in time range predicate

* fix: some fixme

* feat: let Date and DateTime parsing from string value be aware of timezone

* chore: tweak

* test: add datetime from_str test with timezone

* feat: construct function context from query context

* test: add timezone test for to_unixtime and date_format function

* fix: typo

* chore: apply suggestion

* test: adds string with timezone

* chore: apply CR suggestion

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* chore: apply suggestion

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2024-01-22 14:14:03 +00:00
dennis zhuang
5e89472b2e feat: adds parse options for SQL parser (#3193)
* feat: adds parse options for parser and timezone to scan request

* chore: remove timezone in ScanRequest

* feat: remove timezone in parse options and adds type checking to parititon columns

* fix: comment

* chore: apply suggestions

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: format

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-01-19 09:16:36 +00:00
niebayes
63205907fb refactor: introduce common-wal to aggregate wal stuff (#3171)
* refactor: aggregate wal configs

* refactor: move wal options to common-wal

* chore: slim Cargo.toml

* fix: add missing crates

* fix: format

* chore: update comments

* chore: add testing feature gate for test_util

* fix: apply suggestions from code review

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* fix: apply suggestions from code review

* fix: compiling

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
2024-01-18 03:49:37 +00:00
LFC
3cfd60e139 refactor: expose region edit in mito engine (#3179)
* refactor: expose region edit in mito engine

* feat: add a method for editing region directly

* fix: resolve PR comments

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
2024-01-17 14:25:08 +00:00
LFC
93f28c2a37 refactor: make grpc service able to be added dynamically (#3160) 2024-01-15 04:33:27 +00:00
LFC
51a3fbc7bf refactor: change how frontend grpc services are orchestrated (#3134) 2024-01-11 02:26:44 +00:00
Weny Xu
7fad4e8356 fix: incorrect parsing broker_endpoints env variable (#3135) 2024-01-10 13:59:49 +00:00
LFC
fcacb100a2 chore: expose some codes to let other projects use them (#3115) 2024-01-08 06:32:01 +00:00
zyy17
a89840f5f9 refactor(metrics): add 'greptime_' prefix for every metrics (#3093)
* refactor(metrics): add 'greptimedb_' prefix for every metrics

* chore: use 'greptime_' as prefix

* chore: add some prefix for new metrics

* chore: fix format error
2024-01-05 08:12:23 +00:00
WU Jingdi
4d250ed054 fix: Optimize export metric behavior (#3047)
* fix: optimze export metric bahavior

* chor: fix ci

* chore: update config format

* chore: fix format
2024-01-04 06:40:50 +00:00
LFC
ec43b9183d feat: table route for metric engine (#3053)
* feat: table route for metric engine

* feat: register logical regions

* fix: open logical region (#96)

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
2024-01-04 06:30:17 +00:00
LFC
7551432cff refactor: merge standalone and metasrv table metadata allocators (#3035)
* refactor: merge standalone and metasrv table metadata allocators

* Update src/common/meta/src/ddl/table_meta.rs

Co-authored-by: niebayes <niebayes@gmail.com>

* Update src/common/meta/src/ddl/table_meta.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: niebayes <niebayes@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2023-12-29 08:50:59 +00:00
zyy17
b8b1e98399 refactor: use string type instead of Option type for '--store-key-prefix' (#3018)
* refactor: use string type instead of Option type for '--store-key-prefix'

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine for code review comments

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2023-12-27 11:26:30 +00:00
WU Jingdi
d1ee1ba56a feat: support set timezone in db (#2992)
* feat: support set timezone in db

* chore: fix  ci

* chore: fix code advice

* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
LFC
eadde72973 chore: "fix: revert unfinished route table change" (#3009)
Revert "fix: revert unfinished route table change (#3008)"

This reverts commit 8ce8a8f3c7.
2023-12-27 02:40:59 +00:00
Ruihang Xia
8ce8a8f3c7 fix: revert unfinished route table change (#3008)
* Revert "refactor: hide `RegionRoute` behind `TableRouteValue` (#2989)"

This reverts commit 1641fd572a.

* Revert "feat: MetricsEngine table route (part 1) (#2952)"

This reverts commit 6ac47e939c.
2023-12-26 09:56:49 +00:00
Weny Xu
c902d43380 refactor(remote_wal): add StandaloneWalConfig (#3002)
* feat: integrate remote wal to standalone

* fix: test

* refactor: refactor standalone wal config

* chore: change default kafka port to 9092

* chore: apply suggestions from CR

---------

Co-authored-by: niebayes <niebayes@gmail.com>
2023-12-26 04:31:41 +00:00
tison
417be13400 feat: adopt human-panic crash reports (#2996)
* feat: adopt human-panic crash reports

Signed-off-by: tison <wander4096@gmail.com>

* build: fix dep toml conflict

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
2023-12-25 13:23:18 +00:00
LFC
1641fd572a refactor: hide RegionRoute behind TableRouteValue (#2989)
* refactor: hide `RegionRoute` behind `TableRouteValue` (Metric Engine table route, part 2)

* Update src/common/meta/src/ddl/create_table.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* Update src/meta-srv/src/procedure/region_migration/test_util.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: resolve PR comments

* fix: rustfmt

* compatible with the old TableRouteValue

* Update src/common/meta/src/key/table_route.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
2023-12-25 11:37:50 +00:00
zyy17
0d42651047 feat: add '--store-key-prefix' to support multiple meta instances can use the same etcd backend (#2988)
* feat: add '--store-key-prefix' to support multiple meta instances can use the same etcd backend

* refactor: refine lock_key
2023-12-25 09:33:25 +00:00
niebayes
830a91c548 feat(remote_wal): implement topic allocation (#2970)
* chore: implement wal options allocator

* chore: implement round-robin topic selector

* feat: add shuffle to round-robin topic selector

* chore: implement kafka topic manager

* test: add tests for wal options allocator

* test: add wal provider to test config files

* test: leave todos for adding tests for remote wal

* fix: resolve review conversations

* fix: typo
2023-12-22 08:26:48 +00:00
WU Jingdi
675767c023 feat: Support export metric in remote write format (#2928)
* feat: remote write metric task

* chore: pass stanalone task to frontend

* chore: change name to system metric

* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
LFC
8776b1204b feat: use explicitly set table id (#2945)
* feat: make standalone table metadata allocator able to use explicitly set table id

* rebase

* fix: resolve PR comments
2023-12-21 05:51:43 +00:00
niebayes
9da1f236d9 feat(remote_wal): add skeleton for remote wal related to datanode (#2941)
* refactor: refactor wal config

* test: update tests related to wal

* feat: introduce kafka wal config

* chore: augment proto with wal options

* feat: augment region open request with wal options

* feat: augment mito region with wal options

* feat: augment region create request with wal options

* refactor: refactor log store trait

* feat: add skeleton for kafka log store

* feat: generalize building log store when starting datanode

* feat: integrate wal options to region write

* chore: minor update

* refactor: remove wal options from region create/open requests

* fix: compliation issues

* chore: insert wal options into region options upon initializing region server

* chore: integrate wal options into region options

* chore: fill in kafka wal config

* chore: reuse namespaces while writing to wal

* chore: minor update

* chore: fetch wal options from region while handling truncate/flush

* fix: region options test

* fix: resolve some review conversations

* refactor: serde with wal options

* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
niebayes
839e653e0d feat(remote_wal): add skeleton for remote wal related to meta srv (#2933)
* feat: introduce wal config and kafka config

* feat: introduce kafka topic manager and selector

* feat: introduce region wal options

* chore: build region wal options upon starting meta srv

* feat: integrate region wal options allocator into table meta allocator

* chore: add wal config to metasrv.example.toml

* chore: add region wal options map to create table procedure

* feat: augment region create request with wal options

* feat: augment DatanodeTableValue with region wal options map

* chore: encode region wal options upon constructing table creator

* feat: persist region wal options when creating table meta

* fix: sqlness test

* chore: set default wal provider to raft-engine

* refactor: refactor wal options

* chore: update wal options allocator

* refactor: rename region wal options to wal options

* chore: update usages of region wal options

* chore: add some comments to kafka

* chore: fill in kafka config

* test: add tests for serde wal config

* test: add tests for wal options

* refactor: refactor wal options allocator to enum

* refactor: store wal options into the request options instead

* fix: typo

* fix: typo

* refactor: move wal options map to region info

* refactor: refacto serialization and deserialization of wal options

* refactor: use serde_json to encode wal options

* chore: rename wal_options_map to region_wal_options

* chore: resolve some review comments

* fix: typo

* refactor: replace kecab-case with snake_case

* fix: sqlness and converage tests

* fix: typo

* fix: coverage test

* fix: coverage test

* chore: resolve some review conversations

* fix: resolve some review conversations

* chore: format comments in metasrv.example.toml

* chore: update import style

* feat: integrate wal options allocator to standalone mode

* test: add compatible test for OpenRegion

* test: add compatible test for UpdateRegionMetadata

* chore: remove based suffix from topic selector type
2023-12-19 12:43:47 +00:00
LFC
033a065359 refactor: make sequence bounded with max value (#2937)
* refactor: make sequence bounded with max value

(cherry picked from commit 3a8eba6f863327a96b617cd86ee2d39fac30abb2)

* fix: resolve PR comments
2023-12-18 07:05:28 +00:00
LFC
181e16a11a refactor: make instance started separately (#2911)
* refactor: make instance started separately, to support further integrated into other binaries

* fix: resolve PR comments

* fix: resolve PR comments
2023-12-14 03:44:29 +00:00
Weny Xu
cf6bba09fd refactor: use downgrading the region instead of closing region (#2863)
* refactor: use downgrading the region instead of closing region

* feat: enhance the tests for alive keeper

* feat: add a metric to track region lease expired

* chore: apply suggestions from CR

* chore: enable logging for test_distributed_handle_ddl_request

* refactor: simplify lease keeper

* feat: add metrics for lease keeper

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: move OpeningRegionKeeper to common_meta

* feat: register operating regions to MemoryRegionKeeper
2023-12-12 09:24:17 +00:00