Lei, HUANG
2c1b1cecc8
chore: add bound check for raft-engine logstore ( #3073 )
...
* chore: add bound check for raft-engine logstore
* feat: add bound check to append_batch API
* chore: check entry id during replay
* chore: resolve conflicts
* feat: add allow_stale_entries options to force obsolete wal entries
* chore: resolve some comments
2024-01-09 06:42:46 +00:00
Lei, HUANG
62db28b465
feat: add options to enable log recycle and periodical fsync ( #3114 )
...
* feat: add options to enable log recycle and periodical fsync
* fix: resolve review comments
* fix: conflicts
2024-01-09 06:41:23 +00:00
fys
6e860bc0fd
feat: support grpc for otlp trace and metrics ( #3105 )
...
* feat: add grpc support for otlp trace and metrics
* cr: add some comment
* fix: ut
* fix: cr
2024-01-09 05:01:48 +00:00
Yingwen
8bd4a36136
feat(mito): Init the write cache in datanode ( #3100 )
...
* feat: add builder to build cache manager
* refactor: make MitoEngine::new async
* refactor: refactor object store creation
* refactor: add helper fn to attaches layers
* feat: fn to build fs store
* feat: add write cache to engine
* feat: config write cache
* style: fix clippy
* test: fix test
* feat: add warning
* chore: add experimental prefix to configs
* test: fix config test
* test: test weighted size
* feat: add switch to enable write cache
* fix: update cache stats by using get
* style: use then
2024-01-09 04:40:22 +00:00
tison
316d843482
feat: support CSV format in sql HTTP API ( #3062 )
...
* chore: fix typo
Signed-off-by: tison <wander4096@gmail.com >
* add csv format
Signed-off-by: tison <wander4096@gmail.com >
* flatten response
Signed-off-by: tison <wander4096@gmail.com >
* more flatten response
Signed-off-by: tison <wander4096@gmail.com >
* add CSV format
Signed-off-by: tison <wander4096@gmail.com >
* format InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format ErrorResponse
Signed-off-by: tison <wander4096@gmail.com >
* propagate ErrorResponse to InfluxdbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format GreptimedbV1Response
Signed-off-by: tison <wander4096@gmail.com >
* format CsvResponse
Signed-off-by: tison <wander4096@gmail.com >
* impl IntoResponse for QueryResponse
Signed-off-by: tison <wander4096@gmail.com >
* promql
Signed-off-by: tison <wander4096@gmail.com >
* sql
Signed-off-by: tison <wander4096@gmail.com >
* compile
Signed-off-by: tison <wander4096@gmail.com >
* fixup aide
Signed-off-by: tison <wander4096@gmail.com >
* clear debt
Signed-off-by: tison <wander4096@gmail.com >
* fixup UT test_recordbatches_conversion
Signed-off-by: tison <wander4096@gmail.com >
* fixup IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup more IT cases
Signed-off-by: tison <wander4096@gmail.com >
* fixup test-integration cases
Signed-off-by: tison <wander4096@gmail.com >
* update comment
Signed-off-by: tison <wander4096@gmail.com >
* fixup deserialize and most query < 1ms
Signed-off-by: tison <wander4096@gmail.com >
* fixup auth tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup tests
Signed-off-by: tison <wander4096@gmail.com >
* fixup and align X-GreptimeDB headers
Signed-off-by: tison <wander4096@gmail.com >
* fixup compile
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-01-08 10:54:27 +00:00
Weny Xu
f1a4750576
feat(tests-integration): add more region migration integration tests ( #3094 )
2024-01-04 08:18:46 +00:00
WU Jingdi
4d250ed054
fix: Optimize export metric behavior ( #3047 )
...
* fix: optimze export metric bahavior
* chor: fix ci
* chore: update config format
* chore: fix format
2024-01-04 06:40:50 +00:00
LFC
ec43b9183d
feat: table route for metric engine ( #3053 )
...
* feat: table route for metric engine
* feat: register logical regions
* fix: open logical region (#96 )
---------
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2024-01-04 06:30:17 +00:00
ClSlaid
5c66ce6e88
chore: remove unnecessary result wrappings ( #3084 )
...
patch: remove unnecessary result wrappings
Signed-off-by: 蔡略 <cailue@bupt.edu.cn >
2024-01-03 10:20:33 +00:00
Weny Xu
611a8aa2fe
feat(tests-integration): add a naive region migration integration test ( #3078 )
...
* fix: fix heartbeat handler ignore upgrade candidate instruction
* fix: fix handler did not inject wal options
* feat: expose `RegionMigrationProcedureTask`
* feat(tests-integration): add a naive region migration test
* chore: apply suggestions from CR
* feat: add test if the target region has migrated
* chore: apply suggestions from CR
2024-01-03 07:12:59 +00:00
Weny Xu
128d3717fa
test(tests-integration): add a naive test with kafka wal ( #3071 )
...
* chore(tests-integration): add setup tests with kafka wal to README.md
* feat(tests-integration): add meta wal config
* fix(tests-integration): fix sign of both_instances_cases_with_kafka_wal
* chore(tests-integration): set num_topic to 3 for tests
* test(tests-integration): add a naive test with kafka wal
* chore: apply suggestions from CR
2024-01-02 09:05:20 +00:00
Weny Xu
d87ab06b28
feat: add kafka wal integration test utils ( #3069 )
...
* feat(tests-integration): add wal_config
* feat: add kafka wal integration test utils
2024-01-02 07:38:43 +00:00
AntiTopQuark
4460af800f
feat(TableRouteValue): add panic notes and type checks ( #3031 )
...
* refactor(TableRouteValue): add panic notes and type checks
* chore: add deprecate develop branch warning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add error defines and checks
* Update README.md
* update code format and fix tests
* update name of error
* delete unused note
* fix unsafe .expect() for region_route()
* update error name
* update unwrap
* update code format
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-30 13:02:26 +00:00
dennis zhuang
11ae85b1cd
feat: adds information_schema.schemata ( #3051 )
...
* feat: improve information_schema.columns
* feat: adds information_schema.schemata
* fix: instance test
* fix: comment
2023-12-29 09:22:31 +00:00
LFC
7551432cff
refactor: merge standalone and metasrv table metadata allocators ( #3035 )
...
* refactor: merge standalone and metasrv table metadata allocators
* Update src/common/meta/src/ddl/table_meta.rs
Co-authored-by: niebayes <niebayes@gmail.com >
* Update src/common/meta/src/ddl/table_meta.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
---------
Co-authored-by: niebayes <niebayes@gmail.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-12-29 08:50:59 +00:00
Weny Xu
e16f093282
test(remote_wal): add sqlness with kafka wal ( #3027 )
...
* feat(sqlness): add kafka wal config
* chore: add sqlness with kafka wal ci config
* fix: fix config
* chore: apply suggestions from CR
* fix: add metasrv config to sqlness with kafka
* fix: replay memtable should from flushed_entry_id + 1
* fix: should set append flag to fopen
* feat: start wal allocator in standalone meta mode
* feat: append a noop record after kafka topic initialization
* test: ignore tests temporally
* test: change sqlness kafka wal config
2023-12-29 08:17:22 +00:00
WU Jingdi
d1ee1ba56a
feat: support set timezone in db ( #2992 )
...
* feat: support set timezone in db
* chore: fix ci
* chore: fix code advice
* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
LFC
eadde72973
chore: "fix: revert unfinished route table change" ( #3009 )
...
Revert "fix: revert unfinished route table change (#3008 )"
This reverts commit 8ce8a8f3c7
2023-12-27 02:40:59 +00:00
Ruihang Xia
8ce8a8f3c7
fix: revert unfinished route table change ( #3008 )
...
* Revert "refactor: hide `RegionRoute` behind `TableRouteValue` (#2989 )"
This reverts commit 1641fd572a#2952 )"
This reverts commit 6ac47e939c
2023-12-26 09:56:49 +00:00
LFC
196c06db14
feat: make logging to stdout configurable ( #3003 )
...
* feat: make logging to stdout configurable
* fix sqlness
* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
Weny Xu
89129c99c8
chore: setup kafka standalone in coverage test ( #2984 )
...
* chore: setup kafka standalone in coverage test
* test: add a naive test for topic manager
2023-12-25 11:18:54 +00:00
niebayes
7d509e97f6
chore: move some commonly referenced crates to workspace Cargo.toml ( #2981 )
...
fix: resolve conflicts
2023-12-22 09:13:18 +00:00
niebayes
830a91c548
feat(remote_wal): implement topic allocation ( #2970 )
...
* chore: implement wal options allocator
* chore: implement round-robin topic selector
* feat: add shuffle to round-robin topic selector
* chore: implement kafka topic manager
* test: add tests for wal options allocator
* test: add wal provider to test config files
* test: leave todos for adding tests for remote wal
* fix: resolve review conversations
* fix: typo
2023-12-22 08:26:48 +00:00
Ning Sun
43f01cc594
feat: add a default internal schema ( #2974 )
2023-12-22 06:25:19 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
LFC
8776b1204b
feat: use explicitly set table id ( #2945 )
...
* feat: make standalone table metadata allocator able to use explicitly set table id
* rebase
* fix: resolve PR comments
2023-12-21 05:51:43 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
LFC
6ac47e939c
feat: MetricsEngine table route (part 1) ( #2952 )
...
* refactor: make `TableRouteValue` an enum to add the variant of MetricsEngine table route
* fix: resolve PR comments
* Update src/common/meta/src/key/table_route.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-12-20 07:31:59 +00:00
niebayes
839e653e0d
feat(remote_wal): add skeleton for remote wal related to meta srv ( #2933 )
...
* feat: introduce wal config and kafka config
* feat: introduce kafka topic manager and selector
* feat: introduce region wal options
* chore: build region wal options upon starting meta srv
* feat: integrate region wal options allocator into table meta allocator
* chore: add wal config to metasrv.example.toml
* chore: add region wal options map to create table procedure
* feat: augment region create request with wal options
* feat: augment DatanodeTableValue with region wal options map
* chore: encode region wal options upon constructing table creator
* feat: persist region wal options when creating table meta
* fix: sqlness test
* chore: set default wal provider to raft-engine
* refactor: refactor wal options
* chore: update wal options allocator
* refactor: rename region wal options to wal options
* chore: update usages of region wal options
* chore: add some comments to kafka
* chore: fill in kafka config
* test: add tests for serde wal config
* test: add tests for wal options
* refactor: refactor wal options allocator to enum
* refactor: store wal options into the request options instead
* fix: typo
* fix: typo
* refactor: move wal options map to region info
* refactor: refacto serialization and deserialization of wal options
* refactor: use serde_json to encode wal options
* chore: rename wal_options_map to region_wal_options
* chore: resolve some review comments
* fix: typo
* refactor: replace kecab-case with snake_case
* fix: sqlness and converage tests
* fix: typo
* fix: coverage test
* fix: coverage test
* chore: resolve some review conversations
* fix: resolve some review conversations
* chore: format comments in metasrv.example.toml
* chore: update import style
* feat: integrate wal options allocator to standalone mode
* test: add compatible test for OpenRegion
* test: add compatible test for UpdateRegionMetadata
* chore: remove based suffix from topic selector type
2023-12-19 12:43:47 +00:00
LFC
c7b36779c1
fix: correctly decode mysql timestamp binary value ( #2946 )
...
* fix: correctly decode mysql timestamp binary value
* fix: ut
* fix: resolve PR comments
2023-12-19 11:05:28 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
LFC
033a065359
refactor: make sequence bounded with max value ( #2937 )
...
* refactor: make sequence bounded with max value
(cherry picked from commit 3a8eba6f863327a96b617cd86ee2d39fac30abb2)
* fix: resolve PR comments
2023-12-18 07:05:28 +00:00
dennis zhuang
262a79a170
feat: adds some tables to information_schema ( #2935 )
...
* feat: adds engines table to information_schema
* feat: adds COLUMN_PRIVILEGES and COLUMN_STATISTICS
* feat: refactor memory tables
* chore: rename memory_tables
* test: adds unit tests
* chore: format
* chore: style
* fix: by cr comments
* refactor: tables
2023-12-18 06:10:22 +00:00
JeremyHi
99dda93f0e
feat: sql with influxdb v1 result format ( #2917 )
...
* feat: sql with influxdb v1 result format
* chore: add unit tests
* feat: minor refactor
* chore: by comment
* chore; u128 to u64 since serde can't deser u128 in enum
* chore: by comment
* chore: apply suggestion
* chore: revert suggestion
* chore: try again
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-13 16:15:37 +00:00
WU Jingdi
34635558d2
fix: support tailing commas in SQL ( #2913 )
...
* fix: support tailing commas in SQL
* fix: broken ci
2023-12-12 11:56:23 +00:00
Ruihang Xia
8a74bd36f5
style: rename *Adaptor to *Adapter ( #2914 )
...
* rename RecordBatchStreamAdaptor to RecordBatchStreamWrapper
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* rename ServerSqlQueryHandlerAdaptor to ServerSqlQueryHandlerAdapter
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-12 09:45:09 +00:00
Weny Xu
cf6bba09fd
refactor: use downgrading the region instead of closing region ( #2863 )
...
* refactor: use downgrading the region instead of closing region
* feat: enhance the tests for alive keeper
* feat: add a metric to track region lease expired
* chore: apply suggestions from CR
* chore: enable logging for test_distributed_handle_ddl_request
* refactor: simplify lease keeper
* feat: add metrics for lease keeper
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* refactor: move OpeningRegionKeeper to common_meta
* feat: register operating regions to MemoryRegionKeeper
2023-12-12 09:24:17 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
WU Jingdi
aa89d9deef
fix: replace opendal PrometheusLayer ( #2861 )
...
* fix: replace opendal `PrometheusLayer`
* chore: add docs on `PrometheusMetricsLayer`
* chore: fix code advice
* chore: fix bug on `PrometheusMetricsLayer`
2023-12-05 07:15:45 +00:00
LFC
6344b1e0db
fix: fragile integration tests ( #2870 )
2023-12-05 02:35:23 +00:00
LFC
9ce9421850
refactor: add builder for Frontend ( #2849 )
2023-12-01 04:39:47 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
Weny Xu
7547e7ebdf
fix: fix procedure loaders not found issue ( #2824 )
2023-11-27 10:50:28 +00:00
Wei
3a4c9f2b45
feat: supports decimal type in RPC ( #2788 )
...
* refactor: ColumnDataTypeWrapper
* feat: decimal128 grpc
* feat: add test case
* chore: add TODO
* chore: empty line
* chore: remove precision and scale
* refactor: remove precision and scale
* chore: remove sqlness test
* chore: rename
* chore: proto version
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/mito2/src/memtable/time_series.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-24 09:19:33 +00:00
LFC
64a36e9b36
refactor: start datanode more flexibly ( #2800 )
...
* refactor: start datanode more flexibly
* Update src/datanode/src/datanode.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fix: resolve PR comments
* Apply suggestions from code review
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
Co-authored-by: JeremyHi <jiachun_feng@proton.me >
2023-11-24 08:16:21 +00:00
tison
ff8ab6763b
chore: internal ChrootKvBackend refactor and test ( #2799 )
...
* try avoid rate limit
Signed-off-by: tison <wander4096@gmail.com >
* chroot utilities as method
Signed-off-by: tison <wander4096@gmail.com >
* add test
Signed-off-by: tison <wander4096@gmail.com >
* make clippy happy
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-11-24 06:45:09 +00:00
Ning Sun
85eebcb16f
fix: correct mysql timestamp encoding for binary protocol ( #2797 )
...
* fix: correct mysql timestamp encoding for binary protocol
* test: add tests for mysql timestamp encoding
2023-11-23 18:54:59 +00:00
