WU Jingdi
d1ee1ba56a
feat: support set timezone in db ( #2992 )
...
* feat: support set timezone in db
* chore: fix ci
* chore: fix code advice
* fix: rename `time_zone` to `timezone`
2023-12-27 09:19:39 +00:00
LFC
196c06db14
feat: make logging to stdout configurable ( #3003 )
...
* feat: make logging to stdout configurable
* fix sqlness
* fix: resolve PR comments
2023-12-26 07:37:50 +00:00
Weny Xu
c902d43380
refactor(remote_wal): add StandaloneWalConfig ( #3002 )
...
* feat: integrate remote wal to standalone
* fix: test
* refactor: refactor standalone wal config
* chore: change default kafka port to 9092
* chore: apply suggestions from CR
---------
Co-authored-by: niebayes <niebayes@gmail.com >
2023-12-26 04:31:41 +00:00
niebayes
830a91c548
feat(remote_wal): implement topic allocation ( #2970 )
...
* chore: implement wal options allocator
* chore: implement round-robin topic selector
* feat: add shuffle to round-robin topic selector
* chore: implement kafka topic manager
* test: add tests for wal options allocator
* test: add wal provider to test config files
* test: leave todos for adding tests for remote wal
* fix: resolve review conversations
* fix: typo
2023-12-22 08:26:48 +00:00
WU Jingdi
675767c023
feat: Support export metric in remote write format ( #2928 )
...
* feat: remote write metric task
* chore: pass stanalone task to frontend
* chore: change name to system metric
* fix: add header and rename to export metrics
2023-12-22 02:09:12 +00:00
LFC
8776b1204b
feat: use explicitly set table id ( #2945 )
...
* feat: make standalone table metadata allocator able to use explicitly set table id
* rebase
* fix: resolve PR comments
2023-12-21 05:51:43 +00:00
Weny Xu
6c1c7d8d24
feat: allow initializing regions in background ( #2930 )
...
* refactor: open regions in background
* feat: add status code of RegionNotReady
* feat: use RegionNotReady instead of RegionNotFound for a registering region
* chore: apply suggestions from CR
* feat: add status code of RegionBusy
* feat: return RegionBusy for mutually exclusive operations
2023-12-20 11:37:46 +00:00
niebayes
9da1f236d9
feat(remote_wal): add skeleton for remote wal related to datanode ( #2941 )
...
* refactor: refactor wal config
* test: update tests related to wal
* feat: introduce kafka wal config
* chore: augment proto with wal options
* feat: augment region open request with wal options
* feat: augment mito region with wal options
* feat: augment region create request with wal options
* refactor: refactor log store trait
* feat: add skeleton for kafka log store
* feat: generalize building log store when starting datanode
* feat: integrate wal options to region write
* chore: minor update
* refactor: remove wal options from region create/open requests
* fix: compliation issues
* chore: insert wal options into region options upon initializing region server
* chore: integrate wal options into region options
* chore: fill in kafka wal config
* chore: reuse namespaces while writing to wal
* chore: minor update
* chore: fetch wal options from region while handling truncate/flush
* fix: region options test
* fix: resolve some review conversations
* refactor: serde with wal options
* fix: resolve some review conversations
2023-12-20 09:01:17 +00:00
LFC
c7b36779c1
fix: correctly decode mysql timestamp binary value ( #2946 )
...
* fix: correctly decode mysql timestamp binary value
* fix: ut
* fix: resolve PR comments
2023-12-19 11:05:28 +00:00
Ruihang Xia
6b8dbcfb54
chore: update toolchain to 20231219 ( #2932 )
...
* update toolchain file, remove unused feature gates
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update action file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update to 12-19
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-12-19 07:24:08 +00:00
JeremyHi
99dda93f0e
feat: sql with influxdb v1 result format ( #2917 )
...
* feat: sql with influxdb v1 result format
* chore: add unit tests
* feat: minor refactor
* chore: by comment
* chore; u128 to u64 since serde can't deser u128 in enum
* chore: by comment
* chore: apply suggestion
* chore: revert suggestion
* chore: try again
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-13 16:15:37 +00:00
Yingwen
6a57f4975e
perf(mito): scan SSTs and memtables in parallel ( #2852 )
...
* feat: seq scan support parallelism
* feat: scan region by parallelism in config
* feat: enlarge channel size
* chore: parallel builder logs
* feat: use parallel reader accroding to source num
* chore: 128 channel size
* feat: add fetch cost metrics
* feat: add channel size to config
* feat: builder cost
* feat: logs
* feat: compiler error
* feat: fetch cost
* feat: convert cost
* chore: Revert "feat: logs"
This reverts commit 01e0df2c3a
2023-12-11 06:43:17 +00:00
Niwaka
cfe3a2c55e
feat!: support table ddl for custom storage ( #2733 )
...
* feat: support table ddl for custom_storage
* refactor: rename extract_variant_name to name
* chore: add blank
* chore: keep compatible
* feat: rename custom_stores to providers
* chore: rename
* chore: config
* refactor: add should_retry in client Error
* fix: test fail
* chore: remove unused options
* chore: remove unused import
* chore: remove the blanks.
* chore: revert
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2023-12-06 15:59:01 +00:00
Yingwen
abbac46c05
fix: do not expose manifest compression algorithm ( #2835 )
...
* fix: don't allow to set manifest compression algorithm
* docs: update config examples
2023-11-29 05:49:40 +00:00
Wei
3a4c9f2b45
feat: supports decimal type in RPC ( #2788 )
...
* refactor: ColumnDataTypeWrapper
* feat: decimal128 grpc
* feat: add test case
* chore: add TODO
* chore: empty line
* chore: remove precision and scale
* refactor: remove precision and scale
* chore: remove sqlness test
* chore: rename
* chore: proto version
* chore: cr comment.
Co-authored-by: Yingwen <realevenyag@gmail.com >
* Update src/mito2/src/memtable/time_series.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-24 09:19:33 +00:00
Ning Sun
85eebcb16f
fix: correct mysql timestamp encoding for binary protocol ( #2797 )
...
* fix: correct mysql timestamp encoding for binary protocol
* test: add tests for mysql timestamp encoding
2023-11-23 18:54:59 +00:00
tison
102e43aace
test: use EtcdStore in IT cases ( #2734 )
...
* test: use EtcdStore in IT cases
Signed-off-by: tison <wander4096@gmail.com >
* retrigger CI
Signed-off-by: tison <wander4096@gmail.com >
* refactor: KvPair can take etcd KeyValue
Signed-off-by: tison <wander4096@gmail.com >
* temporary use fork
Signed-off-by: tison <wander4096@gmail.com >
* drop cloned
Signed-off-by: tison <wander4096@gmail.com >
* chroot_key_value
Signed-off-by: tison <wander4096@gmail.com >
* chroot and prepend in each point
Signed-off-by: tison <wander4096@gmail.com >
* adjust call points
Signed-off-by: tison <wander4096@gmail.com >
* cargo clippy
Signed-off-by: tison <wander4096@gmail.com >
* point to upstream etcd-client
Signed-off-by: tison <wander4096@gmail.com >
* test etcd chroot
Signed-off-by: tison <wander4096@gmail.com >
* add NO_CHROOT constant
Signed-off-by: tison <wander4096@gmail.com >
* check
Signed-off-by: tison <wander4096@gmail.com >
* handle range end
Signed-off-by: tison <wander4096@gmail.com >
* handle special encoded key or range_end
Signed-off-by: tison <wander4096@gmail.com >
* fixup implementation
Signed-off-by: tison <wander4096@gmail.com >
* clippy
Signed-off-by: tison <wander4096@gmail.com >
* avoid test name conflict
Signed-off-by: tison <wander4096@gmail.com >
* chroot to kvbackend level
Signed-off-by: tison <wander4096@gmail.com >
* fixup all occurances
Signed-off-by: tison <wander4096@gmail.com >
* fix type
Signed-off-by: tison <wander4096@gmail.com >
* Update src/common/meta/src/kv_backend/txn.rs
* make github happy
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com >
2023-11-23 05:47:00 +00:00
Yingwen
b9146c88ff
refactor: Remove usages of the old storage crate ( #2777 )
...
* chore: remove storage from some crate
* feat: remove storage config
* feat: remove storage from cmd
* feat: impl stream_to_parquet
* feat: remove storage from operator
* feat: remove stream writer from mito2
* feat: remove storage from project toml
* test: fix config api test
* docs: remove outdated configs
* refactor: remove storage directory
2023-11-20 20:29:41 +08:00
fys
da68d8ce4b
feat: add random weighted choose in load_based selector ( #2234 )
...
* feat: add random weigted choose in load_based selector
* fix: meta cannot save heartbeats when cluster have no region
* chore: print some log
* chore: remove unused code
* cr
* add some logs when filter result is empty
2023-11-20 06:47:42 +00:00
Yingwen
ce959ddd3f
feat(mito): implements row group level page cache ( #2688 )
...
* feat: add page cache
* docs: update mito config toml
* feat: impl CachedPageReader
* feat: use cache reader to read row group
* feat: do not fetch data if we have pages in cache
* chore: return if nothing to fetch
* feat: enlarge page cache size
* test: test write read parquet
* test: test cache
* docs: update comments
* test: fix config api test
* feat: cache metrics
* feat: change default page cache size
* test: fix config api test
2023-11-20 02:55:50 +00:00
WU Jingdi
500e299e40
feat: Enable distributed tracing in greptimedb ( #2755 )
...
* feat: implement distributed tracing
* fix: change usage of span
* fix: use otlp as exporter
* chore: update dependence
* chore: add span info
* chore: add alias
* chore: use instrument instead of trace
2023-11-17 08:51:57 +00:00
Lanqing Yang
5608035074
fix!: improve user experience on setting compression type ( #2765 )
...
* fixes: https://github.com/GreptimeTeam/greptimedb/issues/2758
Chore: improve user experience on setting compression type
* Update src/common/datasource/src/compression.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-11-17 06:27:13 +00:00
Ruihang Xia
6599bb5a46
feat: do not require worker's number to be power of 2 ( #2732 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-11-13 09:04:44 +00:00
tison
f5eede4ce1
feat: support prometheus format_query endpoint ( #2731 )
...
* feat: support prometheus format_query endpoint
Signed-off-by: tison <wander4096@gmail.com >
* add test
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2023-11-10 10:33:28 +00:00
Lei, HUANG
b53537e69b
fix: unify all sst_write_buffer_size usage ( #2712 )
...
* fix: unify all sst_write_buffer_size usage
* fix: some CR comments
* fix: logs
2023-11-09 11:22:47 +00:00
Niwaka
06d273b75a
fix: GET: /config return all configurations when running standalone ( #2630 )
...
* fix: config api return all configurations when running standalone
* chore: follow same style
* fix: avoid panic
2023-11-08 03:22:19 +00:00
Weny Xu
fb8d0c6ce5
refactor: unify the KvBackend ( #2684 )
...
* refactor: unify the KvBackend
* refactor: rename kv_store to kv_backend
* chore: apply suggestions from CR
* refactor: rename kv_store to kv_backend
* refactor: rename KvStoreConfig to KvBackendConfig
2023-11-03 02:46:43 +00:00
WU Jingdi
180bc64cb0
feat: change metric crate from metrics to prometheus ( #2655 )
...
* feat: change metrics to prometheus
* chore: fix code advise
* chore: resolve merge conflict
* chore: fix code advise
2023-10-31 15:46:57 +00:00
Yun Chen
f859932745
fix: convert to ReadableSize & Durations ( #2594 )
...
* fix: convert to ReadableSize & Durations
* fix: change more grpc sender/recv message size to ReadableSize
fix: format
fix: cargo fmt
fix: change cmd test to use durations
fix: revert metaclient change
fix: convert default fields in meta client options
fix: human serde meta client durations
* fix: remove milisecond postfix in heartbeat option
* fix: humantime serde on heartbeat
* fix: update config example
* fix: update integration test config
* fix: address pr comments
* fix: fix pr comment on default annotation
2023-10-13 03:28:29 +00:00
dennis zhuang
ff15bc41d6
feat: improve object storage cache ( #2522 )
...
* feat: refactor object storage cache with moka
* chore: minor fixes
* fix: concurrent issues and invalidate cache after write/delete
* chore: minor changes
* fix: cargo lock
* refactor: rename
* chore: change DEFAULT_OBJECT_STORE_CACHE_SIZE to 256Mib
* fix: typo
* chore: style
* fix: toml format
* chore: toml
* fix: toml format
* Update src/object-store/src/layers/lru_cache/read_cache.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* chore: update Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: update src/object-store/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: refactor and apply suggestions
* fix: typo
* feat: adds back allow list for caching
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* chore: cr suggestion
Co-authored-by: Yingwen <realevenyag@gmail.com >
* refactor: wrap inner Accessor with Arc
* chore: remove run_pending_task in read and write path
* chore: the arc is unnecessary
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2023-10-08 03:27:49 +00:00
Yingwen
657542c0b8
feat(mito): Cache repeated vector for tags ( #2523 )
...
* feat: add vector_cache to CacheManager
* feat: cache repeated vectors
* feat: skip decoding pk if output doesn't contain tags
* test: add TestRegionMetadataBuilder
* test: test ProjectionMapper
* test: test vector cache
* test: test projection mapper convert
* style: fix clippy
* feat: do not cache vector if it is too large
* docs: update comment
2023-10-07 11:36:00 +00:00
Ning Sun
0ad3fb6040
fix: mysql timezone settings ( #2534 )
...
* fix: restore time zone settings for mysql
* test: add integration test for time zone
* test: fix unit test for check
2023-10-07 10:21:32 +00:00
Ruihang Xia
e352fb4495
fix: check for table scan before expanding ( #2491 )
...
* fix: check for table scan before expanding
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* change assert_ok to unwrap
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix clippy warning
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* don't skip dml
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* uncomment ignored tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2023-09-26 12:12:08 +00:00
Yingwen
a6116bb866
feat(mito): Add cache manager ( #2488 )
...
* feat: add cache manager
* feat: add cache to reader builder
* feat: add AsyncFileReaderCache
* feat: Impl AsyncFileReaderCache
* chore: move moka dep to workspace
* feat: add moka cache to the manager
* feat: implement parquet meta cache
* test: test cache manager
* feat: consider vec size
* style: fix clippy
* test: fix config api test
* feat: divide cache
* test: test disabling meta cache
* test: fix config api test
* feat: remove meta cache if file is purged
2023-09-26 11:46:19 +00:00
LinFeng
621c6f371b
feat: limit grpc message size ( #2459 )
...
* feat: add two grpc config options
Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size
* test: add integration tests for message size limit
2023-09-22 02:07:46 +00:00
shuiyisong
20f4f7971a
refactor: remove source and location in snafu display ( #2428 )
...
* refactor: remove source pt 1
* refactor: remove source pt 2
* refactor: remove source pt 3
* refactor: remove location pt 1
* refactor: remove location pt 2
* chore: remove rustc files
* chore: fix error case
* chore: fix test
* chore: fix test
* chore: fix cr issue
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
---------
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com >
2023-09-21 02:55:24 +00:00
Ruihang Xia
34d6288945
feat: bring back sqlness and integration tests ( #2448 )
...
* enable integration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* disable sqlness region failover
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* enable sqlness in CI
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* sort unstable result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* set require_lease_before_startup to true
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix: fix inconsistent cache
* replace windows path chars
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* ignore some integration cases in windows
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* Revert "ignore some integration cases in windows"
This reverts commit 122478b7c1waynestxia@gmail.com >
* fix: fix close region bug in RegionHeartbeatResponseHandler
* disable failover tests
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: WenyXu <wenymedia@gmail.com >
2023-09-20 09:17:30 +00:00
dennis zhuang
342cc0a4c4
fix: compile error after updating protos ( #2435 )
2023-09-18 12:12:39 +00:00
Zhenchi
d6d46378a1
test: fix some integration tests ( #2432 )
...
* test: fix some integration tests
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* test: add timezone setting
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2023-09-18 10:52:14 +00:00
Zhenchi
3a39215f11
feat: migrate file engine from table to reigon ( #2365 )
...
* feat: migrate file engine from table to reigon
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* Update src/file-engine/src/engine.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* feat: specify ts index for file engine
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: handle time index for external table
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: some integration testsg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add file schema and table schema compatibility
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: compatible file schema to region schema
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* feat: add error msg
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: simplify close
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: implement set_writable
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: tests-integration compilation
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: address comments
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: zhongzc <zhongzc@zhongzcs-MacBook-Pro.local >
2023-09-18 08:02:43 +00:00
LFC
4b13c88752
fix: resolve more integration tests ( #2406 )
...
* fix: resolve more integration tests
* Update tests-integration/tests/http.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2023-09-15 09:43:16 +00:00
dennis zhuang
43e3c94fd1
refactor: catalog managers ( #2405 )
...
* feat: rename catalog::local to catalog::memory
* refactor: catalog managers
* chore: license header
2023-09-15 08:48:14 +00:00
Yingwen
a84a8ad04f
fix: alter table procedure panics while renaming table ( #2397 )
...
* fix: procedure panic on renaming table
* test: fix test_insert_and_select invalid arguments
* test: fix test_standalone_insert_and_query using wrong semantic type
* test: fix test_distributed_insert_delete_and_query semantic type
2023-09-14 11:50:00 +00:00
LFC
d08b05c963
fix: make test-integration able to compile ( #2384 )
...
* fix: make test-integration able to compile
* chore: fmt toml
---------
Co-authored-by: WenyXu <wenymedia@gmail.com >
2023-09-13 12:42:46 +00:00
JeremyHi
58d07e0e62
feat: v04 rm unused exprs ( #2285 )
...
* feat: rm compact and flush exprs
* refactor: continue to rm compact and flush
2023-09-12 07:57:15 -05:00
shuiyisong
f16f58266e
refactor: query_ctx from http middleware ( #2253 )
...
* chore: change userinfo to query_ctx in http handler
* chore: minor change
* chore: move prometheus http to http mod
* chore: fix uni test:
* chore: add back schema check
* chore: minor change
* chore: remove clone
2023-08-25 09:36:33 +00:00
Ning Sun
88247e4284
fix!: resolve residual issues with removing prometheus port ( #2227 )
...
* fix: resolve residual issues when removing prometheus port
* fix: remove prometheus from sample config as well
2023-08-23 01:49:11 +00:00
dennis zhuang
18fa0e01ed
feat: remove checkpoint_on_startup ( #2228 )
...
feat: update flushed manifest version when it is larger
2023-08-22 13:09:34 +00:00
niebayes
e5f4ca2dab
feat: streaming do_get ( #2171 )
...
* feat: rewrite do_get for streaming get flight data
* feat: rewrite do_get call stack but leave the async stream adapter not modified yet
* feat: rewrite the async stream adapter to accept greptime record batch stream
* fix: resolve some PR comments
* feat: rewrite tests to adapt to the streaming do_get
* feat: add unit tests for streaming do_get
* feat: rewrite timer metric of merge scan
* remove unhelpful unit tests for streaming do_get
* add a new metric timer for merge scan and fix some test errors
* rewrite mysql writer to write query results in a streaming manner
* fix: fix fmt errors
* fix: rewrite sqlness runner to take into account the streaming do_get
* fix: fix toml format errors
* fix: resolve some PR comments
* fix: resolve some PR comments
* fix: refactor do_get to increase readability
* fix: refactor mysql try_write_one to increase readability
2023-08-22 02:54:05 +00:00
Weny Xu
e1ce1d86a1
refactor: unite key serialization method ( #2195 )
2023-08-18 09:42:19 +00:00
