Lei, HUANG
7183fa198c
refactor: make MergeTreeMemtable the default choice ( #3430 )
...
* refactor: make MergeTreeMemtable the default choice
* refactor: reformat
* chore: add doc to config
2024-03-05 10:00:08 +00:00
Lei, HUANG
02b18fbca1
feat: decode prom requests to grpc ( #3425 )
...
* hack: inline decode
* move to servers
* fix: samples lost
* add bench
* remove useless functions
* wip
* feat: remove object pools
* fix: minor issues
* fix: remove useless dep
* chore: rebase main
* format
* finish
* fix: format
* feat: introduce request pool
* try to fix license issue
* fix: clippy
* resolve comments
* fix:typo
* remove useless comments
2024-03-05 09:47:32 +00:00
shuiyisong
7b1c3503d0
fix: complete interceptors for all frontend entry ( #3428 )
2024-03-05 09:38:47 +00:00
WU Jingdi
53f2a5846c
feat: support tracing rule sampler ( #3405 )
...
* feat: support tracing rule sampler
* chore: simplify code
2024-03-05 15:40:02 +08:00
Yingwen
49157868f9
feat: Correct server metrics and add more metrics for scan ( #3426 )
...
* feat: drop timer on stream terminated
* refactor: combine metrics into a histogram vec
* refactor: frontend grpc metrics
* feat: add metrics middleware layer to grpc server
* refactor: move http metrics layer to metrics mod
* feat: bucket for grpc/http elapsed
* feat: remove duplicate metrics
* style: fix cilppy
* fix: incorrect bucket of promql series
* feat: more metrics for mito
* feat: convert cost
* test: fix metrics test
2024-03-04 10:15:10 +00:00
dennis zhuang
e6819412c5
refactor: show tables and show databases ( #3423 )
...
* refactor: show tables and show databases
* chore: clean code
2024-03-04 06:15:17 +00:00
JeremyHi
0edf1bbacc
feat: reduce a clone of string ( #3422 )
2024-03-03 08:09:17 +00:00
Eugene Tolbakov
8609977b52
feat: add verbose support for tql explain/analyze ( #3390 )
...
* feat: add verbose support for tql explain/analyze
* chore: apply clippy suggestions
* feat: add sqlness tests
* fix: adjust sqlness replace rules
* fix: address CR (move tql explain/analyze inside common folder)
* fix: address CR(improve comments to indicate that verbose is optional)
2024-03-02 11:18:22 +00:00
JeremyHi
2d975e4f22
feat: tableref cache ( #3420 )
...
* feat: tableref cache
* chore: minor refactor
* chore: avoid to string
* chore: change log level
* feat: add metrics for prometheus remote write decode
2024-03-02 07:37:31 +00:00
Kould
00cbbc97ae
feat: support Create Table ... Like ( #3372 )
...
* feat: support `Create Table ... Like`
* fix: `check_permission` for `Create Table ... Like`
* style: renaming `name` -> `table_name` & `target` -> `source_name` and make `Create Table ... Like` testcase more complicated
* rebase
* avoid _ fn
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
2024-03-02 06:34:13 +00:00
niebayes
7d30c2484b
fix: mitigate memory spike during startup ( #3418 )
...
* fix: fix memory spike during startup
* fix: allocate a region write ctx for each wal entry
2024-03-01 07:46:05 +00:00
Lei, HUANG
376409b857
feat: employ sparse key encoding for shard lookup ( #3410 )
...
* feat: employ short key encoding for shard lookup
* fix: license
* chore: simplify code
* refactor: only enable sparse encoding to speed lookup on metric engine
* fix: names
2024-03-01 06:22:15 +00:00
Ning Sun
d4a54a085b
feat: add configuration for tls watch option ( #3395 )
...
* feat: add configuration for tls watch option
* test: sleep longer to ensure async task run
* test: update config api integration test
* refactor: rename function
2024-03-01 03:49:54 +00:00
dennis zhuang
c1a370649e
fix: show table names not complete from information_schema ( #3417 )
2024-03-01 02:51:46 +00:00
JeremyHi
3cad9d989d
fix: partition region id ( #3414 )
2024-02-29 09:09:59 +00:00
JohnsonLee
a50025269f
feat: Support automatic DNS lookup for kafka bootstrap servers ( #3379 )
...
* feat: Support automatic DNS lookup for kafka bootstrap servers
* Revert "feat: Support automatic DNS lookup for kafka bootstrap servers"
This reverts commit 5baed7b01dniebayes@gmail.com >
* move more common code
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
Co-authored-by: tison <wander4096@gmail.com >
Co-authored-by: niebayes <niebayes@gmail.com >
2024-02-29 07:29:20 +00:00
JeremyHi
a3533c4ea0
feat: zero copy on split rows ( #3407 )
2024-02-28 13:27:52 +00:00
Lei, HUANG
3413fc0781
refactor: move some costly methods in DataBuffer::read out of read lock ( #3406 )
...
* refactor: move some costly methods in DataBuffer::read out of read lock
* refactor: also replace ShardReader with ShardReaderBuilder
2024-02-28 12:22:44 +00:00
tison
dc205a2c5d
feat: enable ArrowFlight compression ( #3403 )
...
* feat: enable ArrowFlight compression
Signed-off-by: tison <wander4096@gmail.com >
* turn on features
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-02-28 08:55:44 +00:00
Lei, HUANG
a0a8e8c587
fix: some read metrics ( #3404 )
...
* fix: some read metrics
* chore: fix some metrics
* fix
2024-02-28 08:47:49 +00:00
Zhenchi
c3c80b92c8
feat(index): measure memory usage in global instead of single-column and add metrics ( #3383 )
...
* feat(index): measure memory usage in global instead of single-column and add metrics
* feat: add leading zeros to streamline memory usage
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: fmt
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: remove println
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
---------
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com >
2024-02-28 06:49:24 +00:00
Weny Xu
a8cbec824c
refactor: refactor TableRouteManager ( #3392 )
...
* feat: introduce TableRouteStorage
* refactor: remove get & batch_get in TableRouteManager
* refactor: move txn related fn to TableRouteStorage
* chore: apply suggestions from CR
* chore(codecov): ingore tests-integration dir
2024-02-28 06:18:09 +00:00
Lei, HUANG
7942b8fae9
chore: add metris for memtable read path ( #3397 )
...
* chore: add metris for read path
* chore: add more metrics
2024-02-28 03:37:19 +00:00
Yingwen
b97f957489
feat: Use a partition level map to look up pk index ( #3400 )
...
* feat: partition level map
* test: test shard and builder
* fix: do not use pk index from shard builder
* feat: add multi key test
* fix: freeze shard before finding pk in shards
2024-02-28 03:17:09 +00:00
dennis zhuang
4b36c285f1
feat: flush or compact table and region functions ( #3363 )
...
* feat: adds Requester to process table flush and compaction request
* feat: admin_fn macros for administration functions
* test: add query result
* feat: impl flush_region, flush_table, compact_region, and flush_region functions
* docs: add Arguments to admin_fn macro
* chore: apply suggestion
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
* chore: apply suggestion
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
* fix: group_requests_by_peer and adds log
* Update src/common/macro/src/admin_fn.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
* feat: adds todo for spawan thread
* feat: rebase with main
---------
Co-authored-by: Zhenchi <zhongzc_arch@outlook.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-02-27 08:57:38 +00:00
discord9
dbb1ce1a9b
feat(flow): impl for MapFilterProject ( #3359 )
...
* feat: mfp impls
* fix: after rebase
* test: temporal filter mfp
* refactor: more comments&test
* test: permute
* fix: check input len when eval
* refactor: err handle&docs: more explain graph
* docs: better flowchart map,filter,project
* refactor: visit_* falliable
* chore: better temp lint allow
* fix: permute partially
* chore: remove duplicated checks
* docs: more explain&tests for clarity
* refactor: use ensure! instead
2024-02-27 08:13:55 +00:00
Ruihang Xia
3544c9334c
feat!: new partition grammar - parser part ( #3347 )
...
* parser part
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix test in sql
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* comment out and ignore some logic
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update sqlness cases
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update region migration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* temporary disable region migration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* allow dead code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* update integration test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-02-27 07:20:16 +00:00
Lei, HUANG
492a00969d
feat: enable zstd compression and encodings in merge tree data part ( #3380 )
...
* feat: enable zstd compression in merge tree data part to save memory
* feat: also enable customized column encoding in DataPartEncoder
2024-02-27 06:54:56 +00:00
Yingwen
206666bff6
feat: Implement partition eviction and only add value size to write buffer size ( #3393 )
...
* feat: track key bytes in dict
* chore: done allocating on finish
* feat: evict keys
* chore: do not add to write buffer
* chore: only count value bytes
* fix: reset key bytes
* feat: remove write buffer manager from shards
* feat: change dict size compute method
* chore: adjust dictionary size by os memory
2024-02-27 06:28:57 +00:00
Weny Xu
7453d9779d
fix: throw errors instead of panic ( #3391 )
...
* fix: throw errors instead of panic
* chore: apply suggestions from CR
2024-02-27 03:46:12 +00:00
dimbtp
b1e290f959
fix: range fix in modulo function tests ( #3389 )
...
fix: range fix for modulo tests
2024-02-26 15:50:23 +00:00
Ning Sun
3887d207b6
feat: make tls certificates/keys reloadable (part 1) ( #3335 )
...
* feat: make tls certificates/keys reloadable (part 1)
* feat: add notify watcher for cert/key files
* test: add unit test for watcher
* fix: correct usage of watcher
* fix: skip watch when tls disabled
2024-02-26 09:37:54 +00:00
Ruihang Xia
ce397ebcc6
feat: change how region id maps to region worker ( #3384 )
...
* feat: change how region id maps to region worker
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* add overflow test
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-02-26 08:42:29 +00:00
Yingwen
26011ed0b6
fix: resets dict builder keys counter and avoid unnecessary pruning ( #3386 )
...
* fix: dict builder resets num_keys on finish
* feat: skip empty shard and builder
* feat: avoid pruning if possible
Implementations:
- Apply all filters on the partition column
- If no filter to prune, skip decoding keys
2024-02-26 08:24:46 +00:00
Lei, HUANG
8087822ab2
refactor: change the receivers of merge tree components ( #3378 )
...
* refactor: change the receivers of Shard::read/DataBuffer::read/DataParts::read to &self instead of &mut self
* refactor: remove allow(dead_code) in merge tree
2024-02-26 06:50:55 +00:00
Yingwen
e481f073f5
feat: Implement dedup for the new memtable and expose the config ( #3377 )
...
* fix: KeyValues num_fields() is incorrect
* chore: fix warnings
* feat: support dedup
* feat: allow using the new memtable
* feat: serde default for config
* fix: resets pk index after finishing a dict
2024-02-25 13:06:01 +00:00
Lei, HUANG
606309f49a
fix: remove unused imports in memtable_util.rs ( #3376 )
2024-02-25 09:23:28 +00:00
Yingwen
8059b95e37
feat: Implement iter for the new memtable ( #3373 )
...
* chore: read shard builder
* chore: reuse pk weights
* chore: prune key
* chore: shard reader wip
* refactor: shard builder DataBatch
* feat: merge shard readers
* feat: return shard id in shard readers
* feat: impl partition reader
* chore: impl partition read
* feat: impl iter tree
* chore: save last yield pk id
* style: fix clippy
* refactor: rename ShardReaderImpl to ShardReader
* chore: address CR comment
2024-02-25 07:42:16 +00:00
Lei, HUANG
afe4633320
feat: merge tree dedup reader ( #3375 )
...
* feat: add dedup option to merge tree component
* feat: impl dedup reader for shard reader
* refactor: DedupReader::new to DedupReader::try_new
* refactor: remove DedupReader::current_key field
* fix: some cr comments
* fix: fmt
* fix: remove shard_id method from DedupSource
2024-02-24 13:50:49 +00:00
Yingwen
abbfd23d4b
feat: Add freeze and fork method to the memtable ( #3374 )
...
* feat: add fork method to the memtable
* feat: allow mark immutable returns result
* feat: use fork to create the mutable memtable
* feat: remove memtable builder from freeze
* chore: warninigs
* fix: inspect error
* feat: iter returns result
* chore: maintains memtable id in region
* chore: update comment
* fix: remove region status if failed to freeze a memtable
* chroe: update comment
* chore: iter should not require sync
* chore: implement freeze and fork for the new memtable
2024-02-24 12:11:16 +00:00
Yingwen
1df64f294b
refactor: Remove Item from merger's Node trait ( #3371 )
...
* refactor: data reader returns reference to data batch
* refactor: use range to create merger
* chore: Reference RecordBatch in DataBatch
* fix: top node not read if no next node
* refactor: move timestamp_array_to_i64_slice to data mod
* style: fix cilppy
* chore: derive copy for DataBatch
* chore: address CR comments
2024-02-24 07:19:48 +00:00
LFC
a6564e72b4
fix: treat "0" and "1" as valid boolean values. ( #3370 )
...
* Treat "0" and "1" as valid boolean values.
* Update src/sql/src/statements.rs
Co-authored-by: tison <wander4096@gmail.com >
* Fix tests.
---------
Co-authored-by: tison <wander4096@gmail.com >
2024-02-23 14:34:27 +00:00
Lei, HUANG
1f1d1b4f57
feat: distinguish between different read paths ( #3369 )
...
* feat: distinguish between different read paths
* fix: reformat code
2024-02-23 12:40:39 +00:00
Yingwen
b144836935
feat: Implement write and fork for the new memtable ( #3357 )
...
* feat: write to a shard or a shard builder
* feat: freeze and fork for partition and shards
* chore: shard builder
* chore: change dict reader to support random access
* test: test write shard
* test: test write
* test: test memtable
* feat: add new and write_row to DataParts
* refactor: partition freeze shards
* refactor: write_with_pk_id
* style: fix clippy
* chore: add methods to get pk weights
* chroe: fix compiler errors
2024-02-23 07:20:55 +00:00
Lei, HUANG
90e9b69035
feat: impl merge reader for DataParts ( #3361 )
...
* feat: impl merge reader for DataParts
* fix: fmt
* fix: sort rows with pk and ts according to sequnce desc
* fix: remove pk weight as pk index are already replace by weights
* fix: format
* fix: some cr comments
* fix: some cr comments
* refactor: simply trait's associated types
* fix: some cr comments
2024-02-23 06:07:55 +00:00
LFC
2035e7bf4c
refactor: set the actual bound port in server handler ( #3353 )
...
* refactor: set the actual bound port so we can use port 0 in testing
* Update src/servers/src/server.rs
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fmt
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-02-23 02:49:11 +00:00
Ruihang Xia
7341f23019
feat: skip filling NULL for put and delete requests ( #3364 )
...
* feat: optimize for sparse data
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* remove old structures
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-02-22 14:30:43 +00:00
tison
41ee0cdd5a
build(deps): Upgrade opensrv to 0.7.0 ( #3362 )
...
* build(deps): Upgrade opensrv to 0.7.0
Signed-off-by: tison <wander4096@gmail.com >
* workaround X is not X by casting
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-02-22 13:11:26 +00:00
Kould
578dd8f87a
feat: add isnull function ( #3360 )
...
* code fmt
* feat: add isnull function
* feat: add isnull function
2024-02-22 12:41:25 +00:00
Weny Xu
1dc4fec662
refactor: allocate table ids in the procedure ( #3293 )
...
* refactor: refactor the create logical tables
* test(create_logical_tables): add tests for on_prepare
* test(create_logical_tables): add tests for on_create_metadata
* refactor: rename to create_logical_tables_metadata
* chore: fmt toml
* chore: apply suggestions from CR
2024-02-22 10:53:28 +00:00
