rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-17 21:40:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,645 Commits 133 Branches 252 Tags
f16ce3ca2749f2135cd225d2cefc49e92209daec
Commit Graph

276 Commits

Author SHA1 Message Date
Ruihang Xia
115c74791d build(deps): bump snafu to 0.8 (#3911) 
* change Cargo.toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias in script engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-10 13:36:25 +00:00
Yingwen
5a0629eaa0 feat: Parquet reader builder supports building multiple ranges to read (#3841) 
* chore: change `&mut self` to `&self`

* feat: define partition and partition context

* refactor: move precise_filter to PartitionContext

* feat: filter  wip

* feat: compute projection and fields in format

* feat: use RowGroupReader to implement ParquetReader

* fix: use expected meta to get column id for filters

* feat: partition returns row group reader

* style: fix clippy

* feat: add build partitions method

* docs: comment

* refactor: rename Partition to FileRange

* chore: address CR comments

* feat: avoid allocating column ids while constructing ReadFormat
 2024-05-10 07:39:38 +00:00
Yingwen
9d8f72d611 fix: add data type to vector cache key (#3876) 
* test: test for null tag

* test: sqlness test

* fix: add type to vector cache key

* test: update sqlness test
 2024-05-08 06:30:28 +00:00
Ruihang Xia
530353785c refactor: remove re-export from logging (#3865) 
* refactor: remove re-export from logging

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge problem

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* run formatter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-06 13:26:01 +00:00
Lei, HUANG
f3b68253c2 chore: remove one slice copy in wal encoding (#3861) 
* chore: remove one slice copy in wal encoding

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: cr comments

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2024-05-06 10:26:40 +00:00
tison
695746193b ci: reduce workflow files by merging actions (#3848) 
* ci: merge license header checker into dev ci

Signed-off-by: tison <wander4096@gmail.com>

* ci: merge nightly-funtional-tests.yml into nightly-ci.yml

Signed-off-by: tison <wander4096@gmail.com>

* fix typos

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-05-01 04:44:00 +00:00
Yingwen
77fc1e6de0 fix: prune row groups correctly for columns with the same name (#3802) 
* test: add prune test case

* fix: use latest region metadata to get column id

* test: sort output
 2024-04-26 06:52:23 +00:00
Lei, HUANG
8d229dda98 chore: always obsolete wal to avoid discontinuous log entries (#3789) 2024-04-24 07:08:01 +00:00
Kould
42e7403fcc feat: support different types for CompatReader (#3745) 
* feat: support different types for `CompatReader`

* chore: only compare whether we need: (data_type)

* fix: optimize code based on review suggestions

- add unit test `test_safe_cast_to_null` to test safely cast
- add DataType to projected_fields
- remove TODO

* fix: assert_eq fail on `projection.rs`

* style: codefmt

* style: fix the code based on review suggestions
 2024-04-24 06:27:52 +00:00
Yingwen
4685b59ef1 feat: write manifests in background tasks (#3709) 
* chore: truncate wip

* feat: truncate and edit write manifest in background

* refactor: wrap in manifest context

* feat: alter write manifest in background

* chore: fix compiler errors

* feat: flush update manifest in background

* feat: compaction update manifest in background

* feat: set dropping state

* feat: reset drop state

* feat: check state before updating manifest

* test: fix compaction test

* refactor: rename method

* chore: update comment

* chore: discard state guard

* refactor: use atomic cell to store state enum

* chore: fix clippy

* chore: update toml

* chore: remove unused type alias

* feat: check state after writing manifest

* chore: address CR comments

* chore: change status code

* chore: Update src/mito2/src/region.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

* fix: executes applier

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-04-24 03:09:48 +00:00
Lei, HUANG
778e195f07 fix: do not remove deletion markers when window time range overlaps (#3773) 
* fix: do not remove deletion markers when window time range overlaps

* chore: fix some minor issues; add compaction test

* chore: add more test

* fix: nitpick master's nitpick
 2024-04-23 08:05:16 +00:00
Ruihang Xia
0c5f4801b7 build: update toolchain to nightly-2024-04-18 (#3740) 
* chore: update toolchain to nightly-2024-04-17

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update fuzz test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update to nightly-2024-04-18

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* avoid unnecessary allow clippy attrs

Signed-off-by: tison <wander4096@gmail.com>

* help the compiler find the clone is unnecessary and make clippy happy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-04-19 05:42:34 +00:00
LFC
314f2704d4 build(deps): update datafusion to latest and arrow to 51.0 (#3661) 
* chore: update datafusion

* update sqlness case of time.sql

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: adjust range query partition

* fix: hisogram incorrect result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: ignore filter pushdown temporarily

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update limit sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: histogram with wrong distribution

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update negative ordinal sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: bump df to cd7a00b

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* ignore test_range_filter

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix promql exec panic

* fix "select count(*)" exec error

* re-enable the "test_range_filter" test since the filter push down seems not necessary to be removed

* fix: range query schema error

* update sqlness results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* resolve conflicts

* update datafusion, again

* fix pyo3 compile error, and update some sqlness results

* update decimal sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: promql literal

* fix udaf tests

* fix filter pushdown sqlness tests

* fix?: test_cast

* fix: rspy test fail due to datafusion `sin` signature change

* rebase main to see if there are any failed tests

* debug ci

* debug ci

* debug ci

* enforce input partition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* debug ci

* fix ci

* fix ci

* debug ci

* debug ci

* debug ci

* fix sqlness

* feat: do not return error while creating a filter

* chore: remove array from error

* chore: replace todo with unimplemented

* Update src/flow/clippy.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: WUJingdi <taylor-lagrange@qq.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-04-18 12:07:18 +00:00
tison
50ae4dc174 refactor: merge RegionHandleResult into RegionHandleResponse (#3721) 
* refactor: merge RegionHandleResult into RegionHandleResponse

Signed-off-by: tison <wander4096@gmail.com>

* RegionResponse to api::region

Signed-off-by: tison <wander4096@gmail.com>

* order

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-04-17 10:03:20 +00:00
Lei, HUANG
5af87baeb0 feat: add filter_deleted option to avoid removing deletion markers (#3707) 
* feat: add `filter_deleted` scan option to avoid removing deletion markers.

* refactor: move sort_batches_and_print to test_util
 2024-04-16 06:34:41 +00:00
Yingwen
2f4726f7b5 refactor: Move manifest manager lock to MitoRegion (#3689) 
* feat: remove manager inner wip

* feat: put manifest lock in region

* feat: don't update manifest if manager is stopped

* chore: address CR comments
 2024-04-15 05:48:25 +00:00
Ruihang Xia
aab7367804 feat: try get pk values from cache when applying predicate to parquet (#3286) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-04-09 12:53:38 +00:00
tison
2c0c7759ee feat: add checksum for checkpoint data (#3651) 
* feat: add checksum for checkpoint data

Signed-off-by: tison <wander4096@gmail.com>

* add test

Signed-off-by: tison <wander4096@gmail.com>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* fix: checksum should calculate on uncompressed data

Signed-off-by: tison <wander4096@gmail.com>

* address comments

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-04-09 08:32:24 +00:00
Yingwen
3e1a125732 feat: add append mode to table options (#3624) 
* feat: add append mode to table options

* test: add append mode test

* test: rename test tables

* chore: Add delete test for append mode
 2024-04-08 13:42:58 +00:00
Yingwen
89db8c18c8 feat: Add timers to more mito methods (#3659) 
* feat: add timers for more mito methods

* refactor: combine methods to get type name
 2024-04-08 05:53:34 +00:00
LFC
aa0af6135d chore: add manifest related metrics (#3634) 
* chore: add two manifest related metrics

* Update src/mito2/src/manifest/manager.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/mito2/src/metrics.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: resolve PR comments

* update cargo lock

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-04-08 05:53:08 +00:00
Lei, HUANG
ddeb73fbb7 fix: mistakely removes compaction inputs on failure (#3635) 
* fix: mistakely removes compaction inputs on failure

* test: add test for compaction failure

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
 2024-04-03 11:54:20 +00:00
Yingwen
bfd32571d9 fix: run purge jobs in another scheduler (#3621) 2024-04-01 03:18:14 +00:00
Yingwen
ffbb132f27 feat: Implement an unordered scanner for append mode (#3598) 
* feat: ScanInput

* refactor: seq scan use scan input

* chore: implement unordered scan

* feat: use unordered scan for append table

* fix: unordered scan panic

* docs: update mermaid

* chore: address comment

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2024-03-29 07:25:35 +00:00
Yingwen
922b1a9b66 feat: Implement append mode for a region (#3558) 
* feat: add dedup option to merge reader

* test: test merger

* feat: append mode option

* feat: implement append mode for regions

* feat: only allow put under append mode

* feat: always create builder

* test: test append mode

* style: fix clippy

* test: trigger compaction

* chore: fix compiler errors
 2024-03-27 03:21:22 +00:00
dimbtp
dd18d8c97b build(deps): remove some unused dependencies (#3582) 
* build(deps): remove some unused dependencies

* add `arc-swap` dependency back
 2024-03-26 12:48:28 +00:00
Ruihang Xia
74565151e9 fix: update pk_cache in compat reader (#3576) 
* fix: update pk_cache in compat reader

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update document

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* avoid mysterious bug

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-03-26 08:31:00 +00:00
Ruihang Xia
2b2fd80bf4 feat: return new added columns in region server's extension response (#3533) 
* feat: adapt the new proto response

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* write columns to extension

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use physical column's schema

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort logical columns by name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* return physical table's column

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/common/meta/src/datanode_manager.rs

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

* implement sort column logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* proxy create table procedure to create logical table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add unit test for sort_columns

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2024-03-23 09:31:16 +00:00
Lei, HUANG
1f0fc40287 fix: performance degradation caused by config change (#3556) 2024-03-21 12:23:52 +00:00
Yingwen
39b69f1e3b refactor!: Renames the new memtable to PartitionTreeMemtable (#3547) 
* refactor: rename mod merge_tree to partition_tree

* refactor: rename merge_tree

* refactor: change merge tree comment

* refactor: rename merge tree struct

* refactor: memtable options
 2024-03-20 06:40:41 +00:00
Lei, HUANG
ddbcff68dd feat: support append-only mode in time-series memtable (#3540) 
* feat: support append-only mode in time-series memtable

* fix: rename sort_and_dedup to sort
 2024-03-19 20:37:54 +00:00
Yingwen
641592644d feat: support per table memtable options (#3524) 
* feat: add memtable builder to region

* refactor: rename memtable_builder in worker to default_memtable_builder

* fix: return error instead of using default compaction options

Support deserializing memtable and compaction options from the option
map

* feat: optional memtable options

* feat: add MemtableBuilderProvider to create builders

* feat: change default memtable and skip deserializing dedup

* chore: update test and comment

* chore: test invalid type

* feat: metric engine use new memtable manually

* feat: expose more memtable configs

* feat: add memtable options to valid option list

* test: add test

* test: sqlness test

* chore: serde workspace

* chore: remove comments
 2024-03-19 08:50:10 +00:00
Yingwen
74862f8c3f feat(mito): Checks whether a region should flush periodically (#3459) 
* feat: handle flush periodically

* chore: call periodical method in loop

* feat: check periodical tasks on channel timeout

* refactor: use time provider to get time

Mock a time provider to test auto flush

* chore: fix typos

* refactor: rename mock time provider

* style: fix cilppy

* chore: address comment
 2024-03-15 06:41:28 +00:00
Yingwen
8ca9e01455 feat: Partition memtables by time if compaction window is provided (#3501) 
* feat: define time partitions

* feat: adapt time partitions to version

* feat: implement non write methods

* feat: add write one to memtable

* feat: implement write

* chore: fix warning

* fix: inner not set

* refactor: add collect_iter_timestamps

* test: test partitions

* chore: debug log

* chore: fix typos

* chore: log memtable id

* fix: empty check

* chore: log total parts

* chore: update comments
 2024-03-14 11:13:01 +00:00
Yingwen
7c895e2605 perf: more benchmarks for memtables (#3491) 
* chore: remove duplicate bench

* refactor: rename bench

* perf: add full scan bench for memtable

* feat: filter bench and add time series to bench group

* chore: comment

* refactor: rename

* style: fix clippy
 2024-03-12 12:02:58 +00:00
Yingwen
9aa8f756ab fix: allow passing extra table options (#3484) 
* fix: do not check options in parser

* test: fix tests

* test: fix sqlness

* test: add sqlness test

* chore: log options

* chore: must specify compaction type

* feat: validate option key

* feat: add option key validation back
 2024-03-12 07:03:52 +00:00
Yingwen
06dcd0f6ed fix: freeze data buffer in shard (#3468) 
* feat: call freeze if the active data buffer in a shard is full

* chore: more metrics

* chore: print metrics

* chore: enlarge freeze threshold

* test: test freeze

* test: fix config test
 2024-03-11 14:51:06 +00:00
gcmutator
21ff3620be chore: remove repetitive words (#3469) 
remove repetitive words

Signed-off-by: gcmutator <329964069@qq.com>
 2024-03-09 04:18:47 +00:00
Yingwen
3ee53360ee perf: Reduce decode overhead during pruning keys in the memtable (#3415) 
* feat: reuse value buf

* feat: skip values to decode

* feat: prune shard

chore: fix compiler errors

refactor: shard prune metrics

* fix: panic on DedupReader::try_new

* fix: prune after next

* chore: num parts metrics

* feat: metrics and logs

* chore: data build cost

* chore: more logs

* feat: cache skip result

* chore: todo

* fix: index out of bound

* test: test codec

* fix: invalid offsets

* fix: skip binary

* fix: offset buffer reuse

* chore: comment

* test: test memtable filter

* style: fix clippy

* chore: fix compiler error
 2024-03-08 02:54:00 +00:00
Lei, HUANG
7183fa198c refactor: make MergeTreeMemtable the default choice (#3430) 
* refactor: make MergeTreeMemtable the default choice

* refactor: reformat

* chore: add doc to config
 2024-03-05 10:00:08 +00:00
Yingwen
49157868f9 feat: Correct server metrics and add more metrics for scan (#3426) 
* feat: drop timer on stream terminated

* refactor: combine metrics into a histogram vec

* refactor: frontend grpc metrics

* feat: add metrics middleware layer to grpc server

* refactor: move http metrics layer to metrics mod

* feat: bucket for grpc/http elapsed

* feat: remove duplicate metrics

* style: fix cilppy

* fix: incorrect bucket of promql series

* feat: more metrics for mito

* feat: convert cost

* test: fix metrics test
 2024-03-04 10:15:10 +00:00
niebayes
7d30c2484b fix: mitigate memory spike during startup (#3418) 
* fix: fix memory spike during startup

* fix: allocate a region write ctx for each wal entry
 2024-03-01 07:46:05 +00:00
Lei, HUANG
376409b857 feat: employ sparse key encoding for shard lookup (#3410) 
* feat: employ short key encoding for shard lookup

* fix: license

* chore: simplify code

* refactor: only enable sparse encoding to speed lookup on metric engine

* fix: names
 2024-03-01 06:22:15 +00:00
Lei, HUANG
3413fc0781 refactor: move some costly methods in DataBuffer::read out of read lock (#3406) 
* refactor: move some costly methods in DataBuffer::read out of read lock

* refactor: also replace ShardReader with ShardReaderBuilder
 2024-02-28 12:22:44 +00:00
Lei, HUANG
a0a8e8c587 fix: some read metrics (#3404) 
* fix: some read metrics

* chore: fix some metrics

* fix
 2024-02-28 08:47:49 +00:00
Zhenchi
c3c80b92c8 feat(index): measure memory usage in global instead of single-column and add metrics (#3383) 
* feat(index): measure memory usage in global instead of single-column and add metrics

* feat: add leading zeros to streamline memory usage

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove println

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-02-28 06:49:24 +00:00
Lei, HUANG
7942b8fae9 chore: add metris for memtable read path (#3397) 
* chore: add metris for read path

* chore: add more metrics
 2024-02-28 03:37:19 +00:00
Yingwen
b97f957489 feat: Use a partition level map to look up pk index (#3400) 
* feat: partition level map

* test: test shard and builder

* fix: do not use pk index from shard builder

* feat: add multi key test

* fix: freeze shard before finding pk in shards
 2024-02-28 03:17:09 +00:00
Lei, HUANG
492a00969d feat: enable zstd compression and encodings in merge tree data part (#3380) 
* feat: enable zstd compression in merge tree data part to save memory

* feat: also enable customized column encoding in DataPartEncoder
 2024-02-27 06:54:56 +00:00
Yingwen
206666bff6 feat: Implement partition eviction and only add value size to write buffer size (#3393) 
* feat: track key bytes in dict

* chore: done allocating on finish

* feat: evict keys

* chore: do not add to write buffer

* chore: only count value bytes

* fix: reset key bytes

* feat: remove write buffer manager from shards

* feat: change dict size compute method

* chore: adjust dictionary size by os memory
 2024-02-27 06:28:57 +00:00