rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-16 21:10:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,495 Commits 133 Branches 252 Tags
8e9f2ffce42e2aa4fed1da166c0a649ed31c450d
Commit Graph

220 Commits

Author SHA1 Message Date
Lei, HUANG
19d2d77b41 fix: parse large timestamp (#2185) 
* feat: support parsing large timestamp values

* chore: update sqlness tests

* fix: tests

* fix: allow larger window
 2023-08-24 03:52:15 +00:00
dennis zhuang
18fa0e01ed feat: remove checkpoint_on_startup (#2228) 
feat: update flushed manifest version when it is larger
 2023-08-22 13:09:34 +00:00
Yingwen
cc3e198975 feat(mito): Implement operations like concat and sort for Batch (#2203) 
* feat: Implement slice and first/last timestamp for Batch

* feat(mito): implements sort/concat for Batch

* chore: fix typo

* chore: remove comments

* feat: sort and dedup

* test: test batch operations

* chore: cast enum to test op type

* test: test filter related api

* sytle: fix clippy

* docs: comment for slice

* chore: address CR comment

Don't return Option in get_timestamp()/get_sequence()
 2023-08-22 12:03:02 +00:00
Lei, HUANG
be1e13c713 feat(mito2): time series memtable (#2208) 
* feat: time series memtable

* feat: add some test

* fix: some clippy warnings

* chore: some rustdoc

* refactor: test

* fix: remove useless functions

* feat: add config for TimeSeriesMemtable

* chore: some optimize

* refactor: remove bucketing

* refactor: avoid cloing RegionMetadataRef across all Series; make initial_builder_capacity a const; sort batch only by timestamp and sequence
 2023-08-22 08:40:46 +00:00
Lei, HUANG
e924b44e83 refactor: KeyValues return ValueRef (#2170) 
* refactor: KeyValues return ValueRef

* 1. Change KeyValues returned value from pb value to ValueRef
2. Replace OpType/SemanticType with pb's OpType and SemanticType to avoid duplicated conversions.

* feat: define min value of OpType as a const

* fix: toml format
 2023-08-14 14:51:13 +00:00
LFC
5c19913a91 build: on windows (#2054) 
* build on windows

* rebase develop

* fix: resolve PR comments
 2023-08-10 08:08:37 +00:00
Vanish
57836e762b feat: truncate table in standalone mode (#2090) 
* feat: impl table procedure in standalone mode

* chore: remove useless changes

* test: add some tests

* Update src/table-procedure/src/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* CR

* Update src/datanode/src/sql/truncate_table.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: fmt

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-08 11:23:36 +00:00
Ning Sun
c39de9072f refactor: use workspace dependencies for internal modules (#2119) 
* refactor: use workspace dependencies for internal modules

* fix: resolve issue with mock module in datanode

* refactor: update test modules
 2023-08-08 11:02:34 +00:00
LFC
46fa3eb629 chore: upgrade rust toolchain to latest nightly (#2049) 
* chore: upgrade rust toolchain to latest nightly

* rebase develop

* update rust toolchain in ci
 2023-08-08 07:17:51 +00:00
Vanish
cf7e8c9142 feat: truncate region (#2097) 
* feat: impl truncate region

* test: test truncate region

* chore: typo

* refactor: table truncate

* chore: remove useless changes

* chore: reset version

* fix: wait for flush task to complete

* fix: clippy

* chore: remove useless changes

* CR

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/tests/truncate.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/region/writer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* CR

* Update src/storage/src/engine.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* Update src/storage/src/manifest/region.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2023-08-04 12:26:25 +00:00
shuiyisong
7efff2d704 fix: introduce taplo.toml and sort Cargo.toml (#2096) 
* fix: add taplo.toml

* fix: introduce taplo.toml & sort cargo.toml

* chore: remove option in ci too
 2023-08-04 06:44:45 +00:00
gongzhengyang
0eac56a442 chore: remove unused dependencies (#2055) 
Co-authored-by: gongzhengyang <gongzhengyang@bolean.com.cn>
 2023-08-01 07:43:03 +00:00
Lei, HUANG
81ea61ba43 fix: window inferer (#2033) 
* fix: window inferer

* chore: rename
 2023-07-26 02:18:19 +00:00
Yingwen
43bde82e28 test(storage): fix schedule_duplicate_tasks test (#1990) 
test: fix schedule_duplicate_tasks test
 2023-07-19 03:05:30 +00:00
Ben Baoyi
9ff7670adf refactor:remove common_error::prelude (#1946) 
* feat:Remove common_error::prelude

* fix merge error

* cr comment

* fix error
 2023-07-13 10:36:36 +08:00
LFC
4fdb6d2f21 refactor: remote catalog uses memory (#1926) 
* refactor: remote catalog uses memory

* rebase develop

* fix: resolve PR comments
 2023-07-12 09:33:33 +00:00
Lei, HUANG
a7ea3bbc16 feat: manual compact api (#1912) 
* merge develop

* chore: merge develop

* fix: some cr commentx

* fix: cr comments
 2023-07-11 04:00:39 +00:00
Yingwen
4ea8a78817 feat: dedup rows while flushing memtables (#1916) 
* test: enlarge meta client timeout

* feat: dedup on flush

* test: enlarge datanode clients timeout

* chore: fix typo
 2023-07-10 15:07:10 +08:00
Lei, HUANG
553530cff4 fix: immediately reschedule a compaction after compaction (#1882) 
* fix: immediately reschedule a compaction after compaction

* refactor: add WriterCompactRequest

* feat: reschedule compaction

* fix: only reschedule compaction when it's triggered by flush

* fix: remove max_files_in_l0

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
 2023-07-10 15:05:31 +08:00
Yingwen
f731193ddc refactor: Define RegionId as a new type (#1903) 
* refactor: Define RegionId as a new type

* chore: use into

* feat: custom debug print for region id

fix: test_show_create_table
 2023-07-07 21:26:03 +08:00
Lei, HUANG
3b6f70cde3 feat: initial twcs impl (#1851) 
* feat: initial twcs impl

* chore: rename SimplePicker to LeveledPicker

* rename some structs

* Remove Compaction strategy

* make compaction picker a trait object

* make compaction picker configurable for every region

* chore: add some test for ttl

* add some tests

* fix: some style issues in cr

* feat: enable twcs when creating tables

* feat: allow config time window when creating tables

* fix: some cr comments
 2023-07-04 16:42:27 +08:00
Yingwen
b8e92292d2 feat: Implement a new scan mode using a chain reader (#1857) 
* feat: add log

* feat: print more info

* feat: use chain reader

* fix: panic on getting first range

* fix: prev not updated

* fix: reverse readers and iter backward

* chore: don't print windows in log

* feat: consider memtable range

Also fix the issue that using incorrect comparision method to sort time
ranges.

* fix: merge memtable window with sst's

* feat: add use_chain_reader option

* feat: skip empty memtables

* chore: change log level

* fix: memtable range not ordered

* style: fix clippy

* chore: address review comments

* chore: print region id in log
 2023-07-04 16:01:34 +08:00
Vanish
563f6e05e2 feat: remove all the manifests in drop_region. (#1834) 
* feat: drop_region delete manifest file

* chore: remove redundant code

* chore: fmt

* chore: clippy

* chore: clippy

* feat: support delete_all in manifest.

* chore:CR

* test: test_drop_basic, test_drop_reopen

* chore: cr

* fix: typo

* chore: cr
 2023-06-30 17:42:11 +08:00
Lei, HUANG
559d1f73a2 feat: push all possible filters down to parquet exec (#1839) 
* feat: push all possible filters down to parquet exec

* fix: project

* test: add ut for DatafusionArrowPredicate

* fix: according to CR comments
 2023-06-28 20:14:37 +08:00
Lei, HUANG
f287d3115b chore: replace result assertions (#1840) 
* s/assert!\((.*)\.is_ok\(\)\);/\1.unwrap\(\);/g

* s/assert!\((.*)\.is_some\(\)\);/\1.unwrap\(\);/g
 2023-06-27 19:14:48 +08:00
LFC
fcff66e039 chore: deny unused results (#1825) 
* chore: deny unused results

* rebase
 2023-06-27 15:33:53 +08:00
Yingwen
5ab0747092 test(storage): wait task before checking scheduled task num (#1811) 2023-06-21 18:04:34 +08:00
Lei, HUANG
960b84262b fix: abort parquet writer (#1785) 
* fix: sst file size

* fix: avoid creating file when no row's been written

* chore: rename tests

* fix: some clippy issues

* fix: some cr comments
 2023-06-19 03:19:31 +00:00
Lei, HUANG
69854c07c5 fix: wait for compaction task to finish (#1783) 2023-06-16 16:45:06 +08:00
Lei, HUANG
fb35e09072 chore: fix compaction caused race condition (#1767) 
fix: unit tests. For real, this time.
 2023-06-13 21:03:09 +08:00
Lei, HUANG
0f1e061f24 fix: compile issue on develop and workaround to fix failing tests cau… (#1771) 
* fix: compile issue on develop and workaround to fix failing tests caused by logstore file lock

* Apply suggestions from code review

Co-authored-by: JeremyHi <jiachun_feng@proton.me>

---------

Co-authored-by: JeremyHi <jiachun_feng@proton.me>
 2023-06-13 07:30:16 +00:00
Lei, HUANG
7961de25ad feat: persist compaction time window (#1757) 
* feat: persist compaction time window

* refactor: remove useless compaction window fields

* chore: revert some useless change

* fix: some CR comments

* fix: comment out unstable sqlness test

* revert commented sqlness
 2023-06-13 10:15:42 +08:00
Lei, HUANG
f7d98e533b chore: fix compaction caused race condition (#1759) 
* fix: set max_files_in_l0 in unit tests to avoid compaction

* refactor: pass while EngineConfig

* fix: comment out unstable sqlness test

* revert commented sqlness
 2023-06-12 11:19:42 +00:00
Yingwen
0e937be3f5 fix(storage): Use region_write_buffer_size as default value (#1760) 2023-06-12 15:05:17 +08:00
Ruihang Xia
8c78368374 refactor: replace #[snafu(backtrace)] with Location (#1753) 
* remove snafu backtrace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clean up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-12 11:55:33 +08:00
Lei, HUANG
67c16dd631 feat: optimize some parquet writer parameter (#1758) 2023-06-12 11:46:45 +08:00
Lei, HUANG
ddcee052b2 fix: order by optimization (#1748) 
* add some debug log

* fix: use lazy parquet reader in MitoTable::scan_to_stream to avoid IO in plan stage

* fix: unit tests

* fix: order-by optimization

* add some tests

* fix: move metric names to metrics.rs

* fix: some cr comments
 2023-06-12 11:45:43 +08:00
dennis zhuang
f08f726bec test: s3 manifest (#1755) 
* feat: change default manifest options

* test: s3 manifest

* feat: revert checkpoint_margin to 10

* Update src/object-store/src/test_util.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
 2023-06-09 10:28:41 +00:00
Yingwen
5b8e54e60e feat: Add HTTP API for cpu profiling (#1694) 
* chore: print source error in mem-prof

* feat(common-pprof): add pprof crate

* feat(servers): Add pprof handler to router

refactor the mem_prof handler to avoid checking feature while
registering router

* feat(servers): pprof handler support different output type

* docs(common-pprof): Add readme

* feat(common-pprof): Build guard using code in pprof-rs's example

* feat(common-pprof): use prost

* feat: don't add timeout to perf api

* feat: add feature pprof

* feat: update readme

* test: fix tests

* feat: close region in TestBase

* feat(pprof): addres comments
 2023-06-07 15:25:16 +08:00
Lei, HUANG
f63ddb57c3 fix: parquet time range predicate panic (#1735) 
fix: parquet reader should use store schema to build time range predicate
 2023-06-06 19:11:45 +08:00
Yingwen
1b4976b077 feat: Adds some metrics for write path and flush (#1726) 
* feat: more metrics

* feat: Add preprocess elapsed

* chore(storage): rename metric

* test: fix tests
 2023-06-05 21:35:44 +08:00
LFC
5004cf6d9a feat: make grpc insert requests in a batch (#1687) 
* feat: make Prometheus remote write in a batch

* rebase

* fix: resolve PR comments

* fix: resolve PR comments

* fix: resolve PR comments
 2023-06-02 09:06:48 +00:00
Lei, HUANG
ccd666aa9b fix: avoid writing manifest and wal if no files are actually flushed (#1698) 
* fix: avoid writing manifest and wal if no files are actually flushed

* fix: simplify log
 2023-06-02 13:16:59 +08:00
Ruihang Xia
e5b6f8654a feat: optimizer rule to pass expected output ordering hint (#1675) 
* move type convertsion rule into optimizer dir

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement order_hint rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* it works!

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use column name instead

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* accomplish test case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update lock file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-06-02 03:43:51 +00:00
Weny Xu
0460f3ae30 test: add write test for region failover (#1673) 
* test: add write test for region failover

* test: add written data assertion after failover

* test: support more storage types
 2023-05-31 15:42:00 +08:00
Yingwen
9d179802b8 feat: Add a global TTL option for all tables (#1679) 
* feat: Add a global TTL option for all tables

* docs: update config examples

* chore: print start command and options when standalone/frontend starts
 2023-05-31 15:36:25 +08:00
Lei, HUANG
72b6bd11f7 feat: adapt window reader to order rules (#1671) 
* feat: adapt window reader to order rules

* fix: add asc sort test case
 2023-05-31 03:36:17 +00:00
Ruihang Xia
9e21632f23 fix: clippy warning (#1669) 
* fix: clippy warning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* restore the removed common sqlness cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2023-05-30 08:55:24 +08:00
Lei, HUANG
78fab08b51 feat: window inferer (#1648) 
* feat: window inferer

* doc: add some doc

* test: add a long missing unit test case for windowed reader

* add more tests

* fix: some CR comments
 2023-05-29 14:41:00 +08:00
Weny Xu
9f67ad8bce fix: fix doesn't release closed regions issue (#1596) 
* fix: fix close region issue

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* chore: apply suggestion from CR

* refactor: remove close method from Region trait

* chore: remove PartialEq from CloseTableResult
 2023-05-23 11:40:12 +08:00