rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-14 17:23:09 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,749 Commits 103 Branches 227 Tags
c2218f8be80e2159fcdc6f0cc239d2ab6491d8fe
Commit Graph

2256 Commits

Author SHA1 Message Date
Ruihang Xia
c2218f8be8 build(deps): bump datafusion 20240528 (#4061) 
* build(deps): bump datafusion 20240528

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* another update

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update expected sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix first/last value

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reformat comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix remaining errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert toml format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 feature

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dead code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* format file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-06-01 14:03:00 +00:00
Ruihang Xia
45fee948e9 fix: display error in correct format (#4082) 
* fix: display error in correct format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add address to RegionServer error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-31 09:25:14 +00:00
discord9
ea49f8a5c4 feat(flow): make write path faster with shared lock (#4073) 
* feat(WIP): make write faster

* feat: read lock on fast path

* chore: per review
 2024-05-31 06:50:22 +00:00
Jeremyhi
43afea1a9d refactor!: remove the tableid in ddl response since tableids is enough (#4080) 
* refactor: remove the tableid in ddl response since tableids is enough

* chore: upgrade proto
 2024-05-31 06:41:53 +00:00
Weny Xu
fcfcf86385 feat: implement WalEntryDistributor, WalEntryReciver (#4031) 
* feat: implement the `WalEntryDistributor` and `WalEntryReceiver`

* test: add tests for `WalEntryDistributor`

* refactor: use bounded channel

* chore: apply suggestions from CR
 2024-05-31 03:03:38 +00:00
Jeremyhi
26b112ab57 refactor: remove upgrade cli tool (#4077) 2024-05-31 00:40:27 +00:00
dennis zhuang
24612f62dd feat: querying from view works (#3952) 
* feat: querying from view works

* feat: use MemoryCatalogProviderList instead of DummyCatalogList

* refactor: revert src/query/src/dummy_catalog.rs

* chore: clean code

* fix: make clippy happy

* fix: toml format

* fix: sqlness

* fix: forgot files

* fix: make sqlness happy

* test: table source, serializer and decoder

* fix: fail to decode plan because of invalid table names

* test: adds more sqlness test for view

* chore: remove unused errors

* fix: comments

* fix: typo

* fix: invalidate view info cache after creating view successfully

* chore: apply suggestion

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: apply suggestion

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: compile error after rebeasing

* chore: style

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* fix: don't export table_name in common-meta

* chore: change ViewInfo::new signature

* docs: leave a TODO for optimize param

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-30 21:45:56 +00:00
Ruihang Xia
85a231850d fix: add tailing separator to prefix (#4078) 
* fix: add tailing separator to prefix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* project select result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-30 15:29:26 +00:00
Jeremyhi
8b6596faa0 feat: avoid some cloning when mirror requests to flownode (#4068) 
* feat: some refactor mirror requests to flownode

* feat: use spawn_bg to avoid impact front-ground write

* feat: add mirror row count metric
 2024-05-30 07:29:13 +00:00
Weny Xu
eab309ff7e fix: avoid acquiring lock during reading stats (#4070) 
* fix: avoid acquiring lock during reading stats

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-05-30 07:08:04 +00:00
discord9
7de336f087 fix(flow): correctness bugs (#4018) 
* fix: optional args of tumble

* fix(WIP): choose

* feat: rename default ts to GREPTIME_TIMESTAMP

* fix: default timestamp name

* fix: reorder write requests

* fix: expire state

* fix: test of tumble

* fix: send buf clear

* fix: ts<start time correct window

* fix: window_start when ts<start_time

* mend

* fix: range begin>range end

* refactor: per reviews

* feat!: ts placeholder rename to __ts_placeholder

* refactor: better condition

* tests(WIP): func sig choose

* tests(WIP): tumble func

* feat: make run_available optional blocking

* tests: tumble transform

* chore: clippy

* fix?: lagged missing data

* fix: flow source break on empty chnl
 2024-05-30 03:49:11 +00:00
Weny Xu
6e9a9dc333 refactor(log_store): remove associated type Namespace and Entry in LogStore (#4038) 
* refactor(log_store): remove associated type `Namespace` in `LogStore`

* fix(test): filter entries

* refactor: ignore incomplete parts

* refactor: simplify `RawEntryReaderFilter`

* chore: avoid cloning

* test: add tests for `maybe_emit_entry`

* refactor: remove `Namespace` trait and rename `LogStoreNamespace` to `Namespace`

* chore: apply suggestions from CR

* refactor: refine `entry` method signature

* feat: ingore  any potential incomplete parts

* refactor: rename `namespace` to `provider`

* chore: add debug assertion

* refactor: associated type `Entry` in `LogStore`

* refactor: renamse `namespace` to `provider`

* refactor: remove unwrap

* refactor: let `remaining_entries` return a optional vector

* test: add basic tests for kafka logstore

* refactor: move `append` method under `cfg(test)`

* refactor: rename `RawEntry` to `Entry`

* refactor: rename `CorruptedLogEntry` to `CorruptedEntry`

* test: add tests for handling corrupted raw entry stream

* refactor: rename `ns` to `provider`

* refactor: remove `entry_stream.rs` file

* chore: remove unused code

* chore: update comments

* chore: apply suggestions from CR

* chore: update comments

* chore: apply suggestions from CR

* chore: remove Deref

* chore: add comments

* fix: ignores tail corrupted data

* chore: add comments

* fix: add `MIN_BATCH_SIZE` limit
 2024-05-29 13:44:01 +00:00
Yingwen
848bd7e553 feat: Implements row group level parallel unordered scanner (#3992) 
* feat: unordered scanner

* feat: support compat

* chore: update debug print

fix: missing ranges in scan parts

* fix: ensure chunk size > 0

* fix: parallel is disabled if there is only one file and memtable

* chore: reader metrics

* chore: remove todo

* refactor: add ScanPartBuilder trait

* chore: pass file meta to the part builder

* chore: make part builder private

* docs: update comment

* chore: remove meta()

* refactor: only prune file ranges in ScanInput

replaces ScanPartBuilder with FileRangeCollector which only collect file
ranges

* chore: address typo

* fix: panic when no partition

* feat: Postpone part distribution

* chore: handle empty partition in mito

* style: fix clippy
 2024-05-29 11:06:08 +00:00
LYZJU2019
f0effd2680 feat: Implement SHOW STATUS (#4050) 
* show status returning empty contents

* return an empty set instead of affected rows

* chore: Update src/query/src/sql.rs

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-05-29 04:49:05 +00:00
dennis zhuang
aafb468547 fix: set local or session time_zone not work (#4064) 
* fix: set local or session time_zone not work

* chore: supports PostgreSQL-specific setting time zone
 2024-05-29 00:06:13 +00:00
Weny Xu
4aa756c896 feat: open region in background (#4052) 
* feat: open region in background

* feat: trace opening regions

* feat: wait for the opening region

* feat: let engine to handle the future open request

* fix: fix `test_region_registering`
 2024-05-28 13:58:15 +00:00
Weny Xu
d3860671a8 chore: add LAST_SENT_HEARTBEAT_ELAPSED metric (#4062) 2024-05-28 08:05:37 +00:00
tison
9dd6e033a7 refactor: move Database to client crate behind testing feature (#4059) 
* refactor: move Database to client crate behind testing feature

Signed-off-by: tison <wander4096@gmail.com>

* partial move

Signed-off-by: tison <wander4096@gmail.com>

* catch up more

Signed-off-by: tison <wander4096@gmail.com>

* fix imports

Signed-off-by: tison <wander4096@gmail.com>

* finish

Signed-off-by: tison <wander4096@gmail.com>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-05-28 03:21:43 +00:00
Weny Xu
048368fd87 feat: invoke flush_table and compact_table in fuzz tests (#4045) 
* feat: invoke `flush_table` and `compact_table` in fuzz tests

* feat: support to flush and compact physical metric table

* fix: avoid to create tables with the same name

* feat: validate values after flushing or compacting table
 2024-05-27 09:26:50 +00:00
tison
f9db5ff0d6 build(deps): upgrade opendal to 0.46 (#4037) 
* build(deps): upgrade opendal to 0.46

Signed-off-by: tison <wander4096@gmail.com>

* migrate writes

Signed-off-by: tison <wander4096@gmail.com>

* migrate reads

Signed-off-by: tison <wander4096@gmail.com>

* fixup object safety

Signed-off-by: tison <wander4096@gmail.com>

* fixup names

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* fixup compilation

Signed-off-by: tison <wander4096@gmail.com>

* a few Buffer to Vec

Signed-off-by: tison <wander4096@gmail.com>

* Make greptime buildable with opendal 0.46 (#5)

Signed-off-by: Xuanwo <github@xuanwo.io>

* fixup toml check

Signed-off-by: tison <wander4096@gmail.com>

* test_orc_opener

Signed-off-by: tison <wander4096@gmail.com>

* Fix lru cache (#6)

Signed-off-by: Xuanwo <github@xuanwo.io>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* improve comments

Signed-off-by: tison <wander4096@gmail.com>

* address comments

Signed-off-by: tison <wander4096@gmail.com>

* reduce buf copy

Signed-off-by: tison <wander4096@gmail.com>

* upgrade to reqwest 0.12

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2024-05-27 09:12:23 +00:00
Weny Xu
20ce7d428d fix(metric-engine): missing catchup implementation (#4048) 
* fix(metric-engine): missing catchup implementation

* fix: should be `metadata_region_id`
 2024-05-27 07:56:46 +00:00
tison
c78043d526 build(deps): merge tower deps to workspace (#4036) 
Signed-off-by: tison <wander4096@gmail.com>
 2024-05-27 07:15:30 +00:00
LFC
297105266b feat: enable tcp keepalive for http server (#4019) 
* feat: enable tcp keepalive for http server

* chore: for enterprise's update

* resolve PR comments
 2024-05-27 04:07:36 +00:00
Ruihang Xia
1de17aec74 feat: change EXPIRE WHEN to EXPIRE AFTER (#4002) 
* feat: change EXPIRE WHEN to EXPIRE AFTER

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change remaining

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename create_if_not_exist to create_if_not_exists

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* parse interval expr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-05-27 04:05:55 +00:00
Weny Xu
389ded93d1 chore: add logs for setting the region to writable (#4044) 
* chore: add logs for setting the region to writable

* fix: ignore redundant logs
 2024-05-27 04:01:40 +00:00
Eugene Tolbakov
af486ec0d0 feat(opertor): check if a database is in use before dropping it (#4035) 
feat(opertor): check if database is in use before dropping it
 2024-05-27 03:31:58 +00:00
irenjj
25d64255a3 feat: support table level comment (#4042) 
* feat: support table level comment

* use constants

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: tison <wander4096@gmail.com>
 2024-05-27 02:28:52 +00:00
tison
3790020d78 build(deps): upgrade promql-parser to 0.4 (#4047) 
* build(deps): upgrade promql-parser to 0.4

Signed-off-by: tison <wander4096@gmail.com>

* lock

Signed-off-by: tison <wander4096@gmail.com>

* catch up upgrades

Signed-off-by: tison <wander4096@gmail.com>

* concise method

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-05-27 01:51:59 +00:00
Weny Xu
5df3d4e5da feat: implement the LogStoreRawEntryReader and RawEntryReaderFilter (#4030) 
* feat: implement the `LogStoreRawEntryReader`

* feat: implement the `RawEntryReaderFilter`

* test: add tests
 2024-05-24 11:53:15 +00:00
Ruihang Xia
a58256d4d3 feat: round-robin selector (#4024) 
* feat: implement round robin peer selector

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document and test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-24 07:29:07 +00:00
Weny Xu
466f7c6448 feat: add RawEntryReader and OneshotWalEntryReader trait (#4027) 
* feat: add `RawEntryReader` and `OneShotWalEntryReader` trait

* chore: rename `OneShot` to `Oneshot`

* refacotr: remove `region_id` from `OneshotWalEntryReader`
 2024-05-24 06:30:50 +00:00
Ruihang Xia
0101657649 feat: remove one clone on constructing partition (#4028) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-24 04:01:19 +00:00
taobo
a3a2c8d063 feat: Add TLS support for gRPC service (#3957) 
* feat: Add tls support for grpc service

* feat: add integration test

* fix: integration test

* fix: revert by suggestion

* fix: typos

* fix: optimize code

* fix: optimize code

* docs: update configs
 2024-05-23 19:00:16 +00:00
Yingwen
dfc1acbb2a fix: notifies all workers once a region is flushed (#4016) 
* fix: notify workers to handle stalled requests if flush is finished

* chore: change stalled count to gauge

* feat: process stalled requests eagerly
 2024-05-23 12:45:00 +00:00
Lei, HUANG
0d055b6ee6 refactor: remove unused log config (#4021) 2024-05-23 08:59:42 +00:00
Ning Sun
b90b7adf6f feat: add fallback logic for vmagent sending wrong content type (#4009) 
* feat: add fallback logic for vmagent sending wrong content type

* fix: resolve lint issues

* Update src/servers/src/http/prom_store.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-05-23 02:40:17 +00:00
Jeremyhi
418090b464 chore: log error for detail (#4011) 
* chore: log error for detail

* chore: by cr
 2024-05-22 12:17:20 +00:00
Lei, HUANG
090b59e8d6 feat: manual compaction (#3988) 
* add compaction udf params

* wip: pass compaction options through grpc

* wip: pass compaction options all the way down to region server

* wip: window compaction task

* feat: trigger major compaction

* refactor: optimize compaction parameter parsing

* chore: rebase main

* chore: update proto

* chore: add some tests

* feat: validate catalog

* chore: fix typo and rebase main

* fix: some cr comments

* fix: file_time_bucket_span

* fix: avoid upper bound overflow

* chore: update proto
 2024-05-22 09:42:21 +00:00
shuiyisong
9e1af79637 chore: add ttl to write_cache (#4010) 
* chore: add ttl to write_cache

* chore: update test & add example config

* chore: fix typo

* chore: fix typo

* chore: fix typo
 2024-05-22 06:50:12 +00:00
zyy17
b86d79b906 fix: can't print log because the tracing guard is dropped (#4005) 
* fix: avoid logging guard drop

* chore: remove unused '#[allow(dead_code)]'
 2024-05-22 03:24:40 +00:00
Lei, HUANG
e070ba3c32 feat: respect time range when building parquet reader (#3947) 
* feat: convert timestamp range filters to predicates

* chore: rebase main

* fix: remove prediactes once they have been added to timestamp filters to avoid duplicate filtering

* fix: some comments

* fix: resolve conflicts
 2024-05-21 16:02:25 +00:00
zyy17
47785756e5 fix: move log_version() into build() of App to fix no log version on bootstrap (#4004) 2024-05-21 09:15:15 +00:00
Jeremyhi
0aa523cd8c feat: make create view procedure simple as others (#4001) 2024-05-21 08:30:57 +00:00
maco
40c585890a refactor: replace Expr with datafusion::Expr (#3995) 
* refactor: replace Expr with datafusion::Expr

* fix: fmt-toml

* fix: cr comment
 2024-05-21 06:40:29 +00:00
Yingwen
179c8c716c feat: Adds RegionScanner trait (#3948) 
* feat: define region scanner

* feat: single partition scanner

* feat: use single partition scanner

* feat: implement ExecutionPlan wip

* feat: mito engine returns single partition scanner

* feat: implement DisplayAs for region server

* feat: dummy table provider use handle_partitioned_query()

* test: update sqlness test

* feat: table provider use ReadFromRegion

* refactor: remove StreamScanAdapter

* chore: update lock

* style: fix clippy

* refactor: remove handle_query from the RegionEngine trait

* chore: address CR comments

* refactor: rename methods

* refactor: rename ReadFromRegion to RegionScanExec
 2024-05-20 11:52:00 +00:00
shuiyisong
19543f9819 feat: support compression on gRPC server (#3961) 
* feat: enable gzip in grpc server side

* feat: add enable_gzip_compression config

* test: add grpc compression test

* feat: support user configured compression on grpc server

* chore: update doc

* chore: add tests

* fix: make config-docs

* chore: fix cr issue

* chore: add test

* refactor: remove config on server side, auto enable all compression support

* chore: minor update

* chore: remove unused code

* refactor: enable zstd compression internally by default

* chore: minor fix
 2024-05-20 11:28:00 +00:00
discord9
533ada70ca chore: remove a dbg! forget to remove (#3990) 
* chore: remove a dbg! forget to remove

* remove other dbg! and add lint

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix pyo3 feature

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-20 08:34:47 +00:00
zyy17
82c3eca25e refactor: make the command entry cleaner (#3981) 
* refactor: move run() in App trait

* refactor: introduce AppBuilder trait

* chore: remove AppBuilder

* refactor: remove Options struct and make the start() clean

* refactor: init once for common_telemetry::init_global_logging
 2024-05-20 03:34:06 +00:00
Yingwen
df13832a59 feat: use cache in compaction (#3982) 2024-05-20 02:36:51 +00:00
Eugene Tolbakov
6a9a92931d chore: change binary array type from LargeBinaryArray to BinaryArray (#3924) 
* chore: change binary array type from LargeBinaryArray to BinaryArray

* fix: adjust try_into_vector logic

* fix: apply CR suggestions, add tests

* chore: fix failing test

* chore: fix integration test

* chore: adjust the assertions according to changed implementation

* chore: add a test with LargeBinary type

* chore: apply CR suggestions

* chore: simplify tests
 2024-05-18 08:04:41 +00:00