rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-26 01:40:36 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,834 Commits 134 Branches 254 Tags
f04f16cec3efae81b3d375aef0b1e9d2a33dedd4
Commit Graph

91 Commits

Author SHA1 Message Date
discord9
3811e3f632 feat: also get index file&expose mito in metrics (#5680) 
* feat: download index file too

* feat: expose mito in metrics

* chore: fmt
 2025-03-10 13:07:08 +00:00
Ruihang Xia
73ca39f37e feat: time series distribution in scanner (#5675) 
* define distribution

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: SeqScan support per series distribution

* probe distribution

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reverse sort order

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more strict check

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change null's ordering

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
 2025-03-10 05:43:17 +00:00
Weny Xu
1d637cad51 fix(metric-engine): group DDL requests (#5628) 
* fix(metric-engine): group DDL requests

* test: add sqlness tests

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2025-03-05 09:17:47 +00:00
Ruihang Xia
d363c8ee3c fix: check physical region before use (#5612) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-28 06:46:48 +00:00
yihong
ff0dcf12c5 perf: close issue 4974 by do not delete columns when drop logical region about 100 times faster (#5561) 
* perf: do not delete columns when drop logical region in drop database

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: make ci happy

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address review comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address some comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: drop stupid comments by copilot

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* chore: minor refactor

* chore: minor refactor

* chore: update grpetime-proto

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
 2025-02-25 09:00:49 +00:00
Ruihang Xia
4f988b5ba9 feat: remove default inverted index for physical table (#5583) 
* feat: remove default inverted index for physical table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-22 06:48:05 +00:00
discord9
1e6d2fb1fa feat: add snapshot seqs field to query context (#5477) 
* TODO: snapshot read

* feat: RegionEngine get last seq

* feat: query context snapshot

* chore: use new proto

* feat: get_region_seqs in region engine

* chore: typo

* chore: toml

* feat: make snapshots modifiable

* feat: add hint for snapshot read

* chore: some typo

* refactor: remove hint as not used

* fix: use commited seqs

* refactor: remove sequences variant on RegionRequest

* refactor: per review

* chore: rebase solve conflict

* refactor: rm unused key

* chore: per review

* chore: per review
 2025-02-14 09:07:48 +00:00
Weny Xu
c56106b883 perf: optimize table alteration speed in metric engine (#5526) 
* feat(metric-engine): introduce batch alter request handling

* refactor: minor refactor

* refactor: push down filter to mito

* chore: apply suggestions from CR
 2025-02-14 08:11:48 +00:00
Weny Xu
bb97f1bf16 perf: optimize table creation speed in metric engine (#5503) 
* feat(metric-engine): introduce batch create request handling

* chore: remove unused code

* test: add more tests

* chore: remove unused error

* chore: apply suggestions from CR
 2025-02-13 07:39:04 +00:00
yihong
79acc9911e fix: Delete statement not supported in metric engine close #4649 (#5473) 
* fix: Delete statement not supported in metric engine close #4649

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: do not include Truncate address review comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comment again

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-07 06:47:53 +00:00
Weny Xu
b107384cc6 feat(metric-engine): support to write rows with sparse primary key encoding (#5424) 
* feat: support to write rows with sparse primary key encoding

* feat: cache decoded primary key

* chore: remove unused code

* feat: create physical table based on the engine config

* chore: log primary key encoding info

* fix: correct sqlness test

* chore: correct config.md

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2025-01-24 06:56:09 +00:00
Weny Xu
965a48656f feat(metric-engine): introduce RowModifier for MetricEngine (#5380) 
* feat(metric-engine): store physical table ColumnIds in `MetricEngineState`

* feat(metric-engine): introduce `RowModifier` for MetricEngine

* chore: upgrade greptime-proto

* feat: introduce `WriteHint` to `RegionPutRequest`

* chore: apply suggestions from CR

* chore: udpate greptime-proto

* chore: apply suggestions from CR

* chore: add comments

* chore: update proto
 2025-01-22 05:16:44 +00:00
Weny Xu
385b1bcbb0 feat(metric-engine): introduce index options from metric engine (#5374) 
* feat(metric-engine): introduce index options from metric engine

* chore: fmt toml

* test: add sqlness test

* fix: ignore internal columns

* chore: remove unused dep

* chore: update sqlness result

* chore: ignore metric engine internal columns

* chore: refine code styling

* test: update sqlness test

* refactor: refactor `create_table_constraints`

* test: show index

* chore: apply suggestions from CR

* fix: set inverted index explicitly

* chore: apply suggestions from CR
 2025-01-20 08:48:00 +00:00
Weny Xu
4323c20d18 feat(metric-engine): introduce experimental_sparse_primary_key_encoding to MetricEngineConfig (#5373) 
* feat: introduce `experimental_sparse_primary_key_encoding` to `MetricEngineConfig`

* fix: unit test
 2025-01-20 03:49:39 +00:00
Yingwen
7eaabb3ca2 fix: increase in progress scan gauge and adjust histogram buckets (#5370) 
* fix: in progress scan doesn't inc

* feat(mito): adjust mito histogram buckets

* chore(metric-engine): adjust metric engine histogram bucket
 2025-01-16 12:53:03 +00:00
discord9
758aef39d8 feat: filter batch by sequence in memtable (#5367) 
* feat: add seq field

* feat: filter by sequence

* chore: per review

* docs: explain why not prune

* chore: correct doc

* test: test filter by seq
 2025-01-16 04:44:28 +00:00
Ruihang Xia
422d18da8b feat: bump opendal and switch prometheus layer to the upstream impl (#5179) 
* feat: bump opendal and switch prometheus layer to the upstream impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused files

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused things

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove root dir on recovering cache

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* filter out non-files entry in test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-19 03:42:05 +00:00
Yingwen
8a5384697b chore: add aquamarine to dep lists (#5181) 2024-12-17 01:45:50 +00:00
Weny Xu
d0245473a9 fix: correct set_region_role_state_gracefully behaviors (#5171) 
* fix: reduce default max rows for fuzz testing

* chore: remove Postgres setup from fuzz test workflow

* chore(fuzz): increase resource limits for GreptimeDB cluster

* chore(fuzz): increase resource limits for kafka

* fix: correct `set_region_role_state_gracefully` behaviors

* chore: remove Postgres setup from fuzz test workflow

* chore(fuzz): redue resource limits for GreptimeDB & kafka
 2024-12-16 14:01:40 +00:00
shuiyisong
9d7fea902e chore: remove unused dep (#5163) 
* chore: remove unused dep

* chore: remove more unused dep
 2024-12-16 06:17:27 +00:00
discord9
8b944268da feat: ttl=0/instant/forever/humantime&ttl refactor (#5089) 
* feat: ttl zero filter

* refactor: use TimeToLive enum

* fix: unit test

* tests: sqlness

* refactor: Option<TTL> None means UNSET

* tests: sqlness

* fix: 10000 years --> forever

* chore: minor refactor from reviews

* chore: rename back TimeToLive

* refactor: split imme request from normal requests

* fix: use correct lifetime

* refactor: rename immediate to instant

* tests: flow sink table default ttl

* refactor: per review

* tests: sqlness

* fix: ttl alter to instant

* tests: sqlness

* refactor: per review

* chore: per review

* feat: add db ttl type&forbid instant for db

* tests: more unit test
 2024-12-06 09:20:42 +00:00
Lanqing Yang
8bdef776b3 fix: allow physical region alter region options (#5046) 
allow physical region alter region options
 2024-11-27 08:24:34 +00:00
Lei, HUANG
fae141ad0a fix(metric-engine): set ttl also on opening metadata regions (#5051) 
* fix/metric-metadata-region-options: Remove APPEND_MODE_KEY and refactor TTL option handling in MetricEngineInner

* fix/metric-metadata-region-options: Refactor metadata region options into a shared function

 • Extract metadata region options into region_options_for_metadata_region function
 • Replace inline options map with a call to the new shared function in both create.rs and open.rs files

* fix: exclude typos

* fix/metric-metadata-region-options:
 Refactor metadata region options to accept original options and remove APPEND_MODE_KEY
 2024-11-26 08:14:41 +00:00
Ruihang Xia
e3927ea6f7 fix: prevent metadata region from inheriting database ttl (#5044) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-11-25 02:13:11 +00:00
dennis zhuang
d7c3c8e124 fix: physical table statistics info (#4975) 
* fix: physical table statistics info

* refactor: is_physical_table

* fix: remove file
 2024-11-13 14:29:44 +00:00
discord9
175fddb3b5 fix: alter table add column id alloc mismatch (#4972) 
* fix: alter table add column id alloc mismatch

* chore: remove debug code

* chore: typos

* chore: error variant

* chore: more checks for invariant

* refactor: better check&explain

* fix: exist column metadata correct

* chore: remove unused error variant
 2024-11-13 07:02:35 +00:00
Ruihang Xia
fcd0ceea94 fix: column already exists (#4961) 
* fix: merge fetched logical metadata with existing cache

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix log acquire

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/metric-engine/src/engine/region_metadata.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-11-07 13:25:05 +00:00
Weny Xu
081c6d9e74 fix: flush metric metadata region (#4852) 
* fix: flush metric metadata region

* chore: apply suggestions from CR
 2024-10-18 07:21:35 +00:00
Ruihang Xia
b61a388d04 refactor: replace info logs with debug logs in region server (#4829) 
* refactor: replace info logs with debug logs in region server

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: update error handling for closing and opening nonexistent regions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-10-14 12:46:07 +00:00
Ruihang Xia
06e565d25a feat: cache logical region's metadata (#4827) 
* feat: cache logical region's metadata

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: implement logical region locking for metadata operations

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: correct typo in comment for MetadataRegion struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-10-14 08:44:13 +00:00
Weny Xu
6e776d5f98 feat: support to reject write after flushing (#4759) 
* refactor: use `RegionRoleState` instead of `RegionState`

* feat: introducing `RegionLeaderState::Downgrading`

* refactor: introduce `set_region_role_state_gracefully`

* refactor: use `set_region_role` instead of `set_writable`

* feat: support to reject write after flushing

* fix: fix unit tests

* test: add unit test for `should_reject_write`

* chore: add comments

* chore: refine comments

* fix: fix unit test

* test: enable fuzz tests for Local WAL

* chore: add logs

* chore: rename `RegionStatus` to `RegionState`

* feat: introduce `DowngradingLeader`

* chore: rename

* refactor: refactor `set_role_state` tests

* test: ensure downgrading region will reject write

* chore: enhance logs

* chore: refine name

* chore: refine comment

* test: add tests for `set_role_role_state`

* fix: fix unit tests

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-09-30 08:28:51 +00:00
Weny Xu
4045298cb2 feat: add region_statistics table (#4771) 
* refactor: introduce `region_statistic`

* refactor: move DatanodeStat related structs to common_meta

* chore: add comments

* feat: implement `list_region_stats` for `ClusterInfo` trait

* feat: add `region_statistics` table

* feat: add table_id and region_number fields

* chore: rename unused snafu

* chore: udpate sqlness results

* chore: avoid to print source in error msg

* chore: move `procedure_info` under `greptime` catalog

* chore: apply suggestions from CR

* Update src/common/meta/src/datanode.rs

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

---------

Co-authored-by: jeremyhi <jiachun_feng@proton.me>
 2024-09-27 09:54:52 +00:00
Ruihang Xia
ff40d512bd fix: support append-only physical table (#4716) 
* fix: support append-only physical table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/metric-engine/src/engine/create.rs

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
 2024-09-10 12:23:23 +00:00
jeremyhi
e88465840d feat: add extension field to HeartbeatRequest (#4688) 
* feat: add extension field to HeartbeatRequest

* chore: extension to extensions

* chore: upgrade proto
 2024-09-06 08:29:20 +00:00
Ruihang Xia
93f202694c refactor: remove unused error variants (#4666) 
* add python script

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unused errors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix all negative cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* setup CI

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-09-03 13:19:38 +00:00
Weny Xu
47657ebbc8 feat: replay WAL entries respect index (#4565) 
* feat(log_store): use new `Consumer`

* feat: add `from_peer_id`

* feat: read WAL entries respect index

* test: add test for `build_region_wal_index_iterator`

* fix: keep the handle

* fix: incorrect last index

* fix: replay last entry id may be greater than expected

* chore: remove unused code

* chore: apply suggestions from CR

* chore: rename `datanode_id` to `location_id`

* chore: rename `from_peer_id` to `location_id`

* chore: rename `from_peer_id` to `location_id`

* chore: apply suggestions from CR
 2024-08-28 11:37:18 +00:00
Ruihang Xia
a3b8d2fe8f chore: bump rust toolchain to 2024-08-21 (#4606) 
* chore: bump rust toolchain to 2024-08-22

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update workflow

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* try 20240606

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-08-22 15:38:10 +00:00
Weny Xu
cb4cffe636 chore: bump opendal version to 0.48 (#4499) 2024-08-04 00:46:04 +00:00
Yingwen
458e5d7e66 feat: add TimeSeriesRowSelector hint (#4327) 
* feat: Add TimeSeriesRowSelector

* feat: scan allow specify series row selector

* chore: Update comment
 2024-07-09 12:29:47 +00:00
Zhenchi
00308218b3 feat(fulltext_index): allow enable full-text index in SQL and gRPC way (#4310) 
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: test_fulltext_intm_path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: explicitly build column options

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: fix error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-08 20:18:48 +00:00
Weny Xu
15ad9f2f6f fix: region logical regions after catching up (#4176) 
* fix: region logical regions after catching up

* test: add metric table migration test

* chore: apply suggestions from CR
 2024-06-21 10:30:18 +00:00
Ruihang Xia
fce65c97e3 feat: make RegionScanner aware of PartitionRange (#4170) 
* define PartitionRange

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add optimizer rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl aggr stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add fallback method

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document and rename struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-21 09:54:22 +00:00
Jeremyhi
b03cb3860e chore: reduce some burden on the write path (#4110) 
* chore: remove unnecessary checking

* chore: avoid do the same thing in a loop
 2024-06-06 06:45:19 +00:00
Lei, HUANG
a80059b47f fix: recover memtable options when opening physical regions (#4102) 
* fix: recover memtable options when opening physical regions

* chore: fmt

* chore: merge data region options
 2024-06-04 08:20:29 +00:00
Weny Xu
eab309ff7e fix: avoid acquiring lock during reading stats (#4070) 
* fix: avoid acquiring lock during reading stats

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-05-30 07:08:04 +00:00
Weny Xu
048368fd87 feat: invoke flush_table and compact_table in fuzz tests (#4045) 
* feat: invoke `flush_table` and `compact_table` in fuzz tests

* feat: support to flush and compact physical metric table

* fix: avoid to create tables with the same name

* feat: validate values after flushing or compacting table
 2024-05-27 09:26:50 +00:00
Weny Xu
20ce7d428d fix(metric-engine): missing catchup implementation (#4048) 
* fix(metric-engine): missing catchup implementation

* fix: should be `metadata_region_id`
 2024-05-27 07:56:46 +00:00
maco
40c585890a refactor: replace Expr with datafusion::Expr (#3995) 
* refactor: replace Expr with datafusion::Expr

* fix: fmt-toml

* fix: cr comment
 2024-05-21 06:40:29 +00:00
Yingwen
179c8c716c feat: Adds RegionScanner trait (#3948) 
* feat: define region scanner

* feat: single partition scanner

* feat: use single partition scanner

* feat: implement ExecutionPlan wip

* feat: mito engine returns single partition scanner

* feat: implement DisplayAs for region server

* feat: dummy table provider use handle_partitioned_query()

* test: update sqlness test

* feat: table provider use ReadFromRegion

* refactor: remove StreamScanAdapter

* chore: update lock

* style: fix clippy

* refactor: remove handle_query from the RegionEngine trait

* chore: address CR comments

* refactor: rename methods

* refactor: rename ReadFromRegion to RegionScanExec
 2024-05-20 11:52:00 +00:00
Ruihang Xia
115c74791d build(deps): bump snafu to 0.8 (#3911) 
* change Cargo.toml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* global replace

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle alias in script engine

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-05-10 13:36:25 +00:00