rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-05-19 14:30:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,564 Commits 133 Branches 253 Tags
751fa4ede906d65542f693c6dfb357a98f74e184
Commit Graph

2901 Commits

Author SHA1 Message Date
Yohan Wal
751fa4ede9 feat(config): make table name of pg backend configurable (#5244) 
* feat: configurable table name

* fix: election sql

* feat: configurable lock_id

* chore: update config file

* perf: useless allocation

* perf: useless allocation

* chore: remove unused type hint

* Apply suggestions from code review

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: update config file

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2025-01-08 14:55:35 +00:00
zyy17
03a2e6d0c1 feat: support elasticsearch _bulk API to ingest logs (#5261) 
* feat: support elasticsearch '_bulk' API to ingest logs

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

* refactor: add metrics

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-01-08 14:54:53 +00:00
Yingwen
c19a56c79f feat!: Remove script crate and python feature (#5321) 
* feat: exclude script crate

* chore: simplify feature

* feat: remove the script crate

* chore: remove python feature and some comments

* chore: fix warning
 2025-01-08 12:11:53 +00:00
Weny Xu
7f307a4cac feat: auto detecting local IP to use as hostname (#5314) 
* feat: auto detecting hostname

* chore: update config.md

* chore: bring back hostname

* chore: apply suggestions from CR

* chore: use const

* chore: apply suggesions from CR

* fix: fix sqlness
 2025-01-08 12:06:31 +00:00
yihong
52eebfce77 feat: support select session_user; (#5313) 
* feat: support `select session_user;`

This commit is part of support DBeaver that support function
select session_user like postgres did.

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: lint problem

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments add tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-08 09:44:54 +00:00
shuiyisong
d1f8ea7880 perf: parse Loki labels in protobuf write path (#5305) 
* chore: parse loki labels

* chore: add bench

* chore: add comment

* chore: add test

* chore: remove unnecessory default value and update test

* fix: typo and test

* chore: cr issue

* chore: cr issue
 2025-01-08 07:55:06 +00:00
LFC
2cd1b08ff7 refactor: optimize out partition split insert requests (#5298) 
* test: optimize out partition split insert requests if there is only one region

* Now that the optimization for single region insert has been lifted up, the original "fast path" can be obsoleted.

* resolve PR comments
 2025-01-08 07:26:29 +00:00
discord9
0ee41339aa feat(flow): flow refill state (Part 1) (#5295) 
* feat(flow): (Part 1) refill utils

* chore: after rebase fix

* chore: more rebase

* rm refill.rs to reduce pr size

* chore: simpler args

* refactor: per review

* docs: more explain for instant requests

* refactor: per review
 2025-01-08 06:45:56 +00:00
yihong
369b59c84a fix: drop unused dep using udeps to minial the size (#5301) 
* fix: drop unused dep using udeps to minial the size

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: adress comments fix the problem

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-08 06:36:22 +00:00
Zhenchi
c305b2b406 fix: show index to display skipping index (#5297) 
* fix: show index to display skipping index

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix sqlness

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix sqlness

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-01-08 04:59:40 +00:00
Weny Xu
3d9df822ad refactor: refactor PgStore (#5309) 
* refactor: refactor PgStore

* fix: election use bytea and txn use Serializable to avoid read unrepeatable (#4)

* fix: election use bytea as well

* fix: use Serializable to avoid read unrepeatable

* chore: remove unused error

* ci: enable pg kvbackend and sqlness

* ci: switch on pg_kvbackend feature

* fix: fix sqlness runner

* chore: add pg_kvbackend feature gate

* build(ci): add feature gate

* fix: add retry for `PgStore` txn

* fix: correct `SET_IDLE_SESSION_TIMEOUT`

---------

Co-authored-by: Yohan Wal <1035325592@qq.com>
Co-authored-by: CookiePieWw <profsyb@gmail.com>
 2025-01-07 07:27:58 +00:00
Weny Xu
05f115e047 fix(pg_backend): correct set idle_in_transaction_session_timeout statement (#5304) 
* fix(metasrv): correct `backend` field configuration

* refactor!: added `#[serde(rename_all = "snake_case")]` macro to the `BackendImpl` enum

* fix(metasrv): correct `set idle_in_transaction_session_timeout` statement

* build: enable `pg_backend` by default
 2025-01-06 12:07:35 +00:00
Zhenchi
5cf9d7b6ca fix(bloom-filter): filter rows with segment precision (#5286) 
* fix(bloom-filter): filter rows with segment precision

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add case

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address TODO

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-01-06 11:45:15 +00:00
shuiyisong
a56c430db0 fix: pass unknown encoding in decompression layer (#5300) 
fix: allow pass unknown in decompression layer
 2025-01-06 09:10:56 +00:00
Ning Sun
04708f10aa feat(pipeline): allow coerce timestamp from integer values (#5270) 
* feat: allow coerce timestamp from integer values

* test: add tests for parsing integer values
 2025-01-06 08:53:09 +00:00
Weny Xu
96b2a5fb28 feat: introduce ParallelFstValuesMapper (#5276) 
* refactor: `RangeReader` to use `&self`

* refactor: `InvertedIndexReader` to use `&self`

* refactor: refactor: `BloomFilterReader` to use `&self`

* feat: introduce `ParallelFstValuesMapper`

* chore: change prefetch size to 8KiB

* chore: add `file_size_hint` for cached blob reader

* chore: fix clippy

* refactor: remove `FstValuesMapper`

* chore: apply suggestions from CR
 2025-01-06 07:33:35 +00:00
shuiyisong
bbbba29afc feat: support Loki JSON write (#5288) 
* perf: small updates

* refactor: move loki to a separate file

* chore: extract content_type pattern matching

* chore: minor update

* feat: loki json write

* chore: add decompression http layer

* fix: label string value instead of to_string

* chore: add test

* fix: typo

* fix: license header

* chore: rename
 2025-01-06 07:09:48 +00:00
Kould
2ad50332cb feat: impl COPY a query resultset to external file (#5250) 
* feat: impl COPY a query resultset to external file

* chore: add more tests for parse `copy_table_to`

* chore: add more tests for parse `copy_table_to`
 2025-01-06 04:32:51 +00:00
Yohan Wal
513569ed5d feat: add Txn for pg kv backend (#5266) 
* feat: txn for pg kv backend

* chore: clippy

* fix: txn uses one client

* test: clean up and txn test

* test: clean up

* test: change lock_id to avoid conflict in test

* test: use different prefix in pg election test

* fix(test): just a fix

* test: aggregate multiple test to avoid concurrency problem

* test: use uuid instead of rng

* perf: batch cmp in txn

* perf: batch same op in txn
 2025-01-06 03:29:09 +00:00
Kould
1067357b72 chore(config)!: refactor configs of write cache (#5259) 
* chore: refactor configs of write cache

* chore: write_cache_size `10GiB` -> `5GiB`
 2025-01-04 07:14:38 +00:00
Ning Sun
2caf003db0 fix: brings back functions required by downstream projects (#5283) 2025-01-04 03:49:01 +00:00
discord9
353c8230db refactor: flow replace check&better error msg (#5277) 
* chore: better error msg

* chore eof newline

* refactor: move replace check to flow worker

* chore: add ctx to insert flow failure

* chore: Update src/flow/src/adapter/flownode_impl.rs

* test: add order by for deterministic

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2025-01-03 10:59:58 +00:00
Lei, HUANG
577d81f14c chore: suppress list warning (#5280) 
chore/suppress-list-warning:
 ### Update logging level in `intermediate.rs`

 - Changed logging level from `warn` to `debug` for unexpected directory entries in index creation.
 - Added `debug` to the `common_telemetry` import to support the logging level change.
 2025-01-03 09:05:03 +00:00
yihong
856bba5d95 fix: better fmt check from 40s to 4s (#5279) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-03 08:12:49 +00:00
Yingwen
89399131dd feat: support add if not exists in the gRPC alter kind (#5273) 
* test: test adding existing columns

* chore: add more checks to AlterKind

* chore: update logs

* fix: check and build table info first

* feat: Add add_if_not_exists flag to alter expr

* feat: skip existing columns when building alter kind

* checks in make_region_alter_kind()
* reuse the alter kind

* test: fix tests in common-meta

* chore: fix typos

* chore: update comments
 2025-01-03 07:23:17 +00:00
discord9
d20b592fe8 fix: flow handle reordered inserts (#5275) 
* fix: reorder correct schema

* tests: reorder insert handled correctly

* chore: rm unused

* refactor: per review

* chore: more comment

* chore: per review
 2025-01-03 06:25:39 +00:00
Yohan Wal
bcb0f14227 refactor: adjust index cache page size (#5267) 
* refactor: adjust index cache page size

* fix: wrong docs

* Update config/datanode.example.toml

* Update config/config.md

* Update config/config.md

* chore: adjust to 64KiB

* Apply suggestions from code review
 2025-01-03 03:26:17 +00:00
discord9
4d6fe31fff fix(flow): flow's table schema cache (#5251) 
* fix: flow schema cache

* refactor: location for `to_meta_err`

* chore: endfile emptyline

* chore: review(partially)

* chore: per review

* refactor: per review

* refactor: per review
 2025-01-02 10:33:23 +00:00
Yohan Wal
1b0b9add90 feat: use connection pool for pg kv backend in preparation for txn (#5260) 
* feat: use connection pool

* chore: follow review comments

* fix: create table before test
 2025-01-02 06:33:21 +00:00
chenmortal
2b89970d45 fix: import tokio-metrics and tokio-metrics-collector (#5264) 2025-01-02 05:58:31 +00:00
Ning Sun
53d006292d fix: correct invalid testing feature gate usage (#5258) 
* fix: correct invalid testing feature gate usage

* test: refactor tests to avoid test code leak

* fix: sync main
 2025-01-02 03:22:54 +00:00
discord9
d18c8b5e16 chore: typo (#5265) 
* fix: a typo

* chore: even more typos
 2025-01-02 03:17:53 +00:00
jeremyhi
e0949c4a11 feat: hints all in one (#5194) 
* feat: hints all in one

* chore: If hints are provided in the x-greptime-hints header, ignore the rest of the headers
 2025-01-02 02:56:33 +00:00
Ruihang Xia
5cf931c417 feat(log-query): implement pagination with limit and offset parameters (#5241) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-31 06:42:46 +00:00
Ruihang Xia
55b7656956 feat: override __sequence on creating SST to save space and CPU (#5252) 
* override memtable sequence

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* override sst sequence

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore changes per to CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use correct sequence number

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* wrap a method to get max sequence

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-31 03:28:02 +00:00
Yingwen
75e4f307c9 feat: update partition duration of memtable using compaction window (#5197) 
* feat: update partition duration of memtable using compaction window

* chore: only use provided duration if it is not None

* test: more tests

* test: test compaction apply window

* style: fix clippy
 2024-12-30 13:06:25 +00:00
Yohan Wal
89f2e15ffb feat: add election logic for PgElection (#5249) 
* feat: init PgElection

* fix: release advisory lock

* fix: handle duplicate keys

* chore: update comments

* fix: unlock if acquired the lock

* chore: add TODO and avoid unwrap

* refactor: check both lock and expire time, add more comments

* test: add unit test for pg election

* chore: fmt

* chore: typo

* fix: add feature gate

* chore: visibility

* chore: follow review comments
 2024-12-30 09:45:04 +00:00
zyy17
13ed10556a refactor: support to convert time string to timestamp in convert_value() (#5242) 
refactor: support to covert time string to timestamp in convert_value()
 2024-12-30 08:36:08 +00:00
Lin Yihai
d1108ab581 feat: add vec_div function (#5245) 2024-12-30 07:08:26 +00:00
Zhenchi
109fe04d17 fix(bloom-filter): skip applying for non-indexed columns (#5246) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-12-30 06:56:58 +00:00
Yingwen
f1eb76f489 fix: implement a CacheStrategy to ensure compaction use cache correctly (#5254) 
* feat: impl CacheStrategy

* refactor: replace Option<CacheManagerRef> with CacheStrategy

* feat: add disabled strategy

* ci: force update taplo

* refactor: rename CacheStrategy::Normal to CacheStrategy::EnableAll

* ci: force install cargo-gc-bin

* ci: force install

* chore: use CacheStrategy::Disabled as ScanInput default

* chore: fix compiler errors
 2024-12-30 06:24:53 +00:00
shuiyisong
588f6755f0 fix: disable path label in opendal for now (#5247) 
* fix: remove path label in opendal for now

* fix: typo

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-27 04:34:19 +00:00
Kould
dad8ac6f71 feat(vector): add vector functions vec_sub & vec_sum & vec_elem_sum (#5230) 
* feat(vector): add sub function

* chore: added check for vector length misalignment

* feat(vector): add `vec_sum` & `vec_elem_sum`

* chore: codefmt

* update lock file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-26 15:07:13 +00:00
Yohan Wal
ef13c52814 feat: init PgElection with candidate registration (#5209) 
* feat: init PgElection

fix: release advisory lock

fix: handle duplicate keys

chore: update comments

fix: unlock if acquired the lock

chore: add TODO and avoid unwrap

refactor: check both lock and expire time, add more comments

chore: fmt

fix: deal with multiple edge cases

feat: init PgElection with candidate registration

chore: fmt

chore: remove

* test: add unit test for pg candidate registration

* test: add unit test for pg candidate registration

* chore: update pg env

* chore: make ci happy

* fix: spawn a background connection thread

* chore: typo

* fix: shadow the election client for now

* fix: fix ci

* chore: readability

* chore: follow review comments

* refactor: use kvbackend for pg election

* chore: rename

* chore: make clippy happy

* refactor: use pg server time instead of local ones

* chore: typo

* chore: rename infancy to leader_infancy for clarification

* chore: clean up

* chore: follow review comments

* chore: follow review comments

* ci: unit test should test all features

* ci: fix

* ci: just test pg
 2024-12-26 12:39:32 +00:00
Zhenchi
7471f55c2e feat(mito): add bloom filter read metrics (#5239) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-12-26 04:44:03 +00:00
Zhenchi
f4b2d393be feat(config): add bloom filter config (#5237) 
* feat(bloom-filter): integrate indexer with mito2

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(config) add bloom filter config

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix docs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* merge

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove cache config

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-12-26 04:38:45 +00:00
localhost
0cf44e1e47 chore: add more info for pipeline dryrun API (#5232) 2024-12-26 03:06:25 +00:00
Ruihang Xia
00ad27dd2e feat(bloom-filter): bloom filter applier (#5220) 
* wip

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* draft search logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use defined BloomFilterReader

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* round the range end

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finish index applier

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* integrate applier into mito2 with cache layer

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix cache key and add unit test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* provide bloom filter index size hint

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert BloomFilterReaderImpl::read_vec

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove dead code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore null on eq

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more tests and fix bloom filter logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-12-26 02:51:18 +00:00
discord9
5ba8bd09fb fix: flow compare null values (#5234) 
* fix: flow compare null values

* fix: fix again ck ty before cmp

* chore: rm comment

* fix: handle null

* chore: typo

* docs: update comment

* refactor: per review

* tests: more sqlness

* tests: sqlness not show create table
 2024-12-25 15:31:27 +00:00
Zhenchi
a9f21915ef feat(bloom-filter): integrate indexer with mito2 (#5236) 
* feat(bloom-filter): integrate indexer with mito2

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* rename skippingindextype

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-12-25 14:30:07 +00:00