Commit Graph

2960 Commits

Author SHA1 Message Date
Lei, HUANG
51a8d0a726 fix: avoid suppress manual compaction (#5399)
* fix/avoid-suppress-manual-compaction:
 **Refactor Compaction Logic**

 - Removed `PendingCompaction` struct and integrated its functionality directly into `CompactionStatus` in `compaction.rs`.
 - Simplified waiter management by consolidating waiter handling logic into `CompactionStatus`.
 - Updated `CompactionRequest` creation to directly handle waiters without intermediate structures.
 - Adjusted test cases in `compaction.rs` to align with the new waiter management approach.

(cherry picked from commit 87e2d1c2cc9bd82c02991d22e429bef25c5ee348)

* fix/avoid-suppress-manual-compaction:
 ### Add Support for Manual Compaction Requests

 - **Compaction Logic Enhancements**:
   - Updated `CompactionScheduler` in `compaction.rs` to handle manual compaction requests using `Options::StrictWindow`.
   - Introduced `PendingCompaction` struct to manage pending manual compaction requests.
   - Added logic to reschedule manual compaction requests once the current compaction task is completed.

 - **Testing**:
   - Added `test_manual_compaction_when_compaction_in_progress` to verify the handling of manual compaction requests during ongoing compaction processes.

 These changes enhance the compaction scheduling mechanism by allowing manual compaction requests to be queued and processed efficiently.

(cherry picked from commit bc38ed0f2f8ba2c4690e0d0e251aeb2acce308ca)

* chore: fix conflicts

* fix/avoid-suppress-manual-compaction:
 ### Add Error Handling for Manual Compaction Override

 - **`compaction.rs`**: Enhanced the `set_pending_request` method to handle manual compaction overrides by sending an error to the waiter if a previous request exists.
 - **`error.rs`**: Introduced a new error variant `ManualCompactionOverride` to represent manual compaction being overridden, and mapped it to the `Cancelled` status code.

* fix: format

* fix/avoid-suppress-manual-compaction:
 **Add Error Handling for Pending Compaction Requests**

 - Enhanced error handling in `compaction.rs` by adding logic to handle errors for pending compaction requests.
 - Introduced a mechanism to send errors using `waiter.send` when a pending compaction request fails, ensuring proper error propagation and context with `CompactRegionSnafu`.

* fix/avoid-suppress-manual-compaction:
 **Fix Typo and Simplify Code Logic in `compaction.rs`**

 - Corrected a typo in the license comment from "langucage" to "language".
 - Simplified the logic for handling `pending_compaction` in `CompactionStatus` by removing unnecessary pattern matching and directly accessing `waiter`.

* fix: typo
2025-01-22 05:36:39 +00:00
Weny Xu
965a48656f feat(metric-engine): introduce RowModifier for MetricEngine (#5380)
* feat(metric-engine): store physical table ColumnIds in `MetricEngineState`

* feat(metric-engine): introduce `RowModifier` for MetricEngine

* chore: upgrade greptime-proto

* feat: introduce `WriteHint` to `RegionPutRequest`

* chore: apply suggestions from CR

* chore: udpate greptime-proto

* chore: apply suggestions from CR

* chore: add comments

* chore: update proto
2025-01-22 05:16:44 +00:00
dennis zhuang
4259975be9 feat: support not-equal matcher for PromQL metric names (#5385)
* feat: make instant_query and range_query to supports not-equal matchers

* feat: impl query_metric_names

* feat: forgot some files and refactor

* chore: test and docs

* fix: typo

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* refactor: parse_query

* chore: improve test

* fix: use current catalog to query information_schema

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-22 03:04:40 +00:00
yihong
d2f3f2e24d fix: vector function for PromQL need to ignore the time index also (#5398)
* fix: vector function for PromQL need to ignore the time index also close #5392

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: do not affect scalar function

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: betteer name for it

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-22 02:51:12 +00:00
Zhenchi
f74a955504 feat: bloom filter as fulltext index v2 (Part 1) (#5406)
* feat: bloom filter as fulltext index v2

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests for tokenizer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor dup vars

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
2025-01-21 23:33:11 +00:00
discord9
9f626ec776 chore: better error msg (#5415) 2025-01-21 07:32:32 +00:00
Niwaka
0163ce8df9 feat: add column if not exists (#5393)
* feat: add column if not exists

* chore: address reviews
2025-01-21 02:38:25 +00:00
shuiyisong
2ab235ec9d chore: extract service_name in OTLP traces by default (#5412)
chore: add service_name in traces
2025-01-21 02:34:56 +00:00
Zhenchi
281d9a5920 fix: matches incorrectly uses byte len as char len (#5411)
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
2025-01-21 02:34:36 +00:00
Weny Xu
385b1bcbb0 feat(metric-engine): introduce index options from metric engine (#5374)
* feat(metric-engine): introduce index options from metric engine

* chore: fmt toml

* test: add sqlness test

* fix: ignore internal columns

* chore: remove unused dep

* chore: update sqlness result

* chore: ignore metric engine internal columns

* chore: refine code styling

* test: update sqlness test

* refactor: refactor `create_table_constraints`

* test: show index

* chore: apply suggestions from CR

* fix: set inverted index explicitly

* chore: apply suggestions from CR
2025-01-20 08:48:00 +00:00
Yohan Wal
5287d46073 refactor: use MetadataKey for kafka topic (#5351)
* refactor: use MetadataKey

* fix: match all prefix

* refactor: introduce TopicPool

* fix: fix test, some rename

* test: add unit test for legacy restore

* fix: add _ between prefix and topic id

* chore: readable legacy topics

* refactor: a refactor

* Apply suggestions from code review

* Apply suggestions from code review

* refactor: introduce TopicPool

* fix: fix unit test

* chore: fix unit test and add some comments

* fix: fix unit test

* refactor: just refactor

* refactor: rename

* chore: rename, comments and remove unnecessary clone
2025-01-20 07:38:22 +00:00
Lei, HUANG
64ce9d3744 chore(http): change authorization header (#5389)
* chore/change-authorization-header:
 ### Add Custom Authorization Header Support

 - **Files Modified**: `http.rs`, `authorize.rs`, `authorize.rs` (tests)
 - **Key Changes**:
   - Introduced a custom authorization header `x-greptime-auth` in `http.rs`.
   - Updated authorization logic in `authorize.rs` to support both `x-greptime-auth` and the standard `Authorization` header.
   - Enhanced test cases in `authorize.rs` to validate the new custom header functionality.

* chore: add more tests
2025-01-20 07:09:44 +00:00
Weny Xu
4323c20d18 feat(metric-engine): introduce experimental_sparse_primary_key_encoding to MetricEngineConfig (#5373)
* feat: introduce `experimental_sparse_primary_key_encoding` to `MetricEngineConfig`

* fix: unit test
2025-01-20 03:49:39 +00:00
discord9
87c21e2baa fix(flow): deal with flow drop leftover (#5391)
* fix: deal with flow drop leftover

* chore: make it warn

* chore: apply suggestion.

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* chore: review

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-19 12:55:25 +00:00
yihong
d072801ad6 fix: drop unused pub fn using to cargo_workspace_unused (#5352)
* fix: drop unused pub fn using to cargo_workspace_unused

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: seems is_sum can delete too

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-18 15:59:21 +00:00
Lei, HUANG
0607b38a20 chore: change default compaction output size limit to 2GB (#5378)
chore/change-default-compaction-output-size-limit:
 ### Update `TwcsOptions` Default Configuration

 - Modified the default value of `max_output_file_size` in `TwcsOptions` to `Some(ReadableSize::gb(2))` in `src/mito2/src/region/options.rs`.
2025-01-18 15:11:36 +00:00
Yingwen
e0384a7d46 feat: overwrites inferred compaction window by region options (#5396)
* feat: use time window in compaction options for compaction window

* test: add tests for overwriting options

* chore: typo

* chore: fix a grammar issue in log
2025-01-18 14:53:56 +00:00
Ning Sun
d73815ba84 feat: pipeline dispatch part 1, add definition (#5359)
* feat: add dispatcher definition

* feat: add dispatcher element in pipelien definition

* fmt: correct format

* test: add negative tests

* fmt: fix format

* refactor: replace consts

* feat: add tostring for dispatcher

* refactor: remove to_string which is actually debug

* Update src/pipeline/src/dispatcher.rs

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-18 11:47:17 +00:00
yihong
c78a492863 refactor: drop useless clone for some better performance using static chek (#5388)
refactor: drop useless clone for some better performance using static check

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-17 13:50:37 +00:00
yihong
52697a9e66 fix: maybe double free from static str in Snafu (#5383)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2025-01-17 08:24:26 +00:00
discord9
f8d26b433e fix: make flow worker actually run in parallel (#5384)
* fix: make flow worker actually run in parallel

* chore: check for underflow

* fix: del duplicate sub

* fix: print server handle error
2025-01-16 14:32:33 +00:00
Zhenchi
1acfb6ed1c feat!: use indirect indices for bloom filter to reduce size (#5377)
* feat!(bloom-filter): use indirect indices to reduce size

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix format

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* update proto

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* nit

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* upgrade proto

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
2025-01-16 13:18:29 +00:00
Yingwen
7eaabb3ca2 fix: increase in progress scan gauge and adjust histogram buckets (#5370)
* fix: in progress scan doesn't inc

* feat(mito): adjust mito histogram buckets

* chore(metric-engine): adjust metric engine histogram bucket
2025-01-16 12:53:03 +00:00
Ruihang Xia
8d5d4000e6 feat: set default compaction parallelism (#5371)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-16 11:16:56 +00:00
Ruihang Xia
a598008ec3 fix: panic when received invalid query string (#5366)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-16 11:05:46 +00:00
Ruihang Xia
86bd54194a feat: digest pipeline processor (#5323)
* feat: basic impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply code review comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Apply suggestions from code review

* follow the naming master

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-16 09:46:37 +00:00
Weny Xu
ccd2b06b7a fix!: correct index cache config (#5381)
* fix: correct index cache config

* chore: update config.md
2025-01-16 09:46:02 +00:00
discord9
317fe9eaa5 feat: flow's http server (#5372)
* feat: flow's http server

* feat: add cli options for http addr

* test: sqlness runner http addr

* feat: metrics

* chore: also shutdown http server
2025-01-16 07:25:30 +00:00
Lanqing Yang
a4761d6245 feat: Alter inverted index (#5131)
feat: support alter inverted index.
2025-01-16 07:09:27 +00:00
discord9
758aef39d8 feat: filter batch by sequence in memtable (#5367)
* feat: add seq field

* feat: filter by sequence

* chore: per review

* docs: explain why not prune

* chore: correct doc

* test: test filter by seq
2025-01-16 04:44:28 +00:00
ZonaHe
4e3dd04f42 feat: update dashboard to v0.7.9-rc.1 (#5368)
feat: update dashboard to v0.7.9-rc

Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2025-01-16 03:43:38 +00:00
Weny Xu
18b77408ae feat: introduce SparsePrimaryKeyCodec and SparsePrimaryKeyFilter (#5365)
* feat(mito): introduce `SparseRowCodec`

* feat(mito): introduce `SparsePrimaryKeyFilter`

* chore: apply suggestions from CR
2025-01-16 03:10:43 +00:00
Ning Sun
725d5a9e68 fix: redirect /dashboard to /dashboard/ (#5369)
* fix: redirect /dashboard to /dashboard/

* test: update integration test
2025-01-16 03:04:08 +00:00
Ruihang Xia
4f29e50ef3 feat: refine log query AST (#5316)
* draft

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl planner part

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* feat: tweak aggr func

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* todo about context

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename log expr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sign todo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-15 12:11:24 +00:00
discord9
0185a65905 feat(flow): refill flow task def(Part 2) (#5317)
* feat: refill task def

* chore: per review

* chore: after rebase
2025-01-15 08:48:54 +00:00
yihong
7e61d1ae27 feat: support pg_database for DBeaver. (#5362)
This patch support pg_database for pg_catalog, also add query replace,
in fixtures.rs for the reason that datafusion do not support sql like
'select 1,1;' more can check issue #5344.

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-15 07:05:34 +00:00
yihong
e56dd20426 fix: panic and interval when do not have keyword interval (#5339)
* fix: panic and interval when do not have keyword

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: wrong pos...

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments drop the unreachable

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments and add sqlness tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-15 06:42:24 +00:00
Weny Xu
b64c075cdb feat: introduce PrimaryKeyEncoding (#5312)
* feat: introduce `PrimaryKeyEncoding`

* fix: fix unit tests

* chore: add empty line

* test: add unit tests

* chore: fmt code

* refactor: introduce new codec trait to support various encoding

* fix: fix unit tests

* chore: update sqlness result

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2025-01-15 06:16:53 +00:00
Weny Xu
57f8afcb70 chore: avoid sending create table requests for already existing tables (#5347)
* chore: avoid sending create table requests for already existing tables

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2025-01-15 02:50:25 +00:00
Ruihang Xia
bd37e086c2 fix: improve error handling in pipeline and log in log store (#5357)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-01-14 13:58:30 +00:00
discord9
66f63ae981 feat: more than one flow workers (#5315)
* feat: more workers

* feat: use round robin

* refactor: per review

* refactor: per bot review

* chore: per review

* docs: example

* docs: update config.md

* docs: update

* chore: per review

* refactor: set workers to cpu/2.max(1)

* fix: flow config in standalone mode

* test: fix config test

* docs: update docs&opt name

* chore: update config.md

* refactor: per review, sanitize at top

* chore: per review

* chore: config.md
2025-01-14 12:28:18 +00:00
discord9
95b20592ac fix: handle insert default value (#5307)
* fix: handle flow inserts with default values

* test: sqlness

* chore: typo

* chore: newline

* feat(WIP): impure default filler

* feat: fill impure default values

* test: add test for default fill impure

* feat: check for impure

* fix: also handle stmt to region

* refactor: per review

* refactor: per review

* chore: rebase fix

* chore: clippy

* chore: per review
2025-01-14 09:06:53 +00:00
ZonaHe
1855dccdf1 feat: update dashboard to v0.7.8 (#5355)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2025-01-14 09:00:08 +00:00
Ning Sun
f5829364a2 fix: security fix, sqlx, hashbrown, idna and CI updates (#5330)
* fix: security fix, sqlx, hashbrown, idna

* ci: optimize ci cache generation

* feat: update pprof
2025-01-14 08:19:33 +00:00
Xuanwo
87bd12d6df refactor: Bump opendal to 0.51.1 (#5354)
* refactor: Bump opendal to 0.51.1

Signed-off-by: Xuanwo <github@xuanwo.io>

* Ignore dirs from cache

Signed-off-by: Xuanwo <github@xuanwo.io>

* Reduce extra alloc

Signed-off-by: Xuanwo <github@xuanwo.io>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
2025-01-14 07:28:09 +00:00
ZonaHe
c370b4b40d feat: update dashboard to v0.7.7 (#5350)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2025-01-13 07:35:12 +00:00
zyy17
3f01f67f94 refactor(elasticsearch): use _index as greptimedb table in log ingestion and add /${index}/_bulk API (#5335)
* refactor(elasticsearch): use `_index` as greptimedb table in log ingestion and add `/${index}/_bulk` API

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
2025-01-13 06:58:04 +00:00
Weny Xu
6eb746d994 fix: skip building indexer when indexed_column_ids are empty (#5348) 2025-01-13 05:15:20 +00:00
yihong
03a144fa56 chore: drop useless import raw_normalize_path in object-store lib (#5349)
chore: drop use less import raw_normalize_path in store lib

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2025-01-13 04:08:36 +00:00
Yohan Wal
f069ea082f fix(test): use different table name (#5334) 2025-01-10 17:09:23 +00:00