Commit Graph

4029 Commits

Author SHA1 Message Date
evenyag
6293bb1f5b feat: try send before send
reduce the send timeout to 10ms
2025-04-24 06:03:06 +08:00
Ruihang Xia
8fa1ebcc3e assign partition_ranges
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-04-24 06:03:06 +08:00
Ruihang Xia
c18c3f5839 Revert "feat: keep parallelize_scan unchanged"
This reverts commit 96ba00d175.
2025-04-24 06:03:06 +08:00
evenyag
629e72d8c0 chore: update comment 2025-04-24 06:03:06 +08:00
evenyag
e4065505ab feat: use smallvec 2025-04-24 06:03:05 +08:00
evenyag
aafd164483 fix: include build merge reader cost to scan cost 2025-04-24 06:03:05 +08:00
evenyag
1386e903d6 fix: address compiler errors 2025-04-24 06:03:05 +08:00
evenyag
12692a940c feat: keep parallelize_scan unchanged 2025-04-24 06:03:05 +08:00
evenyag
4d44cbb8b2 fix: use series scan in PerSeries distribution 2025-04-24 06:03:05 +08:00
evenyag
f4911aa3bb refactor: remove per series scan from SeqScan 2025-04-24 06:03:05 +08:00
evenyag
5ac61f17bc feat: parallelize PerSeries 2025-04-24 06:03:05 +08:00
evenyag
e0d34c6d95 feat: use series scan when distribution is PerSeries 2025-04-24 06:03:04 +08:00
evenyag
8a98b9c433 feat: implement scan logic of each partition 2025-04-24 06:03:03 +08:00
evenyag
1f5d36a203 chore: add to scanner enum 2025-04-24 05:57:37 +08:00
evenyag
6fc7168893 chore: basic methods for SeriesScan 2025-04-24 05:57:37 +08:00
Weny Xu
f3d000f6ec feat: track region failover attempts and adjust timeout (#5952) 2025-04-23 18:19:18 +00:00
discord9
9557b76224 fix: try prune one less (#5965)
* try prune one less

* test: also not add one

* ci: use longer fuzz time

* revert fuzz time&per review

* chore: no (

* docs: add explain to offset used in delete records

* test: fix test_procedure_execution
2025-04-23 16:57:54 +00:00
discord9
a0900f5b90 feat(flow): use batching mode&fix sqlness (#5903)
* feat: use flow batching engine

broken: try using logical plan

fix: use dummy catalog for logical plan

fix: insert plan exec&sqlness grpc addr

feat: use frontend instance in flownode in standalone

feat: flow type in metasrv&fix: flush flow out of sync& column name alias

tests: sqlness update

tests: sqlness flow rebuild udpate

chore: per review

refactor: keep chnl mgr

refactor: use catalog mgr for get table

tests: use valid sql

fix: add more check

refactor: put flow type determine to frontend

* chore: update proto

* chore: update proto to main branch

* fix: add locks for create/drop flow&docs: update docs

* feat: flush_flow flush all ranges now

* test: add align time window test

* docs: explain `nodeid` use in check task

* refactor: AddAutoColumnRewriter check for Projection

* refactor: per review

* fix: query without time window also clean dirty time window

* chore: better logging

* chore: add comments per review

* refactor: per review

* chore: per review

* chore: per review rename args

* refactor: per review partially

* chore: update docs

* chore: use better error variant

* chore: better error variant

* refactor: rename FlowWorkerManager to FlowStreamingEngine

* rename again

* refactor: per review

* chore: rebase after #5963 merged

* refactor: rename all flow_worker_manager occurs

* docs: rm resolved TODO
2025-04-23 15:12:16 +00:00
Yingwen
45a05fb08c docs: fix some units and adds the opendal errors panel (#5962)
* docs: fixes units in the dashboard

* docs: add opendal errors panel

* docs: opendal traffic use decbytes

* docs: update readme

---------

Co-authored-by: zyy17 <zyylsxm@gmail.com>
2025-04-23 13:31:29 +00:00
LFC
71db79c8d6 feat: node excluder (#5964)
* feat: node excluder

* fix ci

* fix ci
2025-04-23 10:48:46 +00:00
discord9
79ed7bbc44 fix: store flow query ctx on creation (#5963)
* fix: store flow schema on creation

* chore: update sqlness

* refactor: save the entire query context to flow info

* chore: sqlness update

* chore: rm pub

* fix: keep old version compatibility
2025-04-23 09:59:09 +00:00
zyy17
02e9a66d7a chore: update dac tools image and docs (#5961) 2025-04-23 05:00:37 +00:00
Weny Xu
55cadcd2c0 feat: introduce flush metadata region task for metric engine (#5951)
* feat: introduce flush metadata region task for metric engine

* docs: generate config.md

* chore: add header

* test: fix unit test

* fix: fix unit tests

* chore: apply suggestions from CR

* chore: remove docs

* fix: fix unit tests
2025-04-23 04:51:22 +00:00
fys
8c4796734a chore: remove unused attribute (#5960) 2025-04-23 03:17:13 +00:00
Yuhan Wang
919956999b fix: use max in flushed entry id and topic latest entry id (#5946) 2025-04-22 23:48:32 +00:00
ZonaHe
7e5f6cbeae feat: update dashboard to v0.9.0 (#5948)
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
2025-04-22 11:35:33 +00:00
shuiyisong
5c07f0dec7 refactor: run_pipeline parameters (#5954)
* refactor: simplify run_pipeline params

* refactor: remove unnecessory function wrap
2025-04-22 11:34:19 +00:00
discord9
9fb0487e67 fix: parse flow expire after interval (#5953)
* fix: parse flow expire after interval

* fix: correct 30.44&comments
2025-04-22 08:44:04 +00:00
discord9
6e407ae4b9 test: use random seed for window sort fuzz test (#5950)
tests: use random seed for window sort fuzz test
2025-04-22 08:14:27 +00:00
Ning Sun
bcefc6b83f feat: add format support for promql http api (not prometheus) (#5939)
* feat: add format support for promql http api (not prometheus)

* test: add csv format test
2025-04-22 08:10:35 +00:00
Weny Xu
0f77135ef9 feat: add exclude_peer_ids to SelectorOptions (#5949)
* feat: add `exclude_peer_ids` to `SelectorOptions`

* chore: apply suggestions from CR

* fix: clippy
2025-04-22 07:49:22 +00:00
Weny Xu
0a4594c9e2 fix: remove obsolete failover detectors after region leader change (#5944)
* fix: remove obsolete failover detectors after region leader change

* chore: apply suggestions from CR

* fix: fix unit tests

* fix: fix unit test

* fix: failover logic
2025-04-22 06:15:47 +00:00
LFC
d9437c6da7 chore: assert plugin uniqueness (#5947) 2025-04-22 06:04:06 +00:00
zyy17
35f4fa3c3e refactor: unify all dashboards and use dac tool to generate intermediate dashboards (#5933)
* refactor: split cluster metrics into multiple dashboards

* chore: merge multiple dashboards into one dashboard

* refactor: add 'dac' tool to generate a intermediate dashboards

* refactor: generate markdown docs for dashboards
2025-04-22 06:03:01 +00:00
jeremyhi
60e4607b64 chore: better buckets for heartbeat stat size histogram (#5945)
chore: better buckets for METRIC_META_HEARTBEAT_STAT_MEMORY_SIZE
2025-04-21 16:12:27 +00:00
shuiyisong
3b8c6d5ce3 chore: use once_cell to avoid parse everytime in pipeline exec (#5943)
* chore: use once_cell to avoid parse everytime

* chore: remove pub on options
2025-04-21 12:55:48 +00:00
Weny Xu
7a8e1bc3f9 feat: support building metasrv with selector from plugins (#5942)
* chore: expose selector

* feat: use f64

* chore: expose selector::common

* feat: build metasrv with selector from plugins
2025-04-21 10:59:24 +00:00
Yuhan Wang
ee07b9bfa8 test: update configs to enable auto wal prune (#5938)
* test: update configs to enable auto wal prune

* fix: add humantime_serde

* fix: enable overwrite_entry_start_id

* fix: not in metasrv

* chore: update default value name

* Apply suggestions from code review

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* fix: kafka use overwrite_entry_start_id

---------

Co-authored-by: jeremyhi <jiachun_feng@proton.me>
2025-04-21 07:57:43 +00:00
Lei, HUANG
90ffaa8a62 feat: implement otel-arrow protocol for GreptimeDB (#5840)
* [wip]: implement arrow service

* add service

* feat/otel-arrow:
 ### Add OpenTelemetry Arrow Support

 - **`Cargo.toml`, `Cargo.lock`**: Updated `otel-arrow-rust` dependency to use a local path and added `arrow-ipc` as a dependency.
 - **`src/servers/src/grpc.rs`, `src/servers/src/grpc/builder.rs`**: Integrated `ArrowMetricsServiceServer` with gRPC server, including support for custom header interception and message compression.
 - **`src/servers/src/otel_arrow.rs`**: Implemented `OtelArrowServiceHandler` for handling OpenTelemetry Arrow metrics and added `HeaderInterceptor` for custom header handling.

* feat/otel-arrow:
 Add error handling for OpenTelemetry Arrow requests

 - **`src/error.rs`**: Introduced a new error variant `HandleOtelArrowRequest` to handle failures in processing OpenTelemetry Arrow requests.
 - **`src/otel_arrow.rs`**: Implemented error handling for receiving and consuming batches from the OpenTelemetry Arrow client. Added logging for errors and updated the response status accordingly.

* feat/otel-arrow:
 Remove `otel_arrow` Module from gRPC Server

 - Deleted the `otel_arrow` module from the gRPC server implementation.
 - Removed the `otel_arrow` module import from `grpc.rs`.
 - Deleted the `otel_arrow.rs` file, which contained the `OtelArrowServer` struct and its implementation.

* feat/otel-arrow:
 ## Remove `Arc` Implementations for Protocol and Pipeline Handlers

 - **Removed `Arc` Implementations**: Deleted `Arc` implementations for `OpenTelemetryProtocolHandler` and `PipelineHandler` traits in `query_handler.rs`. This change simplifies the code by removing redundant async trait implementations for `Arc<T>`.
 - **File Affected**: `src/servers/src/query_handler.rs`

* feat/otel-arrow:
 Improve error handling and metadata processing in `otel_arrow.rs`

 - Updated error handling by ignoring the result of `sender.send` to prevent panic on failure.
 - Enhanced metadata processing in `HeaderInterceptor` by using `Ok` to safely handle `grpc-encoding` entry retrieval.

* fix dependency

* feat/otel-arrow:
 - **Update Dependencies**:
   - Moved `otel-arrow-rust` dependency in `Cargo.toml`.
   - Adjusted workspace dependencies in `src/frontend/Cargo.toml`.

 - **Error Handling**:
   - Removed `MissingQueryContext` error variant from `src/servers/src/error.rs`.

* fix: toml format

* remove useless code

* chore: resolve conflicts
2025-04-21 07:24:23 +00:00
Yingwen
56f319a707 fix: filter doesn't consider default values after schema change (#5912)
* test: sqlness test case

* feat: use correct default while pruning row groups

* fix: consider default in SimpleFilterContext

* test: update sqlness test

* test: add order by
2025-04-21 06:32:26 +00:00
shuiyisong
9df493988b fix: wrong error msg in pipeline (#5937) 2025-04-21 04:05:46 +00:00
dennis zhuang
ad1b77ab04 feat: update readme (#5936)
* fix: title

* chore: format

* chore: format

* chore: format
2025-04-21 02:44:44 +00:00
Yuhan Wang
e817a65d75 feat: enable submitting wal prune procedure periodically (#5867)
* feat: enable submitting wal prune procedure periodically

* chore: fix and add options

* test: add unit test

* test: fix unit test

* test: enable active_wal_pruning in test

* test: update default config

* chore: update config name

* refactor: use semaphore to control the number of prune process

* refactor: use split client for wal prune manager and topic creator

* chore: add configs

* chore: apply review comments

* fix: use tracker properly

* fix: use guard to track semaphore

* test: update unit tests

* chore: update config name

* chore: use prunable_entry_id

* refactor: semaphore to only limit the process of submitting

* chore: remove legacy sort

* chore: better configs

* fix: update config.md

* chore: respect fmt

* test: update unit tests

* chore: use interval_at

* fix: fix unit test

* test: fix unit test

* test: fix unit test

* chore: apply review comments

* docs: update config docs
v0.14.0-nightly-20250421
2025-04-18 16:02:33 +00:00
Yuhan Wang
41814bb49f feat: introduce high_watermark for remote wal logstore (#5877)
* feat: introduce high_watermark_since_flush

* test: add unit test for high watermark

* refactor: submit a request instead

* fix: send reply before submit request

* fix: no need to update twice

* feat: update high watermark in background periodically

* test: update unit tests

* fix: update high watermark periodically

* test: update unit tests

* chore: apply review comments

* chore: rename

* chore: apply review comments

* chore: clean up

* chore: apply review comments
2025-04-18 12:10:47 +00:00
Weny Xu
1e394af583 feat: prevent migrating a leader region to a peer that already has a region follower (#5923)
* feat: prevent migrating a leader region to a peer that already has a region follower

* chore: refine err msg
2025-04-18 11:13:01 +00:00
discord9
a9065f5319 chore: rm dev opt level 3 (#5932)
remove accidentally added dev profile opt level 3 for depend
2025-04-18 11:04:19 +00:00
Weny Xu
b8c6f1c8ed feat: sync region followers after altering regions (#5901)
* feat: close follower regions after dropping leader regions

* chore: upgrade greptime-proto

* feat: sync region followers after alter region operations

* test: add tests

* chore: apply suggestions from CR

* chore: apply suggestions from CR
2025-04-18 10:21:35 +00:00
Ruihang Xia
115e5a03a8 fix: anchor regex string to fully match in promql (#5920)
* fix: anchor regex string to fully match in promql

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test result again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-04-18 10:13:45 +00:00
Yingwen
a5c443f734 perf: keep compiled regex in SimpleFilterEvaluator to avoid re-compiling (#5919)
* feat: cache regex in evaluator

* chore: fix warnings

* chore: add reference

* refactor: address CR comments

* Add negative to state
* Don't create the evaluator if the regex is invalid

* test: add test for maybe_build_regex
2025-04-18 09:36:28 +00:00
discord9
5287b87925 docs: memory profile scripts (#5922)
* docs: memory profile scripts

* chore: typo

* chore: comment

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* chore: newline eof

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-04-18 09:28:05 +00:00