rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-07 13:52:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,489 Commits 104 Branches 226 Tags
f0bec4940f635b56f97625e0fcafcc7822e52d6d
Commit Graph

4489 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yihong
f0bec4940f fix: two label_replace different from promql (#6720) 
* fix: two label_replace different from promql

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>

* fix: another address

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
 2025-08-13 06:27:49 +00:00
yihong
5eb491df12 fix: label_join should work with unknown (#6714) 
* fix: label_join should work with unknown

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>

* fix: address forget comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: Jiachun Feng <jiachun_feng@proton.me>
 2025-08-13 03:45:40 +00:00
Weny Xu
1d84e802d8 feat: add integration tests for table reconciliation procedures part1 (#6705) 
* feat: add integration tests for table reconciliation procedures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-13 03:29:39 +00:00
Ruihang Xia
2992e70393 fix: correct offset's symbol (#6728) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-13 03:20:47 +00:00
Ruihang Xia
8a44137f37 chore: prefix debug_assertion only variables with underscore (#6727) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-13 02:50:16 +00:00
zyy17
777da35b0d refactor: unify the event recorder (#6689) 
* refactor: unify the event recorder

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `table_name()` in `Event` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `slow_query_options` in `Instance`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `EventHandlerOptions` and `options()` in `EventHandler` trait

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: add `aggregate_events_by_type()` and support log mode of slow query

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: support to set ttl by using extension of query context

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine the configs fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: sqlness test errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use `Duration` type instead of `String` for ttl fields

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: use pre-allocation for building RowInsertRequests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix clippy errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: code review

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: polish code for `group_events_by_type()` and `build_row_inserts_request()`, also add the unit tests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: refine comments

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-12 18:26:12 +00:00
Ruihang Xia
9ad9a7d2bc feat: add all partition column to logical table automatically (#6711) 
* feat: add all partition column to logical table automatically

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify builder

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix request builder

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
 2025-08-12 17:25:24 +00:00
Lei, HUANG
ff5d672583 chore: impl cast from primitives to PathType (#6724) 
chore/impl-cast-to-primitives-for-path-type:
 ### Add `num_enum` for Enum Conversion and Update `PathType`

 - **Added `num_enum` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `num_enum` for enum conversion functionality.
   - Files: `Cargo.lock`, `src/store-api/Cargo.toml`

 - **Enhanced `PathType` Enum**: Implemented `TryFromPrimitive` for `PathType` to enable conversion from primitive types.
   - Files: `src/store-api/src/region_request.rs`

 - **Added Unit Tests**: Introduced tests to verify the conversion of `PathType` enum to and from primitive types.
   - Files: `src/store-api/src/region_request.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2025-08-12 13:07:01 +00:00
Ruihang Xia
e495c614f7 perf: improve bloom filter reader's byte reading logic (#6658) 
* perf: improve bloom filter reader's byte reading logic

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* revert toml change

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clearify comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update lock file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* pub util fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* note endian

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-12 11:37:25 +00:00
Ning Sun
e80e4a9ed7 fix: update pgwire to fix windows timeout issue (#6710) 
* test: reproduce windows ci issue

* chore: update sqlx

* chore: update pgwire

* chore: update to a debug version of pgwire

* fix: update pgwire to resolve peek after read on windows

* ci: remove windows task from regular ci
 2025-08-12 08:24:58 +00:00
Yingwen
1977ae50ee feat: Projection mapper for flat schema (#6679) 
* feat: plain projection mapper wip

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: change ProjectionMapper to enum

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: convert plain batch

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add tests for the mapper

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: allow dead code

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: format code

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix compiler errors

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: change PlainProjectionMapper to FlatProjectionMapper

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: fix projection tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fmt code

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: Address  comments

Removes some unwrap()

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fmt

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address comment

as_plain -> as_flat

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-12 07:36:23 +00:00
yihong
5cec0d4e3a fix: http and tql should return the same value for nuknown (#6718) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-08-12 06:38:01 +00:00
fys
d2d6489b2f fix: unit test about trigger labels parse (#6716) 2025-08-12 04:43:42 +00:00
Ruihang Xia
25f926ea7d feat: mito region staging state (#6664) 
* fix: not mark all deleted when partial trunc (#6654)

* fix: not mark all deleted when partial trunc&not update manifest when partial file range is empty

Signed-off-by: discord9 <discord9@163.com>

* docs: note

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* some tests and DdlRequest

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* stage transit

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* address CR comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* correct error type

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
 2025-08-12 03:17:47 +00:00
discord9
f159fcf599 fix: metrics without physical partition columns query push down (#6694) 
* fix: metrics no part cols

Signed-off-by: discord9 <discord9@163.com>

* chore: typos

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* chore: rename stuff

Signed-off-by: discord9 <discord9@163.com>

* refactor: put partition rules in table

Signed-off-by: discord9 <discord9@163.com>

* more tests

Signed-off-by: discord9 <discord9@163.com>

* test: redact more

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-08-12 02:51:38 +00:00
Yingwen
e4454e0c7d feat: Implements last row dedup strategy for flat format (#6695) 
* feat: implement FlatLastRow dedup strategy

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: fix dedup metrics

Add tests for iter with FlatLastRow strategy

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix delete metrics

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address review comments

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: move BatchLastRow position

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix typos

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-11 07:44:45 +00:00
Ruihang Xia
0781adaa3d feat: new HTTP API for formatting SQL (#6691) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-09 07:39:41 +00:00
LFC
253d89b5cc feat: able to set read preference to flownode (#6696) 
fix: correctly compare the opened follower regions in startup

Signed-off-by: luofucong <luofc@foxmail.com>
 2025-08-08 09:08:09 +00:00
localhost
3a2f5413e0 chore: add and/or for log query (#6681) 
* chore: add and/or for log query

* chore: remove impl From<Vec<ColumnFilters>> for Filters
 2025-08-08 08:48:03 +00:00
Yingwen
214ffe7109 feat: Implements an iterator to merge RecordBatches (#6666) 
* feat: Implements the sync merge iterator for RecordBatch

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: test merge iterator

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: fix incorrect column index

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix typos

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-08 07:03:22 +00:00
Ruihang Xia
3b1f172ab8 fix: TQL CTE parser take raw query string (#6671) 
* take raw TQL part

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add order by

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* more order by

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add comment back

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-08 06:22:56 +00:00
LFC
0215b39f61 fix: correctly set extension range source index (#6692) 
refactor: extract the common codes of creating proto ColumnSchema and Row to helper functions

fix: explicitly set the follower max sequence when finding extension ranges to avoid potential concurrency hazard

Signed-off-by: luofucong <luofc@foxmail.com>
 2025-08-08 06:17:25 +00:00
Weny Xu
01dc789816 refactor: refine error status code mappings (#6678) 
Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-07 09:29:32 +00:00
Ning Sun
bbe48e9e8b feat: update pgwire to 0.32 (#6674) 
* feat: update pgwire api

* feat: update pgwire and override on_query/on_execute

* feat: update pgwire to 0.32

* chore: remove code example

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
 2025-08-07 06:17:52 +00:00
Weny Xu
e2015ce1af feat(metric-engine): add metadata region cache (#6657) 
* feat(metric-engine): add metadata region cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: use lru

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: default ttl

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: longer ttl

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-07 06:16:23 +00:00
Yingwen
7bb765af1d chore: pub access layer (#6670) 
Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-06 13:34:38 +00:00
discord9
080b4b5d53 docs(rfc): rfc for gc worker (#6572) 
* docs: rfc for gc worker

Signed-off-by: discord9 <discord9@163.com>

* revise: gc worker now run after compaction&long run queries check

Signed-off-by: discord9 <discord9@163.com>

* more drawback

Signed-off-by: discord9 <discord9@163.com>

* flowchart

Signed-off-by: discord9 <discord9@163.com>

* read consist optional&more detail

Signed-off-by: discord9 <discord9@163.com>

* chore: rephrase

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-08-06 12:03:28 +00:00
Weny Xu
c7c8495a6b feat: add metrics for reconciliation procedures (#6652) 
* feat: add metrics for reconciliation procedures

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: improve error handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(datanode): handle ignore_nonexistent_region flag in open_all_regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: merge metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-06 11:24:03 +00:00
Yingwen
bbab35f285 perf: Reduce fulltext bloom load time (#6651) 
* perf: cached reader do not get page concurrently

Otherwise they will all fetch the same pages in parallel

Signed-off-by: evenyag <realevenyag@gmail.com>

* perf: always disable zstd for bloom

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-06 08:25:31 +00:00
Zhenchi
6c6487ab30 chore: bump version to 0.17.0 (#6663) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-08-06 07:45:39 +00:00
Ruihang Xia
757694ae38 feat: count underscore in English tokenizer and improve performance (#6660) 
* feat: count underscore in English tokenizer and improve performance

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update lock file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* assert lookup table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle utf8 alphanumeric

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* finalize

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-06 07:23:18 +00:00
Yingwen
39e2f122eb feat: EncodedBulkPartIter iters flat format and returns RecordBatch (#6655) 
* feat: implements iter to read bulk part

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: BulkPartEncoder encodes BulkPart instead of mutation

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-06 06:50:01 +00:00
Lei, HUANG
877ce6e893 chore: add methods to catalog manager (#6656) 
* chore/optimize-catalog:
 ### Add `table_id` Method to `CatalogManager`

 - **Files Modified**:
   - `src/catalog/src/kvbackend/manager.rs`
   - `src/catalog/src/lib.rs`

 - **Key Changes**:
   - Introduced a new asynchronous method `table_id` in the `CatalogManager` trait to retrieve the table ID based on catalog, schema, and table name.
   - Implemented the `table_id` method in `KvBackendCatalogManager` to fetch the table ID from the system catalog or cache, with a fallback to `pg_catalog` for Postgres channels.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/optimize-catalog:
 ### Add `table_info_by_id` Method to Catalog Managers

 - **`manager.rs`**: Introduced the `table_info_by_id` method in `KvBackendCatalogManager` to retrieve table information by table ID using the `TableInfoCacheRef`.
 - **`lib.rs`**: Updated the `CatalogManager` trait to include the new `table_info_by_id` method.
 - **`memory/manager.rs`**: Implemented the `table_info_by_id` method in `MemoryCatalogManager` to fetch table information by table ID from in-memory catalogs.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2025-08-06 06:25:32 +00:00
Ruihang Xia
c8da35c7e5 feat(log-query): support binary op, scalar fn & is_true/is_false (#6659) 
* rename symbol

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle binary op

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test results

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/query/src/log_query/planner.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* reduce duplication

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2025-08-06 04:38:25 +00:00
Ruihang Xia
309e9d978c feat: support TQL CTE in planner (#6645) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-06 04:07:38 +00:00
zyy17
3a9f0220b5 fix: unable to record slow query (#6590) 
* refactor: add process manager for prometheus query

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: modify `register_query()` API to accept parsed statement(`catalog::process_manager::QueryStatement`)

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add the slow query timer in the `Tikcet` of ProcessManager

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* test: add integration tests

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add process manager in `do_exec_plan()`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* tests: add `test_postgres_slow_query` integration test

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: create a query ticket and slow query timer if the statement is a query in `query_statement()`

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* fix: sqlness errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-06 03:35:12 +00:00
zyy17
cc35bab5e4 feat: record the migration events in metasrv (#6579) 
* feat: collect procedure event

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* feat: collect region migration events

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* test: add integration test

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix docs error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: fix integration test error

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: change status code for errors

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: add `event()` in Procedure

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* refactor: improve trait design

1. Add `user_metadata()` in `Procedure` trait;

2. Add `Eventable` trait;

Signed-off-by: zyy17 <zyylsxm@gmail.com>

* chore: polish the code

Signed-off-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2025-08-06 03:30:33 +00:00
Ruihang Xia
414db41219 fix: box Explain node in Statement to reduce stack size (#6661) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-06 02:24:45 +00:00
Ruihang Xia
ea024874e7 feat: use column expr with filters in LogQuery (#6646) 
* feat: use column expr with filters in LogQuery

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove some clone

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-05 18:35:09 +00:00
discord9
e64469bbc4 fix: not mark all deleted when partial trunc (#6654) 
* fix: not mark all deleted when partial trunc&not update manifest when partial file range is empty

Signed-off-by: discord9 <discord9@163.com>

* docs: note

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-08-05 11:45:47 +00:00
discord9
875207d26c feat: register all aggregate function to auto step aggr fn (#6596) 
* feat: support generic aggr push down

Signed-off-by: discord9 <discord9@163.com>

* typo

Signed-off-by: discord9 <discord9@163.com>

* fix: type ck in merge wrapper

Signed-off-by: discord9 <discord9@163.com>

* test: update sqlness

Signed-off-by: discord9 <discord9@163.com>

* feat: support all registried aggr func

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-08-05 11:37:45 +00:00
jeremyhi
9871c22740 fix: sequence peek with remote value (#6648) 
* fix: sequence peek with remote value

* chore: more ut

* chore: add more ut
 2025-08-05 08:28:09 +00:00
Yingwen
50f7f61fdc feat: Implements an iterator to read the RecordBatch in BulkPart (#6647) 
* feat: impl RecordBatchIter for BulkPart

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename BulkPartIter to EncodedBulkPartIter

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: add iter benchmark

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: filter by primary key columns

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: move struct definitions

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: bulk iter for flat schema

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: iter filter benchmark

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix compiler errors

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: use corrent sequence array to compare

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove RecordBatchIter

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update comments

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: apply projection first

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address comment

No need to check number of rows after filter

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-05 08:11:28 +00:00
Ruihang Xia
9c3b83e84d feat: use real data to truncate manipulate range (#6649) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-08-05 04:55:24 +00:00
Yingwen
e81d0f5861 feat: implements FlatReadFormat to project parquets with flat schema (#6638) 
* feat: add plain read format

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: reduce unused code

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: reuse code

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: allow dead code

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: change ReadFormat to enum

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: as_primary_key() returns option

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove some allow dead_code

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename WriteFormat to PrimaryKeyWriteFormat

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add tests for read/write format

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: format code

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: dedup column ids in format

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename plain to flat

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: implements FlatReadFormat based on the new format

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: fix tests

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: support override sequence

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: new_override_sequence_array for ReadFormat

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: update comments

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address comment

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-04 12:43:50 +00:00
Ning Sun
29e0092468 feat: schema/database support for label_values (#6631) 
* feat: initial support for __schema__ in label values

* feat: filter database with matches

* refactor: skip unnecessary check

* fix: resolve schema matcher in label values

* test: add a test case for table not exists

* refactor: add matchop check on db label

* chore: merge main
 2025-08-04 11:56:10 +00:00
Weny Xu
67a93a07a2 fix: fix sequence peek method to return correct values when sequence is not initialized (#6643) 
fix: improve sequence peek method to handle uninitialized sequences

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-04 11:31:06 +00:00
discord9
1afa0afc67 feat: add partial truncate (#6602) 
* feat: add partial truncate

Signed-off-by: discord9 <discord9@163.com>

* fix: per review

Signed-off-by: discord9 <discord9@163.com>

* feat: add proto partial truncate kind

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* chore: update branched proto

Signed-off-by: discord9 <discord9@163.com>

* feat: grpc support truncate WIP sql support

Signed-off-by: discord9 <discord9@163.com>

* wip: parse truncate range

Signed-off-by: discord9 <discord9@163.com>

* feat: truncate by range

Signed-off-by: discord9 <discord9@163.com>

* fix: truncate range display

Signed-off-by: discord9 <discord9@163.com>

* chore: resolve todo

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* test: more invalid parse

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: unused

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: update branch

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-08-04 10:50:27 +00:00
Weny Xu
414101fafa feat: introduce reconciliation interface (#6614) 
* feat: introduce reconcile interface

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: upgrade proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-08-04 09:12:48 +00:00
Yingwen
280024d7f8 feat: Add option to limit the files reading simultaneously (#6635) 
* feat: limits the max number of files to scan at the same time

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: make max_concurrent_scan_files configurable

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: reduce concurrent scan files to 128

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update config example

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: add test for max_concurrent_scan_files

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update config test

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-08-04 07:18:58 +00:00