rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-06 13:22:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,797 Commits 106 Branches 226 Tags
7f48184e35fa2d5b844adc8d9780dd11d616180e
Commit Graph

3797 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
LFC
ccfa40dc41 ci: run nightly jobs only on greptimedb repo (#5505) 
ci: skip nightly ci jobs (#9)

(cherry picked from commit 345b4c30474f47a0477263bfba9894d7b4acda2d)
(cherry picked from commit dcd779cd668802fb1ea12fefb4dc3f83f34e30a2)
 2025-02-11 10:57:43 +00:00
Zhenchi
336b941113 feat: change puffin stager eviction policy (#5511) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-02-11 08:16:27 +00:00
yihong
de3f817596 fix: drop useless clone and for loop second (#5507) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-11 06:23:49 +00:00
ZonaHe
d094f48822 feat: update dashboard to v0.7.9 (#5508) 
Co-authored-by: ZonaHex <ZonaHex@users.noreply.github.com>
 2025-02-11 06:19:58 +00:00
yihong
342883e922 ci: safe ci using zizmor check (#5491) 
* ci: safe ci using zizmor check

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: lines empty

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: delete useless code

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-11 02:38:14 +00:00
Zhenchi
5be81abba3 feat: add metadata method to puffin reader (#5501) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-02-10 09:14:54 +00:00
Zhenchi
c19ecd7ea2 refactor: change traversal order during index construction (#5498) 
* refactor: change traversal order during index construction

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chain

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
 2025-02-10 06:31:35 +00:00
Ning Sun
15f4b10065 chore: revert "docs: add TM to logos" (#5495) 
* Revert "docs: add TM to logos (#4789)"

This reverts commit caf5f2c7a5.

* chore: transparent
 2025-02-10 04:00:59 +00:00
yihong
c100a2d1a6 fix: refactor pgkv using prepare_cache about 10% better (#5497) 
fix: refactor pgkv using prepare_cache about 15% better

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-10 03:59:18 +00:00
yihong
ccb1978c98 fix: close issue #5466 by do not shortcut the drop command (#5467) 
fix: close issue #5466 by do not shortcut by back it to READY when fail

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-10 03:28:34 +00:00
Ning Sun
480b05c590 feat: pipeline dispatcher part 2: execution (#5409) 
* fmt: correct format

* test: add negative tests

* feat: Add pipeline dispatching and execution output handling

* refactor: Enhance ingest function to correctly process original data values

custom table names during pipeline execution while optimizing the management of
transformed rows and multiple dispatched pipelines

* refactor: call greptime_identity with intermediate values

* fix: typo

* test: port tests to refactored apis

* refactor: adapt dryrun api call

* refactor: move pipeline execution code to a separated module

* refactor: update otlp pipeline execution path

* fmt: format imports

* fix: compilation

* fix: resolve residual issues

* refactor: address review comments

* chore: use btreemap as pipeline intermediate status trait modify

* refactor: update dispatcher to accept BTreeMap

* refactor: update identity pipeline

* refactor: use new input for pipeline

* chore: wip

* refactor: use updated prepare api

* refactor: improve error and header name

* feat: port flatten to new api

* chore: update pipeline api

* chore: fix transform and some pipeline test

* refactor: reimplement cmcd

* refactor: update csv processor

* fmt: update format

* chore: fix regex and dissect processor

* chore: fix test

* test: add integration test for http pipeline

* refactor: improve regex pipeline

* refactor: improve required field check

* refactor: rename table_part to table_suffix

* fix: resolve merge issue

---------

Co-authored-by: paomian <xpaomian@gmail.com>
v0.12.0-nightly-20250210 		2025-02-08 09:01:54 +00:00
Ruihang Xia
0de0fd80b0 feat: move pipelines to the first-class endpoint (#5480) 
* feat: move pipelines to the first-class endpoint

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change endpoints

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* prefix path with /

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update integration result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-08 03:46:31 +00:00
Yohan Wal
059cb6fdc3 feat: update topic-region map when create and drop table (#5423) 
* feat: update topic-region map

* fix: parse topic correctly

* test: add unit test forraft engine wal

* Update src/common/meta/src/ddl/drop_table.rs

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* test: fix unit tests

* test: fix unit tests

* chore: error handling and tests

* refactor: manage region-topic map in table_metadata_keys

* refactor: use WalOptions instead of String in deletion

* chore: revert unused change

* chore: follow review comments

* Apply suggestions from code review

Co-authored-by: jeremyhi <jiachun_feng@proton.me>

* chore: follow review comments

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
 2025-02-07 15:09:37 +00:00
jeremyhi
29218b5fe7 refactor!: unify the option names across all components part2 (#5476) 
* refactor: part2, replace old options in doc yaml

* chore: remove deprecated options

* chore: update config.md

* fix: ut
 2025-02-07 13:06:50 +00:00
discord9
59e6ec0395 chore: update pprof (#5488) 
dep: update pprof
 2025-02-07 11:43:40 +00:00
Lei, HUANG
79ee230f2a fix: cross compiling for aarch64 targets and allow customizing page size (#5487) 2025-02-07 11:21:16 +00:00
ozewr
0e4bd59fac build: Update Loki proto (#5484) 
* build: mv loki-api to loki-proto

* fmt: fmt toml

* fix: loki-proto using rev

---------

Co-authored-by: wangrui <wangrui@baihai.ai>
 2025-02-07 09:09:39 +00:00
Yingwen
6eccadbf73 fix: force recycle region dir after gc duration (#5485) 2025-02-07 08:39:04 +00:00
discord9
f29a1c56e9 fix: unquote flow_name in create flow expr (#5483) 
* fix: unquote flow_name in create flow expr

* chore: per review

* fix: compat with older version
 2025-02-07 08:26:14 +00:00
shuiyisong
88c3d331a1 refactor: otlp logs insertion (#5479) 
* chore: add test for selector overlapping

* refactor: simplify otlp logs insertion

* fix: use layered extracted value array

* fix: wrong len

* chore: minor renaming and update

* chore: rename

* fix: clippy

* fix: typos

* chore: update test

* chore: address CR comment & update meter-deps version
 2025-02-07 07:21:20 +00:00
yihong
79acc9911e fix: Delete statement not supported in metric engine close #4649 (#5473) 
* fix: Delete statement not supported in metric engine close #4649

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: do not include Truncate address review comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comment again

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-07 06:47:53 +00:00
Yingwen
0a169980b7 fix: lose decimal precision when using decimal type as tag (#5481) 
* fix: replicate() of decimal vector lose precision

* test: add sqlness test

* test: drop table
 2025-02-06 13:17:05 +00:00
Weny Xu
c80d2a3222 fix: introduce gc task for metadata store (#5461) 
* fix: introduce gc task for metadata kvbackend

* refactor: refine KvbackendConfig

* chore: apply suggestions from CR
 2025-02-06 12:12:43 +00:00
Ruihang Xia
116bdaf690 refactor: pull column filling logic out of mito worker loop (#5455) 
* avoid duplicated req catagorisation

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* pull column filling up

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fill columns instead of fill column

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test with metadata

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-06 11:43:28 +00:00
Ruihang Xia
6341fb86c7 feat: write memtable in parallel (#5456) 
* feat: write memtable in parallel

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* some comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unwrap

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* unwrap spawn result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use FuturesUnordered

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-06 09:29:57 +00:00
Ruihang Xia
fa09e181be perf: optimize time series memtable ingestion (#5451) 
* initialize with capacity

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* avoid collect

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* optimize zip

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename variable

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore type checking in the upper level

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change to two-step capacity

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-02-06 09:12:29 +00:00
Zhenchi
ab4663ec2b feat: add vec_add function (#5471) 
* feat: add vec_add function

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix unexpected utf8

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2025-02-06 06:48:50 +00:00
jeremyhi
fac22575aa refactor!: unify the option names across all components (#5457) 
* refactor: rename grpc options

* refactor: make the arg clearly

* chore: comments on server_addr

* chore: fix test

* chore: remove the store_addr alias

* refactor: cli option rpc_server_addr

* chore: keep store-addr alias

* chore: by comment
 2025-02-06 06:37:14 +00:00
Yingwen
0e249f69cd fix: don't transform Limit in TypeConversionRule, StringNormalizationRule and DistPlannerAnalyzer (#5472) 
* fix: do not transform exprs in the limit plan

* chore: keep some logs for debug

* feat: workaround for limit in other rules

* test: add sqlness tests for offset 0

* chore: add fixme
 2025-02-05 11:30:24 +00:00
yihong
5d1761f3e5 docs: fix memory perf command wrong (#5470) 
* docs: fix memory perf command wrong

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: better format

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: make macos right

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* docs: add jeprof install info

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
 2025-02-05 10:45:51 +00:00
Lei, HUANG
dba6da4d00 refactor(mito): Allow creating multiple files in ParquetWriter (#5291) 
* - **Refactored SST File Handling**:
   - Introduced `FilePathProvider` trait and its implementations (`WriteCachePathProvider`, `RegionFilePathFactory`) to manage SST and index file paths.
   - Updated `AccessLayer`, `WriteCache`, and `ParquetWriter` to use `FilePathProvider` for path management.
   - Modified `SstWriteRequest` and `SstUploadRequest` to use path providers instead of direct paths.
   - Files affected: `access_layer.rs`, `write_cache.rs`, `parquet.rs`, `writer.rs`.

 - **Enhanced Indexer Management**:
   - Replaced `IndexerBuilder` with `IndexerBuilderImpl` and made it async to support dynamic indexer creation.
   - Updated `ParquetWriter` to handle multiple indexers and file IDs.
   - Files affected: `index.rs`, `parquet.rs`, `writer.rs`.

 - **Removed Redundant File ID Handling**:
   - Removed `file_id` from `SstWriteRequest` and `CompactionOutput`.
   - Updated related logic to dynamically generate file IDs where necessary.
   - Files affected: `compaction.rs`, `flush.rs`, `picker.rs`, `twcs.rs`, `window.rs`.

 - **Test Adjustments**:
   - Updated tests to align with new path and indexer management.
   - Introduced `FixedPathProvider` and `NoopIndexBuilder` for testing purposes.
   - Files affected: `sst_util.rs`, `version_util.rs`, `parquet.rs`.

* chore: merge main

* refactor/generate-file-id-in-parquet-writer:
 **Enhance Logging in Compactor**

 - Updated `compactor.rs` to improve logging of compaction process.
   - Added `itertools::Itertools` for efficient string joining.
   - Moved logging of compaction inputs and outputs to the async block for better context.
   - Enhanced log message to include both input and output file names for better traceability.
 2025-02-05 09:00:54 +00:00
discord9
59b31372aa feat(cli): add proxy options (#5459) 
* feat(cli): proxy options

* refactor: map
 2025-02-05 03:24:22 +00:00
yihong
d6b8672e63 docs: the year is better to show in 2025 (#5468) 
doc: the year is better to show in 2025

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-02-05 03:01:45 +00:00
Ning Sun
deaa1f9578 ci: move components to flakes so it won't affect builders (#5464) 
* ci: move components to flakes so it won't affect builders

* chore: add gnuplot for benchmark/criterion
v0.12.0-nightly-20250203 		2025-01-31 08:55:59 +00:00
Ruihang Xia
f378d218e9 perf: optimize writing non-null primitive value (#5460) 
* avoid using arrow builder

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* optimize from_vec

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-01-30 14:53:59 +00:00
yihong
5b6279f191 fix: no need for special case since datafusion updated (#5458) 
* fix: no need for special case since datafusion updated

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: drop useless tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
v0.12.0-nightly-20250127 		2025-01-26 05:57:23 +00:00
Ning Sun
698b28c636 feat: provide options to disable or customize http corss-origin settings (#5450) 
* feat: add cors headers for http server

* test: add cors test

* test: add preflight test

* feat: allow customize http cross-origin settings

* chore: typo fix

* test: update tests

* test: fix test for config

* refactor: address review comments
 2025-01-26 03:55:34 +00:00
yihong
c4d10313e6 fix: better error handler for the time range close #5449 (#5453) 
* fix: better error handler for the time range close #5499

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: wrong compare

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-26 03:33:12 +00:00
Ruihang Xia
f165bfb0af fix: remove metric engine's internal column from promql's query (#5032) 
* fix: remove metric engine's internal column from promql's query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unwrap

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* filter out physical table

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add integration test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-01-26 03:10:59 +00:00
Weny Xu
4111c18d44 chore: avoid necessary cloning (#5454) 
* chore: avoid necessary cloning

* Apply suggestions from code review

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2025-01-25 14:16:22 +00:00
Ruihang Xia
5abe4c141a feat: expose http endpoint for flownode and metasrv (#5437) 
* feat: expose http endpoint for flownode and metasrv

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust health check

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-01-25 13:20:25 +00:00
yihong
adb5c3743c fix: flush table panic when table has interval column close #3235 (#5422) 
* fix: flash table panic when table has interval column close #3235

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* Revert "fix: flash table panic when table has interval column close #3235"

This reverts commit ffc63efda39cd6ef525313b60ede061c5ec24b12.

* fix: create table do not support interval type for now close #3235

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: sqlness

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address comments fix conflict and more tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

* fix: address final comments drop useless sqlness tests

Signed-off-by: yihong0618 <zouzou0208@gmail.com>

---------

Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-25 08:09:55 +00:00
Ruihang Xia
7c5ead90ac feat: mirror insert request to flownode in async (#5444) 
* feat: mirror insert request to flownode in async

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-01-24 13:42:27 +00:00
shuiyisong
d870987a65 chore: update toolchain to 2024-12-25 (#5430) 
* chore: update toolchain to 2024-12-25

* chore: fix clippy

* feat: update flakes

* chore: remove `rerun-if-changed` for now

* chore: update shadow-rs

* fix: clippy

* chore: update version in DEV_BUILDER_IMAGE_TAG

---------

Co-authored-by: Ning Sun <sunning@greptime.com>
 2025-01-24 09:30:54 +00:00
Ning Sun
dce4ed9f1d feat: add CORS headers for http interfaces (#5447) 
* feat: add cors headers for http server

* test: add cors test

* test: add preflight test
 2025-01-24 09:28:04 +00:00
yihong
bbfbc9f0f8 fix: drop unused numpy code since pyo3 rustpython do not support any more (#5442) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-24 08:20:01 +00:00
Weny Xu
b107384cc6 feat(metric-engine): support to write rows with sparse primary key encoding (#5424) 
* feat: support to write rows with sparse primary key encoding

* feat: cache decoded primary key

* chore: remove unused code

* feat: create physical table based on the engine config

* chore: log primary key encoding info

* fix: correct sqlness test

* chore: correct config.md

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2025-01-24 06:56:09 +00:00
zyy17
2802c8bf28 ci: update dev-builder version to fix build android image failed (#5445) 
ci: update dev-builder version
 2025-01-24 06:48:26 +00:00
zyy17
9b9784a557 fix: install x86-64 protoc on android dev-builder (#5443) 2025-01-24 04:39:39 +00:00
yihong
1e61d05211 fix: arm actions test failed (#5433) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-01-24 03:43:59 +00:00