rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-08 06:12:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,963 Commits 103 Branches 226 Tags
8ab6136d1cb94f00ce7b7332b42918e376be0ff3
Commit Graph

804 Commits

Author SHA1 Message Date
Yingwen
d6c7b848da chore: bump version to v0.9.0 (#4376) 
* chore: bump version to v0.9.0

* chore: format
 2024-07-16 07:50:45 +00:00
Zhenchi
04ac0c8da0 feat(fulltext_index): integrate full-text indexer with parquet reader (#4348) 
* feat(fulltext_index): integrate full-text indexer with parquet reader

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* disable reload

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: range allow exceeding total row

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: unit tests in index

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: prune row groups

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename creator

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: sst fulltext index

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-15 08:14:44 +00:00
shuiyisong
67dfdd6c61 feat: support text/plain format for log ingestion (#4300) 
* feat: support text/plain format of log input

* refactor: pipeline query and delete using dataframe api

* chore: minor refactor

* refactor: skip jsonify when processing plan/text

* refactor: support array(string) as pipeline engine input
 2024-07-12 09:17:15 +00:00
Ruihang Xia
63acc30ce7 perf: fine–tuned plan steps (#4258) 
* perf: fine–tuned plan steps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle explain plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle explain plan again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-12 06:56:13 +00:00
tison
285ffc5850 fix: build info should use build time env var (#4343) 
* fix: build info should use build time env var

Signed-off-by: tison <wander4096@gmail.com>

* catch up

Signed-off-by: tison <wander4096@gmail.com>

* fixup lifetime

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fix more

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-07-12 03:08:30 +00:00
dennis zhuang
33ed745049 feat: show create view and creating view with columns (#4086) 
* feat: parse column names when creating view

* feat: save the view definition into view info

* feat: supports view columns and show create view

* feat: save plan columns for validation

* fix: typo

* chore: comments and style

* chore: apply suggestions

* test: assert CreateView display result

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: avoid the clone

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: compile error after rebeasing

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-09 18:35:11 +00:00
discord9
1ddf19d886 feat: flownode use Inserter to write to database (#4323) 
* feat: use `Inserter` as Frontend

* fix: enable procedure in flownode

* docs: remove `frontend_addr` opts

* chore: rm fe addr in test runner

* refactor: int test also use inserter invoker

* feat: flow shutdown&refactor: remove `Frontendinvoker`

* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`

* refactor: per review

* refactor: remove a layer of  box

* fix: standalone use `node_manager`

* fix: remove a `Arc` cycle
 2024-07-09 10:44:22 +00:00
Weny Xu
1a9314a581 feat: enhanced the retry logic by adding a random noise (#4320) 
feat: enhanced the retry logic by adding a random noise to the retry delay to avoid retry storms
 2024-07-09 04:30:10 +00:00
tison
d1f1fad440 build(deps): switch to upstream (#4319) 
* build(deps): switch to upstream

* lock

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-07-09 01:56:19 +00:00
Zhenchi
00308218b3 feat(fulltext_index): allow enable full-text index in SQL and gRPC way (#4310) 
* feat(fulltext_index): allow enable full-text index in SQL and gRPC way

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: test_fulltext_intm_path

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: explicitly build column options

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: fix error msg

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-08 20:18:48 +00:00
Lei, HUANG
aa4d10eef7 feat(inverted_index): inverted index cache (#4309) 
* feat/inverted-index-cache:
 Update dependencies and add caching for inverted index reader

 - Updated `atomic` to 0.6.0 and `uuid` to 1.9.1 in `Cargo.lock`.
 - Added `moka` and `uuid` dependencies in `Cargo.toml`.
 - Introduced `seek_read` method in `InvertedIndexBlobReader` for common seek and read operations.
 - Added `cache.rs` module to implement caching for inverted index reader using `moka`.
 - Updated `async-compression` to 0.4.11 in `puffin/Cargo.toml`.

* feat/inverted-index-cache:
 Refactor InvertedIndexReader and Add Index Cache Support

 - Refactored `InvertedIndexReader` to include `seek_read` method and default implementations for `fst` and `bitmap`.
 - Implemented `seek_read` in `InvertedIndexBlobReader` and `CachedInvertedIndexBlobReader`.
 - Introduced `InvertedIndexCache` in `CacheManager` and `SstIndexApplier`.
 - Updated `SstIndexApplierBuilder` to accept and utilize `InvertedIndexCache`.
 - Added `From<FileId> for Uuid` implementation.

* feat/inverted-index-cache:
 Update Cargo.toml and refactor SstIndexApplier

 - Moved `uuid.workspace` entry in Cargo.toml for better organization.

* feat/inverted-index-cache:
 Refactor InvertedIndexCache to use type alias for Arc

 - Replaced `Arc<InvertedIndexCache>` with `InvertedIndexCacheRef` type alias.

* feat/inverted-index-cache:
 Add Prometheus metrics and caching improvements for inverted index

 - Introduced `prometheus` and `puffin` dependencies for metrics.

* feat/inverted-index-cache:
 Refactor InvertedIndexReader and Cache handling

 - Simplified `InvertedIndexReader` trait by removing seek-related comments.

* feat/inverted-index-cache:
 Add configurable cache sizes for inverted index metadata and content
 - Introduced `index_metadata_size` and `index_content_size` in `CacheManagerBuilder`.

* feat/inverted-index-cache:
 Refactor and optimize inverted index caching

 - Removed `metrics.rs` and integrated cache metrics into `index.rs`.

* feat/inverted-index-cache:
 Remove unused dependencies from Cargo.lock and Cargo.toml

 - Removed `moka`, `prometheus`, and `puffin` dependencies from both Cargo.lock and Cargo.toml.

* feat/inverted-index-cache:
 Replace Uuid with FileId in CachedInvertedIndexBlobReader

 - Updated `file_id` type from `Uuid` to `FileId` in `CachedInvertedIndexBlobReader` and related methods.

* feat/inverted-index-cache:
 Refactor cache configuration for inverted index

 - Moved `inverted_index_metadata_cache_size` and `inverted_index_cache_size` from `MitoConfig` to `InvertedIndexConfig`.

* feat/inverted-index-cache:
 Remove unnecessary conversion of `file_id` in `SstIndexApplier`

 - Simplified the initialization of `CachedInvertedIndexBlobReader` by removing the redundant `into()` conversion for `file_id`.
 2024-07-08 12:36:59 +00:00
tison
fe0be1583a build(deps): upgrade opendal to 0.47.3 (#4307) 
Signed-off-by: tison <wander4096@gmail.com>
 2024-07-08 03:33:38 +00:00
Eugene Tolbakov
3f4928effc feat(sql): add iso-8601 format support for intervals (#4291) 
* feat(sql): add iso-8601 format support for intervals

* fix(sql): address CR feedback

* chore(sql): use regex to check the start of iso 8601
 2024-07-05 22:19:05 +00:00
discord9
09fff24ac4 feat: make flow distributed work&tests (#4256) 
feat: flownode frontend client&test

feat: Frontend Client

feat: set frontend invoker for flownode

feat: set frontend invoker for flownode

chore: test script

WIP: test flow distributed

feat: hard coded demo

docs: flownode example toml

feat: add flownode support in runner

docs: comments for node

chore: after rebase

docs: add a todo

tests: move flow tests to common

fix: flownode sqlness dist test

chore: per review

docs: make

fix: make doc
 2024-07-05 14:46:44 +00:00
Weny Xu
30b65ca99e chore: bump OpenDAL to 0.47.2 (#4297) 
chore: bump opendal to 0.47.2
 2024-07-05 13:54:32 +00:00
Weny Xu
8a119aa0b2 feat: add naive region failover test for metric table (#4269) 
* feat: add failover test for metric table

* chore: introduce help marco

* chore: remove incorrect check
 2024-07-05 06:54:23 +00:00
dennis zhuang
d2f6daf7b7 fix: prepare inserting with column defaults not work, #4244 (#4272) 
* fix: prepare inserting with column defaults not work, #4244

* fix: build column_defaults every time when creating adapters

* feat: cache the column_defaults in table

* test: assert ts column

* fix: unit

* chore: style

Co-authored-by: Yingwen <realevenyag@gmail.com>

* fix: typo

* chore: style

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-05 06:50:12 +00:00
shuiyisong
849e0b9249 feat: delete pipeline (#4156) 
* feat: add delete for pipeline

* chore: remove unused code

* refactor: delete pipeline

* chore: add pipeline management api metrics

* chore: minor cr issues

* chore: add unit test

* chore: fix cr issue

* fix: test

* chore: add `GreptimedbManageResponse`

* fix: typo

* fix: typo
 2024-07-05 06:23:49 +00:00
Ning Sun
8399dcada3 refactor: use rwlock for modifiable session data (#4232) 
* chore: update sqlness results

* refactor: use rwlock for modifiable data in session and querycontext

* chore: format toml

* refactor: use mutable_inner structure for mutable fields

* refactor: remove arc wrapper
 2024-07-04 12:53:25 +00:00
Zhenchi
6e2c21dd3f refactor(puffin): adjust generic parameters (#4279) 
* refactor(puffin): adjust generic parameters

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove Box impl

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-04 12:22:04 +00:00
Zhenchi
e64379d4f7 feat(fulltext_index): introduce creator (#4249) 
* feat(fulltext_index): introduce creator

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: return error if writer not found

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: helper function for tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-02 09:06:14 +00:00
tison
2665616f72 build(deps): Upgrade OpenDAL to 0.47 (#4224) 
* catch up changes

Signed-off-by: tison <wander4096@gmail.com>

* fmt

Signed-off-by: tison <wander4096@gmail.com>

* Fix cache for 0471 (#7)

* Fix cache for 0471

Signed-off-by: Xuanwo <github@xuanwo.io>

* Make clippy happy

Signed-off-by: Xuanwo <github@xuanwo.io>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>

* tidy

Signed-off-by: tison <wander4096@gmail.com>

* use opendal's exported type

Signed-off-by: tison <wander4096@gmail.com>

* clippy

Signed-off-by: tison <wander4096@gmail.com>

* fmt

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Co-authored-by: Xuanwo <github@xuanwo.io>
 2024-07-01 17:05:15 +00:00
Zhenchi
b69b24a237 feat(puffin): implement MokaCacheManager (#4211) 
* feat(puffin): implement MokaCacheManager

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: clippy

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: +1s

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: corner case to get a blob

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: keep dir in used

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: add more tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add doc comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: toml format

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename unreleased_dirs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: refine some comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: handle more cornor cases

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: refine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* refactor: simplify

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: more explanation

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: use recycle bin

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove instead

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: remove unnecessary removing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-01 13:10:13 +00:00
discord9
f035a7c79c feat: flow cli for distributed (#4226) 
* feat(WIP): add FlownodeInstance for flow cli

* feat(WIP): cli

* feat: add merge opts func

* refactor: move server&error to src dir

* feat: flownode cli build

* feat: add `flownode` subcmd to cli

* refactor: per review

* refactor!: BREAKING remove alias `metasrv-addr`

* chore: after rebase

* feat: cache invalide flownode cache

* chore: small refactor per review

* chore: fix a typo

* feat!: revert breaking change

* chore: per review

* refactor: not accept `metasrv-addr` only for flownode
 2024-07-01 09:56:15 +00:00
Jeremyhi
214fd38f69 feat: add build info for flow heartbeat task (#4228) 
* chore: refactor load region stats

* feat: add build info for flow heartbeat
 2024-07-01 03:19:25 +00:00
Weny Xu
352cc9ddde test: add e2e test for region failover (#4188) 
* test: add e2e test for region failover

* chore: add ci cfg

* chore: reduce parallelism to 8

* fix(ci): enable region failure

* chore: set sqlx LogLevel to Off

* refactor: move help functions to utils
 2024-06-28 06:49:41 +00:00
discord9
b6585e3581 refactor(flow): make from_substrait_* async& worker handle refactor (#4210) 
* refactor: use oneshot to receive result

* refactor: make from_substrait_* async

* refacrot: remove serde for plan&expr
 2024-06-27 17:17:46 +00:00
Zhenchi
1e815dddf1 feat(puffin): implement CachedPuffinWriter (#4203) 
* feat(puffin): support lz4 compression for footer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(puffin): introduce puffin manager trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat(puffin): implement CachedPuffinWriter

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: polish

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-06-25 08:00:48 +00:00
discord9
a44fe627ce feat: heartbeat task&peer lookup in proc (#4179) 
* feat: herat beat task

* feat: use real flow peer allocator when building

* feat: add peer look up in ddl context

* fix: drop flow test

* refactor: per review(WIP)

* refactor: not check if is alive

* refactor: per review

* refactor: remove useless `reset`

* refactor: per bot advices

* refactor: alive peer

* chore: bot review
 2024-06-24 15:06:33 +00:00
Zhenchi
07cbabab7b feat(puffin): support lz4 compression for footer (#4194) 
* feat(puffin): support lz4 compression for footer

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-06-24 11:39:03 +00:00
discord9
517917453d fix(flow): fix call df func bug&sqlness test (#4165) 
* tests: flow sqlness tests

* tests: WIP df func test

* fix: use schema before expand for transform expr

* tests: some basic flow tests

* tests: unit test

* chore: dep use rev not patch

* fix: wired sqlness error?

* refactor: per review

* fix: temp sqlness bug

* fix: use fixed sqlness

* fix: impl drop as async shutdown

* refactor: per bot's review

* tests: drop worker handler both sync/async

* docs: add rationale for test

* refactor: per review

* chore: fmt
 2024-06-24 07:52:45 +00:00
Ruihang Xia
fce65c97e3 feat: make RegionScanner aware of PartitionRange (#4170) 
* define PartitionRange

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add optimizer rule

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* implement interfaces

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* impl aggr stream

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add fallback method

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document and rename struct

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add more comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix typo

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-21 09:54:22 +00:00
Weny Xu
1e52ba325f feat: introduce chaos crds (#4173) 
feat: add chaos-mesh crds
 2024-06-21 06:28:51 +00:00
Yohan Wal
b739c9fd10 feat: PREPARE and EXECUTE statement from mysql client (#4125) 
* feat: prepare stmt in mysql client

* feat: execute stmt in mysql client

* fix: handle parameters properly

* refactor: use existing funcs to convert expr to scalar value

* refactor: use uuid strings as stmt_key for queries from COM_PREPARE packet

* refactor: take prepare and execute parser as submodule

* test: add unit test for converting expr to scalar value

* feat: deallocate stmt in mysql client

* chore: comments and duplicates

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-06-21 02:02:57 +00:00
Ruihang Xia
21c89f3247 perf: optimize RecordBatch to HttpOutput conversion (#4178) 
* add benchmark

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 70ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add profiler

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 50ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* save 160ms

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix license header

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix windows build

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-20 12:33:58 +00:00
LFC
cc2f7efb98 chore: bump datafusion version to fix last_value regression (#4169) 
* chore: bump datafusion version to fix `last_value` regression

* fix: resolve PR comments

* fix ci
 2024-06-19 07:47:17 +00:00
discord9
cd9705ccd7 feat(flow): support datafusion scalar function (#4142) 
* chore: call df function types

* feat: RelationDesc to DfSchema

* refactor: use RelationDesc instead of Type

* chore: WIP get to phy expr

* feat: custom deserialize

* chore: fmt

* refactor: renaming to DfScalarFunction

* feat: eval df func(untested)

* fix: had to spawn a thread for calling async

* chore: per review advices

* tests: test df scalar function
 2024-06-18 12:34:38 +00:00
Weny Xu
ea2d067cf1 feat: implement the OrderedBatchProducer (#4134) 
* feat: implement the `OrderedBatchProducer`

* test: add test of cancel safety

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* refactor: simplify the `BackgroundProducerWorker`

* feat: implement the OrderedBatchProducer v2

* refactor: switch to `OrderedBatchProducer`

* chore: rename to `MAX_FLUSH_QUEUE_SIZE`

* refactor: switch to `OrderedBatchProducerV2`

* refactor: remove `OrderedBatchProducerV1`

* test: add tests

* refactor: make config configurable

* refactor: minor refactor

* chore: remove unused code

* chore: remove `benchmarks` crate

* chore: update config doc

* chore: remove unused comment

* refactor: refactor client registry

* refactor: rename `max_batch_size` to `max_batch_bytes`

* chore: use constant value

* chore: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: remove the `CHANNEL_SIZE`

* chore: apply suggestions from CR

* fix: ensure serialized meta < ESTIMATED_META_SIZE

* chore: apply suggestions from CR

* chore: apply suggestions from CR
 2024-06-18 07:20:01 +00:00
Jeremyhi
141d017576 feat: enable metasrv to accept flownode's heartbeats (#4160) 
* feat: make metasrv can accept flownode's heartbeat

* chore: proto
 2024-06-18 04:07:46 +00:00
LFC
f4a5a44549 refactor: make region manifest checkpoint ran in background (#4133) 
* refactor: use in-database manifest as checkpoint instead of merging incremental files in object store

* refactor: make region manifest checkpoint ran in background

* reduce unnecessary metrics

* Update src/mito2/src/manifest/checkpointer.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* resolve PR comments

* resolve PR comments

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-17 03:47:18 +00:00
localhost
01e3a24cf7 feat: log ingestion support (#4014) 
* chore: add log http ingester scaffold

* chore: add some example code

* chore: add log inserter

* chore: add log handler file

* chore: add pipeline lib

* chore: import log handler

* chore: add pipelime http handler

* chore: add pipeline private table

* chore: add pipeline API

* chore: improve error handling

* chore: merge main

* chore: add multi content type support for log handler

* refactor: remove servers dep on pipeline

* refactor: move define_into_tonic_status to common-error

* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4

* chore: fix typo

* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c

* chore: fix typo and license header

* refactor: move http event handler to a separate file

* chore: add test for pipeline

* chore: fmt

* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93

* refactor: move `pipeline_operator` to `pipeline` crate

* chore: minor update

* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4

* chore: add log

* chore: add log

* chore: remove open hook

* chore: minor update

* chore: fix fmt

* chore: minor update

* chore: rename desc for pipeline table

* refactor: remove updated_at in pipelines

* chore: add more content type support for log inserter api

* chore: introduce pipeline crate

* chore: update upload pipeline api

* chore: fix by pr commit

* chore: add some doc for pub fn/struct

* chore: some minro fix

* chore: add pipeline version support

* chore: impl log pipeline version

* chore: fix format issue

* fix: make the LogicalPlan of a query pipeline sorted in desc order

* chore: remove some debug log

* chore: replacing hashmap cache with moak

* chore: fix by pr commit

* chore: fix toml format issue

* chore: update Cargo.lock

* chore: fix by pr commit

* chore: fix some issue by pr commit

* chore: add more doc for pipeline version

---------

Co-authored-by: shuiyisong <xixing.sys@gmail.com>
 2024-06-14 17:03:30 +00:00
Weny Xu
54ca06ba08 chore: bump version to v0.8.2 (#4141) 2024-06-14 03:39:08 +00:00
LFC
5a6021e34f refactor: remove substrait ser/de for region query in standalone (#3812) 
* refactor: remove substrait serde for region query in standalone

* fix ci

* move QueryRequest to common-query

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format code

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format toml file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: format toml

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-11 09:33:47 +00:00
Ning Sun
587e99d806 fix: macro crate cannot be compiled alone (#4130) 
* fix: macro crate cannot be compiled alone

* Update src/common/macro/Cargo.toml

Co-authored-by: Yingwen <realevenyag@gmail.com>

---------

Co-authored-by: tison <wander4096@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-06-11 05:36:29 +00:00
Jeremyhi
4719569e4f feat: support gRPC cancellation (#4092) 
* feat: support cancellation

* chore: add unit test for cancellation

* chore: minor refactor

* feat: we do not need to spawn in distributed mode

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-06-06 08:27:25 +00:00
taobo
dd06e107f9 test: add fuzz tests for column data type alteration (#4076) 
* feat: support make fuzz-stable in Makefile

* test: add fuzz tests for column data type alteration

* fix: optimize code by cr
 2024-06-04 13:38:57 +00:00
LFC
c0aed1d267 feat: set global runtime size by config file (#4063) 
* set global runtime size

* fix: resolve PR comments

* fix: log the whole option

* fix ci

* debug ci

* debug ci

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-06-04 10:03:33 +00:00
Weny Xu
a6269397c8 fix: fix EntityTooSmall issue (#4100) 
* fix: fix EntityTooSmall issue

* chore(ci): add minio to coverage

* tests: add test for parquet writer

* chore: move tests to `common-datasource` crate
 2024-06-04 08:43:33 +00:00
Weny Xu
b3a4362626 test: run test_flush_reopen_region and test_region_replay with KafkaLogStore (#4083) 
* feat: add `LogStoreFactory` to `TestEnv`

* feat: add `multiple_log_store_factories` template

* test: run `test_flush_reopen_region` and `test_region_replay` with `KafkaLogStore`

* chore: move deps to workspace

* chore: apply suggestions from CR
 2024-06-04 08:11:15 +00:00
shuiyisong
d1838fb28d refactor: move define_into_tonic_status to common-error (#4095) 
* chore: finish cherry-pick

* chore: remove unused code
 2024-06-04 03:29:15 +00:00