rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-01-07 13:52:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,971 Commits 104 Branches 226 Tags
2ae2a6674e8a780cb93723de9b749b65ea28562e
Commit Graph

2971 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zyy17
2ae2a6674e refactor: add get_storage_path() and get_catalog_and_schema() (#4397) 
refactor: add get_storage_path() and get_catalog_and_schema()
v0.10.0-nightly-20240722 		2024-07-20 01:55:48 +00:00
Lei, HUANG
c8cf3b1677 fix(wal): handle WAL deletion on region drop (#4400) 
Add LogStore trait bound to RegionWorkerLoop and handle WAL deletion on region drop.
 2024-07-19 13:24:10 +00:00
Yingwen
7aae19aa8b fix: dictionary key type use u32 (#4396) 
* fix: dictionary key type use u32

* fix: fix error whle reading content

* fix: bulk memtable dictionary type
 2024-07-19 09:51:29 +00:00
Jeremyhi
b90267dd80 feat: export database data (#4382) 
* feat: export database data

* feat: export data with time range

* feat: refactor the data dir

* feat: by comment
 2024-07-19 09:29:45 +00:00
discord9
9fa9156bde feat: FLOWS table in information_schema&SHOW FLOWS (#4386) 
* feat(WIP): flow info table

refactor: better err handling&log

feat: add flow metadata to info schema provider

feat(WIP): info_schema.flows

feat: info_schema.flows table

* fix: err after rebase

* fix: wrong comparsion op

* feat: SHOW FLOWS&tests

* refactor: per review

* chore: unused

* refactor: json error

* chore: per review

* test: sqlness

* chore: rm inline error

* refactor: per review
 2024-07-19 09:29:36 +00:00
zyy17
ce900e850a fix: user provider can't be configured by config file or environment variables (#4398) 2024-07-19 08:41:29 +00:00
zyy17
5274c5a407 refactor: add &mut Plugins argument in plugins setup api and remove unnecessary mut (#4389) 
refactor: add '&mut Plugins' argument in plugins setup api and remove unnecessary mut

Signed-off-by: zyy17 <zyylsxm@gmail.com>
 2024-07-19 08:12:06 +00:00
Yingwen
0b13ac6e16 ci: disable auto review (#4387) 2024-07-18 08:03:37 +00:00
shuiyisong
8ab6136d1c chore: support pattern as pipeline key name (#4368) 
* chore: add pattern to processor key name

* fix: typo

* refactor: test
 2024-07-18 03:32:26 +00:00
Weny Xu
e39f49fe56 fix: ensure keep alive is completed in time (#4349) 
* fix: ensure keep alive is completed in time

* chore: apply suggestions from CR

* chore: use write runtime

* refactor: set META_LEASE_SECS to 5

* chore: set etcd replicas to 1

* chore: apply suggestions from CR

* chore: apply suggestions from CR

* fix: set `MissedTickBehavior::Delay`

* chore: apply suggestions from CR
 2024-07-17 06:14:45 +00:00
discord9
c595a56ac8 test(flow): ignore flow tests for now (#4377) 
* tests: ignore flow tests for now

* chore: typo

* test: remove file altogether

* fix: last_value sqlness test regex

* fix: last_value part REDACTED
v0.9.0 		2024-07-16 10:07:21 +00:00
Yingwen
d6c7b848da chore: bump version to v0.9.0 (#4376) 
* chore: bump version to v0.9.0

* chore: format
 2024-07-16 07:50:45 +00:00
Lei, HUANG
2010a2a33d feat: Add caching for last row reader and expose cache manager (#4375) 
* Add caching for last row reader and expose cache manager
 - Implement `RowGroupLastRowCachedReader` to handle cache hits and misses for last row reads.

* Add projection field to SelectorResultValue and refactor RowGroupLastRowReader

 - Introduced `projection` field in `SelectorResultValue` to store projection indices.
 2024-07-16 07:13:39 +00:00
dennis zhuang
be3ea0fae7 feat: improve datafusion external error and mysql error (#4362) 
* feat: improve datafusion external error and mysql error

* chore: address CR comments and fix tests

---------

Co-authored-by: evenyag <realevenyag@gmail.com>
 2024-07-16 07:01:09 +00:00
Lei, HUANG
7b28da277d refactor: LastRowReader to use LastRowSelector (#4374) 
Refactor LastRowReader to use LastRowSelector
 - Replaced `last_batch` in `LastRowReader` with `LastRowSelector`.
v0.9.0-b2c5f8eef-20240716-1721102282 		2024-07-16 03:47:41 +00:00
discord9
b2c5f8eefa test: more sleep when flow insert makes it serial (#4373) 
* tests: more sleep

* fix: wait more time

* fix: update result file
 2024-07-16 03:36:31 +00:00
JohnsonLee
072d7c2022 feat: introduce 'pg_catalog.pg_type' (#4332) 
* WIP: pg_catalog

* refactor: move memory_table to crate public level to reuse it in pgcatalog

* refactor: new system_schema mod to manage implementation of information_schema and pg_catalog

* feat: pg_catalog.pg_type

* fix: remove unused code to avoid warning

* test: add pg_catalog sqlness test

* feat: pg_catalog_cache in system_catalog

* fix: integration test

* test: rollback unit test

* refactor: mix pg_catalog table_id with old ones

* fix: add todo information

* tests: rerun sqlness

---------

Co-authored-by: johnsonlee <johnsonlee@localhost.localdomain>
 2024-07-15 17:41:08 +00:00
Yingwen
7900367433 test: replace unstable output of last value test (#4371) 
test: replace unstable output
 2024-07-15 16:51:16 +00:00
Lei, HUANG
9fbc4ba649 feat: add PruneReader for optimized row filtering (#4370) 
* Add PruneReader for optimized row filtering and error handling

 - Introduced `PruneReader` to replace `RowGroupReader` for optimized row filtering.

* Commit Message:

 Make ReaderMetrics fields public for external access

* Add row selection support to SeqScan and FileRange readers

 - Updated `SeqScan::build_part_sources` to accept an optional `TimeSeriesRowSelector`.

* Refactor `scan_region.rs` to remove unnecessary cloning of `series_row_selector`. Enhance `file_range.rs` by adding `select_all` method to check if all rows in a row group are selected, and update the logic in `reader` method to use `LastRowReader` only when all rows are
 selected and no DELETE operations are present.

* Commit Message:

Enhance PruneReader and ParquetReader with reset functionality and metrics handling

Summary:

 • Made Source enum public in prune.rs.

* chore: Update src/mito2/src/sst/parquet/reader.rs

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
 2024-07-15 14:23:34 +00:00
Yingwen
2e7b12c344 feat: add a cache for last value result in row group (#4369) 
* feat: add selector result cache to cache manager

* feat: expose config
 2024-07-15 12:33:36 +00:00
discord9
2b912d93fb feat: flow perf&fix df func call (#4347) 
* feat: flow perf&fix df func call

feat: make source sender `send` non-blocking

feat: better control of flow worker freq

feat: support transform interval

fix: const folding df func args&tests

tests: update cast const fold

chore: adjust flow work's freq

refactor: batch split

feat: adaptive run freq flow worker&check for errors

chore: better debug log

* refactor: per review

* chore: per zc's review

* chore: per bot review

* chore: remove some `TODO` completed

* docs: add comments for a test
 2024-07-15 09:20:04 +00:00
Zhenchi
04ac0c8da0 feat(fulltext_index): integrate full-text indexer with parquet reader (#4348) 
* feat(fulltext_index): integrate full-text indexer with parquet reader

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* disable reload

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix: range allow exceeding total row

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: unit tests in index

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: prune row groups

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: rename creator

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* test: sst fulltext index

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-15 08:14:44 +00:00
dennis zhuang
64cad4e891 feat: tweak error and status codes (#4359) 
* feat: tweak status codes

* fix: typo

* fix: by cr comments
 2024-07-15 07:50:16 +00:00
Yingwen
20d9c0a345 fix: scan hint checks order asc (#4365) 
* fix: order by asc check

* feat: print selector in explain

* test: move last_value opt test to standalone

* test: sqlness remove space

* test: update regex for datetime

* test: fix partitioning

* chore: update comment

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

---------

Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-15 07:49:14 +00:00
Lanqing Yang
9501318ce5 feat: support show views statement (#4360) 
This commit enables show view statement which will display a list of views
names.
 2024-07-15 07:24:27 +00:00
Ning Sun
b8bd8456f0 fix: remove path label for cache store (#4336) 
* fix: remove path label for cache store

* fix: ignore path label for intermediatemanager

* refactor: remove unused object store
 2024-07-15 03:34:19 +00:00
tison
4b8b04ffa2 chore: update project slogan (#4361) 
* chore: update project slogan

Signed-off-by: tison <wander4096@gmail.com>

* keep in oneline

Signed-off-by: tison <wander4096@gmail.com>

* Apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* Update README.md

---------

Signed-off-by: tison <wander4096@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-15 03:02:44 +00:00
Lanqing Yang
15ac8116ea feat: adding information_schema.views table (#4342) 
This commit introduces information_schema.views table. The VIEWS table provides
information about views in databases.
 2024-07-14 09:50:19 +00:00
Zhenchi
377a513690 feat(index): distinguish different types of index metrics (#4337) 
Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-14 08:03:09 +00:00
Yingwen
5a1732279b feat: Implement reader that returns the last row of each series (#4354) 
* feat: last row reader

* feat: scan use last row reader

* test: test last row selector

* chore: update comment
 2024-07-12 14:40:06 +00:00
Ruihang Xia
16075ada67 feat: impl optimizer rule to handle last_value case (#4357) 
* feat: impl optimizer rule to handle last_value case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* rename file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* Update src/query/src/optimizer/scan_hint.rs

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* split methods

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-07-12 14:36:23 +00:00
shuiyisong
67dfdd6c61 feat: support text/plain format for log ingestion (#4300) 
* feat: support text/plain format of log input

* refactor: pipeline query and delete using dataframe api

* chore: minor refactor

* refactor: skip jsonify when processing plan/text

* refactor: support array(string) as pipeline engine input
 2024-07-12 09:17:15 +00:00
irenjj
9f2d53c3df refactor: Remove the StandaloneKafkaConfig struct (#4253) 
* refactor: Remove the StandaloneKafkaConfig struct

* remove the redundant assignment

* remove rudundant struct

* simplify replication_factor

* add KafkaTopicConfig

* fix check

* fix check

* fix check

* add flatten with

* revert config.md

* fix test params

* fix test param

* fix missing params when provider is kafka

* remove unsed files

* remove with prefix

* fix doc

* fix test

* fix clippy
 2024-07-12 08:17:18 +00:00
Weny Xu
05c7d3eb42 docs(config): add enable_region_failover option to configuration (#4355) 
docs(config): Add enable_region_failover option to configuration
 2024-07-12 08:09:25 +00:00
Ruihang Xia
63acc30ce7 perf: fine–tuned plan steps (#4258) 
* perf: fine–tuned plan steps

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle explain plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle explain plan again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-12 06:56:13 +00:00
tison
285ffc5850 fix: build info should use build time env var (#4343) 
* fix: build info should use build time env var

Signed-off-by: tison <wander4096@gmail.com>

* catch up

Signed-off-by: tison <wander4096@gmail.com>

* fixup lifetime

Signed-off-by: tison <wander4096@gmail.com>

* fixup

Signed-off-by: tison <wander4096@gmail.com>

* fix more

Signed-off-by: tison <wander4096@gmail.com>

---------

Signed-off-by: tison <wander4096@gmail.com>
 2024-07-12 03:08:30 +00:00
dennis zhuang
ab22bbac84 feat: impl drop view (#4231) 
* feat: impl drop view

* fix: metric name

* fix: comments

* test: add DropViewProcedure test

* test: drop view meets a table

* test: update sqlness tests by drop view

* feat: apply suggestion from AI

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* chore: apply suggestion

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>

* fix: TYPE_NAME for DropFlowProcedure

---------

Co-authored-by: Jeremyhi <jiachun_feng@proton.me>
 2024-07-11 19:53:54 +00:00
Weny Xu
7ad248d6f6 fix(config): enable file engine by default (#4345) 
* fix: enable file engine by default

* fix: fix tests
 2024-07-11 17:35:18 +00:00
Ning Sun
50e4539667 fix: permission denied is 403 (#4350) 2024-07-11 17:16:17 +00:00
Zhenchi
da1ea253ba perf(puffin): not to stage uncompressed blob (#4333) 
* feat(puffin): not to stage blob

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* feat: back with compressed blob

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
 2024-07-11 12:32:07 +00:00
Lei, HUANG
da0c840261 feat: customize copy to parquet parameter (#4328) 
* feat/copy-to-parquet-parameter: Commit Message:

Enhance Parquet Writer with Column-wise Configuration

Summary:

 • Introduced column_wise_config function to customize per-column properties in Parquet writer.

* feat/copy-to-parquet-parameter: Commit Message:

Enhance Parquet File Format Handling for Specific Data Types

Summary:

 • Added ConcreteDataType import to support specific data type handling.

* feat/copy-to-parquet-parameter: Commit Message:

 Refactor Parquet file format configuration

* feat/copy-to-parquet-parameter:
 Enhance Parquet file format handling for timestamp columns

 - Added logic to disable dictionary encoding and set DELTA_BINARY_PACKED encoding for timestamp columns in the Parquet file format configuration.

* feat/copy-to-parquet-parameter:
 Disable dictionary encoding for timestamp columns in Parquet writer and update default max_active_window_runs in TwcsOptions

 - Modified Parquet writer to disable dictionary encoding for timestamp columns to optimize for increasing timestamp data.

* feat/copy-to-parquet-parameter:
 Update compaction settings in tests

 - Modified `test_compaction_region` to include new compaction options: `compaction.type`,
 `compaction.twcs.max_active_window_runs`, and `compaction.twcs.max_inactive_window_runs`.
 - Updated `test_merge_mode_compaction` to use `compaction.twcs.max_active_window_runs` and
 `compaction.twcs.max_inactive_window_runs` instead of `max_active_window_files` and
 `max_inactive_window_files`.
 2024-07-10 07:52:30 +00:00
zyy17
20417e646a ci: add flownode in docker compose (#4306) 2024-07-10 06:37:21 +00:00
Ruihang Xia
9271b3b7bd docs: remove cargo test workspace command (#4325) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-10 06:36:44 +00:00
Ruihang Xia
374cfe74bf fix(sqlness): relax start time regex to match various precisions (#4326) 
* fix(sqlness): relax start time regex to match various precisions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* exclude ip pattern

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2024-07-10 06:35:55 +00:00
Weny Xu
52a9a748a1 revert: lz4 compression (#4329) 
* Revert "test: revert lz4 compression"

This reverts commit 180dda13fa.

* refactor: remove compression field
 2024-07-10 04:24:40 +00:00
dennis zhuang
33ed745049 feat: show create view and creating view with columns (#4086) 
* feat: parse column names when creating view

* feat: save the view definition into view info

* feat: supports view columns and show create view

* feat: save plan columns for validation

* fix: typo

* chore: comments and style

* chore: apply suggestions

* test: assert CreateView display result

* chore: style

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* chore: avoid the clone

Co-authored-by: Weny Xu <wenymedia@gmail.com>

* fix: compile error after rebeasing

---------

Co-authored-by: Weny Xu <wenymedia@gmail.com>
 2024-07-09 18:35:11 +00:00
Yingwen
458e5d7e66 feat: add TimeSeriesRowSelector hint (#4327) 
* feat: Add TimeSeriesRowSelector

* feat: scan allow specify series row selector

* chore: Update comment
 2024-07-09 12:29:47 +00:00
discord9
1ddf19d886 feat: flownode use Inserter to write to database (#4323) 
* feat: use `Inserter` as Frontend

* fix: enable procedure in flownode

* docs: remove `frontend_addr` opts

* chore: rm fe addr in test runner

* refactor: int test also use inserter invoker

* feat: flow shutdown&refactor: remove `Frontendinvoker`

* refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker`

* refactor: per review

* refactor: remove a layer of  box

* fix: standalone use `node_manager`

* fix: remove a `Arc` cycle
v0.9.0-nightly-20240709 		2024-07-09 10:44:22 +00:00
Ruihang Xia
185953e586 fix: support unary operator in default value, partition rule and prepare statement (#4301) 
* handle unary operator

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add sqlness test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add prepare test

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add test and context

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix rebase error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix merge error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix sqlness

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2024-07-09 08:59:06 +00:00
Ning Sun
7fe3f496ac refactor: do not print error log on PlanQuery error (#4322) 2024-07-09 06:34:30 +00:00