Ning Sun
70d113a355
feat: update default size of bgworkers, add hbworkers ( #4129 )
...
* feat: update default size of bgworkers, add hbworkers
* feat: update frontend heartbeat as well
* chore: update sample config files and default settings
* chore: update config docs
* Revert "chore: update config docs"
This reverts commit 8107f4c120f5ae701c8d
2024-06-18 06:18:37 +00:00
shuiyisong
cb657ae51e
feat(pipeline): join processor ( #4158 )
...
* feat: add join processor
* test: add join simple test
* chore: fix header
* chore: update commit
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* test: add more join test
* chore: fix lint
* chore: update comment
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-18 05:00:34 +00:00
Jeremyhi
141d017576
feat: enable metasrv to accept flownode's heartbeats ( #4160 )
...
* feat: make metasrv can accept flownode's heartbeat
* chore: proto
2024-06-18 04:07:46 +00:00
yuanbohan
0fc18b6865
feat(pipeline): gsub prosessor ( #4121 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* gsub prosessor
* chore: add test
* chore: update commit
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: paomian <xpaomian@gmail.com >
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-17 07:57:47 +00:00
yuanbohan
0aceebf0a3
feat(pipeline): transform support on_failure ( #4123 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* transform on_failure
* chore: add test
* chore: move test to a separate file
* chore: add comment
---------
Co-authored-by: paomian <xpaomian@gmail.com >
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
2024-06-17 06:56:31 +00:00
Yingwen
558272de61
refactor: Decouple dedup and merge ( #4139 )
...
* feat: remove dedup/filter deleted from merge reader
* feat: impl dedup reader
* feat: support filter deleted flag
* test: test dedup reader
* feat: remove put_only field
* chore: fix clippy
* feat: metrics
* test: test empty batch
* perf: optimize dedup strategy
Avoid iterating all timestamps.
* test: fix test
* feat: generic
2024-06-17 04:09:50 +00:00
LFC
f4a5a44549
refactor: make region manifest checkpoint ran in background ( #4133 )
...
* refactor: use in-database manifest as checkpoint instead of merging incremental files in object store
* refactor: make region manifest checkpoint ran in background
* reduce unnecessary metrics
* Update src/mito2/src/manifest/checkpointer.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* resolve PR comments
* resolve PR comments
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-17 03:47:18 +00:00
zyy17
5390603855
refactor: add Compactor trait to abstract the compaction ( #4097 )
...
* refactor: add Compactor trait
* chore: add compact() in Compactor trait and expose compaction module
* refactor: add CompactionRequest and open_compaction_region
* refactor: export the compaction api
* refactor: add DefaultCompactor::new_from_request
* refactor: no need to pass mito_config in open_compaction_region()
* refactor: CompactionRequest -> &CompactionRequest
* fix: typo
* docs: add docs for public apis
* refactor: remove 'Picker' from Compactor
* chore: add logs
* chore: change pub attribute for Picker
* refactor: remove do_merge_ssts()
* refactor: update comments
* refactor: use CompactionRegion argument in Picker
* chore: make compaction module public and remove unnessary clone
* refactor: move build_compaction_task() in CompactionScheduler{}
* chore: use in open_compaction_region() and add some comments for public structure
* refactor: add 'manifest_dir()' in store-api
* refactor: move the default implementation to DefaultCompactor
* refactor: remove Options from MergeOutput
* chore: minor modification
* fix: clippy errors
* fix: unit test errors
* refactor: remove 'manifest_dir()' from store-api crate(already have one in opener)
* refactor: use 'region_dir' in CompactionRequest
* refactor: refine naming
* refactor: refine naming
* refactor: remove clone()
* chore: add comments
* refactor: add PickerOutput field in CompactorRequest
2024-06-17 03:03:47 +00:00
Ruihang Xia
a2e3532a57
docs: add guide for tsbs benchmark ( #4151 )
...
* docs: add guide for tsbs benchmark
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-16 16:02:15 +08:00
Ruihang Xia
2faa6d6c97
ci: align docs with develop ( #4152 )
...
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-15 16:33:57 +00:00
Weny Xu
d6392acd65
fix(sqlness): catch different format timestamp ( #4149 )
2024-06-15 11:13:01 +00:00
localhost
01e3a24cf7
feat: log ingestion support ( #4014 )
...
* chore: add log http ingester scaffold
* chore: add some example code
* chore: add log inserter
* chore: add log handler file
* chore: add pipeline lib
* chore: import log handler
* chore: add pipelime http handler
* chore: add pipeline private table
* chore: add pipeline API
* chore: improve error handling
* chore: merge main
* chore: add multi content type support for log handler
* refactor: remove servers dep on pipeline
* refactor: move define_into_tonic_status to common-error
* refactor: bring in pipeline 3eb890c551b8d7f60c4491fcfec18966e2b210a4
* chore: fix typo
* refactor: bring in pipeline a95c9767d7056ab01dd8ca5fa1214456c6ffc72c
* chore: fix typo and license header
* refactor: move http event handler to a separate file
* chore: add test for pipeline
* chore: fmt
* refactor: bring in pipeline 7d2402701877901871dd1294a65ac937605a6a93
* refactor: move `pipeline_operator` to `pipeline` crate
* chore: minor update
* refactor: bring in pipeline 1711f4d46687bada72426d88cda417899e0ae3a4
* chore: add log
* chore: add log
* chore: remove open hook
* chore: minor update
* chore: fix fmt
* chore: minor update
* chore: rename desc for pipeline table
* refactor: remove updated_at in pipelines
* chore: add more content type support for log inserter api
* chore: introduce pipeline crate
* chore: update upload pipeline api
* chore: fix by pr commit
* chore: add some doc for pub fn/struct
* chore: some minro fix
* chore: add pipeline version support
* chore: impl log pipeline version
* chore: fix format issue
* fix: make the LogicalPlan of a query pipeline sorted in desc order
* chore: remove some debug log
* chore: replacing hashmap cache with moak
* chore: fix by pr commit
* chore: fix toml format issue
* chore: update Cargo.lock
* chore: fix by pr commit
* chore: fix some issue by pr commit
* chore: add more doc for pipeline version
---------
Co-authored-by: shuiyisong <xixing.sys@gmail.com >
2024-06-14 17:03:30 +00:00
Weny Xu
bf3ad44584
fix: fix release CI typo ( #4147 )
...
* fix: ci typo
* chore: use aws registry
2024-06-14 14:29:34 +00:00
Weny Xu
11a903f193
chore(ci): switch to aws registry ( #4145 )
...
chore: switch to aws registry
2024-06-14 11:46:57 +00:00
Weny Xu
acdfaabfa5
fix(ci): use ld_classic on macOS ( #4143 )
...
fix: use ld_classic on macos
2024-06-14 08:09:12 +00:00
Weny Xu
54ca06ba08
chore: bump version to v0.8.2 ( #4141 )
2024-06-14 03:39:08 +00:00
Weny Xu
1f315e300f
fix: retry on unknown error ( #4138 )
2024-06-13 11:15:38 +00:00
Weny Xu
573e25a40f
chore: run fuzz tests with disk cache ( #4118 )
...
* chore: run fuzz tests with disk cache
* fix: print error messages correctly
2024-06-13 09:07:12 +00:00
Lei, HUANG
f8ec46493f
refactor: simplify parquet writer ( #4112 )
...
* refactor: simplify parquet writer
* chore: fix clippy
* refactor: use AsyncArrowWriter instead of BufferedWriter
* refactor: remove BufferedWriter
* fix: add chunk parameter to avoid entity too small issue
* refactor: use AtomicUsize instead of Mutex
* fix: add chunk argument to stream_to_parquet
* chore: fmt
* wip: fail check
* fix: check
* fmt
* refactor: use impl Future instead of async_trait
* fmt
* refactor: use associate types
2024-06-13 07:32:47 +00:00
Weny Xu
14a2d83594
chore: remove unused code ( #4135 )
...
* chore: remove unused code
* Update src/mito2/src/wal/entry_reader.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-12 13:18:33 +00:00
Yingwen
65f8b72d34
feat: Implement RegionScanner for SeqScan ( #4060 )
...
* feat: ordered builder wip
* feat: impl RegionScanner for SeqScan
* feat: implement scan_partition and build_stream
* chore: return SeqScan as RegionScanner
* fix: group parts
* feat: split parts
* chore: reader metrics
* chore: metrics
* chore: remove unused codes
* chore: support holding a group of ranges in ScanPart
* feat: group ScanParts to ScanParts
* feat: impl SeqScanner again
* chore: observe build cost in ScannerMetrics
* chore: fix compiler warnings
* style: fix clippy
* docs: update config docs
* chore: forward DisplayAs to scanner
* test: update sqlness tests
* chore: update debug fmt
* chore: custom debug for timestamp
fix test compiling issue with common-macro when running
cargo nextest -p common-time
* chore: update debug format
* feat: update fmt for scan part
* chore: fix warning
* fix: sanitize parallelism
* feat: split parts
* test: fix config api test
* feat: update logs
* chore: Revert "chore: remove unused codes"
This reverts commit b548b30a01eeded59b1a0a8d89f9293ca63afc41.
* chore: Revert "docs: update config docs"
This reverts commit a7997e78d6ddcf635560574de8c1948c495bdd12.
* feat: each partition scan files in parallel
* test: fix config api test
* docs: fix typo
* chore: address comments, simplify tests
* feat: global semaphore
* feat: always spawn task
* chore: simplify default explain output format
* handle output partiton number is 0
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* fix typo
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-12 08:21:30 +00:00
LFC
9473daab8b
fix: explicitly set config instead of using changable default in tests ( #4132 )
...
* fix: explicitly set config instead of using changable default in tests
* fix: resolve PR comments
2024-06-11 10:51:01 +00:00
LFC
5a6021e34f
refactor: remove substrait ser/de for region query in standalone ( #3812 )
...
* refactor: remove substrait serde for region query in standalone
* fix ci
* move QueryRequest to common-query
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format code
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* format toml file
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore: format toml
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-11 09:33:47 +00:00
discord9
1b00526de5
fix(flow): infer table schema correctly ( #4113 )
...
* refactor: make individual col name optional
* chore: rename TypedPlan's `typ` to `schema`
* feat: add optional col name to typed plan
* feat: pass col name all along
* feat: correct infer output table schema
* chore: unused import
* fix: error when key is not projected
* refactor: per review
* chore: fmt
2024-06-11 08:57:47 +00:00
Yingwen
5533bd9293
chore(common-macro): remove features covered by full ( #4131 )
2024-06-11 07:44:53 +00:00
Ning Sun
587e99d806
fix: macro crate cannot be compiled alone ( #4130 )
...
* fix: macro crate cannot be compiled alone
* Update src/common/macro/Cargo.toml
Co-authored-by: Yingwen <realevenyag@gmail.com >
---------
Co-authored-by: tison <wander4096@gmail.com >
Co-authored-by: Yingwen <realevenyag@gmail.com >
2024-06-11 05:36:29 +00:00
Yingwen
9cae15bd1b
fix: executes pending ddls if region memtable is empty while scheduling next flush ( #4119 )
...
* ci: enable debug log
* chore: test to reproduce panic
* chore: Revert "ci: enable debug log"
This reverts commit 17eff2a045
2024-06-11 00:10:17 +00:00
cjw
d8b51cfaba
refactor: remove double checks of memtable size ( #4117 )
...
* refactor: remove unnecessary unwrap
* Update src/mito2/src/region/version.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
---------
Co-authored-by: Kermit <chenjiawei1@xiaohongshu.com >
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-07 23:34:03 +00:00
Weny Xu
e142ca40d7
feat: invoke handle_batch_open_requests ( #4107 )
...
* feat: open all regions via invoking `handle_batch_open_requests`
* tests: add sqlness tests
* refactor: avoid cloning
* chore: apply suggestions from CR
* chore: update config.md
* chore: apply suggestions from CR
2024-06-07 09:07:45 +00:00
Yingwen
e982d2e55c
fix: Update region Version in the worker loop ( #4114 )
...
* feat: handle region edit result
* feat: handle edit result
* feat: handle truncate result
* feat: flush compaction
* feat: invoke in worker
* feat: remove unused fields
* style: fix clippy
* feat: remove applier
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-07 06:27:16 +00:00
Weny Xu
09e0e1b246
chore: run fuzz tests with kafka remote wal ( #4105 )
...
* chore: add fuzz tests with kafka
* chore(ci): use minio
* chore: add empty line
* chore(ci): refactor
* chore: add empty line
* fix: update config
* fix: add default value for `MetaClientOptions`
* fix: remove redundant `debug_assert`
* chore: run fuzz tests with disk cache
* chore: remove redundant minio setup
* chore: cache targets
* Revert "chore: run fuzz tests with disk cache"
This reverts commit d81783187d09b899eed1
2024-06-07 03:47:40 +00:00
irenjj
9c42825f5d
feat: Implement SHOW CREATE FLOW ( #4040 )
...
* feat: Implement SHOW CREATE FLOW
* fmt
* stmt for display
* Update src/operator/src/statement.rs
Co-authored-by: Yingwen <realevenyag@gmail.com >
* test: add sqlness test
* fix test
* parse query in parser
* test: move test to standalone
* reuse ParserContext::new()
* Update tests/cases/standalone/show_create_flow.result
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* add line breaks
---------
Co-authored-by: Yingwen <realevenyag@gmail.com >
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-07 03:24:56 +00:00
Jeremyhi
4719569e4f
feat: support gRPC cancellation ( #4092 )
...
* feat: support cancellation
* chore: add unit test for cancellation
* chore: minor refactor
* feat: we do not need to spawn in distributed mode
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com >
2024-06-06 08:27:25 +00:00
Jeremyhi
b03cb3860e
chore: reduce some burden on the write path ( #4110 )
...
* chore: remove unnecessary checking
* chore: avoid do the same thing in a loop
2024-06-06 06:45:19 +00:00
shuiyisong
2ade511f26
feat: introduce pipeline crate ( #4109 )
...
* chore: introduce pipeline crate
* chore: fix typo
2024-06-05 17:23:25 +00:00
Weny Xu
16b85b06b6
chore: remove gc before running fuzz tests ( #4108 )
2024-06-05 11:59:29 +00:00
Ruihang Xia
03cacf9948
ci: cargo gc all fuzz test runner ( #4081 )
...
* ci: cargo gc all fuzz test runner
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
* chore(ci): disable caching targets
* chore(ci): remove .tar file after unzip
* fix cargo-gc command
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com >
Co-authored-by: WenyXu <wenymedia@gmail.com >
2024-06-05 09:58:29 +00:00
Weny Xu
c23f8ad113
feat: implement the handle_batch_open_requests ( #4075 )
...
* feat: implement the `handle_batch_open_requests`
* refactor: refactor `handle_batch_open_requests` method signature
* test: add tests for `handle_batch_open_requests`
* chore: fmt code
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
* chore: apply suggestions from CR
2024-06-05 09:22:34 +00:00
Weny Xu
e0a2c5a581
chore(ci): remove redundant sqlness test config ( #4106 )
2024-06-05 08:39:39 +00:00
zyy17
417ab3b779
ci: add 'make run-cluster-with-etcd' to run greptimedb cluster by using docker-compose ( #4103 )
2024-06-05 08:07:29 +00:00
tison
1850fe2956
feat: show create table only for base table ( #4099 )
...
* feat: show create table only for base table
Signed-off-by: tison <wander4096@gmail.com >
* add new cases
Signed-off-by: tison <wander4096@gmail.com >
---------
Signed-off-by: tison <wander4096@gmail.com >
2024-06-04 21:29:07 +00:00
taobo
dd06e107f9
test: add fuzz tests for column data type alteration ( #4076 )
...
* feat: support make fuzz-stable in Makefile
* test: add fuzz tests for column data type alteration
* fix: optimize code by cr
2024-06-04 13:38:57 +00:00
sarailQAQ
98c19ed0fa
feat: implement drop multiple tables ( #4085 )
...
* feat: implement drop multiple tables
* fix: pass fmt and clippy checks
* add: drop multiple sqlness test
* update: accept review suggestions
* update: accept reviem suggestion
Co-authored-by: Weny Xu <wenymedia@gmail.com >
* fix: pass clippy check
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-04 13:11:41 +00:00
LFC
c0aed1d267
feat: set global runtime size by config file ( #4063 )
...
* set global runtime size
* fix: resolve PR comments
* fix: log the whole option
* fix ci
* debug ci
* debug ci
---------
Co-authored-by: Weny Xu <wenymedia@gmail.com >
2024-06-04 10:03:33 +00:00
discord9
0a07130931
fix(flow): mfp operator missing rows ( #4084 )
...
* fix: mfp missing rows if run twice in same tick
* tests: run mfp for multiple times
* refactor: make mfp less hacky
* feat: make channel larger
* chore: typos
2024-06-04 09:07:13 +00:00
Weny Xu
a6269397c8
fix: fix EntityTooSmall issue ( #4100 )
...
* fix: fix EntityTooSmall issue
* chore(ci): add minio to coverage
* tests: add test for parquet writer
* chore: move tests to `common-datasource` crate
2024-06-04 08:43:33 +00:00
Lei, HUANG
a80059b47f
fix: recover memtable options when opening physical regions ( #4102 )
...
* fix: recover memtable options when opening physical regions
* chore: fmt
* chore: merge data region options
2024-06-04 08:20:29 +00:00
Weny Xu
b3a4362626
test: run test_flush_reopen_region and test_region_replay with KafkaLogStore ( #4083 )
...
* feat: add `LogStoreFactory` to `TestEnv`
* feat: add `multiple_log_store_factories` template
* test: run `test_flush_reopen_region` and `test_region_replay` with `KafkaLogStore`
* chore: move deps to workspace
* chore: apply suggestions from CR
2024-06-04 08:11:15 +00:00
Kelvin Wu
51e2b6e728
fix: display the PartitionBound and PartitionDef correctly ( #4101 )
...
* fix: display the PartitionBound and PartitionDef correctly
* Update src/partition/src/partition.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com >
* fix: fix unit test of partition definition
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com >
2024-06-04 08:10:44 +00:00
shuiyisong
d1838fb28d
refactor: move define_into_tonic_status to common-error ( #4095 )
...
* chore: finish cherry-pick
* chore: remove unused code
2024-06-04 03:29:15 +00:00
