mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-07 20:42:28 +00:00
080b4b5d53521fdf029dc22e2c753f141d82b910
318
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
1afa0afc67 |
feat: add partial truncate (#6602)
* feat: add partial truncate Signed-off-by: discord9 <discord9@163.com> * fix: per review Signed-off-by: discord9 <discord9@163.com> * feat: add proto partial truncate kind Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: update branched proto Signed-off-by: discord9 <discord9@163.com> * feat: grpc support truncate WIP sql support Signed-off-by: discord9 <discord9@163.com> * wip: parse truncate range Signed-off-by: discord9 <discord9@163.com> * feat: truncate by range Signed-off-by: discord9 <discord9@163.com> * fix: truncate range display Signed-off-by: discord9 <discord9@163.com> * chore: resolve todo Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: more invalid parse Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: unused Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * chore: update branch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Yingwen
|
1df605ec4b |
feat: more logs and metrics under explain verbose mode (#6575)
* feat: collect region metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: log in info level Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add CoalescePartitionsExec to explain Signed-off-by: evenyag <realevenyag@gmail.com> * fix: finish metrics in partition and add sender full to metrics Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add eof flag on finish Signed-off-by: evenyag <realevenyag@gmail.com> * fix: output cost as string Signed-off-by: evenyag <realevenyag@gmail.com> * feat: log on stream done Signed-off-by: evenyag <realevenyag@gmail.com> * chore: region id as string Signed-off-by: evenyag <realevenyag@gmail.com> * chore: enlarge send channel size Signed-off-by: evenyag <realevenyag@gmail.com> * feat: more log in flight and scan Signed-off-by: evenyag <realevenyag@gmail.com> * chore: logs about rows/batches/bytes Signed-off-by: evenyag <realevenyag@gmail.com> * chore: enlarge channel size Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remote read only log in verbose Signed-off-by: evenyag <realevenyag@gmail.com> * chore: revert channel change Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: get explain verbose in RegionScanExec Signed-off-by: evenyag <realevenyag@gmail.com> * feat: print scan log in verbose mode Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: collect region metrics after finishing one region Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: define StreamMetrics and log in verbose mode Signed-off-by: evenyag <realevenyag@gmail.com> * feat: only log non zero filter and distributor metrics Signed-off-by: evenyag <realevenyag@gmail.com> * chore: revert displaying CoalescePartitions in explain Signed-off-by: evenyag <realevenyag@gmail.com> * feat: collect memtable metrics in partition metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
 Weny Xu
|
ac8493ab4a |
feat: introduce reconcile table procedure (#6584)
* feat: introduce `SyncColumns` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce reconcile table procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggesions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Lin Yihai
|
b6cef77a5c |
feat: add SET DEFAULT syntax (#6421)
* feat: add `SET DEFAULT` syntax Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * test: add `CURRENT_TIMESTAMP()` as default value for `SET DEFAULT` syntax Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * refactor: Make the error types more precise. Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: a minor error display enchancement for `SET DEFAULT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * refactor: Using `MODIFY COLUMN` for `DROP/SET DEFUALT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: update `greptime-proto` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> --------- Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> |
||
 dennis zhuang
|
052033d436 |
feat: supports more db options (#6529)
* feat: supports more db options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: use btree map for consistent results Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: adds compaction keys into valid db options Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Ruihang XiaandYingwen
|
076e20081b |
feat: add RegionId to FileId (#6410)
* refactor: remove staled manifest structures Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add RegionId to FileId Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * rename method Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix test cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix test Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * refactor: introduce RegionFileId - FileId still only consist of an uuid - PathProvider accepts RegionFileId and doesn't need to keep a region id in it - All Index applier takes RegionFileId and respects the region id in the RegionFileId - FileMeta can still derive Serialize/Deserialize - Refactor the CacheManager to accept RegionFileId Signed-off-by: evenyag <realevenyag@gmail.com> * feat: define PathType Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: adding PathType WIP Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: add path_type to region_dir_from_table_dir Move region_dir_from_table_dir to mito and use join_dir internally Signed-off-by: evenyag <realevenyag@gmail.com> * feat: set path type to ApplierBuilder Signed-off-by: evenyag <realevenyag@gmail.com> * style: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix passing incorrect dir to access layer Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove region_dir from CompactionRegion We can get table_dir and path_type from the access layer Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix unit tests Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix typo Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * fix: correct marker path Signed-off-by: evenyag <realevenyag@gmail.com> * feat: use AccessLayer::build_region_dir to get region dir Signed-off-by: evenyag <realevenyag@gmail.com> * chore: log entries in test Signed-off-by: evenyag <realevenyag@gmail.com> * fix: set path type in catchup Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix test_open_region_failure test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix compiler errors Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Co-authored-by: Yingwen <realevenyag@gmail.com> |
||
|
Weny Xu
|
71b564b4aa |
refactor: support multiple index operations in single alter region request (#6487)
* refactor: support multiple index operations in single alter region request Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update greptime-proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Lei, HUANG
|
0cf25f7b05 |
feat: add sst file num in region stat (#6537)
* feat/add-sst-file-num-in-region-stat: ### Add SST File Count to Region Statistics - **Enhancements**: - Added `sst_num` to track the number of SST files in region statistics across multiple modules. - Updated `RegionStat` and `RegionStatistic` structs in `datanode.rs` and `region_engine.rs` to include `sst_num`. - Modified `MitoRegion` and `SstVersion` in `region.rs` and `version.rs` to compute and return the number of SST files. - Adjusted test cases in `collect_leader_region_handler.rs`, `failure_handler.rs`, `region_lease_handler.rs`, and `weight_compute.rs` to initialize `sst_num`. - Updated `get_region_statistic` in `utils.rs` to sum `sst_num` from metadata and data statistics. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/add-sst-file-num-in-region-stat: Add `sst_num` to `region_statistics` - Updated `region_statistics.rs` to include a new constant `SST_NUM` and added it to the schema and builder structures. - Modified `information_schema.result` to reflect the addition of `sst_num` in the `region_statistics` table. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Weny Xu
|
7d17782fd5 |
feat: persist column ids in table metadata (#6457)
* feat: persist column ids in table metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Zhenchi
|
599f289f59 |
feat: add granularity and false_positive_rate options for indexes (#6416)
* feat: add `granularity` and `false_positive_rate` options for indexes Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * upgrade proto Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
 Ruihang XiaandCopilot
|
7a9444c85b |
refactor: remove staled manifest structures (#6382)
* refactor: remove staled manifest structures Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * Update src/store-api/src/lib.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
 LFC
|
bb12be3310 |
refactor: scan Batches directly (#6369)
* refactor: scan `Batch`es directly Signed-off-by: luofucong <luofc@foxmail.com> * fix ci Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
Yingwen
|
39d3e0651d |
feat: Support ListMetadataRequest to retrieve regions' metadata (#6348)
* feat: support list metadata in region server Signed-off-by: evenyag <realevenyag@gmail.com> * test: add test for list region metadata Signed-off-by: evenyag <realevenyag@gmail.com> * feat: return null if region not exists Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
LFC
|
e072726ea8 |
refactor: make scanner creation async (#6349)
* refactor: make scanner creation async Signed-off-by: luofucong <luofc@foxmail.com> * resolve PR comments Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 Ning SunandRuihang Xia
|
e78c3e1eaa |
refactor: make metadata region option opt-in (#6350)
* refactor: make metadata region option opt-in Signed-off-by: Ning Sun <sunning@greptime.com> * fix: preserve wal_options for metadata region Signed-off-by: Ning Sun <sunning@greptime.com> * Update src/metric-engine/src/engine/create.rs Co-authored-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> Co-authored-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Lin Yihai
|
438791b3e4 |
feat: Add DROP DEFAULT (#6290)
* feat: Add `DROP DEFAULT` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> * chore: use `next_token` Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> --------- Signed-off-by: Yihai Lin <yihai-lin@foxmail.com> |
||
 zyy17
|
fdf32a8f46 |
refactor: respect data_home as root data home directory (#6050)
* refactor: initialize logging dir by using data_home * chore: remove tail '/' for dir name |
||
|
Weny Xu
|
80c5af0ecf |
fix: ignore incomplete WAL entries during read (#6251)
* fix: ignore incomplete entry * fix: fix unit tests |
||
|
Lei, HUANG
|
4e615e8906 |
feat(wal): support bulk wal entries (#6178)
* feat/bulk-wal: ### Refactor: Simplify Data Handling in LogStore Implementations - **`kafka/log_store.rs`, `raft_engine/log_store.rs`, `wal.rs`, `raw_entry_reader.rs`, `logstore.rs`:** - Refactored `entry` and `build_entry` functions to accept `Vec<u8>` directly instead of `&mut Vec<u8>`. - Removed usage of `std::mem::take` for data handling, simplifying the code and improving readability. - Updated test cases to align with the new function signatures. * feat/bulk-wal: ### Add Support for Bulk WAL Entries and Flight Data Encoding - **Add `raw_data` field to `BulkPart` and related structs**: Updated `BulkPart` and related structures in `src/mito2/src/memtable/bulk/part.rs`, `src/mito2/src/memtable/simple_bulk_memtable.rs`, `src/mito2/src/memtable/time_partition.rs`, `src/mito2/src/region_write_ctx.rs`, `src/mito2/src/worker/handle_bulk_insert.rs`, and `src/store-api/src/region_request.rs` to include a new `raw_data` field for handling Arrow IPC data. - **Implement Flight Data Encoding**: Added a new module `flight` in `src/common/test-util/src/flight.rs` to encode record batches to Flight data format. - **Update `greptime-proto` dependency**: Changed the revision of the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml`. - **Enhance WAL Writer and Tests**: Modified `src/mito2/src/wal.rs` and related test files to support bulk WAL entries and added tests for encoding and handling bulk data. * feat/bulk-wal: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`. - **Add `common-grpc` Dependency**: Added `common-grpc` as a dependency in `Cargo.lock` and `src/mito2/Cargo.toml`. - **Refactor `BulkPart` Structure**: Removed `num_rows` field and added `num_rows()` method in `src/mito2/src/memtable/bulk/part.rs`. Updated related usages in `src/mito2/src/memtable/simple_bulk_memtable.rs`, `src/mito2/src/memtable/time_partition.rs`, `src/mito2/src/memtable/time_series.rs`, `src/mito2/src/region_write_ctx.rs`, and `src/mito2/src/worker/handle_bulk_insert.rs`. - **Implement `TryFrom` and `From` for `BulkWalEntry`**: Added implementations for converting between `BulkPart` and `BulkWalEntry` in `src/mito2/src/memtable/bulk/part.rs`. - **Handle Bulk Entries in Region Opener**: Added logic to process bulk entries in `src/mito2/src/region/opener.rs`. - **Fix `BulkInsertRequest` Handling**: Corrected `region_id` handling in `src/operator/src/bulk_insert.rs` and `src/store-api/src/region_request.rs`. - **Add Error Variant for `ConvertBulkWalEntry`**: Added a new error variant in `src/mito2/src/error.rs` for handling bulk WAL entry conversion errors. * fix: ci * feat/bulk-wal: Add bulk write operation in `opener.rs` - Enhanced the region write context by adding a call to `write_bulk()` after `write_memtable()` in `opener.rs`. - This change aims to improve the efficiency of writing operations by enabling bulk writes. * feat/bulk-wal: Enhance error handling and metrics in `bulk_insert.rs` - Updated `Inserter` to improve error handling by capturing the result of `datanode.handle(request)` and incrementing the `DIST_INGEST_ROW_COUNT` metric with the number of affected rows. * feat/bulk-wal: ### Remove Encode Error Handling for WAL Entries - **`error.rs`**: Removed the `EncodeWal` error variant and its associated handling. - **`wal.rs`**: Eliminated the `entry_encode_buf` buffer and its usage for encoding WAL entries. Replaced with direct encoding to a vector using `encode_to_vec()`. |
||
|
Lei, HUANG
|
27e339f628 |
perf: optimize bulk encode decode (#6161)
* main: **Enhancements to Flight Data Handling and Error Management** - **Flight Data Handling:** - Added `bytes` dependency in `Cargo.lock` and `Cargo.toml`. - Introduced `try_from_schema_bytes` and `try_decode_record_batch` methods in `FlightDecoder` to handle schema and record batch decoding more efficiently in `src/common/grpc/src/flight.rs`. - Updated `Inserter` in `src/operator/src/bulk_insert.rs` to utilize schema bytes directly, improving bulk insert operations. - **Error Management:** - Added `ArrowError` handling in `src/common/grpc/src/error.rs` to manage errors related to Arrow operations. - **Region Request Processing:** - Modified `make_region_bulk_inserts` in `src/store-api/src/region_request.rs` to use the new `FlightDecoder` methods for decoding Arrow IPC data. * - **Flight Data Handling:** - Added `bytes` dependency in `Cargo.lock` and `Cargo.toml`. - Introduced `try_from_schema_bytes` and `try_decode_record_batch` methods in `FlightDecoder` to handle schema and record batch decoding more efficiently in `src/common/grpc/src/flight.rs`. - Updated `Inserter` in `src/operator/src/bulk_insert.rs` to utilize schema bytes directly, improving bulk insert operations. - **Error Management:** - Added `ArrowError` handling in `src/common/grpc/src/error.rs` to manage errors related to Arrow operations. - **Region Request Processing:** - Modified `make_region_bulk_inserts` in `src/store-api/src/region_request.rs` to use the new `FlightDecoder` methods for decoding Arrow IPC data. * perf/optimize-bulk-encode-decode: Update `greptime-proto` dependency and refactor error handling - **Dependency Update**: Updated the `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`. - **Error Handling Refactor**: Removed the `Prost` error variant from `MetadataError` in `src/store-api/src/metadata.rs`. - **Error Handling Improvement**: Replaced `unwrap` with `context(FlightCodecSnafu)` for error handling in `make_region_bulk_inserts` function in `src/store-api/src/region_request.rs`. * fix: clippy * fix: toml * perf/optimize-bulk-encode-decode: ### Update `Cargo.toml` Dependencies - Updated the `bytes` dependency to use the workspace version in `Cargo.toml`. * perf/optimize-bulk-encode-decode: **Fix payload assignment in `bulk_insert.rs`** - Corrected the assignment of the `payload` field in the `ArrowIpc` struct within the `Inserter` implementation in `bulk_insert.rs`. * use main branch proto |
||
|
Lei, HUANG
|
4b71e493f7 |
feat!: revise compaction picker (#6121)
* - **Refactor `RegionFilePathFactory` to `RegionFilePathProvider`:** Updated references and implementations in `access_layer.rs`, `write_cache.rs`, and related test files to use the new struct name. - **Add `max_file_size` support in compaction:** Introduced `max_file_size` option in `PickerOutput`, `SerializedPickerOutput`, and `WriteOptions` in `compactor.rs`, `picker.rs`, `twcs.rs`, and `window.rs`. - **Enhance Parquet writing logic:** Modified `parquet.rs` and `parquet/writer.rs` to support optional `max_file_size` and added a test case `test_write_multiple_files` to verify writing multiple files based on size constraints. **Refactor Parquet Writer Initialization and File Handling** - Updated `ParquetWriter` in `writer.rs` to handle `current_indexer` as an `Option`, allowing for more flexible initialization and management. - Introduced `finish_current_file` method to encapsulate logic for completing and transitioning between SST files, improving code clarity and maintainability. - Enhanced error handling and logging with `debug` statements for better traceability during file operations. - **Removed Output Size Enforcement in `twcs.rs`:** - Deleted the `enforce_max_output_size` function and related logic to simplify compaction input handling. - **Added Max File Size Option in `parquet.rs`:** - Introduced `max_file_size` in `WriteOptions` to control the maximum size of output files. - **Refactored Indexer Management in `parquet/writer.rs`:** - Changed `current_indexer` from an `Option` to a direct `Indexer` type. - Implemented `roll_to_next_file` to handle file transitions when exceeding `max_file_size`. - Simplified indexer initialization and management logic. - **Refactored SST File Handling**: - Introduced `FilePathProvider` trait and its implementations (`WriteCachePathProvider`, `RegionFilePathFactory`) to manage SST and index file paths. - Updated `AccessLayer`, `WriteCache`, and `ParquetWriter` to use `FilePathProvider` for path management. - Modified `SstWriteRequest` and `SstUploadRequest` to use path providers instead of direct paths. - Files affected: `access_layer.rs`, `write_cache.rs`, `parquet.rs`, `writer.rs`. - **Enhanced Indexer Management**: - Replaced `IndexerBuilder` with `IndexerBuilderImpl` and made it async to support dynamic indexer creation. - Updated `ParquetWriter` to handle multiple indexers and file IDs. - Files affected: `index.rs`, `parquet.rs`, `writer.rs`. - **Removed Redundant File ID Handling**: - Removed `file_id` from `SstWriteRequest` and `CompactionOutput`. - Updated related logic to dynamically generate file IDs where necessary. - Files affected: `compaction.rs`, `flush.rs`, `picker.rs`, `twcs.rs`, `window.rs`. - **Test Adjustments**: - Updated tests to align with new path and indexer management. - Introduced `FixedPathProvider` and `NoopIndexBuilder` for testing purposes. - Files affected: `sst_util.rs`, `version_util.rs`, `parquet.rs`. * chore: rebase main * feat/multiple-compaction-output: ### Add Benchmarking and Refactor Compaction Logic - **Benchmarking**: Added a new benchmark `run_bench` in `Cargo.toml` and implemented benchmarks in `benches/run_bench.rs` using Criterion for `find_sorted_runs` and `reduce_runs` functions. - **Compaction Module Enhancements**: - Made `run.rs` public and refactored the `Ranged` and `Item` traits to be public. - Simplified the logic in `find_sorted_runs` and `reduce_runs` by removing `MergeItems` and related functions. - Introduced `find_overlapping_items` for identifying overlapping items. - **Code Cleanup**: Removed redundant code and tests related to `MergeItems` in `run.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Add Benchmarks - **Compaction Logic Improvements**: - Updated `reduce_runs` function in `src/mito2/src/compaction/run.rs` to remove the target parameter and improve the logic for selecting files to merge based on minimum penalty. - Enhanced `find_overlapping_items` to handle unsorted inputs and improve overlap detection efficiency. - **Benchmark Enhancements**: - Added `bench_find_overlapping_items` in `src/mito2/benches/run_bench.rs` to benchmark the new `find_overlapping_items` function. - Extended existing benchmarks to include larger data sizes. - **Testing Enhancements**: - Updated tests in `src/mito2/src/compaction/run.rs` to reflect changes in `reduce_runs` and added new tests for `find_overlapping_items`. - **Logging and Debugging**: - Improved logging in `src/mito2/src/compaction/twcs.rs` to provide more detailed information about compaction decisions. * feat/multiple-compaction-output: ### Refactor and Enhance Compaction Logic - **Refactor `find_overlapping_items` Function**: Changed the function signature to accept slices instead of mutable vectors in `run.rs`. - **Rename and Update Struct Fields**: Renamed `penalty` to `size` in `SortedRun` struct and updated related logic in `run.rs`. - **Enhance `reduce_runs` Function**: Improved logic to sort runs by size and limit probe runs to 100 in `run.rs`. - **Add `merge_seq_files` Function**: Introduced a new function `merge_seq_files` in `run.rs` for merging sequential files. - **Modify `TwcsPicker` Logic**: Updated the compaction logic to use `merge_seq_files` when only one run is found in `twcs.rs`. - **Remove `enforce_file_num` Function**: Deleted the `enforce_file_num` function and its related test cases in `twcs.rs`. * feat/multiple-compaction-output: ### Enhance Compaction Logic and Testing - **Add `merge_seq_files` Functionality**: Implemented the `merge_seq_files` function in `run.rs` to optimize file merging based on scoring systems. Updated benchmarks in `run_bench.rs` to include `bench_merge_seq_files`. - **Improve Compaction Strategy in `twcs.rs`**: Modified the compaction logic to handle file merging more effectively, considering file size and overlap. - **Update Tests**: Enhanced test coverage in `compaction_test.rs` and `append_mode_test.rs` to validate new compaction logic and file merging strategies. - **Remove Unused Function**: Deleted `new_file_handles` from `test_util.rs` as it was no longer needed. * feat/multiple-compaction-output: ### Refactor TWCS Compaction Options - **Refactor Compaction Logic**: Simplified the TWCS compaction logic by replacing multiple parameters (`max_active_window_runs`, `max_active_window_files`, `max_inactive_window_runs`, `max_inactive_window_files`) with a single `trigger_file_num` parameter in `picker.rs`, `twcs.rs`, and `options.rs`. - **Update Tests**: Adjusted test cases to reflect the new compaction logic in `append_mode_test.rs`, `compaction_test.rs`, `filter_deleted_test.rs`, `merge_mode_test.rs`, and various test files under `tests/cases`. - **Modify Engine Options**: Updated engine option keys to use `trigger_file_num` in `mito_engine_options.rs` and `region_request.rs`. - **Fuzz Testing**: Updated fuzz test generators and translators to accommodate the new compaction parameter in `alter_expr.rs` and related files. This refactor aims to streamline the compaction configuration by reducing the number of parameters and simplifying the codebase. * chore: add trailing space * fix license header * feat/revise-compaction-picker: **Limit File Processing and Optimize Merge Logic in `run.rs`** - Introduced a limit to process a maximum of 100 files in `merge_seq_files` to control time complexity. - Adjusted logic to calculate `target_size` and iterate over files using the limited set of files. - Updated scoring calculations to use the limited file set, ensuring efficient file merging. * feat/revise-compaction-picker: ### Add Compaction Metrics and Remove Debug Logging - **Compaction Metrics**: Introduced new histograms `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` to track compaction input and output file sizes in `metrics.rs`. Updated `compactor.rs` to observe these metrics during the compaction process. - **Logging Cleanup**: Removed debug logging of file ranges during the merge process in `twcs.rs`. * feat/revise-compaction-picker: ## Enhance Compaction Logic and Metrics - **Compaction Logic Improvements**: - Added methods `input_file_size` and `output_file_size` to `MergeOutput` in `compactor.rs` to streamline file size calculations. - Updated `Compactor` implementation to use these methods for metrics tracking. - Modified `Ranged` trait logic in `run.rs` to improve range comparison. - Enhanced test cases in `run.rs` to reflect changes in compaction logic. - **Metrics Enhancements**: - Changed `COMPACTION_INPUT_BYTES` and `COMPACTION_OUTPUT_BYTES` from histograms to counters in `metrics.rs` for better performance tracking. - **Debugging and Logging**: - Added detailed logging for compaction pick results in `twcs.rs`. - Implemented custom `Debug` trait for `FileMeta` in `file.rs` to improve debugging output. - **Testing Enhancements**: - Added new test `test_compaction_overlapping_files` in `compaction_test.rs` to verify compaction behavior with overlapping files. - Updated `merge_mode_test.rs` to reflect changes in file handling during scans. * feat/revise-compaction-picker: ### Update `FileHandle` Debug Implementation - **Refactor Debug Output**: Simplified the `fmt::Debug` implementation for `FileHandle` in `src/mito2/src/sst/file.rs` by consolidating multiple fields into a single `meta` field using `meta_ref()`. - **Atomic Operations**: Updated the `deleted` field to use atomic loading with `Ordering::Relaxed`. * Trigger CI * feat/revise-compaction-picker: **Update compaction logic and default options** - **`twcs.rs`**: Enhanced logging for compaction pick results by improving the formatting for better readability. - **`options.rs`**: Modified the default `max_output_file_size` in `TwcsOptions` from 2GB to 512MB to optimize file handling and performance. * feat/revise-compaction-picker: Refactor `find_overlapping_items` to use an external result vector - Updated `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to accept a mutable result vector instead of returning a new vector, improving memory efficiency. - Modified benchmarks in `src/mito2/benches/bench_compaction_picker.rs` to accommodate the new function signature. - Adjusted tests in `src/mito2/src/compaction/run.rs` to use the updated function signature, ensuring correct functionality with the new approach. * feat/revise-compaction-picker: Improve file merging logic in `run.rs` - Refactor the loop logic in `merge_seq_files` to simplify the iteration over file groups. - Adjust the range for `end_idx` to include the endpoint, allowing for more flexible group selection. - Remove the condition that skips groups with only one file, enabling more comprehensive processing of file sequences. * feat/revise-compaction-picker: Enhance `find_overlapping_items` with `SortedRun` and Update Tests - Refactor `find_overlapping_items` in `src/mito2/src/compaction/run.rs` to utilize the `SortedRun` struct for improved efficiency and clarity. - Introduce a `sorted` flag in `SortedRun` to optimize sorting operations. - Update test cases in `src/mito2/benches/bench_compaction_picker.rs` to accommodate changes in `find_overlapping_items` by using `SortedRun`. - Add `From<Vec<T>>` implementation for `SortedRun` to facilitate easy conversion from vectors. * feat/revise-compaction-picker: **Enhancements in `compaction/run.rs`:** - Added `ReadableSize` import to handle size calculations. - Modified the logic in `merge_seq_files` to clamp the calculated target size to a maximum of 2GB when `max_file_size` is not provided. * feat/revise-compaction-picker: Add Default Max Output Size Constant for Compaction Introduce DEFAULT_MAX_OUTPUT_SIZE constant to define the default maximum compaction output file size as 2GB. Refactor the merge_seq_files function to utilize this constant, ensuring consistent and maintainable code for handling file size limits during compaction. |
||
|
Lei, HUANG
|
f46cdbd66b |
fix: fast path for single region bulk insert (#6104)
* fix/fast-path-for-single-region-bulk-insert: ### Commit Summary - **Refactor `try_decode` Method**: Updated the `try_decode` method in `FlightDecoder` to accept a reference to `FlightData` instead of consuming it. This change affects multiple files including `database.rs`, `region.rs`, `flight.rs`, `bulk_insert.rs`, `stream.rs`, and `region_request.rs`. - **Optimize Bulk Insert Handling**: Added a fast path for handling bulk inserts when only one region is involved in `bulk_insert.rs`. * fix/fast-path-for-single-region-bulk-insert: Improve `FlightDecoder` usage in tests - Updated `try_decode` method calls in `flight.rs` to remove unnecessary references for `d1`, `d2`, and `d3`. - Ensured consistency in handling `FlightMessage` variants within test cases. * fix/fast-path-for-single-region-bulk-insert: **Enhancement: Skip Empty Regions in Bulk Insert** - Updated `bulk_insert.rs` to improve efficiency by skipping regions without data during the bulk insert process. This change ensures that regions with a `true_count` of zero are not processed, optimizing resource usage and performance. * fix/fast-path-for-single-region-bulk-insert: ### Commit Summary - **Refactor `RegionMask` Handling**: - Introduced `RegionMask` struct to encapsulate boolean array and selected rows count. - Updated methods to use `RegionMask` instead of `BooleanArray` for region selection. - Affected files: `bulk_insert.rs`, `multi_dim.rs`, `partition.rs`, `splitter.rs`. - **Optimize Region Selection**: - Removed unnecessary checks for empty regions in `bulk_insert.rs`. - Improved logic for handling default regions in `multi_dim.rs`. - **Update Tests**: - Modified test cases to accommodate `RegionMask` changes. - Affected files: `multi_dim.rs`, `splitter.rs`. * fix/fast-path-for-single-region-bulk-insert: **Enhancements to MultiDimPartitionRule Logic and Tests** - **`multi_dim.rs`**: Improved the logic for selecting rows in `MultiDimPartitionRule` by optimizing the selection process when only one region is present. - **Tests**: Added new test cases to verify the behavior of default regions with unselected rows, existing default regions, and scenarios where all rows are selected. These tests ensure robust handling of partition rules and validate the correct assignment of rows to regions. |
||
 Lei, HUANGandLei
|
5a9023d6b3 |
feat(bulk): write to multiple time partitions (#6086)
* add benchmark for splitting according to time partition * feat/write-to-multiple-time-partitions: **Enhancements to Bulk Processing and Time Partitioning** - **`part.rs`**: Added `Snafu` to imports and introduced `timestamp_index` in `BulkPart` struct. Implemented `timestamps` method for accessing timestamp columns. - **`simple_bulk_memtable.rs`**: Updated tests to include `timestamp_index` initialization. - **`time_partition.rs`**: Enhanced `TimePartition` to support partial writes with `write_record_batch_partial`. Implemented `split_record_batch` for filtering records by timestamp range. Added comprehensive tests for `split_record_batch`. - **`handle_bulk_insert.rs`**: Modified to retrieve timestamp index and column together, updating `BulkPart` initialization with `timestamp_index`. * feat/write-to-multiple-time-partitions: ### Enhance Time Partitioning Logic - **`time_partition.rs`**: - Introduced `HashSet` for efficient partition management. - Refactored `write_bulk` to handle multiple partitions and added `find_partitions_by_time_range` for identifying existing and missing partitions. - Updated `get_or_create_time_partition` to manage partition creation. - Added comprehensive tests for partition finding logic, covering various scenarios including overlapping and non-overlapping time ranges. - **Tests**: - Added `test_find_partitions_by_time_range` to validate new partitioning logic. - Updated `test_split_record_batch` to ensure correct record batch splitting behavior. * feat/write-to-multiple-time-partitions: ### Enhance Time Partitioning and Testing in `time_partition.rs` - **Time Partitioning Enhancements**: - Updated `split_record_batch` to handle multiple timestamp units (`Second`, `Millisecond`, `Microsecond`, `Nanosecond`) by matching on `DataType`. - Improved filtering logic for timestamp arrays to support various time units. - **Testing Enhancements**: - Added `test_write_bulk` to verify writing across multiple partitions and scenarios in `time_partition.rs`. - Updated `test_split_record_batch` to use `TimestampMillisecondArray` for testing timestamp partitioning. - **Imports and Dependencies**: - Added necessary imports for new timestamp array types and testing utilities. * feat/write-to-multiple-time-partitions: ### Refactor and Enhance Time Partition Filtering - **Refactor Filtering Logic**: Consolidated the filtering logic for timestamp arrays using macros in `time_partition.rs` and `bench_filter_time_partition.rs`. This reduces code duplication and improves maintainability. - **Enhance `BulkPart` Struct**: Made fields in `BulkPart` public to facilitate easier access and manipulation in `memtable.rs` and `part.rs`. - **Rename Function**: Renamed `split_record_batch` to `filter_record_batch` for clarity in `time_partition.rs` and `bench_filter_time_partition.rs`. - **Add Feature Flag**: Introduced `int_roundings` feature in `lib.rs` to support new functionality. * refactor tests * feat/write-to-multiple-time-partitions: Improve timestamp handling in `time_partition.rs` - Enhanced safety comments for timestamp conversion to ensure clarity. - Modified logic to prevent overflow by using `div_euclid` for `bulk_start_sec` and `bulk_end_sec` calculations. - Adjusted the `filter_map` logic to correctly compute timestamps using `start_sec` and `part_duration_sec`. * feat/write-to-multiple-time-partitions: **Refactor timestamp handling and add utility function** - **Refactor `time_partition.rs`:** Simplified timestamp handling by replacing direct type access with a utility function to retrieve the timestamp unit. Improved error handling for timestamp conversion. - **Enhance `metadata.rs`:** Added `time_index_type` function to `RegionMetadata` to retrieve the timestamp type of the time index column, ensuring safer and more readable code. * feat/write-to-multiple-time-partitions: Refactor time partition variable names in `time_partition.rs` - Renamed variables for clarity: `bulk_start_sec` to `start_bucket` and `bulk_end_sec` to `end_bucket`. - Updated related logic to use new variable names for improved readability and maintainability. * feat/write-to-multiple-time-partitions: **Refactor variable names in `time_partition.rs`** - Updated variable names from `matching` and `missing` to `matchings` and `missings` for clarity and consistency. - Modified function calls and loop iterations to align with the new variable names. - Affected file: `src/mito2/src/memtable/time_partition.rs` * feat/write-to-multiple-time-partitions: ### Refactor variable names in `time_partition.rs` - Updated variable names for clarity in `time_partition.rs`: - Renamed `matchings` to `matching_parts` - Renamed `missings` to `missing_parts` - Adjusted logic to use new variable names in methods `find_partitions_by_time_range` and `write_record_batch`. * feat/write-to-multiple-time-partitions: ### Enhance Time Partition Handling - **`time_partition.rs`**: - Added `ArrayRef` to handle timestamp arrays, improving the partitioning logic by allowing more efficient timestamp range checks. - Enhanced `find_partitions_by_time_range` to support sparse data and handle different timestamp units (`Second`, `Millisecond`, `Microsecond`, `Nanosecond`). - Updated test cases to cover new scenarios, including sparse data and edge cases, ensuring robustness of partition handling. --------- Co-authored-by: Lei <lei@Leis-MacBook-Pro.local> |
||
|
Ruihang Xia
|
bbb6f8685e |
feat: implement commutativity rule for prom-related plans (#5875)
* feat: implement commutativity rule for prom-related plans Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix range manipulate deserializer Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * blocklist in commutativity rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change dictionary type Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle partition and ordering Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update tests Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * add rate, increase and delta Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * regexp_replace uses empty string instead of null value Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
6ab0f0cc5c |
fix: alter table modify type should also modify default value (#6049)
* fix: select after alter * fix: insert a proper row&catch a bug * fix: alter table modify type modify default value type too * refactor: per review * chore: per review * refactor: per review * refactor: per review |
||
|
Lei, HUANG
|
8685ceb232 |
feat: impl bulk memtable and bridge bulk inserts (#6054)
* feat/bridge-bulk-insert:
## Implement Bulk Insert and Update Dependencies
- **Bulk Insert Implementation**: Added `handle_bulk_inserts` method in `src/operator/src/bulk_insert.rs` to manage bulk insert requests using `FlightDecoder` and `FlightData`.
- **Dependency Updates**: Updated `Cargo.lock` and `Cargo.toml` to use the latest revision of `greptime-proto` and added new dependencies like `arrow`, `arrow-ipc`, `bytes`, and `prost`.
- **gRPC Enhancements**: Modified `put_record_batch` method in `src/frontend/src/instance/grpc.rs` and `src/servers/src/grpc/flight.rs` to handle `FlightData` instead of `RawRecordBatch`.
- **Error Handling**: Added new error types in `src/operator/src/error.rs` for handling Arrow operations and decoding flight data.
- **Miscellaneous**: Updated `src/operator/src/insert.rs` to expose `partition_manager` and `node_manager` as public fields.
* feat/bridge-bulk-insert:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`.
- **Refactor gRPC Query Handling**: Removed `RawRecordBatch` usage from `grpc.rs`, `flight.rs`, `greptime_handler.rs`, and test files, simplifying the gRPC query handling.
- **Enhance Bulk Insert Logic**: Improved bulk insert logic in `bulk_insert.rs` and `region_request.rs` by using `FlightDecoder` and `BooleanArray` for better performance and clarity.
- **Add `common-grpc` Dependency**: Added `common-grpc` as a workspace dependency in `store-api/Cargo.toml` to support gRPC functionalities.
* fix: clippy
* fix schema serialization
* feat/bridge-bulk-insert:
Add error handling for encoding/decoding in `metadata.rs` and `region_request.rs`
- Introduced new error variants `FlightCodec` and `Prost` in `MetadataError` to handle encoding/decoding failures in `metadata.rs`.
- Updated `make_region_bulk_inserts` function in `region_request.rs` to use `context` for error handling with `ProstSnafu` and `FlightCodecSnafu`.
- Enhanced error handling for `FlightData` decoding and `filter_record_batch` operations.
* fix: test
* refactor: rename
* allow empty app_metadata in FlightData
* feat/bridge-bulk-insert:
- **Remove Logging**: Removed unnecessary logging of affected rows in `region_server.rs`.
- **Error Handling Enhancement**: Improved error handling in `bulk_insert.rs` by adding context to `split_record_batch` and handling single datanode fast path.
- **Error Enum Cleanup**: Removed unused `Arrow` error variant from `error.rs`.
* fix: standalone test
* feat/bridge-bulk-insert:
### Enhance Bulk Insert Handling and Metadata Management
- **`lib.rs`**: Enabled the `result_flattening` feature for improved error handling.
- **`request.rs`**: Made `name_to_index` and `has_null` fields public in `WriteRequest` for better accessibility.
- **`handle_bulk_insert.rs`**:
- Added `handle_record_batch` function to streamline processing of bulk insert payloads.
- Improved error handling and task management for bulk insert operations.
- Updated `region_metadata_to_column_schema` to return both column schemas and a name-to-index map for efficient data access.
* feat/bridge-bulk-insert:
- **Refactor `handle_bulk_insert.rs`:**
- Replaced `handle_record_batch` with `handle_payload` for handling payloads.
- Modified the fast path to use `common_runtime::spawn_global` for asynchronous task execution.
- **Optimize `multi_dim.rs`:**
- Added a fast path for single-region scenarios in `MultiDimPartitionRule::partition_record_batch`.
* feat/bridge-bulk-insert:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`.
- **Optimize Memory Allocation**: Increased initial and builder capacities in `time_series.rs` to improve performance.
- **Enhance Data Handling**: Modified `bulk_insert.rs` to use `Bytes` for efficient data handling.
- **Improve Bulk Insert Logic**: Refined the bulk insert logic in `region_request.rs` to handle schema and payload data more effectively and optimize record batch filtering.
- **String Handling Improvement**: Updated string conversion in `helper.rs` for better performance.
* fix: clippy warnings
* feat/bridge-bulk-insert:
**Add Metrics and Improve Error Handling**
- **Metrics Enhancements**: Introduced new metrics for bulk insert operations in `metrics.rs`, `bulk_insert.rs`, `greptime_handler.rs`, and `region_request.rs`. Added `HANDLE_BULK_INSERT_ELAPSED`, `BULK_REQUEST_MESSAGE_SIZE`, and `GRPC_BULK_INSERT_ELAPSED` histograms to
monitor performance.
- **Error Handling Improvements**: Removed unnecessary error handling in `handle_bulk_insert.rs` by eliminating redundant `let _ =` patterns.
- **Dependency Updates**: Added `lazy_static` and `prometheus` to `Cargo.lock` and `Cargo.toml` for metrics support.
- **Code Refactoring**: Simplified function calls in `region_server.rs` and `handle_bulk_insert.rs` for better readability.
* chore: rebase main
* implement simple bulk memtable
* impl write_bulk
* implement simple bulk memtable
* feat/simple-bulk-memtable:
### Enhance Time-Series Memtable and Bulk Insert Handling
- **Visibility Modifications**: Made `mutable_array` in `PrimitiveVectorBuilder` and `StringVectorBuilder` public in `primitive.rs` and `string.rs`.
- **New Module**: Added `builder.rs` to `memtable` for time-series builders, including `FieldBuilder` and `StringBuilder` implementations.
- **Bulk Insert Enhancements**:
- Added `sequence` field to `BulkPart` in `part.rs` and updated its handling in `simple_bulk_memtable.rs` and `region_write_ctx.rs`.
- Introduced metrics for bulk insert operations in `metrics.rs` and `bulk_insert.rs`.
- **Performance Metrics**: Added timing metrics for write operations in `metrics.rs`, `region_write_ctx.rs`, and `handle_write.rs`.
- **Region Request Handling**: Updated `make_region_bulk_inserts` in `region_request.rs` to include performance metrics.
* feat/simple-bulk-memtable:
**Improve Memtable Stats Calculation and Add Metrics Timer**
- **`simple_bulk_memtable.rs`**: Refactored `stats` method to use `num_rows` for checking if rows have been written, improving accuracy in memory table statistics.
- **`handle_bulk_insert.rs`**: Introduced a metrics timer to measure the elapsed time for processing bulk requests, enhancing performance monitoring.
* feat/simple-bulk-memtable:
### Commit Message
**Enhancements and Bug Fixes**
- **Dependency Update**: Updated `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`.
- **Feature Addition**: Implemented `to_mutation` method in `BulkPart` to convert `BulkPart` to `Mutation` for fallback `write_bulk` implementation in `src/mito2/src/memtable/bulk/part.rs`.
- **Functionality Improvement**: Modified `write_bulk` method in `TimeSeriesMemtable` to support default implementation fallback to row iteration in `src/mito2/src/memtable/time_series.rs`.
- **Performance Optimization**: Enhanced `bulk_insert` handling by optimizing region request processing and data partitioning in `src/operator/src/bulk_insert.rs`.
- **Error Handling**: Added `ComputeArrow` error variant for better error management in `src/operator/src/error.rs`.
- **Code Refactoring**: Simplified region bulk insert request processing in `src/store-api/src/region_request.rs`.
* fix: some clippy warnings
* feat/simple-bulk-memtable:
### Commit Summary
- **Refactor Return Types to `Result`:**
Updated the return type of the `ranges` method in `memtable.rs`, `bulk.rs`, `partition_tree.rs`, `simple_bulk_memtable.rs`, `time_series.rs`, and `memtable_util.rs` to return `Result<MemtableRanges>` for better error handling.
- **Enhance Metrics Tracking:**
Improved metrics tracking by adding `num_rows` and `max_sequence` to `WriteMetrics` in `stats.rs`. Updated related methods in `partition_tree.rs`, `simple_bulk_memtable.rs`, `time_series.rs`, and `scan_region.rs` to utilize these metrics.
- **Remove Unused Imports:**
Cleaned up unused imports in `time_series.rs` to streamline the codebase.
* merge main
* remove useless error vairant
* use newer version of proto
* feat/simple-bulk-memtable:
Commit Message
Summary
Enhance FieldBuilder and StringBuilder functionality, add tests, and improve error handling.
Key Changes
• builder.rs:
• Added documentation for FieldBuilder methods.
• Renamed append_string_vector to append_vector in StringBuilder.
• simple_bulk_memtable.rs:
• Added new test cases for write_one, write_bulk, is_empty, stats, fork, and sequence_filter.
• time_series.rs:
• Improved error handling in ValueBuilder for type mismatches.
• memtable_util.rs:
• Removed unused imports and streamlined code.
These changes enhance the robustness and test coverage of the memtable components.
* feat/simple-bulk-memtable:
Improve Time Partition Matching Logic in `time_partition.rs`
- Enhanced the `write_bulk` method in `time_partition.rs` to improve the logic for matching partitions based on time ranges.
- Introduced a new mechanism to filter and select partitions that overlap with the record batch's timestamp range before writing.
* feat/simple-bulk-memtable:
Improve Metrics Handling in `bulk_insert.rs`
- Removed the `group_request_timer` and its associated metric observation to streamline the timing logic.
- Moved the `BULK_REQUEST_ROWS` metric observation to occur after filtering, ensuring accurate row count metrics.
* feat/simple-bulk-memtable:
**Enhance Stalled Requests Calculation and Update Metrics**
- **`worker.rs`**: Updated the `stalled_count` method to include both `reqs` and `bulk_reqs` in the calculation of stalled requests.
- **`bulk_insert.rs`**: Removed duplicate observation of `BULK_REQUEST_MESSAGE_SIZE` metric.
- **`metrics.rs`**: Changed the bucket strategy for `BULK_REQUEST_ROWS` from linear to exponential, improving the granularity of metrics collection.
* feat/simple-bulk-memtable:
**Refactor `StringVector` Usage and Update Method Signatures**
- **`src/datatypes/src/vectors/string.rs`**: Changed `StringVector`'s `array` field from public to private.
- **`src/mito2/src/memtable/builder.rs`**: Refactored `append_vector` method to `append_array`, updating its usage to work directly with `StringArray` instead of `StringVector`.
- **`src/mito2/src/memtable/time_series.rs`**: Updated `ValueBuilder` to handle `StringArray` directly, replacing `StringVector` usage with `StringArray` in the `FieldBuilder::String` case.
* feat/simple-bulk-memtable:
- **Refactor `PrimitiveVectorBuilder`**: Made `mutable_array` private in `src/datatypes/src/vectors/primitive.rs`.
- **Optimize `ValueBuilder`**: Replaced `UInt64VectorBuilder` and `UInt8VectorBuilder` with `Vec<u64>` and `Vec<u8>` for `sequence` and `op_type` in `src/mito2/src/memtable/time_series.rs`.
- **Improve Metrics Initialization**: Updated histogram bucket initialization to use `exponential_buckets` in `src/mito2/src/metrics.rs`.
* feat/simple-bulk-memtable:
Improve error handling in `simple_bulk_memtable.rs` and `time_series.rs`
- Enhanced error handling by using `OptionExt` for more concise error context management in `simple_bulk_memtable.rs` and `time_series.rs`.
- Replaced `ok_or` with `with_context` to streamline error context creation in both files.
* feat/simple-bulk-memtable:
**Enhance Time Partition Handling in `time_partition.rs`**
- Introduced `create_time_partition` function to streamline the creation of new time partitions, ensuring thread safety by acquiring a lock.
- Modified logic to handle cases where no matching time partitions exist, creating new partitions as needed.
- Updated `write_record_batch` and `write_one` methods to utilize the new partition creation logic, improving partition management and data writing efficiency.
* replace proto
* feat/simple-bulk-memtable:
Update `metrics.rs` to adjust the range of exponential buckets for bulk insert message rows from `10 ~ 1_000_000` to `10 ~ 100_000`.
|
||
|
LFCandCopilot
|
e787007eb5 |
feat: scan with sst minimal sequence (#6051)
* feat: scan with sst minimal sequence * Update src/store-api/src/storage/requests.rs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update proto --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Lei, HUANG
|
f298a110f9 |
feat: bridge bulk insert (#5927)
* feat/bridge-bulk-insert: ## Implement Bulk Insert and Update Dependencies - **Bulk Insert Implementation**: Added `handle_bulk_inserts` method in `src/operator/src/bulk_insert.rs` to manage bulk insert requests using `FlightDecoder` and `FlightData`. - **Dependency Updates**: Updated `Cargo.lock` and `Cargo.toml` to use the latest revision of `greptime-proto` and added new dependencies like `arrow`, `arrow-ipc`, `bytes`, and `prost`. - **gRPC Enhancements**: Modified `put_record_batch` method in `src/frontend/src/instance/grpc.rs` and `src/servers/src/grpc/flight.rs` to handle `FlightData` instead of `RawRecordBatch`. - **Error Handling**: Added new error types in `src/operator/src/error.rs` for handling Arrow operations and decoding flight data. - **Miscellaneous**: Updated `src/operator/src/insert.rs` to expose `partition_manager` and `node_manager` as public fields. * feat/bridge-bulk-insert: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in `Cargo.lock` and `Cargo.toml`. - **Refactor gRPC Query Handling**: Removed `RawRecordBatch` usage from `grpc.rs`, `flight.rs`, `greptime_handler.rs`, and test files, simplifying the gRPC query handling. - **Enhance Bulk Insert Logic**: Improved bulk insert logic in `bulk_insert.rs` and `region_request.rs` by using `FlightDecoder` and `BooleanArray` for better performance and clarity. - **Add `common-grpc` Dependency**: Added `common-grpc` as a workspace dependency in `store-api/Cargo.toml` to support gRPC functionalities. * fix: clippy * fix schema serialization * feat/bridge-bulk-insert: Add error handling for encoding/decoding in `metadata.rs` and `region_request.rs` - Introduced new error variants `FlightCodec` and `Prost` in `MetadataError` to handle encoding/decoding failures in `metadata.rs`. - Updated `make_region_bulk_inserts` function in `region_request.rs` to use `context` for error handling with `ProstSnafu` and `FlightCodecSnafu`. - Enhanced error handling for `FlightData` decoding and `filter_record_batch` operations. * fix: test * refactor: rename * allow empty app_metadata in FlightData * feat/bridge-bulk-insert: - **Remove Logging**: Removed unnecessary logging of affected rows in `region_server.rs`. - **Error Handling Enhancement**: Improved error handling in `bulk_insert.rs` by adding context to `split_record_batch` and handling single datanode fast path. - **Error Enum Cleanup**: Removed unused `Arrow` error variant from `error.rs`. * fix: standalone test * feat/bridge-bulk-insert: ### Enhance Bulk Insert Handling and Metadata Management - **`lib.rs`**: Enabled the `result_flattening` feature for improved error handling. - **`request.rs`**: Made `name_to_index` and `has_null` fields public in `WriteRequest` for better accessibility. - **`handle_bulk_insert.rs`**: - Added `handle_record_batch` function to streamline processing of bulk insert payloads. - Improved error handling and task management for bulk insert operations. - Updated `region_metadata_to_column_schema` to return both column schemas and a name-to-index map for efficient data access. * feat/bridge-bulk-insert: - **Refactor `handle_bulk_insert.rs`:** - Replaced `handle_record_batch` with `handle_payload` for handling payloads. - Modified the fast path to use `common_runtime::spawn_global` for asynchronous task execution. - **Optimize `multi_dim.rs`:** - Added a fast path for single-region scenarios in `MultiDimPartitionRule::partition_record_batch`. * feat/bridge-bulk-insert: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`. - **Optimize Memory Allocation**: Increased initial and builder capacities in `time_series.rs` to improve performance. - **Enhance Data Handling**: Modified `bulk_insert.rs` to use `Bytes` for efficient data handling. - **Improve Bulk Insert Logic**: Refined the bulk insert logic in `region_request.rs` to handle schema and payload data more effectively and optimize record batch filtering. - **String Handling Improvement**: Updated string conversion in `helper.rs` for better performance. * fix: clippy warnings * feat/bridge-bulk-insert: **Add Metrics and Improve Error Handling** - **Metrics Enhancements**: Introduced new metrics for bulk insert operations in `metrics.rs`, `bulk_insert.rs`, `greptime_handler.rs`, and `region_request.rs`. Added `HANDLE_BULK_INSERT_ELAPSED`, `BULK_REQUEST_MESSAGE_SIZE`, and `GRPC_BULK_INSERT_ELAPSED` histograms to monitor performance. - **Error Handling Improvements**: Removed unnecessary error handling in `handle_bulk_insert.rs` by eliminating redundant `let _ =` patterns. - **Dependency Updates**: Added `lazy_static` and `prometheus` to `Cargo.lock` and `Cargo.toml` for metrics support. - **Code Refactoring**: Simplified function calls in `region_server.rs` and `handle_bulk_insert.rs` for better readability. * chore: rebase main * chore: merge main |
||
  shuiyisongandNing Sun
|
3c943be189 |
chore: update rust toolchain (#5818)
* chore: update nightly version * chore: sort lint lines * chore: minor fix * chore: update nix * chore: update toolchain to 2024-04-14 * chore: update toolchain to 2024-04-15 * chore: remove unnecessory test * chore: do not assert oid in sqlness test * chore: fix margin issue * chore: fix cr issues * chore: fix cr issues --------- Co-authored-by: Ning Sun <sunning@greptime.com> |
||
 Yuhan Wang
|
41814bb49f |
feat: introduce high_watermark for remote wal logstore (#5877)
* feat: introduce high_watermark_since_flush * test: add unit test for high watermark * refactor: submit a request instead * fix: send reply before submit request * fix: no need to update twice * feat: update high watermark in background periodically * test: update unit tests * fix: update high watermark periodically * test: update unit tests * chore: apply review comments * chore: rename * chore: apply review comments * chore: clean up * chore: apply review comments |
||
|
Weny Xu
|
b8c6f1c8ed |
feat: sync region followers after altering regions (#5901)
* feat: close follower regions after dropping leader regions * chore: upgrade greptime-proto * feat: sync region followers after alter region operations * test: add tests * chore: apply suggestions from CR * chore: apply suggestions from CR |
||
|
Lei, HUANG
|
799c7cbfa9 |
feat(mito): bulk insert request handling on datanode (#5831)
* wip: implement basic request handling * feat/bulk-insert: ### Add Error Handling and Enhance Bulk Insert Functionality - **Error Handling**: Introduced a new error variant `ConvertDataType` in `error.rs` to handle conversion failures from `ConcreteDataType` to `ColumnDataType`. - **Bulk Insert Enhancements**: - Updated `WorkerRequest::BulkInserts` in `request.rs` to include metadata and sender. - Implemented `handle_bulk_inserts` in `worker.rs` to process bulk insert requests with region metadata. - Added functions `region_metadata_to_column_schema` and `record_batch_to_rows` in `handle_bulk_insert.rs` for schema conversion and row processing. - **API Changes**: Modified `RegionBulkInsertsRequest` in `region_request.rs` to include `region_id`. Files affected: `error.rs`, `request.rs`, `worker.rs`, `handle_bulk_insert.rs`, `region_request.rs`. * feat/bulk-insert: **Enhance Error Handling and Add Unit Tests** - Improved error handling in `record_batch_to_rows` function within `handle_bulk_insert.rs` by returning `Result` and handling errors with `context`. - Added unit tests for `region_metadata_to_column_schema` and `record_batch_to_rows` functions in `handle_bulk_insert.rs` to ensure correct functionality and error handling. * chore: update proto version * feat/bulk-insert: - **Refactor Error Handling**: Updated error handling in `error.rs` by modifying the `ConvertDataType` error handling. - **Improve Logging and Error Reporting**: Enhanced logging and error reporting in `worker.rs` by adding error messages for missing region metadata. - **Add New Error Type**: Introduced `DecodeArrowIpc` error in `metadata.rs` to handle Arrow IPC decoding failures. - **Handle Arrow IPC Decoding**: Updated `region_request.rs` to handle Arrow IPC decoding errors using the new `DecodeArrowIpc` error type. * chore: update proto version * feat/bulk-insert: Refactor `handle_bulk_insert.rs` to simplify row construction - Removed the mutable `current_row` vector and refactored `row_at` function to return a new vector directly. - Updated `record_batch_to_rows` to utilize the refactored `row_at` function for constructing rows. * feat/bulk-insert: ### Commit Summary **Enhancements in Region Server Request Handling** - Updated `region_server.rs` to include `RegionRequest::BulkInserts(_)` in the `RegionChange::Ingest` category, improving the handling of bulk insert operations. - Refined the categorization of region requests to ensure accurate mapping to `RegionChange` actions. |
||
|
Zhenchi
|
8d485e9be0 |
feat: support altering fulltext backend (#5896)
* feat: add `greptime_index_type` to `information_schema.key_column_usage` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix: show create Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
|
Weny Xu
|
c522893552 |
fix: ensure logical regions are synced during region sync (#5878)
* fix: ensure logical regions are synced during region sync * chore: apply suggestions from CR * chore: apply suggestions from CR |
||
|
Weny Xu
|
5a36fa5e18 |
fix: alway rejects write while downgrading region (#5842)
* fix: alway rejects write while downgrading region * chore: apply suggestions from CR |
||
|
Lei, HUANG
|
74d8fd00a4 |
fix: remove metadata region options (#5852)
* fix/remove-metadata-region-options: ### Add `SKIP_WAL_KEY` Option to Metric Engine - **Enhancements**: - Introduced `SKIP_WAL_KEY` to the metric engine options in `create.rs` and `mito_engine_options.rs`. - Updated test cases in `create.rs` to include `skip_wal` option and ensure it is removed for metadata regions. - **Refactoring**: - Updated `requests.rs` to use `SKIP_WAL_KEY` from `store_api::mito_engine_options`. These changes enhance the metric engine by allowing the option to skip Write-Ahead Logging (WAL) and ensure consistent usage of option keys across modules. * fix/remove-metadata-region-options: Add note for new options in mito_engine_options.rs • Introduce a comment to remind developers to check if new options should be removed in region_options_for_metadata_region within metric_engine::engine::create. * empty |
||
|
Weny Xu
|
981d51785b |
fix: throw errors instead of ignoring (#5792)
* fix: throw errors instead of ignoring * fix: fix unit tests * refactor: remove schema version check * fix: fix clippy * chore: remove unused error * refactor: remove schema version check * feat: handle mutliple results * feat: introduce consistency guard * fix: release consistency guard on datanode operation completion * test: add tests * chore: remove schema version * refactor: rename * test: add more tests * chore: print all error * tests: query table after alteration * log ignored request * refine fuzz test * chore: fix clippy and log mailbox message * chore: close prepared statement after execution * chore: add comment * chore: remove log * chore: rename to `ConsistencyPoison` * chore: remove unused error * fix: fix unit tests * chore: apply suggestions from CR |
||
|
Weny Xu
|
eab702cc02 |
feat: implement sync_region for metric engine (#5826)
* feat: implement `sync_region` for metric engine * chore: apply suggestions from CR * chore: upgrade proto |
||
|
Yuhan Wang
|
f73b61e767 |
feat(remote-wal): add remote wal prune procedure (#5714)
* feat: add remote wal prune procedure * feat: add retry logic and remove rollback * chore: simplify the logic * fix: remove REMOTE_WAL_LOCK * fix: use in-memory kv * perf: O(n) judgement * chore: add single write lock * test: add unit test * chore: remove unused function * chore: update comments * chore: apply comments * chore: apply comments |
||
|
Zhenchi
|
f797de3497 |
feat: add backend field to fulltext options (#5806)
* feat: add backend field to fulltext options Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * update proto Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix option conv Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix display Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> --------- Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> |
||
|
Weny Xu
|
dbb79c9671 |
feat: introduce CollectLeaderRegionHandler (#5811)
* feat: introduce `CollectLeaderRegionHandler` * feat: add to default handler group * fix: correct unit test * chore: rename |
||
|
Weny Xu
|
d701c18150 |
feat: introduce CustomizedRegionLeaseRenewer (#5762)
* feat: add manifest_version to `GrantedRegion` * chore: upgrade proto * chore: apply review suggestions * chore: apply suggestions from CR * feat: introduce `CustomizedRegionLeaseRenewerRef` * chore: upgrade to `103948` |
||
|
Weny Xu
|
41aee1f1b7 |
feat: implement sync_region for mito engine (#5765)
* chore: upgrade proto to `2d52b` * feat: add `SyncRegion` to `WorkerRequest` * feat: impl `sync_region` for `Engine` trait * test: add tests * chore: fmt code * chore: upgrade proto * chore: unify `RegionLeaderState` and `RegionFollowerState` * chore: check immutable memtable * chore: fix clippy * chore: apply suggestions from CR |
||
|
Yingwen
|
dbc25dd8da |
feat: expose scanner metrics to df execution metrics (#5699)
* feat: add metrics list to scanner * chore: add report metrics method * feat: use df metrics in PartitionMetrics * feat: pass execution metrics to scan partition * refactor: remove PartitionMetricsList * feat: better debug format for ScanMetricsSet * feat: do not expose all metrics to execution metrics by default * refactor: use struct destruction * feat: add metrics list to scanner * chore: Add custom Debug for ScanMetricsSet and partition metrics display * test: update sqlness result |
||
|
Lei, HUANG
|
40b52f3b13 |
feat(mito): allow skipping wal while creating tables (#5740)
* chore: add Noop Wal option * remove: WalOptionsAllocator::alloc method * feat/no-op-wal: ### Add Noop WAL Option - **`engine.rs`, `opener.rs`, `wal.rs`, `entry_reader.rs`, `handle_write.rs`, `provider.rs`**: - Introduced a new `WalOptions::Noop` variant to handle scenarios where no write-ahead logging is required. - Implemented `NoopEntryReader` to provide a no-operation entry reader. - Updated logic to skip WAL operations for regions with `Noop` option. - Added `Provider::Noop` to handle `Noop` operations in the provider logic. * feat/no-op-wal: ### Add `skip_wal` Option to Table Metadata - **Enhancements in `table_meta.rs`**: - Added a `skip_wal` parameter to the `create_wal_options` function to allow skipping WAL writes. - Updated the `create_table_route` function to utilize the `skip_wal` option from `table_info.meta.options`. - **Updates in `wal_options_allocator.rs`**: - Modified `alloc_batch` to handle the `skip_wal` flag, setting WAL options to `Noop` when true. - Added a test case `test_allocator_with_skip_wal` to verify the `skip_wal` functionality. - **Changes in `requests.rs`**: - Introduced `skip_wal` in `TableOptions` and added parsing logic. - Updated `TableOptions` display to include `skip_wal`. These changes introduce the ability to skip WAL writes for tables, enhancing flexibility in table metadata management. * feat/no-op-wal: **Add WAL Option Handling and Table Option Validation** - **`handle_write.rs`**: Introduced a check for `WalOptions::Noop` in the `RegionWorkerLoop` to skip WAL writing for regions with this option. - **`requests.rs`**: Added `SKIP_WAL_KEY` to the list of valid table options for enhanced table configuration validation. * feat/no-op-wal: ### Update WAL Options Allocation - **`key.rs`**: Modified the `allocate_region_wal_options` function to include an additional boolean parameter, enhancing the allocation logic. - **`wal_options_allocator.rs`**: Simplified the `test_allocator_with_skip_wal` test by removing unnecessary variable declarations and directly using `WalOptionsAllocator::RaftEngine`. These changes improve the flexibility and efficiency of WAL options allocation in the system. * chore: reformat code * feat/no-op-wal: **Enhancement:** Conditional Addition of `SKIP_WAL_KEY` in `requests.rs` - Updated `TableOptions` implementation in `requests.rs` to conditionally add `SKIP_WAL_KEY` to `key_vals` only when `self.skip_wal` is true, optimizing the key-value pair generation. * feat/no-op-wal: Update `requests.rs` tests to reflect changes in `skip_wal` option - Modified test assertions in `requests.rs` to remove `skip_wal=false` from expected strings. - Added a new test case to verify `skip_wal=true` is correctly represented in `TableOptions`. * feat/no-op-wal: Add Debug Logging and Improve Error Handling for WAL and Table Options • Introduced debug logging in wal.rs to skip obsolete regions, enhancing traceability. • Improved error handling in requests.rs by replacing warn with error propagation for invalid skip_wal values. • Added new test cases for skip_wal functionality, including SQL scripts and expected results, to ensure correct behavior and validation of the changes. |
||
|
Ruihang Xia
|
e81213728b |
feat: add/correct some kafka-related metrics (#5757)
* feat: add/correct some kafka-related metrics Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix dumb issues Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * per-partition produce latency Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Yingwen
|
352b197be4 |
feat: add hint for logical region in RegionScanner (#5684)
* feat: add a flag to check logical region * feat: sets logical region hint in metric engine * refactor: rename to logical_region |
||
|
Ruihang Xiaandevenyag
|
73ca39f37e |
feat: time series distribution in scanner (#5675)
* define distribution Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * feat: SeqScan support per series distribution * probe distribution Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * reverse sort order Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * more strict check Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * change null's ordering Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> Co-authored-by: evenyag <realevenyag@gmail.com> |
||
 yihongandWenyXu
|
ff0dcf12c5 |
perf: close issue 4974 by do not delete columns when drop logical region about 100 times faster (#5561)
* perf: do not delete columns when drop logical region in drop database Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: make ci happy Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: address review comments Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: address some comments Signed-off-by: yihong0618 <zouzou0208@gmail.com> * fix: drop stupid comments by copilot Signed-off-by: yihong0618 <zouzou0208@gmail.com> * chore: minor refactor * chore: minor refactor * chore: update grpetime-proto --------- Signed-off-by: yihong0618 <zouzou0208@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
286f225e50 |
fix: correct inverted_indexed_column_ids behavior (#5586)
* fix: correct `inverted_indexed_column_ids` * fix: fix unit tests |