* fix/filter-empty-batch-in-bulk-insert-api:
**Add Early Return for Empty Record Batches in `bulk_insert.rs`**
- Implemented an early return in the `Inserter` implementation to handle cases where `record_batch.num_rows()` is zero, improving efficiency by avoiding unnecessary processing.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
**Improve Bulk Insert Handling**
- **`handle_bulk_insert.rs`**: Added a check to handle cases where the batch has zero rows, immediately returning and sending a success response with zero rows processed.
- **`bulk_insert.rs`**: Enhanced logic to skip processing for masks that select none, optimizing the bulk insert operation by avoiding unnecessary iterations.
These changes improve the efficiency and robustness of the bulk insert process by handling edge cases more effectively.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
### Refactor and Error Handling Enhancements
- **Refactored Timestamp Handling**: Introduced `timestamp_array_to_primitive` function in `timestamp.rs` to streamline conversion of timestamp arrays to primitive arrays, reducing redundancy in `handle_bulk_insert.rs` and `bulk_insert.rs`.
- **Error Handling**: Added `InconsistentTimestampLength` error in `error.rs` to handle mismatched timestamp column lengths in bulk insert operations.
- **Bulk Insert Logic**: Updated `handle_bulk_insert.rs` to utilize the new timestamp conversion function and added checks for timestamp length consistency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
**Refactor `bulk_insert.rs` to streamline imports**
- Simplified import statements by removing unused timestamp-related arrays and data types from the `arrow` crate in `bulk_insert.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: actually split window to limit time range
feat: truly limit time range by split window
Update src/flow/src/batching_mode/state.rs
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
* chore: added stalled time window range
Signed-off-by: discord9 <discord9@163.com>
* fix: not flush all time range as too expensive
Signed-off-by: discord9 <discord9@163.com>
* test: make it more robust
Signed-off-by: discord9 <discord9@163.com>
* what
Signed-off-by: discord9 <discord9@163.com>
* feat: denfensively handle surplus
Signed-off-by: discord9 <discord9@163.com>
* refactor: per review,explain flush flow
Signed-off-by: discord9 <discord9@163.com>
* chore: per bugbot
Signed-off-by: discord9 <discord9@163.com>
* fix: a temp fix to make mirror insert go first(still need better fix to sync with mirror insert that happens before
Signed-off-by: discord9 <discord9@163.com>
* chore: add todo
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
* fix: label_replace and label_join functions in expressions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove update_fields
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: tql eval -> TQL EVAL
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: empty regex and not existing source label
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: simplfy test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix/process-manager-skip-fail-nodes:
- **Enhance Error Handling in `process_manager.rs`:**
Improved error handling by adding a warning log for failing nodes in the `list_process` method. This ensures that the process listing continues even if some nodes fail to respond.
- **Add Error Type Import in `process_manager.rs`:**
Included the `Error` type from the `error` module to handle errors more effectively within the `ProcessManager` implementation.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/process-manager-skip-fail-nodes:
**Enhancements to Debugging and Trait Implementation**
- **`process_manager.rs`**: Improved logging by adding more detailed error messages when skipping failing nodes.
- **`selector.rs`**: Enhanced the `FrontendClient` trait by adding the `Debug` trait bound to improve debugging capabilities.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
refactor/building-backend-in-object-store:
### Refactor Object Store Configuration
- **Centralize Object Store Configurations**: Moved object store configurations (`FileConfig`, `S3Config`, `OssConfig`, `AzblobConfig`, `GcsConfig`) to `object-store/src/config.rs`.
- **Error Handling Enhancements**: Introduced `object-store/src/error.rs` for improved error handling related to object store operations.
- **Factory Pattern for Object Store**: Implemented `object-store/src/factory.rs` to create object store instances, consolidating logic from `datanode/src/store.rs`.
- **Remove Redundant Store Implementations**: Deleted individual store files (`azblob.rs`, `fs.rs`, `gcs.rs`, `oss.rs`, `s3.rs`) from `datanode/src/store/`.
- **Update Usage of Object Store Config**: Updated references to `ObjectStoreConfig` in `datanode.rs`, `standalone.rs`, `config.rs`, and `error.rs` to use the new centralized configuration.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
