* fix/filter-empty-batch-in-bulk-insert-api:
**Add Early Return for Empty Record Batches in `bulk_insert.rs`**
- Implemented an early return in the `Inserter` implementation to handle cases where `record_batch.num_rows()` is zero, improving efficiency by avoiding unnecessary processing.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
**Improve Bulk Insert Handling**
- **`handle_bulk_insert.rs`**: Added a check to handle cases where the batch has zero rows, immediately returning and sending a success response with zero rows processed.
- **`bulk_insert.rs`**: Enhanced logic to skip processing for masks that select none, optimizing the bulk insert operation by avoiding unnecessary iterations.
These changes improve the efficiency and robustness of the bulk insert process by handling edge cases more effectively.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
### Refactor and Error Handling Enhancements
- **Refactored Timestamp Handling**: Introduced `timestamp_array_to_primitive` function in `timestamp.rs` to streamline conversion of timestamp arrays to primitive arrays, reducing redundancy in `handle_bulk_insert.rs` and `bulk_insert.rs`.
- **Error Handling**: Added `InconsistentTimestampLength` error in `error.rs` to handle mismatched timestamp column lengths in bulk insert operations.
- **Bulk Insert Logic**: Updated `handle_bulk_insert.rs` to utilize the new timestamp conversion function and added checks for timestamp length consistency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/filter-empty-batch-in-bulk-insert-api:
**Refactor `bulk_insert.rs` to streamline imports**
- Simplified import statements by removing unused timestamp-related arrays and data types from the `arrow` crate in `bulk_insert.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix/reordered-write-cause-incorrect-kv:
- **Enhance Testing in `partition_tree.rs`**: Added comprehensive test functions such as `kv_region_metadata`, `key_values`, and `collect_kvs` to improve the robustness of key-value operations and ensure correct behavior of the `PartitionTreeMemtable`.
- **Improve Key Handling in `dict.rs`**: Modified `KeyDictBuilder` to handle both full and sparse keys, ensuring correct mapping and insertion. Added a new test `test_builder_finish_with_sparse_key` to validate the handling of sparse keys.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/reordered-write-cause-incorrect-kv:
### Refactor `partition_tree.rs` for Improved Key Handling
- **Refactored Key Handling**: Simplified the `key_values` function to accept an iterator of keys, removing hardcoded key-value pairs. This change enhances flexibility and reduces redundancy in key management.
- **Updated Test Cases**: Modified test cases to use the new `key_values` function signature, ensuring they iterate over keys dynamically rather than relying on predefined lists.
Files affected:
- `src/mito2/src/memtable/partition_tree.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/reordered-write-cause-incorrect-kv:
Enhance Testing in `partition_tree.rs`
- Added assertions to verify key-value collection after `memtable` and `forked` operations.
- Refactored key-value writing logic for clarity in `forked` operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/print-series-count-after-wal-replay:
### Add Series Count Functionality and Logging Enhancements
- **`time_partition.rs`**: Introduced `series_count` method to calculate the total timeseries count across all time partitions.
- **`opener.rs`**: Enhanced logging to include the total timeseries replayed during WAL replay.
- **`version.rs`**: Added `series_count` method to `VersionControlData` for approximating timeseries count in the current version.
- **`handler.rs`**: Added entry and exit logging for the `sql` function to trace execution flow.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/print-series-count-after-wal-replay:
### Remove Unused Import
- **File Modified**: `src/servers/src/http/handler.rs`
- **Change Summary**: Removed the unused `info` import from `common_telemetry`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/series-metrics:
### Add Metrics for Active Series and Values in Memtable
- **`simple_bulk_memtable.rs`**: Implemented `Drop` trait for `SimpleBulkMemtable` to decrement `MEMTABLE_ACTIVE_SERIES_COUNT` and `MEMTABLE_ACTIVE_VALUES_COUNT` upon dropping.
- **`time_series.rs`**:
- Introduced `SeriesMap` with `Drop` implementation to manage active series and values count.
- Updated `SeriesSet` and `Iter` to use `SeriesMap`.
- Added `num_values` method in `Series` to calculate the number of values.
- **`metrics.rs`**: Added `MEMTABLE_ACTIVE_SERIES_COUNT` and `MEMTABLE_ACTIVE_VALUES_COUNT` metrics to track active series and values in `TimeSeriesMemtable`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/series-metrics:
- Add metrics for active series and field builders
- Update dashboard
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/series-metrics:
**Add Series Count Tracking in Memtables**
- **`flush.rs`**: Updated `RegionFlushTask` to track and log the series count during memtable flush operations.
- **`memtable.rs`**: Introduced `series_count` in `MemtableStats` and added a method to retrieve it.
- **`partition_tree.rs`, `partition.rs`, `tree.rs`**: Implemented series count calculation in `PartitionTreeMemtable` and its components.
- **`simple_bulk_memtable.rs`, `time_series.rs`**: Integrated series count tracking in `SimpleBulkMemtable` and `TimeSeriesMemtable` implementations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Update src/mito2/src/memtable.rs
Co-authored-by: Yingwen <realevenyag@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
* fix: use `limit` params in jaeger http
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* refactor: only parse `max_duration` and `min_duration` when it's not empty
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* fix: handle the input for empty `limit` string
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* fix: missing the fileter for `service_name`
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* test: fix ci errors
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* fix: incorrect behavior of find_traces
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* fix: the logic of `find_traces()`
The correct logic should be:
1. Get all trace ids that match the filters;
2. Get all traces that match the trace ids from the previous query;
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* fix: integration test errors
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* refactor: add `empty_string_as_none`
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* refactor: refine naming
Signed-off-by: zyy17 <zyylsxm@gmail.com>
---------
Signed-off-by: zyy17 <zyylsxm@gmail.com>
* feat/answer-ctrl-c-in-mysql:
## Implement Connection ID-based Query Killing
### Key Changes:
- **Connection ID Management:**
- Added `connection_id` to `Session` and `QueryContext` in `src/session/src/lib.rs` and `src/session/src/context.rs`.
- Updated `MysqlInstanceShim` and `MysqlServer` to handle `connection_id` in `src/servers/src/mysql/handler.rs` and `src/servers/src/mysql/server.rs`.
- **KILL Statement Enhancements:**
- Introduced `Kill` enum to handle both `ProcessId` and `ConnectionId` in `src/sql/src/statements/kill.rs`.
- Updated `ParserContext` to parse `KILL QUERY <connection_id>` in `src/sql/src/parser.rs`.
- Modified `StatementExecutor` to support killing queries by `connection_id` in `src/operator/src/statement/kill.rs`.
- **Process Management:**
- Refactored `ProcessManager` to include `connection_id` in `src/catalog/src/process_manager.rs`.
- Added `kill_local_process` method for local query termination.
- **Testing:**
- Added tests for `KILL` statement parsing and execution in `src/sql/src/parser.rs`.
### Affected Files:
- `Cargo.lock`, `Cargo.toml`
- `src/catalog/src/process_manager.rs`
- `src/frontend/src/instance.rs`
- `src/frontend/src/stream_wrapper.rs`
- `src/operator/src/statement.rs`
- `src/operator/src/statement/kill.rs`
- `src/servers/src/mysql/federated.rs`
- `src/servers/src/mysql/handler.rs`
- `src/servers/src/mysql/server.rs`
- `src/servers/src/postgres.rs`
- `src/session/src/context.rs`
- `src/session/src/lib.rs`
- `src/sql/src/parser.rs`
- `src/sql/src/statements.rs`
- `src/sql/src/statements/kill.rs`
- `src/sql/src/statements/statement.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Conflicts:
Cargo.lock
Cargo.toml
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/answer-ctrl-c-in-mysql:
### Enhance Process Management and Execution
- **`process_manager.rs`**: Added a new method `find_processes_by_connection_id` to filter processes by connection ID, improving process management capabilities.
- **`kill.rs`**: Refactored the process killing logic to utilize the new `find_processes_by_connection_id` method, streamlining the execution flow and reducing redundant checks.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/answer-ctrl-c-in-mysql:
## Commit Message
### Update Process ID Type and Refactor Code
- **Change Process ID Type**: Updated the process ID type from `u64` to `u32` across multiple files to optimize memory usage. Affected files include `process_manager.rs`, `lib.rs`, `database.rs`, `instance.rs`, `server.rs`, `stream_wrapper.rs`, `kill.rs`, `federated.rs`, `handler.rs`, `server.rs`,
`postgres.rs`, `mysql_server_test.rs`, `context.rs`, `lib.rs`, and `test_util.rs`.
- **Remove Connection ID**: Removed the `connection_id` field and related logic from `process_manager.rs`, `lib.rs`, `instance.rs`, `server.rs`, `stream_wrapper.rs`, `kill.rs`, `federated.rs`, `handler.rs`, `server.rs`, `postgres.rs`, `mysql_server_test.rs`, `context.rs`, `lib.rs`, and `test_util.rs` to
simplify the codebase.
- **Refactor Process Management**: Refactored process management logic to improve clarity and maintainability in `process_manager.rs`, `kill.rs`, and `handler.rs`.
- **Enhance MySQL Server Handling**: Improved MySQL server handling by integrating process management in `server.rs` and `mysql_server_test.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/answer-ctrl-c-in-mysql:
### Add Process Manager to Postgres Server
- **`src/frontend/src/server.rs`**: Updated server initialization to include `process_manager`.
- **`src/servers/src/postgres.rs`**: Modified `MakePostgresServerHandler` to accept `process_id` for session creation.
- **`src/servers/src/postgres/server.rs`**: Integrated `process_manager` into `PostgresServer` for generating `process_id` during connection handling.
- **`src/servers/tests/postgres/mod.rs`** and **`tests-integration/src/test_util.rs`**: Adjusted test server setup to accommodate optional `process_manager`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/answer-ctrl-c-in-mysql:
Update `greptime-proto` Dependency
- Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`.
- `Cargo.lock`: Changed source revision from `d75a56e05a87594fe31ad5c48525e9b2124149ba` to `fdcbe5f1c7c467634c90a1fd1a00a784b92a4e80`.
- `Cargo.toml`: Updated the `greptime-proto` git revision to match the new commit.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
