mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-05-18 05:50:41 +00:00
df53431dffbe4035c55a3e4ee38258ec0bda5b41
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Weny Xu
|
25687bb282 |
feat: add ddl timeout/wait options, repartition WITH parsing, meta-client startup refactor (#7589)
* feat: add ddl request timeouts and unify meta client startup Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: omplement ALTER TABLE repartition DDL options parsing Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add sqlness tests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: pass timeout argument to procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine comments Signed-off-by: WenyXu <wenymedia@gmail.com> * test: assert timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
69865c831d |
feat: batch region migration for failover (#7245)
* refactor: support multiple rows per event in event recorder Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: batch region migration for failover Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
shuiyisong
|
ee5b7ff3c8 |
chore: unify initialization of channel manager (#7159)
* chore: unify initialization of channel manager and extract loading tls Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fix cr issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
Ruihang Xia
|
c9377e7c5a |
build: bump rust edition to 2024 (#6920)
* bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
zyy17
|
777da35b0d |
refactor: unify the event recorder (#6689)
* refactor: unify the event recorder Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: add `table_name()` in `Event` trait Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: add `slow_query_options` in `Instance` Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: add `EventHandlerOptions` and `options()` in `EventHandler` trait Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: add `aggregate_events_by_type()` and support log mode of slow query Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: polish the code Signed-off-by: zyy17 <zyylsxm@gmail.com> * fix: clippy errors Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: support to set ttl by using extension of query context Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: refine the configs fields Signed-off-by: zyy17 <zyylsxm@gmail.com> * fix: sqlness test errors Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: use `Duration` type instead of `String` for ttl fields Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: use pre-allocation for building RowInsertRequests Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: fix clippy errors Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: code review Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: fix integration errors Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: polish code for `group_events_by_type()` and `build_row_inserts_request()`, also add the unit tests Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: refine comments --------- Signed-off-by: zyy17 <zyylsxm@gmail.com> |
||
|
zyy17
|
3a9f0220b5 |
fix: unable to record slow query (#6590)
* refactor: add process manager for prometheus query Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: modify `register_query()` API to accept parsed statement(`catalog::process_manager::QueryStatement`) Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: add the slow query timer in the `Tikcet` of ProcessManager Signed-off-by: zyy17 <zyylsxm@gmail.com> * test: add integration tests Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: add process manager in `do_exec_plan()` Signed-off-by: zyy17 <zyylsxm@gmail.com> * tests: add `test_postgres_slow_query` integration test Signed-off-by: zyy17 <zyylsxm@gmail.com> * chore: polish the code Signed-off-by: zyy17 <zyylsxm@gmail.com> * refactor: create a query ticket and slow query timer if the statement is a query in `query_statement()` Signed-off-by: zyy17 <zyylsxm@gmail.com> * fix: sqlness errors Signed-off-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: zyy17 <zyylsxm@gmail.com> |
||
Lei, HUANG
|
6684200fce |
fix: skip failing nodes when gathering porcess info (#6412)
* fix/process-manager-skip-fail-nodes: - **Enhance Error Handling in `process_manager.rs`:** Improved error handling by adding a warning log for failing nodes in the `list_process` method. This ensures that the process listing continues even if some nodes fail to respond. - **Add Error Type Import in `process_manager.rs`:** Included the `Error` type from the `error` module to handle errors more effectively within the `ProcessManager` implementation. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix/process-manager-skip-fail-nodes: **Enhancements to Debugging and Trait Implementation** - **`process_manager.rs`**: Improved logging by adding more detailed error messages when skipping failing nodes. - **`selector.rs`**: Enhanced the `FrontendClient` trait by adding the `Debug` trait bound to improve debugging capabilities. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Lei, HUANG
|
ecbbd2fbdb |
feat: handle Ctrl-C command in MySQL client (#6320)
* feat/answer-ctrl-c-in-mysql: ## Implement Connection ID-based Query Killing ### Key Changes: - **Connection ID Management:** - Added `connection_id` to `Session` and `QueryContext` in `src/session/src/lib.rs` and `src/session/src/context.rs`. - Updated `MysqlInstanceShim` and `MysqlServer` to handle `connection_id` in `src/servers/src/mysql/handler.rs` and `src/servers/src/mysql/server.rs`. - **KILL Statement Enhancements:** - Introduced `Kill` enum to handle both `ProcessId` and `ConnectionId` in `src/sql/src/statements/kill.rs`. - Updated `ParserContext` to parse `KILL QUERY <connection_id>` in `src/sql/src/parser.rs`. - Modified `StatementExecutor` to support killing queries by `connection_id` in `src/operator/src/statement/kill.rs`. - **Process Management:** - Refactored `ProcessManager` to include `connection_id` in `src/catalog/src/process_manager.rs`. - Added `kill_local_process` method for local query termination. - **Testing:** - Added tests for `KILL` statement parsing and execution in `src/sql/src/parser.rs`. ### Affected Files: - `Cargo.lock`, `Cargo.toml` - `src/catalog/src/process_manager.rs` - `src/frontend/src/instance.rs` - `src/frontend/src/stream_wrapper.rs` - `src/operator/src/statement.rs` - `src/operator/src/statement/kill.rs` - `src/servers/src/mysql/federated.rs` - `src/servers/src/mysql/handler.rs` - `src/servers/src/mysql/server.rs` - `src/servers/src/postgres.rs` - `src/session/src/context.rs` - `src/session/src/lib.rs` - `src/sql/src/parser.rs` - `src/sql/src/statements.rs` - `src/sql/src/statements/kill.rs` - `src/sql/src/statements/statement.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Conflicts: Cargo.lock Cargo.toml Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/answer-ctrl-c-in-mysql: ### Enhance Process Management and Execution - **`process_manager.rs`**: Added a new method `find_processes_by_connection_id` to filter processes by connection ID, improving process management capabilities. - **`kill.rs`**: Refactored the process killing logic to utilize the new `find_processes_by_connection_id` method, streamlining the execution flow and reducing redundant checks. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/answer-ctrl-c-in-mysql: ## Commit Message ### Update Process ID Type and Refactor Code - **Change Process ID Type**: Updated the process ID type from `u64` to `u32` across multiple files to optimize memory usage. Affected files include `process_manager.rs`, `lib.rs`, `database.rs`, `instance.rs`, `server.rs`, `stream_wrapper.rs`, `kill.rs`, `federated.rs`, `handler.rs`, `server.rs`, `postgres.rs`, `mysql_server_test.rs`, `context.rs`, `lib.rs`, and `test_util.rs`. - **Remove Connection ID**: Removed the `connection_id` field and related logic from `process_manager.rs`, `lib.rs`, `instance.rs`, `server.rs`, `stream_wrapper.rs`, `kill.rs`, `federated.rs`, `handler.rs`, `server.rs`, `postgres.rs`, `mysql_server_test.rs`, `context.rs`, `lib.rs`, and `test_util.rs` to simplify the codebase. - **Refactor Process Management**: Refactored process management logic to improve clarity and maintainability in `process_manager.rs`, `kill.rs`, and `handler.rs`. - **Enhance MySQL Server Handling**: Improved MySQL server handling by integrating process management in `server.rs` and `mysql_server_test.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/answer-ctrl-c-in-mysql: ### Add Process Manager to Postgres Server - **`src/frontend/src/server.rs`**: Updated server initialization to include `process_manager`. - **`src/servers/src/postgres.rs`**: Modified `MakePostgresServerHandler` to accept `process_id` for session creation. - **`src/servers/src/postgres/server.rs`**: Integrated `process_manager` into `PostgresServer` for generating `process_id` during connection handling. - **`src/servers/tests/postgres/mod.rs`** and **`tests-integration/src/test_util.rs`**: Adjusted test server setup to accommodate optional `process_manager`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/answer-ctrl-c-in-mysql: Update `greptime-proto` Dependency - Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`. - `Cargo.lock`: Changed source revision from `d75a56e05a87594fe31ad5c48525e9b2124149ba` to `fdcbe5f1c7c467634c90a1fd1a00a784b92a4e80`. - `Cargo.toml`: Updated the `greptime-proto` git revision to match the new commit. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Lei, HUANG
|
6da633e70d |
feat: support killing process (#6309)
* feat/kill-process: ### Add Cancellation Support and Enhance Process Management - **Cancellation Handle Implementation**: Introduced `CancellationHandle` in `cancellation_handle.rs` to facilitate cancellation of futures and streams. - **Process Management Enhancements**: - Updated `ProcessManager` in `process_manager.rs` to support cancellable processes using `CancellableProcess`. - Added `kill_process` method for terminating processes. - **Stream Wrapper Update**: - Replaced `StreamWrapper` with `CancellableStreamWrapper` in `stream_wrapper.rs` and `instance.rs` to handle stream cancellation. - **Error Handling**: - Added `StreamCancelled` error variant in `error.rs` to handle stream cancellation scenarios. - **gRPC Handler Update**: - Added `kill_process` gRPC method in `frontend_grpc_handler.rs` to allow external process termination. - **Dependency Updates**: - Updated `Cargo.lock` and `Cargo.toml` to include `common-base` and `tokio-util`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: **Enhancements and Bug Fixes** - **Dependency Update**: Updated `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision. - **Error Handling Improvements**: - Modified error variants in `src/catalog/src/error.rs` and `src/common/frontend/src/error.rs` to improve error messages and handling. - Added `FrontendNotFound` error variant for better error specificity. - **Process Management Enhancements**: - Updated `ProcessManager` in `src/catalog/src/process_manager.rs` to include `kill_process` functionality with server address validation. - Enhanced `FrontendClient` trait in `src/common/frontend/src/selector.rs` to support `kill_process` requests. - **gRPC Handler Update**: - Refactored `FrontendGrpcHandler` in `src/servers/src/grpc/frontend_grpc_handler.rs` to handle `kill_process` requests asynchronously and return process status. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: ### Add Kill Process Functionality - **`Cargo.lock`, `Cargo.toml`**: Added `common-frontend` as a dependency. - **`server.rs`, `builder.rs`, `instance.rs`**: Updated `FrontendInvoker` and `FrontendBuilder` to support process management. - **`error.rs`**: Introduced `InvalidProcessId` error for handling invalid process IDs. - **`statement.rs`, `kill.rs`**: Implemented `execute_kill` method in `StatementExecutor` to handle the `KILL` statement. - **`parser.rs`, `statement.rs`**: Updated SQL parser to recognize and parse the `KILL` statement. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: ## Add Cancellation Support to Query Execution - **`process_manager.rs`**: Updated `CancellationHandle` initialization to use `default()` method. - **`cancellation_handle.rs`**: Implemented `Debug` trait for `CancellationHandle` and added `Cancellation` and `CancellableFuture` structs to support cancellable futures. - **`error.rs`**: Introduced `Cancelled` error variant to handle query cancellations. - **`instance.rs`**: Integrated `CancellableFuture` to manage query execution with cancellation support. - **`stream_wrapper.rs`**: Modified `CancellableStreamWrapper` to use the new `waker()` method for cancellation handling. - **`statement.rs`**: Added `#[allow(clippy::too_many_arguments)]` to `StatementExecutor::new` to suppress clippy warnings. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: - **Add `MetaClientMissing` Error**: Introduced a new error variant `MetaClientMissing` in `error.rs` to handle missing meta client scenarios. - **Refactor Cancellation Handling**: Merged `cancellation_handle.rs` into `cancellation.rs` and updated related logic in `process_manager.rs`, `instance.rs`, and `stream_wrapper.rs`. - **Enhance Process Management**: Improved process management logic in `process_manager.rs` to handle process cancellation more effectively. - **Update Tests**: Added and updated tests in `cancellation.rs` and `stream_wrapper.rs` to cover new cancellation logic and error handling. - **Cargo.toml Update**: Adjusted workspace settings in `Cargo.toml` for `common-frontend`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: - **Add Tests for Process Management**: Introduced multiple async tests in `process_manager.rs` to verify query registration, deregistration, cancellation, and process killing functionalities. - **Update Error Message in SQL Parser**: Modified the expected error message in `parser.rs` to clarify the expected token as a "process id string literal". Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: ### Add Process Count Metrics to Catalog - **`metrics.rs`**: Introduced a new metric `PROCESS_LIST_COUNT` to track the count of running processes per catalog using `IntGaugeVec`. - **`process_manager.rs`**: Updated `CancellableProcess` to increment and decrement `PROCESS_LIST_COUNT` upon creation and destruction, respectively. Added a `Drop` implementation for `CancellableProcess` to handle metric updates. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: ### Fix process removal logic in `process_manager.rs` - Corrected the condition for removing an entry from the catalog in `ProcessManager` by using `o.get()` instead of `o.get_mut()`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: - **Error Handling Improvements**: - Updated status codes for `Error::FrontendNotFound` and `Error::MetaClientMissing` to `StatusCode::Unexpected` in `src/catalog/src/error.rs`. - Changed `InvokeFrontend` error display message and status code in `src/common/frontend/src/error.rs`. - Added `ProcessManagerMissing` error in `src/operator/src/error.rs` and updated its handling in `src/operator/src/statement/kill.rs`. - **Process Management Enhancements**: - Added documentation for `ProcessManager` and `register_query` in `src/catalog/src/process_manager.rs`. - Modified `kill_process` response handling in `src/servers/src/grpc/frontend_grpc_handler.rs`. - **Cancellation Logic Update**: - Improved cancellation logic in `src/common/base/src/cancellation.rs` to use `compare_exchange` for atomic operations. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: ### Add Process Kill Count Metric and Refactor Cancellation Handle - **Metrics Update**: Added a new metric `PROCESS_KILL_COUNT` in `metrics.rs` to track the count of completed kill process requests per catalog. - **Refactor Cancellation Handle**: Renamed `cancellation_handler` to `cancellation_handle` across multiple files for consistency: - `process_manager.rs` - `instance.rs` - `stream_wrapper.rs` - **Process Management**: Updated process management logic in `process_manager.rs` to increment the `PROCESS_KILL_COUNT` metric upon successful process termination. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: Update metric description in `metrics.rs` - Changed the description of `PROCESS_KILL_COUNT` to reflect the count of killed processes instead of running processes in `metrics.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/kill-process: Update `greptime-proto` Dependency and Fix Response Field - **Updated Dependency**: Changed the `greptime-proto` Git revision in `Cargo.lock` and `Cargo.toml` to `f0913f1`. - **Code Fix**: Modified `frontend_grpc_handler.rs` to correct the response field from `found` to `success` in `KillProcessResponse`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Lei, HUANG
|
05b708ed2e |
feat: implement process manager and information_schema.process_list (#5865)
* ### Add Process List Management - **Error Handling Enhancements**: * refactor: Update test IP addresses to include ports in ProcessKey * feat/show-process-list: Refactor Process Management in Meta Module - Introduced `ProcessManager` for handling process registration and deregistration. - Added methods for managing and querying process states, including `register_query`, `deregister_query`, and `list_all_processes`. - Removed redundant process management code from the query module. - Updated error handling to reflect changes in process management. - Enhanced test coverage for process management functionalities. * chore: rebase main * add information schema process list table * integrate process list table to system catalog * build ProcessManager on frontend and standalone mode * feat/show-process-list: **Add Process Management Enhancements** - **`manager.rs`**: Introduced `process_manager` to `SystemCatalog` and `KvBackendCatalogManager` for improved process handling. - **`information_schema.rs`**: Updated table insertion logic to conditionally include `PROCESS_LIST`. - **`frontend.rs`, `standalone.rs`**: Enhanced `StartCommand` to clone `process_manager` for better resource management. - **`instance.rs`, `builder.rs`**: Integrated `ProcessManager` into `Instance` and `FrontendBuilder` to manage query * feat/show-process-list: ### Add Process Listing and Error Handling Enhancements - **Error Handling**: Introduced a new error variant `ListProcess` in `error.rs` to handle failures when listing running processes. - **Process List Implementation**: Enhanced `InformationSchemaProcessList` in `process_list.rs` to track running queries, including defining column names and implementing the `make_process_list` function to build the process list. - **Frontend Builder**: Added a `#[allow(clippy::too_many_arguments)]` attribute in `builder.rs` to suppress Clippy warnings for the `FrontendBuilder::new` function. These changes improve error handling and process tracking capabilities within the system. * feat/show-process-list: Refactor imports in `process_list.rs` - Updated import paths for `Predicates` and `InformationTable` in `process_list.rs` to align with the new module structure. * feat/show-process-list: Refactor process list generation in `process_list.rs` - Simplified the process list generation by removing intermediate row storage and directly building vectors. - Updated `process_to_row` function to use a mutable vector for current row data, improving memory efficiency. - Removed `rows_to_record_batch` function, integrating its logic directly into the main loop for streamlined processing. * wip: move ProcessManager to catalog crate * feat/show-process-list: - **Refactor Row Construction**: Updated row construction in multiple files to use references for `Value` objects, improving memory efficiency. Affected files include: - `cluster_info.rs` - `columns.rs` - `flows.rs` - `key_column_usage.rs` - `partitions.rs` - `procedure_info.rs` - `process_list.rs` - `region_peers.rs` - `region_statistics.rs` - `schemata.rs` - `table_constraints.rs` - `tables.rs` - `views.rs` - `pg_class.rs` - `pg_database.rs` - `pg_namespace.rs` - **Remove Unused Code**: Deleted unused functions and error variants related to process management in `process_list.rs` and `error.rs`. - **Predicate Evaluation Update**: Modified predicate evaluation functions in `predicate.rs` to work with references, enhancing performance. * feat/show-process-list: ### Implement Process Management Enhancements - **Error Handling Enhancements**: - Added new error variants `BumpSequence`, `StartReportTask`, `ReportProcess`, and `BuildProcessManager` in `error.rs` to improve error handling for process management tasks. - Updated `ErrorExt` implementations to handle new error types. - **Process Manager Improvements**: - Introduced `ProcessManager` enhancements in `process_manager.rs` to manage process states using `ProcessWithState` and `ProcessState` enums. - Implemented periodic task `ReportTask` to report running queries to the KV backend. - Modified `register_query` and `deregister_query` methods to use the new state management system. - **Testing and Validation**: - Updated tests in `process_manager.rs` to validate new process management logic. - Replaced `dump` method with `list_all_processes` for listing processes. - **Integration with Frontend and Standalone**: - Updated `frontend.rs` and `standalone.rs` to handle `ProcessManager` initialization errors using `BuildProcessManager` error variant. - **Schema Adjustments**: - Modified `process_list.rs` in `system_schema/information_schema` to use the updated process listing method. - **Key-Value Conversion**: - Added `TryFrom` implementation for converting `Process` to `KeyValue` in `process_list.rs`. * chore: remove register * fix: sqlness tests * merge main Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision. - **Refactor `ProcessManager`**: Simplified the `ProcessManager` implementation by removing the use of `KvBackendRef` and `SequenceRef`, and replaced them with `AtomicU64` and `RwLock` for managing process IDs and catalogs in `process_manager.rs`. - **Remove Process List Metadata**: Deleted the `process_list.rs` file and removed related metadata key definitions in `key.rs`. - **Update Process List Logic**: Modified the process list logic in `process_list.rs` to use the new `ProcessManager` structure. - **Adjust Frontend and Standalone Start Commands**: Updated `frontend.rs` and `standalone.rs` to use the new `ProcessManager` constructor. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency version in `Cargo.lock` and `Cargo.toml` to a new commit hash. - **Refactor Error Handling**: Removed unused error variants and added a new `ParseProcessId` error in `src/catalog/src/error.rs`. - **Enhance Process Management**: Introduced `DisplayProcessId` struct for better process ID representation and parsing in `src/catalog/src/process_manager.rs`. - **Revise Process List Schema**: Updated the schema and logic for process listing in `src/catalog/src/system_schema/information_schema/process_list.rs` to include new fields like `client` and `frontend`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Enhancements and Refactoring** - **Process Management:** - Refactored `ProcessManager` to list local processes with an optional catalog filter in `process_manager.rs`. - Updated related tests in `process_manager.rs` and `process_list.rs`. - **Client Enhancements:** - Added `frontend_client` method in `client.rs` to support gRPC communication with the frontend. - **Error Handling:** - Extended error handling in `error.rs` to include gRPC and Meta errors. - **Frontend Module:** - Introduced `selector.rs` for frontend client selection and process listing. - Updated `Cargo.toml` to include new dependencies and dev-dependencies. - **gRPC Server:** - Integrated `FrontendServer` in `builder.rs` for enhanced gRPC server capabilities. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Refactor Process Management and Frontend Integration** - **Add `common-frontend` Dependency**: - Updated `Cargo.lock`, `Cargo.toml` files to include `common-frontend` as a dependency. - **Refactor Process Management**: - Moved `ProcessManager` trait and `DisplayProcessId` struct to `common-frontend`. - Updated `process_manager.rs` to use `MetaProcessManager` and `ProcessManagerRef`. - Removed `ParseProcessId` error variant from `error.rs` in `catalog` and `frontend`. - **Frontend gRPC Service**: - Added `frontend_grpc_handler.rs` to handle gRPC requests for frontend processes. - Updated `grpc.rs` and `builder.rs` to integrate `FrontendGrpcHandler`. - **Update Tests**: - Modified tests in `process_manager.rs` to align with new `ProcessManager` implementation. - **Remove Unused Code**: - Removed `DisplayProcessId` and related parsing logic from `process_manager.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add `MetaClientRef` to `MetaProcessManager` and Update Instantiation - **Files Modified**: - `src/catalog/src/process_manager.rs` - `src/cmd/src/frontend.rs` - `src/cmd/src/standalone.rs` - **Key Changes**: - Added `MetaClientRef` as an optional parameter to the `MetaProcessManager::new` method. - Updated instantiation of `MetaProcessManager` to include `MetaClientRef` where applicable. ### Update `ProcessManagerRef` Usage - **Files Modified**: - `src/catalog/src/kvbackend/manager.rs` - `src/catalog/src/system_schema/information_schema.rs` - `src/catalog/src/system_schema/information_schema/process_list.rs` - `src/frontend/src/instance.rs` - `src/frontend/src/instance/builder.rs` - **Key Changes**: - Ensured consistent usage of `ProcessManagerRef` across various modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ## Refactor Process Management - **Unified Process Manager**: - Replaced `MetaProcessManager` with `ProcessManager` across the codebase. - Updated `ProcessManager` to use `Arc` for shared references and introduced a `Ticket` struct for query registration and deregistration. - Affected files: `manager.rs`, `process_manager.rs`, `frontend.rs`, `standalone.rs`, `frontend_grpc_handler.rs`, `instance.rs`, `builder.rs`, `cluster.rs`, `standalone.rs`. - **Stream Wrapper Implementation**: - Added `StreamWrapper` to handle record batch streams with process management. - Affected file: `stream_wrapper.rs`. - **Test Adjustments**: - Updated tests to align with the new `ProcessManager` implementation. - Affected file: `tests-integration/src/cluster.rs`, `tests-integration/src/standalone.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling and Process Management - **Error Handling Enhancements**: - Added new error variants `ListProcess` and `CreateChannel` in `error.rs` to handle specific gRPC service invocation failures. - Updated error handling in `selector.rs` to use the new error variants for better context and error propagation. - **Process Management Integration**: - Introduced `process_manager` method in `instance.rs` to access the process manager. - Integrated `FrontendGrpcHandler` with process management in `server.rs` to handle gRPC requests related to process management. - **gRPC Server Enhancements**: - Made `frontend_grpc_handler` public in `grpc.rs` to allow external access and integration with other modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` dependency and enhance process management - **Dependency Update**: Updated `greptime-proto` in `Cargo.lock` and `Cargo.toml` to a new revision. - **Process Management**: - Modified `process_manager.rs` to include catalog filtering in `list_process`. - Updated `frontend_grpc_handler.rs` to handle catalog filtering in `list_process` requests. - **System Schema**: Added a TODO comment in `process_list.rs` for future user catalog filtering implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update Workspace Dependencies**: - Modified `Cargo.toml` files in `src/catalog`, `src/common/frontend`, and `src/servers` to adjust workspace dependencies. - **Refactor `ProcessManager` Logic**: - Updated `process_manager.rs` to simplify the condition in the `select` method. - **Remove Unused Error Variants**: - Deleted `BuildProcessManager` error variant from `error.rs` in `src/cmd`. - Removed `InvalidProcessKey` error variant from `error.rs` in `src/common/meta`. - **Add License Header**: - Added Apache License header to `stream_wrapper.rs` in `src/frontend`. - **Update Test Results**: - Adjusted expected results in `information_schema.result` to reflect changes in the schema. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling for Process Listing - **`src/catalog/src/error.rs`**: Introduced a new error variant `ListProcess` to handle failures in listing frontend nodes. - **`src/catalog/src/process_manager.rs`**: Updated `local_processes` and `list_all_processes` methods to return the new error type, adding context for error handling. - **`src/catalog/src/system_schema/information_schema/process_list.rs`**: Modified `make_process_list` to propagate errors using the new error handling mechanism. - **`src/servers/src/grpc/frontend_grpc_handler.rs`**: Enhanced error handling in the `list_process` method to log errors and return appropriate gRPC status codes. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` Dependency and Remove `frontend_client` Method - **Cargo.lock** and **Cargo.toml**: Updated the `greptime-proto` dependency to a new revision (`5f6119ac7952878d39dcde0343c4bf828d18ffc8`). - **src/client/src/client.rs**: Removed the `frontend_client` method from the `Client` implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Query Registration with Pre-Generated ID - **`process_manager.rs`**: Introduced `register_query_with_id` method to allow registering queries with a pre-generated ID. This includes creating a `ProcessInfo` instance and inserting it into the catalog. Added `next_id` method to generate the next process ID. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update Process List Retrieval Method - **File**: `process_list.rs` - Updated the method for retrieving process lists from `local_processes` to `list_all_processes` to support asynchronous operations. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update error handling in `error.rs` - Refined status code handling for `CreateChannel` error by delegating to `source.status_code()`. - Separated `ListProcess` and `CreateChannel` error handling for clarity. Signed-off-by: Lei, HUANG <lhuang@greptime.com> --------- Signed-off-by: Lei, HUANG <lhuang@greptime.com> |
||
|
shuiyisong
|
9d7fea902e |
chore: remove unused dep (#5163)
* chore: remove unused dep * chore: remove more unused dep |
||
discord9
|
1ddf19d886 |
feat: flownode use Inserter to write to database (#4323)
* feat: use `Inserter` as Frontend * fix: enable procedure in flownode * docs: remove `frontend_addr` opts * chore: rm fe addr in test runner * refactor: int test also use inserter invoker * feat: flow shutdown&refactor: remove `Frontendinvoker` * refactor: rename `RemoteFrontendInvoker` to `FrontendInvoker` * refactor: per review * refactor: remove a layer of box * fix: standalone use `node_manager` * fix: remove a `Arc` cycle |
||
|
Ruihang Xia
|
115c74791d |
build(deps): bump snafu to 0.8 (#3911)
* change Cargo.toml Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * global replace Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * handle alias in script engine Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * fix clippy Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Weny Xu
|
08263995f6 |
feat: implement the FrontendInvoker (#3824)
* chore: add `common-frontend` * feat: add `FrontendInvoker` trait * feat: implement the `FrontendInvoker` |