mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2025-12-27 16:32:54 +00:00
05b708ed2e
* ### Add Process List Management - **Error Handling Enhancements**: * refactor: Update test IP addresses to include ports in ProcessKey * feat/show-process-list: Refactor Process Management in Meta Module - Introduced `ProcessManager` for handling process registration and deregistration. - Added methods for managing and querying process states, including `register_query`, `deregister_query`, and `list_all_processes`. - Removed redundant process management code from the query module. - Updated error handling to reflect changes in process management. - Enhanced test coverage for process management functionalities. * chore: rebase main * add information schema process list table * integrate process list table to system catalog * build ProcessManager on frontend and standalone mode * feat/show-process-list: **Add Process Management Enhancements** - **`manager.rs`**: Introduced `process_manager` to `SystemCatalog` and `KvBackendCatalogManager` for improved process handling. - **`information_schema.rs`**: Updated table insertion logic to conditionally include `PROCESS_LIST`. - **`frontend.rs`, `standalone.rs`**: Enhanced `StartCommand` to clone `process_manager` for better resource management. - **`instance.rs`, `builder.rs`**: Integrated `ProcessManager` into `Instance` and `FrontendBuilder` to manage query * feat/show-process-list: ### Add Process Listing and Error Handling Enhancements - **Error Handling**: Introduced a new error variant `ListProcess` in `error.rs` to handle failures when listing running processes. - **Process List Implementation**: Enhanced `InformationSchemaProcessList` in `process_list.rs` to track running queries, including defining column names and implementing the `make_process_list` function to build the process list. - **Frontend Builder**: Added a `#[allow(clippy::too_many_arguments)]` attribute in `builder.rs` to suppress Clippy warnings for the `FrontendBuilder::new` function. These changes improve error handling and process tracking capabilities within the system. * feat/show-process-list: Refactor imports in `process_list.rs` - Updated import paths for `Predicates` and `InformationTable` in `process_list.rs` to align with the new module structure. * feat/show-process-list: Refactor process list generation in `process_list.rs` - Simplified the process list generation by removing intermediate row storage and directly building vectors. - Updated `process_to_row` function to use a mutable vector for current row data, improving memory efficiency. - Removed `rows_to_record_batch` function, integrating its logic directly into the main loop for streamlined processing. * wip: move ProcessManager to catalog crate * feat/show-process-list: - **Refactor Row Construction**: Updated row construction in multiple files to use references for `Value` objects, improving memory efficiency. Affected files include: - `cluster_info.rs` - `columns.rs` - `flows.rs` - `key_column_usage.rs` - `partitions.rs` - `procedure_info.rs` - `process_list.rs` - `region_peers.rs` - `region_statistics.rs` - `schemata.rs` - `table_constraints.rs` - `tables.rs` - `views.rs` - `pg_class.rs` - `pg_database.rs` - `pg_namespace.rs` - **Remove Unused Code**: Deleted unused functions and error variants related to process management in `process_list.rs` and `error.rs`. - **Predicate Evaluation Update**: Modified predicate evaluation functions in `predicate.rs` to work with references, enhancing performance. * feat/show-process-list: ### Implement Process Management Enhancements - **Error Handling Enhancements**: - Added new error variants `BumpSequence`, `StartReportTask`, `ReportProcess`, and `BuildProcessManager` in `error.rs` to improve error handling for process management tasks. - Updated `ErrorExt` implementations to handle new error types. - **Process Manager Improvements**: - Introduced `ProcessManager` enhancements in `process_manager.rs` to manage process states using `ProcessWithState` and `ProcessState` enums. - Implemented periodic task `ReportTask` to report running queries to the KV backend. - Modified `register_query` and `deregister_query` methods to use the new state management system. - **Testing and Validation**: - Updated tests in `process_manager.rs` to validate new process management logic. - Replaced `dump` method with `list_all_processes` for listing processes. - **Integration with Frontend and Standalone**: - Updated `frontend.rs` and `standalone.rs` to handle `ProcessManager` initialization errors using `BuildProcessManager` error variant. - **Schema Adjustments**: - Modified `process_list.rs` in `system_schema/information_schema` to use the updated process listing method. - **Key-Value Conversion**: - Added `TryFrom` implementation for converting `Process` to `KeyValue` in `process_list.rs`. * chore: remove register * fix: sqlness tests * merge main Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision. - **Refactor `ProcessManager`**: Simplified the `ProcessManager` implementation by removing the use of `KvBackendRef` and `SequenceRef`, and replaced them with `AtomicU64` and `RwLock` for managing process IDs and catalogs in `process_manager.rs`. - **Remove Process List Metadata**: Deleted the `process_list.rs` file and removed related metadata key definitions in `key.rs`. - **Update Process List Logic**: Modified the process list logic in `process_list.rs` to use the new `ProcessManager` structure. - **Adjust Frontend and Standalone Start Commands**: Updated `frontend.rs` and `standalone.rs` to use the new `ProcessManager` constructor. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency version in `Cargo.lock` and `Cargo.toml` to a new commit hash. - **Refactor Error Handling**: Removed unused error variants and added a new `ParseProcessId` error in `src/catalog/src/error.rs`. - **Enhance Process Management**: Introduced `DisplayProcessId` struct for better process ID representation and parsing in `src/catalog/src/process_manager.rs`. - **Revise Process List Schema**: Updated the schema and logic for process listing in `src/catalog/src/system_schema/information_schema/process_list.rs` to include new fields like `client` and `frontend`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Enhancements and Refactoring** - **Process Management:** - Refactored `ProcessManager` to list local processes with an optional catalog filter in `process_manager.rs`. - Updated related tests in `process_manager.rs` and `process_list.rs`. - **Client Enhancements:** - Added `frontend_client` method in `client.rs` to support gRPC communication with the frontend. - **Error Handling:** - Extended error handling in `error.rs` to include gRPC and Meta errors. - **Frontend Module:** - Introduced `selector.rs` for frontend client selection and process listing. - Updated `Cargo.toml` to include new dependencies and dev-dependencies. - **gRPC Server:** - Integrated `FrontendServer` in `builder.rs` for enhanced gRPC server capabilities. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Commit Message **Refactor Process Management and Frontend Integration** - **Add `common-frontend` Dependency**: - Updated `Cargo.lock`, `Cargo.toml` files to include `common-frontend` as a dependency. - **Refactor Process Management**: - Moved `ProcessManager` trait and `DisplayProcessId` struct to `common-frontend`. - Updated `process_manager.rs` to use `MetaProcessManager` and `ProcessManagerRef`. - Removed `ParseProcessId` error variant from `error.rs` in `catalog` and `frontend`. - **Frontend gRPC Service**: - Added `frontend_grpc_handler.rs` to handle gRPC requests for frontend processes. - Updated `grpc.rs` and `builder.rs` to integrate `FrontendGrpcHandler`. - **Update Tests**: - Modified tests in `process_manager.rs` to align with new `ProcessManager` implementation. - **Remove Unused Code**: - Removed `DisplayProcessId` and related parsing logic from `process_manager.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add `MetaClientRef` to `MetaProcessManager` and Update Instantiation - **Files Modified**: - `src/catalog/src/process_manager.rs` - `src/cmd/src/frontend.rs` - `src/cmd/src/standalone.rs` - **Key Changes**: - Added `MetaClientRef` as an optional parameter to the `MetaProcessManager::new` method. - Updated instantiation of `MetaProcessManager` to include `MetaClientRef` where applicable. ### Update `ProcessManagerRef` Usage - **Files Modified**: - `src/catalog/src/kvbackend/manager.rs` - `src/catalog/src/system_schema/information_schema.rs` - `src/catalog/src/system_schema/information_schema/process_list.rs` - `src/frontend/src/instance.rs` - `src/frontend/src/instance/builder.rs` - **Key Changes**: - Ensured consistent usage of `ProcessManagerRef` across various modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ## Refactor Process Management - **Unified Process Manager**: - Replaced `MetaProcessManager` with `ProcessManager` across the codebase. - Updated `ProcessManager` to use `Arc` for shared references and introduced a `Ticket` struct for query registration and deregistration. - Affected files: `manager.rs`, `process_manager.rs`, `frontend.rs`, `standalone.rs`, `frontend_grpc_handler.rs`, `instance.rs`, `builder.rs`, `cluster.rs`, `standalone.rs`. - **Stream Wrapper Implementation**: - Added `StreamWrapper` to handle record batch streams with process management. - Affected file: `stream_wrapper.rs`. - **Test Adjustments**: - Updated tests to align with the new `ProcessManager` implementation. - Affected file: `tests-integration/src/cluster.rs`, `tests-integration/src/standalone.rs`. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling and Process Management - **Error Handling Enhancements**: - Added new error variants `ListProcess` and `CreateChannel` in `error.rs` to handle specific gRPC service invocation failures. - Updated error handling in `selector.rs` to use the new error variants for better context and error propagation. - **Process Management Integration**: - Introduced `process_manager` method in `instance.rs` to access the process manager. - Integrated `FrontendGrpcHandler` with process management in `server.rs` to handle gRPC requests related to process management. - **gRPC Server Enhancements**: - Made `frontend_grpc_handler` public in `grpc.rs` to allow external access and integration with other modules. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` dependency and enhance process management - **Dependency Update**: Updated `greptime-proto` in `Cargo.lock` and `Cargo.toml` to a new revision. - **Process Management**: - Modified `process_manager.rs` to include catalog filtering in `list_process`. - Updated `frontend_grpc_handler.rs` to handle catalog filtering in `list_process` requests. - **System Schema**: Added a TODO comment in `process_list.rs` for future user catalog filtering implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: - **Update Workspace Dependencies**: - Modified `Cargo.toml` files in `src/catalog`, `src/common/frontend`, and `src/servers` to adjust workspace dependencies. - **Refactor `ProcessManager` Logic**: - Updated `process_manager.rs` to simplify the condition in the `select` method. - **Remove Unused Error Variants**: - Deleted `BuildProcessManager` error variant from `error.rs` in `src/cmd`. - Removed `InvalidProcessKey` error variant from `error.rs` in `src/common/meta`. - **Add License Header**: - Added Apache License header to `stream_wrapper.rs` in `src/frontend`. - **Update Test Results**: - Adjusted expected results in `information_schema.result` to reflect changes in the schema. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Error Handling for Process Listing - **`src/catalog/src/error.rs`**: Introduced a new error variant `ListProcess` to handle failures in listing frontend nodes. - **`src/catalog/src/process_manager.rs`**: Updated `local_processes` and `list_all_processes` methods to return the new error type, adding context for error handling. - **`src/catalog/src/system_schema/information_schema/process_list.rs`**: Modified `make_process_list` to propagate errors using the new error handling mechanism. - **`src/servers/src/grpc/frontend_grpc_handler.rs`**: Enhanced error handling in the `list_process` method to log errors and return appropriate gRPC status codes. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: Update `greptime-proto` Dependency and Remove `frontend_client` Method - **Cargo.lock** and **Cargo.toml**: Updated the `greptime-proto` dependency to a new revision (`5f6119ac7952878d39dcde0343c4bf828d18ffc8`). - **src/client/src/client.rs**: Removed the `frontend_client` method from the `Client` implementation. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Add Query Registration with Pre-Generated ID - **`process_manager.rs`**: Introduced `register_query_with_id` method to allow registering queries with a pre-generated ID. This includes creating a `ProcessInfo` instance and inserting it into the catalog. Added `next_id` method to generate the next process ID. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update Process List Retrieval Method - **File**: `process_list.rs` - Updated the method for retrieving process lists from `local_processes` to `list_all_processes` to support asynchronous operations. Signed-off-by: Lei, HUANG <lhuang@greptime.com> * feat/show-process-list: ### Update error handling in `error.rs` - Refined status code handling for `CreateChannel` error by delegating to `source.status_code()`. - Separated `ListProcess` and `CreateChannel` error handling for clarity. Signed-off-by: Lei, HUANG <lhuang@greptime.com> --------- Signed-off-by: Lei, HUANG <lhuang@greptime.com>
Sqlness Test
Sqlness manual
Case file
Sqlness has two types of file:
.sql: test input, SQL only.result: expected test output, SQL and its results
.result is the output (execution result) file. If you see .result files is changed,
it means this test gets a different result and indicates it fails. You should
check change logs to solve the problem.
You only need to write test SQL in .sql file, and run the test.
Case organization
The root dir of input cases is tests/cases. It contains several subdirectories stand for different test
modes. E.g., standalone/ contains all the tests to run under greptimedb standalone start mode.
Under the first level of subdirectory (e.g. the cases/standalone), you can organize your cases as you like.
Sqlness walks through every file recursively and runs them.
Kafka WAL
Sqlness supports Kafka WAL. You can either provide a Kafka cluster or let sqlness to start one for you.
To run test with kafka, you need to pass the option -w kafka. If no other options are provided, sqlness will use conf/kafka-cluster.yml to start a Kafka cluster. This requires docker and docker-compose commands in your environment.
Otherwise, you can additionally pass the your existing kafka environment to sqlness with -k option. E.g.:
cargo sqlness -w kafka -k localhost:9092
In this case, sqlness will not start its own kafka cluster and the one you provided instead.
Run the test
Unlike other tests, this harness is in a binary target form. You can run it with:
cargo sqlness
It automatically finishes the following procedures: compile GreptimeDB, start it, grab tests and feed it to
the server, then collect and compare the results. You only need to check if the .result files are changed.
If not, congratulations, the test is passed 🥳!