* chore: gate soft-drop table behind the enterprise feature
Soft-drop table becomes an enterprise-only feature:
- metasrv rejects gc.experimental_soft_drop.enable=true at startup in
non-enterprise builds, and ddl_soft_drop_enabled is hard-disabled
without the enterprise feature as a second line of defense
- the UNDROP TABLE parser/AST/statement variant, ADMIN purge_table()
registration, and information_schema.recycle_bin registration are
compiled out unless the enterprise feature is enabled
- common-meta procedures, tombstone keys, and DdlTask serde stay
unconditional for persisted-procedure recovery and wire compatibility
- the [gc.experimental_soft_drop] section is removed from the OSS
example config and generated docs (moving to the enterprise repo)
- the soft-drop sqlness cases and their CI job are removed from OSS
(moving to the enterprise repo); affected information_schema .result
files are regenerated
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: limit unused_variables allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch accidental unused variables in register_admin_only.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: include the config key in the soft-drop enterprise gate error
Addresses review comment: name gc.experimental_soft_drop.enable in the
startup validation error so users can locate the setting quickly when
it is set via env vars or layered config.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: limit unused_mut allow to non-enterprise builds
Addresses review comment: apply the allow via cfg_attr so enterprise
builds still catch unused mut in the table_ddl_event test setup.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: reject soft-drop DDL submissions in non-enterprise builds
Addresses review comment: clients could bypass the SQL-level gates by
submitting DdlTask::UndropTable or DdlTask::PurgeDroppedTable directly
to the procedure service. Reject fresh submissions at the DdlManager
boundary in non-enterprise builds while keeping the procedure loaders
registered for crash recovery and wire compatibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: stop --enable-gc from enabling soft drop in the sqlness template
Addresses review comment: the metasrv test template rendered
[gc.experimental_soft_drop] enable = true under the generic --enable-gc
flag, which non-enterprise metasrv now rejects at startup, making the
documented --enable-gc mode unusable in OSS. Keep the flag scoped to
plain GC; enterprise soft-drop coverage moves to the enterprise repo.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate fresh soft-drop procedures
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test: gate soft-drop fallback coverage
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: gate soft-drop procedure implementation
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate drop table soft-drop behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: gate expired soft-drop gc behavior
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: test enterprise table ddl lifecycle
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark purge_table as enterprise licensed
The purge_table module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark recycle_bin as enterprise licensed
The recycle_bin module is compiled only with the enterprise feature, so
apply the Enterprise License header and register it with both license
header configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: mark soft-drop procedure sources as enterprise licensed
The purge and undrop procedure implementations plus the recycle-bin test
module compile only with the enterprise feature. Apply the Enterprise
License header and register them with both license configurations.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: support full WAL retirement
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: complete close request migration
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: guard Kafka provider setup behind index collector check
Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.
Affected file:
- `src/log-store/src/kafka/log_store.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: avoid to_vec
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: clean up soft-dropped regions offline
Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them.
Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): reject file-engine soft drop
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: preserve soft-drop cleanup split state
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: remove obsolete CleanUp match arm from RegionRequest
The `CleanUp` variant in the `region_request::Body` match is now handled
exclusively by `RegionServer` via a separate path. This arm would have
returned an unexpected error, so removing it eliminates dead code.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): clean every soft-dropped region replica
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(meta): order soft-drop replica cleanup
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): order soft-drop replica cleanup"
This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* Revert "fix(meta): clean every soft-dropped region replica"
This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(catalog): expose soft drops in recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(sql): add UNDROP TABLE
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sql): cover UNDROP TABLE execution
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(sql): keep successful UNDROP result
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject stale undrop by name-based lookup after tombstone consumed
When a table is dropped, recreated under the same name (consuming the name
tombstone), and then the recreated table is dropped, an undrop procedure
that was built before the first drop and holds a stale original table name
should fail with TableNotFound instead of silently matching a different
table.
Changed `UndropTableProcedure::on_prepare` to perform a name-based lookup
when `table_name` is available and filter by table ID, ensuring that a
dropped table can only be recovered when its name tombstone still maps to
the expected ID.
- `src/common/meta/src/ddl/undrop_table.rs`: name-first lookup in on_prepare
- `src/common/meta/src/ddl/tests/drop_table.rs`: test for the stale-id
rejection case
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(log-store): keep Kafka obsolete_all as no-op
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): hide purging tables from recycle bin
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(catalog): scope recycle bin scans by catalog
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(sqlness): update recycle bin expectations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: avoid redundant recycle bin allocations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: table semantic layer information_schema view (Phase 3)
Add `information_schema.table_semantics`, a queryable view over the table
semantic layer. One row per table that carries at least one
`greptime.semantic.*` option: the signal-agnostic keys
(signal_type/source/pipeline/metadata_quality) are promoted to columns and
the remaining signal-specific keys are folded into a `semantic_options`
JSON string. Tables with no semantic key are excluded.
Stacked on Phase 2.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: address PR review on table_semantics
- fold JSON serialization failure into None instead of unwrap/panic
- drop per-row Vec allocation in predicate eval; use a fixed array
- align RFC view name with the shipped `table_semantics`
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: update pgwire
* feat: add special parser for copy to stdout
* feat: implement copy to stdout
* fix: improve code
* fix: expect optional with
* fix: lint
* feat: correct encoder using and refactor
* chore: fmt
* refactor: update api
* chore: use released dependencies
* fix: update datafusion-pg-catalog to support schema query
* fix: support for double quoted identifier
* feat: update datafusion-postgres to support schema.table
* refactor: use pgsqlparser container
* refactor: remove unquote which is no longer needed
* fix: correctly handle invalid query
* fix: correct handle null in nano timestamp
* test: add a new test for additional close )
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: show tables for mysql
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partitions table in information_schema and add starrocks external catalog compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: async udf
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: set warnings
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: impl pg_my_temp_schema and make description functions simple
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add test for issue 7313
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partition_expression and partition_description
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: test
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: unit tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: saerch_path only works for pg
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: improve warnings processing
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: warnings while writing affected rows and refactor
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: improve ShobjDescriptionFunction signature
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: array_to_boolean
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: add updated_on to tablemeta with a default of created_on
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: support the update_on on alter procedure
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: add updated_on into information_schema.tables
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* fix: make sqlness happy
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* test: add test case for tablemeta update
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* fix: fix failing test for ALTER TABLE
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: use created_on as default for updated_on when missing
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
---------
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* refactor: add datafusion-postgres dependency
* refactor: move and include pg_catalog udfs
* chore: update upstream
* feat: register table function pg_get_keywords
* feat: bridge CatalogInfo for our CatalogManager
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: convert pg_catalog table to our system table
* feat: bridge system catalog with datafusion-postgres
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: add more udfs
* feat: add compatibility rewriter to postgres handler
* fix: various fix
* fmt: fix
* fix: use functions from pg_catalog library
* fmt
* fix: sqlness runner
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: adopt arrow 56.0 to 56.1 memory size change
* fix: add additional udfs
* chore: format
* refactor: return None when creating system table failed
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: provide safety comments about expect usage
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat/add-sst-file-num-in-region-stat:
### Add SST File Count to Region Statistics
- **Enhancements**:
- Added `sst_num` to track the number of SST files in region statistics across multiple modules.
- Updated `RegionStat` and `RegionStatistic` structs in `datanode.rs` and `region_engine.rs` to include `sst_num`.
- Modified `MitoRegion` and `SstVersion` in `region.rs` and `version.rs` to compute and return the number of SST files.
- Adjusted test cases in `collect_leader_region_handler.rs`, `failure_handler.rs`, `region_lease_handler.rs`, and `weight_compute.rs` to initialize `sst_num`.
- Updated `get_region_statistic` in `utils.rs` to sum `sst_num` from metadata and data statistics.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/add-sst-file-num-in-region-stat:
Add `sst_num` to `region_statistics`
- Updated `region_statistics.rs` to include a new constant `SST_NUM` and added it to the schema and builder structures.
- Modified `information_schema.result` to reflect the addition of `sst_num` in the `region_statistics` table.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* ### Add Process List Management
- **Error Handling Enhancements**:
* refactor: Update test IP addresses to include ports in ProcessKey
* feat/show-process-list:
Refactor Process Management in Meta Module
- Introduced `ProcessManager` for handling process registration and deregistration.
- Added methods for managing and querying process states, including `register_query`, `deregister_query`, and `list_all_processes`.
- Removed redundant process management code from the query module.
- Updated error handling to reflect changes in process management.
- Enhanced test coverage for process management functionalities.
* chore: rebase main
* add information schema process list table
* integrate process list table to system catalog
* build ProcessManager on frontend and standalone mode
* feat/show-process-list:
**Add Process Management Enhancements**
- **`manager.rs`**: Introduced `process_manager` to `SystemCatalog` and `KvBackendCatalogManager` for improved process handling.
- **`information_schema.rs`**: Updated table insertion logic to conditionally include `PROCESS_LIST`.
- **`frontend.rs`, `standalone.rs`**: Enhanced `StartCommand` to clone `process_manager` for better resource management.
- **`instance.rs`, `builder.rs`**: Integrated `ProcessManager` into `Instance` and `FrontendBuilder` to manage query
* feat/show-process-list:
### Add Process Listing and Error Handling Enhancements
- **Error Handling**: Introduced a new error variant `ListProcess` in `error.rs` to handle failures when listing running processes.
- **Process List Implementation**: Enhanced `InformationSchemaProcessList` in `process_list.rs` to track running queries, including defining column names and implementing the `make_process_list` function to build the process list.
- **Frontend Builder**: Added a `#[allow(clippy::too_many_arguments)]` attribute in `builder.rs` to suppress Clippy warnings for the `FrontendBuilder::new` function.
These changes improve error handling and process tracking capabilities within the system.
* feat/show-process-list:
Refactor imports in `process_list.rs`
- Updated import paths for `Predicates` and `InformationTable` in `process_list.rs` to align with the new module structure.
* feat/show-process-list:
Refactor process list generation in `process_list.rs`
- Simplified the process list generation by removing intermediate row storage and directly building vectors.
- Updated `process_to_row` function to use a mutable vector for current row data, improving memory efficiency.
- Removed `rows_to_record_batch` function, integrating its logic directly into the main loop for streamlined processing.
* wip: move ProcessManager to catalog crate
* feat/show-process-list:
- **Refactor Row Construction**: Updated row construction in multiple files to use references for `Value` objects, improving memory efficiency. Affected files include:
- `cluster_info.rs`
- `columns.rs`
- `flows.rs`
- `key_column_usage.rs`
- `partitions.rs`
- `procedure_info.rs`
- `process_list.rs`
- `region_peers.rs`
- `region_statistics.rs`
- `schemata.rs`
- `table_constraints.rs`
- `tables.rs`
- `views.rs`
- `pg_class.rs`
- `pg_database.rs`
- `pg_namespace.rs`
- **Remove Unused Code**: Deleted unused functions and error variants related to process management in `process_list.rs` and `error.rs`.
- **Predicate Evaluation Update**: Modified predicate evaluation functions in `predicate.rs` to work with references, enhancing performance.
* feat/show-process-list:
### Implement Process Management Enhancements
- **Error Handling Enhancements**:
- Added new error variants `BumpSequence`, `StartReportTask`, `ReportProcess`, and `BuildProcessManager` in `error.rs` to improve error handling for process management tasks.
- Updated `ErrorExt` implementations to handle new error types.
- **Process Manager Improvements**:
- Introduced `ProcessManager` enhancements in `process_manager.rs` to manage process states using `ProcessWithState` and `ProcessState` enums.
- Implemented periodic task `ReportTask` to report running queries to the KV backend.
- Modified `register_query` and `deregister_query` methods to use the new state management system.
- **Testing and Validation**:
- Updated tests in `process_manager.rs` to validate new process management logic.
- Replaced `dump` method with `list_all_processes` for listing processes.
- **Integration with Frontend and Standalone**:
- Updated `frontend.rs` and `standalone.rs` to handle `ProcessManager` initialization errors using `BuildProcessManager` error variant.
- **Schema Adjustments**:
- Modified `process_list.rs` in `system_schema/information_schema` to use the updated process listing method.
- **Key-Value Conversion**:
- Added `TryFrom` implementation for converting `Process` to `KeyValue` in `process_list.rs`.
* chore: remove register
* fix: sqlness tests
* merge main
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision.
- **Refactor `ProcessManager`**: Simplified the `ProcessManager` implementation by removing the use of `KvBackendRef` and `SequenceRef`, and replaced them with `AtomicU64` and `RwLock` for managing process IDs and catalogs in `process_manager.rs`.
- **Remove Process List Metadata**: Deleted the `process_list.rs` file and removed related metadata key definitions in `key.rs`.
- **Update Process List Logic**: Modified the process list logic in `process_list.rs` to use the new `ProcessManager` structure.
- **Adjust Frontend and Standalone Start Commands**: Updated `frontend.rs` and `standalone.rs` to use the new `ProcessManager` constructor.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency version in `Cargo.lock` and `Cargo.toml` to a new commit hash.
- **Refactor Error Handling**: Removed unused error variants and added a new `ParseProcessId` error in `src/catalog/src/error.rs`.
- **Enhance Process Management**: Introduced `DisplayProcessId` struct for better process ID representation and parsing in `src/catalog/src/process_manager.rs`.
- **Revise Process List Schema**: Updated the schema and logic for process listing in `src/catalog/src/system_schema/information_schema/process_list.rs` to include new fields like `client` and `frontend`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Commit Message
**Enhancements and Refactoring**
- **Process Management:**
- Refactored `ProcessManager` to list local processes with an optional catalog filter in `process_manager.rs`.
- Updated related tests in `process_manager.rs` and `process_list.rs`.
- **Client Enhancements:**
- Added `frontend_client` method in `client.rs` to support gRPC communication with the frontend.
- **Error Handling:**
- Extended error handling in `error.rs` to include gRPC and Meta errors.
- **Frontend Module:**
- Introduced `selector.rs` for frontend client selection and process listing.
- Updated `Cargo.toml` to include new dependencies and dev-dependencies.
- **gRPC Server:**
- Integrated `FrontendServer` in `builder.rs` for enhanced gRPC server capabilities.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Commit Message
**Refactor Process Management and Frontend Integration**
- **Add `common-frontend` Dependency**:
- Updated `Cargo.lock`, `Cargo.toml` files to include `common-frontend` as a dependency.
- **Refactor Process Management**:
- Moved `ProcessManager` trait and `DisplayProcessId` struct to `common-frontend`.
- Updated `process_manager.rs` to use `MetaProcessManager` and `ProcessManagerRef`.
- Removed `ParseProcessId` error variant from `error.rs` in `catalog` and `frontend`.
- **Frontend gRPC Service**:
- Added `frontend_grpc_handler.rs` to handle gRPC requests for frontend processes.
- Updated `grpc.rs` and `builder.rs` to integrate `FrontendGrpcHandler`.
- **Update Tests**:
- Modified tests in `process_manager.rs` to align with new `ProcessManager` implementation.
- **Remove Unused Code**:
- Removed `DisplayProcessId` and related parsing logic from `process_manager.rs`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add `MetaClientRef` to `MetaProcessManager` and Update Instantiation
- **Files Modified**:
- `src/catalog/src/process_manager.rs`
- `src/cmd/src/frontend.rs`
- `src/cmd/src/standalone.rs`
- **Key Changes**:
- Added `MetaClientRef` as an optional parameter to the `MetaProcessManager::new` method.
- Updated instantiation of `MetaProcessManager` to include `MetaClientRef` where applicable.
### Update `ProcessManagerRef` Usage
- **Files Modified**:
- `src/catalog/src/kvbackend/manager.rs`
- `src/catalog/src/system_schema/information_schema.rs`
- `src/catalog/src/system_schema/information_schema/process_list.rs`
- `src/frontend/src/instance.rs`
- `src/frontend/src/instance/builder.rs`
- **Key Changes**:
- Ensured consistent usage of `ProcessManagerRef` across various modules.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
## Refactor Process Management
- **Unified Process Manager**:
- Replaced `MetaProcessManager` with `ProcessManager` across the codebase.
- Updated `ProcessManager` to use `Arc` for shared references and introduced a `Ticket` struct for query registration and deregistration.
- Affected files: `manager.rs`, `process_manager.rs`, `frontend.rs`, `standalone.rs`, `frontend_grpc_handler.rs`, `instance.rs`, `builder.rs`, `cluster.rs`, `standalone.rs`.
- **Stream Wrapper Implementation**:
- Added `StreamWrapper` to handle record batch streams with process management.
- Affected file: `stream_wrapper.rs`.
- **Test Adjustments**:
- Updated tests to align with the new `ProcessManager` implementation.
- Affected file: `tests-integration/src/cluster.rs`, `tests-integration/src/standalone.rs`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Error Handling and Process Management
- **Error Handling Enhancements**:
- Added new error variants `ListProcess` and `CreateChannel` in `error.rs` to handle specific gRPC service invocation failures.
- Updated error handling in `selector.rs` to use the new error variants for better context and error propagation.
- **Process Management Integration**:
- Introduced `process_manager` method in `instance.rs` to access the process manager.
- Integrated `FrontendGrpcHandler` with process management in `server.rs` to handle gRPC requests related to process management.
- **gRPC Server Enhancements**:
- Made `frontend_grpc_handler` public in `grpc.rs` to allow external access and integration with other modules.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
Update `greptime-proto` dependency and enhance process management
- **Dependency Update**: Updated `greptime-proto` in `Cargo.lock` and `Cargo.toml` to a new revision.
- **Process Management**:
- Modified `process_manager.rs` to include catalog filtering in `list_process`.
- Updated `frontend_grpc_handler.rs` to handle catalog filtering in `list_process` requests.
- **System Schema**: Added a TODO comment in `process_list.rs` for future user catalog filtering implementation.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update Workspace Dependencies**:
- Modified `Cargo.toml` files in `src/catalog`, `src/common/frontend`, and `src/servers` to adjust workspace dependencies.
- **Refactor `ProcessManager` Logic**:
- Updated `process_manager.rs` to simplify the condition in the `select` method.
- **Remove Unused Error Variants**:
- Deleted `BuildProcessManager` error variant from `error.rs` in `src/cmd`.
- Removed `InvalidProcessKey` error variant from `error.rs` in `src/common/meta`.
- **Add License Header**:
- Added Apache License header to `stream_wrapper.rs` in `src/frontend`.
- **Update Test Results**:
- Adjusted expected results in `information_schema.result` to reflect changes in the schema.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Error Handling for Process Listing
- **`src/catalog/src/error.rs`**: Introduced a new error variant `ListProcess` to handle failures in listing frontend nodes.
- **`src/catalog/src/process_manager.rs`**: Updated `local_processes` and `list_all_processes` methods to return the new error type, adding context for error handling.
- **`src/catalog/src/system_schema/information_schema/process_list.rs`**: Modified `make_process_list` to propagate errors using the new error handling mechanism.
- **`src/servers/src/grpc/frontend_grpc_handler.rs`**: Enhanced error handling in the `list_process` method to log errors and return appropriate gRPC status codes.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
Update `greptime-proto` Dependency and Remove `frontend_client` Method
- **Cargo.lock** and **Cargo.toml**: Updated the `greptime-proto` dependency to a new revision (`5f6119ac7952878d39dcde0343c4bf828d18ffc8`).
- **src/client/src/client.rs**: Removed the `frontend_client` method from the `Client` implementation.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Query Registration with Pre-Generated ID
- **`process_manager.rs`**: Introduced `register_query_with_id` method to allow registering queries with a pre-generated ID. This includes creating a `ProcessInfo` instance and inserting it into the catalog. Added `next_id` method to generate the next process ID.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Update Process List Retrieval Method
- **File**: `process_list.rs`
- Updated the method for retrieving process lists from `local_processes` to `list_all_processes` to support asynchronous operations.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Update error handling in `error.rs`
- Refined status code handling for `CreateChannel` error by delegating to `source.status_code()`.
- Separated `ListProcess` and `CreateChannel` error handling for clarity.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
---------
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* fix unit tests
* fix: sqlness
* fix/default-time-window:
## Add Helper Functions and Enhance Compaction Tests
- **Refactor Compaction Logic**: Introduced helper functions `flush` and `compact` in `compaction_test.rs` to streamline compaction operations.
- **Enhance Compaction Tests**: Added a new test `test_infer_compaction_time_window` in `compaction_test.rs` to verify compaction time window inference.
- **Testing Improvements**: Added `#[cfg(test)]` attribute to `new_multi_partitions` in `time_partition.rs` to ensure it's only included in test builds.
* fix/default-time-window:
- **Refactor `TimePartition` Struct**: Removed unnecessary comments regarding `time_range` in `time_partition.rs`.
- **Enhance `TimePartitions` Functionality**: Added a method `part_duration_or_default` to provide a default partition duration in `time_partition.rs`.
- **Update SQL Test Cases**: Modified SQL operations and expected results in `scan_big_varchar.result` and `scan_big_varchar.sql` to reflect changes in data manipulation logic.
* fix/default-time-window:
### Update Time Partition Default Duration
- **Refactor Default Duration**: Introduced `INITIAL_TIME_WINDOW` constant to define the default time window duration as `Duration::from_days(1)`. This change replaces multiple instances of the hardcoded default duration across the `time_partition.rs` file.
- **Files Affected**: `time_partition.rs`
* fix/default-time-window:
## Update Partition Duration Handling
- **`time_partition.rs`**: Refactored `part_duration` to be non-optional, removing `Option` wrapper. Updated logic to use `unwrap_or` with `INITIAL_TIME_WINDOW` where necessary. Adjusted related methods and tests to accommodate this change.
- **`version.rs` (memtable and region)**: Updated handling of `part_duration` to align with changes in `time_partition.rs`, ensuring consistent use of non-optional `Duration`.
* fix/default-time-window:
### Improve Error Context in `time_partition.rs`
- Enhanced error context message in `time_partition.rs` to provide clearer information on partition time range issues, including bucket size details.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
---------
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* fix: fix region follower procedure
* feat: add table related info to region peers table and follower regions
* feat: impl show region
* chore: apply suggestions from CR
* feat: enhancement information_schema.flows
* feat: enhancement information_schema.flows
* u
* u
* u
* u
* u
* u
* u
* u
* u
* update
* update
* update
* delete unused code
* u
* u
* Update src/flow/src/adapter/worker.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* Update src/common/meta/src/key/flow/flow_state.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* Update src/common/meta/src/key/flow/flow_info.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* Update src/common/meta/src/key/flow/flow_state.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* Update src/common/meta/src/key/flow/flow_info.rs
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* u
* u
* u
* u
* u
* u
* chore: fix sqlness
* chore: update proto
* fix: remove date time
* fix: update result of information_schema test
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: discord9 <discord9@163.com>
discord9
Lei, HUANG
Weny Xu
shuiyisong
dennis zhuang
LFC
QuakeWanganddennis zhuang
Yingwen
Ning Sun
discord9
WaterWhisperer
fys
Sicong Hu
zyy17andYingwen
Zhenchi
Alan Tang
Ruihang Xia
Yan Tingwangandevenyag
Lin Yihai
