refactor/building-backend-in-object-store:
### Refactor Object Store Configuration
- **Centralize Object Store Configurations**: Moved object store configurations (`FileConfig`, `S3Config`, `OssConfig`, `AzblobConfig`, `GcsConfig`) to `object-store/src/config.rs`.
- **Error Handling Enhancements**: Introduced `object-store/src/error.rs` for improved error handling related to object store operations.
- **Factory Pattern for Object Store**: Implemented `object-store/src/factory.rs` to create object store instances, consolidating logic from `datanode/src/store.rs`.
- **Remove Redundant Store Implementations**: Deleted individual store files (`azblob.rs`, `fs.rs`, `gcs.rs`, `oss.rs`, `s3.rs`) from `datanode/src/store/`.
- **Update Usage of Object Store Config**: Updated references to `ObjectStoreConfig` in `datanode.rs`, `standalone.rs`, `config.rs`, and `error.rs` to use the new centralized configuration.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* ### Add Process List Management
- **Error Handling Enhancements**:
* refactor: Update test IP addresses to include ports in ProcessKey
* feat/show-process-list:
Refactor Process Management in Meta Module
- Introduced `ProcessManager` for handling process registration and deregistration.
- Added methods for managing and querying process states, including `register_query`, `deregister_query`, and `list_all_processes`.
- Removed redundant process management code from the query module.
- Updated error handling to reflect changes in process management.
- Enhanced test coverage for process management functionalities.
* chore: rebase main
* add information schema process list table
* integrate process list table to system catalog
* build ProcessManager on frontend and standalone mode
* feat/show-process-list:
**Add Process Management Enhancements**
- **`manager.rs`**: Introduced `process_manager` to `SystemCatalog` and `KvBackendCatalogManager` for improved process handling.
- **`information_schema.rs`**: Updated table insertion logic to conditionally include `PROCESS_LIST`.
- **`frontend.rs`, `standalone.rs`**: Enhanced `StartCommand` to clone `process_manager` for better resource management.
- **`instance.rs`, `builder.rs`**: Integrated `ProcessManager` into `Instance` and `FrontendBuilder` to manage query
* feat/show-process-list:
### Add Process Listing and Error Handling Enhancements
- **Error Handling**: Introduced a new error variant `ListProcess` in `error.rs` to handle failures when listing running processes.
- **Process List Implementation**: Enhanced `InformationSchemaProcessList` in `process_list.rs` to track running queries, including defining column names and implementing the `make_process_list` function to build the process list.
- **Frontend Builder**: Added a `#[allow(clippy::too_many_arguments)]` attribute in `builder.rs` to suppress Clippy warnings for the `FrontendBuilder::new` function.
These changes improve error handling and process tracking capabilities within the system.
* feat/show-process-list:
Refactor imports in `process_list.rs`
- Updated import paths for `Predicates` and `InformationTable` in `process_list.rs` to align with the new module structure.
* feat/show-process-list:
Refactor process list generation in `process_list.rs`
- Simplified the process list generation by removing intermediate row storage and directly building vectors.
- Updated `process_to_row` function to use a mutable vector for current row data, improving memory efficiency.
- Removed `rows_to_record_batch` function, integrating its logic directly into the main loop for streamlined processing.
* wip: move ProcessManager to catalog crate
* feat/show-process-list:
- **Refactor Row Construction**: Updated row construction in multiple files to use references for `Value` objects, improving memory efficiency. Affected files include:
- `cluster_info.rs`
- `columns.rs`
- `flows.rs`
- `key_column_usage.rs`
- `partitions.rs`
- `procedure_info.rs`
- `process_list.rs`
- `region_peers.rs`
- `region_statistics.rs`
- `schemata.rs`
- `table_constraints.rs`
- `tables.rs`
- `views.rs`
- `pg_class.rs`
- `pg_database.rs`
- `pg_namespace.rs`
- **Remove Unused Code**: Deleted unused functions and error variants related to process management in `process_list.rs` and `error.rs`.
- **Predicate Evaluation Update**: Modified predicate evaluation functions in `predicate.rs` to work with references, enhancing performance.
* feat/show-process-list:
### Implement Process Management Enhancements
- **Error Handling Enhancements**:
- Added new error variants `BumpSequence`, `StartReportTask`, `ReportProcess`, and `BuildProcessManager` in `error.rs` to improve error handling for process management tasks.
- Updated `ErrorExt` implementations to handle new error types.
- **Process Manager Improvements**:
- Introduced `ProcessManager` enhancements in `process_manager.rs` to manage process states using `ProcessWithState` and `ProcessState` enums.
- Implemented periodic task `ReportTask` to report running queries to the KV backend.
- Modified `register_query` and `deregister_query` methods to use the new state management system.
- **Testing and Validation**:
- Updated tests in `process_manager.rs` to validate new process management logic.
- Replaced `dump` method with `list_all_processes` for listing processes.
- **Integration with Frontend and Standalone**:
- Updated `frontend.rs` and `standalone.rs` to handle `ProcessManager` initialization errors using `BuildProcessManager` error variant.
- **Schema Adjustments**:
- Modified `process_list.rs` in `system_schema/information_schema` to use the updated process listing method.
- **Key-Value Conversion**:
- Added `TryFrom` implementation for converting `Process` to `KeyValue` in `process_list.rs`.
* chore: remove register
* fix: sqlness tests
* merge main
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision.
- **Refactor `ProcessManager`**: Simplified the `ProcessManager` implementation by removing the use of `KvBackendRef` and `SequenceRef`, and replaced them with `AtomicU64` and `RwLock` for managing process IDs and catalogs in `process_manager.rs`.
- **Remove Process List Metadata**: Deleted the `process_list.rs` file and removed related metadata key definitions in `key.rs`.
- **Update Process List Logic**: Modified the process list logic in `process_list.rs` to use the new `ProcessManager` structure.
- **Adjust Frontend and Standalone Start Commands**: Updated `frontend.rs` and `standalone.rs` to use the new `ProcessManager` constructor.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency version in `Cargo.lock` and `Cargo.toml` to a new commit hash.
- **Refactor Error Handling**: Removed unused error variants and added a new `ParseProcessId` error in `src/catalog/src/error.rs`.
- **Enhance Process Management**: Introduced `DisplayProcessId` struct for better process ID representation and parsing in `src/catalog/src/process_manager.rs`.
- **Revise Process List Schema**: Updated the schema and logic for process listing in `src/catalog/src/system_schema/information_schema/process_list.rs` to include new fields like `client` and `frontend`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Commit Message
**Enhancements and Refactoring**
- **Process Management:**
- Refactored `ProcessManager` to list local processes with an optional catalog filter in `process_manager.rs`.
- Updated related tests in `process_manager.rs` and `process_list.rs`.
- **Client Enhancements:**
- Added `frontend_client` method in `client.rs` to support gRPC communication with the frontend.
- **Error Handling:**
- Extended error handling in `error.rs` to include gRPC and Meta errors.
- **Frontend Module:**
- Introduced `selector.rs` for frontend client selection and process listing.
- Updated `Cargo.toml` to include new dependencies and dev-dependencies.
- **gRPC Server:**
- Integrated `FrontendServer` in `builder.rs` for enhanced gRPC server capabilities.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Commit Message
**Refactor Process Management and Frontend Integration**
- **Add `common-frontend` Dependency**:
- Updated `Cargo.lock`, `Cargo.toml` files to include `common-frontend` as a dependency.
- **Refactor Process Management**:
- Moved `ProcessManager` trait and `DisplayProcessId` struct to `common-frontend`.
- Updated `process_manager.rs` to use `MetaProcessManager` and `ProcessManagerRef`.
- Removed `ParseProcessId` error variant from `error.rs` in `catalog` and `frontend`.
- **Frontend gRPC Service**:
- Added `frontend_grpc_handler.rs` to handle gRPC requests for frontend processes.
- Updated `grpc.rs` and `builder.rs` to integrate `FrontendGrpcHandler`.
- **Update Tests**:
- Modified tests in `process_manager.rs` to align with new `ProcessManager` implementation.
- **Remove Unused Code**:
- Removed `DisplayProcessId` and related parsing logic from `process_manager.rs`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add `MetaClientRef` to `MetaProcessManager` and Update Instantiation
- **Files Modified**:
- `src/catalog/src/process_manager.rs`
- `src/cmd/src/frontend.rs`
- `src/cmd/src/standalone.rs`
- **Key Changes**:
- Added `MetaClientRef` as an optional parameter to the `MetaProcessManager::new` method.
- Updated instantiation of `MetaProcessManager` to include `MetaClientRef` where applicable.
### Update `ProcessManagerRef` Usage
- **Files Modified**:
- `src/catalog/src/kvbackend/manager.rs`
- `src/catalog/src/system_schema/information_schema.rs`
- `src/catalog/src/system_schema/information_schema/process_list.rs`
- `src/frontend/src/instance.rs`
- `src/frontend/src/instance/builder.rs`
- **Key Changes**:
- Ensured consistent usage of `ProcessManagerRef` across various modules.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
## Refactor Process Management
- **Unified Process Manager**:
- Replaced `MetaProcessManager` with `ProcessManager` across the codebase.
- Updated `ProcessManager` to use `Arc` for shared references and introduced a `Ticket` struct for query registration and deregistration.
- Affected files: `manager.rs`, `process_manager.rs`, `frontend.rs`, `standalone.rs`, `frontend_grpc_handler.rs`, `instance.rs`, `builder.rs`, `cluster.rs`, `standalone.rs`.
- **Stream Wrapper Implementation**:
- Added `StreamWrapper` to handle record batch streams with process management.
- Affected file: `stream_wrapper.rs`.
- **Test Adjustments**:
- Updated tests to align with the new `ProcessManager` implementation.
- Affected file: `tests-integration/src/cluster.rs`, `tests-integration/src/standalone.rs`.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Error Handling and Process Management
- **Error Handling Enhancements**:
- Added new error variants `ListProcess` and `CreateChannel` in `error.rs` to handle specific gRPC service invocation failures.
- Updated error handling in `selector.rs` to use the new error variants for better context and error propagation.
- **Process Management Integration**:
- Introduced `process_manager` method in `instance.rs` to access the process manager.
- Integrated `FrontendGrpcHandler` with process management in `server.rs` to handle gRPC requests related to process management.
- **gRPC Server Enhancements**:
- Made `frontend_grpc_handler` public in `grpc.rs` to allow external access and integration with other modules.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
Update `greptime-proto` dependency and enhance process management
- **Dependency Update**: Updated `greptime-proto` in `Cargo.lock` and `Cargo.toml` to a new revision.
- **Process Management**:
- Modified `process_manager.rs` to include catalog filtering in `list_process`.
- Updated `frontend_grpc_handler.rs` to handle catalog filtering in `list_process` requests.
- **System Schema**: Added a TODO comment in `process_list.rs` for future user catalog filtering implementation.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
- **Update Workspace Dependencies**:
- Modified `Cargo.toml` files in `src/catalog`, `src/common/frontend`, and `src/servers` to adjust workspace dependencies.
- **Refactor `ProcessManager` Logic**:
- Updated `process_manager.rs` to simplify the condition in the `select` method.
- **Remove Unused Error Variants**:
- Deleted `BuildProcessManager` error variant from `error.rs` in `src/cmd`.
- Removed `InvalidProcessKey` error variant from `error.rs` in `src/common/meta`.
- **Add License Header**:
- Added Apache License header to `stream_wrapper.rs` in `src/frontend`.
- **Update Test Results**:
- Adjusted expected results in `information_schema.result` to reflect changes in the schema.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Error Handling for Process Listing
- **`src/catalog/src/error.rs`**: Introduced a new error variant `ListProcess` to handle failures in listing frontend nodes.
- **`src/catalog/src/process_manager.rs`**: Updated `local_processes` and `list_all_processes` methods to return the new error type, adding context for error handling.
- **`src/catalog/src/system_schema/information_schema/process_list.rs`**: Modified `make_process_list` to propagate errors using the new error handling mechanism.
- **`src/servers/src/grpc/frontend_grpc_handler.rs`**: Enhanced error handling in the `list_process` method to log errors and return appropriate gRPC status codes.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
Update `greptime-proto` Dependency and Remove `frontend_client` Method
- **Cargo.lock** and **Cargo.toml**: Updated the `greptime-proto` dependency to a new revision (`5f6119ac7952878d39dcde0343c4bf828d18ffc8`).
- **src/client/src/client.rs**: Removed the `frontend_client` method from the `Client` implementation.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Add Query Registration with Pre-Generated ID
- **`process_manager.rs`**: Introduced `register_query_with_id` method to allow registering queries with a pre-generated ID. This includes creating a `ProcessInfo` instance and inserting it into the catalog. Added `next_id` method to generate the next process ID.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Update Process List Retrieval Method
- **File**: `process_list.rs`
- Updated the method for retrieving process lists from `local_processes` to `list_all_processes` to support asynchronous operations.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat/show-process-list:
### Update error handling in `error.rs`
- Refined status code handling for `CreateChannel` error by delegating to `source.status_code()`.
- Separated `ListProcess` and `CreateChannel` error handling for clarity.
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
---------
Signed-off-by: Lei, HUANG <lhuang@greptime.com>
* feat: implement Arrow Flight "DoPut" in Frontend
* support auth for "do_put"
* set request_id in DoPut requests and responses
* set "db" in request header
* refactor: remove mode option in configuration files
* chore: remove mode in configuration file
* remvoe mode field in FlownodeOptions
* add comment for test
* update config.md
* remove mode field in standalone options
* fix: ci
* chore: resolve conflicts
* chore: merge main
* test: add compatibility test for DatanodeLeaseKey with missing cluster_id
* test: add compatibility test for DatanodeLeaseKey without cluster_id
* refactor/remove-cluster-id:
- **Update `greptime-proto` Dependency**: Updated the `greptime-proto` dependency in `Cargo.lock` and `Cargo.toml` to a new revision.
- **Remove `cluster_id` Usage**: Removed the `cluster_id` field and its related logic from various files, including `cluster.rs`, `datanode.rs`, `rpc.rs`,
`adapter.rs`, `client.rs`, `ask_leader.rs`, `heartbeat.rs`, `procedure.rs`, `store.rs`, `handler.rs`, `response_header_handler.rs`, `key.rs`, `datanode.rs`,
`lease.rs`, `metrics.rs`, `cluster.rs`, `heartbeat.rs`, `procedure.rs`, and `store.rs`.
- **Refactor Tests**: Updated tests in `client.rs`, `response_header_handler.rs`, `store.rs`, and `service` modules to reflect the removal of `cluster_id`.
* fix: clippy
* refactor/remove-cluster-id:
**Refactor and Cleanup in Meta Server**
- **`response_header_handler.rs`**: Removed unused import of `HeartbeatResponse` and cleaned up the test function by eliminating the creation of an unused `HeartbeatResponse` object.
- **`node_lease.rs`**: Simplified parameter handling in `HttpHandler` implementation by using an underscore for unused parameters.
* refactor/remove-cluster-id:
### Remove `TableMetadataAllocatorContext` and Refactor Code
- **Removed `TableMetadataAllocatorContext`**: Eliminated the `TableMetadataAllocatorContext` struct and its usage across multiple files, including `ddl.rs`, `create_table.rs`, `create_view.rs`, `table_meta.rs`, `test_util.rs`, `create_logical_tables.rs`,
`drop_table.rs`, and `table_meta_alloc.rs`.
- **Refactored Function Signatures**: Updated function signatures to remove the `TableMetadataAllocatorContext` parameter in methods like `create`, `create_view`, and `alloc` in `table_meta.rs` and `table_meta_alloc.rs`.
- **Updated Imports**: Adjusted import statements to reflect the removal of `TableMetadataAllocatorContext` in affected files.
These changes simplify the codebase by removing an unnecessary context struct and updating related function calls.
* refactor/remove-cluster-id:
### Update `datanode.rs` to Modify Key Prefix
- **File Modified**: `src/common/meta/src/datanode.rs`
- **Key Changes**:
- Updated `DatanodeStatKey::prefix_key` and `From<DatanodeStatKey>` to remove the cluster ID from the key prefix.
- Adjusted comments to reflect the changes in key prefix handling.
* reformat code
* refactor/remove-cluster-id:
### Commit Summary
- **Refactor `Pusher` Initialization**: Removed the `RequestHeader` parameter from the `Pusher::new` method across multiple files, including `handler.rs`, `test_util.rs`, and `heartbeat.rs`. This change simplifies the `Pusher` initialization process by eliminating th
unnecessary parameter.
- **Update Imports**: Adjusted import statements in `handler.rs` and `test_util.rs` to remove unused `RequestHeader` references, ensuring cleaner and more efficient code.
* chore: update proto
* change dep
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* feat: adapt to arrow's interval array
* chore: fix compile errors in datatypes crate
* chore: fix api crate compiler errors
* chore: fix compiler errors in common-grpc
* chore: fix common-datasource errors
* chore: fix deprecated code in common-datasource
* fix promql and physical plan related
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* wip: upgrading network deps
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* block on updating `sqlparser`
* upgrade sqlparser
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* adapt new df's trait requirements
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* chore: fix compiler errors in mito2
* chore: fix common-function crate errors
* chore: fix catalog errors
* change import path
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* chore: fix some errors in query crate
* chore: fix some errors in query crate
* aggr expr and some other tiny fixes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* chore: fix expr related errors in query crate
* chore: fix query serializer and admin command
* chore: fix grpc services
* feat: axum serve
* chore: fix http server
* remove handle_error handler
* refactor timeout layer
* serve axum
* chore: fix flow aggr functions
* chore: fix flow
* feat: fix errors in meta-srv
* boxed()
* use TokioIo
* feat!: Remove script crate and python feature (#5321)
* feat: exclude script crate
* chore: simplify feature
* feat: remove the script crate
* chore: remove python feature and some comments
* chore: fix warning
* chore: fix servers tests compiler errors
* feat: fix tests-integration errors
* chore: fix unused
* test: fix catalog test
* chore: fix compiler errors for crates using common-meta
testing feature is enabled when check with --workspace
* test: use display for logical plan test
* test: implement rewrite for ScanHintRule
* fix: http server build panic
* test: fix mito test
* fix: sql parser type alias error
* test: fix TestClient not listen
* test: some flow tests
* test(flow): more fix
* fix: test_otlp_logs
* test: fix promql test that using deprecated method fun()
* fix: sql type replace supports Int8 ~ Int64, UInt8 ~ UInt64
* test: fix infer schema test case
* test: fix tests related to plan display
* chore: fix last flow test
* test: fix function format related assertion
* test: use larger port range for tests
* fix: test_otlp_traces
* fix: test_otlp_metrics
* fix range query and dist plan
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: flow handle distinct use deprecated field
* fix: can't pass Join plan expressions to LogicalPlan::with_new_exprs
* test: fix deserialize test
* test: reduce split key case num
* tests: lower case aggr func name
* test: fix some sqlness tests
* tests: more sqlness fix
* tests: fixed sqlness test
* commit non-bug changes
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: make our udf correct
* fix: implement empty methods of ContextProvider for DfContextProviderAdapter
* test: update sqlness test result
* chore: remove unused
* fix: provide alias name for AggregateExprBuilder in range plan
* test: update range query result
* fix: implement missing ContextProvider methods for DfContextProviderAdapter
* test: update timestamps, cte result
* fix: supports empty projection in mito
* test: update comment for cte test
* fix: support projection for numbers
* test: update test cases after projection fix
* fix: fix range select first_value/last_value
* fix: handle CAST and time index conflict
* fix: handle order by correctly in range first_value/last_value
* test: update sqlness result
* test: update view test result
* test: update decimal test
wait for https://github.com/apache/datafusion/pull/14126 to fix this
* feat: remove redundant physical optimization
todo(ruihang): Check if we can remove this.
* test: update sqlness test result
* chore: range select default sort use nulls_first = false
* test: update filter push down test result
* test: comment deciaml test to avoid different panic message
* test: update some distributed test result
* test: update test for distributed count and filter push down
* test: update subqueries test
* fix: SessionState may overwrite our UDFs
* chore: fix compiler errors after merging main
* fix: fix elasticsearch and dashboard router panic
* chore: fix common-functions tests
* chore: update sqlness result
* test: fix id keyword and update sqlness result
* test: fix flow_null test
* fix: enlarge thread size in debug mode to avoid overflow
* chore: fix warnings in common-function
* chore: fix warning in flow
* chore: fix warnings in query crate
* chore: remove unused warnings
* chore: fix deprecated warnings for parquet
* chore: fix deprecated warning in servers crate
* style: fix clippy
* test: enlarge mito cache tttl test ttl time
* chore: fix typo
* style: fmt toml
* refactor: reimplement PartialOrd for RangeSelect
* chore: remove script crate files introduced by merge
* fix: return error if sql option is not kv
* chore: do not use ..default::default()
* chore: per review
* chore: update error message in BuildAdminFunctionArgsSnafu
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
* refactor: typed precision
* update sqlness view case
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* chore: flow per review
* chore: add example in comment
* chore: warn if parquet stats of timestamp is not INT64
* style: add a newline before derive to make the comment more clear
* test: update sqlness result
* fix: flow from substrait
* chore: change update_range_context log to debug level
* chore: move axum-extra axum-macros to workspace
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: luofucong <luofc@foxmail.com>
Co-authored-by: discord9 <discord9@163.com>
Co-authored-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
* fix: use SchemaCache to locate database metadata
* main:
Refactor SchemaMetadataManager to use TableInfoCacheRef
- Replace TableInfoManagerRef with TableInfoCacheRef in SchemaMetadataManager
- Update DatanodeBuilder to pass TableInfoCacheRef to SchemaMetadataManager
- Rename error MissingCacheRegistrySnafu to MissingCacheSnafu in datanode module
- Adjust tests to use new mock_schema_metadata_manager with TableInfoCacheRef
* fix/schema-cache-invalidation: Add cache module and integrate cache registry into datanode
• Implement build_datanode_cache_registry function to create cache registry for datanode
• Integrate cache registry into datanode by modifying DatanodeBuilder and HeartbeatTask
• Refactor InvalidateTableCacheHandler to InvalidateCacheHandler and move to common-meta crate
• Update Cargo.toml to include cache as a dev-dependency for datanode
• Adjust related modules (flownode, frontend, tests-integration, standalone) to use new cache handler and registry
• Remove obsolete handler module from frontend crate
* fix: fuzz imports
* chore: add some doc for cahce builder functions
* refactor: change table info cache to table schema cache
* fix: remove unused variants
* fix fuzz
* chore: apply suggestion
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* chore: apply suggestion
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* fix: compile
---------
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* feat: add cache for schema options
* fix/use-cache-kv-manager: Add cache invalidation handling to Datanode's heartbeat task
• Implement InvalidateSchemaCacheHandler in heartbeat.rs to handle cache invalidation instructions.
• Update HeartbeatTask constructor to accept cached_kv_backend and pass it to InvalidateSchemaCacheHandler.
• Modify DatanodeBuilder to clone cached_kv_backend when creating schema_metadata_manager.
• Refactor MetasrvCacheInvalidator in cache_invalidator.rs to reuse MailboxMessage for broadcasting to different channels.
* fix: only remove schema related cache entries
* chore: add more tests
* fix/use-cache-kv-manager: Moved InvalidateSchemaCacheHandler to a separate module
• Extracted InvalidateSchemaCacheHandler and associated tests into a new file cache_invalidator.rs
• Removed async_trait and CacheInvalidator related code from heartbeat.rs
• Added cache_invalidator module declaration in handler.rs
* fix: unit tests
* fix/use-cache-kv-manager:
Standardize TODO comment format in CachedKvBackend txn method
* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs
* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs
* Update src/datanode/src/heartbeat/handler/cache_invalidator.rs
---------
Co-authored-by: jeremyhi <jiachun_feng@proton.me>
* fix: make Influxdb lines able to be inserted into last created tables
* Update src/servers/src/influxdb.rs
* add an option to control the time index alignment behavior
* fix ci
* refactor: use interceptor to handle timestamp align
* Apply suggestions from code review
Co-authored-by: dennis zhuang <killme2008@gmail.com>
---------
Co-authored-by: tison <wander4096@gmail.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
* feat: enable gzip in grpc server side
* feat: add enable_gzip_compression config
* test: add grpc compression test
* feat: support user configured compression on grpc server
* chore: update doc
* chore: add tests
* fix: make config-docs
* chore: fix cr issue
* chore: add test
* refactor: remove config on server side, auto enable all compression support
* chore: minor update
* chore: remove unused code
* refactor: enable zstd compression internally by default
* chore: minor fix
