* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* feat: use arrow-pg for encode_row
* refactor: remove bytea and datetime module
* feat: port more encodings to arrow-pg
* feat: implement intervalstyle
* chore: format
* chore: remove error that is no longer used
* chore: use released arrow-pg
* Apply suggestions from code review
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
---------
Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com>
* feat: add repartition procedure factory support to DdlManager
- Introduce RepartitionProcedureFactory trait for creating and registering
repartition procedures
- Implement DefaultRepartitionProcedureFactory for metasrv with full support
- Implement StandaloneRepartitionProcedureFactory for standalone (unsupported)
- Add procedure loader registration for RepartitionProcedure and
RepartitionGroupProcedure
- Add helper methods to TableMetadataAllocator for allocator access
- Add error types for repartition procedure operations
- Update DdlManager to accept and use RepartitionProcedureFactoryRef
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: integrate repartition procedure into DdlManager
- Add submit_repartition_task() to handle repartition from alter table
- Route Repartition operations in submit_alter_table_task() to repartition factory
- Refactor: rename submit_procedure() to execute_procedure_and_wait()
- Make all DDL operations wait for completion by default
- Add submit_procedure() for fire-and-forget submissions
- Add CreateRepartitionProcedure error type
- Add placeholder Repartition handling in grpc-expr (unsupported)
- Update greptime-proto dependency
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: implement ALTER TABLE REPARTITION procedure submission
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor(repartition): handle central region in apply staging manifest
- Introduce ApplyStagingManifestInstructions struct to organize instructions
- Add special handling for central region when applying staging manifests
- Transition state from UpdateMetadata to RepartitionEnd after applying staging manifests
- Remove next_state() method in RepartitionStart and inline state transitions
- Improve logging and expression serialization in DDL statement executor
- Move repartition tests from standalone to distributed test suite
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: apply suggestions from CR
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update proto
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* refactor/bulk-insert-service:
refactor: decode FlightData early in put_record_batch pipeline
- Move FlightDecoder usage from Inserter up to PutRecordBatchRequestStream,
passing decoded RecordBatch and schema bytes instead of raw FlightData.
- Eliminate redundant per-request decoding/encoding in Inserter; encode
once and reuse for all region requests.
- Streamline GrpcQueryHandler trait and implementations to accept
PutRecordBatchRequest containing pre-decoded data.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
feat: stream-based bulk insert with per-batch responses
- Introduce handle_put_record_batch_stream() to process Flight DoPut streams
- Resolve table & permissions once, yield (request_id, AffectedRows) per batch
- Replace loop-over-request with async-stream in frontend & server
- Make PutRecordBatchRequestStream public for cross-crate usage
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
fix: propagate request_id with errors in bulk insert stream
Changes the bulk-insert stream item type from
Result<(i64, AffectedRows), E> to (i64, Result<AffectedRows, E>)
so every emitted tuple carries the request_id even on failure,
letting callers correlate errors with the originating request.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
refactor: unify DoPut response stream to return DoPutResponse
Replace the tuple (i64, Result<AffectedRows>) with Result<DoPutResponse>
throughout the gRPC bulk-insert path so the handler, adapter and server
all speak the same type.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
feat: add elapsed_secs to DoPutResponse for bulk-insert timing
- DoPutResponse now carries elapsed_secs field
- Frontend measures and attaches insert duration
- Server observes GRPC_BULK_INSERT_ELAPSED metric from response
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
refactor: unify Bytes import in flight module
- Replace `bytes::Bytes` with `Bytes` alias for consistency
- Remove redundant `ProstBytes` alias
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
fix: terminate gRPC stream on error and optimize FlightData handling
- Stop retrying on stream errors in gRPC handler
- Replace Vec1 indexing with into_iter().next() for FlightData
- Remove redundant clones in bulk_insert and flight modules
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
Improve permission check placement in `grpc.rs`
- Moved the permission check for `BulkInsert` to occur before resolving the table reference in `GrpcQueryHandler` implementation.
- Ensures permission validation is performed earlier in the process, potentially avoiding unnecessary operations if permission is denied.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
**Refactor Bulk Insert Handling in gRPC**
- **`grpc.rs`**:
- Switched from `async_stream::stream` to `async_stream::try_stream` for error handling.
- Removed `body_size` parameter and added `flight_data` to `handle_bulk_insert`.
- Simplified error handling and permission checks in `GrpcQueryHandler`.
- **`bulk_insert.rs`**:
- Added `raw_flight_data` parameter to `handle_bulk_insert`.
- Calculated `body_size` from `raw_flight_data` and removed redundant encoding logic.
- **`flight.rs`**:
- Replaced `body_size` with `flight_data` in `PutRecordBatchRequest`.
- Updated memory usage calculation to include `flight_data` components.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
perf(bulk_insert): encode record batch once per datanode
Move FlightData encoding outside the per-region loop so the same
encoded bytes are reused when mask.select_all(), eliminating redundant
serialisation work.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: show tables for mysql
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partitions table in information_schema and add starrocks external catalog compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: async udf
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: set warnings
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: impl pg_my_temp_schema and make description functions simple
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add test for issue 7313
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partition_expression and partition_description
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: test
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: unit tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: saerch_path only works for pg
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: improve warnings processing
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: warnings while writing affected rows and refactor
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: improve ShobjDescriptionFunction signature
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: array_to_boolean
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat/manual-compaction-parallelism:
### Add Parallelism Support to Compaction Requests
- **`Cargo.lock` & `Cargo.toml`**: Updated `greptime-proto` dependency to a new revision.
- **`flush_compact_table.rs`**: Enhanced `parse_compact_params` to support a new `parallelism` parameter, allowing users to
specify the level of parallelism for table compaction.
- **`handle_compaction.rs`**: Integrated `parallelism` into the compaction scheduling process, defaulting to 1 if not
specified.
- **`request.rs` & `region_request.rs`**: Modified `CompactRequest` to include `parallelism`, with logic to handle unspecifie
values.
- **`requests.rs`**: Updated `CompactTableRequest` structure to include an optional `parallelism` field.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/manual-compaction-parallelism:
### Commit Message
Enhance Compaction Request Handling
- **`flush_compact_table.rs`**:
- Renamed `parse_compact_params` to `parse_compact_request`.
- Introduced `DEFAULT_COMPACTION_PARALLELISM` constant.
- Updated parsing logic to handle keyword arguments for `strict_window` and `regular` compaction types, including `parallelism` and `window`.
- Modified tests to reflect changes in parsing logic and default parallelism handling.
- **`request.rs`**:
- Updated `parallelism` handling in `RegionRequestBody::Compact` to use the new default value.
- **`requests.rs`**:
- Changed `CompactTableRequest` to use a non-optional `parallelism` field with a default value of 1.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/manual-compaction-parallelism:
### Update `flush_compact_table.rs` Parameter Validation
- Modified parameter validation in `flush_compact_table.rs` to restrict the maximum number of parameters from 4 to 3 in the `parse_compact_request` function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/manual-compaction-parallelism:
Update `greptime-proto` dependency
- Updated the `greptime-proto` dependency to a new revision in both `Cargo.lock` and `Cargo.toml`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add updated_on to tablemeta with a default of created_on
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: support the update_on on alter procedure
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: add updated_on into information_schema.tables
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* fix: make sqlness happy
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* test: add test case for tablemeta update
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* fix: fix failing test for ALTER TABLE
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: use created_on as default for updated_on when missing
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
---------
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: struct value
Signed-off-by: Ning Sun <sunning@greptime.com>
* feat: update for proto module
* feat: wip struct type
* feat: implement more vector operations
* feat: make datatype and api
* feat: reoslve some compilation issues
* feat: resolve all compilation issues
* chore: format update
* test: resolve tests
* test: test and refactor value-to-pb
* feat: add more tests and fix for value types
* chore: remove dbg
* feat: test and fix iterator
* fix: resolve struct_type issue
* refactor: use vec for struct items
* chore: update proto to main branch
* refactor: address some of review issues
* refactor: update for further review
* Add validation on new methods
* feat: update struct/list json serialization
* refactor: reimplement get in struct_vector
* refactor: struct vector functions
* refactor: fix lint issue
* refactor: address review comments
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
