* refactor/bulk-insert-service:
refactor: decode FlightData early in put_record_batch pipeline
- Move FlightDecoder usage from Inserter up to PutRecordBatchRequestStream,
passing decoded RecordBatch and schema bytes instead of raw FlightData.
- Eliminate redundant per-request decoding/encoding in Inserter; encode
once and reuse for all region requests.
- Streamline GrpcQueryHandler trait and implementations to accept
PutRecordBatchRequest containing pre-decoded data.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
feat: stream-based bulk insert with per-batch responses
- Introduce handle_put_record_batch_stream() to process Flight DoPut streams
- Resolve table & permissions once, yield (request_id, AffectedRows) per batch
- Replace loop-over-request with async-stream in frontend & server
- Make PutRecordBatchRequestStream public for cross-crate usage
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
fix: propagate request_id with errors in bulk insert stream
Changes the bulk-insert stream item type from
Result<(i64, AffectedRows), E> to (i64, Result<AffectedRows, E>)
so every emitted tuple carries the request_id even on failure,
letting callers correlate errors with the originating request.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
refactor: unify DoPut response stream to return DoPutResponse
Replace the tuple (i64, Result<AffectedRows>) with Result<DoPutResponse>
throughout the gRPC bulk-insert path so the handler, adapter and server
all speak the same type.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
feat: add elapsed_secs to DoPutResponse for bulk-insert timing
- DoPutResponse now carries elapsed_secs field
- Frontend measures and attaches insert duration
- Server observes GRPC_BULK_INSERT_ELAPSED metric from response
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
refactor: unify Bytes import in flight module
- Replace `bytes::Bytes` with `Bytes` alias for consistency
- Remove redundant `ProstBytes` alias
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
fix: terminate gRPC stream on error and optimize FlightData handling
- Stop retrying on stream errors in gRPC handler
- Replace Vec1 indexing with into_iter().next() for FlightData
- Remove redundant clones in bulk_insert and flight modules
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
Improve permission check placement in `grpc.rs`
- Moved the permission check for `BulkInsert` to occur before resolving the table reference in `GrpcQueryHandler` implementation.
- Ensures permission validation is performed earlier in the process, potentially avoiding unnecessary operations if permission is denied.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
**Refactor Bulk Insert Handling in gRPC**
- **`grpc.rs`**:
- Switched from `async_stream::stream` to `async_stream::try_stream` for error handling.
- Removed `body_size` parameter and added `flight_data` to `handle_bulk_insert`.
- Simplified error handling and permission checks in `GrpcQueryHandler`.
- **`bulk_insert.rs`**:
- Added `raw_flight_data` parameter to `handle_bulk_insert`.
- Calculated `body_size` from `raw_flight_data` and removed redundant encoding logic.
- **`flight.rs`**:
- Replaced `body_size` with `flight_data` in `PutRecordBatchRequest`.
- Updated memory usage calculation to include `flight_data` components.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor/bulk-insert-service:
perf(bulk_insert): encode record batch once per datanode
Move FlightData encoding outside the per-region loop so the same
encoded bytes are reused when mask.select_all(), eliminating redundant
serialisation work.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(mysql): add SHOW WARNINGS support and return warnings for unsupported SET variables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(function): add MySQL IF() function and PostgreSQL description functions for connector compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: show tables for mysql
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partitions table in information_schema and add starrocks external catalog compatibility
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: async udf
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: set warnings
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: impl pg_my_temp_schema and make description functions simple
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add test for issue 7313
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: partition_expression and partition_description
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: test
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: unit tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: saerch_path only works for pg
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: improve warnings processing
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: warnings while writing affected rows and refactor
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: improve ShobjDescriptionFunction signature
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: array_to_boolean
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: gc ctx&procedure
Signed-off-by: discord9 <discord9@163.com>
* fix: handle region not found case
Signed-off-by: discord9 <discord9@163.com>
* docs: more explain&todo
Signed-off-by: discord9 <discord9@163.com>
* per review
Signed-off-by: discord9 <discord9@163.com>
* chore: add time for region gc
Signed-off-by: discord9 <discord9@163.com>
* fix: explain why loader for gc region should fail
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* feat(expr): support vec_elem_avg function
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: support vec_avg function
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* test: add more query test for avg aggregator
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* fix: fix the merge batch mode
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* refactor: use sum and count as state for avg function
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* refactor: refactor merge batch mode for avg function
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
* feat: add additional vector restrictions for validation
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
---------
Signed-off-by: Alan Tang <jmtangcs@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
* mito2: add unit test for flat single-range append_mode dedup behavior
Verify memtable_flat_sources skips dedup when append_mode is true and
performs dedup otherwise for single-range flat memtables, preventing
regressions in the new append_mode path.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix/flat-source-merge:
### Improve Column Metadata Extraction Logic
- **File**: `src/common/meta/src/ddl/utils.rs`
- Modified the `extract_column_metadatas` function to use `swap_remove` for extracting the first schema and decode column metadata for comparison instead of raw bytes. This ensures that the extension map is considered during
verification, enhancing the robustness of metadata consistency checks across datanodes.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
