* fix: window sort off by one precision TimeRange&better alias track (#8019)
* fix: window sort track alias&off by one precision TimeRange
Signed-off-by: discord9 <discord9@163.com>
* chore: more test
Signed-off-by: discord9 <discord9@163.com>
* refactor: clear helper
Signed-off-by: discord9 <discord9@163.com>
* dedup a bit
Signed-off-by: discord9 <discord9@163.com>
* feat: even more guard
Signed-off-by: discord9 <discord9@163.com>
* fix: case insensitive
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 9fafd879ed)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(server): describe EXPLAIN statements so bind parameters work (#8035)
* fix(server): describe EXPLAIN statements so bind parameters work
`do_describe_inner` only planned `Insert`/`Query`/`Delete`, so
`EXPLAIN` and `EXPLAIN ANALYZE` fell through to the non-plan branch
and had no parameter-type inference. At Bind time the Postgres
handler then reported `unsupported_parameter_type` even though the
inner query would have worked on its own.
Recurse one level into `Statement::Explain` so that an EXPLAIN
wrapping a plannable statement goes through the same describe path.
Adds a tokio-postgres integration test that exercises
`EXPLAIN`/`EXPLAIN ANALYZE` over the extended query protocol.
Fixes#8029
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
* refactor(server): extract plannable-inner check into closure
Reduce duplication between the direct match and the EXPLAIN inner match
by factoring out is_inner_plannable. Behaviour unchanged.
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
(cherry picked from commit 793545d8e6)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: windows windowed sort ci (#8039)
* fix: windows windowed sort ci
Signed-off-by: discord9 <discord9@163.com>
* chore
Signed-off-by: discord9 <discord9@163.com>
* c
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 760581b2a0)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: batched prometheus ingest row metric (#8054)
* fix: count batched prometheus ingest rows
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: align batched ingest metrics
Use actual affected rows when updating `DIST_INGEST_ROW_COUNT` and cache the flush database label to avoid repeated `get_db_string` allocation.
Files: `src/servers/src/pending_rows_batcher.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
(cherry picked from commit f0b3ee4830)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: preserve case in database name from connection string (#8062)
`parse_optional_catalog_and_schema_from_db_string` unconditionally
lowercased database/schema names, causing quoted database names (e.g.
`CREATE DATABASE "TestQuery"`) to be stored with preserved case but
looked up as lowercase on connection, resulting in "Database not found".
Fixes#8059
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
(cherry picked from commit f5c1d5d9bc)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(metric-engine): validate column types and require time index in verify_rows (#8018)
* fix(metric-engine): validate column types and require time index in verify_rows
The remote-write path into the metric engine previously bypassed schema
validation. When a row's time index column carried a non-timestamp
datatype (e.g. a string), the request reached mito's ValueBuilder::push
for the timestamp builder and panicked instead of surfacing a typed
error.
Cache the (column_id, data_type, semantic_type) tuple for each physical
column on PhysicalRegionState and use it in verify_rows to:
- reject columns whose datatype or semantic type disagrees with the
physical region's schema (mirrors mito's WriteRequest::check_schema)
- reject requests that omit the time index column entirely
Field columns stay optional; tag completeness needs per-logical-region
metadata that verify_rows doesn't have and is left to a follow-up.
Fixes#7990.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* refactor(metric-engine): simplify PhysicalColumnInfo construction
- Add From<ColumnMetadata> and From<&ColumnMetadata> for PhysicalColumnInfo
so call sites can use metadata.into() instead of repeating the field list.
- Replace the four struct-literal constructions in create.rs, open.rs and
alter.rs with the conversion.
- In verify_rows, pass &col.column_name to ColumnNotFoundSnafu instead of
cloning it explicitly (snafu's context handles the conversion).
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* perf(metric-engine): cache time index column name in PhysicalRegionState
verify_rows previously scanned every physical column on each row batch to
find the timestamp column. Since the time index is fixed at region
creation and never changes, stash its name on PhysicalRegionState when
the region is first registered and read it directly from there.
add_physical_columns carries a debug_assert to document the invariant
that alter never introduces a new time index.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* perf(metric-engine): borrow physical column names when building name_to_id
On the row-write path we built a HashMap<String, ColumnId> by cloning
every column name out of the physical region's cached state. The map is
scoped to the block that holds the state's read guard, so there's no
need to own the keys.
Switch the map to HashMap<&str, ColumnId> and widen RowsIter::new /
IterIndex::new to accept any key type that borrows as str. Existing
test helpers that pass HashMap<String, ColumnId> keep working through
the Borrow<str> bound.
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* fix: validate metric rows against physical schema
Cache physical column metadata in the metric engine state so row validation and row modification can use the same source of truth for column IDs, data types, and semantic types.
Validate incoming metric rows against the physical schema before writes. Put requests now require the time index and the expected field column, while delete requests keep accepting primary-key-plus-timestamp payloads by skipping the field completeness check.
Pass physical column metadata directly into RowsIter instead of rebuilding a name-to-column-id map at each call site, and cover the new validation paths with tests for missing time indexes, missing fields, and duplicate field columns.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: do not allow adding a new field
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: fill default value for fields
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: fill default for nullable fields
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit d1873ca31d)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: type inference for sql rewrite (#8052)
fix: type inference for rewrited sql
(cherry picked from commit 5b47ec24ec)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: infer time index from column meta on derived table (#8013)
* rough fix
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* reorganize
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* simplification
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix format
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* add comment
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* enhance default by infer
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* supply comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* update sqlness result
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
(cherry picked from commit 0d90f7407c)
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: pre-cast constants (#7926)
* init impl
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* handle no cast
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* refactor using common-expr
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* extend matching pattern
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* more tests
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* simplification
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix zero timestamp
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: normalize sqlness partition count output
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: normalize remaining sqlness plan output
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: normalize sqlness repartition details in tql explain
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* fix: tighten const normalization casts
* test: normalize standalone tql explain repartition output
* resolve cr comments
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
* simplify
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
---------
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
(cherry picked from commit 9133d0464f)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito): ignore compaction override in enum option validation (#8094)
* fix(mito): ignore compaction override in enum option validation
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* test: cover compaction override without compaction type
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
* fix(mito): short-circuit enum option validation
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
---------
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
(cherry picked from commit 73c267e641)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): drop unsound time-filter cache-key stripping (#8105)
* fix(mito2): drop unsound time-filter cache-key stripping
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: update comments and test
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit 5e468190a5)
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: remap batch table route addresses (#8109)
(cherry picked from commit a04fa52486)
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: bump version to v1.0.2
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: avoid stale route update during repartition allocation (#8115)
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update sqlness result
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: QuakeWang <45645138+QuakeWang@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* feat/auto-schema-align:
- **Error Handling Improvements**:
- Removed `CatalogSnafu` context from various `.await` calls in `dashboard.rs`, `influxdb.rs`, `jaeger.rs`, `prometheus.rs`, `event.rs`, and `pipeline.rs` to streamline error handling.
- **Prometheus Store Enhancements**:
- Added support for auto-creating tables and adding missing Prometheus tag columns in `prom_store.rs` and `pending_rows_batcher.rs`.
- Introduced `PendingRowsSchemaAlterer` trait for schema alterations in `pending_rows_batcher.rs`.
- **Test Additions**:
- Added tests for new Prometheus store functionalities in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Message Improvements**:
- Enhanced error messages for catalog access in `error.rs`.
- **Server Configuration Updates**:
- Updated server configuration to include Prometheus store options in `server.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* reformat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add DataTypes Error Handling and Column Renaming Logic
- **`error.rs`**: Introduced a new `DataTypes` error variant to handle errors from `datatypes::error::Error`. Updated `ErrorExt` implementation to include `DataTypes`.
- **`pending_rows_batcher.rs`**: Added functions `find_prom_special_column_names` and `rename_prom_special_columns_for_existing_schema` to handle renaming of special Prometheus columns. Updated `build_prom_create_table_schema` to simplify error handling with
`ConcreteDataType`.
- **Tests**: Added a test case `test_rename_prom_special_columns_for_existing_schema` to verify the renaming logic for Prometheus special columns.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- Refactored `PendingRowsBatcher` to accommodate Prometheus record batches:
- Introduced `accommodate_record_batch_for_target_schema` to normalize incoming record batches against existing table schemas.
- Removed `collect_missing_prom_tag_columns` and `rename_prom_special_columns_for_existing_schema` in favor of the new function.
- Added `unzip_logical_region_schema` to extract schema components.
- Updated tests in `pending_rows_batcher.rs`:
- Added tests for `accommodate_record_batch_for_target_schema` to verify handling of missing tag columns and renaming of special columns.
- Ensured error handling for missing timestamp and field columns in target schema.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Enhancement in Table Creation Logic**: Updated `prom_store.rs` to modify the handling of `table_options` during table creation. Specifically, `table_options` are now extended differently based on the `AutoCreateTableType`. For `Physical` tables, enforced
`sst_format=flat` to optimize pending-rows writes by leveraging bulk memtables.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Enhance Performance Monitoring in `pending_rows_batcher.rs`
- Added performance monitoring timers to various stages of the `PendingRowsBatcher` process, including schema cache checks, table resolution, schema creation, and record batch alignment.
- Improved schema handling by adding timers around schema alteration and missing column addition processes.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Enhance Concurrent Write Handling**: Introduced `FlushRegionWrite` and `FlushWriteResult` structs to manage region writes and their results. Added `flush_region_writes_concurrently` function to handle concurrent flushing of region writes based on
`should_dispatch_concurrently` logic in `pending_rows_batcher.rs`.
- **Testing Enhancements**: Added tests for concurrent dispatching of region writes and the logic for determining concurrent dispatch in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Histogram for Flush Stage Elapsed Time
- **`metrics.rs`**: Introduced a new `HistogramVec` named `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to track the elapsed time of pending rows batch flush stages.
- **`pending_rows_batcher.rs`**: Replaced instances of `PENDING_ROWS_BATCH_INGEST_STAGE_ELAPSED` with `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to measure the elapsed time for various flush stages, including `flush_write_region`, `flush_concat_table_batches`,
`flush_resolve_table`, `flush_fetch_partition_rule`, `flush_split_record_batch`, `flush_filter_record_batch`, `flush_resolve_region_leader`, and `flush_encode_ipc`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add design doc for physical table batching in PendingRowsBatcher
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add implementation plan for physical table batching in PendingRowsBatcher
* feat/auto-schema-align:
### Commit Message
**Enhance Metric Engine with Physical Batch Processing**
- **Add `metric-engine` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `metric-engine` as a workspace dependency.
- **Expose Batch Modifier Functions**: Changed visibility of `TagColumnInfo`, `compute_tsid_array`, and `modify_batch_sparse` in `batch_modifier.rs` to public, and made `batch_modifier` a public module in `lib.rs`.
- **Implement Physical Batch Processing**:
- Added functions `bulk_insert_physical_region` and `bulk_insert_logical_region` in `bulk_insert.rs` to handle physical and logical batch insertions.
- Updated `pending_rows_batcher.rs` to attempt physical batch processing before falling back to logical processing, including new functions `flush_batch_physical` and `flush_batch_per_logical_table`.
- **Enhance Testing**:
- Added tests for physical region passthrough and empty batch handling in `bulk_insert.rs`.
- Introduced `with_mito_config` in `test_util.rs` for customized test environments.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing for Table Creation and Alteration
- **`prom_store.rs`**:
- Added `create_tables_if_missing_batch` and
`add_missing_prom_tag_columns_batch` methods to handle batch creation of tables
and batch alteration to add missing tag columns.
- Implemented logic to determine missing tables and columns, and perform batch
operations accordingly.
- **`pending_rows_batcher.rs`**:
- Updated `PendingRowsBatcher` to utilize batch methods for creating tables an
adding missing columns.
- Enhanced logic to resolve table schemas and accommodate record batches after
batch operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: concurrent catalog lookups and eliminate redundant concat_batches on ingest path
Replace sequential catalog_manager.table() calls with concurrent
futures::future::join_all in align_table_batches_to_region_schema.
This affects all three lookup loops: initial table resolution,
post-create resolution, and post-alter schema refresh. Reduces
O(N) sequential RPC latency to O(1) wall-clock time for requests
with many distinct logical tables (e.g. Prometheus remote_write).
Remove the per-logical-table concat_batches in flush_batch_physical.
Instead of merging all chunks of a table into one RecordBatch before
calling modify_batch_sparse, apply modify_batch_sparse directly to
each chunk and collect all modified chunks for a single final concat.
This eliminates one full data copy per logical table on the flush path.
* refactor: extract Prometheus schema alignment helpers into prom_row_builder module
Move six functions and their eight unit tests from pending_rows_batcher.rs
(~2386 lines) into a new prom_row_builder.rs module (~776 lines), leaving
the batcher at ~1665 lines focused on flush/worker machinery.
Extracted functions:
- accommodate_record_batch_for_target_schema (normalize incoming batch
against existing table schema)
- unzip_logical_region_schema (extract ts/field/tag columns)
- build_prom_create_table_schema (build ColumnSchema vec for table creation)
- align_record_batch_to_schema (reorder/fill/cast columns to target schema)
- rows_to_record_batch (convert proto Rows to Arrow RecordBatch)
- build_arrow_array (build Arrow arrays from proto values)
Cleaned up 12 now-unused imports from pending_rows_batcher.rs.
* feat/auto-schema-align:
### Enhance `PendingRowsBatcher` and `prom_row_builder` for Efficient Schema Handling
- **`pending_rows_batcher.rs`:**
- Refactored `submit` method to integrate table batch building and alignment into a single method `build_and_align_table_batches`.
- Removed intermediate `RecordBatch` creation, optimizing the process by directly converting proto `RowInsertRequests` into aligned `RecordBatch`es.
- Enhanced schema handling by identifying missing columns directly from proto schemas.
- **`prom_row_builder.rs`:**
- Introduced `rows_to_aligned_record_batch` for direct conversion of proto `Rows` into aligned `RecordBatch`es.
- Added `identify_missing_columns_from_proto` to detect absent tag columns without intermediate `RecordBatch`.
- Implemented `build_prom_create_table_schema_from_proto` to construct table schemas directly from proto schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add elapsed time metrics for bulk insert operations
- Updated `bulk_insert` method in `bulk_insert.rs` to record elapsed time metrics using `MITO_OPERATION_ELAPSED` for both physical and logical regions.
- Added a new test `test_bulk_insert_records_elapsed_metric` to verify that the elapsed time metric is recorded correctly during bulk insert operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* remove flush per logical region
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch` and `flush_batch_physical` functions**
- Removed unused `catalog` and `schema` variables from `flush_batch` in `pending_rows_batcher.rs`.
- Updated `flush_batch_physical` to directly use `ctx.current_catalog()` and `ctx.current_schema()` for resolving table names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Unused Function and Associated Test
- **File:** `src/servers/src/prom_row_builder.rs`
- Removed the unused function `build_prom_create_table_schema` which was responsible for building a `Vec<ColumnSchema>` from an Arrow schema.
- Deleted the associated test `test_build_prom_create_table_schema_from_request_schema` that validated the removed function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Remove Test**: Deleted the `test_bulk_insert_records_elapsed_metric` test from `bulk_insert.rs`.
- **Refactor Table Resolution**: Introduced `TableResolutionPlan` struct and refactored table resolution logic in `pending_rows_batcher.rs`.
- **Enhance Table Handling**: Added functions for collecting non-empty table rows, unique table schemas, and handling table creation and alteration in `pending_rows_batcher.rs`.
- **Add Tests**: Implemented tests for `collect_non_empty_table_rows` and `collect_unique_table_schemas` in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor Error Handling**: Updated error handling in `pending_rows_batcher.rs` and `prom_row_builder.rs` to use `Snafu` error context for more descriptive error messages.
- **Remove Unused Functionality**: Eliminated the `rows_to_record_batch` function and related test in `prom_row_builder.rs` as it was redundant.
- **Simplify Function Return Types**: Modified `rows_to_aligned_record_batch` in `prom_row_builder.rs` to return only `RecordBatch` without missing columns, simplifying the function's interface and related tests.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Helper Function for Table Options in `prom_store.rs`
- Introduced `fill_metric_physical_table_options` function to encapsulate logic for setting table options, ensuring the use of flat SST format and physical table metadata.
- Updated `Instance` implementation to utilize the new helper function for setting table options.
- Added a unit test `test_metric_physical_table_options_forces_flat_sst_format` to verify the correct application of table options.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `PendingRowsBatcher`**: Simplified worker retrieval logic in `get_or_spawn_worker` method by using a more concise conditional check.
- **Metrics Update**: Added `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` metric in `pending_rows_batcher.rs`.
- **Remove Unused Code**: Deleted multiple test functions related to record batch alignment and schema preparation in `pending_rows_batcher.rs` and `prom_row_builder.rs`.
- **Function Visibility Change**: Made `build_prom_create_table_schema_from_proto` public in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: remove plan
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor and Simplify Schema Alteration Logic
- **Removed Unused Methods**: Deleted `create_table_if_missing` and `add_missing_prom_tag_columns` methods from `PendingRowsSchemaAlterer` trait in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Handling Improvement**: Enhanced error handling in `create_tables_if_missing_batch` method to return a specific error message for unsupported `AutoCreateTableType` in `prom_store.rs`.
- **Visibility Change**: Made `as_str` method public in `AutoCreateTableType` enum in `insert.rs` to support external access.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Improve safety in `prom_row_builder.rs`
- Updated `unzip_logical_region_schema` to use `saturating_sub` for safer capacity calculation of `tag_columns`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add TODO comments for future improvements in `pending_rows_batcher.rs`
- Added a TODO comment to consider bounding the `flush_region_writes_concurrently` function.
- Added a TODO comment to potentially limit the maximum rows to concatenate in the `flush_batch_physical` function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Enhance error handling in `pending_rows_batcher.rs`
- Updated `collect_unique_table_schemas` to return a `Result` type, enabling error handling for duplicate table names.
- Modified the function to return an error when duplicate table names are found in `table_rows`.
- Adjusted test cases to handle the new `Result` return type in `collect_unique_table_schemas`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `partition_columns` Method**: Updated the `partition_columns` method in `multi_dim.rs`, `partition.rs`, and `splitter.rs` to return a slice reference instead of a cloned vector, improving performance by avoiding unnecessary cloning.
- **Enhance Partition Handling**: Added functions `collect_tag_columns_and_non_tag_indices` and `strip_partition_columns_from_batch` in `pending_rows_batcher.rs` to manage partition columns more efficiently, including stripping partition columns from record batches.
- **Update Tests**: Modified existing tests and added new ones in `pending_rows_batcher.rs` to verify the functionality of partition column handling, ensuring correct behavior of the new methods.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Schema Handling and Validation in `pending_rows_batcher.rs`
- **Schema Validation Enhancements**:
- Added checks for essential columns (`timestamp`, `value`) in `collect_tag_columns_and_non_tag_indices`.
- Introduced `PHYSICAL_REGION_ESSENTIAL_COLUMN_COUNT` to ensure minimum column count in `strip_partition_columns_from_batch`.
- Improved error handling for unexpected data types and duplicated columns.
- **Function Modifications**:
- Updated `strip_partition_columns_from_batch` to project essential columns without lookup.
- Modified `flush_batch_physical` to use `essential_col_indices` instead of `non_tag_indices`.
- **Test Enhancements**:
- Added tests for schema validation, including checks for unexpected data types and duplicated columns.
- Verified correct projection of essential columns in `strip_partition_columns_from_batch`.
Files affected: `pending_rows_batcher.rs`, `tests`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Add `smallvec` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `smallvec` as a workspace dependency.
- **Refactor Function**: Renamed `collect_tag_columns_and_non_tag_indices` to `columns_taxonomy` in `pending_rows_batcher.rs` and updated its return type to use `SmallVec`.
- **Update Tests**: Modified test cases in `pending_rows_batcher.rs` to reflect changes in function name and return type.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `pending_rows_batcher.rs` to Simplify Table ID Handling**
- Updated `TableBatch` struct to use `TableId` directly instead of `Option<u32>` for `table_id`.
- Simplified logic in `flush_batch_physical` by removing the check for `None` in `table_id`.
- Adjusted related logic in `start_worker` to accommodate the change in `table_id` handling.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing Logic
- **`pending_rows_batcher.rs`**:
- Moved column taxonomy resolution inside the loop to handle schema variations across batches.
- Added checks to skip processing if both tag columns and essential column indices are empty.
- **Tests**:
- Added `test_modify_batch_sparse_with_taxonomy_per_batch` to verify batch modification logic with varying schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Primary Key Column Check in `pending_rows_batcher.rs`
- Removed the check for the primary key column and other essential column names in the function `strip_partition_columns_from_batch` within `pending_rows_batcher.rs`.
- Simplified the logic by eliminating the validation of column order against expected essential names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor error handling and iteration in `otlp.rs` and `pending_rows_batcher.rs`
- **`otlp.rs`**: Simplified error handling by removing `CatalogSnafu` context when awaiting table retrieval.
- **`pending_rows_batcher.rs`**: Streamlined iteration over tables by removing unnecessary `into_iter()` calls, improving code readability and efficiency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/metrics-for-bulk:
Add timing metrics for batch processing in `pending_rows_batcher.rs`
- Introduced `modify_elapsed` and `columns_taxonomy_elapsed` to measure time spent in `modify_batch_sparse` and `columns_taxonomy` functions.
- Updated `flush_batch_physical` to record these metrics using `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Remove Unused Code**: Eliminated the `#[allow(dead_code)]` attribute from the `compute_tsid_array` function in `batch_modifier.rs`.
- **Error Handling Improvement**: Enhanced error handling in `flush_batch_physical` function by adjusting the `match` block in `pending_rows_batcher.rs`.
- **Simplify Logic**: Streamlined the logic in `rows_to_aligned_record_batch` by removing unnecessary type casting in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch_physical` in `pending_rows_batcher.rs`:**
- Moved partition column stripping logic to a single location before processing region batches.
- Updated the use of `combined_batch` to `stripped_batch` for consistency in batch processing.
- Removed redundant partition column stripping logic within the region batch loop.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Update `batch_modifier.rs` Documentation and Parameter Naming
- Enhanced documentation for `compute_tsid_array` and `modify_batch_sparse` functions to clarify their logic and parameters.
- Renamed parameter `non_tag_column_indices` to `extra_column_indices` in `modify_batch_sparse` for better clarity.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* bench(promql): add range-function benchmark suite
* perf(promql): use flat buffers in range function hot loops
* perf(promql): reuse quantile scratch buffers
* feat: metric batch 2s PoC
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: max_concurrent_flushes
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: work channel size
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(servers): add metrics and logs for pending rows batch flush
Add the `FLUSH_ELAPSED` histogram metric to track the duration of pending
rows batch flushes in the Prometheus store protocol handler. This provides
better observability into the performance and latency of the batcher.
Also update telemetry by:
- Recording elapsed time for both successful and failed flush operations.
- Adding an informational log upon successful flush including row count and duration.
- Including elapsed time in error logs when a flush fails.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): implement columnar batching for pending rows
Refactor PendingRowsBatcher to use columnar batching for the metrics
store. Incoming RowInsertRequests are now converted to RecordBatches,
partitioned, and flushed via BulkInsert requests to datanodes.
- Enhance MultiDimPartitionRule to handle scalar boolean predicates.
- Add metrics for tracking flush failures and dropped rows.
- Update dependencies to support columnar batching in servers.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): add backpressure for pending rows
Implement backpressure in PendingRowsBatcher by limiting in-flight
requests with a semaphore and making the submission wait for the flush
result. This ensures Prometheus write requests are throttled and only
return once the data has been successfully flushed to datanodes.
- Add max_inflight_requests to PromStoreOptions.
- Use oneshot channels to notify submitters of flush completion.
- Limit concurrent requests using a new inflight_semaphore.
- Update PendingRowsBatcher::submit to wait for the flush outcome.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add stage-level metrics for bulk ingestion
Introduce histograms to track the elapsed time of various stages in the
metric engine bulk insert path and the server's pending rows batcher.
This provides better observability into the performance bottlenecks
of the ingestion pipeline.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* - `src/metric-engine/src/engine/bulk_insert.rs`: Removed the fallback mechanism that converted record batches to rows when bulk inserts were unsupported, along with related helper functions and unused imports.
- `src/operator/src/insert.rs`: Removed an unused import (`common_time::TimeToLive::Instant`).
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): columnar Prom remote write
Optimize the Prometheus remote write path by allowing direct conversion
from decoded Prometheus samples to Arrow RecordBatches. This bypasses
intermediate row-based representations when `PendingRowsBatcher` is
active and no pipeline is used, improving ingestion efficiency.
- Implement `as_record_batch_groups` in `TablesBuilder` and `PromWriteRequest`.
- Add `submit_prom_record_batch_groups` to `PendingRowsBatcher`.
- Introduce `DecodedPromWriteRequest` in `prom_store`.
- Implement row-to-RecordBatch conversion logic in `prom_row_builder`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Revert "feat(servers): columnar Prom remote write"
This reverts commit efbb63c12a3e7fcec03858ea0351efd94fec8242.
* refactor(servers): improve row to RecordBatch conversion
- Use `snafu::ensure` for row validation in `rows_to_record_batch`.
- Add explicit type hint for `MutableVector` to improve clarity.
- Reorganize and clean up imports in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(servers): use arrow builders for row conversion
This commit optimizes the conversion from `api::v1::Rows` to `RecordBatch`
by using Arrow builders directly. This avoids the overhead of
`MutableVector` and `common_recordbatch`, leading to better performance
in the `pending_rows_batcher`.
Additionally, the `#[allow(dead_code)]` attribute is removed from
`modify_batch_sparse` in the metric engine as it is now utilized.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(metric-engine): optimize batch modification
Optimize `modify_batch_sparse` by reusing buffers, using Arrow
builders, and employing fast-path encoding methods. This reduces
allocations and avoids redundant downcasting and serializer overhead.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-support-bulk:
**Add Environment Variable for Batch Sync Control**
- `pending_rows_batcher.rs`: Introduced an environment variable `PENDING_ROWS_BATCH_SYNC` to control the synchronization behavior of batch processing. If set to true, the function will wait for the flush result; otherwise, it will return immediatel
with the total rows count.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* wip
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: update and fix clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: failing test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Commit Message
Remove Unused Code and Simplify Error Handling
- **`src/error.rs`**: Removed the `BatcherQueueFull` error variant and its associated logic, simplifying the error handling by removing unused code.
- **`src/http/prom_store.rs`**: Eliminated the `try_decompress` function, streamlining the decompression logic by directly using `snappy_decompress` in `decode_remote_read_request`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: parse PENDING_ROWS_BATCH_SYNC once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: revert unrelated changes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* **Refactor Prometheus Write Handling**
- **`prom_store.rs`**: Introduced `pre_write` method in `PromStoreProtocolHandler` to handle pre-write checks for Prometheus remote write requests. Updated `write` method to utilize `pre_write`.
- **`server.rs`**: Modified `PendingRowsBatcher` initialization to conditionally create a batcher based on `with_metric_engine` flag.
- **`http/prom_store.rs`**: Integrated `pre_write` checks before submitting requests to `PendingRowsBatcher`.
- **`query_handler.rs`**: Added `pre_write` method to `PromStoreProtocolHandler` trait for pre-write operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
- **Fix Label Typo**: Corrected a typo in the label value from `"flush_wn ite_region"` to `"flush_write_region"` in `pending_rows_batcher.rs`.
- **Refactor Array Building Logic**: Introduced a macro `build_array!` to streamline the construction of `ArrayRef` for different data types, reducing code duplication in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* format toml
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update PromStore and PendingRowsBatcher Configuration
- **`prom_store.rs`**: Set `pending_rows_flush_interval` to `Duration::ZERO` to disable automatic flushing.
- **`pending_rows_batcher.rs`**: Enhance validation to disable the batcher when `flush_interval` is zero or configuration values like `max_batch_rows`, `max_concurrent_flushes`, `worker_channel_capacity`, or `max_inflight_requests` are zero, preventing potential panics or deadlocks.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update `pending_rows_flush_interval` to Zero
- **Files Modified**:
- `src/frontend/src/service_config/prom_store.rs`
- `tests-integration/tests/http.rs`
- **Key Changes**:
- Updated `pending_rows_flush_interval` from `Duration::from_secs(2)` to `Duration::ZERO` in `prom_store.rs`.
- Changed `pending_rows_flush_interval` configuration from `"2s"` to `"0s"` in `http.rs`.
These changes set the flush interval to zero, potentially affecting how frequently pending rows are flushed.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
**Add Worker Management Enhancements**
- **`metrics.rs`**: Introduced `PENDING_WORKERS` gauge to track active pending rows batch workers.
- **`pending_rows_batcher.rs`**:
- Added worker idle timeout logic with `WORKER_IDLE_TIMEOUT_MULTIPLIER`.
- Implemented worker management functions: `spawn_worker`, `remove_worker_if_same_channel`, and `should_close_worker_on_idle_timeout`.
- Enhanced worker lifecycle management to handle idle workers and ensure proper cleanup.
- **Tests**: Added unit tests for worker removal and idle timeout logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: jeremyhi <fengjiachun@gmail.com>
* fix: resolve optimization issue for extended query
* fix: type cast from subquery
* chore: update error information in sqlness
* chore: switch to released pgwire
* refactor: remove optimize function completely
* chore: add more tests
* test: attempt to fix the fuzz issue
* fix: try to resolve the test issue
* feat(metric-engine): support bulk inserts
Implement `RegionRequest::BulkInserts` to support efficient columnar data
ingestion in the metric engine.
Key changes:
- Implement `bulk_insert_region` to handle logical-to-physical region mapping
and dispatch writes.
- Add `batch_modifier` for `RecordBatch` transformations, specifically for
`__tsid` generation and sparse primary key encoding.
- Integrate `BulkInserts` into the `MetricEngine` request handling logic.
- Provide a row-based fallback mechanism if the underlying storage doesn't
support bulk writes.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
### Update `bulk_insert.rs` to Support Partition Expression Version
- **Enhancements**:
- Added support for `partition_expr_version` in `RegionBulkInsertsRequest` and `RegionPutRequest`.
- Modified the handling of `partition_expr_version` to be dynamically set from the `request` object.
Files affected:
- `src/metric-engine/src/engine/bulk_insert.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: cargo lock revert
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* add doc for conversions
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: simplify test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
### Refactor `bulk_insert.rs` in `metric-engine`
- **Refactor Functionality**:
- Replaced `resolve_tag_columns` with `resolve_tag_columns_from_metadata` to streamline tag column resolution.
- Moved logic for resolving tag columns directly into `resolve_tag_columns_from_metadata`, removing the need for an external function call.
- **Enhancements**:
- Improved error handling and context provision for missing physical regions and columns.
- Optimized tag column sorting and index management within the batch processing logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
### Refactor `record_batch_to_rows` Function in `bulk_insert.rs`
- Simplified the `record_batch_to_rows` function by removing the `logical_metadata` parameter and directly validating column types within the function.
- Enhanced error handling for timestamp, value, and tag columns by checking their data types and providing detailed error messages.
- Replaced the use of `Helper::try_into_vector` with direct downcasting to `TimestampMillisecondArray`, `Float64Array`, and `StringArray` for improved type safety and clarity.
- Updated the construction of `api::v1::Rows` to directly handle null values and construct `api::v1::Value` objects accordingly.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
## Commit Message
Refactor `bulk_insert.rs` to optimize state access
- Moved the state read operation inside a new block to limit its scope and improve code clarity.
- Adjusted logic for processing `tag_columns` and `non_tag_indices` to work within the new block structure.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
### Refactor `compute_tsid_array` Function
- **Refactored `compute_tsid_array` function**: Modified the function signature to accept `tag_arrays` as a parameter instead of building it internally. This change affects the following files:
- `src/metric-engine/src/batch_modifier.rs`
- **Updated test cases**: Adjusted test cases to accommodate the new `compute_tsid_array` function signature by passing `tag_arrays` explicitly.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* docs: add doc for bulk_insert_region
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
### Commit Message
Refactor `bulk_insert.rs` in `metric-engine`:
- Removed error handling for unsupported status codes in `write_data` method.
- Eliminated `record_batch_to_rows` function, simplifying the data insertion process.
- Streamlined the `write_data` method by removing fallback logic for unsupported operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
- **Optimize Primary Key Construction**: Refactored `modify_batch_sparse` in `batch_modifier.rs` to use `BinaryBuilder` for more efficient primary key construction.
- **Add Fallback for Unsupported Bulk Inserts**: Updated `bulk_insert.rs` to handle unsupported bulk inserts by converting record batches to rows and using `RegionPutRequest`.
- **Implement Record Batch to Rows Conversion**: Added `record_batch_to_rows` function in `bulk_insert.rs` to convert `RecordBatch` to `api::v1::Rows` for fallback operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
Add test for handling null values in `record_batch_to_rows`
- Added a new test `test_record_batch_to_rows_with_null_values` in `bulk_insert.rs` to verify the handling of null values in the `record_batch_to_rows` function.
- The test checks the conversion of a `RecordBatch` with null values in various fields to ensure correct row creation and schema handling.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-bulk-insert:
Add fallback path for unsupported status and improve error context handling
- **`bulk_insert.rs`**:
- Added a fallback path for `PartitionTreeMemtable` in case of unsupported status code.
- Enhanced error handling by using `with_context` for better error messages when timestamp and value columns are not found in `RecordBatch`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: cast filters type for scanbench
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: pub file_range mod
So we can use the pub struct FileRange in other places
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: add api as dev-dependency to cmd for clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support profiling after warmup
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* perf/prom-decode:
**Refactor `PromLabel` to Use Raw Byte Slices**
- Updated `PromLabel` struct in `proto.rs` to use `RawBytes` for `name` and `value` fields, replacing `Bytes` with static byte slices.
- Modified test cases in `prom_row_builder.rs` to accommodate changes in `PromLabel` by using byte literals.
- Simplified `merge_bytes` function in `proto.rs` to directly assign byte slices, removing unnecessary memory operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf/prom-decode:
- **Add UTF-8 Validation**: Introduced `validate_bytes` method in `http.rs` to validate UTF-8 encoding using `simdutf8` for `PromValidationMode::Strict`.
- **Update Column Indexing**: Modified `prom_row_builder.rs` to use `Vec<u8>` for `col_indexes` keys, ensuring UTF-8 validation for label names.
- **Dependency Update**: Added `simdutf8` version `0.1.5` to `Cargo.toml` and updated `Cargo.lock` to include this new dependency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: style issues
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: update pgwire
* feat: add special parser for copy to stdout
* feat: implement copy to stdout
* fix: improve code
* fix: expect optional with
* fix: lint
* feat: correct encoder using and refactor
* chore: fmt
* refactor: update api
* chore: use released dependencies
* fix: update datafusion-pg-catalog to support schema query
* fix: support for double quoted identifier
* feat: update datafusion-postgres to support schema.table
* refactor: use pgsqlparser container
* refactor: remove unquote which is no longer needed
* fix: correctly handle invalid query
* fix: correct handle null in nano timestamp
* test: add a new test for additional close )
