mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-05-21 15:30:40 +00:00
0ef54511f710f0ef2c05941c8c600bb4c1fd46c8
5314 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yingwen
|
0ef54511f7 |
chore: pick fixes and bump version to v1.0.2 (#8116)
* fix: window sort off by one precision TimeRange&better alias track (#8019) * fix: window sort track alias&off by one precision TimeRange Signed-off-by: discord9 <discord9@163.com> * chore: more test Signed-off-by: discord9 <discord9@163.com> * refactor: clear helper Signed-off-by: discord9 <discord9@163.com> * dedup a bit Signed-off-by: discord9 <discord9@163.com> * feat: even more guard Signed-off-by: discord9 <discord9@163.com> * fix: case insensitive Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> (cherry picked from commitv1.0.2 |
||
Weny Xu
|
8d2f92c01a |
chore: cherry pick fixes and bum version to v1.0.1 (#8024)
* fix: remap peer addresses during retries (#7933) * fix: remap peer addresses during retries Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: styling Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: using uint64 datatype for postgres prepared statement parameters (#7942) * feat: add support for decimal parameter type, remove string replacement fallback * chore: format * fix: add support for using unsigned bigint in postgres * chore: format toml * refactor: cleanup duplicated code * fix: rescale decimal Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix current version comparison logic for pre-releases (#7946) Signed-off-by: liyang <daviderli614@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(index): intersect bitmaps before early exit in predicates applier (#7867) * fix(index): intersect bitmaps before early exit in predicates applier The loop skipped intersecting when the next bitmap was empty, which left the accumulator unchanged instead of zeroing it. Intersect first, then break when the result is empty. Signed-off-by: Weixie Cui <cuiweixie@gmail.com> * per gemini * style(index): format predicates applier loop * fix(index): remove unused mut in predicates applier --------- Signed-off-by: Weixie Cui <cuiweixie@gmail.com> Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: randomize standalone test ports in cli export test (#7955) fix/flaky-test: ### Add Dynamic Port Selection for Standalone Tests - **`cli.rs`**: Implemented functions `random_standalone_addrs` and `choose_random_unused_port_offset` to dynamically select unused ports for standalone tests, enhancing test reliability. - Updated `test_export_create_table_with_quoted_names` to use dynamically assigned ports for HTTP, RPC, MySQL, and PostgreSQL addresses. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: fix git cliff errors in latest version (#7947) * chore: fix git cliff errors in latest version - Fix errors in v2.12.0 - Do not generate logs for beta/rc tags between the compared commits Signed-off-by: evenyag <realevenyag@gmail.com> * chore: preserve blank line before release date in changelog Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: match term zh (#7952) * fix: match term zh Signed-off-by: discord9 <discord9@163.com> * chore: per gemini Signed-off-by: discord9 <discord9@163.com> * chore: revert accident change Signed-off-by: discord9 <discord9@163.com> * feat: unicode script han Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: set upload timeout for uploading artifacts to S3 (#7958) * ci: set upload timeout for uploading artifacts to S3 Signed-off-by: liyang <daviderli614@gmail.com> * Update upload-artifacts-to-s3.sh --------- Signed-off-by: liyang <daviderli614@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: cargo check -p common-meta (#7964) fix: moka feature Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: always skip field pruning when using merge mode (#7957) * test: add prefilter regressions for last_row null filters Signed-off-by: evenyag <realevenyag@gmail.com> * fix: skip fields in all merge mode Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: simplify pre-filter skip fields handling Signed-off-by: evenyag <realevenyag@gmail.com> * test: update test Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: mysql prepare correctly returns error instead of panic (#7963) feat: mysql writer support multiple statement execution Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: relax azblob validation requirements (#7970) Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito2): allow CompactionOutput to succeed independently (#7948) * refactor(mito2): improve compaction error handling and file removal Refactor compaction task execution to enhance error handling and robustness. - Implemented parallel execution of compaction tasks with proper error capture and logging for individual task failures. - Ensured JoinSnafu is no longer directly used in error propagation, instead handling errors within the task processing loop. - Adjusted file removal logic to correctly include expired SSTs after compaction merges. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(mito2): extract SstMerger trait for testability in compaction Extract SstMerger trait and DefaultSstMerger implementation to improve the testability of DefaultCompactor. The DefaultCompactor is now generic over SstMerger, allowing mock implementations to be injected for unit testing without relying on the full object storage access layer. This refactoring separates the concerns of SST file merging from the overall compaction orchestration logic. Additionally: - Updated CompactionScheduler to use DefaultCompactor::default(). - Added unit tests for DefaultCompactor using a MockMerger. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix(compaction): propagate join error during sst flush Correctly propagates the error when joining SST flush handles during compaction. Previously, the error was logged but not returned, leading to potential silent failures. Also reorders some imports for consistency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf(compaction): pre-allocate capacity for compacted_inputs Pre-allocates capacity for the compacted_inputs vector based on the estimated total size of inputs and expired SSTs. This optimization aims to reduce vector reallocations during the compaction process. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/allow-partial-compaction: ### Commit Message Enhance `DefaultCompactor` and `MockMerger` for Improved Flexibility - **`compactor.rs`**: - Added `Clone` trait to `DefaultSstMerger` and `MockMerger` to allow cloning. - Removed `Arc` wrapping from `DefaultCompactor`'s `merger` field for direct usage. - Updated `merge_ssts` method to require `Clone` trait for `SstMerger`. - Modified `MockMerger` to use `Arc<Mutex>` for `results` and `call_idx` to ensure thread safety. - Adjusted error handling to use `error::InvalidMetaSnafu` directly. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate staging leader through lease and heartbeat (#7950) * feat(mito): expose staging leader role state * fix(region): clear staging metadata on leader exit * feat: propagate staging leader role through heartbeat and metasrv * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(region): unify staging exit role transitions * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: cancel local compaction for enter staging (#7885) * feat(mito2): support cancelling active local compaction Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover compaction cancellation return paths Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: cancel remaining tasks Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: move group rollback ownership to parent repartition (#7967) * refactor(meta-srv): move group rollback ownership to parent repartition procedure - Parent procedure now owns partial rollback based on failed/unknown subprocedures - rollback order: group metadata first, then allocated-region cleanup - original_target_routes captured during build-plan, persisted in RepartitionPlanEntry - rollback_group_metadata_routes moved to utils as parent-owned helper - Group subprocedure no longer supports rollback (rollback_supported = false) - Removed UpdateMetadata::RollbackStaging from group state machine - Deleted redundant group rollback tests and helpers BREAKING CHANGE: group Procedure no longer handles rollback; parent procedure is responsible for crash recovery and selecting which plans to roll back. Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: use PreFilterMode::All if only one source in the partition range (#7973) * feat: use PrefilterMode::All if only one source in the partition range Signed-off-by: evenyag <realevenyag@gmail.com> * fix: consider append_mode Signed-off-by: evenyag <realevenyag@gmail.com> * chore: skip merge if only one source Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix test Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): renew operating region leases from keeper roles (#7971) * refactor(meta): store operating region roles in memory keeper Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): register operating region roles from region routes Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): require explicit operating region roles Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): renew operating region leases from keeper roles Signed-off-by: WenyXu <wenymedia@gmail.com> * test(common-meta): cover region route role helpers Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta): cover operating region role propagation Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): prefer metadata roles for region lease renewal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add an index page (#7975) * feat: include an index page * fix: address code review * fix: let / auth gated * refactor: rename public-apis to public-api-prefix Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: remove redundant error messages in admin functions (#7953) Closes #7938 Signed-off-by: yxrxy <yxrxytrigger@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: better jieba cut (#7984) * perf: better jieba cut Signed-off-by: discord9 <discord9@163.com> * fix: also filter pun mark Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * docs: explain why Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: allow ipv4_num_to_string to accept valid integers (#7994) * fix: allow ipv4_num_to_string to accept valid integers Signed-off-by: Johannes Sluis <joesluis51@gmail.com> * test: update sqlness result file Signed-off-by: Johannes Sluis <joesluis51@gmail.com> * fix: use coercible integer signature for ipv4_num_to_string Signed-off-by: Johannes Sluis <joesluis51@gmail.com> --------- Signed-off-by: Johannes Sluis <joesluis51@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update manifest state before deleting delta files (#8001) * fix: update state before deleting deltas Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update log level Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: upgrade mysql metadata value limit to mediumblob (#7985) * fix: upgrade mysql metadata values to mediumblob * fix: fail mysql metadata startup on upgrade check errors Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: zh same underscore behavior (#8002) * fix: zh same underscore behavior Signed-off-by: discord9 <discord9@163.com> * fix: only add token with _ from en analyzer Signed-off-by: discord9 <discord9@163.com> * test: neg sqlness case Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: manifest recovery scans after last version if possible (#8009) * feat: suppport scan with start after Signed-off-by: evenyag <realevenyag@gmail.com> * test: add start_after test Signed-off-by: evenyag <realevenyag@gmail.com> * chore: adjust remove dir warning Signed-off-by: evenyag <realevenyag@gmail.com> * test: test list_with_start_after Signed-off-by: evenyag <realevenyag@gmail.com> * fix: update get_paths call with start_after arg in checkpoint test Signed-off-by: evenyag <realevenyag@gmail.com> * feat: log scan metrics Signed-off-by: evenyag <realevenyag@gmail.com> * fix: fix start_after on manifest dir Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add a standalone flag in plugins during startup (#7974) * chore: add a standalone flag in plugins during startup Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add derive Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: bump version to v1.0.1 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: liyang <daviderli614@gmail.com> Signed-off-by: Weixie Cui <cuiweixie@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: evenyag <realevenyag@gmail.com> Signed-off-by: discord9 <discord9@163.com> Signed-off-by: luofucong <luofc@foxmail.com> Signed-off-by: yxrxy <yxrxytrigger@gmail.com> Signed-off-by: Johannes Sluis <joesluis51@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Ning Sun <sunng@protonmail.com> Co-authored-by: liyang <daviderli614@gmail.com> Co-authored-by: cui <cuiweixie@gmail.com> Co-authored-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@163.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Yingwen <realevenyag@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: LFC <990479+MichaelScofield@users.noreply.github.com> Co-authored-by: yxrxy <1532529704@qq.com> Co-authored-by: Joe Sluis <43276756+JoeS51@users.noreply.github.com> Co-authored-by: shuiyisong <113876041+shuiyisong@users.noreply.github.com>v1.0.1 |
||
|
Yingwen
|
f3dbf34c74 |
chore: bump version to 1.0.0 (#7935)
* chore: bump version to 1.0.0 Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix sqlness test Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix cluster info sqlness Signed-off-by: evenyag <realevenyag@gmail.com> * test: reorder regex in cluster_info Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix pg catalog Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>v1.0.0 |
||
|
Weny Xu
|
6cc68ee8e1 |
fix(repartition): harden repartition rollback paths (#7918)
* fix(meta-srv): restore repartition group metadata on rollback Signed-off-by: WenyXu <wenymedia@gmail.com> * test(meta-srv): add repartition group rollback coverage * fix(meta-srv): rollback allocated regions on repartition failure * test(meta-srv): cover repartition parent rollback flow * test(meta-srv): cover repartition retry paths * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: persist repartition allocate state for retry and rollback Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: retry repartition mailbox channel close Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine logs * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
Ning Sun
|
70ad412092 | fix: resolve postgres format and sync cleanup issues (#7928) | ||
Lei, HUANG
|
2f8607138d |
docs(metric-engine): update prom_store example configs (#7920)
docs: update prom_store example configs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
discord9
|
b623cb1aa2 |
perf: no longer window sort when limit (#7912)
* perf: no longer window sort when limit Signed-off-by: discord9 <discord9@163.com> * test: confusing vector sqlness Signed-off-by: discord9 <discord9@163.com> * chore: redact sqlness Signed-off-by: discord9 <discord9@163.com> * chore: redact every thing Signed-off-by: discord9 <discord9@163.com> * REDACTED Signed-off-by: discord9 <discord9@163.com> * what Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
jeremyhi
|
6e0f5c5042 |
chore: memory limit comment (#7914)
* chore: memory limit comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by gemini comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
6c72dc8e57 |
fix: add overflow check before interleave() (#7921)
* fix: add overflow check before interleave() Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: pass batches and column index to check_interleave_bytes_overflow Refactor check_interleave_bytes_overflow to accept batches and a column index directly, avoiding the intermediate Vec collection of arrays. Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Ning Sun
|
1df9837538 |
refactor!: update arrow-ipc output to stream format (#7922)
* refactor!: update arrow-ipc output to stream format * chore: format |
||
|
Yingwen
|
233e35c0c9 |
feat!: switch default sst format to flat (#7909)
* feat: support alter from primary_key to flat Signed-off-by: evenyag <realevenyag@gmail.com> * chore: alter flat to primary_key Signed-off-by: evenyag <realevenyag@gmail.com> * feat: change default_experimental_flat_format to true Signed-off-by: evenyag <realevenyag@gmail.com> * feat: compute channel size from splitted batch size Signed-off-by: evenyag <realevenyag@gmail.com> * test: add tests for split and channel size Signed-off-by: evenyag <realevenyag@gmail.com> * fix: always set sst_format from manifest on region open sanitize_region_options did not set options.sst_format when the default (PrimaryKey) matched the manifest value, leaving it as None after reopen. This caused the alter format change to appear lost. Signed-off-by: evenyag <realevenyag@gmail.com> * test: fix tests Signed-off-by: evenyag <realevenyag@gmail.com> * test: show create table after alteration Signed-off-by: evenyag <realevenyag@gmail.com> * refactor!: rename default_experimental_flat_format to default_flat_format The flat format is no longer experimental. Remove "experimental" from the config field name, doc comments, and all references. Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>v1.0.0-rc.2-nightly-20260406 |
||
shuiyisong
|
a9256f0310 |
refactor: extract otel helper (#7910)
* refactor: extract otel helper Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: move to submodule Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
a424ee1c0a |
refactor(metric-engine): Refactor PendingRowsBatcher for better testability and benchmarking (#7902)
* perf/schema-align: **Refactor and Enhance Error Handling in `pending_rows_batcher.rs`** - **Refactored `record_failure` Macro**: Moved the `record_failure` macro outside of the `flush_batch_physical` function to improve code reuse and maintainability. - **Enhanced Batch Transformation**: Introduced `transform_logical_batches_to_physical` function to handle the transformation of logical table batches into physical format. - **Batch Concatenation**: Added `concat_modified_batches` function to concatenate modified batches into a single batch. - **Region Write Splitting**: Implemented `split_and_encode_region_writes` function to split combined batches into region-specific writes based on partition rules. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: Add tests for `transform_logical_batches_to_physical` in `pending_rows_batcher.rs` - Implemented `mock_tag_batch` function to create mock `RecordBatch` instances for testing. - Added multiple test cases for `transform_logical_batches_to_physical`: - `test_transform_logical_batches_to_physical_success`: Verifies successful transformation of logical to physical batches. - `test_transform_logical_batches_to_physical_taxonomy_failure`: Tests failure scenario when column IDs are missing. - `test_transform_logical_batches_to_physical_multiple_batches`: Checks handling of multiple batches. - `test_transform_logical_batches_to_physical_mixed_success_failure`: Tests mixed success and failure scenarios. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: refactor `flush_batch_physical` for better testability Introduced several traits to abstract dependencies on CatalogManager, PartitionRuleManager, and NodeManager, enabling easier unit testing with mock implementations. - Added `PhysicalFlushCatalogProvider`, `PhysicalFlushPartitionProvider`, and `PhysicalFlushNodeRequester` traits. - Implemented adapters for existing managers to satisfy the new traits. - Refactored `flush_batch_physical` to use these traits instead of concrete manager references. - Modularized region write planning, resolution, and encoding into standalone functions. - Added comprehensive unit tests for the refactored logic, including edge cases for table lookup and region routing. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: ### Enhance Error Handling and Simplify Code in `error.rs` and `pending_rows_batcher.rs` - **Error Handling Improvements**: - Added new error variants `Partition` and `MetricEngine` in `error.rs` to handle specific error cases. - Updated error propagation using `ResultExt` and `context` for better error messages and handling in `pending_rows_batcher.rs`. - **Code Simplification**: - Removed `FlushWriteResult` enum and refactored `flush_region_writes_concurrently` to return `Result<()>`. - Simplified error handling in `flush_batch_physical` and related functions by removing `first_error` and using `Result` for error propagation. - **Test Adjustments**: - Updated tests to align with the new error handling approach, ensuring they check for specific error messages and conditions. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: refactor `PendingBatch` to use `Option` for cleaner state management Refactored `PendingBatch` in `pending_rows_batcher.rs` to use `Option<PendingBatch>` within the worker loop. This change simplifies initialization and cleanup logic by leveraging `Option::get_or_insert_with` and `Option::take`. - Updated `PendingBatch` fields `created_at` and `ctx` to be non-optional. - Modified `drain_batch` to take `&mut Option<PendingBatch>` and return the drained batch, removing the need for `flush_with_error`. - Simplified the worker loop logic for batch creation and flushing. - Added a unit test `test_drain_batch_takes_initialized_pending_batch_from_option` to verify the new draining logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: share errors across waiters using `Arc<Error>` Enhanced error reporting in `PendingRowsBatcher` by using `Arc<Error>` in `FlushWaiter` and `WorkerCommand`. This allows the same error instance to be shared among all waiters of a batch, avoiding redundant error string conversions and providing more structured error information. - Added `SubmitBatch` variant to `Error` in `error.rs`. - Updated `FlushWaiter` and `WorkerCommand` to use `std::result::Result<(), Arc<Error>>`. - Refactored `notify_waiters` to distribute the shared `Arc<Error>`. - Added `SubmitBatchSnafu` context when receiving results from the worker. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf/schema-align: export types for benchmarking Exported several internal types and traits from `pending_rows_batcher.rs` to enable external benchmarking of the physical batch flushing logic. - Made `PhysicalTableMetadata`, `PhysicalFlushCatalogProvider`, `PhysicalFlushPartitionProvider`, `PhysicalFlushNodeRequester`, `TableBatch`, and `flush_batch_physical` public. - Added a new criterion benchmark `flush_batch_physical.rs` to measure the performance of physical batch flushing with varying numbers of logical tables and rows per table. - Registered the new benchmark in `src/servers/Cargo.toml`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: typo Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(servers): improve error handling and documentation in batcher Refactored error handling in `pending_rows_batcher.rs` by using `ArrowSnafu` for RecordBatch projection errors and simplified partition rule fetching. Added comprehensive documentation for `flush_batch_physical` and updated error display for `SubmitBatch`. - Added `Location` to `Arrow` error variant for better traceability. - Updated `SubmitBatch` display to include source error. - Replaced manual error mapping with `context(error::ArrowSnafu)` in `strip_partition_columns_from_batch`. - Added doc comments to `flush_batch_physical` outlining the pipeline steps. - Optimized capacity allocation in `transform_logical_batches_to_physical`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(servers): clarify physical table metadata and simplify planned batch Renamed `name_to_ids` to `col_name_to_ids` in `PhysicalTableMetadata` to better reflect its purpose. Refactored `PlannedRegionBatch` to use a `num_rows()` method instead of storing a redundant `row_count` field. - Updated `PhysicalTableMetadata` and its usages in `pending_rows_batcher.rs` and benchmarks. - Removed `row_count` field from `PlannedRegionBatch` and added a `num_rows()` helper. - Cleaned up manual `with_context` closures for table lookups. - Fixed a minor formatting issue in worker command processing. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor(servers): simplify flush write structs and centralize metrics Removed redundant `row_count` fields from `FlushRegionWrite` and `PlannedRegionBatch` (made the helper method test-only). Centralized the incrementing of `FLUSH_TOTAL` and `FLUSH_ROWS` metrics into `flush_batch` to avoid duplication and ensure consistency. - Removed `row_count` from `FlushRegionWrite` and `PlannedRegionBatch`. - Marked `PlannedRegionBatch::num_rows()` as `#[cfg(test)]`. - Updated `flush_batch` to handle `FLUSH_TOTAL` and `FLUSH_ROWS` metrics. - Simplified concurrent and sequential flush logic by removing local metric updates. - Cleaned up related tests to match the structural changes. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
jeremyhi
|
f0ea87f52f |
fix: windows ci (#7905)
* fix: windows ci Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: typo Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: use common create_temp_dir Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
shuiyisong
|
8d495909d3 |
feat: auto alter table during trace ingestion from int to float (#7871)
* feat: impl alter table Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor refactor Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: address issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: address issues Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
2af59ed386 |
feat: always use flat scan path for both format (#7901)
* feat: remove primary_key format scan path Signed-off-by: evenyag <realevenyag@gmail.com> * feat: remove flat format flag Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove CompatReader tests Signed-off-by: evenyag <realevenyag@gmail.com> * chore: show whether the format is flat in explain Signed-off-by: evenyag <realevenyag@gmail.com> * test: stable series scan result Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
shuiyisong
|
ba32c5fe9e |
chore: remove unused deps using udeps (#7906)
* chore: remove unused deps using udeps Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: fmt toml Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
d9736407f2 |
fix: return empty when promql gets non-exist label name (#7899)
* fix: return empty when promql gets non-exist label name Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: minor refactor Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: typo Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
shuiyisong
|
3f3407fa24 |
feat: partial success in trace ingestion (#7892)
* feat: impl partial success Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: grouping by resource and scope Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: remove unused code Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: rebase main & fix clippy Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add trace ingestion failure counter Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: address comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: update status list and remove TODO Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: address comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add more tests Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: fmt Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
b75a112561 |
feat: implement prefilter for bulk memtable (#7895)
* feat: prefilter in memtable Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fmt code Signed-off-by: evenyag <realevenyag@gmail.com> * feat: bulk part reader also do prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: extract pk filters check Signed-off-by: evenyag <realevenyag@gmail.com> * fix: scanbench support explain verbose Signed-off-by: evenyag <realevenyag@gmail.com> * feat: add metrics for mem prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: address review comment Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove dead code Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Lei, HUANG
|
2b4e12c358 |
feat: auto-align Prometheus schemas in pending rows batching (#7877)
* feat/auto-schema-align: - **Error Handling Improvements**: - Removed `CatalogSnafu` context from various `.await` calls in `dashboard.rs`, `influxdb.rs`, `jaeger.rs`, `prometheus.rs`, `event.rs`, and `pipeline.rs` to streamline error handling. - **Prometheus Store Enhancements**: - Added support for auto-creating tables and adding missing Prometheus tag columns in `prom_store.rs` and `pending_rows_batcher.rs`. - Introduced `PendingRowsSchemaAlterer` trait for schema alterations in `pending_rows_batcher.rs`. - **Test Additions**: - Added tests for new Prometheus store functionalities in `prom_store.rs` and `pending_rows_batcher.rs`. - **Error Message Improvements**: - Enhanced error messages for catalog access in `error.rs`. - **Server Configuration Updates**: - Updated server configuration to include Prometheus store options in `server.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * reformat Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Add DataTypes Error Handling and Column Renaming Logic - **`error.rs`**: Introduced a new `DataTypes` error variant to handle errors from `datatypes::error::Error`. Updated `ErrorExt` implementation to include `DataTypes`. - **`pending_rows_batcher.rs`**: Added functions `find_prom_special_column_names` and `rename_prom_special_columns_for_existing_schema` to handle renaming of special Prometheus columns. Updated `build_prom_create_table_schema` to simplify error handling with `ConcreteDataType`. - **Tests**: Added a test case `test_rename_prom_special_columns_for_existing_schema` to verify the renaming logic for Prometheus special columns. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - Refactored `PendingRowsBatcher` to accommodate Prometheus record batches: - Introduced `accommodate_record_batch_for_target_schema` to normalize incoming record batches against existing table schemas. - Removed `collect_missing_prom_tag_columns` and `rename_prom_special_columns_for_existing_schema` in favor of the new function. - Added `unzip_logical_region_schema` to extract schema components. - Updated tests in `pending_rows_batcher.rs`: - Added tests for `accommodate_record_batch_for_target_schema` to verify handling of missing tag columns and renaming of special columns. - Ensured error handling for missing timestamp and field columns in target schema. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Commit Summary - **Enhancement in Table Creation Logic**: Updated `prom_store.rs` to modify the handling of `table_options` during table creation. Specifically, `table_options` are now extended differently based on the `AutoCreateTableType`. For `Physical` tables, enforced `sst_format=flat` to optimize pending-rows writes by leveraging bulk memtables. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: Enhance Performance Monitoring in `pending_rows_batcher.rs` - Added performance monitoring timers to various stages of the `PendingRowsBatcher` process, including schema cache checks, table resolution, schema creation, and record batch alignment. - Improved schema handling by adding timers around schema alteration and missing column addition processes. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Enhance Concurrent Write Handling**: Introduced `FlushRegionWrite` and `FlushWriteResult` structs to manage region writes and their results. Added `flush_region_writes_concurrently` function to handle concurrent flushing of region writes based on `should_dispatch_concurrently` logic in `pending_rows_batcher.rs`. - **Testing Enhancements**: Added tests for concurrent dispatching of region writes and the logic for determining concurrent dispatch in `pending_rows_batcher.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Add Histogram for Flush Stage Elapsed Time - **`metrics.rs`**: Introduced a new `HistogramVec` named `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to track the elapsed time of pending rows batch flush stages. - **`pending_rows_batcher.rs`**: Replaced instances of `PENDING_ROWS_BATCH_INGEST_STAGE_ELAPSED` with `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to measure the elapsed time for various flush stages, including `flush_write_region`, `flush_concat_table_batches`, `flush_resolve_table`, `flush_fetch_partition_rule`, `flush_split_record_batch`, `flush_filter_record_batch`, `flush_resolve_region_leader`, and `flush_encode_ipc`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Add design doc for physical table batching in PendingRowsBatcher Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Add implementation plan for physical table batching in PendingRowsBatcher * feat/auto-schema-align: ### Commit Message **Enhance Metric Engine with Physical Batch Processing** - **Add `metric-engine` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `metric-engine` as a workspace dependency. - **Expose Batch Modifier Functions**: Changed visibility of `TagColumnInfo`, `compute_tsid_array`, and `modify_batch_sparse` in `batch_modifier.rs` to public, and made `batch_modifier` a public module in `lib.rs`. - **Implement Physical Batch Processing**: - Added functions `bulk_insert_physical_region` and `bulk_insert_logical_region` in `bulk_insert.rs` to handle physical and logical batch insertions. - Updated `pending_rows_batcher.rs` to attempt physical batch processing before falling back to logical processing, including new functions `flush_batch_physical` and `flush_batch_per_logical_table`. - **Enhance Testing**: - Added tests for physical region passthrough and empty batch handling in `bulk_insert.rs`. - Introduced `with_mito_config` in `test_util.rs` for customized test environments. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Enhance Batch Processing for Table Creation and Alteration - **`prom_store.rs`**: - Added `create_tables_if_missing_batch` and `add_missing_prom_tag_columns_batch` methods to handle batch creation of tables and batch alteration to add missing tag columns. - Implemented logic to determine missing tables and columns, and perform batch operations accordingly. - **`pending_rows_batcher.rs`**: - Updated `PendingRowsBatcher` to utilize batch methods for creating tables an adding missing columns. - Enhanced logic to resolve table schemas and accommodate record batches after batch operations. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: concurrent catalog lookups and eliminate redundant concat_batches on ingest path Replace sequential catalog_manager.table() calls with concurrent futures::future::join_all in align_table_batches_to_region_schema. This affects all three lookup loops: initial table resolution, post-create resolution, and post-alter schema refresh. Reduces O(N) sequential RPC latency to O(1) wall-clock time for requests with many distinct logical tables (e.g. Prometheus remote_write). Remove the per-logical-table concat_batches in flush_batch_physical. Instead of merging all chunks of a table into one RecordBatch before calling modify_batch_sparse, apply modify_batch_sparse directly to each chunk and collect all modified chunks for a single final concat. This eliminates one full data copy per logical table on the flush path. * refactor: extract Prometheus schema alignment helpers into prom_row_builder module Move six functions and their eight unit tests from pending_rows_batcher.rs (~2386 lines) into a new prom_row_builder.rs module (~776 lines), leaving the batcher at ~1665 lines focused on flush/worker machinery. Extracted functions: - accommodate_record_batch_for_target_schema (normalize incoming batch against existing table schema) - unzip_logical_region_schema (extract ts/field/tag columns) - build_prom_create_table_schema (build ColumnSchema vec for table creation) - align_record_batch_to_schema (reorder/fill/cast columns to target schema) - rows_to_record_batch (convert proto Rows to Arrow RecordBatch) - build_arrow_array (build Arrow arrays from proto values) Cleaned up 12 now-unused imports from pending_rows_batcher.rs. * feat/auto-schema-align: ### Enhance `PendingRowsBatcher` and `prom_row_builder` for Efficient Schema Handling - **`pending_rows_batcher.rs`:** - Refactored `submit` method to integrate table batch building and alignment into a single method `build_and_align_table_batches`. - Removed intermediate `RecordBatch` creation, optimizing the process by directly converting proto `RowInsertRequests` into aligned `RecordBatch`es. - Enhanced schema handling by identifying missing columns directly from proto schemas. - **`prom_row_builder.rs`:** - Introduced `rows_to_aligned_record_batch` for direct conversion of proto `Rows` into aligned `RecordBatch`es. - Added `identify_missing_columns_from_proto` to detect absent tag columns without intermediate `RecordBatch`. - Implemented `build_prom_create_table_schema_from_proto` to construct table schemas directly from proto schemas. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: Add elapsed time metrics for bulk insert operations - Updated `bulk_insert` method in `bulk_insert.rs` to record elapsed time metrics using `MITO_OPERATION_ELAPSED` for both physical and logical regions. - Added a new test `test_bulk_insert_records_elapsed_metric` to verify that the elapsed time metric is recorded correctly during bulk insert operations. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * remove flush per logical region Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: **Refactor `flush_batch` and `flush_batch_physical` functions** - Removed unused `catalog` and `schema` variables from `flush_batch` in `pending_rows_batcher.rs`. - Updated `flush_batch_physical` to directly use `ctx.current_catalog()` and `ctx.current_schema()` for resolving table names. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Remove Unused Function and Associated Test - **File:** `src/servers/src/prom_row_builder.rs` - Removed the unused function `build_prom_create_table_schema` which was responsible for building a `Vec<ColumnSchema>` from an Arrow schema. - Deleted the associated test `test_build_prom_create_table_schema_from_request_schema` that validated the removed function. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Remove Test**: Deleted the `test_bulk_insert_records_elapsed_metric` test from `bulk_insert.rs`. - **Refactor Table Resolution**: Introduced `TableResolutionPlan` struct and refactored table resolution logic in `pending_rows_batcher.rs`. - **Enhance Table Handling**: Added functions for collecting non-empty table rows, unique table schemas, and handling table creation and alteration in `pending_rows_batcher.rs`. - **Add Tests**: Implemented tests for `collect_non_empty_table_rows` and `collect_unique_table_schemas` in `pending_rows_batcher.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Refactor Error Handling**: Updated error handling in `pending_rows_batcher.rs` and `prom_row_builder.rs` to use `Snafu` error context for more descriptive error messages. - **Remove Unused Functionality**: Eliminated the `rows_to_record_batch` function and related test in `prom_row_builder.rs` as it was redundant. - **Simplify Function Return Types**: Modified `rows_to_aligned_record_batch` in `prom_row_builder.rs` to return only `RecordBatch` without missing columns, simplifying the function's interface and related tests. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Add Helper Function for Table Options in `prom_store.rs` - Introduced `fill_metric_physical_table_options` function to encapsulate logic for setting table options, ensuring the use of flat SST format and physical table metadata. - Updated `Instance` implementation to utilize the new helper function for setting table options. - Added a unit test `test_metric_physical_table_options_forces_flat_sst_format` to verify the correct application of table options. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Refactor `PendingRowsBatcher`**: Simplified worker retrieval logic in `get_or_spawn_worker` method by using a more concise conditional check. - **Metrics Update**: Added `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` metric in `pending_rows_batcher.rs`. - **Remove Unused Code**: Deleted multiple test functions related to record batch alignment and schema preparation in `pending_rows_batcher.rs` and `prom_row_builder.rs`. - **Function Visibility Change**: Made `build_prom_create_table_schema_from_proto` public in `prom_row_builder.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: remove plan Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Refactor and Simplify Schema Alteration Logic - **Removed Unused Methods**: Deleted `create_table_if_missing` and `add_missing_prom_tag_columns` methods from `PendingRowsSchemaAlterer` trait in `prom_store.rs` and `pending_rows_batcher.rs`. - **Error Handling Improvement**: Enhanced error handling in `create_tables_if_missing_batch` method to return a specific error message for unsupported `AutoCreateTableType` in `prom_store.rs`. - **Visibility Change**: Made `as_str` method public in `AutoCreateTableType` enum in `insert.rs` to support external access. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Commit Message Improve safety in `prom_row_builder.rs` - Updated `unzip_logical_region_schema` to use `saturating_sub` for safer capacity calculation of `tag_columns`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: Add TODO comments for future improvements in `pending_rows_batcher.rs` - Added a TODO comment to consider bounding the `flush_region_writes_concurrently` function. - Added a TODO comment to potentially limit the maximum rows to concatenate in the `flush_batch_physical` function. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Commit Message Enhance error handling in `pending_rows_batcher.rs` - Updated `collect_unique_table_schemas` to return a `Result` type, enabling error handling for duplicate table names. - Modified the function to return an error when duplicate table names are found in `table_rows`. - Adjusted test cases to handle the new `Result` return type in `collect_unique_table_schemas`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Refactor `partition_columns` Method**: Updated the `partition_columns` method in `multi_dim.rs`, `partition.rs`, and `splitter.rs` to return a slice reference instead of a cloned vector, improving performance by avoiding unnecessary cloning. - **Enhance Partition Handling**: Added functions `collect_tag_columns_and_non_tag_indices` and `strip_partition_columns_from_batch` in `pending_rows_batcher.rs` to manage partition columns more efficiently, including stripping partition columns from record batches. - **Update Tests**: Modified existing tests and added new ones in `pending_rows_batcher.rs` to verify the functionality of partition column handling, ensuring correct behavior of the new methods. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Enhance Schema Handling and Validation in `pending_rows_batcher.rs` - **Schema Validation Enhancements**: - Added checks for essential columns (`timestamp`, `value`) in `collect_tag_columns_and_non_tag_indices`. - Introduced `PHYSICAL_REGION_ESSENTIAL_COLUMN_COUNT` to ensure minimum column count in `strip_partition_columns_from_batch`. - Improved error handling for unexpected data types and duplicated columns. - **Function Modifications**: - Updated `strip_partition_columns_from_batch` to project essential columns without lookup. - Modified `flush_batch_physical` to use `essential_col_indices` instead of `non_tag_indices`. - **Test Enhancements**: - Added tests for schema validation, including checks for unexpected data types and duplicated columns. - Verified correct projection of essential columns in `strip_partition_columns_from_batch`. Files affected: `pending_rows_batcher.rs`, `tests`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: - **Add `smallvec` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `smallvec` as a workspace dependency. - **Refactor Function**: Renamed `collect_tag_columns_and_non_tag_indices` to `columns_taxonomy` in `pending_rows_batcher.rs` and updated its return type to use `SmallVec`. - **Update Tests**: Modified test cases in `pending_rows_batcher.rs` to reflect changes in function name and return type. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: **Refactor `pending_rows_batcher.rs` to Simplify Table ID Handling** - Updated `TableBatch` struct to use `TableId` directly instead of `Option<u32>` for `table_id`. - Simplified logic in `flush_batch_physical` by removing the check for `None` in `table_id`. - Adjusted related logic in `start_worker` to accommodate the change in `table_id` handling. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Enhance Batch Processing Logic - **`pending_rows_batcher.rs`**: - Moved column taxonomy resolution inside the loop to handle schema variations across batches. - Added checks to skip processing if both tag columns and essential column indices are empty. - **Tests**: - Added `test_modify_batch_sparse_with_taxonomy_per_batch` to verify batch modification logic with varying schemas. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Remove Primary Key Column Check in `pending_rows_batcher.rs` - Removed the check for the primary key column and other essential column names in the function `strip_partition_columns_from_batch` within `pending_rows_batcher.rs`. - Simplified the logic by eliminating the validation of column order against expected essential names. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Refactor error handling and iteration in `otlp.rs` and `pending_rows_batcher.rs` - **`otlp.rs`**: Simplified error handling by removing `CatalogSnafu` context when awaiting table retrieval. - **`pending_rows_batcher.rs`**: Streamlined iteration over tables by removing unnecessary `into_iter()` calls, improving code readability and efficiency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/metrics-for-bulk: Add timing metrics for batch processing in `pending_rows_batcher.rs` - Introduced `modify_elapsed` and `columns_taxonomy_elapsed` to measure time spent in `modify_batch_sparse` and `columns_taxonomy` functions. - Updated `flush_batch_physical` to record these metrics using `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Commit Summary - **Remove Unused Code**: Eliminated the `#[allow(dead_code)]` attribute from the `compute_tsid_array` function in `batch_modifier.rs`. - **Error Handling Improvement**: Enhanced error handling in `flush_batch_physical` function by adjusting the `match` block in `pending_rows_batcher.rs`. - **Simplify Logic**: Streamlined the logic in `rows_to_aligned_record_batch` by removing unnecessary type casting in `prom_row_builder.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: **Refactor `flush_batch_physical` in `pending_rows_batcher.rs`:** - Moved partition column stripping logic to a single location before processing region batches. - Updated the use of `combined_batch` to `stripped_batch` for consistency in batch processing. - Removed redundant partition column stripping logic within the region batch loop. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/auto-schema-align: ### Update `batch_modifier.rs` Documentation and Parameter Naming - Enhanced documentation for `compute_tsid_array` and `modify_batch_sparse` functions to clarify their logic and parameters. - Renamed parameter `non_tag_column_indices` to `extra_column_indices` in `modify_batch_sparse` for better clarity. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
github-actions[bot]
|
b4492ee39d |
ci: update dev-builder image tag (#7894)
* chore: Update Dockerfile * ci: update dev-builder image tag Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com> --------- Signed-off-by: greptimedb-ci <greptimedb-ci@greptime.com> Co-authored-by: liyang <daviderli614@gmail.com> Co-authored-by: greptimedb-ci <greptimedb-ci@greptime.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
dependabot[bot]
|
0bd0df0e88 |
chore(deps): bump tar from 0.4.44 to 0.4.45 (#7890)
Bumps [tar](https://github.com/alexcrichton/tar-rs) from 0.4.44 to 0.4.45. - [Commits](https://github.com/alexcrichton/tar-rs/compare/0.4.44...0.4.45) --- updated-dependencies: - dependency-name: tar dependency-version: 0.4.45 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
liyang
|
03b2f94821 |
chore: Update Dockerfile (#7893)
* chore: Update Dockerfile * Update update-dev-builder-version.sh |
||
|
dependabot[bot]
|
58ac889818 |
chore(deps): bump rustls-webpki from 0.103.3 to 0.103.10 (#7891)
Bumps [rustls-webpki](https://github.com/rustls/webpki) from 0.103.3 to 0.103.10. - [Release notes](https://github.com/rustls/webpki/releases) - [Commits](https://github.com/rustls/webpki/compare/v/0.103.3...v/0.103.10) --- updated-dependencies: - dependency-name: rustls-webpki dependency-version: 0.103.10 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
jeremyhi
|
ab10696657 |
feat: implement export-v2 chunked data export flow (#7841)
* feat: implement export-v2 chunked data export flow Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by codex comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by gemini comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clippy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: handle empty export ranges consistently Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate resume config Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: file-uri paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: check args on schema-only mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
shuiyisong
|
358524566a |
chore: update ignore list for AI related (#7896)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Yingwen
|
dde1edcdb4 |
fix: incorrect prefilter check (#7886)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
Yao Noel Achi
|
fb2dd862d5 |
fix: avoid cloning serialized view plans on resolve (#7882)
fix: avoid cloning serialized view plan on resolve - Change `ViewInfoValue.view_info` from `Vec<u8>` to `common_base::bytes::Bytes` so resolving a view no longer clones the full serialized plan buffer on every decode. - To keep the change narrow, the metadata write boundary still accepts `Vec<u8>` and converts once when constructing/updating `ViewInfoValue`. The hot read path now uses a cheap clone of the stored bytes. - The benches introduced revealed up to 82% resolution time improvment. Signed-off-by: Yao ACHI <achi.noel@hotmail.com> |
||
|
Ning Sun
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
|
shuiyisong
|
a8fe6b5e44 |
fix: allow auto type upscale conversion in trace ingestion (#7870)
* fix: allow auto type upscale conversion in trace ingestion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: immediate return when parse fails Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: typos Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: add integration test and fix Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: add Int/Float/Bool to String conversion Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: coerce rows together Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: extract coerce Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: save clone Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: unify in- and cross-batch check Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com>v1.0.0-rc.2-a8fe6b5e4-20260330-1774862317 |
||
|
Ning Sun
|
6bd14aaf9f | fix: correct app-name for dashboard (#7884) | ||
|
Yingwen
|
92e2d71f48 |
feat: implement prefilter framework and primary key prefilter (#7862)
* feat: prefilter basic framework Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: move arguments to RowGroupBuildContext Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: skip prefiltered exprs in FlatPruneReader Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove unused functions Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update comment Signed-off-by: evenyag <realevenyag@gmail.com> * feat: handle partition columns in prefilter Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: apply prefiltered selection by and_then Signed-off-by: evenyag <realevenyag@gmail.com> * chore: fix clippy Signed-off-by: evenyag <realevenyag@gmail.com> * fix: handle last row cache Signed-off-by: evenyag <realevenyag@gmail.com> * fix: don't ignore error in PrimaryKeyFilter Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
jeremyhi
|
3904df5397 |
chore: refine track memory metrics semantics (#7874)
Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
cui
|
8e7e4a91d2 |
fix(datatypes): correct ConstantVector rhs comparison in vector equality (#7866)
* fix(datatypes): compare ConstantVector rhs inner in vector equality When either operand is a ConstantVector, the recursive equal() call must compare lhs.inner() against rhs.inner(). The second argument incorrectly used lhs twice, breaking equality when only the rhs was constant. Signed-off-by: Weixie Cui <cuiweixie@gmail.com> * fix: review Signed-off-by: Weixie Cui <cuiweixie@gmail.com> --------- Signed-off-by: Weixie Cui <cuiweixie@gmail.com> |
||
Ruihang Xia
|
fe45ae446c |
perf: optimize promql range functions (#7878)
* bench(promql): add range-function benchmark suite * perf(promql): use flat buffers in range function hot loops * perf(promql): reuse quantile scratch buffersv1.0.0-rc.2-nightly-20260330 |
||
|
Weny Xu
|
6f2ec12059 |
feat(partition): add expression split utility (#7822)
* feat(partition): add expression split utility Implement MVP split logic with checker-safe degrade paths and move module under utils/split with aligned split naming and tests. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: header Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: styling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): degrade split when branch becomes unsatisfiable Detect empty conjunction branches after split and return EmptyBranch instead of silently succeeding. This keeps split behavior aligned with expected partition semantics and adds regression tests for contradictory cuts. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): tighten empty-branch split detection Handle Eq/NotEq contradictions and discrete-gap unsatisfiable ranges in split empty-branch checks. Add regression tests for equality conflicts and impossible int/date intervals. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): degrade singleton and uint impossible split branches Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): enforce finite float bounds in split degradation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): drop date and timestamp support from expr split Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): reject nan and infinity in expr split Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(partition): reuse conjunction bound collection in expr split Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: fmt Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(partition): respect null-first semantics in empty branch checks Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(partition): restrict expr split to range-only shapes Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(partition): clarify split helper scope and test names Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
b57dfc18dc |
feat: pending rows batching for metrics (#7831)
* feat: metric batch 2s PoC Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: max_concurrent_flushes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: work channel size Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(servers): add metrics and logs for pending rows batch flush Add the `FLUSH_ELAPSED` histogram metric to track the duration of pending rows batch flushes in the Prometheus store protocol handler. This provides better observability into the performance and latency of the batcher. Also update telemetry by: - Recording elapsed time for both successful and failed flush operations. - Adding an informational log upon successful flush including row count and duration. - Including elapsed time in error logs when a flush fails. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat(servers): implement columnar batching for pending rows Refactor PendingRowsBatcher to use columnar batching for the metrics store. Incoming RowInsertRequests are now converted to RecordBatches, partitioned, and flushed via BulkInsert requests to datanodes. - Enhance MultiDimPartitionRule to handle scalar boolean predicates. - Add metrics for tracking flush failures and dropped rows. - Update dependencies to support columnar batching in servers. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat(servers): add backpressure for pending rows Implement backpressure in PendingRowsBatcher by limiting in-flight requests with a semaphore and making the submission wait for the flush result. This ensures Prometheus write requests are throttled and only return once the data has been successfully flushed to datanodes. - Add max_inflight_requests to PromStoreOptions. - Use oneshot channels to notify submitters of flush completion. - Limit concurrent requests using a new inflight_semaphore. - Update PendingRowsBatcher::submit to wait for the flush outcome. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add stage-level metrics for bulk ingestion Introduce histograms to track the elapsed time of various stages in the metric engine bulk insert path and the server's pending rows batcher. This provides better observability into the performance bottlenecks of the ingestion pipeline. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * - `src/metric-engine/src/engine/bulk_insert.rs`: Removed the fallback mechanism that converted record batches to rows when bulk inserts were unsupported, along with related helper functions and unused imports. - `src/operator/src/insert.rs`: Removed an unused import (`common_time::TimeToLive::Instant`). Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat(servers): columnar Prom remote write Optimize the Prometheus remote write path by allowing direct conversion from decoded Prometheus samples to Arrow RecordBatches. This bypasses intermediate row-based representations when `PendingRowsBatcher` is active and no pipeline is used, improving ingestion efficiency. - Implement `as_record_batch_groups` in `TablesBuilder` and `PromWriteRequest`. - Add `submit_prom_record_batch_groups` to `PendingRowsBatcher`. - Introduce `DecodedPromWriteRequest` in `prom_store`. - Implement row-to-RecordBatch conversion logic in `prom_row_builder`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * Revert "feat(servers): columnar Prom remote write" This reverts commit efbb63c12a3e7fcec03858ea0351efd94fec8242. * refactor(servers): improve row to RecordBatch conversion - Use `snafu::ensure` for row validation in `rows_to_record_batch`. - Add explicit type hint for `MutableVector` to improve clarity. - Reorganize and clean up imports in `pending_rows_batcher.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf(servers): use arrow builders for row conversion This commit optimizes the conversion from `api::v1::Rows` to `RecordBatch` by using Arrow builders directly. This avoids the overhead of `MutableVector` and `common_recordbatch`, leading to better performance in the `pending_rows_batcher`. Additionally, the `#[allow(dead_code)]` attribute is removed from `modify_batch_sparse` in the metric engine as it is now utilized. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf(metric-engine): optimize batch modification Optimize `modify_batch_sparse` by reusing buffers, using Arrow builders, and employing fast-path encoding methods. This reduces allocations and avoids redundant downcasting and serializer overhead. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/metric-engine-support-bulk: **Add Environment Variable for Batch Sync Control** - `pending_rows_batcher.rs`: Introduced an environment variable `PENDING_ROWS_BATCH_SYNC` to control the synchronization behavior of batch processing. If set to true, the function will wait for the flush result; otherwise, it will return immediatel with the total rows count. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * wip Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: update and fix clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: failing test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * picking-pending-rows-batcher: ### Commit Message Remove Unused Code and Simplify Error Handling - **`src/error.rs`**: Removed the `BatcherQueueFull` error variant and its associated logic, simplifying the error handling by removing unused code. - **`src/http/prom_store.rs`**: Eliminated the `try_decompress` function, streamlining the decompression logic by directly using `snappy_decompress` in `decode_remote_read_request`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: parse PENDING_ROWS_BATCH_SYNC once Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: revert unrelated changes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * **Refactor Prometheus Write Handling** - **`prom_store.rs`**: Introduced `pre_write` method in `PromStoreProtocolHandler` to handle pre-write checks for Prometheus remote write requests. Updated `write` method to utilize `pre_write`. - **`server.rs`**: Modified `PendingRowsBatcher` initialization to conditionally create a batcher based on `with_metric_engine` flag. - **`http/prom_store.rs`**: Integrated `pre_write` checks before submitting requests to `PendingRowsBatcher`. - **`query_handler.rs`**: Added `pre_write` method to `PromStoreProtocolHandler` trait for pre-write operations. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * picking-pending-rows-batcher: - **Fix Label Typo**: Corrected a typo in the label value from `"flush_wn ite_region"` to `"flush_write_region"` in `pending_rows_batcher.rs`. - **Refactor Array Building Logic**: Introduced a macro `build_array!` to streamline the construction of `ArrayRef` for different data types, reducing code duplication in `pending_rows_batcher.rs`. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * format toml Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * picking-pending-rows-batcher: ### Update PromStore and PendingRowsBatcher Configuration - **`prom_store.rs`**: Set `pending_rows_flush_interval` to `Duration::ZERO` to disable automatic flushing. - **`pending_rows_batcher.rs`**: Enhance validation to disable the batcher when `flush_interval` is zero or configuration values like `max_batch_rows`, `max_concurrent_flushes`, `worker_channel_capacity`, or `max_inflight_requests` are zero, preventing potential panics or deadlocks. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * picking-pending-rows-batcher: ### Update `pending_rows_flush_interval` to Zero - **Files Modified**: - `src/frontend/src/service_config/prom_store.rs` - `tests-integration/tests/http.rs` - **Key Changes**: - Updated `pending_rows_flush_interval` from `Duration::from_secs(2)` to `Duration::ZERO` in `prom_store.rs`. - Changed `pending_rows_flush_interval` configuration from `"2s"` to `"0s"` in `http.rs`. These changes set the flush interval to zero, potentially affecting how frequently pending rows are flushed. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * picking-pending-rows-batcher: **Add Worker Management Enhancements** - **`metrics.rs`**: Introduced `PENDING_WORKERS` gauge to track active pending rows batch workers. - **`pending_rows_batcher.rs`**: - Added worker idle timeout logic with `WORKER_IDLE_TIMEOUT_MULTIPLIER`. - Implemented worker management functions: `spawn_worker`, `remove_worker_if_same_channel`, and `should_close_worker_on_idle_timeout`. - Enhanced worker lifecycle management to handle idle workers and ensure proper cleanup. - **Tests**: Added unit tests for worker removal and idle timeout logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Co-authored-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Yingwen
|
d3c8df70f5 |
fix: fix SeriesScan verbose mode mising metrics (#7872)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Ning Sun
|
08ded45c7a |
feat: add common_version customization (#7869)
* feat: add product name customization * chore: update tests |
||
|
Ruihang Xia
|
1118fe243f |
test: filter on region_peers table (#7864)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
discord9
|
d7bc5ad16b |
feat: add incremental read context and scan boundaries (#7848)
* feat: add incremental read context and scan boundaries Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> * docs: explain field Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
jeremyhi
|
8058ce7cf2 |
refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make confg-docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by codex review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: track_with_policy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: mem granularity mb to kb Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by scan_memory_on_exhausted comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: typo Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
fys
|
8d40b129f1 |
chore: remove unused rexpect dev-dependency (#7865)
* chore: remove unused rexpect dev-dependency * fix: taplo fmt |
||
|
Ning Sun
|
73b48b14c1 |
feat: update postgres ParameterDescription size limit (#7861)
* feat: update postgres ParameterDescription size limit * chore: don't log io error |
||
|
Ruihang Xia
|
59dd418629 |
feat: simplify nested aggr inside count query (#7859)
* as optimizer rule Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * dump changes Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * perf: tighten count-count optimizer rewrite * extend inner op set Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * simplify and more coverage Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * remove prom-non-null Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * preserve value column through pruning Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * more sqlness cases Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * rename Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * enforce is not null before inner aggr Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * finalize Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * update sqlness result Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
Boudewijn van Groos
|
ec9d57cecc |
fix: nested views not working (#7857)
Signed-off-by: Boudewijn van Groos <boudewijn.vangroos@foundationzero.org> |
||
|
Lei, HUANG
|
35c5a4adb7 |
fix(mito2): accept post-truncate flush for skip-wal tables (#7858)
Allow flush edits with equal entry ids when flushed sequence advances, so close-time flush after truncate still succeeds for skip-wal regions while stale pre-truncate flushes are rejected. Add a regression test for create->truncate->write->close timing. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Yingwen
|
04aa84af62 |
feat: use ArrowReaderBuilder instead of the RowGroups API (#7853)
* feat: use ArrowReaderBuilder instead of the RowGroups API Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: make row_group_idx required Signed-off-by: evenyag <realevenyag@gmail.com> * chore: remove unsed variant Signed-off-by: evenyag <realevenyag@gmail.com> * fix: collect total_fetch_elapsed metrics Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com>v1.0.0-rc.2-13cdfa9b5-20260325-1774407105 |
||
|
Ning Sun
|
13cdfa9b59 | fix: update 8-bit int to smallint in postgres (#7854) |