mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-13 07:22:32 +00:00
0c8dedea2b66bf14eb4fd80c61118da41d6b658e
1706
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ning Sun
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
 dennis zhuang
|
526dd48bda |
fix(promql): restore exact filtering for nullable label matchers (#9418)
* fix(promql): restore exact filtering for nullable label matchers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: verify compound filter semantics and last-row selection Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: consolidate label filter regression coverage Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: separate filter evaluation and timestamp adaptation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: inline predicate and timestamp tests Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor: keep direct IN predicates outside simple filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: coerce partition bounds before simple filtering Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: cover partition bound overflow after timestamp widening Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 jeremyhi
|
3c0e2a8d55 |
feat: export Metric snapshots with packed Parquet objects (#9382)
* feat: export Metric snapshots with packed Parquet objects Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject empty packed export time ranges Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover cancellation during packed export I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
dennis zhuang
|
079ffec116 |
chore: remove dead code left behind by removed features (#9377)
* chore: remove dead code left behind by removed features Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: remove unused RouteInfoCorrupted error Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
jeremyhi
|
03823a9a01 |
feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the object store WAL config, validated by the datanode and the store. In the `enqueued` mode `append_batch` returns on admission with the entry ids assigned and the object is created in the background. At a backlog threshold the next append is held back until an upload completes. A transient create failure is repeated under the same sequence with the same bytes; any conflicting object poisons the store. `stop` uploads the backlog, or returns the error that dropped it once stop began. `obsolete` clamps the watermark to the durable entry id, and an id the store handed out needs no sequence floor. Add `LogStore::wait_durable` with a default that returns at once. The object store WAL answers it once the region is durable and indexed through the entry id, and fails it after a backlog was lost. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison an enqueued store on permanent create failures Repeat a failed create in the enqueued mode only when the storage error is retryable; any other storage error poisons the store. A conflicting object poisons an enqueued store without reading its epoch, so a failed header read cannot turn the conflict into a retry. A durability wait for an id above the highest id the store handed out now waits for the handed-out ids of the region below it instead of returning at once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison on permanent create failures after stop begins A create that fails with a storage error that is not retryable poisons an enqueued store even after stop began; only a transient failure drops the backlog without poisoning. Durability waiters whose callers stopped waiting are pruned before a new waiter is queued. The backlog age test no longer depends on a follow-up append finishing within the threshold. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): answer a durability wait once no earlier entry is pending A durability wait now returns once the region holds no entry at or below the target that is handed out but not durable, instead of waiting for the largest id handed out to the region. A later object of the region that is still being created no longer holds back a wait whose target it does not cover. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): order the pending durability wait check after the actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(store-api): state that the default wait_durable keeps each store's guarantee The default `LogStore::wait_durable` returns at once, which keeps each log store's own acknowledgement guarantee; Raft Engine with `sync_write = false` acknowledges before its periodic sync, so the documentation no longer claims that every entry id a caller holds is durable. The object store WAL configuration test now also serializes the new options and reads them back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): limit the acknowledgement guarantees to the durable mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): repeat enqueued creates that a retry layer marks persistent An object store wrapped in the OpenDAL retry layer reports a temporary error that outlasted its retries as persistent rather than temporary. The enqueued mode now repeats a create after any storage error that is not permanent, so a transient outage behind the retry layer no longer poisons the store and drops the acknowledged backlog; after stop began such a failure still drops the backlog without poisoning. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a persistent create failure after stop begins Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
  DeviousCardiandClaude Opus 5.5
|
d5a6a9ac05 |
fix(auth): follow symlink chains in watch_file_user_provider (#9365)
* fix(auth): follow symlink chains in watch_file_user_provider The shared file watcher only watched the immediate parent directory of the configured path, so swapping a directory symlink elsewhere in the chain (e.g. `ln -sfn secrets.2 secrets`, as sops-nix does) never produced an event and the new users file was not loaded until restart. The watcher now resolves each path component by component and also watches the directory holding every symlink it meets, plus the directory of the final file. After each relevant event the chain is re-resolved and the watches are re-armed, so later in-place edits of the new target are seen. Only the final file's directory is required at startup; a link directory that cannot be watched (e.g. mode 0711) is logged and retried later. Events are now filtered to the chain's links and final files, so that watching e.g. /tmp does not reload on unrelated files. Names are compared case-insensitively on macOS and Windows. TLS cert reload uses the same helper. Close #9310 Signed-off-by: Aarav <aaravsjadav@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL * fix(common-config): warn when the file watcher watches the root dir On macOS /tmp, /var and /etc are symlinks under /, so following a symlink chain there adds / to the watched directories, and the FSEvents backend then sees every event on the volume. Log a warning when that happens. Also shorten the comment on skipped link directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Bvab9gpXYjKyLN8Mo6JWWL Signed-off-by: Aarav <aaravsjadav@gmail.com> --------- Signed-off-by: Aarav <aaravsjadav@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
jeremyhi
|
bf67246645 |
perf: bound concurrent Metric export writers (#9296)
* fix: preserve objects after conditional Metric export collisions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: share Metric export writer and payload budgets Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover bounded Metric export conversion and mixed roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve export output after ambiguous close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: align HTTP export timeout with managed cancellation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: trim redundant Metric export roundtrips Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: bound pending Metric export writer handles Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: protect export files on failed writes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve ordinary export cleanup and large rows Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: report Metric export collisions as invalid arguments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up failed conditional export closes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support view arrays in managed ordinary export Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: expect successful cleanup after failed close Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: clean up unsynced overwrite after close failure Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: scope export JSON expansion estimates to JSON columns Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: rechunk ordinary export batches within the write budget Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Weny Xu
|
5ffd01a70a |
fix: preserve primary key order when syncing columns (#9189)
* fix: preserve primary key order when syncing columns Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: reject invalid SyncColumns metadata Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Yingwen
|
3c2aac0a55 |
feat: add manual series index reconciliation (#9323)
* feat(mito): support manual series index reconciliation Signed-off-by: evenyag <realevenyag@gmail.com> * feat(storage): route series index build requests Signed-off-by: evenyag <realevenyag@gmail.com> * feat(admin): add BUILD_SERIES_INDEX Signed-off-by: evenyag <realevenyag@gmail.com> * test: specify compaction type in series index fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * test: correct series index SQL fixtures and error assertions Signed-off-by: evenyag <realevenyag@gmail.com> * test(compat): preserve legacy index rebuild across upgrades Signed-off-by: evenyag <realevenyag@gmail.com> * test(sql): cover series index admin validation Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): bound series index maintenance queue Signed-off-by: evenyag <realevenyag@gmail.com> * docs: remove series index how-to guide Signed-off-by: evenyag <realevenyag@gmail.com> * test: remove index build upgrade compatibility case Signed-off-by: evenyag <realevenyag@gmail.com> * fix: address series index reconciliation review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * fix: bound manual series index reconciliation admission Signed-off-by: evenyag <realevenyag@gmail.com> * chore: update greptime-proto to merged index build options Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
dennis zhuang
|
357c935804 |
fix: make vector aggregates work with GROUP BY and partial aggregation (#9338)
* fix: make vector aggregates work with GROUP BY and partial aggregation Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: make partitioned vec_avg case distinguish weighted averages Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Ning Sun
|
20619273ee |
chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates (#9303)
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates Move the workspace from the pinned nightly-2026-03-21 to stable 1.96.1 and drop all 23 '#![feature]' gates across 13 crates, rewriting the still-unstable API usages with stable equivalents: - try_blocks: closures / an async block (table, query, common-function, servers) - duration_constructors: Duration::from_secs(n * 86400) / (n * 60) - iterator_try_collect: collect::<Result<Vec<_>, _>>() - box_patterns: as_deref() + matches! chains (sql) - error_iter: error_chain_root() source-chain walker (common-error); sources() includes the error itself, so the walker never panics - int_roundings: div_floor -> div_euclid (equal for positive divisors) - iter_partition_in_place: stable sort_by_key partition helper (index) - hash_set_entry: HashSet::insert bool / contains+insert - trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin) - string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned() - never_type: Infallible (common-recordbatch) - debug_closure_helpers: closure-backed DebugFmt newtype (mito2) - binary_heap_pop_if: peek().is_some_and() + pop() - exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds - stmt_expr_attributes: stale gate, no usages Also fix release-dev-builder-images.yaml, which parsed rust-toolchain.toml with a date-only regex and would produce empty image versions with a stable channel; it now extracts the full channel token. Dev-builder images verified against stable 1.96.1 (image build, default-toolchain behavior, binstall/nextest, riscv64 and android targets, in-image cargo check). Validated on 1.96.1: cargo check --workspace --all-targets, clippy --workspace --all-targets --all-features -D warnings, cargo fmt --check, and nextest on all 13 affected crates (4586 passed). Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging first. Signed-off-by: Ning Sun <sunning@greptime.com> * chore: update flake checksum * chore: use wild for linker in flake --------- Signed-off-by: Ning Sun <sunning@greptime.com> |
||
|
dennis zhuang
|
affc0a1b1d |
fix: align ordered aggregate state type with the accumulator output (#9340)
* fix: align ordered aggregate state type with the accumulator output Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: reject mismatched aggregate states and keep hard-ordered aggregates unsplit Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: keep WITHIN GROUP aggregates splittable and relabel state fields by position Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cc1be82858 |
fix: merge every state row in geo_path and json_encode_path (#9339)
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 shuiyisong
|
2bc18c483d |
feat: add schema metadata stream to SchemaManager (#9325)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Weny Xu
|
a35f9d5c78 |
fix: address Windows test failures and run full Windows CI (#9305)
* fix: use relative object keys for Windows filesystem access Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: cover Windows path and time limits in full test CI Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: use relative keys in metadata snapshot tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
   
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
dennis zhuang
|
f3eb8e6c72 |
test: cut integration test time and make the storage matrix meaningful (#9308)
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
Lei, HUANG
|
d0f8f4b80c |
feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
Dhruv Vaishnav
|
f1e9a74f00 |
fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
dennis zhuang
|
e65f10c8ad |
fix(query): stop encoding oversized dynamic filters (#9267)
* fix(query): stop encoding oversized dynamic filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): stop bounded encoding partway through an IN list Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 LFC
|
16978cf6c2 |
feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2 Signed-off-by: luofucong <luofc@foxmail.com> * fix: remove unused annotation context import Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
jeremyhi
|
19c127d9c3 |
feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: restore packed metric snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stream large packed parquet entries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: allow default S3 endpoint in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: skip unconfigured S3 in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use portable file URLs in packed restore fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate packed snapshot structure before restore Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve strict manifest decoding and verify fixture Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject packed layout on both database export paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: implement file size in coordinator test storage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use expect_err for rejected export layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 shuiyisongandLei, HUANG
|
66d38e8e1c |
feat: add database ingestion admission through metering (#9239)
* feat: add `ingest_rows_rate_limit` database option Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add `InsertLimitInterceptor` hook to `Inserter` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: attribute insert limit checks to the target table's database Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: unify write admission through metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce write admission for pending row batches Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat: distinguish internal requests for ingestion metering Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: admit split ingestion requests once per database Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: OpenTSDB throws error reason Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: use meter crate main rev Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: exclude database ingest rate limit from table options Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: reserve channel 255 for internal requests Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
dennis zhuang
|
b332bb67d6 |
perf(telemetry): remove the shared lock from TraceLayer (#9269)
* perf(telemetry): remove the shared lock from TraceLayer Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(telemetry): stop collecting span data while tracing is disabled Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 Sainath Singineedi
|
fbbc017be4 |
feat: expose region min/max timestamp in region_statistics (#9060)
* feat: expose region min/max timestamp in region_statistics Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> * test: cover region_statistic time range assembly and projected values Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> --------- Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> |
||
  discord9anddiscord9
|
5ef8a46e3f |
feat(function): add mergeable binary average states (#9062)
* feat(function): add mergeable binary average states Signed-off-by: discord9 <discord9@163.com> * feat(function): expose avg_calc as an OSS scalar Signed-off-by: discord9 <discord9@163.com> * fix(function): borrow invalid AVG scalar argument type Signed-off-by: discord9 <discord9@163.com> * test(function): verify public AVG state SQL finalization Signed-off-by: discord9 <discord9@163.com> * fix(function): return canonical AVG1 state for empty window frames DataFusion's plain-aggregate window executor bypasses the accumulator and calls default_value() directly when a window frame contains no rows. create_udaf's SimpleAggregateUDF derives default_value from the return type, which yields SQL NULL for Binary output instead of the canonical AVG1 empty state, so empty frames and frames over only NULL inputs became observable differently (IS NULL, direct state saves, state comparisons). Register avg_state/avg_merge through a small AggregateUDFImpl (AvgUdaf) that keeps the existing accumulator and declares the canonical AVG1 empty state as default_value, restoring the documented contract. Add a unit test asserting default_value equals the empty accumulator's evaluate() and an sqlness case covering the empty-frame window. Signed-off-by: discord9 <discord9@outlook.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
LFC
|
697cc5fa29 |
fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic Signed-off-by: luofucong <luofc@foxmail.com> * fix(json): align integer extraction with JSON2 conversions Signed-off-by: luofucong <luofc@foxmail.com> * style: format jsonb dependency declaration Signed-off-by: luofucong <luofc@foxmail.com> * fix(otlp): report concrete unsupported JSONB type names Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
6fa375021f |
feat(flow): expose extension-owned batching execution hooks (#9171)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: merge aggregate states in incremental flows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to state merge execution tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): expose extension-owned batching execution hooks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): correct extension matcher borrowing and regression assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: prove executed empty exact deltas keep incremental mode Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to empty exact delta regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover aggregate state merge through the full flow path Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to flow batching task regression calls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
jeremyhi
|
7fb0d9350c |
feat(wal): add the object store WAL provider identity and configuration (#9203)
* feat(wal): add the object store WAL provider identity and configuration Add the identity and configuration of the experimental object store WAL without the log store implementation: - store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a remote WAL. - common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`) with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and `on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as `object_store` with the key `wal.object_store.prefix`. The metasrv config conversion rejects the new provider. - common-meta, meta-srv, mito2: handle the new variants, map the region WAL options to the provider and reject them on a Raft Engine or Kafka log store. - datanode: validate the configuration and fail with a "not supported yet" error until the log store lands. - Example configs and the generated config docs. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(wal): accept object store WAL options when re-enabling WAL Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
e0915a8380 |
refactor: reuse common batching components in Prom ingestion (#9114)
* refactor: compose Prom ingestion with common batcher components Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: share bulk insert IPC encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: centralize batcher worker channel creation Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: share timestamp extraction for batched writes Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: preserve sharded batcher worker lookup Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker lookup and configuration validation Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: validate batcher limits inside fallible constructors Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
d185a8bf76 |
feat: use exact sequence ranges for incremental Flow reads (#9165)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dhruv Vaishnav
|
7e2a75f771 |
feat: allow re-enabling WAL after disabling (#9130)
* feat: allow re-enabling WAL after disabling Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: regenerate skip_wal sqlness result The previous commit changed tests/cases/standalone/common/skip_wal.sql without regenerating the matching .result, so every Sqlness suite failed on the mismatch. tests/cases/distributed/common is a symlink to standalone/common, so the single stale file accounted for all five failing variants. Regenerated from a real run. The recorded output now covers the cases the .sql added: * A table created with skip_wal = 'true' has no real WAL provider, so enabling WAL is refused with Unsupported rather than the previous InvalidArguments. * A RaftEngine-backed table accepts the true -> false transition, and repeating it is a no-op. * SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including after a restart. * A row written while WAL was skipped is absent after restart, while a row written after WAL was restored survives. Verified locally with nothing else running on the machine: sqlness skip_wal passes in both the standalone and distributed environments, and the regenerated file is byte-identical to the output of an independent earlier run. Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: strengthen skip_wal provider coverage Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Fix WAL provider handling during re-enable Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Refactor WAL provider check Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
discord9
|
8af3a04ed7 |
fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
d7ada1761d |
feat: export logical tables from Metric physical scans (#9159)
* feat: add physical Metric table exporter Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: drain Metric export writes before cancellation cleanup Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: construct Metric export error context lazily Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: name the logical table export entry point Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: share Parquet writer for logical table exports Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve Parquet destinations and cancellation boundaries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify logical table export field names Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: clarify logical table export helper responsibilities Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate logical export membership by table ID Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify logical table export coverage and strengthen assertions Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
dennis zhuang
|
7cbd20a053 |
fix(mysql): strip leading comments before the federated statement filter (#9156)
* fix(mysql): strip leading comments before the federated statement filter JDBC clients prefix every statement with a comment. DataGrip sends `/* ApplicationName=DataGrip <version> */` in front of each one, and every pattern in the federated filter is anchored with `^`, so the prefix makes all of them miss. Two failures follow. `SET TRANSACTION READ WRITE` reaches the SQL parser and is rejected. Worse, the DBeaver-specific entries such as `^(/\* ApplicationName=(.*)SELECT @@(.*))` do match DataGrip's prefix, so `SELECT @@GLOBAL.event_scheduler` and `SHOW VARIABLES LIKE ...` are absorbed into a zero-column output, which the MySQL writer sends as an OK packet. The JDBC driver reports that the statement returned no cursor. Strip leading whitespace and comments before matching, and drop the ApplicationName-specific entries the stripping makes redundant. The three that had no unprefixed counterpart (`SHOW PLUGINS`, `SHOW ENGINES`, `SHOW @@...`) keep their behaviour as plain patterns. Also covers gaps found while probing the same path: - `BEGIN` is absorbed like `START TRANSACTION`/`COMMIT`/`ROLLBACK`. - `SHOW [GLOBAL|SESSION|LOCAL] VARIABLES|STATUS` parses; the scope is ignored because GreptimeDB keeps no global/session split. - `USER()`, `CURRENT_USER()`, `SYSTEM_USER()` and `SCHEMA()` are registered. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): keep executable comments, reject multi-statement absorption Review follow-up on the comment stripping, plus the remaining MySQL compatibility gaps from #9155. `/*!...*/` is an executable comment: mysqldump emits its initialization as `/*!40101 SET NAMES ... */`, and the patterns match those verbatim. Stripping it left an empty statement, so nothing matched and the original SQL reached the parser, which rejects `SetNames` and `MultipleAssignments`. Leave executable comments in place. Absorbing a request also has to stop at a statement boundary, because every pattern ends in `(.*)`. `BEGIN; INSERT INTO t VALUES (1)` used to fail on the unsupported `BEGIN`; once `BEGIN` became absorbable the whole request would report success and write nothing. A request is now scanned for a second statement and handed to the query engine if it has one. The scan skips plain comments and string literals so a `;` inside either is not a boundary, and treats a `/*!...*/` that is not the request itself as a statement, since it carries SQL. `check()` now dispatches on the leading statement keyword, so INSERT, UPDATE, CREATE and ordinary SELECTs run no regex at all — this replaces the narrower hand-rolled INSERT shortcut. The statement scan runs only for a request the patterns already matched, which is always a short one. New compatibility surface: - `SELECT CURRENT_USER|SESSION_USER|SYSTEM_USER|USER` without parentheses, and `SELECT @var`, are answered here rather than in the parser. Both are anchored to the whole statement, so `SELECT user FROM t` still reads the column. - `information_schema.plugins`, `user_privileges` and `processlist` are registered as empty tables, like the other MySQL-shape tables around them. Sessions are reported through `information_schema.process_list` and `SHOW PROCESSLIST`; `processlist` carries the column shape only. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Yingwen
|
23a1a5ecaa |
refactor: remove constant vector and replicate operation (#8999)
* refactor: remove constant vector Signed-off-by: evenyag <realevenyag@gmail.com> * refactor: remove vector replicate operation Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve scalar vector types with optional type hints Signed-off-by: evenyag <realevenyag@gmail.com> * fix: remove obsolete mutable vector helper and import Signed-off-by: evenyag <realevenyag@gmail.com> * fix: preserve typed nulls in struct scalar conversion Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
jeremyhi
|
0059508e29 |
perf: avoid directory listing for explicit COPY input files (#9126)
* perf: avoid directory listing for explicit COPY input files Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve COPY input symlink filtering Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
13c69cdaee |
perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): address packed reference review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): stop without retry when maintenance is enabled Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(meta): avoid logging malformed mailbox payloads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
555485c40e |
perf(promql): avoid per-window allocations in simple range functions (#9104)
* perf(promql): avoid per-window allocations in simple range functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): avoid copying smoothing window values Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover smoothing copy removal across window layouts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip null samples in simple range functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
21aed0371f |
feat(pprof): switch CPU profiler to framehop unwinder (#9125)
* feat(pprof): switch CPU profiler to framehop unwinder Replace the default libgcc-based unwinder in pprof-rs with the framehop unwinder, which is designed to be async-signal-safe: - framehop performs no heap allocation during unwinding (MustNotAllocateDuringUnwind) - It handles prologue/epilogue interruption correctly - It falls back to frame-pointer unwinding when CFI is unavailable - It does not depend on libgcc's unwind implementation, which is documented as not signal-safe (see tikv/pprof-rs#36) Bump pprof from 0.14 to 0.15 in all three consumers (common-pprof, cmd, servers) to unify on a single version. pprof 0.15 also replaces parking_lot with spin-rs to avoid a potential profiler deadlock (#268). This addresses the libgcc_s.so.1 #GP crash observed in production when CPU profiling is active, by eliminating the libgcc unwinder from the signal handler path entirely. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(pprof): gate framehop-unwinder to supported targets framehop-unwinder is only available on x86_64/aarch64 Linux/macOS. Enabling it unconditionally for all Unix targets breaks the build on riscv64 and other platforms: pprof disables its backtrace-rs fallback when framehop-unwinder is set, but the framehop module is not compiled on unsupported targets, leaving no TraceImpl implementation. Split the pprof dependency: the base target.'cfg(unix)' block carries the common features (flamegraph, prost-codec, protobuf), and a separate target block adds framehop-unwinder only on supported targets. Cargo unions features from both blocks on matching targets, so x86_64 and aarch64 Linux/macOS get the full feature set while other Unix targets fall back to the default backtrace-rs implementation. Addresses review comment discussion_r3987313632. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
1c4d3a1729 |
fix(ci): check Windows test targets before merge (#9116)
* fix(ci): check Windows test targets before merge Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): use standard Windows runner for checks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): omit dashboard assets from Windows checks Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Dhruv Vaishnav
|
83ff0d8138 |
feat(meta): record catalog and database reconciliation events (#8896)
* feat(meta): record catalog and database reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): join catalog and database reconciliation events Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
Lei, HUANG
|
ab0b1f5cce |
fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix (#9103)
* fix: bump jemalloc crates to 0.7 and patch tikv-jemalloc-sys with tcache init fix Upgrade tikv-jemallocator / tikv-jemalloc-ctl / tikv-jemalloc-sys from 0.6 to 0.7, which embeds jemalloc 5.3.1 (includes a056c20d 'Handle tcache init failures gracefully'). On top of that, patch tikv-jemalloc-sys to the GreptimeTeam fork that adds the remaining upstream fix 54f22c83 'Initialize TSD tcache before enabling it' (GreptimeTeam/jemalloc#1, GreptimeTeam/jemallocator#1). Without the ordering fix, a reentrant allocation during TSD bootstrap (e.g. heap-profiling prof_tdata init / sampled backtrace when prof:true is active) can observe an enabled-but-uninitialized tcache, corrupting per-thread tcache metadata and crashing the process in arena_stats_merge, calloc, or the libgcc unwinder. The patch is pinned by rev and should be removed once tikv/jemallocator ships a jemalloc snapshot that includes 54f22c83. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump tikv-jemalloc-sys patch rev to merged release-5.3.1 GreptimeTeam/jemalloc#1 has been merged; point the patch at the jemallocator commit referencing the merge commit on release-5.3.1. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: point tikv-jemalloc-sys patch at GreptimeTeam/jemallocator main GreptimeTeam/jemallocator#1 has been merged; reference the merge commit e1846d8c on main instead of the PR head branch. Jemalloc source content is unchanged. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(servers): bump tikv-jemallocator dev-dependency to 0.7 main added a target.'cfg(not(windows))'.dev-dependencies entry on tikv-jemallocator 0.6 for servers after this branch diverged. On the merge ref it pulled tikv-jemalloc-sys 0.6 from crates.io, which conflicts with the patched 0.7 (links = "jemalloc" may only appear once in the dependency graph), failing version selection in CI. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Weny Xu
|
adda50e03f |
feat: add repartition partition count hint (#9080)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
38577c8854 |
refactor: support plugin-backed admin functions (#9083)
* feat: add bulk load SQL extension points Add `CREATE BULK LOAD` parsing and `BulkLoadHandler` dispatch. Expose private system table factories through `src/catalog`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: support plugin-backed admin functions Replace the custom bulk-load SQL hooks with access to process `Plugins` through `FunctionState`. Preserve typed external errors from ADMIN functions in `src/operator/src/error.rs`. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
6fa1023b7f |
feat(mito2): introduce TWCS active window compaction (#9011)
* feat(mito2): support independent TWCS trigger_file_num for active and inactive windows Split the single TWCS trigger_file_num into per-window-state thresholds: the active window keeps the existing trigger (default 4, legacy compaction.twcs.trigger_file_num stays a compatible alias), while inactive windows use a new trigger (default 2). Inactive windows additionally fall back from balanced L0-only/L1-only candidates to a progress-making unbalanced mixed candidate so historical windows can converge; the active window retains the row/byte balance guards. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): bound inactive TWCS window convergence by rewrite budget Inactive windows that cannot compact within one level previously either stayed stuck (a threshold-qualified but unbalanced level returned no candidate without trying any fallback) or fell back to a mixed merge with no balance checks at all, which could rewrite a huge compacted file to absorb tiny fresh files. Inactive windows now converge progressively: threshold-qualified balanced picks, sub-threshold balanced single-level picks, a mixed merge whose total rewrite must fit in the output file budget, and finally an L0-only merge without balance checks. Windows that qualify for none of these are left uncompacted, bounding write amplification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover TWCS window trigger options in alter_table_options sqlness case Exercise SET/UNSET of compaction.twcs.active_window.trigger_file_num and compaction.twcs.inactive_window.trigger_file_num end to end, including that setting the canonical active key removes the legacy compaction.twcs.trigger_file_num alias from the table options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): derive TWCS active window from the max-sequence file The active window was determined by the max event-time window among level-0 files. Between an L0 compaction removing its inputs and the next flush landing, level 0 is empty, so the active window transiently became None and every window fell back to the inactive rules - triggering full-window convergence merges during ongoing ingestion whose outputs are then superseded by new data. Flush and compaction outputs both inherit the max input sequence, so the file with the highest sequence across all levels always tracks the most recent write. Use its window as the active window, falling back to the previous L0-based rule when no file carries a sequence (legacy files). The new helper deliberately computes window keys with the assign_to_windows convention (truncate to seconds, then align up), because the result is compared against window keys produced there; the older ceil-based helper is kept unchanged for the legacy fallback path. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): add active-window L1 compaction safety trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(compat): cover TWCS active window options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): align TWCS window trigger validation Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve database TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): prioritize newer compaction windows Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): repick serial compaction outputs Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): configure inactive-window L1 trigger Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): prioritize TWCS compaction candidates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): preserve TWCS trigger downgrade compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(sql): distinguish invalid database option values Separate database option key and value validation so recognized keys report the invalid value and its constraint. Add parser coverage for invalid, valid, and unknown options. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(meta): explain TWCS legacy key compatibility Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify active window trigger field Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): normalize TWCS trigger aliases Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): ignore ineligible files for active window Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): mark explicit TWCS options as overrides Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): normalize zero compaction output size Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(options): validate database TWCS trigger values Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(store-api): collapse TWCS alias conflict condition Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |