mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-03 18:45:35 +00:00
c7fa48ef95bc5da1f245d9e292ee53b28a4f2136
1085
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Weny Xu
|
4c98fa5265 |
ci: add optional AWS runners for observability benchmarks (#9322)
* ci: add optional AWS runners for observability benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: restore automatic benchmark disk sizing Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 dennis zhuang
|
88197f4019 |
fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings A vector-vector binary operation projected one operand's whole tag set and inner-joined without any cardinality check, so `on()`/`ignoring()` did not reduce the result labels, `group_left`/`group_right` changed nothing, and a non-unique match group produced a cross product that PromQL cannot represent. Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the matching labels, `ignoring(...)` drops them, and a group modifier keeps the many side's labels plus the `group_x(...)` labels taken from the one side. A label the one side does not carry is deleted from the result. The reduced label set no longer identifies the operand series, so `__tsid` is dropped from the context on this path. Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)` window and a scalar UDF that fails the query on a repeated group: on the one side before the join, and on the result labels after it, matching where Prometheus raises each of its three errors. Series are unique by their whole tag set, so the window is only planted when the match keys drop a tag; plain arithmetic and `on(<all tags>)` plan exactly as before. Closes #9207, closes #9208, closes #9209. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): keep __tsid when the result labels are an operand's whole tag set Deriving the result labels dropped `__tsid` from the context unconditionally, so an enclosing operation fell back to joining on the tag columns even where the column still identified the result series. Keep it when every result label comes from one operand and covers that operand's whole tag set: no other operand value reaches the labels, and the matching gives each of its rows a single partner, so its `__tsid` is still one per result series. That is the common `on(<all tags>)` and bare `group_left` shape; a matching that actually drops a tag still clears it. The column is re-qualified as the result's own, which is how the enclosing expression and the context look it up. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep the match group count column unambiguous The cardinality check aliased its row count to a fixed `__promql_match_group_count`. An operand carrying a label of that name made the window output two fields with the same name, and planning failed with "Schema contains qualified field name collide_right.__promql_match_group_count and unqualified field name __promql_match_group_count which would be ambiguous". Pick a name the operand does not already have, the way the `or` operator allocates its match key columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover a match group spread over several regions The cardinality check runs above the merge of the region scans, so it counts a match group globally. Nothing pinned that: every table in these cases holds a single region, and a check evaluated per region would pass them all. Partition the operand on a column outside the match keys, which puts the two series of one match group in different regions, and assert both the pre-join and the post-join check still reject it. The case runs in the distributed environment too, where the regions sit on different datanodes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): group an outer aggregate on the labels the operand kept `by`/`without` planning topped up a missing grouping column by walking down to the table scan and re-projecting it. That is right for a column the scan pruned for efficiency, but the labels a matching modifier deletes are also absent from the operand's output, and they were restored the same way: sum without(host) (a / on(host) b) `on(host)` leaves the operand with `host` alone, so the sum covers everything and Prometheus answers `{} 10`. Instead `device` came back from the scan under `a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for `sum by(device)` of that operand, which has no `device` to group on at all. Take the grouping labels from the operand's own label set rather than from the row keys of the scan beneath it. A label pruned from the plan is still in that set and still gets restored; a label the operand dropped is not. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): drop the unreachable aggregation tag top-up `by`/`without` planning could restore a grouping column that the plan no longer carried by rewriting the scan underneath it. Once the grouping labels come from the operand's own label set, there is nothing left for it to restore: a scan projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds matcher columns to that set), so a label in the set is always in the schema. Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the standalone and the distributed environment. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): drop cases that guarded the removed tag top-up Three plain selector aggregates were there to show that restoring a pruned grouping column still worked. With the restore gone they only repeat what the aggregate cases already cover. Also fix a comment that still said the metric engine scan prunes tag columns: it projects every label of the operand. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): note the duplicate a propagated matcher hides A matcher copied onto the one-side operand removes groups without a partner before the cardinality check sees them, so a duplicate in such a group is not reported. Prometheus checks every group of the one side and fails the query. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 fys
|
5a78e07be6 |
fix(json2): restrict JSON2 type hints (#9316)
* fix(sql): restrict JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): allow equivalent 64-bit aliases in JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * fix(sql): support UInt64 conversion and preserve JSON2 hint aliases Signed-off-by: fys <fengys1996@gmail.com> * refactor(sql): remove redundant JSON2 type hint normalization Signed-off-by: fys <fengys1996@gmail.com> * fix(datatypes): restrict JSON2 hint types in JsonSettings::try_new Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
   
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
d0f8f4b80c |
feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
fys
|
045441e3cc |
feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
dennis zhuang
|
25bca4609f |
perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Lei, HUANG
|
b5199bc59a |
feat: add experimental Jev SQL filtering (#9265)
* feat: add experimental Jev SQL filtering Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: gate Jev filtering behind an opt-in Cargo feature Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: verify Jev registration with default features Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: clarify Jev concurrency scope and stabilization work Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: enable renamed ai-functions feature by default Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * ci: remove extra AI feature-off checks Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: rename AI feature to ai_functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 LFC
|
16978cf6c2 |
feat(trace): support Semantic Graph for Trace V2 (follow-up to #9192) (#9278)
* feat(trace): support Semantic Graph for Trace V2 Signed-off-by: luofucong <luofc@foxmail.com> * fix: remove unused annotation context import Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
 jeremyhi
|
19c127d9c3 |
feat: restore packed metric snapshots (#9250)
* test: cover snapshot parquet restore compatibility Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: restore packed metric snapshots Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: stream large packed parquet entries Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: allow default S3 endpoint in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: skip unconfigured S3 in packed restore test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use portable file URLs in packed restore fixtures Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: validate packed snapshot structure before restore Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: preserve strict manifest decoding and verify fixture Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject packed layout on both database export paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: implement file size in coordinator test storage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: use expect_err for rejected export layouts Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9
|
233e23b352 |
fix(promql): keep value-field grouping labels out of matching filter propagation (#9242)
* fix(promql): keep value-field grouping labels out of matching filter propagation #9202 propagates matching-label matchers to the scanned selector using the planned operands' tag columns. For an aggregated operand those are its grouping labels, and agg_modifier_to_col resolves by(...) names against the input schema only, so a value field named in by(...) is reported as a tag. A value field varies between the samples of one series, so lowering a matcher on it below sample selection (PromInstantManipulate) can drop the newest sample, promote a stale one from the lookback window, and fabricate a match the un-rewritten query does not produce. Track the by(...) labels that name value fields of the aggregated operand's input in PromPlannerContext::aggregation_field_labels, and refuse to propagate matchers on them while still requiring a matcher to name a grouping label to cross an aggregation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: sync matching_filter result fixture comment with the PR number Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Ning Sun
|
4df557bb9e |
test: exclude testing feature completely (#9072)
* test: exclude testing feature completely * chore: fmt |
||
|
discord9
|
9e9a8cac20 |
fix(query): insert MergeScan into nested scalar subqueries (#9261)
* fix(query): insert MergeScan into nested scalar subqueries DataFusion 55 keeps uncorrelated scalar subqueries as expression subqueries (enable_physical_uncorrelated_scalar_subquery, default true) instead of decorrelating them into joins, and executes them via the new physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked the plan with a plain TreeNode transform that does not descend into expression subqueries, so MergeScan was only inserted for depth-1 subqueries. A scalar subquery nested inside another scalar subquery kept a bare frontend DistTable TableScan and failed at execution with "Unsupported operation: get stream from a distributed table". Use the subquery-aware transform so handle_subquery (PlanRewriter / MergeScan insertion) runs for subquery plans at every nesting depth. Fixes #9260. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): strengthen nested scalar subquery regression coverage Address review on #9261: - Replace the ineffective 'no bare TableScan' string check with a real subquery-aware plan walk (apply_with_subqueries); MergeScan hides its remote input from traversal, so any TableScan the walk reaches was genuinely left unwrapped. - Add a distributed regression case on a range-partitioned table so the nested inner aggregate must merge partial results across regions (global AVG feeding an outer SUM filter). Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
fys
|
e82cb0af74 |
feat(json2): respect type hints during JSON2 type concretization (#9222)
* fix: prefer JSON2 type hints for uncast read pushdown * feat(query): materialize JSON get result types before planning * fix(query): apply JSON2 type hints to parsed paths * fix(query): apply JSON2 type hints before distributed planning * chore: remove f * refactor: the code style of json_get_type_hint * chore: add sqlness case * fix: add json expr planner, remove json get type hint * chore: add sqlness test * fix(query): respect JSON2 type hints in query planning * chore: add more sqlness cases * fix: cargo clippy * fix: cargo clippy * chore: update sqlness test result |
||
|
dennis zhuang
|
007836ba7a |
fix(prometheus): honor label matchers in __name__ values query (#9134)
* fix(prometheus): honor label matchers in __name__ values query
`/api/v1/label/__name__/values?match[]={pod="abc"}` dropped every matcher
other than `__name__` and returned all metrics in the schema. No error,
just the wrong list. Grafana's metrics browser sends this request, so
picking a label value there did nothing.
Selectors that only constrain `__name__` keep answering from table
metadata. A selector constraining an ordinary label now goes to the data:
scan each metric engine physical table for distinct `__table_id` in the
time range, map the ids back to metric names, then apply the selector's
own `__name__` matchers.
Only metric engine tables are covered; other engines share no column space
to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): batch-resolve metric names by table id
Building a full table-id-to-name map meant walking every table in the
schema and holding all of them in memory, just to name the handful the
scan returned. Use `tables_by_ids` instead — one batch KV read over the
ids the scan actually produced.
The catalog walk stays, but only to find the physical tables to scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): read an absent label as the empty string
A matcher on a label the series does not carry only worked when the table
had no column for it at all. Where the column exists but is NULL on that
row -- the norm for logical metrics sharing a metric engine physical
table, which holds the union of their label columns -- three-valued logic
dropped the row, so `host!="host1"` and `host=""` missed every metric
without a host label.
Coalesce nullable string label columns to "" for matchers that accept the
empty string, rather than only for the OTLP temporality marker. Equality
matchers are untouched; they cannot match NULL either way.
This is the Prometheus compatibility fix #8970 deliberately kept out of
its own scope. The cost is visible in the regex sqlness plan: the
predicate becomes a CASE, so the scan loses its LastRow selector and
grows a FilterExec. Only negative and empty-accepting matchers pay it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): don't panic on pre-epoch label value bounds
`rewrite_label_values_query` unwrapped `duration_since(UNIX_EPOCH)`, which
returns an error for an instant before the epoch. `start=1969-12-31T23:59:59Z`
parses as valid RFC3339, so the request panicked instead of answering.
Recover the sign from the error branch, and report a value beyond i64
milliseconds as an error rather than wrapping the cast.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(prometheus): drop applicable_matchers, share the distinct scan
With the planner reading an absent label as empty, the frontend no longer
needs to pre-filter matchers per physical table. Removing that exposed a
second problem: a physical table that never took a column from a logical
table exposes no `__table_id`, and projecting it failed the whole request.
Skip those tables; the only thing that can miss is a metric with no labels.
Also pulls out the plan-build-execute-collect sequence the two label value
scans had in common.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
 Sainath Singineedi
|
fbbc017be4 |
feat: expose region min/max timestamp in region_statistics (#9060)
* feat: expose region min/max timestamp in region_statistics Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> * test: cover region_statistic time range assembly and projected values Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> --------- Signed-off-by: Sainath Singineedi <44405294+sainad2222@users.noreply.github.com> |
||
 discord9anddiscord9
|
983738101a |
feat(flow): admit mergeable average states in incremental plans (#9235)
Incremental batching flows already merge sink state for scalar aggregates and the HLL/UDDSketch/stddev state families. AVG now exposes a mergeable Binary state (avg_state / avg_merge with __avg_state_delta_merge), so admit those aggregates too instead of forcing a full snapshot for flows whose only state column is an average. merge_op_for_aggregate_expr takes the aggregate input schema so the avg_merge arm can require a Binary state argument; the state form is the aggregate result persisted by the sink, so no extra coercion is needed. Other input types keep being rejected. Also cover avg_state, avg_merge and duplicate AVG projections in the incremental plan analysis tests, extend the mixed state-family rewrite test with an average column, and extend the standalone partitioned state-merge SQLness case with avg_state/avg_merge compared against a direct avg over the source. Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
discord9
|
27090496d1 |
test(flow): stabilize FLUSH_FLOW assertions after async source-table mirrors (#9230)
* test(flow): wait for async source-table mirrors before FLUSH_FLOW Streaming flows execute inline in the flownode insert handler since #8976, and source-table inserts are mirrored from the frontend as detached tasks. Tests that INSERT then ADMIN FLUSH_FLOW then SELECT intermittently miss rows on slow CI runners because the flush no longer implies the mirror reached the flownode (flow_no_aggr lost row 'l', flow_advance_ttl lost row '23' after the post-restart reinsert). Add SQLNESS SLEEP 3s between those mirror inserts and the flush, the established pattern in the flow suite, so the assertions are stable. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): also guard first-section flushes in flow_advance_ttl Local reproduction without sleeps failed in a window the previous commit did not cover: the first INSERT (20,20,22) of each section is followed immediately by ADMIN FLUSH_FLOW, and the distributed run observed an empty sink on that SELECT (~1 in 8 iterations). Add the same SLEEP 3s guard before those two flushes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
09a9d9088d |
feat(otlp): preserve trace v2 events and links as JSON (follow-up to #9192) (#9232)
feat(otlp): preserve trace v2 events and links as JSON Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
dennis zhuang
|
b1106a9c5a |
perf(promql): propagate matching-label filters between binary operands (#9202)
* perf(promql): propagate matching-label filters between binary operands A one-to-one arithmetic binary expression inner-joins its operands on the matching labels, so every row that survives the join already satisfies the other operand's equality matchers on those labels. Copy those matchers to the other operand so both scans drop non-joining series before execution instead of feeding them to the join. Both operands are planned before the rewrite: a selector matcher can also constrain a value field, and only the planned contexts tell tags and fields apart. A matcher is copied only when its name is a tag column on both sides, its value is non-empty, and it is one of the matching labels. Operands are limited to vector selectors, parentheses, and label-preserving range functions applied directly to a matrix selector; `ignoring(...)`, group modifiers, fill values, set and comparison operators, regex matchers and `or` matcher groups are left alone. Selector matchers are now deduplicated in place instead of through a `HashSet`, so the generated scan filter keeps a stable order once a selector carries more than one matcher. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): propagate ignoring, non-equality and aggregated matchers Widens the matcher propagation added in the previous commit to the shapes it was leaving on the table. The join compares matching labels with plain column equality (`normalized_match_key_expr` and its coalescing are confined to `or`), so any predicate on a matching label is already enforced on both sides for every surviving pair: it originates on one operand, and the join carries it to the pairs it forms. Copying it to the other operand can only drop rows that had no surviving partner. That argument does not depend on the matcher kind, so regular expressions, negations and empty values now propagate too. `=~".*"` stays out: it lowers to no filter at all, so copying it would only force a re-plan. `ignoring(...)` is no longer rejected. A label is a join key exactly when it is a tag on both sides and not named in `ignoring`, which is what `binary_join_key_columns` computes and what the caller can now answer from the two planned contexts. Operands may now be aggregations that partition by their grouping labels, which is the shape most real queries use. `agg_modifier_to_col` rewrites `ctx.tag_columns` to the grouping labels, so a label found in an aggregated operand's tags is a group key, and filtering the aggregate's input by it drops exactly the corresponding output groups. `topk`, `bottomk` and `limitk` are excluded because they select across a group and carry input labels through -- `prom_topk_bottomk_to_plan` leaves `ctx.tag_columns` alone, so the tag check cannot catch them. `count_values` is excluded because it adds an output label that does not exist in its input. The rollup whitelist now covers every label-preserving range function the planner implements, and finds the matrix argument by position so multi-argument rollups such as `quantile_over_time` and `predict_linear` qualify. `absent_over_time` stays out: it synthesizes a series from the matchers when its input has none. Re-running the sqlness case against an unmodified planner produces a byte-identical `.result` for all 27 queries. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin that excluded matching labels stay on their own operand `ignoring(device)` and `on(host)` both leave `device` out of the join keys, so a `device` matcher must not reach the other operand. Neither case was covered end to end: a regression there silently drops the left operand's `eth1` series instead of returning them, which the two added queries now catch. Also corrects the comment at the rewrite site, which still described re-planning as touching a leaf selector. Aggregated operands are re-planned as a whole; what holds is that the rewrite only ever adds matchers to a selector, so the enclosing operand's table reference, time index and field columns are unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): log why matching-filter propagation was skipped Splits the guard chain into `try_propagate`, whose `Err` names the reason the rewrite does not apply, and logs it once in `propagate`. Debugging a query that did not get the filter no longer means stepping through the guards. The reasons also replace the comments that used to explain the same conditions, and the remaining comments lose the parts that restated the code or repeated each other. Two of them were wrong rather than verbose. Saying `topk`'s input "must not be filtered" reads as a claim about PromQL: `topk(1, m{host="x"})` is perfectly legal, and what matters is that filtering before `topk` changes the candidate set it ranks. Saying the subset-matching case "keeps its many-to-many result" read as an endorsement of behaviour Prometheus rejects outright; it now records that the query is a cross product here and points at #9209. The propagation cannot change whether such a check would fire: series in one match group agree on every matching label, so a matcher over one of those labels keeps all of them or drops all of them. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep copied matchers out of the operand's selector metadata Re-planning an operand with a copied matcher also rebuilt its `PromPlannerContext::selector_matcher`, and that context is what an enclosing expression reads. `create_absent_plan` turns the equality matchers found there into the labels `absent()` reports, so absent(counter_metric{host="missing"} / on(host, device) gauge_metric) gained a `host="missing"` label that `main` does not produce. The inner expression was empty either way; only the reported label set changed. The copied matcher belongs to the scan, not to the operand's identity, so both re-planned contexts now keep the matchers their operand was written with. `selector_matcher` has one other reader, `create_table_scan_plan`, which consumes it while the operand is being planned and is unaffected. Reported by @discord9. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
fys
|
5b631fd8eb |
feat(json2)!: remove nullable and default type hint options (#9213)
feat(json2): remove nullable and default type hint options |
||
|
discord9anddiscord9
|
5ef8a46e3f |
feat(function): add mergeable binary average states (#9062)
* feat(function): add mergeable binary average states Signed-off-by: discord9 <discord9@163.com> * feat(function): expose avg_calc as an OSS scalar Signed-off-by: discord9 <discord9@163.com> * fix(function): borrow invalid AVG scalar argument type Signed-off-by: discord9 <discord9@163.com> * test(function): verify public AVG state SQL finalization Signed-off-by: discord9 <discord9@163.com> * fix(function): return canonical AVG1 state for empty window frames DataFusion's plain-aggregate window executor bypasses the accumulator and calls default_value() directly when a window frame contains no rows. create_udaf's SimpleAggregateUDF derives default_value from the return type, which yields SQL NULL for Binary output instead of the canonical AVG1 empty state, so empty frames and frames over only NULL inputs became observable differently (IS NULL, direct state saves, state comparisons). Register avg_state/avg_merge through a small AggregateUDFImpl (AvgUdaf) that keeps the existing accumulator and declares the canonical AVG1 empty state as default_value, restoring the documented contract. Add a unit test asserting default_value equals the empty accumulator's evaluate() and an sqlness case covering the empty-frame window. Signed-off-by: discord9 <discord9@outlook.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <discord9@outlook.com> Co-authored-by: discord9 <discord9@outlook.com> |
||
|
discord9
|
8d8ebd3cc5 |
perf: add flight coalesce regression case with high-cardinality aggregations (#9214)
* perf: add flight coalesce regression case with high-cardinality aggregations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf: add flight coalesce aggregations bench case Add a direct_readable_sst case exercising grouped aggregation over coalesced batches: 16 hosts x 4096 instances, 32 SSTs of 32768 rows, timestamp-major series layout, three SQL queries (aggregation, topk, count_by_host) each with a 10% max candidate latency regression threshold. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
697cc5fa29 |
fix(json): fix JSONPath panic with jsonb 0.5.6 (follow-up to #9192) (#9228)
* fix(json): upgrade jsonb to fix unterminated JSONPath panic Signed-off-by: luofucong <luofc@foxmail.com> * fix(json): align integer extraction with JSON2 conversions Signed-off-by: luofucong <luofc@foxmail.com> * style: format jsonb dependency declaration Signed-off-by: luofucong <luofc@foxmail.com> * fix(otlp): report concrete unsupported JSONB type names Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
6fa375021f |
feat(flow): expose extension-owned batching execution hooks (#9171)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: merge aggregate states in incremental flows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to state merge execution tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(flow): expose extension-owned batching execution hooks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): correct extension matcher borrowing and regression assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: prove executed empty exact deltas keep incremental mode Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to empty exact delta regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover aggregate state merge through the full flow path Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: pass query engine to flow batching task regression calls Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
20d87cff48 |
feat(ci): add long-range metrics benchmark on ECS (#9218)
* feat(ci): add long-range metrics benchmark on ECS Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(ci): run warm and lukewarm long-range benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): simplify long-range inputs and increase disk budget Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: run only warm long-range benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
2c531c62ee |
feat(ci): add observability benchmark and lifecycle summaries (#9215)
* fix(ci): authenticate private o11ybench checkout Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(ci): summarize observability queries and lifecycle evidence Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: update observability runtime with timing and evidence fixes Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
e9beb62eef |
perf(promql): avoid concatenating constant series tags (#9108)
* perf(promql): experiment with constant-tag series concat Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): verify logical constant-tag concat equivalence Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): qualify constant-tag series concat Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover fragmented millisecond series concat Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): compact constant dictionary tags at construction Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): construct constant string dictionaries directly Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): benchmark ordinary TQL queries for constant tags Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): address constant-tag review and cardinality coverage Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): scope concat optimization to string dictionaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): move high-cardinality constant-tag cases to the heavy set The 10k and 100k constant-tag direct-SST cases repeatedly kill the self-hosted query-regression runner (lost communication during the run), while the default-cardinality case passes. Move them out of the default 'all' set into the heavy set so they only run on demand (case=heavy or the heavy-regression label), and qualify them locally instead. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
LFC
|
aacf04cf6e |
feat(otlp): add trace v2 ingestion with JSON2 attributes (#9192)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
discord9
|
7c7132ea65 |
refactor(flow): execute streaming flows with DataFusion (#8976)
* test(mito2): cover regex inverted index pruning
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): execute streaming flows with DataFusion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(flow): remove legacy streaming runtime
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): avoid retrying stateless sink inserts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): align stateless writes with sink schema
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): reject stale stateless source schemas
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(flow): validate stateless flow routing
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* Revert "test(mito2): cover regex inverted index pruning"
This reverts commit
|
||
|
Weny Xu
|
604c88e7e2 |
fix(ci): repair agent observability dispatch and runner cleanup (#9204)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 discord9andDennis Zhuang
|
528ceb7733 |
perf(promql): reuse sliding min and max candidates (#9099)
* perf(promql): reuse sliding min and max candidates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): simplify extrema benchmark parameters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): record baseline sliding extrema SQL results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): rescan windows that barely overlap Reusing candidates loses to a plain scan when consecutive windows overlap little: the deque bookkeeping then costs more than the rescan it replaces. A local Criterion run on 4096 samples at width 240 / step 240 measured 10.79 -> 22.14 us for min and 12.83 -> 19.85 us for max. Pick the evaluator once per batch from the first two windows. RangeManipulate emits one window length and one step per batch, so that sample decides for all of them, and both evaluators return identical bits, so a wrong pick costs time only. Batches that do not qualify fold each window on its own. Move the incremental state into SlidingExtrema so tests can drive it directly: the exhaustive four-sample differential test cannot reach it through a UDF call, because such a batch never qualifies for reuse. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * fix(promql): select the extrema evaluator from batch averages Reading the window shape off the first two windows misreads the batch. RangeManipulate starts a series at max(query start, first aligned sample), so a series that begins inside the query range gets a first window covering roughly one step, and a window covering no sample at all is emitted as (0, 0). Either one closed the gate for the whole batch, including the one-hour window at a 15s step that candidate reuse was written for. Compare the batch averages instead: at least 32 samples per window, and a step advancing at most a quarter of that. Uniform batches select exactly as before, so the thresholds keep the meaning they were measured with. The 32-sample rule had also moved most of the benchmark and query-regression shapes onto the rescan, including the case built to measure reset and rebuild. Widen those windows to 40 samples, add a step at the selection boundary, and add an end-to-end case with 40-sample windows advancing 5. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * fix(promql): ignore empty windows when measuring batch advance The advance was read from the first and last window offsets, but a window covering no sample is emitted as (0, 0). A query whose last evaluation lands exactly one window past the last sample ends on such a window, and its zero offset made a batch of disjoint windows look like one that never moved, which selected the evaluator built for overlap. Results stayed correct; the cost was deque bookkeeping on the shape the scan fallback exists for. Take the offset span over the windows that cover a sample. Empty windows stay in the window count, where they only make both conditions stricter. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> Co-authored-by: Dennis Zhuang <xzhuang@greptime.com> |
||
|
Weny Xu
|
4ecec69bee |
ci: add manual agent observability benchmarks on Aliyun ECS (#9179)
* ci: add manual agent observability benchmarks on Aliyun ECS Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: configure observability ECS budgets and reuse runner actions Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: default observability runners to ecs.c9i.2xlarge Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): prepare observability Docker access during ECS bootstrap Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: remove standalone observability CI guide Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
f5428f8a6a |
test: regenerate expired TLS certificates for integration fixtures (#9196)
* test: regenerate expired TLS certificates for integration fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore root.srl serial file for TLS fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(ci): update compatibility test window to v1.2.1 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add proper TLS extensions to regenerated integration certificates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
d185a8bf76 |
feat: use exact sequence ranges for incremental Flow reads (#9165)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: use exact sequence ranges for capable incremental flow sources Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: format exact sequence recovery tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: exercise dispatched exact delta failure recovery Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: record flushed exact sequence flow results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dhruv Vaishnav
|
7e2a75f771 |
feat: allow re-enabling WAL after disabling (#9130)
* feat: allow re-enabling WAL after disabling Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: regenerate skip_wal sqlness result The previous commit changed tests/cases/standalone/common/skip_wal.sql without regenerating the matching .result, so every Sqlness suite failed on the mismatch. tests/cases/distributed/common is a symlink to standalone/common, so the single stale file accounted for all five failing variants. Regenerated from a real run. The recorded output now covers the cases the .sql added: * A table created with skip_wal = 'true' has no real WAL provider, so enabling WAL is refused with Unsupported rather than the previous InvalidArguments. * A RaftEngine-backed table accepts the true -> false transition, and repeating it is a no-op. * SHOW CREATE TABLE reports skip_wal = 'false' after the transition, including after a restart. * A row written while WAL was skipped is absent after restart, while a row written after WAL was restored survives. Verified locally with nothing else running on the machine: sqlness skip_wal passes in both the standalone and distributed environments, and the regenerated file is byte-identical to the output of an independent earlier run. Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test: strengthen skip_wal provider coverage Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Fix WAL provider handling during re-enable Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * Refactor WAL provider check Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> |
||
|
discord9
|
8af3a04ed7 |
fix: preserve structured query errors through distributed execution (#9161)
* fix: preserve structured query errors through distributed execution Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update SQL expectations for preserved query error codes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
LFC
|
9f6a79da30 |
fix: support native JSON2 row inserts over gRPC (#9145)
* fix: support native JSON2 row inserts over gRPC Signed-off-by: luofucong <luofc@foxmail.com> * test: cover unknown JSON2 schema compatibility Signed-off-by: luofucong <luofc@foxmail.com> --------- Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
dennis zhuang
|
7cbd20a053 |
fix(mysql): strip leading comments before the federated statement filter (#9156)
* fix(mysql): strip leading comments before the federated statement filter JDBC clients prefix every statement with a comment. DataGrip sends `/* ApplicationName=DataGrip <version> */` in front of each one, and every pattern in the federated filter is anchored with `^`, so the prefix makes all of them miss. Two failures follow. `SET TRANSACTION READ WRITE` reaches the SQL parser and is rejected. Worse, the DBeaver-specific entries such as `^(/\* ApplicationName=(.*)SELECT @@(.*))` do match DataGrip's prefix, so `SELECT @@GLOBAL.event_scheduler` and `SHOW VARIABLES LIKE ...` are absorbed into a zero-column output, which the MySQL writer sends as an OK packet. The JDBC driver reports that the statement returned no cursor. Strip leading whitespace and comments before matching, and drop the ApplicationName-specific entries the stripping makes redundant. The three that had no unprefixed counterpart (`SHOW PLUGINS`, `SHOW ENGINES`, `SHOW @@...`) keep their behaviour as plain patterns. Also covers gaps found while probing the same path: - `BEGIN` is absorbed like `START TRANSACTION`/`COMMIT`/`ROLLBACK`. - `SHOW [GLOBAL|SESSION|LOCAL] VARIABLES|STATUS` parses; the scope is ignored because GreptimeDB keeps no global/session split. - `USER()`, `CURRENT_USER()`, `SYSTEM_USER()` and `SCHEMA()` are registered. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(mysql): keep executable comments, reject multi-statement absorption Review follow-up on the comment stripping, plus the remaining MySQL compatibility gaps from #9155. `/*!...*/` is an executable comment: mysqldump emits its initialization as `/*!40101 SET NAMES ... */`, and the patterns match those verbatim. Stripping it left an empty statement, so nothing matched and the original SQL reached the parser, which rejects `SetNames` and `MultipleAssignments`. Leave executable comments in place. Absorbing a request also has to stop at a statement boundary, because every pattern ends in `(.*)`. `BEGIN; INSERT INTO t VALUES (1)` used to fail on the unsupported `BEGIN`; once `BEGIN` became absorbable the whole request would report success and write nothing. A request is now scanned for a second statement and handed to the query engine if it has one. The scan skips plain comments and string literals so a `;` inside either is not a boundary, and treats a `/*!...*/` that is not the request itself as a statement, since it carries SQL. `check()` now dispatches on the leading statement keyword, so INSERT, UPDATE, CREATE and ordinary SELECTs run no regex at all — this replaces the narrower hand-rolled INSERT shortcut. The statement scan runs only for a request the patterns already matched, which is always a short one. New compatibility surface: - `SELECT CURRENT_USER|SESSION_USER|SYSTEM_USER|USER` without parentheses, and `SELECT @var`, are answered here rather than in the parser. Both are anchored to the whole statement, so `SELECT user FROM t` still reads the column. - `information_schema.plugins`, `user_privileges` and `processlist` are registered as empty tables, like the other MySQL-shape tables around them. Sessions are reported through `information_schema.process_list` and `SHOW PROCESSLIST`; `processlist` carries the column shape only. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
ba0f7acd93 |
feat(mito2): add opt-in byte-stream-split encoding for float SST fields (#9069)
* feat(mito2): add opt-in byte stream split encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): correct float encoding checks Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): cover float SST encoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile float encoding tests Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): release parquet test writer Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): register float test primary key Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify BSS write lifecycles Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(metric-engine): verify BSS physical SST Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): verify bulk BSS lifecycle Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(mito2): compile bulk BSS test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding constructors Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float upgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(compat): accept generated float downgrade output Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(mito2): narrow bulk encoding builder Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add default versus BSS storage comparison Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): align BSS reader benchmarks with prior study Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): parse current read benchmark averages Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): retain default float encoding in direct SST fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): isolate BSS user SSTs and benchmark every file Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): record measured BSS storage and reader tradeoffs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): expose warm scan variability and evidence limits Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): clarify BSS baseline and storage measurement scope Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): model bounded mixed integer and fractional metric series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): report bounded mixed BSS measurements and query regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(perf): qualify timings affected by concurrent host builds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): include float BSS comparison in default regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(perf): omit unsupported float encoding option from baseline setup Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
7152ca9264 |
fix(promql): skip NULL samples and fix counter extrapolation order (#9118)
* fix(promql): treat NULL field values as absent samples in range functions Range functions read the value column through `Float64Array::values()`, which returns the raw buffer and ignores the null bitmap. A NULL field value means the series has no sample at that timestamp, so the padding under a null slot (0.0 in practice) was counted as a real sample. `rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`, `quantile_over_time` and `avg_over_time` now work on samples instead of slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL slot, and tokio swallowed the panic so the query returned success with that series missing. Null handling is gated on `null_count() == 0` over the whole backing array, checked once per batch, so tables without NULLs keep the existing code path. `deriv` and `predict_linear` already had this guard through `linear_regression_slices` and are untouched. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): keep other fields when one has no sample in the window Review follow-up. Two things surfaced once range functions started returning NULL for a window without samples. The filter after a function call required every field column to be non-NULL, so on a multi-field table one field with no samples in a window would drop the other fields' results with it. It now keeps a row when any field has a sample, which is the shape a selector already emits. On a single field column the two predicates are identical. `quantile_over_time` returned NaN rather than NULL for a window without samples, so the row survived that filter. Prometheus returns an empty vector there, so the emptiness check now sits in the range UDF; the shared quantile kernel still yields NaN for an empty slice, matching upstream's `quantile()` helper that `quantile_aggr` depends on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(promql): correct two comments on the null handling The planner one did not say why `preserve_any_value` is hardcoded at that call site, which is the question a reader arrives with. The `quantile_over_time` one described the empty-window behaviour while sitting on the `has_nulls` line, and that behaviour had moved into `window_quantile`. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): clamp extrapolation before snapping a counter to zero Prometheus clamps `durationToStart` to half an average interval once the first sample is past the extrapolation threshold, and only then lets the counter zero-snap shorten it further, so the snap can never lengthen the leading extrapolation. Running the snap first let it rescue a duration the clamp should have cut, and `rate` and `increase` over-extrapolated to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream gives 0.375 and this returned 0.5. `extrapolation_matches_prometheus_on_seeded_windows` carries a line-by-line port of upstream `extrapolatedRate` as an oracle and diffs it against the UDF over seeded windows, so the order stays pinned. `factor` also picked up upstream's guard against a zero sampled interval, which previously divided by zero. Two sqlness results move, both verified against the upstream algorithm. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): fold range_presence_null into null_samples #9104 landed its own NULL-sample case whose data is the same series this one already used: one host with interior NULLs, one host with nothing but NULLs. Keeping both means two files asserting the same semantics on the same rows. The merged case keeps every query from both, so the presence functions still cover the trailing-NULL window, the count of two, the empty left-open window at t=7, and the all-NULL windows. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
7f949f48c0 |
fix(promql): preserve native timestamps through sample selection (#9070)
* fix(promql): preserve native timestamps through sample selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): retain column indices in instant plan ordering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): update native precision plan expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): preserve selector output column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): verify preserved selector output order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): refresh native timestamp explain expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: apply PromQL offsets without native timestamp overflow Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover negative PromQL offsets at native timestamp bounds Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat: allow native precision instant LastRow selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: discard unused bounds for empty range intersections Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve native time bounds independently for LastRow Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify native LastRow predicates and overflow through SQL Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: explain native PromQL selection and scan invariants Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): address timestamp helper and stream review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): pass selector offsets explicitly from planner Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): retain explicit offset in payload overflow regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): record inner-offset subquery SQL results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Lei, HUANG
|
743261f05e |
fix(mito2): prevent JSON2 SWCS data loss from misaligned Parquet statistics due to projection (#9129)
* fix(mito2): look up row group stats by parquet leaf index for nested columns On flat-format tables a logical column can expand to multiple parquet leaf columns (e.g. a JSON2 struct stores the remainder and one leaf per promoted path). ParquetFlat used the logical column index in the SST schema directly as the leaf index when reading row group statistics, so min/max/null stats of every column after a nested column were read from wrong leaves. When the misplaced leaf held order-compatible statistics (e.g. a small Int64 JSON path vs. the timestamp window predicate), min-max pruning dropped whole row groups by mistake. SWCS compaction reads inputs with a time window predicate, so it silently lost all rows of such files; plain queries with time-range predicates were affected as well. Map each column to its first parquet leaf column and report NoStats for columns with multiple leaves, which makes pruning conservative for them. Add a unit test and a sqlness regression case that reproduces the data loss on the unfixed binary. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): skip nested root stats and correct SWCS regression baseline Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): resolve statistics leaves for primary-key SST readers Resolve scalar roots against the actual Parquet schema in shared statistics helpers, covering both flat and primary-key readers. Remove flat-side translation to avoid mapping twice and align encoded primary-key statistics as well. Cover dense flat, legacy dense and sparse layouts with statistics and time-pruning regressions. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): distinguish known null counts from unknown statistics Assert validity before reading timestamp null counts and add a nullable scalar after the nested root with a known nonzero count. Exercise the assertions for flat and primary-key SST layouts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): check JSON2 time pruning before SWCS compaction Query the first time window immediately after FLUSH to cover predicate reads on flush-written SSTs independently of compaction outputs. Regenerate the sqlness expectation and retain the post-compaction checks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): validate all business columns after JSON2 SWCS Expand the final regression query to all eight business columns so the generated expectation verifies complete rows, including tags and scalar fields, after repeated compaction. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
555485c40e |
perf(promql): avoid per-window allocations in simple range functions (#9104)
* perf(promql): avoid per-window allocations in simple range functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): avoid copying smoothing window values Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover smoothing copy removal across window layouts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip null samples in simple range functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
97648525cf |
perf(mito2): skip proven all-match prefilters (#9066)
* perf(mito2): skip proven all-match prefilters Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): add manual all-match prefilter reproduction Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): cover all-match prefilter execution paths Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(mito2): match prefilter fixture to sparse SST schema Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(mito2): address all-match prefilter lint findings Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(perf): cover all-match prefilters in default regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
a673e084b2 |
test: cover request-level insert WAL skipping end to end (#9093)
* test: cover request-level WAL skipping end to end Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover session WAL policy and COPY recovery in sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(test): isolate Mito test feature in dev dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> * test: parameterize WAL protocol cases and make setup explicit Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover skip-WAL hints across streaming messages Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
5d5f0d6d70 |
fix(query): prevent incomplete aggregate dynamic filtering (#9102)
* fix(query): prevent incomplete aggregate dynamic filtering Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): cover mixed expression and column maxima Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): pin upstream aggregate regression backport Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): assert aggregate filter behavior on datanode scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): pin merged aggregate dynamic filter fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
fys
|
536f42e4a2 |
feat(json2): support ALTER syntax for JSON column settings (#9094)
feat(json2): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> This is the commit message #3: |
||
|
dennis zhuang
|
0f625a7e92 |
fix(promql): correct counter reset accumulation in rate windows (#9089)
* fix(promql): correct counter reset accumulation in rate windows `prom_rate` and `prom_increase` reused the previous window's counter-reset correction when the next window slid forward by exactly one sample, adding the entering reset and subtracting the leaving one. Running a sum through addition and subtraction does not restore the earlier terms in f64: a large reset absorbs the smaller ones that must survive it, and an expired infinity leaves a NaN that no later window can clear. `prom_delta` shares the code but is not a counter function, so it never took that path. Index the reset positions of the value array once instead, and reduce each window over the resets it contains, in sample order. The result is bit-identical to scanning the window directly, so windows keep the direct reduction when they request fewer sample pairs than the input has. Also sweep the query step in the rate benchmarks: the cost of the reset correction depends on how much the windows overlap, which no existing case varied. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(promql): cover counter reset precision over adjacent rate windows The unit tests build the range windows directly, so they do not show that a plain PromQL range query produces the window layout that lost the correction. This case does: with a query step equal to the sample interval, `increase` over the second window returns 1.333 before the fix and 2.667 after it. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * perf(promql): advance the counter reset bounds instead of searching Locating a window's resets with two binary searches costs more than the reduction it replaces once a series resets often enough for the searches to get deep: on a 20k-sample counter resetting every 37 samples, stepping the windows by one sample was 2.7x slower than the previous code, against 1.2x for a counter that never resets. Windows normally advance, so walk the bounds forward from the previous window and only search when they move back. The cost then no longer depends on the reset density. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * perf(promql): cut the per-window cost of the counter reset index Two costs the index added showed up on a one-sample query step, where the removed fast path used to answer each window with two comparisons. Cache the two reset positions that bound the active slice. A window that only advanced and reached neither of them covers the same resets as the previous one, so the common case is four integer comparisons and no lookup at all. Stop summing the requested sample pairs once they exceed one pass over the values. The sum only decides which side of that comparison the input falls on, and a query with a short lookback and a long step settles it after a few windows instead of after every key. Together these take the one-sample step from 25-32% slower than the previous code down to 6-11%, measured as before / after / before to bound drift. No other step value regresses, and a ten-sample step stays about 88% faster. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * fix(promql): accumulate counter resets into the running result `prom_rate` and `prom_increase` summed a window's counter-reset corrections on their own and added that sum to `last - first`. Prometheus folds each reset into the running result instead, and so did this code before #7880. The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against the first sample and reports no increase at all, where folding the resets in one at a time keeps the 1.0. Restore the original order. The reset index accumulates into the result the same way, so it still matches a direct scan of the window bit for bit, but a window's contribution can no longer be cached as a standalone value and is re-added from its own difference each time. The bounds are still cached, so a window that did not cross a reset skips the lookup, and one that holds no resets returns without touching the index at all. Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> * test(promql): note which reset boundaries the stride of one walks Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> --------- Signed-off-by: Dennis Zhuang <xzhuang@greptime.com> |