* fix: only push down aggregates grouped by the partition columns themselves
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: keep grouping sets on the frontend for partitioned tables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(mito): limit series index disk usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): enforce series index quota during reconciliation
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): remove series index disk budget layer
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): simplify series index limit to estimated usage
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): trust index catalogs when loading snapshots
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito): minimize series index disk limit changes
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): keep series index cleanup running at capacity
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(mito2): avoid no-op index clones and stabilize capacity tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: update config API expectation and stabilize index build tests
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(promql): derive vector matching result labels and reject ambiguous matchings
A vector-vector binary operation projected one operand's whole tag set and
inner-joined without any cardinality check, so `on()`/`ignoring()` did not
reduce the result labels, `group_left`/`group_right` changed nothing, and a
non-unique match group produced a cross product that PromQL cannot represent.
Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the
matching labels, `ignoring(...)` drops them, and a group modifier keeps the
many side's labels plus the `group_x(...)` labels taken from the one side.
A label the one side does not carry is deleted from the result. The reduced
label set no longer identifies the operand series, so `__tsid` is dropped
from the context on this path.
Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)`
window and a scalar UDF that fails the query on a repeated group: on the one
side before the join, and on the result labels after it, matching where
Prometheus raises each of its three errors. Series are unique by their whole
tag set, so the window is only planted when the match keys drop a tag; plain
arithmetic and `on(<all tags>)` plan exactly as before.
Closes#9207, closes#9208, closes#9209.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): keep __tsid when the result labels are an operand's whole tag set
Deriving the result labels dropped `__tsid` from the context unconditionally,
so an enclosing operation fell back to joining on the tag columns even where
the column still identified the result series.
Keep it when every result label comes from one operand and covers that
operand's whole tag set: no other operand value reaches the labels, and the
matching gives each of its rows a single partner, so its `__tsid` is still one
per result series. That is the common `on(<all tags>)` and bare `group_left`
shape; a matching that actually drops a tag still clears it.
The column is re-qualified as the result's own, which is how the enclosing
expression and the context look it up.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep the match group count column unambiguous
The cardinality check aliased its row count to a fixed `__promql_match_group_count`.
An operand carrying a label of that name made the window output two fields with
the same name, and planning failed with "Schema contains qualified field name
collide_right.__promql_match_group_count and unqualified field name
__promql_match_group_count which would be ambiguous".
Pick a name the operand does not already have, the way the `or` operator
allocates its match key columns.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover a match group spread over several regions
The cardinality check runs above the merge of the region scans, so it counts a
match group globally. Nothing pinned that: every table in these cases holds a
single region, and a check evaluated per region would pass them all.
Partition the operand on a column outside the match keys, which puts the two
series of one match group in different regions, and assert both the pre-join
and the post-join check still reject it. The case runs in the distributed
environment too, where the regions sit on different datanodes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): group an outer aggregate on the labels the operand kept
`by`/`without` planning topped up a missing grouping column by walking down to
the table scan and re-projecting it. That is right for a column the scan pruned
for efficiency, but the labels a matching modifier deletes are also absent from
the operand's output, and they were restored the same way:
sum without(host) (a / on(host) b)
`on(host)` leaves the operand with `host` alone, so the sum covers everything
and Prometheus answers `{} 10`. Instead `device` came back from the scan under
`a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for
`sum by(device)` of that operand, which has no `device` to group on at all.
Take the grouping labels from the operand's own label set rather than from the
row keys of the scan beneath it. A label pruned from the plan is still in that
set and still gets restored; a label the operand dropped is not.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): drop the unreachable aggregation tag top-up
`by`/`without` planning could restore a grouping column that the plan no longer
carried by rewriting the scan underneath it. Once the grouping labels come from
the operand's own label set, there is nothing left for it to restore: a scan
projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds
matcher columns to that set), so a label in the set is always in the schema.
Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the
standalone and the distributed environment.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): drop cases that guarded the removed tag top-up
Three plain selector aggregates were there to show that restoring a pruned
grouping column still worked. With the restore gone they only repeat what the
aggregate cases already cover. Also fix a comment that still said the metric
engine scan prunes tag columns: it projects every label of the operand.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(promql): note the duplicate a propagated matcher hides
A matcher copied onto the one-side operand removes groups without a partner
before the cardinality check sees them, so a duplicate in such a group is not
reported. Prometheus checks every group of the one side and fails the query.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: use relative object keys for Windows filesystem access
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: cover Windows path and time limits in full test CI
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: use relative keys in metadata snapshot tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* refactor(flow): expose incremental aggregate plan analysis
Signed-off-by: discord9 <discord9@163.com>
* feat(flow): allow execution hook to rewrite completed plans
Signed-off-by: discord9 <discord9@163.com>
* test(flow): cover the execution hook receiving the completed plan
The hook must see the plan that is dispatched after the incremental
delta-sink merge, so the test records the plan a collaborator is handed and
asserts the merge is already part of it.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* feat: serialize struct to json in postgres
* fix: support view scalars and preserve null structs in scalar-to-value conversion
Address PR review:
- Utf8View/BinaryView ScalarValues now convert like their non-view forms
instead of failing row extraction for struct columns
- a null struct scalar converts to Value::Null so a null struct inside a
list stays null in the serialized JSON
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: return errors instead of panics for unsupported arrow field types
Struct-typed query results with arrow field types greptimedb cannot
represent (e.g. Decimal256) used to panic during schema conversion and
row extraction, dropping the client connection. They now surface as
query errors:
- ConcreteDataType::try_from builds struct types fallibly via the new
StructType::try_from_arrow_fields
- Value::try_from(ScalarValue::Struct) uses the same fallible path
- new try_value_from_array converts an arrow element to Value with
error propagation, used by the postgres struct encoding
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* test: cut integration test time and make the storage matrix meaningful
tests-integration is ~85% of workspace test CPU, and 81% of that is the
S3/S3WithCache variants of the HTTP and gRPC suites. Those suites do not
touch the object store: of the 70 matrix HTTP tests only one flushed and
read back an SST, so the matrix was paying real AWS round trips to
re-prove protocol parsing.
- Point the PR CI object-store matrix at the MinIO already started by
tests-integration/fixtures. Three GT_S3_* consumers did not read
GT_S3_ENDPOINT_URL and would have hit real AWS with MinIO credentials;
they now do.
- Add a nightly Linux job against real AWS S3, and pass GT_S3_* into the
release integration-test container. The release previously ran every
remote-backend case as a skip and only exercised the file backend.
- Give each S3WithCache test its own read cache directory. They shared
/tmp/greptimedb_cache, which the datanode wipes on startup, so a
starting test deleted the read cache of a running one.
- Add flush -> read-back assertions to the tests whose columns have a
non-trivial SST representation: JSON/JSON2 columns, native histograms,
metric-engine logical tables, and tables carrying fulltext or skipping
indexes whose puffin files only exist after a flush.
- Move eight tests that create no table out of the storage matrix.
- Make the event recorder flush interval a constructor parameter and
shorten it in the event tests, which otherwise wait a 5s window per DDL
they assert on. It is skipped by serde and never reaches config files.
- Drop duplicates: test_grpc_zstd_compression was a verbatim copy of
test_grpc_message_size_ok and is now rewritten to assert the negotiated
grpc-encoding; test_execute_copy_to_{s3,oss,gcs,azblob} were strict
prefixes of their copy_from siblings; two standalone/distributed event
test pairs shared one assertion body.
- Fix and un-ignore stddev_by_label. stddev_pop merges partial aggregates
in a parallelism-dependent order, so its last digits are unstable; the
test now compares values with a tolerance.
- Rebase the jaeger v1 fixture on the current instant. It carries
ttl=7d with 2025 timestamps, so its rows were only readable as long as
they stayed in the memtable.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: address review — wire nightly real-S3 job into check-status, keep the short event interval
The nightly `check-status` job did not depend on the new real-S3 job, so a
failure there would not have reached the status or Slack notification.
In database_ddl_event the short interval was set by a first
`with_event_recorder_options` call and then overwritten by the pre-existing
one, which carries `..Default::default()`. Merged into a single call.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: update cargo fuzz command to use nightly toolchain explicitly
* ci: honor RUSTUP_TOOLCHAIN pin in fuzz orchestration script
An explicit `+toolchain` argument overrides the RUSTUP_TOOLCHAIN env var
in rustup precedence, so the hard-coded `cargo +nightly` in
run-fuzz-targets.sh bypassed the pinned FUZZ_RUST_TOOLCHAIN
(nightly-2026-03-21) configured in the workflow.
- Invoke `cargo +"${RUSTUP_TOOLCHAIN:-nightly}" fuzz run` in the script
so CI uses the pinned toolchain and local runs fall back to the
floating nightly
- Pass RUSTUP_TOOLCHAIN through to all four fuzz-test action invocations,
covering the no-prebuilt-binaries path and making the reproduce command
in the summary print the exact pinned toolchain
- Add test_rustup_toolchain_env_is_honored covering the pinned-env
scenario for both the cargo invocation args and the summary text
Addresses #9298 (review).
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(query): stop encoding oversized dynamic filters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(query): stop bounded encoding partway through an IN list
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add experimental Jev SQL filtering
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: gate Jev filtering behind an opt-in Cargo feature
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: verify Jev registration with default features
Run the existing registry regression without the jev feature in both PR tests and merge-queue coverage, alongside the existing feature-enabled test runs.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: clarify Jev concurrency scope and stabilization work
Document the per-expression/batch concurrency bound and track process-wide limiting, rate-limit backoff, and request budgets and metrics as stabilization prerequisites.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: enable renamed ai-functions feature by default
Rename the Jev Cargo feature across the command, query, and function crates and enable it in their defaults. Update CI and documentation, retaining an isolated no-default-features registry check and the runtime API opt-in.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* ci: remove extra AI feature-off checks
Use the regular AI-enabled unit and coverage runs for the default feature configuration. Keep feature-off validation available locally and update the usage guide to match.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: rename AI feature to ai_functions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: reject duplicate region engine configurations
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: reuse canonical engine names in config validation
Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: clarify zero-based duplicate engine config indices
State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(telemetry): remove the shared lock from TraceLayer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(telemetry): stop collecting span data while tracing is disabled
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep value-field grouping labels out of matching filter propagation
#9202 propagates matching-label matchers to the scanned selector using the
planned operands' tag columns. For an aggregated operand those are its
grouping labels, and agg_modifier_to_col resolves by(...) names against the
input schema only, so a value field named in by(...) is reported as a tag.
A value field varies between the samples of one series, so lowering a
matcher on it below sample selection (PromInstantManipulate) can drop the
newest sample, promote a stale one from the lookback window, and fabricate
a match the un-rewritten query does not produce.
Track the by(...) labels that name value fields of the aggregated operand's
input in PromPlannerContext::aggregation_field_labels, and refuse to
propagate matchers on them while still requiring a matcher to name a
grouping label to cross an aggregation.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test: sync matching_filter result fixture comment with the PR number
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(servers): stream Prometheus HTTP responses incrementally
Replace try_collect + full-buffer conversion in the Stream branch of
from_query_result with incremental per-batch merging via the shared
merge_batch path, so record batches are dropped as soon as they are
consumed instead of all coexisting in memory.
- Add ColumnLayout to dedupe schema inference shared by both paths
- Add SeriesKeyLookup (Equivalent-based borrow lookup) so series keys
are looked up by borrowed label slices without materializing owned
keys per lookup
- Empty streams short-circuit to empty data before schema inference
- Add stream-side mixed histogram + Vector dual-path equivalence test
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): address review comments on stream response tests
- Inline single-caller tags_capacity helper
- Drop timestamp column from empty-stream test so it pins the
short-circuit happens before schema inference
- Fix cross-batch merge test data so a series actually spans the
batch boundary (second batch carries earlier timestamps)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(servers): add lazy stream release assertion and pinned cross-batch content
- LazyDropCheckingStream creates one batch per poll and asserts via
Weak that the previous batch was released before the next poll,
protecting the core property that batches are dropped eagerly
- Pin series "a" merged samples to exact timestamps and values in
stream_result_matches_record_batches_result, so a regression in
shared merge_batch logic cannot make both paths agree on wrong output
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Rebase the GreptimeDB DataFusion fork from official 55.0.0 to 55.1.0.
DataFusion 55.1.0 is a patch release on branch-55 containing eleven
cherry-picked fixes (schema-adaptation struct filters, cast/projection
metadata propagation, nested-nullability aggregation adaptation,
UnnestExec batch_size, RightMark join ordering panic, empty-struct
ScalarValue, and FFI codec fixes). All twenty GreptimeDB fork patches
rebase onto it with no textual or semantic overlap; none of them is
absorbed upstream, so all are retained.
Fork pin moves to discord9/datafusion branch greptimedb-55.1.0,
commit 2aa87d52cdce7006af492330064738f33ed294c1 (55.1.0 +
20 patches).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* fix(query): insert MergeScan into nested scalar subqueries
DataFusion 55 keeps uncorrelated scalar subqueries as expression
subqueries (enable_physical_uncorrelated_scalar_subquery, default true)
instead of decorrelating them into joins, and executes them via the new
physical ScalarSubqueryExec. DistPlannerAnalyzer::try_push_down walked
the plan with a plain TreeNode transform that does not descend into
expression subqueries, so MergeScan was only inserted for depth-1
subqueries. A scalar subquery nested inside another scalar subquery kept
a bare frontend DistTable TableScan and failed at execution with
"Unsupported operation: get stream from a distributed table".
Use the subquery-aware transform so handle_subquery (PlanRewriter /
MergeScan insertion) runs for subquery plans at every nesting depth.
Fixes#9260.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* test(query): strengthen nested scalar subquery regression coverage
Address review on #9261:
- Replace the ineffective 'no bare TableScan' string check with a real
subquery-aware plan walk (apply_with_subqueries); MergeScan hides its
remote input from traversal, so any TableScan the walk reaches was
genuinely left unwrapped.
- Add a distributed regression case on a range-partitioned table so the
nested inner aggregate must merge partial results across regions
(global AVG feeding an outer SUM filter).
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
dennis zhuang
Yingwen
Weny Xu
shuiyisong
discord9
fys
jeremyhi
LFC
