* fix(ci): teach check-builder-rust-version.sh to handle stable channels
The script extracted a YYYY-MM-DD date from rust-toolchain.toml to
compare against the rustc build date inside the dev-builder image —
a nightly-era design. With channel = "1.96.1" there is no date in
the file, so every release build failed with 'Error: No rust toolchain
version found in rust-toolchain.toml'.
Extract the channel token instead and branch on it:
- stable channel (X.Y[.Z]): require the builder image's rustc to
exactly match the pinned version
- nightly-YYYY-MM-DD: keep the legacy date-difference check
Verified against a mocked docker for all four paths (stable match /
mismatch, nightly fresh / stale).
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore(toolchain): finish stable-migration cleanup in docs and query-regression pin
- README/AGENTS: the toolchain is now stable Rust pinned by
rust-toolchain.toml, not nightly
- query-regression: align the benchmark toolchain pin with the
workspace (nightly-2026-03-21 = 1.96.0-nightly -> stable 1.96.1),
including the exact-version assertions (cargo 356927216, rustc
31fca3adb, both 2026-06-26) and the runner image default
The query-regression runner image must be rebuilt and
QUERY_REGRESSION_ECS_IMAGE_ID bumped together with these pins
(per .github/runner-scale-sets/query-regression/README.md) before
the next regression run.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the Rust toolchain pin from rust-toolchain.toml
Replace the hard-coded RUSTUP_TOOLCHAIN value and the hard-coded
version strings in the runner Verify assertions with a pin resolved
from rust-toolchain.toml:
- the always-running test-tooling job exports the channel parsed from
rust-toolchain.toml as a job output
- query-regression sets RUSTUP_TOOLCHAIN from that output
- the Verify step escapes the pin into the cargo/rustc/active-toolchain
regexes at runtime; the exact commit hash and date are asserted
generically since a stable version identifies the release
Removing the redundant require_eq (workflow yaml vs runner env) since
both now flow from the single source of truth. When rust-toolchain.toml
is bumped, the run fails with a clear signal until the runner image is
rebuilt with the new toolchain, keeping the existing lockstep contract.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the runner image toolchain from rust-toolchain.toml
Remove the hard-coded 'ARG RUST_TOOLCHAIN=1.96.1' from the
query-regression runner Dockerfile. The pin is now parsed from a
COPY'd rust-toolchain.toml at build time (the bootstrap script builds
with the repo root as context, so the file is in the build context):
- rustup-init installs the parsed channel as the default toolchain
- the baked ENV RUSTUP_TOOLCHAIN is dropped: the rustup default makes
bare cargo/rustc resolve correctly without it, and the workflow
supplies RUSTUP_TOOLCHAIN explicitly at run time
- the build-time self-verification asserts the active toolchain
against the same parsed pin
With this, rust-toolchain.toml is the single source of truth for the
benchmark toolchain end to end: the image bakes whatever the toml says
at build time and the workflow asserts against the toml at run time.
A toolchain bump now only requires rebuilding the image.
The changed mechanics were verified natively with the real rustup-init
1.29.0 and the real 1.96.1 toolchain (registry pulls are unavailable
in this sandbox): parsing, default-toolchain installation without
RUSTUP_TOOLCHAIN env, bare cargo/rustc resolution, and the
active-toolchain assertion for both '(default)' and '(overridden)'
output forms.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): rebuild the runner image automatically on toolchain changes
Mirror the dev-builder automation for the query-regression ECS runner
image: a new rebuild-query-regression-runner-image.yaml workflow runs
whenever rust-toolchain.toml or the query-regression runner directory
changes on main (or via manual dispatch). It drives the existing
build-ecs-image.py ops tool, then completes the documented lockstep
updates in order: bump RUNNER_IMAGE_EPOCH in query-regression.yml and
push the commit to main, and only then point the
QUERY_REGRESSION_ECS_IMAGE_ID repo variable at the new image, so the
next regression run picks up image and epoch together.
Also fix build-ecs-image.py to stage rust-toolchain.toml into the
temporary docker build context: the AMI path builds the embedded
Dockerfile from an empty /tmp/image-context, which would break on the
Dockerfile's COPY of rust-toolchain.toml introduced earlier. The
user-data now base64-stages the toml next to the Dockerfile before
docker build.
Verified: py_compile, render_user_data round-trip (mkdir -> stage ->
docker build ordering), and the RUNNER_IMAGE_EPOCH bump sed against
the real workflow file. Requires a new ALIYUN_ECS_BASE_IMAGE_ID repo
variable (Ubuntu 24.04 public image id in the region); all other
secrets/vars are shared with the provisioning job.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: fold the query-regression runner rebuild into release-dev-builder-images.yaml
Merge the standalone rebuild workflow into the existing builder-image
release workflow, as one entry point for all builder artifacts:
- push paths extended with .github/runner-scale-sets/query-regression/**
- new 'release_query_regression_runner_image' dispatch input
- a 'changes' job diffs the pushed range (github.event.before..sha,
with an everything-changed fallback for dispatch or unknown bases)
so each expensive rebuild only fires for its own paths:
rust-toolchain.toml gates both, docker/dev-builder/** gates the
dev-builder images, the query-regression runner directory gates the
ECS image rebuild
- the rebuild job itself is unchanged from the standalone workflow
(build-ecs-image.py, then RUNNER_IMAGE_EPOCH commit to main, then
the QUERY_REGRESSION_ECS_IMAGE_ID variable update)
The dev-builder jobs, their ECR/CN/tag-update dependents, and the
runner rebuild now share one workflow; the changes filter preserves
the previous on-push behavior for the dev-builder images while the
runner rebuild keeps its own trigger. The epoch-bump commit only
touches query-regression.yml, which is outside the trigger paths, so
no re-trigger loop.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): auto-resolve the ECS base image for the runner rebuild
The automated rebuild failed with 'Missing required configuration:
--base-image-id' because the ALIYUN_ECS_BASE_IMAGE_ID repo variable
does not exist yet (it was flagged as a one-time setup item).
Remove the setup dependency instead: build-ecs-image.py now defaults
--base-image-id to the latest public Ubuntu 24.04 x86_64 system image
in the region (DescribeImages with image_owner_alias=system), so no
manual variable is required. The runner Dockerfile pins every tool
version itself, so base-image drift is low-risk; --base-image-id or
the ALIYUN_ECS_BASE_IMAGE_ID variable still pin a specific base image
deterministically, and the workflow only passes the flag when the
variable is set.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(query-regression): clarify what an image rebuild requires
A routine runner-image rebuild needs no manual file updates: the
rebuild job updates QUERY_REGRESSION_ECS_IMAGE_ID and
RUNNER_IMAGE_EPOCH; the toolchain derives from rust-toolchain.toml;
uv, sccache, otelgen, rustup, and the runner base are pinned by
digest/sha/commit in the Dockerfile. Only an apt package revision
bump (mold, protoc, python3) between rebuilds requires bumping the
corresponding Verify pins, and that failure is loud with the observed
version.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): correct SDK field names in the base-image resolver
DescribeImagesRequest takes 'ostype' (not 'os_type') and the image
items expose 'osname'/'osname_en' (not 'os_name') in the pinned
alibabacloud_ecs20140526 SDK range, so the auto-resolution added in
5a9fd2c769 crashed with a TypeError before describing anything.
Fix the request fields, move the architecture filter server-side, and
paginate (page_size=100 until a short page) instead of relying on a
single default-sized response. Match Ubuntu 24.04 on the localized
osname or the English osname_en.
Verified against the real SDK models (uv run --with
'alibabacloud_ecs20140526>=4.1.0,<6'): a two-page fake client picks
the newest Ubuntu 24.04 via osname_en and rejects 22.04/Windows
decoys.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): correct the repo-root path in build-ecs-image.py
ASSETS_DIR.parent.parent lands on runner-scale-sets, not the repo
root -- the toml lookup failed with FileNotFoundError. The root is
four levels above ecs-image; express it as an explicit REPO_ROOT
constant (ASSETS_DIR.parents[3]).
Verified every path main() reads against the real checkout layout
(Dockerfile, rust-toolchain.toml, start-runner.sh, the systemd unit,
plus REPO_ROOT sanity against Cargo.toml/.git), re-checked the
user-data toml staging round-trip, and re-ran the base-image
resolver regression test.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
* chore: check enterprise-gated files are listed in both license configs
A file reachable only through `#[cfg(feature = "enterprise")] mod ...;` is
governed by the GreptimeDB Enterprise License, so it must appear in the
`includes` of licenserc-enterprise.toml and the `excludes` of licenserc.toml.
hawkeye stays silent when it does not: the file keeps its Apache-2.0 header and
passes the default check precisely because it was never excluded from it.
scripts/check-enterprise-license.py walks enterprise-gated `mod` declarations,
resolves them to files (submodules included) and diffs that set against both
configs, also reporting stale entries. It runs in the license job in CI and as
`make check-enterprise-license`.
Documents the split it cannot decide for you — whole enterprise features get
their own file, a gated match arm stays inline — in
.agents/architecture-invariants.md.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: tighten enterprise license checks
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: do not remove time filters
* chore: remove `time_range` from parquet reader
* chore: print more message in the check script
* chore: fix unused error
* build: Download assets to cargo output dir
Also remove the output from the build script and only print the output
on failure
* chore: Update src/servers/build.rs
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
* build: replace pushd by cd
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
* ci: set whether it is the latest release by using 'ncipollo/release-action'
* ci: modify greptimedb install script to use the latest nightly version binary
Ning Sun
shuiyisong
Yan Tingwang
yihong
Ruihang Xia
Yingwen
tison
LFC