* docs: link the observability database definition from README
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: add the new required checks to the Docs CI placeholders
The main ruleset now requires five more checks. Their workflows skip
docs-only PRs via paths-ignore, so without placeholders those PRs can
never satisfy the required checks.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: add relational region engine skeleton and registration
Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered.
Signed-off-by: tian1220A <1573612521@qq.com>
* feat: add relational connection and idempotent region lifecycle
Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder.
Signed-off-by: tian1220A <1573612521@qq.com>
* refactor: address review comments on relational region engine
- remove connection.rs (only used by its own tests); it will be
reintroduced when connection handling is wired up
- drop RelationalOptions::version (not validated or used to select a
deserialization format)
- validate an idempotent create against the existing region definition
(region id / schema / primary key / options) and return Unexpected on
mismatch, with a reason field on RegionExists
- use serde.workspace and remove the empty [features] section
- cargo fmt
Signed-off-by: tian1220A <1573612521@qq.com>
* fix: adapt relational engine to latest main
- implement RegionEngine::set_region_role_with_lease (no-op lease, the
engine takes no writes so no lease is needed)
- pass node_id to build_object_store_manager in the registration test
- drop stale url dependency from the relational-engine lock entry
Signed-off-by: tian1220A <1573612521@qq.com>
* ci: register relational-engine in scope rules
Signed-off-by: tian1220A <1573612521@qq.com>
* fix: remove table self-dependency from Cargo.lock
Signed-off-by: tian1220A <1573612521@qq.com>
---------
Signed-off-by: tian1220A <1573612521@qq.com>
* ci: gate the integration CI long tail by diff scope
`integration.yml` starts its long tail on every pull request: the three fuzz
families, the chaos legs, the export/import E2E job and the two sqlness
KvBackend legs. Most diffs cannot affect them, so a `scope` job now classifies
the pull request and tells those jobs whether to start.
The classifier is deterministic and fail-open. Forced modes (merge queue,
`schedule`, `workflow_dispatch`, `ci:full`, `CI_SCOPE_FORCE_ALL`) and three
floors -- always-run paths, an uncovered `src/` change, and a PromQL/flow
filter -- can only widen the result; everything else is a glob match against the
suite catalog. Every step of the job is `continue-on-error` and every gate reads
`needs.scope.outputs.<key> != 'false'`, so a classifier crash runs the jobs
instead of skipping them. The rules are checked out from the pull request base,
so a pull request cannot relax its own gate.
The policy is data: `.github/ci/scope-rules.toml` holds the path groups, the
suite catalog and the always-run list, with a `reason` on every exclusion.
`ci_scope_rules.py` loads it with the stdlib `tomllib` and validates the shape
(unknown or cyclic `@group` references, a `family` that disagrees with the suite
id, an empty reason) and that every glob's literal prefix still exists, so a
renamed crate cannot leave a rule that matches nothing. `make check-ci-scope`
runs that check locally; the `CI Scope Rules` job runs it on `ubuntu-slim`, the
same image the classifier uses. `.github/ci/scope-cases.toml` holds golden cases
that `test_ci_scope.py` replays through the real classifier.
The PostgreSQL and MySQL sqlness legs move into their own `sqlness-kvbackend`
job so they can be gated: a job-level `if:` cannot read the matrix, and gating
them inside `sqlness` would skip required checks and block the merge. They run
the same cases as the required Basic leg, so only the backend itself matters.
The verdict is visible where triage looks: one log line per suite, the exact
gate values, `::warning::` when a lookup fails and the gate widens, and a job
summary that lists each suite with the rule that decided it and the changed
files that matched.
Measured on 200 merged and open pull requests: 27.5% always run, 41% decided by
path rules, 31.5% skip every gated job; 995 of 2600 suite selections are
skipped, with no selection that a path rule makes. The classifier itself runs
in about ten seconds on a single-vCPU runner.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): let the scope job read pull requests
The `scope` job declared `permissions: contents: read`, and an explicit block
sets every unlisted permission to `none`. `GET /pulls/{n}` therefore answered
403, `build_plan` widened to a full run as designed, and every gate read an
absent output as "run": the gate could never skip anything.
- `pull-requests: read` on the `scope` job.
- `tests/runner/**` joins `@kv-backend`: the runner implements
`--setup-pg`/`--setup-mysql`, which only the KV-backend legs pass, so a
change there must not skip the only job that exercises it. The
`UNMAPPED_CRATES` excuse is gone with it.
- The README no longer claims a queue failure blocks the merge: the gated jobs
are not required checks.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): run every suite when the kind registry script changes
The distributed and chaos fuzz jobs start their cluster through
`.github/actions/setup-kind`, which runs `.github/scripts/kind-with-registry.sh`.
`.github/actions/**` is in `ALWAYS_RUN` but the script is not, so a change
confined to it selected no suite at all and every gate stayed `false`: the
jobs that exercise the changed cluster setup were skipped.
The script joins `ALWAYS_RUN`, consistent with the action that calls it, and a
golden case pins it.
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(ci): widen the gate for the fuzz corpus seeding script
`fuzz-test/action.yaml` runs `.github/scripts/prepare-fuzz-corpus.sh` before
the cache restore in every fuzz job, but the script was not in `ALWAYS_RUN`.
A change confined to it therefore selected no suite: `any_fuzz` stayed
`false` and every fuzz job that runs it was skipped.
The script joins `ALWAYS_RUN` next to the other scripts the fuzz jobs execute,
with a golden case for it.
Both misses so far (this one and `kind-with-registry.sh`) came through a
composite action, which the workflow never names. `WorkflowContractTests`
now resolves `uses: ./.github/actions/<name>` transitively for the gated jobs
and fails when one of the scripts they run is not in `ALWAYS_RUN`.
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
The `datanodeStorage` field belongs directly under `monitoring.standalone`
(GreptimeDBStandaloneSpec), not under `monitoring.standalone.base`
(PodTemplateSpec). Nesting it under `base` renders an invalid
GreptimeDBCluster resource and the CRD schema rejects it during
server-side apply:
.spec.monitoring.standalone.base.datanodeStorage: field not declared in schema
* ci: install protoc with taiki-e/install-action
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: run multi-language SDK tests on Python 3.13
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: drop the unused libfuzzer-14-dev package from fuzz builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: build fuzz targets with the bundled libFuzzer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
The merge_group coverage job did not set RUST_MIN_STACK, so test threads
ran with the default 2MiB stack while the regular test job (and the
production runtime in debug) use 8MiB. The deeper PromQL planning chain
from #9494 overflows 2MiB under llvm-cov instrumentation, aborting
tests::promql_test::anon_promql_ratio_repro::case_1_test_with_standalone
with SIGABRT (12/12 failures across three merge-queue candidates).
Align the coverage job with the existing 8MiB test policy.
Signed-off-by: discord9 <discord9@163.com>
* test: add repeatable metric snapshot S3 benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: use only existing CI S3 permissions in the metric snapshot benchmark
The CI S3 user is not allowed GetBucketLocation or ListBucketMultipartUploads,
so the bucket probe and the prefix cleanup failed with AccessDenied. Use only
list, put, get and delete under the run prefix, as the object store WAL
benchmark does, and probe S3 before allocating the EC2 runner.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: scope metric snapshot inventory to the current round
The full round's S3 inventory listed the whole run prefix, so it also counted
the smoke snapshot. List only the round's snapshot prefix while cleanup still
covers the run, and assert that the teardown's S3 cleanup step runs always()
with the validated run prefix.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: record the AWS validation run of the metric snapshot benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: benchmark the object store WAL against S3 from an EC2 runner
Add a workflow_dispatch workflow that loads the o11ybench agent observability
dataset into a fresh GreptimeDB container per cell and compares three WAL
configurations that differ only in [wal]: Raft Engine with sync_write on the
instance store, and the object store WAL in durable and enqueued mode, with
the data files on S3 in the runner's region for all three. Rounds alternate
the configuration order. A summary reports rows per second, D/R and E/R as
the geometric mean over rounds, append acknowledgement p50 and p99, and the
object store WAL counters. A smoke mode runs the same chain on ubuntu-latest
against the MinIO fixture.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scrape WAL metrics on failed loads and keep logic out of the benchmark YAML
Scrape the object store WAL metrics before stopping a cell's server whether
the load succeeded or failed, so failed cells keep their stall, poison and
create failure counters. Build the summary from the plan so that cells which
never ran are listed and leave their ratios undefined. Move budget, region,
runtime image, manifest and credential scan logic from the workflow into the
helper scripts, and drive the cell lifecycle in the tests through fake docker
and curl binaries.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop redundant required flags from benchmark inputs
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a benchmark cell whose metrics scrape fails and zero untouched WAL counters
A load that succeeds without its metrics scrape now fails the cell with exit
code 4 and records metrics_scraped in cell.json. The summary reports an
event counter that never registered as zero when the cell is an object store
WAL cell whose scrape succeeded and created objects, and as unknown otherwise.
Planned cells that never ran are read through the same path as recorded ones.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: register the S3 benchmark workflow for dispatch from its branch
A workflow_dispatch workflow can only be dispatched once GitHub has
registered it, which happens on the default branch or when one of its
triggers fires. A push trigger limited to this branch registers it; validate
runs only for workflow_dispatch, so a push run skips every job.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: delete the benchmark S3 prefix one object at a time
boto3 now sends a CRC checksum instead of Content-MD5 with DeleteObjects,
which the MinIO fixture of the smoke run rejects. DeleteObject needs neither
and the prefix of one run holds at most a few thousand objects.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: default the S3 benchmark to c6id.2xlarge
c7id is not offered in the runner region ap-southeast-1, where DescribeInstanceTypes
rejects it. c6id is the compute optimized x86 family with a local NVMe instance
store that the region offers, with the same 8 vCPU and 16 GiB at 2xlarge.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: drop the registration trigger of the S3 benchmark workflow
The workflow is registered now, so dispatches no longer need a push trigger
on its branch.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail an S3 benchmark run on EC2 without a local instance store
Without an instance store the Raft Engine WAL of the R cells silently stayed
on the root EBS volume, so the run measured a different baseline. Only a smoke
run may keep it on the root volume now.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let the instance store check fail its workflow step
The step piped wal-disk into tee without pipefail, so a rejected host still
passed. wal-disk now writes its record itself and the step runs it alone.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: make query regression non-blocking for scheduled nightly releases
Scheduled nightly releases no longer gate publishing on the query
regression release test. The test still runs (validation policy stays
'all' for automatic releases) and uploads its report, but any result -
failure, timeout, cancellation, or skip - no longer blocks image
publishing or the GitHub release for schedule-triggered runs.
Since reusable-workflow caller jobs cannot use continue-on-error, the
gate is relaxed at the consumers instead: the if conditions of
release-images-to-dockerhub and publish-github-release now also pass
when github.event_name == 'schedule' regardless of the query-regression
result. All other gates (runner allocation, artifact builds,
prepare-release-validation) are unchanged.
Tag-push and manual-dispatch releases remain fully blocking unless a
skip policy is explicitly chosen via the release_validation input.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: preserve downstream nightly release jobs after regression failures
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: evenyag <realevenyag@gmail.com>
Co-authored-by: evenyag <realevenyag@gmail.com>
Previously the nightly release was scheduled only on Mondays, so a
failed run left users without nightly builds for a whole week.
Now the schedule triggers every weekday at 00:00 UTC, but the release
only proceeds when the latest published nightly release is older than
NIGHTLY_RELEASE_MAX_AGE_DAYS (5) days. The check runs in
allocate-runners before any EC2 runner is allocated, and gates all
schedule-driven jobs (including the Slack notification, so a skipped
nightly is silent). Tag pushes and manual dispatches are unaffected.
Signed-off-by: Ning Sun <sunning@greptime.com>
* revert(ci): pin the query-regression runner toolchain to nightly-2026-03-21
The query-regression benchmark compiles both the candidate and the
BASE checkout (the previous nightly build). Base refs can predate the
stable-toolchain migration and still use #![feature] gates, so the
runner toolchain must stay a nightly that can build historical
revisions; deriving it from the workspace rust-toolchain.toml (now
stable 1.96.1) breaks base builds.
Revert the toolchain-toml coupling introduced in #9369 and keep the
parts that were correct:
- query-regression.yml: RUSTUP_TOOLCHAIN hard-pinned to
nightly-2026-03-21 again (with a comment explaining why), Verify
assertions back to the exact nightly versions, and the
test-tooling pin-derivation machinery removed
- runner Dockerfile: ARG RUST_TOOLCHAIN=nightly-2026-03-21 + baked
ENV restored; the COPY rust-toolchain.toml parsing removed
- build-ecs-image.py: the toml staging in the builder user-data
removed; base-image auto-resolution kept but retargeted to Ubuntu
26.04 to match the Verify tool pins (python3 3.14 etc.)
- the rebuild job keeps the epoch-bump lockstep, now as a PR from a
timestamped ci/ branch mirroring update-dev-builder-version.sh
(direct pushes to main fail GH006 under branch protection)
Complements #9376 (actions-runner bump), which requires a rebuilt
image with the non-deprecated runner.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: skip qreg for rust-toolchain change
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): teach check-builder-rust-version.sh to handle stable channels
The script extracted a YYYY-MM-DD date from rust-toolchain.toml to
compare against the rustc build date inside the dev-builder image —
a nightly-era design. With channel = "1.96.1" there is no date in
the file, so every release build failed with 'Error: No rust toolchain
version found in rust-toolchain.toml'.
Extract the channel token instead and branch on it:
- stable channel (X.Y[.Z]): require the builder image's rustc to
exactly match the pinned version
- nightly-YYYY-MM-DD: keep the legacy date-difference check
Verified against a mocked docker for all four paths (stable match /
mismatch, nightly fresh / stale).
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore(toolchain): finish stable-migration cleanup in docs and query-regression pin
- README/AGENTS: the toolchain is now stable Rust pinned by
rust-toolchain.toml, not nightly
- query-regression: align the benchmark toolchain pin with the
workspace (nightly-2026-03-21 = 1.96.0-nightly -> stable 1.96.1),
including the exact-version assertions (cargo 356927216, rustc
31fca3adb, both 2026-06-26) and the runner image default
The query-regression runner image must be rebuilt and
QUERY_REGRESSION_ECS_IMAGE_ID bumped together with these pins
(per .github/runner-scale-sets/query-regression/README.md) before
the next regression run.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the Rust toolchain pin from rust-toolchain.toml
Replace the hard-coded RUSTUP_TOOLCHAIN value and the hard-coded
version strings in the runner Verify assertions with a pin resolved
from rust-toolchain.toml:
- the always-running test-tooling job exports the channel parsed from
rust-toolchain.toml as a job output
- query-regression sets RUSTUP_TOOLCHAIN from that output
- the Verify step escapes the pin into the cargo/rustc/active-toolchain
regexes at runtime; the exact commit hash and date are asserted
generically since a stable version identifies the release
Removing the redundant require_eq (workflow yaml vs runner env) since
both now flow from the single source of truth. When rust-toolchain.toml
is bumped, the run fails with a clear signal until the runner image is
rebuilt with the new toolchain, keeping the existing lockstep contract.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): derive the runner image toolchain from rust-toolchain.toml
Remove the hard-coded 'ARG RUST_TOOLCHAIN=1.96.1' from the
query-regression runner Dockerfile. The pin is now parsed from a
COPY'd rust-toolchain.toml at build time (the bootstrap script builds
with the repo root as context, so the file is in the build context):
- rustup-init installs the parsed channel as the default toolchain
- the baked ENV RUSTUP_TOOLCHAIN is dropped: the rustup default makes
bare cargo/rustc resolve correctly without it, and the workflow
supplies RUSTUP_TOOLCHAIN explicitly at run time
- the build-time self-verification asserts the active toolchain
against the same parsed pin
With this, rust-toolchain.toml is the single source of truth for the
benchmark toolchain end to end: the image bakes whatever the toml says
at build time and the workflow asserts against the toml at run time.
A toolchain bump now only requires rebuilding the image.
The changed mechanics were verified natively with the real rustup-init
1.29.0 and the real 1.96.1 toolchain (registry pulls are unavailable
in this sandbox): parsing, default-toolchain installation without
RUSTUP_TOOLCHAIN env, bare cargo/rustc resolution, and the
active-toolchain assertion for both '(default)' and '(overridden)'
output forms.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci(query-regression): rebuild the runner image automatically on toolchain changes
Mirror the dev-builder automation for the query-regression ECS runner
image: a new rebuild-query-regression-runner-image.yaml workflow runs
whenever rust-toolchain.toml or the query-regression runner directory
changes on main (or via manual dispatch). It drives the existing
build-ecs-image.py ops tool, then completes the documented lockstep
updates in order: bump RUNNER_IMAGE_EPOCH in query-regression.yml and
push the commit to main, and only then point the
QUERY_REGRESSION_ECS_IMAGE_ID repo variable at the new image, so the
next regression run picks up image and epoch together.
Also fix build-ecs-image.py to stage rust-toolchain.toml into the
temporary docker build context: the AMI path builds the embedded
Dockerfile from an empty /tmp/image-context, which would break on the
Dockerfile's COPY of rust-toolchain.toml introduced earlier. The
user-data now base64-stages the toml next to the Dockerfile before
docker build.
Verified: py_compile, render_user_data round-trip (mkdir -> stage ->
docker build ordering), and the RUNNER_IMAGE_EPOCH bump sed against
the real workflow file. Requires a new ALIYUN_ECS_BASE_IMAGE_ID repo
variable (Ubuntu 24.04 public image id in the region); all other
secrets/vars are shared with the provisioning job.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* ci: fold the query-regression runner rebuild into release-dev-builder-images.yaml
Merge the standalone rebuild workflow into the existing builder-image
release workflow, as one entry point for all builder artifacts:
- push paths extended with .github/runner-scale-sets/query-regression/**
- new 'release_query_regression_runner_image' dispatch input
- a 'changes' job diffs the pushed range (github.event.before..sha,
with an everything-changed fallback for dispatch or unknown bases)
so each expensive rebuild only fires for its own paths:
rust-toolchain.toml gates both, docker/dev-builder/** gates the
dev-builder images, the query-regression runner directory gates the
ECS image rebuild
- the rebuild job itself is unchanged from the standalone workflow
(build-ecs-image.py, then RUNNER_IMAGE_EPOCH commit to main, then
the QUERY_REGRESSION_ECS_IMAGE_ID variable update)
The dev-builder jobs, their ECR/CN/tag-update dependents, and the
runner rebuild now share one workflow; the changes filter preserves
the previous on-push behavior for the dev-builder images while the
runner rebuild keeps its own trigger. The epoch-bump commit only
touches query-regression.yml, which is outside the trigger paths, so
no re-trigger loop.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): auto-resolve the ECS base image for the runner rebuild
The automated rebuild failed with 'Missing required configuration:
--base-image-id' because the ALIYUN_ECS_BASE_IMAGE_ID repo variable
does not exist yet (it was flagged as a one-time setup item).
Remove the setup dependency instead: build-ecs-image.py now defaults
--base-image-id to the latest public Ubuntu 24.04 x86_64 system image
in the region (DescribeImages with image_owner_alias=system), so no
manual variable is required. The runner Dockerfile pins every tool
version itself, so base-image drift is low-risk; --base-image-id or
the ALIYUN_ECS_BASE_IMAGE_ID variable still pin a specific base image
deterministically, and the workflow only passes the flag when the
variable is set.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* docs(query-regression): clarify what an image rebuild requires
A routine runner-image rebuild needs no manual file updates: the
rebuild job updates QUERY_REGRESSION_ECS_IMAGE_ID and
RUNNER_IMAGE_EPOCH; the toolchain derives from rust-toolchain.toml;
uv, sccache, otelgen, rustup, and the runner base are pinned by
digest/sha/commit in the Dockerfile. Only an apt package revision
bump (mold, protoc, python3) between rebuilds requires bumping the
corresponding Verify pins, and that failure is loud with the observed
version.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): correct SDK field names in the base-image resolver
DescribeImagesRequest takes 'ostype' (not 'os_type') and the image
items expose 'osname'/'osname_en' (not 'os_name') in the pinned
alibabacloud_ecs20140526 SDK range, so the auto-resolution added in
5a9fd2c769 crashed with a TypeError before describing anything.
Fix the request fields, move the architecture filter server-side, and
paginate (page_size=100 until a short page) instead of relying on a
single default-sized response. Match Ubuntu 24.04 on the localized
osname or the English osname_en.
Verified against the real SDK models (uv run --with
'alibabacloud_ecs20140526>=4.1.0,<6'): a two-page fake client picks
the newest Ubuntu 24.04 via osname_en and rejects 22.04/Windows
decoys.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(ci): correct the repo-root path in build-ecs-image.py
ASSETS_DIR.parent.parent lands on runner-scale-sets, not the repo
root -- the toml lookup failed with FileNotFoundError. The root is
four levels above ecs-image; express it as an explicit REPO_ROOT
constant (ASSETS_DIR.parents[3]).
Verified every path main() reads against the real checkout layout
(Dockerfile, rust-toolchain.toml, start-runner.sh, the systemd unit,
plus REPO_ROOT sanity against Cargo.toml/.git), re-checked the
user-data toml staging round-trip, and re-ran the base-image
resolver regression test.
Part of #9289.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore(toolchain): switch to stable Rust 1.96.1 and remove all nightly feature gates
Move the workspace from the pinned nightly-2026-03-21 to stable
1.96.1 and drop all 23 '#![feature]' gates across 13 crates,
rewriting the still-unstable API usages with stable equivalents:
- try_blocks: closures / an async block (table, query, common-function, servers)
- duration_constructors: Duration::from_secs(n * 86400) / (n * 60)
- iterator_try_collect: collect::<Result<Vec<_>, _>>()
- box_patterns: as_deref() + matches! chains (sql)
- error_iter: error_chain_root() source-chain walker (common-error);
sources() includes the error itself, so the walker never panics
- int_roundings: div_floor -> div_euclid (equal for positive divisors)
- iter_partition_in_place: stable sort_by_key partition helper (index)
- hash_set_entry: HashSet::insert bool / contains+insert
- trait_alias: lifetime-parameterized dyn FnOnce type aliases (puffin)
- string_from_utf8_lossy_owned: from_utf8_lossy(&v).into_owned()
- never_type: Infallible (common-recordbatch)
- debug_closure_helpers: closure-backed DebugFmt newtype (mito2)
- binary_heap_pop_if: peek().is_some_and() + pop()
- exclusive_wrapper: drop Exclusive; C: Send + Unpin already in bounds
- stmt_expr_attributes: stale gate, no usages
Also fix release-dev-builder-images.yaml, which parsed
rust-toolchain.toml with a date-only regex and would produce empty
image versions with a stable channel; it now extracts the full
channel token. Dev-builder images verified against stable 1.96.1
(image build, default-toolchain behavior, binstall/nextest, riscv64
and android targets, in-image cargo check).
Validated on 1.96.1: cargo check --workspace --all-targets, clippy
--workspace --all-targets --all-features -D warnings, cargo fmt
--check, and nextest on all 13 affected crates (4586 passed).
Part of #9289. Depends on #9298 (fuzz nightly quarantine) merging
first.
Signed-off-by: Ning Sun <sunning@greptime.com>
* chore: update flake checksum
* chore: use wild for linker in flake
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: use relative object keys for Windows filesystem access
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: cover Windows path and time limits in full test CI
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: use relative keys in metadata snapshot tests
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks
cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.
- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
misplaced deps to the correct sections, drop orphaned
[workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
for dependencies that are structurally required despite no textual
reference: sqlparser (required by sqlparser_derive expansions in
datatypes, common-query), common-error (required by common-macro's
stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
(feature-pinning for the transitive kube dependency in tests-fuzz),
tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
common-mem-prof), protobuf (required by build.rs-generated bindings in
log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections
Part of #9289
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix(meta): populate physical metric table column ids (#9286)
* fix(meta): populate physical metric table column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* test(meta): verify physical metric column ids
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
---------
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
* fix(postgres): return empty responses for comment-only SQL (#9295)
fix(postgres): handle parsed empty queries in both protocols
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
* ci: create docs follow-up issue on PR merge instead of on label (#9237)
* ci: create docs follow-up issue on PR merge instead of on label
The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.
Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
carries the docs-required label
This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: address review comments on docs issue creation timing
- Only touch docs labels when the docs checkbox state actually changed
in an edit. Previously, editing any other part of the PR body while
the checkbox stayed checked removed the docs-required label, silently
dropping the docs follow-up now that issue creation happens at merge.
Unchanged checkbox now leaves labels untouched, which also preserves
manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
re-read the live PR via the API and create the docs issue if the
docs-required label is present OR the checkbox is ticked in the
current body.
- Make the workflow concurrency group action-aware so a merge run does
not cancel an in-flight label update from an edit run.
Signed-off-by: Ning Sun <sunning@greptime.com>
* fix: make docs-required label the single source of truth at merge
The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.
At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.
Signed-off-by: Ning Sun <sunning@greptime.com>
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
* perf(promql): push label filters into grouped join inputs (#9280)
* perf(promql): propagate matching filters through grouped joins
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* perf(promql): check matcher safety on the receiving operand
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): spell out the shapes a filter may cross
`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.
Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.
No behavior change.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): assert the new pushdowns reach the scan
The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.
Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(ci): build tests-integration lib with meta-srv/mock (#9299)
* fix(ci): build tests-integration lib with meta-srv/mock
tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.
Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(repartition): split per-case repartition tests
test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.
Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(json2): support altering JSON2 settings (#9029)
* feat(sql): support alter syntax for JSON2 columns
Signed-off-by: fys <fengys1996@gmail.com>
* fix(json2): preserve rows on type hint mismatch during compaction
* refactor(json2): simplify alter settings handling
* fix(json2): preserve coerced values during compaction
* chore: remove unnecessary clone
* chor: reduce memory allocations
* fix: cargo clippy
* chore: update greptime-proto to main branch
* refactor(datatypes): unify string handling with other JSON type hints
* fix: cr
---------
Signed-off-by: fys <fengys1996@gmail.com>
* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)
* fix: run compaction pruner tasks on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep compaction metadata and index work on compact runtime
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add AI matching, classification, and scoring functions (#9300)
* feat: return matching scores from jev
Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: add Jev choice and score functions
Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor: use generic AI SQL function names
Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: share constant AI criteria within each batch
Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: preserve AI score uncertainty in JSONB results
Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs: explain reuse of volatile AI evaluations
Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: share logical table batching with OTLP metrics (#9288)
* feat: share logical table batching with OTLP metrics
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: unify pending rows batch acknowledgement policy
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align logical batcher example configuration expectations
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: align batcher worker channel defaults to 65536
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* perf(mito2): lazily decode dense primary key columns (#9226)
* perf(mito2): lazily decode dense primary key columns
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(mito2): bypass lazy decoding for full primary keys
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(mito-codec): preserve prefix decoding errors
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): align encoded length helper naming
Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito2): clarify conditional dense key decoding
Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(mito-codec): share string framing in row converter
Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: sync lock
* fix: shear and check issues
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
Weny Xu
LFC
liyang
dennis zhuang
zhaiAohan
Dhruv Vaishnav
Yingwen
jeremyhi
yihong
localhost
discord9
Ning Sunandevenyag
