revert(ci): pin the query-regression runner toolchain to nightly-2026-03-21 (#9389)

* revert(ci): pin the query-regression runner toolchain to nightly-2026-03-21

The query-regression benchmark compiles both the candidate and the
BASE checkout (the previous nightly build). Base refs can predate the
stable-toolchain migration and still use #![feature] gates, so the
runner toolchain must stay a nightly that can build historical
revisions; deriving it from the workspace rust-toolchain.toml (now
stable 1.96.1) breaks base builds.

Revert the toolchain-toml coupling introduced in #9369 and keep the
parts that were correct:

- query-regression.yml: RUSTUP_TOOLCHAIN hard-pinned to
  nightly-2026-03-21 again (with a comment explaining why), Verify
  assertions back to the exact nightly versions, and the
  test-tooling pin-derivation machinery removed
- runner Dockerfile: ARG RUST_TOOLCHAIN=nightly-2026-03-21 + baked
  ENV restored; the COPY rust-toolchain.toml parsing removed
- build-ecs-image.py: the toml staging in the builder user-data
  removed; base-image auto-resolution kept but retargeted to Ubuntu
  26.04 to match the Verify tool pins (python3 3.14 etc.)
- the rebuild job keeps the epoch-bump lockstep, now as a PR from a
  timestamped ci/ branch mirroring update-dev-builder-version.sh
  (direct pushes to main fail GH006 under branch protection)

Complements #9376 (actions-runner bump), which requires a rebuilt
image with the non-deprecated runner.

Part of #9289.

Signed-off-by: Ning Sun <sunning@greptime.com>

* chore: skip qreg for rust-toolchain change

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
This commit is contained in:
Ning Sun
2026-09-29 02:21:27 +00:00
committed by GitHub
parent 0686f72dc6
commit 28f01d2ffe
5 changed files with 100 additions and 83 deletions
@@ -22,14 +22,14 @@ ARG SCCACHE_SHA256=aec995a83ad3dff3d14b6314e08858b7b73d35ca85a5bcf3d3a9ec07dee35
ARG RUSTUP_INIT_VERSION=1.29.0
ARG RUSTUP_INIT_TARGET=x86_64-unknown-linux-gnu
ARG RUSTUP_INIT_SHA256=4acc9acc76d5079515b46346a485974457b5a79893cfb01112423c89aeb5aa10
# Single source of truth for the Rust toolchain: parsed from
# rust-toolchain.toml at build time, so the image always bakes the
# workspace toolchain without a second hard-coded pin here.
COPY rust-toolchain.toml /opt/rust-toolchain.toml
# Pinned to a nightly deliberately: the benchmark compiles historical base
# checkouts (previous nightly builds) that still use #![feature] gates, so
# the runner cannot follow the workspace's rust-toolchain.toml (now stable).
ARG RUST_TOOLCHAIN=nightly-2026-03-21
ENV RUSTUP_HOME=/opt/rustup \
CARGO_HOME=/opt/cargo \
RUSTUP_TOOLCHAIN=${RUST_TOOLCHAIN} \
RUSTUP_AUTO_INSTALL=0 \
PATH=/opt/cargo/bin:${PATH}
@@ -73,14 +73,12 @@ RUN curl --fail --location --silent --show-error \
&& install --mode=0755 /tmp/sccache/sccache /usr/local/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache
RUN rust_toolchain="$(grep -E '^channel' /opt/rust-toolchain.toml | cut -d'"' -f2)" \
&& test -n "${rust_toolchain}" \
&& curl --fail --location --silent --show-error \
RUN curl --fail --location --silent --show-error \
--output /tmp/rustup-init \
"https://static.rust-lang.org/rustup/archive/${RUSTUP_INIT_VERSION}/${RUSTUP_INIT_TARGET}/rustup-init" \
&& echo "${RUSTUP_INIT_SHA256} /tmp/rustup-init" | sha256sum --check --status - \
&& chmod 0755 /tmp/rustup-init \
&& /tmp/rustup-init -y --profile minimal --default-toolchain "${rust_toolchain}" --no-modify-path \
&& /tmp/rustup-init -y --profile minimal --default-toolchain "${RUST_TOOLCHAIN}" --no-modify-path \
&& rm -f /tmp/rustup-init \
&& chown -R root:root /opt/rustup /opt/cargo \
&& chmod -R go-w /opt/rustup /opt/cargo \
@@ -91,8 +89,6 @@ USER runner
RUN set -eu \
&& test "$(id -u)" = "1001" \
&& rust_toolchain="$(grep -E '^channel' /opt/rust-toolchain.toml | cut -d'"' -f2)" \
&& test -n "${rust_toolchain}" \
&& temporary_cargo_home="$(mktemp --directory)" \
&& temporary_proto_dir="" \
&& cleanup() { rm -rf "${temporary_cargo_home}" "${temporary_proto_dir}"; } \
@@ -104,7 +100,7 @@ RUN set -eu \
&& rustc --version \
&& active_toolchain="$(rustup show active-toolchain)" \
&& printf 'Active toolchain: %s\n' "${active_toolchain}" \
&& case "${active_toolchain}" in "${rust_toolchain}-x86_64-unknown-linux-gnu"|"${rust_toolchain}-x86_64-unknown-linux-gnu "*) ;; *) exit 1;; esac \
&& case "${active_toolchain}" in "${RUST_TOOLCHAIN}-x86_64-unknown-linux-gnu"|"${RUST_TOOLCHAIN}-x86_64-unknown-linux-gnu "*) ;; *) exit 1;; esac \
&& test ! -w /opt/rustup \
&& test ! -w /opt/cargo/bin \
&& test -r /usr/include/google/protobuf/any.proto \
@@ -88,12 +88,17 @@ sweep.
The runner `Dockerfile` in the parent directory stays the single source of the
tool contract. The image is rebuilt **automatically** by the
`rebuild-query-regression-runner-image` job in
`.github/workflows/release-dev-builder-images.yaml` whenever
`rust-toolchain.toml` or anything under this directory changes on main (or via
manual dispatch): it runs the ops tool below, then bumps
`RUNNER_IMAGE_EPOCH` in `query-regression.yml` and points the
`QUERY_REGRESSION_ECS_IMAGE_ID` repo variable at the new image, so the next
regression run picks up image and epoch together.
`.github/workflows/release-dev-builder-images.yaml` whenever anything under
this directory changes on main (or via manual dispatch): it runs the ops tool
below, opens an epoch-bump PR for `RUNNER_IMAGE_EPOCH` in
`query-regression.yml`, and points the `QUERY_REGRESSION_ECS_IMAGE_ID` repo
variable at the new image.
The runner toolchain is **pinned inside the Dockerfile** to
`nightly-2026-03-21` and deliberately does NOT follow the workspace
`rust-toolchain.toml`: the benchmark also compiles the BASE checkout (the
previous nightly build), which may predate the stable-toolchain migration and
still require a nightly compiler.
The manual fallback (also what the workflow runs):
@@ -105,10 +110,10 @@ uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \
```
`--base-image-id` is optional: the script defaults to the latest public
Ubuntu 24.04 image in the region (the Dockerfile pins every tool version
itself, so base drift is low-risk); pass it — or set the
`ALIYUN_ECS_BASE_IMAGE_ID` repo variable consumed by the automated job —
to pin a specific base image.
Ubuntu 26.04 image in the region (26.04 matches the tool-version pins the
Verify step asserts, e.g. python3 3.14; keep the two in sync); pass it — or
set the `ALIYUN_ECS_BASE_IMAGE_ID` repo variable consumed by the automated
job — to pin a specific base image.
The script boots a temporary builder instance, `docker build`s the runner
image, materializes `/opt/rustup`, `/opt/cargo`, `/usr/local/bin` tools, and
@@ -173,11 +178,10 @@ overridable via `QUERY_REGRESSION_RUNNER_UID`/`QUERY_REGRESSION_RUNNER_GID`)
and exact tool versions: `libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.40.4`,
`Python 3.14.4`, `sccache 0.16.0`, `otelgen` commit
`863a3f395d062c7322cc1de08a38774b7fdaa6c8`, root-owned `rustup 1.29.0`, and
the image-baked Rust toolchain matching `rust-toolchain.toml` (the image
parses the pin from the toml at build time, and the workflow asserts it
dynamically at run time — there is no separately pinned toolchain version).
the image-baked Rust toolchain `nightly-2026-03-21` pinned in the runner
Dockerfile (deliberately independent of `rust-toolchain.toml` — see above).
`mold` and `python3`
come from apt at image-build time (not Ubuntu 24.04's default 3.12); if the
come from apt at image-build time; if the
Ubuntu archive ships a newer package revision between rebuilds, the Verify
step fails with the observed version — bump those pins in `query-regression.yml`
when that happens. Everything else (toolchain, uv, sccache, otelgen, rustup,
@@ -25,7 +25,7 @@
"""Build the query-regression ECS custom image (manual ops tool).
Boots a temporary pay-as-you-go ECS instance from a public Ubuntu 24.04 image,
Boots a temporary pay-as-you-go ECS instance from a public Ubuntu 26.04 image,
builds the existing runner container image (the Dockerfile in the parent
directory remains the single source of the tool contract), materializes the
tool directories onto the host filesystem so the workflow's "Verify runner
@@ -52,17 +52,15 @@ import time
from pathlib import Path
ASSETS_DIR = Path(__file__).resolve().parent
# Repo root: ecs-image -> query-regression -> runner-scale-sets -> .github -> root.
REPO_ROOT = ASSETS_DIR.parents[3]
DONE_MARKER = "QREG_IMAGE_BUILD_DONE"
FAILED_MARKER = "QREG_IMAGE_BUILD_FAILED"
POLL_INTERVAL_SECONDS = 15
CONSOLE_POLL_INTERVAL_SECONDS = 30
BUILD_TIMEOUT_SECONDS = 60 * 60
# Same apt package contract as the runner Dockerfile; the base
# actions-runner image is Ubuntu 24.04, so an Ubuntu 24.04 host resolves the
# same tool versions (protoc 3.21.12, mold 2.40.4, Python 3.14.4).
# Same apt package contract as the runner Dockerfile; the base is an
# Ubuntu 26.04 image, so an Ubuntu 26.04 host resolves the same tool
# versions (protoc 3.21.12, mold 2.40.4, Python 3.14.4).
# Docker itself comes from Docker's official repository (docker-ce), not the
# distribution-packaged docker.io.
APT_PACKAGES = [
@@ -94,9 +92,8 @@ APT_PACKAGES = [
DOCKER_CE_PACKAGES = "docker-ce docker-ce-cli containerd.io docker-buildx-plugin"
def render_user_data(dockerfile: str, rust_toolchain_toml: str, start_runner: str, unit: str) -> str:
def render_user_data(dockerfile: str, start_runner: str, unit: str) -> str:
dockerfile_b64 = base64.b64encode(dockerfile.encode()).decode()
rust_toolchain_b64 = base64.b64encode(rust_toolchain_toml.encode()).decode()
start_runner_b64 = base64.b64encode(start_runner.encode()).decode()
unit_b64 = base64.b64encode(unit.encode()).decode()
packages = " ".join(APT_PACKAGES)
@@ -125,11 +122,6 @@ base64 -d > /tmp/Dockerfile <<'EOF'
{dockerfile_b64}
EOF
mkdir -p /tmp/image-context
# The Dockerfile COPYies rust-toolchain.toml (single source of truth for
# the baked toolchain); stage it into the build context.
base64 -d > /tmp/image-context/rust-toolchain.toml <<'EOF'
{rust_toolchain_b64}
EOF
docker build --platform linux/amd64 -f /tmp/Dockerfile -t qreg-runner:local /tmp/image-context
# Materialize the tool contract onto the host filesystem.
@@ -238,20 +230,22 @@ def call_api_with_retry(fn, description: str, attempts: int = 5):
def resolve_base_image_id(client, region_id: str) -> str:
"""Resolve the latest public Ubuntu 24.04 x86_64 system image.
"""Resolve the latest public Ubuntu 26.04 x86_64 system image.
Used as the default for --base-image-id: the runner Dockerfile pins
every tool version itself, so a current stock Ubuntu 24.04 base is all
every tool version itself, so a current stock Ubuntu LTS base is all
the builder needs. Pass --base-image-id (or ALIYUN_ECS_BASE_IMAGE_ID)
to pin a specific base image deterministically.
"""
from alibabacloud_ecs20140526 import models as ecs_models
def _is_ubuntu_2404(image) -> bool:
# osname is localized (e.g. "Ubuntu 24.04 64位"), osname_en the
# English form; accept either.
def _is_target_ubuntu(image) -> bool:
# osname is localized (e.g. "Ubuntu 26.04 64位"), osname_en the
# English form; accept either. Keep the Ubuntu version in sync with
# the tool-version pins asserted by the Verify step in
# query-regression.yml (e.g. python3 3.14 comes from 26.04).
for os_name in (image.osname_en, image.osname):
if os_name and "Ubuntu" in os_name and "24.04" in os_name:
if os_name and "Ubuntu" in os_name and "26.04" in os_name:
return True
return False
@@ -277,10 +271,10 @@ def resolve_base_image_id(client, region_id: str) -> str:
break
page_number += 1
candidates = [image for image in images if _is_ubuntu_2404(image)]
candidates = [image for image in images if _is_target_ubuntu(image)]
if not candidates:
raise SystemExit(
"No public Ubuntu 24.04 x86_64 system image found in region "
"No public Ubuntu 26.04 x86_64 system image found in region "
f"{region_id}; pass --base-image-id explicitly"
)
candidates.sort(key=lambda image: image.creation_time or "", reverse=True)
@@ -324,7 +318,7 @@ def main() -> int:
from alibabacloud_ecs20140526 import models as ecs_models
client = make_ecs_client(args.region_id)
# --base-image-id is optional: default to the latest public Ubuntu 24.04
# --base-image-id is optional: default to the latest public Ubuntu 26.04
# image in the region (the Dockerfile pins every tool version itself, so
# base drift is low-risk; pass --base-image-id or set
# ALIYUN_ECS_BASE_IMAGE_ID to pin deterministically).
@@ -337,7 +331,6 @@ def main() -> int:
user_data = base64.b64encode(
render_user_data(
(ASSETS_DIR.parent / "Dockerfile").read_text(),
(REPO_ROOT / "rust-toolchain.toml").read_text(),
(ASSETS_DIR / "start-runner.sh").read_text(),
(ASSETS_DIR / "ephemeral-github-runner.service").read_text(),
).encode()
+14 -18
View File
@@ -120,23 +120,12 @@ jobs:
# Ordinary PRs also run the same tests from checks.yml.
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
# The Rust toolchain pin, resolved from rust-toolchain.toml so the
# benchmark always runs the workspace toolchain without a second
# hard-coded copy in this workflow.
rust_toolchain: ${{ steps.rust-toolchain.outputs.pin }}
steps:
- name: Checkout
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Resolve Rust toolchain pin
id: rust-toolchain
run: |
pin="$(grep -E '^channel' rust-toolchain.toml | cut -d'"' -f2)"
echo "pin=${pin}" >> "$GITHUB_OUTPUT"
- name: Test query regression tooling
run: |
python3 tests/perf/test_query_regression_runner_compaction_toctou.py
@@ -201,7 +190,13 @@ jobs:
CARGO_HOME: /home/runner/.cargo
UV_CACHE_DIR: /home/runner/.cargo/uv-cache
RUSTUP_HOME: /opt/rustup
RUSTUP_TOOLCHAIN: ${{ needs.test-tooling.outputs.rust_toolchain }}
# Deliberately NOT derived from rust-toolchain.toml: the benchmark
# also compiles the BASE checkout (the previous nightly build, which
# may predate the stable-toolchain migration and still uses
# #![feature] gates), so the runner must keep a nightly toolchain
# that can build historical revisions. This pin moves only via a
# deliberate runner-image rebuild.
RUSTUP_TOOLCHAIN: nightly-2026-03-21
RUSTUP_AUTO_INSTALL: "0"
CARGO_TARGET_DIR: /home/runner/query-regression-target
QUERY_REGRESSION_CACHE_META: /home/runner/query-regression-cache-meta
@@ -515,16 +510,17 @@ jobs:
require_eq cargo_path "$(command -v cargo || true)" "/opt/cargo/bin/cargo"
require_eq rustc_path "$(command -v rustc || true)" "/opt/cargo/bin/rustc"
require_match rustup "$(capture rustup --version)" '^rustup[[:space:]]1\.29\.0([[:space:]]|$)'
# The toolchain pin flows from rust-toolchain.toml (resolved in the
# test-tooling job); escape it for the regex assertions below.
pin_regex="${RUSTUP_TOOLCHAIN//./\\.}"
# The runner toolchain is pinned to nightly-2026-03-21 (see the
# RUSTUP_TOOLCHAIN comment): it must compile historical base
# checkouts, so it intentionally does not follow rust-toolchain.toml.
require_match cargo "$(capture cargo --version)" \
"^cargo[[:space:]]${pin_regex}[[:space:]]\([0-9a-f]+[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$"
'^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match rustc "$(capture rustc --version)" \
"^rustc[[:space:]]${pin_regex}[[:space:]]\([0-9a-f]+[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$"
'^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match active_toolchain "$(capture rustup show active-toolchain)" \
"^${pin_regex}-x86_64-unknown-linux-gnu([[:space:]]|$)"
'^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$)'
require_eq RUSTUP_HOME "${RUSTUP_HOME}" "/opt/rustup"
require_eq RUSTUP_TOOLCHAIN "${RUSTUP_TOOLCHAIN}" "nightly-2026-03-21"
require_eq RUSTUP_AUTO_INSTALL "${RUSTUP_AUTO_INSTALL}" "0"
require "readable /opt/rustup" test -r /opt/rustup
require "executable /opt/rustup" test -x /opt/rustup
@@ -71,9 +71,13 @@ jobs:
files="$(git diff --name-only "${base}" "${head}")"
dev_builder=false
query_regression_runner=false
# rust-toolchain.toml only gates the dev-builder images: they bake
# the workspace toolchain. The query-regression runner toolchain is
# pinned inside its Dockerfile (nightly-2026-03-21) and must NOT
# follow the workspace pin, so its rebuild triggers only on changes
# under its own directory.
if grep -qx 'rust-toolchain.toml' <<<"${files}"; then
dev_builder=true
query_regression_runner=true
fi
if grep -q '^docker/dev-builder/' <<<"${files}"; then
dev_builder=true
@@ -335,18 +339,21 @@ jobs:
# Rebuilds the query-regression ECS runner image via the ops tool in
# .github/runner-scale-sets/query-regression/ecs-image/: boots a
# temporary pay-as-you-go ECS builder, snapshots a new custom image,
# then completes the documented lockstep updates in order:
# then completes the documented lockstep updates:
# 1. bumps RUNNER_IMAGE_EPOCH in query-regression.yml (target-cache
# invalidation) and pushes the commit to main, and only then
# invalidation) via a PR from a timestamped ci/ branch (main is
# branch-protected; mirrors update-dev-builder-version.sh), and
# 2. points the repo variable QUERY_REGRESSION_ECS_IMAGE_ID at the
# new image, so the next regression run picks up image and epoch
# together.
# new image. The runner toolchain is pinned inside the Dockerfile
# (nightly-2026-03-21, independent of rust-toolchain.toml — the
# benchmark must compile historical base checkouts), so image
# rebuilds do not change the cache ABI.
#
# Required repository configuration (same names the provisioning job
# uses): vars ALIYUN_ECS_REGION_ID, ALIYUN_ECS_VSWITCH_ID,
# ALIYUN_ECS_SECURITY_GROUP_ID, optionally ALIYUN_ECS_RESOURCE_GROUP_ID
# and ALIYUN_ECS_BASE_IMAGE_ID (deterministic base-image pin; otherwise
# the rebuild auto-resolves the latest public Ubuntu 24.04 image).
# the rebuild auto-resolves the latest public Ubuntu 26.04 image).
# Secrets: ALICLOUD_ECS_ACCESS_KEY_ID, ALICLOUD_ECS_ACCESS_KEY_SECRET,
# GH_PERSONAL_ACCESS_TOKEN (repo push + actions-variable write).
name: Rebuild query-regression runner image
@@ -392,7 +399,7 @@ jobs:
echo "## Rebuilt runner image" >> "$GITHUB_STEP_SUMMARY"
echo "New image: \`${image_id}\`" >> "$GITHUB_STEP_SUMMARY"
- name: Bump RUNNER_IMAGE_EPOCH and update the image id variable
- name: Open the epoch-bump PR and update the image id variable
env:
GH_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
run: |
@@ -405,21 +412,42 @@ jobs:
exit 1
fi
new_epoch=$((old_epoch + 1))
# main is branch-protected (required reviews + status checks), so
# the epoch bump cannot be pushed there directly. Mirror
# .github/scripts/update-dev-builder-version.sh: timestamped bot
# branch, plain push, and a PR with reviewers. Do NOT use [skip ci]
# on the commit: the required checks must be able to run for the PR
# to become mergeable.
BRANCH="ci/update-query-regression-epoch-$(date +%Y%m%d%H%M%S)"
git config user.name "greptimedb-ci"
git config user.email "greptimedb-ci@greptime.com"
git fetch origin main
git checkout -b "${BRANCH}" origin/main
sed -i "s/readonly RUNNER_IMAGE_EPOCH=\"${old_epoch}\"/readonly RUNNER_IMAGE_EPOCH=\"${new_epoch}\"/" \
.github/workflows/query-regression.yml
git config user.name "greptimedb-ci"
git config user.email "greptimedb-ci@users.noreply.github.com"
git add .github/workflows/query-regression.yml
git commit -m "ci(query-regression): bump RUNNER_IMAGE_EPOCH to ${new_epoch} for image ${image_id} [skip ci]"
git pull --rebase origin main
git push origin HEAD:main
git commit -s -m "ci(query-regression): bump RUNNER_IMAGE_EPOCH to ${new_epoch} for image ${image_id}"
git push origin "${BRANCH}"
# Only point the variable at the new image after the epoch commit
# landed, so the next run picks up image and epoch together.
gh pr create \
--title "ci(query-regression): bump RUNNER_IMAGE_EPOCH to ${new_epoch}" \
--body "Auto-generated by the \`Rebuild query-regression runner image\` job in ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID} for image \`${image_id}\`. Merging this invalidates the query-regression target cache (RUNNER_IMAGE_EPOCH ${old_epoch} → ${new_epoch}). The \`QUERY_REGRESSION_ECS_IMAGE_ID\` repo variable already points at the new image; the epoch bump is belt-and-suspenders for image-content drift." \
--base main \
--head "${BRANCH}" \
--reviewer sunng87 \
--reviewer daviderli614 \
--reviewer killme2008 \
--reviewer evenyag \
--reviewer fengjiachun \
--reviewer WenyXu
# Point the variable at the new image once the PR exists. Image
# rebuilds keep the same pinned toolchain, so the cache ABI is
# unchanged and not waiting for the PR merge is fine.
gh api -X PATCH "repos/${{ github.repository }}/actions/variables/QUERY_REGRESSION_ECS_IMAGE_ID" \
-f value="${image_id}"
echo "## Lockstep updates" >> "$GITHUB_STEP_SUMMARY"
echo "- QUERY_REGRESSION_ECS_IMAGE_ID → \`${image_id}\`" >> "$GITHUB_STEP_SUMMARY"
echo "- RUNNER_IMAGE_EPOCH → \`${new_epoch}\` (committed to main)" >> "$GITHUB_STEP_SUMMARY"
echo "- RUNNER_IMAGE_EPOCH → \`${new_epoch}\` (PR on \`${BRANCH}\`, pending review/merge)" >> "$GITHUB_STEP_SUMMARY"