greptimedb/.github/runner-scale-sets/query-regression

Query regression self-hosted runners

The Query Regression workflow targets self-hosted GitHub Actions runners via runner labels or ARC runner scale set names:

• perf-regression-8-cores

The name intentionally avoids generic labels such as ubuntu-22.04-8-cores, which may already be used by GitHub-hosted larger runners or other runner pools.

For Kubernetes-based runners, ARC runner pods run inside the target Kubernetes cluster and connect outbound to GitHub; GitHub can then dispatch jobs whose runs-on value matches the runner scale set name or runner labels.

Prerequisites

Install the ARC scale set controller in the Kubernetes cluster if it is not already installed:

helm upgrade --install arc \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
  --namespace arc-systems \
  --create-namespace \
  --version 0.14.2

Create the GitHub App or PAT secret in the namespace that will host the runner scale sets. Prefer a GitHub App with access limited to GreptimeTeam/greptimedb.

kubectl -n arc-runners create secret generic greptimedb-arc-github-app \
  --from-literal=github_app_id=<app-id> \
  --from-literal=github_app_installation_id=<installation-id> \
  --from-file=github_app_private_key=<private-key.pem>

The values files in this directory reference that secret by name.

Build the runner image

Use a derived ARC runner image instead of the minimal upstream runner image. The image keeps the official /home/runner/run.sh entrypoint layout and adds the tools this workflow expects to be present on a normal CI host, including wget, uv, C/C++ build tools, OpenSSL headers, and protobuf tooling.

docker build \
  -f .github/runner-scale-sets/query-regression/Dockerfile \
  -t greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest \
  .github/runner-scale-sets/query-regression

docker push greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest

Deploy the runner image by digest rather than by a mutable tag. Update values-8-cores.yaml after pushing a rebuilt image.

The workflow still runs setup actions for pinned Rust and uv behavior. mold is installed in the image and selected through CARGO_BUILD_RUSTFLAGS, so jobs do not need privileged package installation at runtime.

Decide whether the registry repository is public or private. Public pull access avoids distributing registry credentials to the runner namespace. If the image must be private, create a dedicated read-only image-pull secret for this image and attach it only as imagePullSecrets; do not expose registry credentials to runner containers.

Install the query-regression scale set

Install the runner scale set. The Helm release name and runnerScaleSetName should match the runs-on value used by the workflow.

helm upgrade --install perf-regression-8-cores \
  oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
  --namespace arc-runners \
  --create-namespace \
  --version 0.14.2 \
  -f .github/runner-scale-sets/query-regression/values-8-cores.yaml

Check registration and pods:

kubectl -n arc-runners get pods
kubectl -n arc-runners get autoscalingrunnersets

The scale sets should also appear under repository Actions runner settings.

Security notes for fork PRs

Maintainer-approved fork PRs can run on self-hosted runners. Approval only lets the workflow execute; it does not make the fork code trusted.

The query-regression label is the explicit trigger for PR runs. Updating a PR does not rerun the benchmark automatically; remove and re-add the label after reviewing the updated changes.

These runners execute PR code and should be treated as untrusted execution capacity:

• keep them isolated from sensitive internal services unless explicitly required;
• do not mount host paths, Docker socket, kubeconfig, or long-lived credentials;
• use ephemeral runner pods and no shared work directory with trusted jobs;
• disable service account token mounting in runner pods unless Kubernetes API access is intentionally required;
• use a runner image whose default user is non-root, disable privilege escalation, drop Linux capabilities, and use the runtime-default seccomp profile;
• keep GitHub tokens least-privilege and rely on normal pull_request behavior for fork PRs, where repository secrets are withheld and GITHUB_TOKEN is read-only;
• review fork workflow changes before approving the run.

If stronger isolation is required, install a separate runner group/namespace just for query-regression PR workloads and restrict repository/workflow access to the Query Regression workflow.

Use namespace or cluster network policy to restrict runner egress where the CNI supports the needed controls. Query-regression runners need outbound access to GitHub Actions services, GitHub artifact/cache endpoints, Rust/crate/toolchain endpoints, DNS, and the configured image registries. Block access to unrelated cluster services, private network ranges, and cloud metadata endpoints unless a case explicitly needs them.

Build cache

The workflow builds base and candidate in the same job, the same source path, and a shared CARGO_TARGET_DIR. It checks out the base ref into src, builds and copies the base binary aside, then resets that same src checkout to the candidate ref before building the candidate binary. Keeping the workspace path stable improves Cargo incremental reuse for local workspace crates compared with building separate base-src and candidate-src checkouts.

The workflow also uses the GitHub Actions Rust cache for restore-only cache reuse. PR and dispatch runs do not save cache entries. Refresh shared caches from trusted maintenance workflows only.

The ARC values in this directory do not configure a cross-run runner-local compiler cache. If cross-run compile time still dominates the benchmark, deployers may add one of the following at the runner infrastructure layer:

• a custom runner image with Rust tooling, mold, and sccache preinstalled;
• RUSTC_WRAPPER=sccache plus a shared sccache backend such as object storage, Redis, or a Kubernetes storage class that is safe for concurrent runner pods;
• separate cache namespaces/buckets for untrusted PR code and trusted branches to avoid cache poisoning across trust boundaries.

Do not mount a shared writable host path, Docker socket, kubeconfig, or other privileged credentials into runners that execute PR code.