rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-09-08 22:48:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci: backport Rust query regression and ephemeral ECS runners

Browse Source 
Backport #8651, #8937, #8986 and #9005. Preserve the five v1.2 routine cases and map tooling checks into the release CI workflow.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
This commit is contained in:
discord9
2026-09-08 20:52:51 +08:00
parent ad2b75a0c6
commit c8ace87cf0
42 changed files with 7492 additions and 2881 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/runner-scale-sets/query-regression/README.md
+158 -295
View File
@@ -1,321 +1,184 @@
# Query regression self-hosted runners
The `Query Regression` workflow uses the dedicated ARC runner scale set
`perf-regression-8-cores`. ARC runner Pods run in the target Kubernetes cluster
and connect outbound to GitHub. The live scale set is currently **paused**:
`minRunners=0`, `maxRunners=0`, and no runner Pods. Do not resume it without
explicit approval.
The `Query Regression` workflow runs on **Aliyun ECS ephemeral runners**
(`aliyun-ecs`, the default and only automated path): a `provision` job on
`ubuntu-latest` creates one pay-as-you-go ECS instance per run, and the
instance registers itself as an ephemeral GitHub runner with a per-run
label. A `teardown` job (`if: always()`) deletes the instance;
`query-regression-janitor.yml` sweeps tagged leftovers older than 4 hours
daily.
## Prerequisites and trust admission
Build caches live on the instance's system disk, so **every run compiles
cold**; only the within-run reuse (base build warms the candidate build
through the shared target dir and sccache) applies. There is no retained
data disk.
Install the ARC scale set controller if it is not already installed:
Dispatching with any other `runner` value uses it as a literal self-hosted
runner label, which is how a manually prepared host (see
`ecs-image/bootstrap-runner-host.sh`) runs the workflow. For PR labels, set
the repository variable `QUERY_REGRESSION_PR_RUNNER` to such a label to
redirect PR runs away from ECS.
The office ARC scale set `perf-regression-8-cores` that previously ran this
workflow is retired; see git history for its values files and pause/deploy
procedures. Tearing down the office cluster (helm release, runner namespace,
and the `query-regression-build-cache` PVC) is a manual operator action
outside this repository. This directory keeps its historical
`runner-scale-sets` name for path stability.
## Nightly vs previous nightly
`query-regression-nightly.yml` runs after a successful `GreptimeDB Nightly
Build` (`workflow_run`). It resolves that run's `head_sha` as the candidate
and the previous successful nightly (same branch, typically Friday when
Monday's nightly fires) as the base, then calls `query-regression.yml` with
those immutable SHAs. Both binaries are still compiled on the ECS runner;
this is not an artifact-download path. `workflow_dispatch` can pass explicit
`base_ref` / `candidate_ref` or a nightly run id. If there is no previous
nightly, or both nightlies built the same commit, the comparison is skipped.
## Aliyun ECS path
Configuration lives in repository variables/secrets:
| Kind | Name | Purpose |
| --- | --- | --- |
| secret | `ALICLOUD_ECS_ACCESS_KEY_ID` / `ALICLOUD_ECS_ACCESS_KEY_SECRET` | RAM user scoped to ECS RunInstances/DeleteInstances/Describe*/CreateImage/RunCommand. Used only by provision/teardown jobs on `ubuntu-latest`; never reaches the ECS instance. |
| secret | `GH_PERSONAL_ACCESS_TOKEN` | Creates the short-lived runner registration token (shared with the jsonbench EC2 path). |
| vars | `ALIYUN_ECS_REGION_ID` / `ALIYUN_ECS_VSWITCH_ID` / `ALIYUN_ECS_SECURITY_GROUP_ID` | Network placement. The security group should allow egress only; no inbound rules are needed. The vSwitch pins the zone. |
| vars | `ALIYUN_ECS_INSTANCE_TYPE` | Dedicated (non-burstable, non-shared) instance family. Prefer 32 GiB (e.g. `ecs.g8i.2xlarge`); `ecs.c9i.2xlarge` is 8c16g and nightly thin-LTO of greptime peaks above that. Both base and candidate clusters run on the same machine, so noisy neighbors break thresholds. |
| vars | `QUERY_REGRESSION_ECS_IMAGE_ID` | Custom image built by `ecs-image/build-ecs-image.py`. |
The system disk is 50 GiB, which covers the image, a 16 GiB swapfile, the
checkout, and cold build caches (target dir, cargo registry, sccache). ENOSPC
stops the runner itself from writing logs, which GitHub reports as `The
operation was canceled` with no telemetry, indistinguishable from a
platform-side cancellation.
cloud-init masks `systemd-oomd`, disables `unattended-upgrades` /
`apt-daily-upgrade`, creates `/swapfile`, and sets `OOMPolicy=continue`
on the runner unit. Ubuntu 24.04 defaults to `DefaultOOMPolicy=stop`,
which SIGTERM-s the whole unit when rustc is OOM-killed and GitHub
reports `The operation was canceled` with no telemetry. Unattended
upgrades can do the same via `systemctl restart` of the runner after a
library update; GitHub then records `UserCancelled` even though the job
was still valid. Swap is a safety net for 16 GiB types, not a substitute
for 32 GiB; linking on swap is slow.
When a run dies with an unexplained `The operation was canceled` (no
"Canceled by" banner, healthy machine), re-dispatch with `keep_instance`
checked: the teardown job is skipped and the instance survives for
post-mortem inspection. The security group is egress-only, so inspect via
Cloud Assistant (`RunCommand`) or VNC: the runner's `_diag` logs under the
runner home record reconnects and worker crashes, `journalctl -u
ephemeral-github-runner.service` mirrors the console stream, `dmesg -T`
shows kernel OOM kills, and `machine-telemetry.log` in the job workspace
has the 30 s sampler history. The janitor sweep still deletes the instance
after its TTL, so finish the inspection within that window.
Trust model on the ECS path: the instance receives only the one-hour runner
registration token via user data and holds no cloud credentials; the Aliyun
AK/SK exist only in the control-plane jobs. Instance tags
(`managed-by=query-regression-ci`, `query-regression-run-id`) feed the janitor
sweep.
### Building and updating the ECS image
The runner `Dockerfile` in the parent directory stays the single source of the
tool contract. Build a new ECS image from it:
```bash
helm upgrade --install arc \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
--namespace arc-systems \
--create-namespace \
--version 0.14.2
ALIBABA_CLOUD_ACCESS_KEY_ID=... ALIBABA_CLOUD_ACCESS_KEY_SECRET=... \
uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \
--region-id <region> --vswitch-id <vsw-...> --security-group-id <sg-...> \
--base-image-id <ubuntu-24.04-image-id>
```
Create the GitHub App secret in the runner namespace. Prefer an App limited to
`GreptimeTeam/greptimedb`:
The script boots a temporary builder instance, `docker build`s the runner
image, materializes `/opt/rustup`, `/opt/cargo`, `/usr/local/bin` tools, and
`/home/runner` (actions-runner) onto the host, installs the ephemeral-runner
systemd unit from `ecs-image/`, and snapshots a custom image. It prints the
image id; set it as `QUERY_REGRESSION_ECS_IMAGE_ID`, and bump
`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time so the target
cache invalidates. Builder sentinel polling requires the Cloud Assistant
agent, which Aliyun public Ubuntu images include.
```bash
kubectl -n arc-runners create secret generic greptimedb-arc-github-app \
--from-literal=github_app_id=<app-id> \
--from-literal=github_app_installation_id=<installation-id> \
--from-file=github_app_private_key=<private-key.pem>
```
The values files here reference that secret by name.
## Trust admission for PR runs
A maintainer applying the `query-regression` or `heavy-regression` label is
**trust admission for that exact PR revision**. `query-regression` runs the five
routine default cases; `heavy-regression` runs only the high-cardinality
`prom_remote_write_7913` remote-write case. The admitted job may use this scale
set's dedicated, writable persistent cache. `pull_request: labeled` is the only
PR trigger: the label event snapshots its merge, head, and base SHAs. A queued
job fetches that immutable event merge SHA directly, verifies it is a two-parent
merge whose parents include the snapshotted head exactly once, and uses its
other parent as the actual base build revision. The snapshotted event base is
retained for audit only, so a difference from the merge's non-head parent is
not a failure. The job never follows a newer mutable PR merge ref. An
unavailable event merge, or one that does not contain exactly one snapshotted
head parent, fails closed. A later PR head change does not retarget an already
queued run: it may execute only its previously trusted event revision if that
revision remains fetchable. To run the new revision, the maintainer must review
it, remove the label, and re-add the desired regression label; cancel the old
run if it is no longer wanted. An existing label does not automatically rerun
the benchmark.
**trust admission for that exact PR revision**. `query-regression` runs the
five routine default cases; `heavy-regression` runs only the high-cardinality
`prom_remote_write_7913` remote-write case. `pull_request: labeled` is the only PR
trigger: the label event snapshots its merge, head, and base SHAs. A queued
job fetches that immutable event merge SHA directly, verifies it is a
two-parent merge whose parents include the snapshotted head exactly once, and
uses its other parent as the actual base build revision. The snapshotted
event base is retained for audit only, so a difference from the merge's
non-head parent is not a failure. The job never follows a newer mutable PR
merge ref. An unavailable event merge, or one that does not contain exactly
one snapshotted head parent, fails closed. A later PR head change does not
retarget an already queued run: it may execute only its previously trusted
event revision if that revision remains fetchable. To run the new revision,
the maintainer must review it, remove the label, and re-add the desired
regression label; cancel the old run if it is no longer wanted. An existing
label does not automatically rerun the benchmark.
Admission does not relax runner hardening or GitHub permissions. Keep
service-account token mounting disabled; do not mount host paths, the Docker
socket, kubeconfig, or long-lived credentials. The runner and cache initializer
use UID/GID 1001, disallow privilege escalation, drop all capabilities, and use
the RuntimeDefault seccomp profile. Keep GitHub tokens least-privilege and
review workflow changes before admission. Where the CNI supports it, restrict
egress to required GitHub Actions, artifact/cache, Rust/crate/toolchain, DNS,
and image-registry endpoints; block unrelated cluster services, private ranges,
and metadata endpoints unless a case requires them.
### Network routing prerequisite
Required split routing is an **external environment-specific prerequisite**. The
responsible network operator must route GitHub Actions, GitHub content,
artifact/cache, crates.io, Rust toolchain, and image-registry traffic through
the approved path rather than the VPN where applicable. Neither this repository
nor Kubernetes configures that route. Verify it with the responsible network
operator before any canary.
Admission does not relax runner hardening or GitHub permissions. Keep the ECS
instance free of cloud credentials and long-lived tokens, keep the security
group egress-only, keep GitHub tokens least-privilege, and review workflow
changes before admission.
## Runner image and workflow tools
Build and push the derived runner image; it preserves the official
`/home/runner/run.sh` entrypoint and supplies CI tools needed at runtime. The
image builds `otelgen` from
The runner `Dockerfile` builds `otelgen` from
[`WenyXu/otelgen`](https://github.com/WenyXu/otelgen) commit
[`863a3f395d062c7322cc1de08a38774b7fdaa6c8`](https://github.com/WenyXu/otelgen/commit/863a3f395d062c7322cc1de08a38774b7fdaa6c8)
so trace cases do not download or compile tools during a benchmark run:
so trace cases do not download or compile tools during a benchmark run. It is
built only as a toolchain factory: `build-ecs-image.py` and
`bootstrap-runner-host.sh` both materialize its contents onto a host, and the
benchmark itself runs host-native.
```bash
docker build \
--platform linux/amd64 \
-f .github/runner-scale-sets/query-regression/Dockerfile \
-t greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest \
.github/runner-scale-sets/query-regression
docker push greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest
```
Deploy by digest, not mutable tag, by updating both image references in
`values-8-cores.yaml` after a rebuild. Update `RUNNER_IMAGE_DIGEST` and bump
`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time. If the registry
is private, use a dedicated read-only pull secret only as `imagePullSecrets`;
never expose registry credentials to runner containers.
Both digest-pinned init and runner containers use `IfNotPresent`: the immutable
digest makes a cached image safe and avoids adding a registry dependency to every
runner startup.
The runner optionally imports only the non-sensitive `HTTP_PROXY`,
`HTTPS_PROXY`, and `NO_PROXY` variables from the
`query-regression-runner-local-env` ConfigMap. Manage that ConfigMap locally in
the target namespace; private endpoint configuration must not be committed, and
credentials or secrets must never be placed in a ConfigMap.
Before builds, the workflow asserts UID/GID 1001 and exact image tool versions:
`libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.30.0`, `Python 3.12.3`, `sccache
0.16.0`, `otelgen` commit `863a3f395d062c7322cc1de08a38774b7fdaa6c8`,
root-owned `rustup 1.29.0`, and the image-baked
`nightly-2026-03-21` Rust toolchain. Rustup, Cargo, and Rustc must resolve from
`/opt/cargo/bin`; the runner cannot write `/opt/rustup` or `/opt/cargo/bin`.
Protobuf well-known includes, including `google/protobuf/any.proto` and
`google/protobuf/empty.proto`, are an image contract and must compile with
`protoc`.
Before builds, the workflow asserts the runner UID/GID (1001 in the ECS image;
overridable via `QUERY_REGRESSION_RUNNER_UID`/`QUERY_REGRESSION_RUNNER_GID`)
and exact tool versions: `libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.40.4`,
`Python 3.14.4`, `sccache 0.16.0`, `otelgen` commit
`863a3f395d062c7322cc1de08a38774b7fdaa6c8`, root-owned `rustup 1.29.0`, and
the image-baked `nightly-2026-03-21` Rust toolchain. `mold` and `python3`
come from apt at image-build time (not Ubuntu 24.04's default 3.12); bump
the Verify pins together with `QUERY_REGRESSION_ECS_IMAGE_ID` when the
image is rebuilt. Rustup, Cargo, and Rustc
must resolve from `/opt/cargo/bin`; the runner cannot write `/opt/rustup` or
`/opt/cargo/bin`. Protobuf well-known includes, including
`google/protobuf/any.proto` and `google/protobuf/empty.proto`, are an image
contract and must compile with `protoc`.
`actions-rust-lang/setup-rust-toolchain@v1` is intentionally removed. The
workflow sets its warning-denying mold `RUSTFLAGS` directly, disables automatic
Rustup installation, and performs no runtime toolchain downloads.
workflow sets its warning-denying mold `RUSTFLAGS` directly, disables
automatic Rustup installation, and performs no runtime toolchain downloads.
The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`, or
runtime Rust setup: the image establishes immutable executable state and the PVC
supplies only reusable Cargo data. Do not reintroduce those actions unless the
corresponding cache or image contract changes.
The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`,
or runtime Rust setup: the image establishes immutable executable state and
each instance's system disk holds only that run's Cargo data. Do not
reintroduce those actions unless the image contract changes.
## Capacity and persistent cache
## Capacity
`values-8-cores.yaml` is normal operation: `minRunners=0`, `maxRunners=1`.
`values-paused.yaml` is the mandatory pause overlay: `minRunners=0`,
`maxRunners=0`. The job uses group `query-regression-persistent-cache-v1`,
`queue: max`, and `cancel-in-progress: false`; admitted jobs queue rather than
replacing older pending jobs. During maintenance, cancel admitted queued runs as
well as pausing ARC. Runner Pods have `activeDeadlineSeconds=12600`.
The runner requests 6 CPU and limits at 8 CPU to preserve `minipc-3`
allocatable-capacity scheduling headroom; do not reset the request to 8 CPU
without revalidating scheduling capacity.
Each run provisions its own ECS instance, so overlapping dispatches proceed
in parallel. There is no workflow `concurrency` group. All Cargo state
(`CARGO_HOME` including registry/git, `CARGO_TARGET_DIR`, sccache, cache
metadata) lives on the 50 GiB system disk and is discarded with the VM.
`RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are image-owned. The runner
sets `RUSTC_WRAPPER=/usr/local/bin/sccache`,
`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=10G`, and
`CARGO_INCREMENTAL=0`. sccache uses its local disk backend and self-evicts
at 10G. Base and candidate builds share the target on that disk; Cargo
fingerprints invalidate source and dependency changes.
The cache claim `query-regression-build-cache` is a nominal 600Gi `local-path`
PVC in `arc-runners`. It is `ReadWriteOnce`; `local-path` uses
WaitForFirstConsumer binding and Delete reclaim behavior, produces a
node-affine local PV, is non-expandable, and the 600Gi request is not a hard
storage quota. The runner's `minipc-3` selector is its only consumer candidate.
The initializer mounts the PVC root at `/cache`, creates and write-tests these
versioned subpaths as non-root UID/GID 1001, and the runner mounts them as:
| Persistent state | PVC subpath | Runner mount |
| --- | --- | --- |
| Ephemeral Cargo home | `emptyDir` | `/home/runner/.cargo` |
| Cargo registry data | `cargo-registry-v1` | `/home/runner/.cargo/registry` |
| Cargo Git data | `cargo-git-v1` | `/home/runner/.cargo/git` |
| Cargo target | `query-regression-target-v1` | `/home/runner/query-regression-target` |
| Cache metadata | `meta-v1` | `/home/runner/query-regression-cache-meta` |
| sccache local disk cache | `sccache-v1` | `/home/runner/.cache/sccache` |
| Immutable Rust toolchain | image-owned | `/opt/rustup`, `/opt/cargo/bin` |
The Pod security context uses UID/GID and `fsGroup` 1001 with
`fsGroupChangePolicy: OnRootMismatch`; no privileged `chown` or raw `hostPath`
is used. `CARGO_HOME` is a per-Pod `emptyDir`; only its nested `registry` and
`git` mounts are persistent. `RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are
image-owned immutable paths, while `CARGO_TARGET_DIR`, cache metadata, and
`SCCACHE_DIR` are persistent absolute paths. The runner sets
`RUSTC_WRAPPER=/usr/local/bin/sccache`,
`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=40G`, and
`CARGO_INCREMENTAL=0`. sccache uses its local PVC disk backend and self-evicts
at 40G; do not add runtime downloads, object storage, or a shared backend.
The repository's `.cargo/config.toml` remains a trusted per-revision build input.
In contrast, `$CARGO_HOME/config*`, credentials, installed bins, and Cargo
metadata outside the persistent `registry` and `git` data mounts are ephemeral
and cannot survive to another Pod.
The local disk backend has a one-server constraint. `maxRunners=1` and the
unchanged `query-regression-persistent-cache-v1` workflow concurrency group
serialize runs; do not increase runner capacity or relax that serialization
while this backend is in use. Base and candidate builds share the target; Cargo
fingerprints invalidate source and dependency changes. The workflow records the
sccache version and relevant environment in the target ABI marker, starts and
zeros sccache after cache and toolchain checks, shows initial/base/candidate
statistics, and resets statistics between base and candidate builds.
### Disk preflight and cleanup contract
Before applying or unpausing, verify the backing filesystem on `minipc-3` has
at least 900GiB free. The current local-path provisioner source is
`/opt/local-path-provisioner`; measure the filesystem containing it:
```bash
df -PB1G /opt/local-path-provisioner
```
The workflow reports `du`, `df -P`, human-readable free space, and inode
availability before builds and in an always-run report. Its cleanup is narrow
and non-destructive:
- warn at target size 400GiB; at 450GiB clear only the complete target root;
- warn at Cargo registry-plus-Git data size 60GiB; at 80GiB remove only
`registry/src` and `git/checkouts`, then abort if that persistent data remains
at least 80GiB;
- below 300GiB backing free space, clear the complete target root first,
remeasure, then remove only those Cargo extracted trees and checkouts; abort
if free space is still below 300GiB;
- never automatically remove Cargo registry cache/index, Git database, the
image-owned Cargo bin or Rustup toolchain, cache metadata, the self-evicting
sccache directory, or the PVC.
The target clear uses fixed absolute roots and removes all entries, including
dotfiles. After migration validation, remove obsolete `cargo-home-v1` and
`rustup-home-v1` only in explicit maintenance while ARC is 0/0 and no runner Pod
exists; they are not mounted by the current configuration.
## Deploy and pause safely
First verify the configured external network route and the disk preflight. Apply the PVC; while
the scale set is paused, it is expected to remain `Pending` because
WaitForFirstConsumer has no scheduled runner:
```bash
kubectl apply --dry-run=server \
-f .github/runner-scale-sets/query-regression/cache-pvc.yaml
kubectl apply -f .github/runner-scale-sets/query-regression/cache-pvc.yaml
```
Render normal and paused configurations. Normal values are always first; the
pause overlay is always last:
```bash
helm template perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --version 0.14.2 \
--set controllerServiceAccount.name=arc-gha-rs-controller \
--set controllerServiceAccount.namespace=arc-systems \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
helm template perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --version 0.14.2 \
--set controllerServiceAccount.name=arc-gha-rs-controller \
--set controllerServiceAccount.namespace=arc-systems \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
-f .github/runner-scale-sets/query-regression/values-paused.yaml
```
The **first post-merge Helm deployment must reconcile the release in paused
mode**. Keep the pause overlay last:
```bash
# First post-merge deployment and every return to paused mode: 0/0.
helm upgrade --install perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --create-namespace --version 0.14.2 \
--reset-values --wait \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml \
-f .github/runner-scale-sets/query-regression/values-paused.yaml
# Expect 0/0 and no runner resources before considering normal mode.
kubectl -n arc-runners get autoscalingrunnerset perf-regression-8-cores \
-o jsonpath='{.spec.minRunners}{"/"}{.spec.maxRunners}{"\n"}'
kubectl -n arc-runners get ephemeralrunners,pods \
-l actions.github.com/scale-set-name=perf-regression-8-cores
```
Only after that verification and separate explicit approval, apply normal 0/1
operation without the pause overlay:
```bash
helm upgrade --install perf-regression-8-cores \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--namespace arc-runners --create-namespace --version 0.14.2 \
--reset-values --wait \
-f .github/runner-scale-sets/query-regression/values-8-cores.yaml
```
Do not use bare `helm rollback`, `--atomic`, or `--reuse-values`: a stored
revision can restore nonzero runner capacity. Inspect rendered manifests for
capacity, the `minipc-3` selector, cache claim and mounts, initializer,
security context, and resources. After approved normal mode receives its first
canary, the PVC binds to `minipc-3`.
For local-PV node loss, cache recovery is intentionally disposable: return to
0/0, recreate the PVC on a healthy node, cold-fill it, and run a new canary.
## Canary and rollback
With explicit approval, run two identical `workflow_dispatch` canaries on
`perf-regression-8-cores`, using immutable full base and candidate commit SHAs
and `cargo_profile=nightly`. The first is the cold fill; the second verifies warm
reuse. Record the workflow's base/candidate build elapsed logs and cache
ABI-marker output, initial/base/candidate sccache statistics, and cache report.
Confirm the image tool contract (root-owned Rustup/Cargo paths, baked nightly,
and non-writable `/opt` roots) and that the ephemeral Cargo home contains only
the mounted registry/Git data before Cargo creates per-Pod state.
Obtain dependency and tool network byte counters from the configured
environment counter source, filtered to `minipc-3` and the dependency/tool
destinations.
Accept the canary only when all of the following hold:
- exactly one runner Pod runs on `minipc-3`, and both jobs use the same bound PV;
- UID/GID 1001 cache mounts are writable; the warm run does not invalidate the
target or bulk-redownload crates or toolchains; sccache reports separate base
and candidate build statistics without server or cache-path errors; immutable
Rustup/Cargo roots remain non-writable and only registry/Git data persists;
- warm base build time is at most 50% of cold base build time;
- warm dependency/tool network bytes are at most 10% of cold fill bytes;
- cache sizes remain below soft watermarks, node free space remains at least
300GiB, and the benchmark is correct without TLS EOFs or timeouts;
- the configured environment counter source confirms this traffic is outside VPN
accounting.
Immediately return to 0/0 after either canary unless ongoing normal operation
has been explicitly approved; return immediately on any traffic, cache, disk,
TLS, or correctness failure. To roll back, use the paused Helm upgrade above,
or another explicit `helm upgrade --install` with known-good values followed by
`values-paused.yaml`, `--reset-values`, and `--wait`. Do not delete the PVC
automatically; preserve it for diagnosis unless intentionally discarding cache.
The workflow reports `du` / `df` in telemetry. It does not try to reclaim
space across runs: a cold 50 GiB disk that fills up fails the build.
## Future optional phases
The current phase uses a digest-pinned image with sccache 0.16.0 and no shared
cache service. Optional follow-ups are an image additionally seeded with the
exact Rust toolchain and `cargo fetch --locked`; or an internal read/write
sccache backend or Cargo/Git mirror. Evaluate them only if persistent PVC reuse
is insufficient.
The current phase uses a materialized runner toolchain with sccache 0.16.0 and
no shared cache service. Optional follow-ups are an image additionally seeded
with `cargo fetch --locked` results, or an internal read/write sccache
backend or Cargo/Git mirror. Evaluate them only if cold compile time on the
system disk becomes the bottleneck.
.github/runner-scale-sets/query-regression/cache-pvc.yaml
-15
View File
@@ -1,15 +0,0 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: query-regression-build-cache
namespace: arc-runners
labels:
app.kubernetes.io/part-of: perf-regression-8-cores
spec:
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 600Gi
.github/runner-scale-sets/query-regression/ecs-image/bootstrap-runner-host.sh
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Bootstrap an Ubuntu 24.04 host as a query-regression self-hosted runner.
#
# This is the host-native equivalent of the ECS image contract: it
# installs docker-ce from Docker's official repository, builds the runner
# image from the sibling Dockerfile (the single source of the tool contract),
# materializes the toolchain onto the host, and optionally registers the
# runner and installs its systemd service.
#
# Large state lives under --data-root (a data disk): docker's data-root, the
# runner home, and the rustup/cargo toolchain roots, symlinked back to the
# hard-coded contract paths (/home/runner, /opt/rustup, /opt/cargo).
#
# The script is idempotent: re-running refreshes the toolchain in place, and
# the runner registration (.runner/.credentials) and _work survive. A live
# runner service is stopped first and restarted at the end.
#
# Usage:
# sudo bash bootstrap-runner-host.sh --repo-dir /path/to/greptimedb
# sudo RUNNER_TOKEN=<token> bash bootstrap-runner-host.sh --repo-dir . \
# --register --repo-url https://github.com/<owner>/<repo>
#
# Options:
# --repo-dir PATH greptimedb checkout containing the runner Dockerfile (required)
# --data-root PATH data disk mount point (default: /data)
# --uid / --gid N runner user id (default: 3141; the ECS image contract is 1001)
# --register also register the runner and install the systemd service
# --repo-url URL repository URL for registration (required with --register)
# --runner-name N runner name (default: qreg-host)
# --labels L runner labels (default: perf-regression-8-cores)
#
# With --register, provide a fresh registration token via RUNNER_TOKEN.
set -euo pipefail
REPO_DIR=""
DATA_ROOT="/data"
RUNNER_UID="3141"
RUNNER_GID="3141"
REGISTER="false"
REPO_URL=""
RUNNER_NAME="qreg-host"
RUNNER_LABELS="perf-regression-8-cores"
IMAGE_TAG="qreg-runner:manual"
while [[ $# -gt 0 ]]; do
case "$1" in
--repo-dir) REPO_DIR="$2"; shift 2 ;;
--data-root) DATA_ROOT="$2"; shift 2 ;;
--uid) RUNNER_UID="$2"; shift 2 ;;
--gid) RUNNER_GID="$2"; shift 2 ;;
--register) REGISTER="true"; shift ;;
--repo-url) REPO_URL="$2"; shift 2 ;;
--runner-name) RUNNER_NAME="$2"; shift 2 ;;
--labels) RUNNER_LABELS="$2"; shift 2 ;;
-h | --help) sed -n '17,40p' "$0"; exit 0 ;;
*) echo "Unknown argument: $1" >&2; exit 2 ;;
esac
done
[[ $EUID -eq 0 ]] || { echo "Run as root, e.g. sudo bash $0 ..." >&2; exit 1; }
[[ -n "${REPO_DIR}" ]] || { echo "--repo-dir is required" >&2; exit 2; }
DOCKERFILE="${REPO_DIR}/.github/runner-scale-sets/query-regression/Dockerfile"
[[ -f "${DOCKERFILE}" ]] || { echo "Dockerfile not found at ${DOCKERFILE}" >&2; exit 1; }
if [[ "${REGISTER}" == "true" ]]; then
[[ -n "${REPO_URL}" ]] || { echo "--repo-url is required with --register" >&2; exit 2; }
[[ -n "${RUNNER_TOKEN:-}" ]] || { echo "RUNNER_TOKEN env is required with --register" >&2; exit 2; }
fi
step() { printf '\n==> %s\n' "$*"; }
# A re-run refreshes files under the (possibly live) runner service, so stop
# it first and restart it at the end (unless --register re-creates it).
RUNNER_SERVICE="$(systemctl list-units --type=service --all --no-legend 'actions.runner.*' 2>/dev/null | awk '{print $1}' | head -n 1 || true)"
RUNNER_WAS_ACTIVE="false"
if [[ -n "${RUNNER_SERVICE}" ]] && systemctl is-active --quiet "${RUNNER_SERVICE}"; then
step "Stop running runner service ${RUNNER_SERVICE}"
systemctl stop "${RUNNER_SERVICE}"
RUNNER_WAS_ACTIVE="true"
fi
step "Install docker-ce from Docker's official repository"
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
apt-get remove -y "${pkg}" 2>/dev/null || true
done
apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
# shellcheck disable=SC1091
. /etc/os-release
tee /etc/apt/sources.list.d/docker.sources > /dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: ${UBUNTU_CODENAME:-$VERSION_CODENAME}
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
step "Move docker data-root to ${DATA_ROOT}/docker"
systemctl stop docker containerd
mkdir -p "${DATA_ROOT}/docker" "${DATA_ROOT}/containerd"
if [[ -f /etc/docker/daemon.json ]] && ! grep -q '"data-root"' /etc/docker/daemon.json; then
echo "ERROR: /etc/docker/daemon.json exists without data-root; merge manually." >&2
exit 1
fi
echo "{ \"data-root\": \"${DATA_ROOT}/docker\" }" > /etc/docker/daemon.json
# containerd keeps its own content store (where build-time image data lands);
# move it off the system disk too.
mkdir -p /etc/containerd
containerd config default | sed "s|root = \"/var/lib/containerd\"|root = \"${DATA_ROOT}/containerd\"|" > /etc/containerd/config.toml
systemctl start containerd docker
docker info | grep "Docker Root Dir"
step "Build the runner image"
docker build --platform linux/amd64 -f "${DOCKERFILE}" -t "${IMAGE_TAG}" "${REPO_DIR}"
step "Materialize the toolchain onto the host"
mkdir -p "${DATA_ROOT}/opt" "${DATA_ROOT}/runner" /opt /home
container="$(docker create "${IMAGE_TAG}")"
trap 'docker rm -f "${container}" >/dev/null 2>&1 || true' EXIT
# The `/.` suffix copies directory *contents*, so re-running over an existing
# materialization refreshes it in place instead of nesting runner/runner.
# Files absent from the image (runner registration, _work) are preserved.
docker cp "${container}:/home/runner/." "${DATA_ROOT}/runner"
docker cp "${container}:/opt/rustup/." "${DATA_ROOT}/opt/rustup"
docker cp "${container}:/opt/cargo/." "${DATA_ROOT}/opt/cargo"
for tool in uv uvx otelgen sccache; do
docker cp "${container}:/usr/local/bin/${tool}" "/usr/local/bin/${tool}"
done
docker rm "${container}" > /dev/null
trap - EXIT
ln -sfn "${DATA_ROOT}/runner" /home/runner
ln -sfn "${DATA_ROOT}/opt/rustup" /opt/rustup
ln -sfn "${DATA_ROOT}/opt/cargo" /opt/cargo
docker image rm "${IMAGE_TAG}" > /dev/null
step "Install system packages"
apt-get install -y --no-install-recommends \
build-essential clang cmake git gzip jq libprotobuf-dev libssl-dev mold \
openssh-client pkg-config protobuf-compiler python3 sudo tar unzip wget xz-utils zip zstd
step "Create runner user (${RUNNER_UID}:${RUNNER_GID}) and environment"
getent group "${RUNNER_GID}" > /dev/null || groupadd -g "${RUNNER_GID}" runner
id -u runner > /dev/null 2>&1 || useradd -u "${RUNNER_UID}" -g "${RUNNER_GID}" -d /home/runner -s /bin/bash runner
chown -R "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner"
echo 'PATH=/opt/cargo/bin:/usr/local/bin:/usr/bin:/bin' > "${DATA_ROOT}/runner/.env"
chown "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner/.env"
if [[ "${REGISTER}" == "true" ]]; then
step "Register runner ${RUNNER_NAME} (labels: ${RUNNER_LABELS})"
if [[ -n "${RUNNER_SERVICE}" ]]; then
(cd /home/runner && ./svc.sh uninstall) || true
fi
runuser -u runner -- bash -c "cd /home/runner && HOME=/home/runner ./config.sh \
--url '${REPO_URL}' --token '${RUNNER_TOKEN}' --name '${RUNNER_NAME}' \
--labels '${RUNNER_LABELS}' --unattended --replace --disableupdate"
step "Install and start the runner service"
cd /home/runner
./svc.sh install runner
./svc.sh start
./svc.sh status
else
step "Skipping registration (pass --register --repo-url ... with RUNNER_TOKEN to enable)"
if [[ "${RUNNER_WAS_ACTIVE}" == "true" ]]; then
step "Restart runner service ${RUNNER_SERVICE}"
systemctl start "${RUNNER_SERVICE}"
systemctl --no-pager status "${RUNNER_SERVICE}" || true
fi
fi
step "Done"
echo "Runner home: /home/runner -> ${DATA_ROOT}/runner"
echo "Before each workflow run on this persistent host, clean transient cargo state:"
echo " sudo rm -f /home/runner/.cargo/.package-cache"
echo " sudo rm -rf /home/runner/.cargo/.global-cache"
.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py
+392
View File
@@ -0,0 +1,392 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# PEP 723 inline metadata (see .github/scripts/aliyun-ecs-runner-provision.py
# for the convention).
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Build the query-regression ECS custom image (manual ops tool).
Boots a temporary pay-as-you-go ECS instance from a public Ubuntu 24.04 image,
builds the existing runner container image (the Dockerfile in the parent
directory remains the single source of the tool contract), materializes the
tool directories onto the host filesystem so the workflow's "Verify runner
image tools" step holds unchanged, installs the ephemeral-runner systemd unit,
and snapshots the result as a custom image. The temporary instance is deleted
afterwards.
Sentinel polling reads the instance's serial console output
(GetInstanceConsoleOutput) and looks for marker lines the user data writes to
/dev/console. This has no in-guest agent dependency.
Usage:
uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \
--region-id cn-hangzhou --vswitch-id vsw-... --security-group-id sg-... \
--base-image-id ubuntu_24_04_x64_20G_alibase_*.vhd
"""
from __future__ import annotations
import argparse
import base64
import os
import time
from pathlib import Path
ASSETS_DIR = Path(__file__).resolve().parent
DONE_MARKER = "QREG_IMAGE_BUILD_DONE"
FAILED_MARKER = "QREG_IMAGE_BUILD_FAILED"
POLL_INTERVAL_SECONDS = 15
CONSOLE_POLL_INTERVAL_SECONDS = 30
BUILD_TIMEOUT_SECONDS = 60 * 60
# Same apt package contract as the runner Dockerfile; the base
# actions-runner image is Ubuntu 24.04, so an Ubuntu 24.04 host resolves the
# same tool versions (protoc 3.21.12, mold 2.40.4, Python 3.14.4).
# Docker itself comes from Docker's official repository (docker-ce), not the
# distribution-packaged docker.io.
APT_PACKAGES = [
"build-essential",
"ca-certificates",
"clang",
"cmake",
"curl",
"git",
"gpg",
"gzip",
"jq",
"libprotobuf-dev",
"libssl-dev",
"mold",
"openssh-client",
"pkg-config",
"protobuf-compiler",
"python3",
"sudo",
"tar",
"unzip",
"wget",
"xz-utils",
"zip",
"zstd",
]
DOCKER_CE_PACKAGES = "docker-ce docker-ce-cli containerd.io docker-buildx-plugin"
def render_user_data(dockerfile: str, start_runner: str, unit: str) -> str:
dockerfile_b64 = base64.b64encode(dockerfile.encode()).decode()
start_runner_b64 = base64.b64encode(start_runner.encode()).decode()
unit_b64 = base64.b64encode(unit.encode()).decode()
packages = " ".join(APT_PACKAGES)
return f"""#!/bin/bash
set -euo pipefail
trap 'echo "{FAILED_MARKER} at line $LINENO" > /dev/console' ERR
# Stream the full build log to the serial console so the poller (and anyone
# watching GetInstanceConsoleOutput) sees real progress, not a silent login
# prompt for the whole build.
exec > >(tee -a /dev/console) 2>&1
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {packages}
# Docker from the official repository, not the distribution-packaged docker.io.
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
. /etc/os-release
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu ${{VERSION_CODENAME}} stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {DOCKER_CE_PACKAGES}
base64 -d > /tmp/Dockerfile <<'EOF'
{dockerfile_b64}
EOF
mkdir -p /tmp/image-context
docker build --platform linux/amd64 -f /tmp/Dockerfile -t qreg-runner:local /tmp/image-context
# Materialize the tool contract onto the host filesystem.
container="$(docker create qreg-runner:local)"
trap 'docker rm -f "${{container}}" >/dev/null 2>&1 || true' EXIT
docker cp "${{container}}:/opt/rustup" /opt/rustup
docker cp "${{container}}:/opt/cargo" /opt/cargo
for tool in uv uvx otelgen sccache; do
docker cp "${{container}}:/usr/local/bin/${{tool}}" "/usr/local/bin/${{tool}}"
done
docker cp "${{container}}:/home/runner" /home/runner
# Runner identity mirrors the container image (UID/GID 1001).
groupadd --gid 1001 runner 2>/dev/null || true
useradd --uid 1001 --gid 1001 --home-dir /home/runner --shell /bin/bash runner 2>/dev/null || true
chown -R 1001:1001 /home/runner
mkdir -p /opt/ephemeral-github-runner
base64 -d > /opt/ephemeral-github-runner/start-runner.sh <<'EOF'
{start_runner_b64}
EOF
chmod 0755 /opt/ephemeral-github-runner/start-runner.sh
base64 -d > /etc/systemd/system/ephemeral-github-runner.service <<'EOF'
{unit_b64}
EOF
systemctl daemon-reload
systemctl enable ephemeral-github-runner.service
# Keep the image free of the build-time docker state (also shrinks the
# snapshot: buildkit cache is several GB).
docker rm -f "${{container}}" >/dev/null
docker system prune -af >/dev/null
trap - EXIT
echo "{DONE_MARKER}" > /dev/console
"""
def make_ecs_client(region_id: str):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
return EcsClient(
OpenApiConfig(
access_key_id=os.environ["ALIBABA_CLOUD_ACCESS_KEY_ID"],
access_key_secret=os.environ["ALIBABA_CLOUD_ACCESS_KEY_SECRET"],
region_id=region_id,
endpoint=f"ecs.{region_id}.aliyuncs.com",
)
)
def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None:
from alibabacloud_ecs20140526 import models as ecs_models
import json
while time.monotonic() < deadline:
response = call_api_with_retry(
lambda: client.describe_instances(
ecs_models.DescribeInstancesRequest(
region_id=region_id, instance_ids=json.dumps([instance_id])
)
),
"DescribeInstances",
)
instances = response.body.instances.instance
if instances and instances[0].status == wanted:
return
time.sleep(POLL_INTERVAL_SECONDS)
raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time")
# Aliyun error codes worth retrying: throttling and server-side faults. Client
# errors (4xx: permissions, bad parameters) are configuration problems and must
# fail fast instead of being retried.
TRANSIENT_ERROR_CODES = {"Throttling", "Throttling.User", "InternalError", "ServiceUnavailable"}
def is_transient(error: Exception) -> bool:
# UnretryableException comes from the darabonba network layer: connection
# reset, read timeout, DNS blip.
if type(error).__name__ == "UnretryableException":
return True
code = getattr(error, "code", "") or ""
status = getattr(error, "statusCode", None)
return code in TRANSIENT_ERROR_CODES or (isinstance(status, int) and status >= 500)
def call_api_with_retry(fn, description: str, attempts: int = 5):
"""Retry transient API/network failures; the build is too long to die on a blip.
Only call this with idempotent or safely-repeatable requests (reads,
StopInstance, CreateImage). Never with RunInstances: if the request
succeeded but the response was lost, a retry double-creates instances.
"""
for attempt in range(1, attempts + 1):
try:
return fn()
except Exception as error: # noqa: BLE001
if attempt == attempts or not is_transient(error):
raise
print(f"{description} failed (attempt {attempt}/{attempts}): {error}", flush=True)
time.sleep(POLL_INTERVAL_SECONDS)
raise RuntimeError("unreachable: retry loop exited without returning")
def read_console_output(client, region_id: str, instance_id: str) -> str:
"""Fetch the instance's serial console output; no in-guest agent needed."""
from alibabacloud_ecs20140526 import models as ecs_models
response = call_api_with_retry(
lambda: client.get_instance_console_output(
ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id)
),
"GetInstanceConsoleOutput",
)
return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID"))
parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID"))
parser.add_argument("--base-image-id", default=os.environ.get("ALIYUN_ECS_BASE_IMAGE_ID"))
parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID"))
parser.add_argument("--instance-type", default="ecs.g7.xlarge")
parser.add_argument("--image-name", default=None, help="Defaults to a timestamped name.")
args = parser.parse_args()
for name in ("region_id", "vswitch_id", "security_group_id", "base_image_id"):
if not getattr(args, name):
raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}")
from alibabacloud_ecs20140526 import models as ecs_models
client = make_ecs_client(args.region_id)
image_name = args.image_name or time.strftime(
"greptimedb-query-regression-runner-%Y%m%d%H%M%S", time.gmtime()
)
user_data = base64.b64encode(
render_user_data(
(ASSETS_DIR.parent / "Dockerfile").read_text(),
(ASSETS_DIR / "start-runner.sh").read_text(),
(ASSETS_DIR / "ephemeral-github-runner.service").read_text(),
).encode()
).decode()
instance_id = None
try:
response = client.run_instances(
ecs_models.RunInstancesRequest(
region_id=args.region_id,
image_id=args.base_image_id,
resource_group_id=args.resource_group_id,
instance_type=args.instance_type,
v_switch_id=args.vswitch_id,
security_group_id=args.security_group_id,
instance_name=f"build-{image_name}",
description="Temporary builder for the query-regression ECS image",
amount=1,
instance_charge_type="PostPaid",
spot_strategy="NoSpot",
internet_charge_type="PayByTraffic",
internet_max_bandwidth_out=100,
# Peak usage is ~18G (OS+apt, docker image, and the materialized
# toolchain coexist briefly): deliberately tight, and a smaller
# disk makes the image snapshot faster. Instances created from
# the image get a larger system disk from the provision side.
system_disk=ecs_models.RunInstancesRequestSystemDisk(
category="cloud_essd", size="20"
),
user_data=user_data,
tag=[
ecs_models.RunInstancesRequestTag(key="managed-by", value="query-regression-ci"),
ecs_models.RunInstancesRequestTag(key="role", value="image-builder"),
],
)
)
instance_id = response.body.instance_id_sets.instance_id_set[0]
print(f"Builder instance: {instance_id}", flush=True)
wait_for_instance_status(client, args.region_id, instance_id, "Running", time.monotonic() + 10 * 60)
deadline = time.monotonic() + BUILD_TIMEOUT_SECONDS
while time.monotonic() < deadline:
console = read_console_output(client, args.region_id, instance_id)
if DONE_MARKER in console:
break
if FAILED_MARKER in console:
tail = "\n".join(console.splitlines()[-20:])
raise RuntimeError(f"Image build failed on the builder; console tail:\n{tail}")
lines = console.splitlines()
print(f"Build in progress; last console line: {lines[-1] if lines else '(none yet)'}", flush=True)
time.sleep(CONSOLE_POLL_INTERVAL_SECONDS)
else:
raise TimeoutError("Image build did not finish in time")
print("Stopping builder before image creation", flush=True)
try:
call_api_with_retry(
lambda: client.stop_instance(
ecs_models.StopInstanceRequest(
instance_id=instance_id,
# The instance is deleted right after the snapshot, but
# there is no reason to keep billing vCPU during the stop.
stopped_mode="StopCharging",
)
),
"StopInstance",
)
except Exception as error: # noqa: BLE001
# Instance families with local disks do not support StopCharging.
print(f"StopCharging unavailable ({error}); stopping with default mode", flush=True)
call_api_with_retry(
lambda: client.stop_instance(ecs_models.StopInstanceRequest(instance_id=instance_id)),
"StopInstance",
)
wait_for_instance_status(client, args.region_id, instance_id, "Stopped", time.monotonic() + 10 * 60)
image = call_api_with_retry(
lambda: client.create_image(
ecs_models.CreateImageRequest(
region_id=args.region_id, instance_id=instance_id, image_name=image_name
)
),
"CreateImage",
)
image_id = image.body.image_id
deadline = time.monotonic() + 60 * 60
while time.monotonic() < deadline:
description = call_api_with_retry(
lambda: client.describe_images(
ecs_models.DescribeImagesRequest(region_id=args.region_id, image_id=image_id)
),
"DescribeImages",
)
images = description.body.images.image
if images:
status = images[0].status
print(f"Image {image_id} status: {status} (progress {images[0].progress})", flush=True)
if status == "Available":
break
time.sleep(POLL_INTERVAL_SECONDS)
else:
raise TimeoutError(
f"Image {image_id} did not become Available in time. The snapshot "
f"continues server-side: check its status in the console and reuse it "
f"once Available instead of rebuilding."
)
print(f"Custom image ready: {image_id} ({image_name})", flush=True)
print(f"Set the repo variable QUERY_REGRESSION_ECS_IMAGE_ID={image_id}", flush=True)
return 0
finally:
if instance_id:
try:
client.delete_instance(
ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True)
)
print(f"Deleted builder instance {instance_id}", flush=True)
except Exception as error: # noqa: BLE001
print(f"Failed to delete builder instance {instance_id}: {error}", flush=True)
if __name__ == "__main__":
raise SystemExit(main())
.github/runner-scale-sets/query-regression/ecs-image/ephemeral-github-runner.service
+45
View File
@@ -0,0 +1,45 @@
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
[Unit]
Description=Ephemeral GitHub Actions runner
After=network-online.target
Wants=network-online.target
# cloud-final runs the provision user data, which writes
# /etc/ephemeral-github-runner.env and the drop-ins below. Ordering after it
# makes the runner come up configured on first boot, no restart needed.
# (cloud-init's `systemctl restart` in user data remains as the bridge for
# images built before this ordering existed.)
After=cloud-final.service
[Service]
Type=simple
# The toolchain lives in the image at /opt/cargo/bin (the Dockerfile's ENV
# PATH does not survive materialization onto the host). Publish it here so
# every runner job inherits it. /etc/ephemeral-github-runner.env currently
# carries the same PATH line as a bridge for images built before this
# directive existed; EnvironmentFile is applied after Environment=, and the
# two values are kept identical, so there is no conflict.
Environment=PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
EnvironmentFile=/etc/ephemeral-github-runner.env
ExecStart=/opt/ephemeral-github-runner/start-runner.sh
Restart=no
# If the kernel OOM killer kills a job child (rustc/cargo), keep the runner
# service alive so the job is reported as a normal failure with logs instead
# of the runner vanishing and GitHub reporting a cancellation. cloud-init's
# oom.conf drop-in carries the same directive for images built before this.
OOMPolicy=continue
[Install]
WantedBy=multi-user.target
.github/runner-scale-sets/query-regression/ecs-image/start-runner.sh
+37
View File
@@ -0,0 +1,37 @@
#!/bin/bash
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configure and start the ephemeral GitHub Actions runner on an ECS instance.
# Baked into the custom image at /opt/ephemeral-github-runner/ by
# build-ecs-image.py; the per-run environment is written by cloud-init to
# /etc/ephemeral-github-runner.env.
set -euo pipefail
# shellcheck disable=SC1091
source /etc/ephemeral-github-runner.env
export HOME=/home/runner
cd /home/runner
if [[ ! -f /home/runner/.runner ]]; then
runuser -u runner -- ./config.sh \
--url "${REPO_URL}" \
--token "${RUNNER_TOKEN}" \
--name "${RUNNER_NAME}" \
--labels "${RUNNER_LABELS}" \
--ephemeral --unattended --replace --disableupdate
fi
exec runuser -u runner -- ./run.sh
.github/runner-scale-sets/query-regression/values-8-cores.yaml
-146
View File
@@ -1,146 +0,0 @@
githubConfigUrl: "https://github.com/GreptimeTeam/greptimedb"
githubConfigSecret: greptimedb-arc-github-app
runnerScaleSetName: "perf-regression-8-cores"
minRunners: 0
maxRunners: 1
template:
spec:
automountServiceAccountToken: false
activeDeadlineSeconds: 12600
nodeSelector:
kubernetes.io/hostname: minipc-3
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
volumes:
- name: cargo-home
emptyDir: {}
- name: build-cache
persistentVolumeClaim:
claimName: query-regression-build-cache
initContainers:
- name: initialize-build-cache
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -ec
- |
umask 0002
for directory in cargo-registry-v1 cargo-git-v1 query-regression-target-v1 meta-v1 sccache-v1; do
cache_directory="/cache/${directory}"
mkdir -p "${cache_directory}"
test -w "${cache_directory}"
probe_file="${cache_directory}/.write-probe"
: > "${probe_file}"
test -f "${probe_file}"
rm "${probe_file}"
done
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources:
requests:
cpu: 100m
memory: 128Mi
ephemeral-storage: 1Gi
limits:
cpu: 500m
memory: 512Mi
ephemeral-storage: 1Gi
volumeMounts:
- name: build-cache
mountPath: /cache
containers:
- name: runner
image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11
imagePullPolicy: IfNotPresent
command: ["/home/runner/run.sh"]
env:
- name: CARGO_HOME
value: /home/runner/.cargo
- name: HTTP_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: HTTP_PROXY
optional: true
- name: HTTPS_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: HTTPS_PROXY
optional: true
- name: NO_PROXY
valueFrom:
configMapKeyRef:
name: query-regression-runner-local-env
key: NO_PROXY
optional: true
- name: UV_CACHE_DIR
value: /home/runner/.cargo/uv-cache
- name: RUSTUP_HOME
value: /opt/rustup
- name: RUSTUP_TOOLCHAIN
value: nightly-2026-03-21
- name: RUSTUP_AUTO_INSTALL
value: "0"
- name: CARGO_TARGET_DIR
value: /home/runner/query-regression-target
- name: QUERY_REGRESSION_CACHE_META
value: /home/runner/query-regression-cache-meta
- name: RUSTC_WRAPPER
value: /usr/local/bin/sccache
- name: SCCACHE_DIR
value: /home/runner/.cache/sccache
- name: SCCACHE_CACHE_SIZE
value: 40G
- name: CARGO_INCREMENTAL
value: "0"
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
resources:
requests:
cpu: "6"
memory: 24Gi
ephemeral-storage: 80Gi
limits:
cpu: "8"
memory: 24Gi
ephemeral-storage: 80Gi
volumeMounts:
- name: cargo-home
mountPath: /home/runner/.cargo
- name: build-cache
mountPath: /home/runner/.cargo/registry
subPath: cargo-registry-v1
- name: build-cache
mountPath: /home/runner/.cargo/git
subPath: cargo-git-v1
- name: build-cache
mountPath: /home/runner/query-regression-target
subPath: query-regression-target-v1
- name: build-cache
mountPath: /home/runner/query-regression-cache-meta
subPath: meta-v1
- name: build-cache
mountPath: /home/runner/.cache/sccache
subPath: sccache-v1
.github/runner-scale-sets/query-regression/values-paused.yaml
-2
View File
@@ -1,2 +0,0 @@
minRunners: 0
maxRunners: 0
.github/scripts/aliyun-ecs-runner-provision.py
+488
View File
@@ -0,0 +1,488 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is the first `.github/scripts/` user of PEP 723 inline script metadata:
# `uv run` installs the pinned Aliyun SDK before executing. The SDK import stays
# lazy (inside `make_ecs_client`) so unit tests can import this module with a
# plain stdlib interpreter.
# /// script
# requires-python = ">=3.10"
# dependencies = [
# # Pin exactly after the first live run against the Aliyun API.
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Provision an ephemeral Aliyun ECS query-regression runner.
Creates one pay-as-you-go ECS instance from the query-regression custom image
and waits until the instance registers itself as an ephemeral GitHub Actions
runner with a per-run label. Build caches live on the instance system disk
and disappear with the VM. The instance id is written to job outputs as soon
as the VM exists so the teardown job is the single DeleteInstance caller (a
timeout here used to delete and then teardown deleted again). On runner
online timeout the console is dumped and the job fails; teardown releases
the instance.
The Aliyun credentials are used only in this control-plane job (ubuntu-latest)
and are never passed to the ECS instance; the instance receives only a
short-lived GitHub runner registration token via user data.
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
RUNNER_NAME_PREFIX = "qreg-ecs"
RUNNER_LABEL_PREFIX = "query-regression-ecs"
MANAGED_BY_TAG_KEY = "managed-by"
MANAGED_BY_TAG_VALUE = "query-regression-ci"
RUN_TAG_KEY = "query-regression-run-id"
# Runner cache paths on the instance system disk. They are created empty
# every provision and deleted with the VM.
CACHE_PATHS = (
"/home/runner/.cargo/registry",
"/home/runner/.cargo/git",
"/home/runner/query-regression-target",
"/home/runner/query-regression-cache-meta",
"/home/runner/.cache/sccache",
)
RUNNER_ONLINE_TIMEOUT_SECONDS = 10 * 60
POLL_INTERVAL_SECONDS = 5
# Nightly thin-LTO of greptime peaks above ecs.c9i.2xlarge's 16 GiB. A swap
# file plus masking systemd-oomd lets the kernel reclaim rustc pages instead
# of SIGTERM-ing the runner service cgroup (GitHub then reports the step as
# "The operation was canceled"). Sized to match that 16 GiB instance; a
# 32 GiB type still benefits as a safety net. Lives on the system disk next
# to the cold build caches.
SWAP_FILE = "/swapfile"
SWAP_SIZE_GIB = 16
# Image (~12G), 16G swap, checkout, base+candidate data homes, and cold
# build caches. Deleted with the instance.
SYSTEM_DISK_GIB = 50
@dataclass(frozen=True)
class ProvisionConfig:
region_id: str
vswitch_id: str
security_group_id: str
image_id: str
instance_type: str
repo: str
run_id: str
github_token: str
# Optional resource group; required when the RAM grant is scoped to one.
resource_group_id: str | None = None
# Runner identity inside the image; the workflow asserts the same values.
runner_uid: str = "1001"
runner_gid: str = "1001"
def runner_name_for_run(run_id: str) -> str:
return f"{RUNNER_NAME_PREFIX}-{run_id}"
def runner_label_for_run(run_id: str) -> str:
return f"{RUNNER_LABEL_PREFIX}-{run_id}"
def render_user_data(
runner_name: str,
runner_label: str,
runner_token: str,
repo: str,
runner_uid: str = "1001",
runner_gid: str = "1001",
) -> str:
"""Render the cloud-init shell script for the runner instance."""
destinations = " ".join(f'"{dst}"' for dst in CACHE_PATHS)
cache_setup = f"""# Caches live on the system disk, are deleted with the instance, and every
# run compiles cold. Within-run reuse (base warming candidate via the shared
# target dir and sccache) still applies.
mkdir -p {destinations}
chown -R {runner_uid}:{runner_gid} /home/runner"""
swap_setup = f"""# Mask systemd-oomd before enabling swap: Ubuntu 24.04 kills the whole
# service cgroup on PSI pressure, and swap thrashing looks like pressure.
systemctl disable --now systemd-oomd.socket systemd-oomd.service || true
systemctl mask systemd-oomd.socket systemd-oomd.service || true
# apt-daily-upgrade / unattended-upgrades can `systemctl restart` services
# whose libraries were updated. That SIGTERM-s the runner mid-job; GitHub
# records UserCancelled (job still valid, child exit 143). These VMs live
# ~1h; security updates belong in the image, not at job time.
systemctl disable --now unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true
systemctl mask unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true
systemctl stop unattended-upgrades.service || true
killall -9 unattended-upgr || true
cat > /etc/apt/apt.conf.d/99disable-auto-updates <<'APTEOF'
APT::Periodic::Update-Package-Lists "0";
APT::Periodic::Unattended-Upgrade "0";
APT::Periodic::Download-Upgradeable-Packages "0";
APTEOF
if [[ ! -f "{SWAP_FILE}" ]]; then
fallocate --length {SWAP_SIZE_GIB}G "{SWAP_FILE}"
chmod 600 "{SWAP_FILE}"
mkswap "{SWAP_FILE}"
fi
swapon "{SWAP_FILE}"
sysctl --write vm.swappiness=10
swapon --show
free --human"""
return f"""#!/bin/bash
set -euo pipefail
{cache_setup}
{swap_setup}
cat > /etc/ephemeral-github-runner.env <<'ENVEOF'
RUNNER_NAME={runner_name}
RUNNER_LABELS={runner_label}
RUNNER_TOKEN={runner_token}
REPO_URL=https://github.com/{repo}
# The container image baked /opt/cargo/bin into PATH via Dockerfile ENV; on
# the host nothing inherits that, so publish it through the unit's
# EnvironmentFile: runner job processes inherit the runner's environment.
PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENVEOF
# Stream the runner service's own logs (config output, job lifecycle, crash
# messages) to the serial console, next to the kernel's OOM-killer records.
# Teardown dumps the console tail before deleting the instance, so the
# witness of what killed the runner survives the machine.
mkdir -p /etc/systemd/system/ephemeral-github-runner.service.d
cat > /etc/systemd/system/ephemeral-github-runner.service.d/console.conf <<'CONFEOF'
[Service]
StandardOutput=journal+console
StandardError=journal+console
CONFEOF
# systemd's DefaultOOMPolicy=stop SIGTERM-s the whole unit when *any*
# process in the cgroup is OOM-killed (typically rustc during thin LTO).
# That is the "Session terminated, killing shell..." / job Canceled path:
# runuser dies, GitHub deletes the ephemeral registration, always() steps
# never run. continue keeps the runner up so cargo can report signal 9.
cat > /etc/systemd/system/ephemeral-github-runner.service.d/oom.conf <<'CONFEOF'
[Service]
OOMPolicy=continue
CONFEOF
systemctl daemon-reload
# restart (not start): the image enables this unit, so it may already
# have come up without the drop-ins if it raced cloud-init. restart
# applies OOMPolicy=continue to the running cgroup. --no-block matters:
# newer image units carry After=cloud-final.service, and this script IS
# cloud-final — a blocking restart would deadlock until TimeoutStartSec.
systemctl restart --no-block ephemeral-github-runner.service
"""
def encode_user_data(script: str) -> str:
return base64.b64encode(script.encode("utf-8")).decode("ascii")
def github_api(token: str, method: str, path: str, body: dict | None = None) -> dict:
data = json.dumps(body).encode("utf-8") if body is not None else None
request = urllib.request.Request(
f"https://api.github.com{path}",
data=data,
method=method,
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
# GitHub's error body says exactly why (e.g. "Must have admin rights to
# Repository" for a PAT without the required scope); surface it instead
# of a bare "HTTP Error 403".
body = error.read().decode("utf-8", "replace")
raise SystemExit(
f"GitHub API {method} {path} failed: HTTP {error.code}: {body}\n"
"The token comes from the GH_PERSONAL_ACCESS_TOKEN secret; it needs "
"'repo' scope (classic PAT) or 'Administration: write' on the "
"repository (fine-grained PAT)."
) from error
def create_registration_token(github_token: str, repo: str) -> str:
response = github_api(
github_token,
"POST",
f"/repos/{repo}/actions/runners/registration-token",
body={},
)
return response["token"]
def find_runner_by_name(github_token: str, repo: str, name: str) -> dict | None:
page = 1
while True:
response = github_api(
github_token,
"GET",
f"/repos/{repo}/actions/runners?per_page=100&page={page}",
)
runners = response.get("runners", [])
for runner in runners:
if runner.get("name") == name:
return runner
if len(runners) < 100:
return None
page += 1
def make_ecs_client(config: ProvisionConfig):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "")
access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "")
if not access_key_id or not access_key_secret:
raise SystemExit(
"ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the "
"ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or "
"misnamed secret expands to an empty string)."
)
return EcsClient(
OpenApiConfig(
access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=config.region_id,
endpoint=f"ecs.{config.region_id}.aliyuncs.com",
)
)
def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None:
from alibabacloud_ecs20140526 import models as ecs_models
while time.monotonic() < deadline:
response = client.describe_instances(
ecs_models.DescribeInstancesRequest(
region_id=region_id, instance_ids=json.dumps([instance_id])
)
)
instances = response.body.instances.instance
if instances and instances[0].status == wanted:
return
time.sleep(POLL_INTERVAL_SECONDS)
raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time")
def fetch_console_output(client, region_id: str, instance_id: str) -> str:
from alibabacloud_ecs20140526 import models as ecs_models
response = client.get_instance_console_output(
ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id)
)
return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace")
class ConsoleTailer:
"""Incrementally prints new serial-console lines so cloud-init progress is
visible in the CI log while waiting, not only after a failure dump."""
def __init__(self, client, region_id: str, instance_id: str) -> None:
self.client = client
self.region_id = region_id
self.instance_id = instance_id
self.printed_lines = 0
def poll(self) -> None:
try:
lines = fetch_console_output(self.client, self.region_id, self.instance_id).splitlines()
except Exception as error: # noqa: BLE001
print(f"[console] unable to fetch console output: {error}", flush=True)
return
# The API returns a bounded tail; if it ever shrinks, restart from the
# beginning of the new buffer rather than skipping lines.
if len(lines) < self.printed_lines:
self.printed_lines = 0
for line in lines[self.printed_lines :]:
print(f"[console] {line}", flush=True)
self.printed_lines = len(lines)
def dump_console_output(client, region_id: str, instance_id: str) -> None:
try:
output = fetch_console_output(client, region_id, instance_id)
except Exception as error: # noqa: BLE001
print(f"Unable to fetch console output for {instance_id}: {error}", flush=True)
return
tail = "\n".join(output.splitlines()[-80:])
print(f"::group::Console output tail for {instance_id}\n{tail}\n::endgroup::", flush=True)
summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
if summary_path:
with open(summary_path, "a", encoding="utf-8") as summary:
summary.write(f"<details><summary>ECS console output ({instance_id})</summary>\n\n```\n")
summary.write(tail)
summary.write("\n```\n</details>\n")
def append_github_output(name: str, value: str) -> None:
output_path = os.environ.get("GITHUB_OUTPUT")
if output_path:
with open(output_path, "a", encoding="utf-8") as output:
output.write(f"{name}={value}\n")
def run_instance(client, config: ProvisionConfig, user_data: str) -> str:
from alibabacloud_ecs20140526 import models as ecs_models
request = ecs_models.RunInstancesRequest(
region_id=config.region_id,
image_id=config.image_id,
resource_group_id=config.resource_group_id,
instance_type=config.instance_type,
v_switch_id=config.vswitch_id,
security_group_id=config.security_group_id,
instance_name=runner_name_for_run(config.run_id),
host_name=runner_name_for_run(config.run_id),
description=f"Ephemeral query-regression runner for run {config.run_id}",
amount=1,
instance_charge_type="PostPaid",
spot_strategy="NoSpot",
internet_charge_type="PayByTraffic",
internet_max_bandwidth_out=100,
system_disk=ecs_models.RunInstancesRequestSystemDisk(
category="cloud_essd",
size=str(SYSTEM_DISK_GIB),
),
user_data=user_data,
tag=[
ecs_models.RunInstancesRequestTag(key=MANAGED_BY_TAG_KEY, value=MANAGED_BY_TAG_VALUE),
ecs_models.RunInstancesRequestTag(key=RUN_TAG_KEY, value=config.run_id),
],
)
response = client.run_instances(request)
instance_id = response.body.instance_id_sets.instance_id_set[0]
return instance_id
def provision(config: ProvisionConfig) -> int:
client = make_ecs_client(config)
runner_name = runner_name_for_run(config.run_id)
runner_label = runner_label_for_run(config.run_id)
registration_token = create_registration_token(config.github_token, config.repo)
user_data = encode_user_data(
render_user_data(
runner_name,
runner_label,
registration_token,
config.repo,
config.runner_uid,
config.runner_gid,
)
)
print(f"::group::Run instance {runner_name}", flush=True)
instance_id = run_instance(client, config, user_data)
print(f"Instance id: {instance_id}", flush=True)
# Teardown is the only DeleteInstance caller. Write outputs immediately
# so a later status/online failure still releases this VM.
append_github_output("label", runner_label)
append_github_output("instance_id", instance_id)
append_github_output("runner_name", runner_name)
wait_for_instance_status(client, config.region_id, instance_id, "Running", time.monotonic() + 5 * 60)
print("::endgroup::", flush=True)
print("::group::Wait for runner online", flush=True)
deadline = time.monotonic() + RUNNER_ONLINE_TIMEOUT_SECONDS
tailer = ConsoleTailer(client, config.region_id, instance_id)
last_console_poll = 0.0
while time.monotonic() < deadline:
runner = find_runner_by_name(config.github_token, config.repo, runner_name)
if runner is not None:
status = runner.get("status")
print(f"Runner {runner_name} status: {status}", flush=True)
if status == "online":
print("::endgroup::", flush=True)
print(f"Runner {runner_name} is online with label {runner_label}", flush=True)
return 0
# The serial console lags the guest by a minute or so; 30s polling is
# enough to follow cloud-init without tripping API throttling.
if time.monotonic() - last_console_poll >= 30:
tailer.poll()
last_console_poll = time.monotonic()
time.sleep(POLL_INTERVAL_SECONDS)
print("::endgroup::", flush=True)
print(
f"Runner {runner_name} did not come online in time; "
"leaving instance for the teardown job to delete",
flush=True,
)
dump_console_output(client, config.region_id, instance_id)
return 1
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID"))
parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID"))
parser.add_argument("--image-id", default=os.environ.get("QUERY_REGRESSION_ECS_IMAGE_ID"))
parser.add_argument("--instance-type", default=os.environ.get("ALIYUN_ECS_INSTANCE_TYPE"))
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument("--run-id", default=os.environ.get("GITHUB_RUN_ID"))
parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN"))
parser.add_argument("--runner-uid", default=os.environ.get("QUERY_REGRESSION_RUNNER_UID", "1001"))
parser.add_argument("--runner-gid", default=os.environ.get("QUERY_REGRESSION_RUNNER_GID", "1001"))
parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID"))
args = parser.parse_args()
missing = [
name
for name, value in vars(args).items()
if name not in ("resource_group_id",)
and (value is None or (isinstance(value, str) and not value))
]
if missing:
flags = ", ".join(f"--{name.replace('_', '-')}" for name in missing)
raise SystemExit(f"Missing required configuration: {flags} (or their env defaults)")
config = ProvisionConfig(
region_id=args.region_id,
vswitch_id=args.vswitch_id,
security_group_id=args.security_group_id,
image_id=args.image_id,
instance_type=args.instance_type,
repo=args.repo,
run_id=args.run_id,
github_token=args.github_token,
resource_group_id=args.resource_group_id or None,
runner_uid=args.runner_uid,
runner_gid=args.runner_gid,
)
return provision(config)
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/aliyun-ecs-runner-teardown.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# PEP 723 inline metadata (see aliyun-ecs-runner-provision.py for the
# convention). The SDK import stays lazy so unit tests run on a plain
# stdlib interpreter.
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "alibabacloud_ecs20140526>=4.1.0,<6",
# "alibabacloud_tea_openapi>=0.3.12,<1",
# ]
# ///
"""Tear down ephemeral Aliyun ECS query-regression runners.
Two modes:
- Targeted (per workflow run): delete one instance by id and deregister its
runner by name. Both steps are idempotent and best-effort; a missing
instance or runner is not an error.
- Sweep (scheduled janitor): delete every instance tagged as managed by
query-regression CI whose creation time is older than the given TTL, and
deregister the matching runners. This is the safety net for runs whose
teardown job never executed.
"""
from __future__ import annotations
import argparse
import importlib.util
import os
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
# Reuse the GitHub API client, runner lookup, and tag constants from the
# provision script.
_PROVISION_SPEC = importlib.util.spec_from_file_location(
"aliyun_ecs_runner_provision",
Path(__file__).resolve().parent / "aliyun-ecs-runner-provision.py",
)
assert _PROVISION_SPEC is not None and _PROVISION_SPEC.loader is not None
provision = importlib.util.module_from_spec(_PROVISION_SPEC)
sys.modules[_PROVISION_SPEC.name] = provision
_PROVISION_SPEC.loader.exec_module(provision)
SWEEP_TTL = timedelta(hours=4)
# ECS DescribeInstances creation_time is ISO8601 UTC, e.g. "2026-08-17T02:13Z".
CREATION_TIME_FORMATS = ("%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%dT%H:%MZ")
def parse_creation_time(value: str) -> datetime:
for fmt in CREATION_TIME_FORMATS:
try:
return datetime.strptime(value, fmt).replace(tzinfo=timezone.utc)
except ValueError:
continue
raise ValueError(f"Unparsable ECS creation time: {value}")
def expired_instance_names(
instances: list[tuple[str, str, str]], now: datetime, ttl: timedelta
) -> list[tuple[str, str]]:
"""Pick (instance_id, instance_name) pairs whose creation is older than ttl.
`instances` items are (instance_id, instance_name, creation_time).
"""
expired = []
for instance_id, instance_name, creation_time in instances:
age = now - parse_creation_time(creation_time)
if age >= ttl:
expired.append((instance_id, instance_name))
return expired
def make_ecs_client(region_id: str):
from alibabacloud_ecs20140526.client import Client as EcsClient
from alibabacloud_tea_openapi.models import Config as OpenApiConfig
access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "")
access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "")
if not access_key_id or not access_key_secret:
raise SystemExit(
"ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the "
"ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or "
"misnamed secret expands to an empty string)."
)
return EcsClient(
OpenApiConfig(
access_key_id=access_key_id,
access_key_secret=access_key_secret,
region_id=region_id,
endpoint=f"ecs.{region_id}.aliyuncs.com",
)
)
def delete_instance(client, instance_id: str, region_id: str | None = None) -> bool:
import time
from alibabacloud_ecs20140526 import models as ecs_models
# Export the serial console before destroying the evidence: a cancelled
# run still runs this teardown job, and the console tail (cloud-init,
# runner service logs, kernel OOM records) is the only witness of what
# happened on the machine.
if region_id is not None:
provision.dump_console_output(client, region_id, instance_id)
# DeleteInstance rejects instances that are still Initializing (or in
# another transitional status). Teardown runs right after a cancelled
# run, when the instance may be only seconds old, so retry the transient
# status errors for a few minutes instead of leaking the instance.
deadline = time.monotonic() + 5 * 60
while True:
try:
client.delete_instance(ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True))
print(f"Deleted instance {instance_id}", flush=True)
return True
except Exception as error: # noqa: BLE001
message = str(error)
if "InvalidInstanceId.NotFound" in message:
print(f"Instance {instance_id} already gone", flush=True)
return True
if "IncorrectInstanceStatus" in message and time.monotonic() < deadline:
print(f"Instance {instance_id} is in a transitional status; retrying delete", flush=True)
time.sleep(15)
continue
print(f"Failed to delete instance {instance_id}: {error}", flush=True)
return False
def deregister_runner(token: str, repo: str, runner_name: str) -> bool:
runner = provision.find_runner_by_name(token, repo, runner_name)
if runner is None:
print(f"Runner {runner_name} is not registered", flush=True)
return True
try:
provision.github_api(token, "DELETE", f"/repos/{repo}/actions/runners/{runner['id']}")
print(f"Deregistered runner {runner_name} (id {runner['id']})", flush=True)
return True
except Exception as error: # noqa: BLE001
print(f"Failed to deregister runner {runner_name}: {error}", flush=True)
return False
def list_managed_instances(client, region_id: str) -> list[tuple[str, str, str]]:
from alibabacloud_ecs20140526 import models as ecs_models
result: list[tuple[str, str, str]] = []
next_token = None
while True:
request = ecs_models.DescribeInstancesRequest(
region_id=region_id,
tag=[
ecs_models.DescribeInstancesRequestTag(
key=provision.MANAGED_BY_TAG_KEY, value=provision.MANAGED_BY_TAG_VALUE
)
],
max_results=100,
next_token=next_token,
)
response = client.describe_instances(request)
for instance in response.body.instances.instance:
result.append((instance.instance_id, instance.instance_name, instance.creation_time))
next_token = response.body.next_token
if not next_token:
return result
def sweep(client, region_id: str, repo: str, github_token: str, ttl: timedelta) -> int:
instances = list_managed_instances(client, region_id)
print(f"Found {len(instances)} managed instance(s) in {region_id}", flush=True)
expired = expired_instance_names(instances, datetime.now(timezone.utc), ttl)
ok = True
for instance_id, instance_name in expired:
print(f"Instance {instance_id} ({instance_name}) exceeds TTL {ttl}; deleting", flush=True)
# Runner names mirror instance names by construction in the provision
# script (both are qreg-ecs-<run_id>).
ok &= delete_instance(client, instance_id, region_id)
ok &= deregister_runner(github_token, repo, instance_name)
return 0 if ok else 1
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID"))
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN"))
parser.add_argument("--instance-id", default=os.environ.get("QUERY_REGRESSION_ECS_INSTANCE_ID"))
parser.add_argument("--runner-name", default=os.environ.get("QUERY_REGRESSION_ECS_RUNNER_NAME"))
parser.add_argument(
"--sweep",
action="store_true",
help="Janitor mode: delete all managed instances older than --sweep-ttl-hours.",
)
parser.add_argument("--sweep-ttl-hours", type=float, default=SWEEP_TTL.total_seconds() / 3600)
args = parser.parse_args()
for name in ("region_id", "repo", "github_token"):
if not getattr(args, name):
raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}")
client = make_ecs_client(args.region_id)
if args.sweep:
return sweep(
client, args.region_id, args.repo, args.github_token, timedelta(hours=args.sweep_ttl_hours)
)
ok = True
if args.instance_id:
ok &= delete_instance(client, args.instance_id, args.region_id)
else:
print("No instance id given; skipping instance deletion", flush=True)
if args.runner_name:
ok &= deregister_runner(args.github_token, args.repo, args.runner_name)
else:
print("No runner name given; skipping runner deregistration", flush=True)
return 0 if ok else 1
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/query-regression-nightly-refs.py
+270
View File
@@ -0,0 +1,270 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve consecutive successful Nightly Build SHAs for query-regression.
Query regression builds both binaries from git; this script only picks the
head SHAs of two Nightly Build workflow runs (today vs the previous success).
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from typing import Any
NIGHTLY_WORKFLOW = "nightly-build.yml"
NIGHTLY_EVENTS = frozenset({"schedule", "workflow_dispatch"})
@dataclass(frozen=True)
class WorkflowRun:
id: int
head_sha: str
head_branch: str
html_url: str
created_at: str
conclusion: str
event: str
@dataclass(frozen=True)
class RefPair:
base: WorkflowRun | None
candidate: WorkflowRun | None
skip: bool
reason: str
def parse_run(payload: dict[str, Any]) -> WorkflowRun:
return WorkflowRun(
id=int(payload["id"]),
head_sha=str(payload["head_sha"]),
head_branch=str(payload.get("head_branch") or ""),
html_url=str(payload.get("html_url") or ""),
created_at=str(payload.get("created_at") or ""),
conclusion=str(payload.get("conclusion") or ""),
event=str(payload.get("event") or ""),
)
def github_get(token: str, path: str, query: dict[str, str] | None = None) -> dict[str, Any]:
encoded = f"?{urllib.parse.urlencode(query)}" if query else ""
request = urllib.request.Request(
f"https://api.github.com{path}{encoded}",
method="GET",
headers={
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {token}",
"X-GitHub-Api-Version": "2022-11-28",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.loads(response.read().decode("utf-8"))
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", "replace")
raise SystemExit(
f"GitHub API GET {path} failed: HTTP {error.code}: {body}"
) from error
def list_successful_nightly_runs(
token: str,
repo: str,
*,
branch: str | None = None,
per_page: int = 30,
) -> list[WorkflowRun]:
query = {
"status": "success",
"per_page": str(per_page),
}
if branch:
query["branch"] = branch
payload = github_get(
token,
f"/repos/{repo}/actions/workflows/{NIGHTLY_WORKFLOW}/runs",
query,
)
runs = [parse_run(item) for item in payload.get("workflow_runs") or []]
return [
run
for run in runs
if run.conclusion == "success" and run.event in NIGHTLY_EVENTS and run.head_sha
]
def fetch_run(token: str, repo: str, run_id: int) -> WorkflowRun:
payload = github_get(token, f"/repos/{repo}/actions/runs/{run_id}")
return parse_run(payload)
def select_base_and_candidate(
runs: list[WorkflowRun],
*,
candidate_run_id: int | None = None,
candidate: WorkflowRun | None = None,
) -> RefPair:
"""Pick candidate (newest / requested) and the previous successful run.
`runs` must be newest-first. Same-SHA consecutive nightlies are skipped
(no commits that day).
"""
if candidate is None:
if candidate_run_id is not None:
candidate = next((run for run in runs if run.id == candidate_run_id), None)
if candidate is None:
return RefPair(
None,
None,
True,
f"candidate nightly run {candidate_run_id} was not a successful "
f"{NIGHTLY_WORKFLOW} run",
)
elif runs:
candidate = runs[0]
else:
return RefPair(None, None, True, "no successful Nightly Build runs")
branch = candidate.head_branch
older = [
run
for run in runs
if run.id != candidate.id
and (not branch or not run.head_branch or run.head_branch == branch)
and (not candidate.created_at or run.created_at <= candidate.created_at)
]
if not older:
return RefPair(
None,
candidate,
True,
"no previous successful Nightly Build to compare against",
)
base = older[0]
if base.head_sha.lower() == candidate.head_sha.lower():
return RefPair(
base,
candidate,
True,
f"previous nightly SHA {base.head_sha} matches candidate; nothing to compare",
)
return RefPair(base, candidate, False, "")
def _override_run(sha: str) -> WorkflowRun:
return WorkflowRun(
id=0,
head_sha=sha,
head_branch="",
html_url="",
created_at="",
conclusion="success",
event="workflow_dispatch",
)
def override_pair(base_ref: str, candidate_ref: str) -> RefPair:
return RefPair(_override_run(base_ref), _override_run(candidate_ref), False, "explicit refs")
def write_outputs(pair: RefPair) -> None:
values = {
"skip": "true" if pair.skip else "false",
"reason": pair.reason,
"base_sha": pair.base.head_sha if pair.base else "",
"candidate_sha": pair.candidate.head_sha if pair.candidate else "",
"base_run_url": pair.base.html_url if pair.base else "",
"candidate_run_url": pair.candidate.html_url if pair.candidate else "",
"base_run_id": str(pair.base.id) if pair.base and pair.base.id else "",
"candidate_run_id": str(pair.candidate.id) if pair.candidate and pair.candidate.id else "",
}
output_path = os.environ.get("GITHUB_OUTPUT")
if output_path:
with open(output_path, "a", encoding="utf-8") as handle:
for key, value in values.items():
handle.write(f"{key}={value}\n")
for key, value in values.items():
print(f"{key}={value}")
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY"))
parser.add_argument(
"--token",
default=os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN"),
)
parser.add_argument("--branch", default=os.environ.get("NIGHTLY_BRANCH") or "")
parser.add_argument(
"--candidate-run-id",
default=os.environ.get("CANDIDATE_RUN_ID") or "",
)
parser.add_argument("--base-ref", default=os.environ.get("BASE_REF") or "")
parser.add_argument("--candidate-ref", default=os.environ.get("CANDIDATE_REF") or "")
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
base_ref = args.base_ref.strip()
candidate_ref = args.candidate_ref.strip()
if base_ref and candidate_ref:
write_outputs(override_pair(base_ref, candidate_ref))
return 0
if bool(base_ref) != bool(candidate_ref):
print("Provide both --base-ref and --candidate-ref, or neither.", file=sys.stderr)
return 2
if not args.repo or not args.token:
print("--repo and --token (or GITHUB_REPOSITORY / GITHUB_TOKEN) are required.", file=sys.stderr)
return 2
candidate_run_id = int(args.candidate_run_id) if str(args.candidate_run_id).strip() else None
candidate: WorkflowRun | None = None
branch = args.branch.strip() or None
if candidate_run_id is not None:
candidate = fetch_run(args.token, args.repo, candidate_run_id)
if candidate.conclusion != "success":
write_outputs(
RefPair(
None,
candidate,
True,
f"nightly run {candidate_run_id} conclusion is {candidate.conclusion}",
)
)
return 0
branch = branch or candidate.head_branch or None
runs = list_successful_nightly_runs(args.token, args.repo, branch=branch)
pair = select_base_and_candidate(
runs,
candidate_run_id=candidate_run_id if candidate is None else None,
candidate=candidate,
)
write_outputs(pair)
return 0
if __name__ == "__main__":
raise SystemExit(main())
.github/scripts/query-regression-run.py
+501 -26
View File
@@ -18,10 +18,17 @@
from __future__ import annotations
import argparse
import json
import os
import re
import signal
import socket
import subprocess
import time
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from typing import Any
DEFAULT_CASES = [
@@ -99,38 +106,479 @@ def append_step_summary(summary: Path) -> None:
out.write(summary.read_text())
@dataclass(frozen=True)
class RunTarget:
name: str
binary: Path
work_dir: Path
http_port: int
grpc_port: int
mysql_port: int
postgres_port: int
metasrv_rpc_port: int
metasrv_http_port: int
datanode_rpc_port: int
datanode_http_port: int
datanode_data_dir: Path
frontend_config: Path | None = None
def allocate_ports(n: int) -> list[int]:
socks = []
try:
for _ in range(n):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("127.0.0.1", 0))
socks.append(sock)
return [sock.getsockname()[1] for sock in socks]
finally:
for sock in socks:
sock.close()
def make_target(
name: str,
binary: Path,
root: Path,
ports: list[int],
frontend_config: Path | None = None,
) -> RunTarget:
work_dir = root / name
return RunTarget(
name=name,
binary=binary,
work_dir=work_dir,
http_port=ports[4],
grpc_port=ports[5],
mysql_port=ports[6],
postgres_port=ports[7],
metasrv_rpc_port=ports[0],
metasrv_http_port=ports[1],
datanode_rpc_port=ports[2],
datanode_http_port=ports[3],
datanode_data_dir=work_dir / "datanode-0" / "data",
frontend_config=frontend_config,
)
def wait_health(port: int, timeout_s: float = 60.0) -> None:
deadline = time.monotonic() + timeout_s
last: Exception | None = None
while time.monotonic() < deadline:
try:
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=2) as response:
if response.status < 500:
return
except Exception as err: # noqa: BLE001 - retain the last health diagnostic
last = err
time.sleep(0.5)
raise TimeoutError(f"health check timed out on port {port}: {last}")
def component_log_dir(target: RunTarget, name: str) -> Path:
return target.work_dir / "logs" / ("datanode-0" if name == "datanode" else name)
def component_command(target: RunTarget, name: str) -> list[str]:
log_dir = component_log_dir(target, name)
if name == "metasrv":
return [
str(target.binary), "metasrv", "start",
"--grpc-bind-addr", f"127.0.0.1:{target.metasrv_rpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.metasrv_rpc_port}",
"--http-addr", f"127.0.0.1:{target.metasrv_http_port}",
"--backend", "memory-store", "--enable-region-failover", "false",
"--log-dir", str(log_dir),
]
if name == "datanode":
return [
str(target.binary), "datanode", "start",
"--grpc-bind-addr", f"127.0.0.1:{target.datanode_rpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.datanode_rpc_port}",
"--http-addr", f"127.0.0.1:{target.datanode_http_port}",
"--data-home", str(target.datanode_data_dir), "--log-dir", str(log_dir),
"--node-id", "0", "--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}",
]
if name == "frontend":
command = [str(target.binary), "frontend", "start"]
if target.frontend_config is not None:
command.extend(["--config-file", str(target.frontend_config)])
command.extend([
"--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}",
"--http-addr", f"127.0.0.1:{target.http_port}",
"--grpc-bind-addr", f"127.0.0.1:{target.grpc_port}",
"--grpc-server-addr", f"127.0.0.1:{target.grpc_port}",
"--mysql-addr", f"127.0.0.1:{target.mysql_port}",
"--postgres-addr", f"127.0.0.1:{target.postgres_port}",
"--log-dir", str(log_dir),
])
return command
raise ValueError(f"unknown component: {name}")
def start_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
if name != "metasrv":
metasrv = procs.get((target.name, "metasrv"))
if metasrv is None or metasrv.poll() is not None:
raise RuntimeError("metasrv exited; memory-store metadata is no longer valid")
if name == "datanode":
target.datanode_data_dir.mkdir(parents=True, exist_ok=True)
logs = component_log_dir(target, name)
logs.mkdir(parents=True, exist_ok=True)
with (logs / "stdout.log").open("ab") as out, (logs / "stderr.log").open("ab") as err:
procs[(target.name, name)] = subprocess.Popen(
component_command(target, name),
stdout=out,
stderr=err,
start_new_session=True,
)
wait_health({
"metasrv": target.metasrv_http_port,
"datanode": target.datanode_http_port,
"frontend": target.http_port,
}[name])
def stop_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
proc = procs.pop((target.name, name), None)
if proc is None or proc.poll() is not None:
return
try:
os.killpg(proc.pid, signal.SIGTERM)
except ProcessLookupError:
return
try:
proc.wait(timeout=20)
except subprocess.TimeoutExpired:
try:
os.killpg(proc.pid, signal.SIGKILL)
except ProcessLookupError:
return
proc.wait(timeout=20)
def restart_component(
target: RunTarget,
name: str,
procs: dict[tuple[str, str], subprocess.Popen[bytes]],
) -> None:
stop_component(target, name, procs)
start_component(target, name, procs)
def stop_all(targets: list[RunTarget], procs: dict[tuple[str, str], subprocess.Popen[bytes]]) -> None:
for target in reversed(targets):
for name in ("frontend", "datanode", "metasrv"):
stop_component(target, name, procs)
def load_plan(fixture_generator: Path, case_path: Path) -> dict[str, Any]:
result = subprocess.run(
[str(fixture_generator), "plan", "--case", str(case_path)],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
check=False,
)
if result.returncode != 0:
raise RuntimeError(f"query_perf_fixture plan failed: {result.stderr[:2000]}")
return json.loads(result.stdout)
def run_direct_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
ports = allocate_ports(16)
targets = [
make_target("base", base_bin, work_dir, ports[:8]),
make_target("candidate", candidate_bin, work_dir, ports[8:]),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
try:
for target in targets:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
prepare = [
str(runner), "prepare-direct", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--fixture-dir", str(work_dir / "fixture"),
"--output", str(work_dir / "prepare-direct.json"),
"--http-timeout", str(args.http_timeout),
]
if parse_bool(args.allow_large_fixture):
prepare.append("--allow-large-fixture")
prepare_status = subprocess.run(prepare, check=False).returncode
if prepare_status != 0:
return prepare_status
for target in targets:
stop_component(target, "datanode", procs)
prepared = json.loads((work_dir / "prepare-direct.json").read_text(encoding="utf-8"))
fixtures = prepared.get("fixtures")
if not isinstance(fixtures, list):
raise RuntimeError("prepare-direct report has no fixtures array")
for target in targets:
destination = target.work_dir / "materialize-destination.toml"
destination.write_text(
f"data_home = {json.dumps(str(target.datanode_data_dir))}\n"
"object_store = { type = \"File\" }\n",
encoding="utf-8",
)
for fixture in fixtures:
fixture_dir = fixture.get("fixture_dir") if isinstance(fixture, dict) else None
if not isinstance(fixture_dir, str):
raise RuntimeError("prepare-direct fixture record has no fixture_dir")
materialize = [
str(runner), "materialize", "--fixture-dir", fixture_dir,
"--destination", str(destination),
]
materialize_status = subprocess.run(materialize, check=False).returncode
if materialize_status != 0:
return materialize_status
for target in targets:
start_component(target, "datanode", procs)
measure = [
str(runner), "measure", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(work_dir / "query-regression-report.json"),
"--http-timeout", str(args.http_timeout),
]
status = subprocess.run(measure, check=False).returncode
if status != 0:
for target in targets:
restart_component(target, "frontend", procs)
status = subprocess.run(measure, check=False).returncode
return status
finally:
stop_all(targets, procs)
def run_remote_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
ports = allocate_ports(16)
targets = [
make_target(
"base",
base_bin,
work_dir,
ports[:8],
work_dir / "base" / "frontend-prom-store.toml",
),
make_target(
"candidate",
candidate_bin,
work_dir,
ports[8:],
work_dir / "candidate" / "frontend-prom-store.toml",
),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
if target.frontend_config is None:
raise RuntimeError(f"remote target has no frontend config path: {target.name}")
render = [
str(runner), "render-remote-config", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--output", str(target.frontend_config),
]
render_status = subprocess.run(render, check=False).returncode
if render_status != 0:
return render_status
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
report = work_dir / "query-regression-report.json"
try:
for target in targets:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
prepare = [
str(runner), "prepare-remote", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(work_dir / "prepare-remote.json"),
"--http-timeout", str(args.http_timeout),
]
prepare_status = subprocess.run(prepare, check=False).returncode
if prepare_status != 0:
return prepare_status
measure = [
str(runner), "measure", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-http-port", str(targets[0].http_port),
"--candidate-http-port", str(targets[1].http_port),
"--output", str(report), "--http-timeout", str(args.http_timeout),
]
measure_status = subprocess.run(measure, check=False).returncode
if measure_status != 0 and not report.exists():
return measure_status
for target in targets:
stop_component(target, "datanode", procs)
finalize = [
str(runner), "finalize-remote", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--candidate-bin", str(candidate_bin),
"--base-data-home", str(targets[0].datanode_data_dir),
"--candidate-data-home", str(targets[1].datanode_data_dir),
"--report", str(report),
]
finalize_status = subprocess.run(finalize, check=False).returncode
if finalize_status != 0:
return finalize_status
final_report = json.loads(report.read_text(encoding="utf-8"))
return 1 if measure_status != 0 or final_report.get("status") == "failed" else 0
finally:
stop_all(targets, procs)
def run_otlp_case(
args: argparse.Namespace,
case_path: Path,
work_dir: Path,
base_bin: Path,
candidate_bin: Path,
fixture_generator: Path,
runner: Path,
) -> int:
if args.otelgen_bin is None:
raise ValueError("--otelgen-bin is required for otlp_trace_load")
ports = allocate_ports(16)
targets = [
make_target("base", base_bin, work_dir, ports[:8]),
make_target("candidate", candidate_bin, work_dir, ports[8:]),
]
for target in targets:
if target.work_dir.exists() and any(target.work_dir.iterdir()):
raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}")
target.work_dir.mkdir(parents=True, exist_ok=True)
procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {}
try:
for target in targets:
try:
start_component(target, "metasrv", procs)
start_component(target, "datanode", procs)
start_component(target, "frontend", procs)
command = [
str(runner), "run-otlp-target", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--otelgen-bin", str(args.otelgen_bin),
"--http-port", str(target.http_port),
"--target-name", target.name,
"--work-dir", str(target.work_dir),
"--output", str(target.work_dir / "report.json"),
"--http-timeout", str(args.http_timeout),
]
status = subprocess.run(command, check=False).returncode
finally:
stop_all([target], procs)
if status != 0:
return status
output = work_dir / "query-regression-report.json"
finalize = [
str(runner), "finalize-otlp", "--case", str(case_path),
"--fixture-generator", str(fixture_generator),
"--base-result", str(targets[0].work_dir / "report.json"),
"--candidate-result", str(targets[1].work_dir / "report.json"),
"--output", str(output),
]
status = subprocess.run(finalize, check=False).returncode
if status != 0:
return status
report = json.loads(output.read_text(encoding="utf-8"))
return 1 if report.get("status") == "failed" else 0
finally:
stop_all(targets, procs)
def run_case(args: argparse.Namespace, case_path: Path, work_dir: Path) -> int:
target_dir = profile_dir(args.cargo_profile)
base_bin = args.base_bin or args.base_src / "target" / target_dir / "greptime"
candidate_bin = args.candidate_bin or args.candidate_src / "target" / target_dir / "greptime"
fixture_generator = args.fixture_generator or args.candidate_src / "target" / target_dir / "query_perf_fixture"
cmd = [
"uv",
"run",
"--no-project",
"python",
str(args.candidate_src / "tests/perf/query_regression_runner.py"),
"--case",
str(case_path),
"--base-bin",
str(base_bin),
"--candidate-bin",
str(candidate_bin),
"--fixture-generator",
str(fixture_generator),
"--work-dir",
str(work_dir),
"--http-timeout",
str(args.http_timeout),
]
if parse_bool(args.allow_large_fixture):
cmd.append("--allow-large-fixture")
if args.otelgen_bin is not None:
cmd.extend(["--otelgen-bin", str(args.otelgen_bin)])
runner = args.runner or args.candidate_src / "target" / target_dir / "query_regression_runner"
plan = load_plan(fixture_generator, case_path)
scenario = plan.get("scenario")
kind = scenario.get("kind") if isinstance(scenario, dict) else None
print(f"::group::Query regression case: {case_path}", flush=True)
try:
return subprocess.run(cmd, check=False).returncode
if kind == "direct_readable_sst":
return run_direct_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
if kind == "prom_remote_write_then_query":
return run_remote_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
if kind == "otlp_trace_load":
return run_otlp_case(
args,
case_path,
work_dir,
base_bin,
candidate_bin,
fixture_generator,
runner,
)
raise ValueError(
f"unsupported scenario kind {kind!r}; supported: "
"'direct_readable_sst', 'prom_remote_write_then_query', 'otlp_trace_load'"
)
finally:
print("::endgroup::", flush=True)
@@ -156,6 +604,27 @@ def write_summary(args: argparse.Namespace, reports: list[Path]) -> int:
return subprocess.run(cmd, check=False).returncode
def write_failed_report(work_dir: Path, case_path: Path, error: Exception) -> None:
work_dir.mkdir(parents=True, exist_ok=True)
path = work_dir / "query-regression-report.json"
try:
report = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(report, dict) or not isinstance(report.get("targets"), list) or not isinstance(report.get("thresholds"), list):
raise ValueError("existing report is malformed")
except (OSError, ValueError, json.JSONDecodeError):
report = {
"case_path": str(case_path),
"targets": [],
"thresholds": [],
}
report["status"] = "failed"
report["error"] = repr(error)
path.write_text(
json.dumps(report, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--cases", action="append", help="'all', 'heavy', or comma/space separated case paths")
@@ -169,6 +638,7 @@ def main() -> int:
default=configured_path(os.environ.get("FIXTURE_GENERATOR")),
)
parser.add_argument("--otelgen-bin", type=Path, default=configured_path(os.environ.get("OTELGEN_BIN")))
parser.add_argument("--runner", type=Path, default=configured_path(os.environ.get("QUERY_REGRESSION_RUNNER")))
parser.add_argument("--cargo-profile", default=os.environ.get("CARGO_PROFILE", "nightly"))
parser.add_argument("--work-dir", default=Path("query-regression-work"), type=Path)
parser.add_argument("--http-timeout", default=os.environ.get("HTTP_TIMEOUT", "300"))
@@ -198,7 +668,12 @@ def main() -> int:
case_path = resolve_case_path(args.candidate_src, case)
work_dir = args.work_dir / case_slug(case_path)
reports.append(work_dir / "query-regression-report.json")
case_status = run_case(args, case_path, work_dir)
try:
case_status = run_case(args, case_path, work_dir)
except Exception as err: # noqa: BLE001 - preserve an aggregate report for the summary step
print(f"error: query regression case {case_path}: {err}", flush=True)
write_failed_report(work_dir, case_path, err)
case_status = 1
if case_status != 0:
status = case_status or 1
.github/workflows/develop.yml
+8
View File
@@ -71,6 +71,14 @@ jobs:
version: "latest"
- name: Run check-version script tests
run: uv run --no-project python .github/scripts/check-version-test.py
- name: Run query-regression Python tooling tests
run: |
uv run --no-project --python 3.12 python tests/perf/test_query_regression_runner_compaction_toctou.py
uv run --no-project --python 3.12 python tests/perf/test_query_regression_runner_otlp_trace_load.py
uv run --no-project --python 3.12 python tests/perf/test_query_regression_summary_otlp.py
uv run --no-project --python 3.12 python tests/perf/test_query_regression_case_selection.py
uv run --no-project --python 3.12 python tests/perf/test_query_regression_nightly_refs.py
uv run --no-project --python 3.12 python tests/perf/test_aliyun_ecs_runner_scripts.py
check:
if: ${{ github.repository == 'GreptimeTeam/greptimedb' }}
.github/workflows/query-regression-janitor.yml
+36
View File
@@ -0,0 +1,36 @@
name: Query Regression ECS Janitor
# Safety net for the aliyun-ecs query-regression path: deletes managed ECS
# instances (and their runner registrations) that outlived their workflow run,
# so a failed teardown never leaks pay-as-you-go spend.
on:
schedule:
- cron: "23 3 * * *"
workflow_dispatch:
permissions:
contents: read
jobs:
sweep:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout teardown script
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Sweep expired ECS runners
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-teardown.py
--sweep --sweep-ttl-hours 4
.github/workflows/query-regression-nightly.yml
+104
View File
@@ -0,0 +1,104 @@
name: Query Regression Nightly
# After a successful GreptimeDB Nightly Build, compare that commit against
# the previous successful nightly. The reusable Query Regression workflow
# still compiles both SHAs; this wrapper only resolves which two SHAs.
on:
workflow_run:
workflows:
- GreptimeDB Nightly Build
types:
- completed
workflow_dispatch:
inputs:
base_ref:
description: Base ref/SHA (empty = previous successful nightly)
required: false
type: string
default: ""
candidate_ref:
description: Candidate ref/SHA (empty = latest successful nightly)
required: false
type: string
default: ""
candidate_run_id:
description: Nightly Build run id to treat as candidate (empty = latest)
required: false
type: string
default: ""
case:
description: Query perf case path(s), all, or heavy
required: false
type: string
default: all
permissions:
contents: read
actions: read
jobs:
resolve-refs:
name: Resolve previous vs current nightly SHAs
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
skip: ${{ steps.refs.outputs.skip }}
reason: ${{ steps.refs.outputs.reason }}
base_sha: ${{ steps.refs.outputs.base_sha }}
candidate_sha: ${{ steps.refs.outputs.candidate_sha }}
steps:
- name: Checkout ref resolver
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Resolve nightly SHAs
id: refs
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
CANDIDATE_RUN_ID: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.id || inputs.candidate_run_id || '' }}
BASE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.base_ref || '' }}
CANDIDATE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.candidate_ref || '' }}
run: python3 .github/scripts/query-regression-nightly-refs.py
- name: Summarize comparison
if: always()
env:
SKIP: ${{ steps.refs.outputs.skip }}
REASON: ${{ steps.refs.outputs.reason }}
BASE_SHA: ${{ steps.refs.outputs.base_sha }}
CANDIDATE_SHA: ${{ steps.refs.outputs.candidate_sha }}
BASE_RUN_URL: ${{ steps.refs.outputs.base_run_url }}
CANDIDATE_RUN_URL: ${{ steps.refs.outputs.candidate_run_url }}
run: |
set -euo pipefail
{
if [[ "${SKIP}" == "true" ]]; then
printf 'Skipping query-regression nightly: %s\n' "${REASON}"
else
printf 'Comparing previous nightly `%s` -> current nightly `%s`\n' \
"${BASE_SHA}" "${CANDIDATE_SHA}"
if [[ -n "${BASE_RUN_URL}" ]]; then
printf -- '- Previous Nightly Build: %s\n' "${BASE_RUN_URL}"
fi
if [[ -n "${CANDIDATE_RUN_URL}" ]]; then
printf -- '- Current Nightly Build: %s\n' "${CANDIDATE_RUN_URL}"
fi
fi
} | tee -a "${GITHUB_STEP_SUMMARY}"
query-regression:
name: Query regression nightly
needs: [resolve-refs]
if: ${{ needs.resolve-refs.outputs.skip != 'true' && needs.resolve-refs.outputs.base_sha != '' && needs.resolve-refs.outputs.candidate_sha != '' }}
uses: ./.github/workflows/query-regression.yml
secrets: inherit
with:
case: ${{ github.event_name == 'workflow_dispatch' && inputs.case || 'all' }}
base_ref: ${{ needs.resolve-refs.outputs.base_sha }}
candidate_ref: ${{ needs.resolve-refs.outputs.candidate_sha }}
cargo_profile: nightly
runner: aliyun-ecs
.github/workflows/query-regression.yml
+333 -130
View File
@@ -32,10 +32,10 @@ on:
type: string
default: nightly
runner:
description: Self-hosted runner label or ARC runner scale set for this query regression run
description: Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance per run
required: false
type: string
default: perf-regression-8-cores
default: aliyun-ecs
workflow_dispatch:
inputs:
case:
@@ -68,12 +68,20 @@ on:
- release
- dev
runner:
description: Self-hosted runner label or ARC runner scale set for this query regression run
description: >-
Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance
per run, any other value is used as a literal runner label
required: true
type: choice
default: perf-regression-8-cores
options:
- perf-regression-8-cores
type: string
default: aliyun-ecs
keep_instance:
description: >-
Debug: keep the ECS instance after the run (skip teardown) so its
runner _diag logs, journal, and telemetry can be inspected; the
janitor still sweeps it after the TTL
required: false
type: boolean
default: false
pull_request:
types: [labeled]
@@ -81,19 +89,99 @@ permissions:
contents: read
jobs:
query-regression:
test-tooling:
# Stdlib unittests for the workflow Python (case selection, report
# helpers, nightly SHA picking, and rendered ECS user-data). They do
# not talk to Aliyun or the Actions runner process; running them on
# ubuntu-latest fails fast before any ECS spend. Not gated on the
# regression labels: those labels boot a VM, these tests should not.
# Ordinary PRs also run the same tests from checks.yml, because this
# workflow only starts on `labeled` (or dispatch / workflow_call).
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Test query regression tooling
run: |
python3 tests/perf/test_query_regression_runner_compaction_toctou.py
python3 tests/perf/test_query_regression_runner_otlp_trace_load.py
python3 tests/perf/test_query_regression_summary_otlp.py
python3 tests/perf/test_query_regression_case_selection.py
python3 tests/perf/test_query_regression_nightly_refs.py
python3 tests/perf/test_aliyun_ecs_runner_scripts.py
provision:
# Runs when the aliyun-ecs path is selected: explicitly via the runner
# input, or for PR labels by default (a QUERY_REGRESSION_PR_RUNNER
# repository variable set to another value redirects PRs to that literal
# runner label instead). Uses trusted scripts from the PR base (or the
# dispatched ref), never from candidate code. Waits for test-tooling so
# a broken user-data template does not still create a VM.
needs: [test-tooling]
if: >-
${{ github.event_name != 'pull_request' ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression')) }}
runs-on: ${{ github.event_name != 'pull_request' && inputs.runner || 'perf-regression-8-cores' }}
${{ !failure() && !cancelled() &&
((github.event_name != 'pull_request' && inputs.runner == 'aliyun-ecs') ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression') &&
(vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs') == 'aliyun-ecs')) }}
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
label: ${{ steps.provision.outputs.label }}
instance_id: ${{ steps.provision.outputs.instance_id }}
runner_name: ${{ steps.provision.outputs.runner_name }}
steps:
- name: Checkout trusted provisioning scripts
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Provision ECS runner
id: provision
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
ALIYUN_ECS_VSWITCH_ID: ${{ vars.ALIYUN_ECS_VSWITCH_ID }}
ALIYUN_ECS_SECURITY_GROUP_ID: ${{ vars.ALIYUN_ECS_SECURITY_GROUP_ID }}
ALIYUN_ECS_INSTANCE_TYPE: ${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}
ALIYUN_ECS_RESOURCE_GROUP_ID: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }}
QUERY_REGRESSION_ECS_IMAGE_ID: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}
QUERY_REGRESSION_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
QUERY_REGRESSION_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-provision.py
query-regression:
needs: [provision, test-tooling]
# `!failure() && !cancelled()` both suppresses the implicit success() and
# lets the job run when provision was intentionally skipped because a
# literal runner label was selected; a failed or cancelled provision still
# blocks the run because no ECS runner would be waiting.
if: >-
${{ !failure() && !cancelled() &&
(github.event_name != 'pull_request' ||
(github.event_name == 'pull_request' &&
!github.event.pull_request.draft &&
(github.event.label.name == 'query-regression' ||
github.event.label.name == 'heavy-regression'))) }}
runs-on: >-
${{ needs.provision.outputs.label ||
(github.event_name != 'pull_request' && inputs.runner ||
(vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs')) }}
timeout-minutes: 180
concurrency:
group: query-regression-persistent-cache-v1
queue: max
cancel-in-progress: false
env:
CARGO_PROFILE: ${{ github.event_name == 'pull_request' && 'nightly' || inputs.cargo_profile }}
CARGO_HOME: /home/runner/.cargo
@@ -105,7 +193,8 @@ jobs:
QUERY_REGRESSION_CACHE_META: /home/runner/query-regression-cache-meta
RUSTC_WRAPPER: /usr/local/bin/sccache
SCCACHE_DIR: /home/runner/.cache/sccache
SCCACHE_CACHE_SIZE: 40G
# Caps the local sccache on the system disk next to target and cargo.
SCCACHE_CACHE_SIZE: 10G
CARGO_INCREMENTAL: "0"
RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold
QUERY_REGRESSION_CACHE_EPOCH: "1"
@@ -114,7 +203,60 @@ jobs:
EVENT_MERGE_SHA: ${{ github.event_name == 'pull_request' && github.sha || '' }}
EVENT_HEAD_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || '' }}
EVENT_BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }}
# Runner identity contract. The ECS image uses 1001; a manually prepared
# host may override via repo variables when 1001 is already taken.
EXPECTED_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }}
EXPECTED_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }}
steps:
- name: Report provisioned ECS runner
if: ${{ needs.provision.outputs.instance_id != '' }}
shell: bash
run: |
set -euo pipefail
{
printf -- '- ECS instance: `%s` (type `%s`, image `%s`, region `%s`)\n' \
"${{ needs.provision.outputs.instance_id }}" \
"${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}" \
"${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}" \
"${{ vars.ALIYUN_ECS_REGION_ID }}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Start machine telemetry sampler
shell: bash
# The ECS instance is deleted when the run ends, so machine state must
# be captured while the job runs. A background sampler appends load,
# memory, disk, and top-memory processes to a log every 30s; the
# "Dump machine telemetry" step (if: always()) prints and uploads it.
# Note: when the job is *cancelled*, even always() steps are killed —
# telemetry covers failures, not cancellations.
run: |
set -euo pipefail
log="${GITHUB_WORKSPACE}/machine-telemetry.log"
{
echo "== baseline $(date -u +%Y-%m-%dT%H:%M:%SZ) =="
nproc
free -h
df -h / /home/runner
} >> "${log}"
{
printf '## Disk (start)\n\n```\n'
df -hP / /home/runner
printf '```\n'
} >> "${GITHUB_STEP_SUMMARY}"
nohup bash -c "
while true; do
{
date -u '+== %Y-%m-%dT%H:%M:%SZ =='
uptime
free -m
df -h / /home/runner
ps -eo pid,comm,%mem,%cpu --sort=-%mem | head -6
} >> '${log}' 2>&1
sleep 30
done
" >/dev/null 2>&1 &
echo "Telemetry sampler started (pid $!), logging to ${log}"
- name: Checkout base source
uses: actions/checkout@v4
with:
@@ -263,14 +405,64 @@ jobs:
shell: bash
run: |
set -euo pipefail
[[ "$(id -u)" == "1001" ]]
[[ "$(id -g)" == "1001" ]]
[[ "${UV_CACHE_DIR}" == "/home/runner/.cargo/uv-cache" ]]
[[ "$(protoc --version)" == "libprotoc 3.21.12" ]]
errors=()
record() {
printf 'FAIL %s\n' "$*"
errors+=("$*")
}
require_eq() {
local name="$1" actual="$2" expected="$3"
printf 'check %s: %s\n' "${name}" "${actual}"
[[ "${actual}" == "${expected}" ]] || record "${name}: expected '${expected}', got '${actual}'"
}
require_match() {
local name="$1" actual="$2" pattern="$3"
printf 'check %s: %s\n' "${name}" "${actual}"
[[ "${actual}" =~ $pattern ]] || record "${name}: expected to match ${pattern}, got '${actual}'"
}
require() {
local name="$1"
shift
printf 'check %s\n' "${name}"
"$@" || record "${name}"
}
capture() {
local out
if out="$("$@" 2>&1)"; then
printf '%s' "${out}"
else
printf '<failed to run: %s>' "$*"
fi
}
printf 'uid=%s gid=%s PATH=%s CARGO_HOME=%s\n' "$(id -u)" "$(id -g)" "${PATH}" "${CARGO_HOME}"
ls -la "${CARGO_HOME}" 2>&1 || printf '(CARGO_HOME missing)\n'
# Image/host hygiene: CARGO_HOME must be empty of config and
# install-state before cargo/rustup run. Those tools create
# .package-cache / bin on first use, so this cannot come after.
cargo_home_dirty=()
for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do
if [[ -e "${CARGO_HOME}/${entry}" ]]; then
cargo_home_dirty+=("${entry}")
record "CARGO_HOME must not contain ${entry}"
fi
done
if (( ${#cargo_home_dirty[@]} > 0 )); then
printf 'CARGO_HOME contents:\n' >&2
ls -la "${CARGO_HOME}" >&2 || true
fi
require_eq uid "$(id -u)" "${EXPECTED_RUNNER_UID}"
require_eq gid "$(id -g)" "${EXPECTED_RUNNER_GID}"
require_eq UV_CACHE_DIR "${UV_CACHE_DIR}" "/home/runner/.cargo/uv-cache"
require_eq protoc "$(capture protoc --version)" "libprotoc 3.21.12"
temporary_proto_dir="$(mktemp --directory)"
trap 'rm -rf "${temporary_proto_dir}"' EXIT
test -r /usr/include/google/protobuf/any.proto
test -r /usr/include/google/protobuf/empty.proto
require "readable /usr/include/google/protobuf/any.proto" \
test -r /usr/include/google/protobuf/any.proto
require "readable /usr/include/google/protobuf/empty.proto" \
test -r /usr/include/google/protobuf/empty.proto
printf '%s\n' \
'syntax = "proto3";' \
'package smoke;' \
@@ -278,47 +470,51 @@ jobs:
'import "google/protobuf/empty.proto";' \
'message Smoke { google.protobuf.Any any = 1; google.protobuf.Empty empty = 2; }' \
> "${temporary_proto_dir}/smoke.proto"
protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \
if protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \
--descriptor_set_out="${temporary_proto_dir}/smoke.pb" \
"${temporary_proto_dir}/smoke.proto"
test -s "${temporary_proto_dir}/smoke.pb"
[[ "$(uv --version)" =~ ^uv[[:space:]]0\.11\.26([[:space:]]|$) ]]
mold_version="$(mold --version)"
[[ "${mold_version}" =~ ^mold[[:space:]]2\.30\.0([[:space:]]|$) ]]
[[ "$(python3 --version)" == "Python 3.12.3" ]]
"${temporary_proto_dir}/smoke.proto"; then
require "protoc smoke descriptor is non-empty" test -s "${temporary_proto_dir}/smoke.pb"
else
record "protoc smoke compile failed"
fi
require_match uv "$(capture uv --version)" '^uv[[:space:]]0\.11\.26([[:space:]]|$)'
require_match mold "$(capture mold --version)" '^mold[[:space:]]2\.40\.4([[:space:]]|$)'
require_eq python3 "$(capture python3 --version)" "Python 3.14.4"
otelgen_path="$(command -v otelgen || true)"
otelgen_version="<unavailable>"
if [[ -n "${otelgen_path}" ]]; then
otelgen_version="$("${otelgen_path}" --version 2>&1 || true)"
otelgen_version="$(capture "${otelgen_path}" --version)"
fi
printf 'otelgen path: %s\n' "${otelgen_path:-<missing>}"
printf 'otelgen version: %s\n' "${otelgen_version}"
[[ "${otelgen_path}" == "/usr/local/bin/otelgen" ]]
[[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]]
sccache_version="$(sccache --version)"
[[ "${sccache_version}" =~ ^sccache[[:space:]]0\.16\.0([[:space:]]|$) ]]
[[ "$(command -v rustup)" == "/opt/cargo/bin/rustup" ]]
[[ "$(command -v cargo)" == "/opt/cargo/bin/cargo" ]]
[[ "$(command -v rustc)" == "/opt/cargo/bin/rustc" ]]
[[ "$(rustup --version)" =~ ^rustup[[:space:]]1\.29\.0([[:space:]]|$) ]]
cargo_version="$(cargo --version)"
[[ "${cargo_version}" =~ ^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
rustc_version="$(rustc --version)"
[[ "${rustc_version}" =~ ^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]]
active_toolchain="$(rustup show active-toolchain)"
[[ "${active_toolchain}" =~ ^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$) ]]
[[ "${RUSTUP_HOME}" == "/opt/rustup" ]]
[[ "${RUSTUP_TOOLCHAIN}" == "nightly-2026-03-21" ]]
[[ "${RUSTUP_AUTO_INSTALL}" == "0" ]]
test -r /opt/rustup && test -x /opt/rustup
test ! -w /opt/rustup
test ! -w /opt/cargo/bin
for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do
test ! -e "${CARGO_HOME}/${entry}"
done
require_eq otelgen_path "${otelgen_path}" "/usr/local/bin/otelgen"
printf 'check otelgen_version: %s\n' "${otelgen_version}"
[[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]] \
|| record "otelgen_version: expected commit 863a3f395d062c7322cc1de08a38774b7fdaa6c8, got '${otelgen_version}'"
require_match sccache "$(capture sccache --version)" '^sccache[[:space:]]0\.16\.0([[:space:]]|$)'
require_eq rustup_path "$(command -v rustup || true)" "/opt/cargo/bin/rustup"
require_eq cargo_path "$(command -v cargo || true)" "/opt/cargo/bin/cargo"
require_eq rustc_path "$(command -v rustc || true)" "/opt/cargo/bin/rustc"
require_match rustup "$(capture rustup --version)" '^rustup[[:space:]]1\.29\.0([[:space:]]|$)'
require_match cargo "$(capture cargo --version)" \
'^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match rustc "$(capture rustc --version)" \
'^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$'
require_match active_toolchain "$(capture rustup show active-toolchain)" \
'^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$)'
require_eq RUSTUP_HOME "${RUSTUP_HOME}" "/opt/rustup"
require_eq RUSTUP_TOOLCHAIN "${RUSTUP_TOOLCHAIN}" "nightly-2026-03-21"
require_eq RUSTUP_AUTO_INSTALL "${RUSTUP_AUTO_INSTALL}" "0"
require "readable /opt/rustup" test -r /opt/rustup
require "executable /opt/rustup" test -x /opt/rustup
require "runner cannot write /opt/rustup" test ! -w /opt/rustup
require "runner cannot write /opt/cargo/bin" test ! -w /opt/cargo/bin
mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git"
if (( ${#errors[@]} > 0 )); then
printf '\nVerify runner image tools failed (%d checks):\n' "${#errors[@]}" >&2
printf ' - %s\n' "${errors[@]}" >&2
exit 1
fi
- name: Prepare persistent query regression cache
- name: Prepare query regression cache
shell: bash
working-directory: src
run: |
@@ -362,24 +558,6 @@ jobs:
find "${root}" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} +
}
clear_cargo_extracted_trees() {
require_expected_root CARGO_REGISTRY "${CARGO_HOME}/registry" "${EXPECTED_CARGO_REGISTRY}"
require_expected_root CARGO_GIT "${CARGO_HOME}/git" "${EXPECTED_CARGO_GIT}"
rm -rf -- "${CARGO_HOME}/registry/src" "${CARGO_HOME}/git/checkouts"
}
cargo_size_kib() {
du -sk -- "${CARGO_HOME}/registry" "${CARGO_HOME}/git" | awk '{ total += $1 } END { print total }'
}
target_size_kib() {
du -sk -- "${CARGO_TARGET_DIR}" | cut -f1
}
free_kib() {
df -Pk "${CARGO_TARGET_DIR}" | awk 'NR == 2 { print $4 }'
}
report_cache_usage() {
du -sh -- "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \
"${RUSTUP_HOME}" "${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}"
@@ -400,7 +578,7 @@ jobs:
printf 'Refusing unexpected RUSTC_WRAPPER: %s\n' "${RUSTC_WRAPPER}" >&2
exit 1
}
[[ "${SCCACHE_CACHE_SIZE}" == "40G" ]] || {
[[ "${SCCACHE_CACHE_SIZE}" == "10G" ]] || {
printf 'Refusing unexpected SCCACHE_CACHE_SIZE: %s\n' "${SCCACHE_CACHE_SIZE}" >&2
exit 1
}
@@ -487,46 +665,10 @@ jobs:
report_cache_usage
target_size="$(target_size_kib)"
if (( target_size >= 400 * 1024 * 1024 )); then
printf 'Warning: target cache is at least 400 GiB (%s KiB)\n' "${target_size}" >&2
fi
if (( target_size >= 450 * 1024 * 1024 )); then
printf 'Target cache is at least 450 GiB; clearing complete target root\n' >&2
clear_directory "${CARGO_TARGET_DIR}"
fi
cargo_size="$(cargo_size_kib)"
if (( cargo_size >= 60 * 1024 * 1024 )); then
printf 'Warning: Cargo cache is at least 60 GiB (%s KiB)\n' "${cargo_size}" >&2
fi
if (( cargo_size >= 80 * 1024 * 1024 )); then
printf 'Cargo cache is at least 80 GiB; removing extracted sources and checkouts\n' >&2
clear_cargo_extracted_trees
cargo_size="$(cargo_size_kib)"
if (( cargo_size >= 80 * 1024 * 1024 )); then
printf 'Cargo cache remains at least 80 GiB after cleanup (%s KiB)\n' "${cargo_size}" >&2
exit 1
fi
fi
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space is below 300 GiB; clearing complete target root\n' >&2
clear_directory "${CARGO_TARGET_DIR}"
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space remains below 300 GiB; removing Cargo extracted sources and checkouts\n' >&2
clear_cargo_extracted_trees
free_space="$(free_kib)"
if (( free_space < 300 * 1024 * 1024 )); then
printf 'Free space remains below 300 GiB after cleanup (%s KiB)\n' "${free_space}" >&2
exit 1
fi
fi
fi
report_cache_usage
# Every run starts from a fresh system disk, so size/free-space
# watermarks from the retired retained-disk era are not restored.
# A run that overflows the disk fails the build outright, which the
# telemetry step makes diagnosable.
sccache --start-server
sccache --zero-stats
@@ -567,24 +709,17 @@ jobs:
git reset --hard "${VERIFIED_CANDIDATE_SHA}"
git clean -ffdx
- name: Test query regression tooling
working-directory: src
run: |
uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py
uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py
uv run --no-project python tests/perf/test_query_regression_summary_otlp.py
uv run --no-project python tests/perf/test_query_regression_case_selection.py
- name: Build candidate greptime and fixture generators
- name: Build candidate greptime and query regression helpers
working-directory: src
run: |
set -euo pipefail
SECONDS=0
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools
cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools
candidate_build_elapsed="${SECONDS}"
printf 'Candidate greptime and fixture generator cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
printf -- '- Candidate greptime and fixture generator cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
printf 'Candidate greptime and query regression helper cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}"
printf -- '- Candidate greptime and query regression helper cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}"
sccache --show-stats
target_dir="${CARGO_PROFILE}"
if [[ "${CARGO_PROFILE}" == "dev" ]]; then
@@ -595,6 +730,8 @@ jobs:
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime"
cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture"
cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \
"${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner"
- name: Run query regression
id: run
@@ -609,6 +746,7 @@ jobs:
FIXTURE_GENERATOR: ${{ github.workspace }}/query-regression-bins/candidate/query_perf_fixture
OTELGEN_BIN: /usr/local/bin/otelgen
SUMMARY_SCRIPT: ${{ github.event_name == 'pull_request' && 'query-regression-trusted-scripts/query-regression-summary.py' || 'src/.github/scripts/query-regression-summary.py' }}
QUERY_REGRESSION_RUNNER: ${{ github.workspace }}/query-regression-bins/candidate/query_regression_runner
run: >-
uv run --no-project python src/.github/scripts/query-regression-run.py
--base-src src
@@ -639,6 +777,7 @@ jobs:
query-regression-work/**/logs/**
query-regression-work/**/otelgen/**
query-regression-summary.md
machine-telemetry.log
if-no-files-found: warn
retention-days: 7
@@ -653,7 +792,7 @@ jobs:
if-no-files-found: warn
retention-days: 7
- name: Report persistent cache usage
- name: Report cache usage
if: ${{ always() }}
shell: bash
run: |
@@ -687,12 +826,76 @@ jobs:
df -P "${report_root}"
df -hP "${report_root}"
df -Pi "${report_root}"
{
printf '## Disk (end)\n\n```\n'
df -hP / /home/runner "${report_root}"
printf '\n'
du -sh -- \
/home/runner/.cargo \
/home/runner/.cargo/registry \
/home/runner/.cargo/git \
/opt/rustup \
/home/runner/query-regression-target \
/home/runner/query-regression-cache-meta \
/home/runner/.cache/sccache \
2>/dev/null || true
printf '```\n'
} >> "${GITHUB_STEP_SUMMARY}"
if command -v sccache >/dev/null 2>&1; then
sccache --show-stats || printf 'Unable to show sccache statistics (report only)\n' >&2
else
printf 'sccache is unavailable (report only)\n' >&2
fi
- name: Dump machine telemetry
if: ${{ always() }}
shell: bash
run: |
log="${GITHUB_WORKSPACE}/machine-telemetry.log"
if [[ -f "${log}" ]]; then
echo "::group::Machine telemetry (last 200 lines)"
tail -n 200 "${log}"
echo "::endgroup::"
else
echo "No telemetry log found (sampler never started?)"
fi
echo "::group::dmesg tail (OOM killer records)"
sudo dmesg -T 2>/dev/null | tail -n 50 || dmesg -T 2>/dev/null | tail -n 50 || \
echo "dmesg unavailable without root"
echo "::endgroup::"
- name: Fail on regression failure
if: ${{ steps.run.outputs.status != '0' }}
run: exit 1
teardown:
# Releases the dynamically provisioned ECS runner. Runs even when the
# benchmark job fails or is cancelled; skipped when a literal runner label
# was selected because the provision outputs are empty, or when the
# dispatch set keep_instance to preserve the machine for post-mortem
# debugging (the janitor sweep still reclaims it after the TTL).
if: ${{ always() && needs.provision.outputs.instance_id != '' && !inputs.keep_instance }}
needs: [provision, query-regression]
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout trusted teardown scripts
uses: actions/checkout@v4
with:
repository: ${{ github.repository }}
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }}
persist-credentials: false
- name: Install uv
uses: astral-sh/setup-uv@v6
- name: Teardown ECS runner
env:
ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }}
ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }}
GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }}
ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }}
QUERY_REGRESSION_ECS_INSTANCE_ID: ${{ needs.provision.outputs.instance_id }}
QUERY_REGRESSION_ECS_RUNNER_NAME: ${{ needs.provision.outputs.runner_name }}
run: >-
uv run .github/scripts/aliyun-ecs-runner-teardown.py
.github/workflows/release.yml
+1 -1
View File
@@ -464,7 +464,7 @@ jobs:
base_ref: ${{ needs.prepare-release-validation.outputs.previous-release-tag }}
candidate_ref: ${{ needs.prepare-release-validation.outputs.candidate-ref }}
cargo_profile: nightly
runner: perf-regression-8-cores
runner: aliyun-ecs
release-images-to-dockerhub:
name: Build and push images to DockerHub
Cargo.lock
Generated
+2
View File
@@ -2195,6 +2195,8 @@ dependencies = [
"mito2",
"moka",
"object-store",
"object_store",
"object_store_opendal",
"parquet",
"plugins",
"pprof",
src/cmd/Cargo.toml
+7
View File
@@ -14,6 +14,11 @@ name = "query_perf_fixture"
path = "src/bin/query_perf_fixture.rs"
required-features = ["dev-tools"]
[[bin]]
name = "query_regression_runner"
path = "src/bin/query_regression_runner.rs"
required-features = ["dev-tools"]
[features]
default = [
"servers/pprof",
@@ -68,6 +73,7 @@ common-wal.workspace = true
datafusion.workspace = true
datafusion-common.workspace = true
datafusion-physical-plan.workspace = true
datafusion_object_store.workspace = true
datanode.workspace = true
datatypes.workspace = true
either = "1.15"
@@ -87,6 +93,7 @@ mito-codec.workspace = true
mito2.workspace = true
moka = { workspace = true, features = ["future"] }
object-store.workspace = true
object_store_opendal.workspace = true
parquet = { workspace = true, features = ["object_store"] }
plugins.workspace = true
prometheus.workspace = true
src/cmd/src/bin/query_perf_fixture/inspect_footer.rs
+344 -54
View File
@@ -13,17 +13,39 @@
// limitations under the License.
use std::collections::BTreeSet;
use std::fs::{self, File};
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use clap::Args as ClapArgs;
use parquet::file::reader::{FileReader, SerializedFileReader};
use serde::Serialize;
use datafusion_object_store::path::Path as StorePath;
use datafusion_object_store::{ObjectMeta, ObjectStore};
use futures::{StreamExt, TryStreamExt};
use object_store::config::ObjectStoreConfig;
use object_store::factory::new_raw_object_store;
use object_store::services::Fs;
use parquet::arrow::async_reader::ParquetObjectReader;
use parquet::file::metadata::ParquetMetaDataReader;
use serde::{Deserialize, Serialize};
/// Same shape as `query_regression_runner::model::DestinationConfig`. The two
/// binaries are separate crate roots, so the serde-compatible struct is defined
/// here instead of being shared; it reuses the `object-store` crate's
/// `ObjectStoreConfig` deserialization.
#[derive(Debug, Deserialize)]
struct DestinationConfig {
data_home: String,
object_store: ObjectStoreConfig,
}
#[derive(Debug, ClapArgs)]
pub(super) struct InspectFooterArgs {
/// Local data home (convenience shortcut for a File destination).
#[arg(long)]
root: PathBuf,
root: Option<PathBuf>,
/// TOML file: data_home = "..." and object_store = { type = "File" | "S3" | ... }.
#[arg(long)]
destination: Option<PathBuf>,
#[arg(long, default_value = "greptime_value")]
column: String,
#[arg(long)]
@@ -68,64 +90,140 @@ struct FooterColumnChunkReport {
num_values: i64,
}
pub(super) fn run_inspect_footer(
/// A parquet data file discovered by listing the store, with the metadata needed
/// to read its footer without any extra stat/head call.
#[derive(Debug)]
struct ListedFile {
location: StorePath,
size: u64,
path: String,
relative_path: String,
}
pub(super) async fn run_inspect_footer(
args: InspectFooterArgs,
) -> Result<(), Box<dyn std::error::Error>> {
let files = collect_parquet_files(&args.root, args.include_metadata_files)?;
let reports = files
.iter()
.map(|p| inspect_file(&args.root, p, &args.column))
.collect::<Result<Vec<_>, _>>()?;
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let (destination, root) = match (args.destination, args.root) {
(Some(_), Some(_)) => {
return Err("--destination and --root are mutually exclusive; pass exactly one".into());
}
(None, None) => return Err("one of --destination or --root is required".into()),
(Some(path), None) => {
let destination: DestinationConfig = toml::from_str(&fs::read_to_string(path)?)?;
(Some(destination), None)
}
(None, root) => (None, root),
};
let store = build_store(destination.as_ref(), root.as_deref()).await?;
let root_display = root
.as_deref()
.map(|root| root.display().to_string())
.unwrap_or_else(|| {
destination
.as_ref()
.expect("destination is set when root is not")
.data_home
.clone()
});
let files =
collect_parquet_files(store.as_ref(), root.as_deref(), args.include_metadata_files).await?;
let mut reports = futures::stream::iter(files.iter())
.map(|file| inspect_file(Arc::clone(&store), file, &args.column))
.buffer_unordered(8)
.try_collect::<Vec<_>>()
.await?;
reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
println!(
"{}",
serde_json::to_string_pretty(&FooterReport {
root: args.root.display().to_string(),
root: root_display,
summary: summarize_footer(&args.column, &reports),
files: reports
})?
);
Ok(())
}
fn collect_parquet_files(
root: &Path,
/// Builds an arrow `object_store::ObjectStore` for the given destination. The
/// File backend (and the `--root` shortcut) uses the opendal `Fs` builder
/// directly to avoid the create_dir_all / clean_temp_dir side effects of
/// `object_store::factory::new_fs_object_store`; every other backend goes
/// through `object_store::factory::new_raw_object_store`.
async fn build_store(
destination: Option<&DestinationConfig>,
root: Option<&Path>,
) -> Result<Arc<dyn ObjectStore>, Box<dyn std::error::Error + Send + Sync>> {
let operator = match destination {
Some(destination) => match &destination.object_store {
ObjectStoreConfig::File(_) => {
object_store::ObjectStore::new(Fs::default().root(&destination.data_home))?.finish()
}
_ => new_raw_object_store(&destination.object_store, &destination.data_home).await?,
},
None => {
let root = root.expect("root must be set when destination is None");
object_store::ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish()
}
};
Ok(Arc::new(object_store_opendal::OpendalStore::new(operator)))
}
async fn collect_parquet_files(
store: &dyn ObjectStore,
root: Option<&Path>,
include_metadata_files: bool,
) -> Result<Vec<PathBuf>, Box<dyn std::error::Error>> {
) -> Result<Vec<ListedFile>, Box<dyn std::error::Error + Send + Sync>> {
let metas = store.list(None).try_collect::<Vec<ObjectMeta>>().await?;
let mut files = Vec::new();
collect_parquet_files_inner(root, root, include_metadata_files, &mut files)?;
files.sort();
for meta in metas {
if !is_parquet_data_file(&meta) {
continue;
}
let is_metadata = meta
.location
.parts()
.any(|part| part.as_ref() == "metadata");
if !include_metadata_files && is_metadata {
continue;
}
let relative_path = meta.location.to_string();
let path = match root {
Some(root) => root.join(&relative_path).display().to_string(),
None => relative_path.clone(),
};
files.push(ListedFile {
location: meta.location,
size: meta.size,
path,
relative_path,
});
}
files.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
Ok(files)
}
fn collect_parquet_files_inner(
root: &Path,
path: &Path,
include_metadata_files: bool,
files: &mut Vec<PathBuf>,
) -> Result<(), Box<dyn std::error::Error>> {
if path.is_dir() {
for entry in fs::read_dir(path)? {
collect_parquet_files_inner(root, &entry?.path(), include_metadata_files, files)?;
}
} else if path.extension().is_some_and(|ext| ext == "parquet")
&& (include_metadata_files
|| !path
.strip_prefix(root)
.unwrap_or(path)
.components()
.any(|c| c.as_os_str() == "metadata"))
{
files.push(path.to_path_buf());
}
Ok(())
/// Keeps only real parquet data files: `.parquet` keys with a non-zero size.
/// Zero-size keys (and keys ending in `/`) are directory markers emitted by
/// some backends and are skipped even when they happen to end in `.parquet`,
/// since an empty parquet file has no readable footer anyway.
fn is_parquet_data_file(meta: &ObjectMeta) -> bool {
meta.location.extension() == Some("parquet")
&& meta.size > 0
&& !meta.location.as_ref().ends_with('/')
}
fn inspect_file(
root: &Path,
path: &Path,
async fn inspect_file(
store: Arc<dyn ObjectStore>,
file: &ListedFile,
column: &str,
) -> Result<FooterFileReport, Box<dyn std::error::Error>> {
let file_size = fs::metadata(path)?.len();
let reader = SerializedFileReader::new(File::open(path)?)?;
let metadata = reader.metadata().file_metadata();
let row_groups = reader.metadata().row_groups();
) -> Result<FooterFileReport, Box<dyn std::error::Error + Send + Sync>> {
let mut reader =
ParquetObjectReader::new(store, file.location.clone()).with_file_size(file.size);
let metadata = ParquetMetaDataReader::new()
.load_and_finish(&mut reader, file.size)
.await?;
let file_metadata = metadata.file_metadata();
let row_groups = metadata.row_groups();
let mut columns = Vec::new();
for (row_group_index, rg) in row_groups.iter().enumerate() {
for (column_index, chunk) in rg.columns().iter().enumerate() {
@@ -145,14 +243,10 @@ fn inspect_file(
}
}
Ok(FooterFileReport {
path: path.display().to_string(),
relative_path: path
.strip_prefix(root)
.unwrap_or(path)
.display()
.to_string(),
file_size,
num_rows: metadata.num_rows(),
path: file.path.clone(),
relative_path: file.relative_path.clone(),
file_size: file.size,
num_rows: file_metadata.num_rows(),
num_row_groups: row_groups.len(),
columns,
})
@@ -179,3 +273,199 @@ fn summarize_footer(column: &str, files: &[FooterFileReport]) -> FooterSummary {
s.unique_encodings = encodings.into_iter().collect();
s
}
#[cfg(test)]
mod tests {
use std::fs::{self, File};
use parquet::column::writer::ColumnWriter;
use parquet::file::properties::WriterProperties;
use parquet::file::writer::SerializedFileWriter;
use parquet::schema::parser::parse_message_type;
use tempfile::TempDir;
use super::*;
/// Writes a minimal valid parquet file with a single `value` Int64 column
/// and `rows` values, so the footer can be read back through the object
/// store path.
fn write_parquet_file(path: &Path, rows: i64) {
let schema = Arc::new(
parse_message_type("message schema { REQUIRED INT64 value; }").expect("valid schema"),
);
let props = Arc::new(WriterProperties::builder().build());
let file = File::create(path).expect("create parquet file");
let mut writer = SerializedFileWriter::new(file, schema, props).expect("init writer");
let mut row_group = writer.next_row_group().expect("next row group");
let mut column = row_group
.next_column()
.expect("next column")
.expect("has column");
match column.untyped() {
ColumnWriter::Int64ColumnWriter(typed) => {
typed
.write_batch(&(1..=rows).collect::<Vec<_>>(), None, None)
.expect("write batch");
}
_ => panic!("unexpected column writer"),
}
column.close().expect("close column");
row_group.close().expect("close row group");
writer.close().expect("close writer");
}
fn fixture_tree(dir: &Path) {
// Nested data tree with a `metadata` subdirectory, a non-parquet file
// and a zero-size fake file.
let table = dir.join("data/table");
fs::create_dir_all(&table).expect("create table dir");
fs::create_dir_all(table.join("metadata")).expect("create metadata dir");
write_parquet_file(&table.join("0001.parquet"), 3);
write_parquet_file(&table.join("0002.parquet"), 5);
write_parquet_file(&table.join("metadata/0001.parquet"), 7);
fs::write(table.join("notes.txt"), b"not parquet").expect("write notes");
File::create(table.join("empty.parquet")).expect("create empty file");
}
/// Runs the full list -> filter -> footer-read path against a File backend
/// rooted at `dir` and returns the collected file reports.
async fn inspect_fixture(dir: &Path, include_metadata_files: bool) -> Vec<FooterFileReport> {
let root = Some(dir.to_path_buf());
let store = build_store(None, root.as_deref())
.await
.expect("build store");
let files = collect_parquet_files(store.as_ref(), root.as_deref(), include_metadata_files)
.await
.expect("collect files");
let mut reports = futures::stream::iter(files.iter())
.map(|file| inspect_file(Arc::clone(&store), file, "value"))
.buffer_unordered(8)
.try_collect::<Vec<_>>()
.await
.expect("inspect files");
reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path));
reports
}
#[tokio::test]
async fn lists_and_inspects_footer_files_without_metadata_by_default() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let reports = inspect_fixture(dir.path(), false).await;
let relative_paths = reports
.iter()
.map(|report| report.relative_path.as_str())
.collect::<Vec<_>>();
// metadata/0001.parquet excluded, notes.txt and empty.parquet excluded.
assert_eq!(
relative_paths,
vec!["data/table/0001.parquet", "data/table/0002.parquet"]
);
assert!(reports.iter().all(|report| report.file_size > 0));
assert!(reports.iter().all(|report| report.num_row_groups == 1));
assert!(reports.iter().all(|report| report.num_rows > 0));
assert!(
reports
.iter()
.all(|report| report.columns.len() == 1 && report.columns[0].column_path == "value")
);
// Default writer properties use dictionary encoding, so the column chunk
// reports PLAIN (dictionary page) plus RLE_DICTIONARY (data pages).
assert!(
reports
.iter()
.all(|report| report.columns[0].encodings.contains(&"PLAIN".to_string()))
);
assert!(
reports
.iter()
.all(|report| report.columns[0].compression == "UNCOMPRESSED")
);
}
#[tokio::test]
async fn includes_metadata_files_when_requested() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let reports = inspect_fixture(dir.path(), true).await;
let relative_paths = reports
.iter()
.map(|report| report.relative_path.as_str())
.collect::<Vec<_>>();
assert_eq!(
relative_paths,
vec![
"data/table/0001.parquet",
"data/table/0002.parquet",
"data/table/metadata/0001.parquet"
]
);
let metadata_report = reports
.iter()
.find(|report| report.relative_path == "data/table/metadata/0001.parquet")
.expect("metadata report");
assert_eq!(metadata_report.num_rows, 7);
}
#[tokio::test]
async fn accepts_a_file_destination_toml() {
let dir = TempDir::new().expect("tempdir");
fixture_tree(dir.path());
let destination = dir.path().join("destination.toml");
fs::write(
&destination,
format!(
"data_home = \"{}\"\n[object_store]\ntype = \"File\"\n",
dir.path().display()
),
)
.expect("write destination toml");
let destination: DestinationConfig =
toml::from_str(&fs::read_to_string(&destination).expect("read toml"))
.expect("parse toml");
let store = build_store(Some(&destination), None)
.await
.expect("build store");
let files = collect_parquet_files(store.as_ref(), None, false)
.await
.expect("collect files");
// Destination mode has no root prefix: paths are store-relative keys.
assert_eq!(
files
.iter()
.map(|file| file.relative_path.as_str())
.collect::<Vec<_>>(),
vec!["data/table/0001.parquet", "data/table/0002.parquet"]
);
assert!(files.iter().all(|file| file.path == file.relative_path));
}
#[test]
fn requires_exactly_one_of_root_or_destination() {
use futures::FutureExt;
let both = run_inspect_footer(InspectFooterArgs {
root: Some(PathBuf::from("/tmp/root")),
destination: Some(PathBuf::from("/tmp/destination.toml")),
column: "value".to_string(),
include_metadata_files: false,
})
.now_or_never();
assert!(matches!(both, Some(Err(_))));
let neither = run_inspect_footer(InspectFooterArgs {
root: None,
destination: None,
column: "value".to_string(),
include_metadata_files: false,
})
.now_or_never();
assert!(matches!(neither, Some(Err(_))));
}
}
src/cmd/src/bin/query_perf_fixture/mod.rs
+3 -3
View File
@@ -95,9 +95,9 @@ pub async fn run() {
Some(Command::PromRemoteWrite(rw)) => run_prom_remote_write(rw)
.await
.expect("prom remote write failed"),
Some(Command::InspectFooter(inspect)) => {
run_inspect_footer(inspect).expect("inspect footer failed")
}
Some(Command::InspectFooter(inspect)) => run_inspect_footer(inspect)
.await
.expect("inspect footer failed"),
Some(Command::Plan(plan)) => run_plan(plan).expect("plan failed"),
None => run_direct_sst(args.legacy).await,
}
src/cmd/src/bin/query_regression_runner.rs
+23
View File
@@ -0,0 +1,23 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(clippy::print_stderr, clippy::print_stdout)]
#[path = "query_regression_runner/mod.rs"]
mod query_regression_runner;
#[tokio::main]
async fn main() {
query_regression_runner::run().await;
}
src/cmd/src/bin/query_regression_runner/direct.rs
+501
View File
@@ -0,0 +1,501 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, Instant};
use reqwest::Client;
use serde::Serialize;
use serde_json::{Value, json};
use crate::query_regression_runner::model::{Scenario, Table};
use crate::query_regression_runner::plan::{load_plan, validate_direct_tables};
use crate::query_regression_runner::sql::{
extract_rows, http_post_sql, row_u64, row_value, sql_ident, sql_string, value_text,
};
use crate::query_regression_runner::{PrepareDirectArgs, Result};
#[derive(Clone, Debug, Serialize)]
struct Discovery {
table: String,
catalog: String,
schema: String,
table_id: u64,
region_id: u64,
region_seq: u64,
table_dir: String,
region_dir: String,
peer_id: u64,
peer_addr: Value,
is_leader: Value,
status: Value,
discovery_queries: Value,
}
#[derive(Debug)]
struct PreparedTarget {
creates: Vec<Value>,
discoveries: Vec<Discovery>,
}
pub(super) async fn run_prepare_direct(args: PrepareDirectArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let case_path = args.case.canonicalize()?;
let plan = load_plan(&args.fixture_generator, &case_path)?;
let scenario_value = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
let scenario: Scenario = serde_json::from_value(scenario_value)?;
let tables = direct_tables(scenario)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = prepare_target("base", args.base_http_port, &tables, &client).await?;
let candidate = prepare_target("candidate", args.candidate_http_port, &tables, &client).await?;
if base.discoveries.len() != candidate.discoveries.len()
|| base
.discoveries
.iter()
.zip(&candidate.discoveries)
.any(|(base, candidate)| {
base.table != candidate.table
|| base.region_id != candidate.region_id
|| base.table_dir != candidate.table_dir
})
{
return Err(format!(
"base/candidate metadata mismatch: base={:?}, candidate={:?}",
base.discoveries, candidate.discoveries
)
.into());
}
let mut fixtures = Vec::with_capacity(tables.len());
for (index, (table, discovery)) in tables.iter().zip(&base.discoveries).enumerate() {
let fixture_dir = table_fixture_dir(&args.fixture_dir, &tables, table, index);
fixtures.push(generate_direct_fixture(
&args.fixture_generator,
&case_path,
&fixture_dir,
table,
discovery,
tables.len() > 1,
args.allow_large_fixture,
)?);
}
let report = json!({
"case_path": case_path,
"scenario": "direct_readable_sst",
"base": { "create_table": base.creates, "discovery": base.discoveries },
"candidate": { "create_table": candidate.creates, "discovery": candidate.discoveries },
"fixtures": fixtures,
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
Ok(())
}
fn direct_tables(scenario: Scenario) -> Result<Vec<Table>> {
match scenario {
Scenario::DirectReadableSst { tables, layout, .. } => {
validate_direct_tables(tables, layout)
}
Scenario::PromRemoteWriteThenQuery { .. } => {
Err("prepare-direct requires scenario kind direct_readable_sst".into())
}
Scenario::OtlpTraceLoad { .. } => {
Err("prepare-direct requires scenario kind direct_readable_sst".into())
}
}
}
async fn prepare_target(
name: &str,
port: u16,
tables: &[Table],
client: &Client,
) -> Result<PreparedTarget> {
let mut creates = Vec::with_capacity(tables.len());
let mut discoveries = Vec::with_capacity(tables.len());
for table in tables {
let sql = create_table_sql(table)?;
let result = http_post_sql(client, port, &sql, &table.database).await;
if !result["ok"].as_bool().unwrap_or(false) {
return Err(format!("CREATE TABLE {} failed for {name}: {result}", table.name).into());
}
creates.push(json!({ "table": table.name, "sql": sql, "result": result }));
discoveries.push(discover_region(client, port, table).await?);
}
Ok(PreparedTarget {
creates,
discoveries,
})
}
fn create_table_sql(table: &Table) -> Result<String> {
if table.engine != "mito" {
return Err("prepare-direct requires table engine mito".into());
}
let time_index = table
.time_index
.as_deref()
.ok_or("direct-SST table requires time_index")?;
if table.columns.is_empty() {
return Err("direct-SST table requires columns".into());
}
let columns = table
.columns
.iter()
.map(|column| format!("{} {}", sql_ident(&column.name), column.data_type))
.collect::<Vec<_>>()
.join(",\n ");
let primary_key = table
.primary_key
.iter()
.map(|column| sql_ident(column))
.collect::<Vec<_>>()
.join(", ");
let mut options = Vec::new();
if let Some(append_mode) = table.append_mode {
options.push(("append_mode", append_mode.to_string()));
}
if let Some(sst_format) = table
.sst_format
.as_deref()
.filter(|value| !value.is_empty())
{
options.push(("sst_format", sst_format.to_string()));
}
let with = (!options.is_empty()).then(|| {
format!(
"\nWITH ({})",
options
.iter()
.map(|(key, value)| format!("'{key}'='{value}'"))
.collect::<Vec<_>>()
.join(", ")
)
});
Ok(format!(
"CREATE TABLE {} (\n {columns},\n TIME INDEX ({}),\n PRIMARY KEY ({primary_key})\n) ENGINE=mito{};",
sql_ident(&table.name),
sql_ident(time_index),
with.unwrap_or_default()
))
}
async fn discover_region(client: &Client, port: u16, table: &Table) -> Result<Discovery> {
let schema = sql_string(&table.database);
let table_name = sql_string(&table.name);
let table_sql = format!(
"SELECT table_id FROM information_schema.tables WHERE table_schema = {schema} AND table_name = {table_name}"
);
let table_result = http_post_sql(client, port, &table_sql, &table.database).await;
if !table_result["ok"].as_bool().unwrap_or(false) {
return Err(format!("table_id discovery failed: {table_result}").into());
}
let table_rows = extract_rows(table_result.get("response").unwrap_or(&Value::Null));
if table_rows.len() != 1 {
return Err(format!(
"expected one information_schema.tables row, got {}: {table_result}",
table_rows.len()
)
.into());
}
let table_id = row_u64(&table_rows[0], 0, "table_id")?;
let region_sql = format!(
"SELECT region_id, peer_id, peer_addr, is_leader, status FROM information_schema.region_peers WHERE table_schema = {schema} AND table_name = {table_name}"
);
let region_result = http_post_sql(client, port, &region_sql, &table.database).await;
if !region_result["ok"].as_bool().unwrap_or(false) {
return Err(format!("region_peers discovery failed: {region_result}").into());
}
let region_rows = extract_rows(region_result.get("response").unwrap_or(&Value::Null));
if region_rows.len() != 1 {
return Err(format!(
"expected one information_schema.region_peers row, got {}: {region_result}",
region_rows.len()
)
.into());
}
let row = &region_rows[0];
let region_id = row_u64(row, 0, "region_id")?;
let peer_id = row_u64(row, 1, "peer_id")?;
let peer_addr = row_value(row, 2, "peer_addr")
.cloned()
.unwrap_or(Value::Null);
let is_leader = row_value(row, 3, "is_leader")
.cloned()
.unwrap_or(Value::Null);
let status = row_value(row, 4, "status").cloned().unwrap_or(Value::Null);
if !matches!(
value_text(&is_leader).to_ascii_lowercase().as_str(),
"yes" | "true"
) {
return Err(format!(
"expected leader region peer, got is_leader={is_leader}: {region_result}"
)
.into());
}
if !value_text(&status).eq_ignore_ascii_case("ALIVE") {
return Err(
format!("expected ALIVE region peer, got status={status}: {region_result}").into(),
);
}
if peer_id != 0 {
return Err(format!("expected region leader on datanode peer_id=0, got {peer_id}").into());
}
if region_id >> 32 != table_id {
return Err(format!(
"table_id mismatch: tables={table_id}, region_id-derived={}",
region_id >> 32
)
.into());
}
let region_seq = region_id & u32::MAX as u64;
let (table_dir, region_dir) = storage_paths(&table.database, table_id, region_seq);
Ok(Discovery {
table: table.name.clone(),
catalog: "greptime".to_string(),
schema: table.database.clone(),
table_id,
region_id,
region_seq,
table_dir: table_dir.clone(),
region_dir,
peer_id,
peer_addr,
is_leader,
status,
discovery_queries: json!({ "table": table_result, "region": region_result }),
})
}
fn storage_paths(database: &str, table_id: u64, region_seq: u64) -> (String, String) {
let table_dir = format!("data/greptime/{database}/{table_id}/");
let region_dir = format!("{table_dir}{table_id}_{region_seq:010}");
(table_dir, region_dir)
}
fn table_fixture_dir(root: &Path, tables: &[Table], table: &Table, index: usize) -> PathBuf {
if tables.len() == 1 {
root.to_path_buf()
} else {
root.join(fixture_subdir(table, index))
}
}
fn fixture_subdir(table: &Table, index: usize) -> String {
let raw = format!("{index:02}_{}_{}", table.database, table.name);
let mut safe = String::new();
let mut replaced = false;
for character in raw.chars() {
if character.is_ascii_alphanumeric() || matches!(character, '_' | '.' | '-') {
safe.push(character);
replaced = false;
} else if !replaced {
safe.push('_');
replaced = true;
}
}
let safe = safe.trim_matches(['.', '_', '-']);
if safe.is_empty() {
format!("table_{index:02}")
} else {
safe.to_string()
}
}
fn generate_direct_fixture(
generator: &Path,
case_path: &Path,
fixture_dir: &Path,
table: &Table,
discovery: &Discovery,
multi_table: bool,
allow_large_fixture: bool,
) -> Result<Value> {
if fixture_dir.exists() {
fs::remove_dir_all(fixture_dir)?;
}
fs::create_dir_all(fixture_dir)?;
let mut command = vec![
generator.to_string_lossy().to_string(),
"direct-sst".to_string(),
"--case".to_string(),
case_path.to_string_lossy().to_string(),
"--out-dir".to_string(),
fixture_dir.to_string_lossy().to_string(),
];
if multi_table {
command.extend(["--table".to_string(), table.name.clone()]);
}
command.extend([
"--region-id".to_string(),
discovery.region_id.to_string(),
"--table-dir".to_string(),
discovery.table_dir.clone(),
]);
if allow_large_fixture {
command.push("--allow-large".to_string());
}
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"fixture generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let summary: Value = serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?;
assert_fixture_summary(&summary, table, discovery)?;
Ok(json!({
"status": "ok",
"fixture_dir": fixture_dir,
"command": command,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
"summary": summary,
}))
}
fn assert_fixture_summary(summary: &Value, table: &Table, discovery: &Discovery) -> Result<()> {
if summary.get("table").and_then(Value::as_str) != Some(&table.name) {
return Err(format!(
"fixture table mismatch: {:?} != {}",
summary.get("table"),
table.name
)
.into());
}
if summary.get("database").and_then(Value::as_str) != Some(&table.database) {
return Err(format!(
"fixture database mismatch: {:?} != {}",
summary.get("database"),
table.database
)
.into());
}
if summary.get("region_id").and_then(|value| {
value
.as_u64()
.or_else(|| value.as_str().and_then(|value| value.parse().ok()))
}) != Some(discovery.region_id)
{
return Err(format!(
"fixture region_id mismatch: {:?} != {}",
summary.get("region_id"),
discovery.region_id
)
.into());
}
if summary.get("table_dir").and_then(Value::as_str) != Some(&discovery.table_dir) {
return Err(format!(
"fixture table_dir mismatch: {:?} != {}",
summary.get("table_dir"),
discovery.table_dir
)
.into());
}
if summary
.get("region_dir")
.and_then(Value::as_str)
.map(|value| value.trim_matches('/'))
!= Some(discovery.region_dir.trim_matches('/'))
{
return Err(format!(
"fixture region_dir mismatch: {:?} != {}",
summary.get("region_dir"),
discovery.region_dir
)
.into());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query_regression_runner::model::Column;
#[test]
fn creates_exact_direct_sst_table_sql() {
let table = Table {
database: "public".to_string(),
name: "metric\"name".to_string(),
engine: "mito".to_string(),
columns: vec![
Column {
name: "host".to_string(),
data_type: "STRING".to_string(),
},
Column {
name: "ts".to_string(),
data_type: "TIMESTAMP(9)".to_string(),
},
],
primary_key: vec!["host".to_string()],
time_index: Some("ts".to_string()),
append_mode: Some(true),
sst_format: Some("flat".to_string()),
validate_show_create_engine: true,
};
assert_eq!(
create_table_sql(&table).unwrap(),
"CREATE TABLE \"metric\"\"name\" (\n \"host\" STRING,\n \"ts\" TIMESTAMP(9),\n TIME INDEX (\"ts\"),\n PRIMARY KEY (\"host\")\n) ENGINE=mito\nWITH ('append_mode'='true', 'sst_format'='flat');"
);
}
#[test]
fn extracts_named_and_positional_discovery_rows() {
let rows = extract_rows(&json!({"output": [{"data": [{"TABLE_ID": "7"}]}]}));
assert_eq!(rows, vec![json!({"TABLE_ID": "7"})]);
assert_eq!(row_u64(&rows[0], 0, "table_id").unwrap(), 7);
let rows = extract_rows(&json!({"data": [[30064771073_u64, 0, "addr", "YES", "ALIVE"]]}));
assert_eq!(rows.len(), 1);
assert_eq!(row_u64(&rows[0], 0, "region_id").unwrap(), 30_064_771_073);
assert_eq!(row_u64(&rows[0], 1, "peer_id").unwrap(), 0);
assert_eq!(
value_text(row_value(&rows[0], 3, "is_leader").unwrap()),
"YES"
);
}
#[test]
fn storage_paths_include_the_table_directory() {
assert_eq!(
storage_paths("public", 1024, 0),
(
"data/greptime/public/1024/".to_string(),
"data/greptime/public/1024/1024_0000000000".to_string(),
)
);
}
}
src/cmd/src/bin/query_regression_runner/materialize.rs
+225
View File
@@ -0,0 +1,225 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::Path;
use futures::{AsyncWriteExt as _, TryStreamExt};
use object_store::ObjectStore;
use object_store::factory::new_raw_object_store;
use object_store::services::Fs;
use crate::query_regression_runner::model::{
CopyCounts, DestinationConfig, FixtureSummary, MaterializeResult,
};
use crate::query_regression_runner::{MaterializeArgs, Result};
pub(super) async fn run_materialize(args: MaterializeArgs) -> Result<()> {
let destination: DestinationConfig = toml::from_str(&fs::read_to_string(args.destination)?)?;
let result = materialize(&args.fixture_dir, destination).await?;
println!("{}", serde_json::to_string(&result)?);
Ok(())
}
async fn materialize(
fixture_dir: &Path,
destination: DestinationConfig,
) -> Result<MaterializeResult> {
let summary: FixtureSummary =
serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?;
let region_dir = validate_region_dir(&summary.region_dir)?;
let object_source = fs_operator(&fixture_dir.join("object-store"))?;
let manifest_source = fs_operator(&fixture_dir.join("manifest"))?;
let destination =
new_raw_object_store(&destination.object_store, &destination.data_home).await?;
let region_prefix = format!("{region_dir}/");
destination
.delete_with(&region_prefix)
.recursive(true)
.await?;
let object_store = copy_tree(&object_source, &destination, "/", "").await?;
let manifest = copy_tree(
&manifest_source,
&destination,
"/",
&format!("{region_dir}/manifest/"),
)
.await?;
Ok(MaterializeResult {
region_dir,
object_store,
manifest,
})
}
fn fs_operator(root: &Path) -> Result<ObjectStore> {
Ok(ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish())
}
fn validate_region_dir(region_dir: &str) -> Result<String> {
let region_dir = region_dir.trim_end_matches('/');
if region_dir.is_empty()
|| region_dir.starts_with('/')
|| region_dir.contains('\\')
|| region_dir
.split('/')
.any(|component| component.is_empty() || matches!(component, "." | ".."))
{
return Err(
"region_dir must be a non-empty relative OpenDAL key without dot components".into(),
);
}
Ok(region_dir.to_string())
}
async fn copy_tree(
source: &ObjectStore,
destination: &ObjectStore,
source_prefix: &str,
destination_prefix: &str,
) -> Result<CopyCounts> {
let mut lister = source.lister_with(source_prefix).recursive(true).await?;
let mut counts = CopyCounts::default();
while let Some(entry) = lister.try_next().await? {
if entry.metadata().is_dir() {
continue;
}
let source_path = entry.path().to_string();
let reader = source
.reader(&source_path)
.await?
.into_futures_async_read(0..entry.metadata().content_length())
.await?;
let mut writer = destination
.writer(&format!("{destination_prefix}{source_path}"))
.await?
.into_futures_async_write();
let bytes = futures::io::copy(reader, &mut writer).await?;
writer.close().await?;
counts.files += 1;
counts.bytes += bytes;
}
Ok(counts)
}
#[cfg(test)]
mod tests {
use std::fs;
use object_store::config::ObjectStoreConfig;
use super::*;
#[test]
fn region_dir_must_be_a_relative_opendal_key() {
assert_eq!(
validate_region_dir("data/db/region/").unwrap(),
"data/db/region"
);
for region_dir in [
"",
"/region",
".",
"a/../region",
"a/./region",
"a\\region",
"a//region",
] {
assert!(validate_region_dir(region_dir).is_err(), "{region_dir}");
}
}
#[tokio::test]
async fn materializes_fixture_from_fs_to_fs() {
let fixture = tempfile::tempdir().unwrap();
let destination = tempfile::tempdir().unwrap();
let region_dir = "data/public/metrics/00000000000000000001";
let sst = fixture
.path()
.join("object-store")
.join(region_dir)
.join("00000000000000000001.parquet");
fs::create_dir_all(sst.parent().unwrap()).unwrap();
fs::write(&sst, b"sst").unwrap();
let checkpoint = fixture
.path()
.join("manifest/00000000000000000001.checkpoint");
fs::create_dir_all(checkpoint.parent().unwrap()).unwrap();
fs::write(&checkpoint, b"checkpoint").unwrap();
fs::write(fixture.path().join("manifest/_last_checkpoint"), b"last").unwrap();
fs::write(
fixture.path().join("summary.json"),
format!(r#"{{"region_dir":"{region_dir}"}}"#),
)
.unwrap();
fs::write(fixture.path().join("files.jsonl"), "metadata").unwrap();
fs::create_dir_all(destination.path().join(region_dir)).unwrap();
fs::write(destination.path().join(region_dir).join("stale"), "stale").unwrap();
fs::write(destination.path().join("unrelated"), "keep").unwrap();
let result = materialize(
fixture.path(),
DestinationConfig {
data_home: destination.path().to_string_lossy().to_string(),
object_store: ObjectStoreConfig::default(),
},
)
.await
.unwrap();
assert_eq!(result.object_store.files, 1);
assert_eq!(result.object_store.bytes, 3);
assert_eq!(result.manifest.files, 2);
assert_eq!(result.manifest.bytes, 14);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("00000000000000000001.parquet")
)
.unwrap(),
b"sst"
);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("manifest/00000000000000000001.checkpoint")
)
.unwrap(),
b"checkpoint"
);
assert_eq!(
fs::read(
destination
.path()
.join(region_dir)
.join("manifest/_last_checkpoint")
)
.unwrap(),
b"last"
);
assert!(!destination.path().join(region_dir).join("stale").exists());
assert_eq!(
fs::read(destination.path().join("unrelated")).unwrap(),
b"keep"
);
assert!(!destination.path().join("summary.json").exists());
assert!(!destination.path().join("files.jsonl").exists());
}
}
src/cmd/src/bin/query_regression_runner/measure.rs
+423
View File
@@ -0,0 +1,423 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::time::Duration;
use std::{fs, io};
use reqwest::Client;
use serde_json::{Map, Value, json};
use crate::query_regression_runner::model::{Measurement, Query, QueryResult, Scenario, Table};
use crate::query_regression_runner::plan::{load_plan, normalize_scenario};
use crate::query_regression_runner::sql::{http_post_sql, sql_ident};
use crate::query_regression_runner::{MeasureArgs, Result};
pub(super) async fn run_measure(args: MeasureArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let case_path = args.case.canonicalize()?;
let plan = load_plan(&args.fixture_generator, &case_path)?;
let case_text = fs::read_to_string(&case_path)?;
let raw_case: toml::Value = toml::from_str(&case_text)?;
let case_metadata = raw_case
.get("case")
.cloned()
.unwrap_or(toml::Value::Table(toml::map::Map::new()));
let case_metadata = serde_json::to_value(case_metadata)?;
let scenario_value = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
let scenario: Scenario = serde_json::from_value(scenario_value.clone())?;
let (tables, configured_queries) = normalize_scenario(scenario)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = run_target(args.base_http_port, &tables, &configured_queries, &client).await;
let candidate = run_target(
args.candidate_http_port,
&tables,
&configured_queries,
&client,
)
.await;
let thresholds = enforce_thresholds(&configured_queries, &base, &candidate)?;
let status = if base.status == "failed"
|| candidate.status == "failed"
|| thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
{
"failed"
} else {
"ok"
};
let report = json!({
"case_path": case_path,
"case": case_metadata,
"scenario": scenario_value,
"queries": configured_queries,
"query_mode": "endpoint",
"http_timeout": args.http_timeout,
"targets": [target_report("base", args.base_http_port, base), target_report("candidate", args.candidate_http_port, candidate)],
"thresholds": thresholds,
"status": status,
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
if status == "failed" {
std::process::exit(1);
}
Ok(())
}
fn target_report(name: &str, http_port: u16, result: QueryResult) -> Value {
let status = if result.status == "ok" {
"measured"
} else {
"failed"
};
json!({
"name": name,
"http_port": http_port,
"validation": result.validation,
"validation_errors": result.validation_errors,
"measurements": result.measurements,
"status": status,
})
}
async fn run_target(
port: u16,
tables: &[Table],
configured_queries: &[Query],
client: &Client,
) -> QueryResult {
let mut queries = configured_queries.to_vec();
if queries.is_empty() {
queries.push(Query {
name: Some("count_all".to_string()),
kind: Some("sql".to_string()),
query: format!("SELECT count(*) FROM {}", sql_ident(&tables[0].name)),
warmup: 0,
iterations: 1,
thresholds: Map::new(),
});
}
let db = &tables[0].database;
let mut validation = Vec::new();
let mut validation_errors = Vec::new();
for table in tables {
let sql = format!("SHOW CREATE TABLE {}", sql_ident(&table.name));
let sample = http_post_sql(client, port, &sql, &table.database).await;
if !sample["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": sql,
"error": sample.get("error"),
"response": sample.get("response"),
}));
} else {
for error in validate_show_create(&sample, table) {
validation_errors.push(json!({
"sql": sql,
"error": error,
"response": sample.get("response"),
}));
}
}
validation.push(sample);
}
let first = http_post_sql(client, port, &queries[0].query, db).await;
if !first["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": queries[0].query,
"error": first.get("error"),
"response": first.get("response"),
}));
}
validation.push(first);
let mut measurements = Vec::with_capacity(queries.len());
for query in &queries {
for _ in 0..query.warmup {
let warmup = http_post_sql(client, port, &query.query, db).await;
if !warmup["ok"].as_bool().unwrap_or(false) {
validation_errors.push(json!({
"sql": query.query,
"phase": "warmup",
"error": warmup.get("error"),
"response": warmup.get("response"),
}));
}
}
let mut samples = Vec::with_capacity(query.iterations);
let mut good_latencies = Vec::with_capacity(query.iterations);
for _ in 0..query.iterations {
let mut sample = http_post_sql(client, port, &query.query, db).await;
let execution_time = sample
.get("response")
.and_then(extract_execution_time)
.cloned()
.unwrap_or(Value::Null);
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("execution_time_ms".to_string(), execution_time);
if sample["ok"].as_bool().unwrap_or(false) {
good_latencies.push(sample["latency_ms"].as_f64().unwrap_or_default());
}
samples.push(sample);
}
let median = (!good_latencies.is_empty()).then(|| median(&good_latencies));
let p95 = (!good_latencies.is_empty()).then(|| percentile(&good_latencies, 95.0));
let status = if good_latencies.len() == samples.len() {
"ok"
} else {
"failed"
};
measurements.push(Measurement {
name: query.name.clone(),
kind: query.kind.clone(),
iterations: samples.len(),
samples,
latency_ms_median: median,
latency_ms_p95: p95,
status: status.to_string(),
});
}
let failed = !validation_errors.is_empty() || measurements.iter().any(|m| m.status == "failed");
QueryResult {
validation,
validation_errors,
measurements,
status: if failed { "failed" } else { "ok" }.to_string(),
}
}
fn validate_show_create(result: &Value, table: &Table) -> Vec<&'static str> {
let text = result
.get("response")
.map(response_text)
.unwrap_or_default()
.to_lowercase();
let mut errors = Vec::new();
if !text.contains(&table.name.to_lowercase()) {
errors.push("SHOW CREATE output does not contain table name");
}
if table.validate_show_create_engine && (!text.contains("engine") || !text.contains("mito")) {
errors.push("SHOW CREATE output does not mention ENGINE=mito");
}
if table.append_mode.is_some() && !text.contains("append_mode") {
errors.push("SHOW CREATE output does not mention append_mode");
}
if table.sst_format.is_some() && !text.contains("sst_format") {
errors.push("SHOW CREATE output does not mention sst_format");
}
errors
}
fn response_text(body: &Value) -> String {
body.as_str()
.map(ToOwned::to_owned)
.unwrap_or_else(|| serde_json::to_string(body).unwrap_or_default())
}
fn extract_execution_time(body: &Value) -> Option<&Value> {
match body {
Value::Object(map) => {
for key in ["execution_time_ms", "execution_time", "elapsed"] {
if let Some(value) = map.get(key) {
return Some(value);
}
}
map.values().find_map(extract_execution_time)
}
Value::Array(values) => values.iter().find_map(extract_execution_time),
_ => None,
}
}
pub(super) fn median(values: &[f64]) -> f64 {
let mut ordered = values.to_vec();
ordered.sort_by(f64::total_cmp);
let middle = ordered.len() / 2;
if ordered.len().is_multiple_of(2) {
(ordered[middle - 1] + ordered[middle]) / 2.0
} else {
ordered[middle]
}
}
fn percentile(values: &[f64], pct: f64) -> f64 {
if values.is_empty() {
return 0.0;
}
let mut ordered = values.to_vec();
ordered.sort_by(f64::total_cmp);
let index = round_ties_even((pct / 100.0) * (ordered.len() - 1) as f64)
.clamp(0, ordered.len() as isize - 1) as usize;
ordered[index]
}
pub(super) fn round_ties_even(value: f64) -> isize {
let floor = value.floor();
let fraction = value - floor;
if fraction < 0.5 {
floor as isize
} else if fraction > 0.5 {
floor as isize + 1
} else if (floor as isize) % 2 == 0 {
floor as isize
} else {
floor as isize + 1
}
}
fn enforce_thresholds(
queries: &[Query],
base: &QueryResult,
candidate: &QueryResult,
) -> Result<Vec<Value>> {
let base_by_name: HashMap<_, _> = base
.measurements
.iter()
.map(|measurement| (measurement.name.as_deref(), measurement))
.collect();
let mut results = Vec::new();
for candidate_measurement in &candidate.measurements {
let query = queries
.iter()
.find(|query| query.name == candidate_measurement.name);
let thresholds = query.map_or_else(Map::new, |query| query.thresholds.clone());
let base_measurement = base_by_name.get(&candidate_measurement.name.as_deref());
if let Some(limit) = thresholds
.get("max_candidate_latency_regression_pct")
.filter(|value| !value.is_null())
.map(value_as_f64)
.transpose()?
{
let result = match base_measurement {
None => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing base measurement"})
}
Some(base) if matches!(base.latency_ms_median, None | Some(0.0)) => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "base median latency is missing or zero", "base_latency_ms_median": base.latency_ms_median})
}
Some(_) if candidate_measurement.latency_ms_median.is_none() => {
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing candidate measurement"})
}
Some(base) => {
let actual = (candidate_measurement.latency_ms_median.unwrap()
- base.latency_ms_median.unwrap())
/ base.latency_ms_median.unwrap()
* 100.0;
json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit})
}
};
results.push(result);
}
for key in thresholds
.keys()
.filter(|key| *key != "max_candidate_latency_regression_pct")
{
results.push(json!({"query": candidate_measurement.name, "threshold": key, "status": "failed", "reason": "unsupported threshold"}));
}
}
Ok(results)
}
fn value_as_f64(value: &Value) -> Result<f64> {
value.as_f64().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidData, "threshold must be numeric").into()
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn median_and_p95_match_python() {
assert_eq!(median(&[1.0, 8.0, 3.0, 4.0]), 3.5);
assert_eq!(
percentile(
&[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0],
95.0
),
10.0
);
assert_eq!(percentile(&[], 95.0), 0.0);
}
#[test]
fn finds_nested_execution_time_in_priority_order() {
let body = json!({"output": [{"elapsed": 3}], "execution_time": 2});
assert_eq!(extract_execution_time(&body), Some(&json!(2)));
assert_eq!(
extract_execution_time(&json!({"output": [{"elapsed": 3}]})),
Some(&json!(3))
);
}
#[test]
fn threshold_rejects_unknown_keys_and_zero_base() {
let query = Query {
name: Some("q".to_string()),
kind: None,
query: "SELECT 1".to_string(),
warmup: 0,
iterations: 1,
thresholds: Map::from_iter([
("max_candidate_latency_regression_pct".to_string(), json!(0)),
("other".to_string(), json!(1)),
]),
};
let measurement = |median| Measurement {
name: Some("q".to_string()),
kind: None,
iterations: 1,
samples: vec![],
latency_ms_median: median,
latency_ms_p95: median,
status: "ok".to_string(),
};
let base = QueryResult {
validation: vec![],
validation_errors: vec![],
measurements: vec![measurement(Some(0.0))],
status: "ok".to_string(),
};
let candidate = QueryResult {
validation: vec![],
validation_errors: vec![],
measurements: vec![measurement(Some(1.0))],
status: "ok".to_string(),
};
let results = enforce_thresholds(&[query], &base, &candidate).unwrap();
assert_eq!(
results[0]["reason"],
"base median latency is missing or zero"
);
assert_eq!(results[1]["reason"], "unsupported threshold");
}
}
src/cmd/src/bin/query_regression_runner/mod.rs
+204
View File
@@ -0,0 +1,204 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::error::Error;
use std::path::PathBuf;
use clap::{Parser, Subcommand};
mod direct;
mod materialize;
mod measure;
mod model;
mod otlp;
mod plan;
mod remote;
mod sql;
pub(super) type Result<T> = std::result::Result<T, Box<dyn Error>>;
#[derive(Debug, Parser)]
#[command(about = "Measure query endpoints or materialize a direct-SST fixture")]
struct Cli {
#[command(subcommand)]
command: RunnerCommand,
}
#[derive(Debug, Subcommand)]
enum RunnerCommand {
/// Run normalized query regression cases against two HTTP endpoints.
Measure(MeasureArgs),
/// Create direct-SST tables, discover their regions, and generate fixtures.
PrepareDirect(PrepareDirectArgs),
/// Render the frontend Prometheus store configuration for a remote-write case.
RenderRemoteConfig(RenderRemoteConfigArgs),
/// Create a database and ingest a normalized Prometheus remote-write case.
PrepareRemote(PrepareRemoteArgs),
/// Inspect stopped remote-write storage and finalize the aggregate report.
FinalizeRemote(FinalizeRemoteArgs),
/// Run one externally managed OTLP trace-load target.
RunOtlpTarget(RunOtlpTargetArgs),
/// Combine externally managed OTLP trace-load target results.
FinalizeOtlp(FinalizeOtlpArgs),
/// Copy a direct-SST fixture into an object store destination.
Materialize(MaterializeArgs),
}
#[derive(Debug, Parser)]
struct MeasureArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct PrepareDirectArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
fixture_dir: PathBuf,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
#[arg(long)]
allow_large_fixture: bool,
}
#[derive(Debug, Parser)]
struct RenderRemoteConfigArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
}
#[derive(Debug, Parser)]
struct PrepareRemoteArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long)]
base_http_port: u16,
#[arg(long)]
candidate_http_port: u16,
#[arg(long, value_name = "PATH")]
output: Option<PathBuf>,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct FinalizeRemoteArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
candidate_bin: PathBuf,
#[arg(long, value_name = "PATH")]
base_data_home: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
candidate_data_home: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
base_destination: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
candidate_destination: Option<PathBuf>,
#[arg(long, value_name = "PATH")]
report: PathBuf,
}
#[derive(Debug, Parser)]
struct RunOtlpTargetArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
otelgen_bin: PathBuf,
#[arg(long)]
http_port: u16,
#[arg(long)]
target_name: String,
#[arg(long, value_name = "PATH")]
work_dir: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
#[arg(long, default_value_t = 120.0)]
http_timeout: f64,
}
#[derive(Debug, Parser)]
struct FinalizeOtlpArgs {
#[arg(long, value_name = "PATH")]
case: PathBuf,
#[arg(long, value_name = "PATH")]
fixture_generator: PathBuf,
#[arg(long, value_name = "PATH")]
base_result: PathBuf,
#[arg(long, value_name = "PATH")]
candidate_result: PathBuf,
#[arg(long, value_name = "PATH")]
output: PathBuf,
}
#[derive(Debug, Parser)]
struct MaterializeArgs {
#[arg(long, value_name = "PATH")]
fixture_dir: PathBuf,
#[arg(
long,
value_name = "PATH",
help = "TOML file: data_home = \"...\" and object_store = { type = \"File\" }; object_store uses object_store::config::ObjectStoreConfig"
)]
destination: PathBuf,
}
pub(super) async fn run() {
if let Err(error) = run_inner().await {
eprintln!("query_regression_runner: {error}");
std::process::exit(1);
}
}
async fn run_inner() -> Result<()> {
match Cli::parse().command {
RunnerCommand::Measure(args) => measure::run_measure(args).await,
RunnerCommand::PrepareDirect(args) => direct::run_prepare_direct(args).await,
RunnerCommand::RenderRemoteConfig(args) => remote::run_render_remote_config(args).await,
RunnerCommand::PrepareRemote(args) => remote::run_prepare_remote(args).await,
RunnerCommand::FinalizeRemote(args) => remote::run_finalize_remote(args).await,
RunnerCommand::RunOtlpTarget(args) => otlp::run_otlp_target(args).await,
RunnerCommand::FinalizeOtlp(args) => otlp::run_finalize_otlp(args).await,
RunnerCommand::Materialize(args) => materialize::run_materialize(args).await,
}
}
src/cmd/src/bin/query_regression_runner/model.rs
+235
View File
@@ -0,0 +1,235 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use object_store::config::ObjectStoreConfig;
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
#[derive(Debug, Deserialize)]
pub(super) struct DestinationConfig {
pub(super) data_home: String,
pub(super) object_store: ObjectStoreConfig,
}
#[derive(Debug, Deserialize)]
pub(super) struct FixtureSummary {
pub(super) region_dir: String,
}
#[derive(Debug, Default, Serialize)]
pub(super) struct CopyCounts {
pub(super) files: u64,
pub(super) bytes: u64,
}
#[derive(Debug, Serialize)]
pub(super) struct MaterializeResult {
pub(super) region_dir: String,
pub(super) object_store: CopyCounts,
pub(super) manifest: CopyCounts,
}
#[derive(Debug, Deserialize)]
#[serde(tag = "kind")]
pub(super) enum Scenario {
#[serde(rename = "direct_readable_sst")]
DirectReadableSst {
tables: Vec<Table>,
layout: Layout,
#[serde(default)]
queries: Vec<Query>,
},
#[serde(rename = "prom_remote_write_then_query")]
PromRemoteWriteThenQuery {
remote_write: RemoteWrite,
#[serde(default)]
queries: Vec<Query>,
},
#[serde(rename = "otlp_trace_load")]
OtlpTraceLoad { load: OtlpTraceLoad },
}
#[derive(Debug, Deserialize)]
pub(super) struct Layout {
pub(super) regions: usize,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct Table {
pub(super) database: String,
pub(super) name: String,
#[serde(default)]
pub(super) engine: String,
#[serde(default)]
pub(super) columns: Vec<Column>,
#[serde(default)]
pub(super) primary_key: Vec<String>,
#[serde(default)]
pub(super) time_index: Option<String>,
#[serde(default)]
pub(super) append_mode: Option<bool>,
#[serde(default)]
pub(super) sst_format: Option<String>,
#[serde(default = "default_show_create_engine")]
pub(super) validate_show_create_engine: bool,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct Column {
pub(super) name: String,
#[serde(rename = "type")]
pub(super) data_type: String,
}
const fn default_show_create_engine() -> bool {
true
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct RemoteWrite {
pub(super) database: String,
pub(super) metric: String,
pub(super) physical_table: String,
pub(super) series_count: u64,
pub(super) samples_per_series: u64,
pub(super) start_unix_millis: i64,
pub(super) step_millis: i64,
pub(super) chunk_series_count: u64,
pub(super) timeout_seconds: u64,
pub(super) sample_chunk_size: Option<u64>,
pub(super) flush_every_sample_chunks: u64,
pub(super) visibility_timeout_seconds: u64,
pub(super) prom_store: PromStore,
pub(super) value: RemoteValue,
pub(super) storage: Option<StorageConfig>,
pub(super) read_bench: Option<ReadBenchConfig>,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct PromStore {
pub(super) pending_rows_flush_interval: String,
pub(super) max_batch_rows: u64,
pub(super) max_concurrent_flushes: u64,
pub(super) worker_channel_capacity: u64,
pub(super) max_inflight_requests: u64,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct RemoteValue {
pub(super) pattern: String,
pub(super) base: f64,
pub(super) step: f64,
pub(super) cardinality: u64,
pub(super) seed: u64,
pub(super) run_length: u64,
pub(super) stall_every: u64,
pub(super) stall_length: u64,
pub(super) mixed_every: u64,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpTraceLoad {
pub(super) load: OtlpLoad,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpLoad {
pub(super) database: String,
pub(super) table: String,
pub(super) pipeline: String,
pub(super) duration_seconds: u64,
pub(super) warmup_seconds: u64,
pub(super) rate: u64,
pub(super) workers: usize,
pub(super) exporter_shards: usize,
pub(super) workload: String,
pub(super) visibility_timeout_seconds: u64,
pub(super) thresholds: OtlpThresholds,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct OtlpThresholds {
pub(super) max_candidate_throughput_regression_pct: f64,
pub(super) max_candidate_mean_latency_regression_pct: f64,
pub(super) max_failure_count: u64,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct StorageConfig {
pub(super) inspect: bool,
pub(super) column: String,
pub(super) root_suffix: Option<String>,
pub(super) include_metadata_files: bool,
pub(super) min_files: u64,
pub(super) min_files_with_column: u64,
pub(super) require_encodings: Vec<String>,
pub(super) forbid_encodings: Vec<String>,
pub(super) max_total_file_size_bytes: Option<u64>,
pub(super) max_column_compressed_size_bytes: Option<u64>,
pub(super) max_column_uncompressed_size_bytes: Option<u64>,
pub(super) max_candidate_total_file_size_regression_pct: Option<f64>,
pub(super) max_candidate_column_compressed_size_regression_pct: Option<f64>,
pub(super) max_candidate_column_uncompressed_size_regression_pct: Option<f64>,
}
#[derive(Clone, Debug, Deserialize)]
pub(super) struct ReadBenchConfig {
pub(super) enabled: bool,
pub(super) parquetbench: bool,
pub(super) scanbench: bool,
pub(super) iterations: u64,
pub(super) projection: Vec<String>,
pub(super) parquet_reader: String,
pub(super) scan_scanner: String,
pub(super) parallelism: u64,
pub(super) max_files: Option<usize>,
}
#[derive(Clone, Debug, Deserialize, Serialize)]
pub(super) struct Query {
#[serde(default)]
pub(super) name: Option<String>,
#[serde(default)]
pub(super) kind: Option<String>,
pub(super) query: String,
#[serde(default)]
pub(super) warmup: usize,
#[serde(default = "one")]
pub(super) iterations: usize,
#[serde(default)]
pub(super) thresholds: Map<String, Value>,
}
const fn one() -> usize {
1
}
#[derive(Debug, Serialize)]
pub(super) struct QueryResult {
pub(super) validation: Vec<Value>,
pub(super) validation_errors: Vec<Value>,
pub(super) measurements: Vec<Measurement>,
pub(super) status: String,
}
#[derive(Debug, Serialize)]
pub(super) struct Measurement {
pub(super) name: Option<String>,
pub(super) kind: Option<String>,
pub(super) iterations: usize,
pub(super) samples: Vec<Value>,
pub(super) latency_ms_median: Option<f64>,
pub(super) latency_ms_p95: Option<f64>,
pub(super) status: String,
}
src/cmd/src/bin/query_regression_runner/otlp.rs
+618
View File
@@ -0,0 +1,618 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashMap;
use std::fs;
use std::path::Path;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use regex::Regex;
use reqwest::Client;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use crate::query_regression_runner::measure::round_ties_even;
use crate::query_regression_runner::model::{OtlpLoad, OtlpThresholds};
use crate::query_regression_runner::plan::normalized_otlp_load;
use crate::query_regression_runner::sql::{
extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64,
};
use crate::query_regression_runner::{FinalizeOtlpArgs, Result, RunOtlpTargetArgs};
const OTLP_ROWS: &str = "greptime_frontend_otlp_traces_rows";
const OTLP_FAILURES: &str = "greptime_frontend_otlp_traces_failure_count";
const OTLP_ELAPSED_SUM: &str = "greptime_servers_http_otlp_traces_elapsed_sum";
const OTLP_ELAPSED_COUNT: &str = "greptime_servers_http_otlp_traces_elapsed_count";
#[derive(Clone, Debug, Deserialize, Serialize)]
struct OtlpSnapshot {
captured_monotonic_seconds: f64,
values: HashMap<String, f64>,
}
pub(super) async fn run_otlp_target(args: RunOtlpTargetArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let (_, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let result = match run_otlp_target_inner(&args, &load, &client).await {
Ok(result) => result,
Err(error) => {
json!({ "name": args.target_name, "status": "failed", "error": error.to_string() })
}
};
let text = format!("{}\n", serde_json::to_string_pretty(&result)?);
fs::write(&args.output, &text)?;
print!("{text}");
if result["status"] == "failed" {
std::process::exit(1);
}
Ok(())
}
async fn run_otlp_target_inner(
args: &RunOtlpTargetArgs,
load: &OtlpLoad,
client: &Client,
) -> Result<Value> {
let create_database = http_post_sql(
client,
args.http_port,
&format!(
"CREATE DATABASE IF NOT EXISTS {}",
sql_ident(&load.database)
),
"public",
)
.await;
if !create_database["ok"].as_bool().unwrap_or(false) {
return Ok(
json!({ "name": args.target_name, "create_database": create_database, "status": "failed" }),
);
}
let otelgen = run_otelgen_load(
&args.otelgen_bin,
args.http_port,
&args.work_dir,
load,
client,
)
.await?;
let metrics = summarize_otlp_metrics(&otelgen)?;
let flush = http_post_sql(
client,
args.http_port,
&format!("ADMIN FLUSH_TABLE({})", sql_string(&load.table)),
&load.database,
)
.await;
let visibility = poll_otlp_visibility(
client,
args.http_port,
&load.table,
&load.database,
metrics["accepted_spans"].as_u64().unwrap_or_default(),
load.visibility_timeout_seconds,
)
.await?;
let checks_ok = otelgen["status"] == "ok"
&& metrics["missing_metrics"]
.as_array()
.is_some_and(Vec::is_empty)
&& metrics["accepted_spans"].as_u64().unwrap_or_default() > 0
&& metrics["http_requests"].as_u64().unwrap_or_default() > 0
&& flush["ok"].as_bool().unwrap_or(false)
&& visibility["ok"].as_bool().unwrap_or(false)
&& visibility["row_count_ok"].as_bool().unwrap_or(false);
Ok(json!({
"name": args.target_name,
"otelgen": otelgen,
"metrics": metrics,
"create_database": create_database,
"flush": flush,
"visibility": visibility,
"status": if checks_ok { "measured" } else { "failed" },
}))
}
fn otelgen_command(otelgen_bin: &Path, http_port: u16, load: &OtlpLoad) -> Vec<String> {
vec![
otelgen_bin.to_string_lossy().to_string(),
"--protocol".to_string(),
"http".to_string(),
"--otel-exporter-otlp-endpoint".to_string(),
format!("127.0.0.1:{http_port}"),
"--otel-exporter-otlp-url-path".to_string(),
"/v1/otlp/v1/traces".to_string(),
"--header".to_string(),
format!("x-greptime-pipeline-name={}", load.pipeline),
"--header".to_string(),
format!("x-greptime-db-name={}", load.database),
"--header".to_string(),
format!("x-greptime-trace-table-name={}", load.table),
"--log-level".to_string(),
"error".to_string(),
"--insecure".to_string(),
"--duration".to_string(),
load.duration_seconds.to_string(),
"--rate".to_string(),
load.rate.to_string(),
"traces".to_string(),
"multi".to_string(),
"--workers".to_string(),
load.workers.to_string(),
"--scenarios".to_string(),
load.workload.clone(),
"--exporter-shards".to_string(),
load.exporter_shards.to_string(),
]
}
async fn run_otelgen_load(
otelgen_bin: &Path,
http_port: u16,
work_dir: &Path,
load: &OtlpLoad,
client: &Client,
) -> Result<Value> {
let command = otelgen_command(otelgen_bin, http_port, load);
let clock = Instant::now();
let initial = fetch_otlp_metrics(client, http_port, &clock).await?;
let log_dir = work_dir.join("otelgen");
fs::create_dir_all(&log_dir)?;
let stdout_path = log_dir.join("stdout.log");
let stderr_path = log_dir.join("stderr.log");
let mut child = Command::new(otelgen_bin)
.args(&command[1..])
.stdout(Stdio::from(fs::File::create(&stdout_path)?))
.stderr(Stdio::from(fs::File::create(&stderr_path)?))
.spawn()?;
let started = Instant::now();
if load.warmup_seconds > 0 {
let _ = wait_for_child(&mut child, Duration::from_secs(load.warmup_seconds)).await?;
}
let warmed = fetch_otlp_metrics(client, http_port, &clock).await?;
let mut timed_out = false;
if child.try_wait()?.is_none() {
let remaining = load
.duration_seconds
.saturating_sub(load.warmup_seconds)
.saturating_add(60)
.max(60);
if !wait_for_child(&mut child, Duration::from_secs(remaining)).await? {
timed_out = true;
child.kill()?;
let _ = child.wait()?;
}
}
let final_snapshot = fetch_otlp_metrics(client, http_port, &clock).await?;
let returncode = match child.try_wait()? {
Some(status) => status.code(),
None => {
child.kill()?;
child.wait()?.code()
}
};
let elapsed_seconds = started.elapsed().as_secs_f64();
Ok(json!({
"status": if returncode == Some(0) && !timed_out && elapsed_seconds + 1.0 >= load.duration_seconds as f64 { "ok" } else { "failed" },
"cmd": command,
"returncode": returncode,
"timed_out": timed_out,
"elapsed_seconds": elapsed_seconds,
"stdout_path": stdout_path,
"stderr_path": stderr_path,
"snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot },
}))
}
async fn wait_for_child(child: &mut std::process::Child, timeout: Duration) -> Result<bool> {
let deadline = Instant::now() + timeout;
loop {
if child.try_wait()?.is_some() {
return Ok(true);
}
if Instant::now() >= deadline {
return Ok(false);
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
}
async fn fetch_otlp_metrics(
client: &Client,
http_port: u16,
clock: &Instant,
) -> Result<OtlpSnapshot> {
let text = client
.get(format!("http://127.0.0.1:{http_port}/metrics"))
.send()
.await?
.error_for_status()?
.text()
.await?;
Ok(OtlpSnapshot {
captured_monotonic_seconds: clock.elapsed().as_secs_f64(),
values: parse_prometheus_metrics(&text)?,
})
}
fn parse_prometheus_metrics(text: &str) -> Result<HashMap<String, f64>> {
let sample = Regex::new(r"^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{[^}]*\})?\s+(\S+)")?;
let mut values = HashMap::new();
for line in text.lines() {
let Some(captures) = sample.captures(line.trim()) else {
continue;
};
let (Some(name), Some(value)) = (captures.get(1), captures.get(2)) else {
continue;
};
let name = name.as_str();
if !matches!(
name,
OTLP_ROWS | OTLP_FAILURES | OTLP_ELAPSED_SUM | OTLP_ELAPSED_COUNT
) {
continue;
}
let value: f64 = value.as_str().parse()?;
if !value.is_finite() {
return Err(format!("non-finite Prometheus sample for {name}").into());
}
*values.entry(name.to_string()).or_default() += value;
}
Ok(values)
}
fn metric_delta(after: &OtlpSnapshot, before: &OtlpSnapshot, name: &str) -> Result<f64> {
let delta = after.values.get(name).copied().unwrap_or_default()
- before.values.get(name).copied().unwrap_or_default();
if delta < 0.0 {
return Err(format!("metric {name} decreased by {}", -delta).into());
}
Ok(delta)
}
fn summarize_otlp_metrics(run: &Value) -> Result<Value> {
let snapshots = run
.get("snapshots")
.ok_or("otelgen result has no snapshots")?;
let initial: OtlpSnapshot = serde_json::from_value(snapshots["initial"].clone())?;
let warmed: OtlpSnapshot = serde_json::from_value(snapshots["warmup"].clone())?;
let final_snapshot: OtlpSnapshot = serde_json::from_value(snapshots["final"].clone())?;
let missing_metrics = [OTLP_ROWS, OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT]
.into_iter()
.filter(|name| !final_snapshot.values.contains_key(*name))
.collect::<Vec<_>>();
let accepted_spans =
round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_ROWS)?) as u64;
let measurement_accepted_spans =
round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ROWS)?) as u64;
let http_requests =
round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_COUNT)?) as u64;
let latency_seconds = metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_SUM)?;
let measurement_seconds =
final_snapshot.captured_monotonic_seconds - warmed.captured_monotonic_seconds;
Ok(json!({
"accepted_spans": accepted_spans,
"measurement_accepted_spans": measurement_accepted_spans,
"accepted_spans_per_second": if measurement_seconds > 0.0 { Some(measurement_accepted_spans as f64 / measurement_seconds) } else { None },
"http_requests": http_requests,
"mean_http_latency_ms": if http_requests > 0 { Some(latency_seconds / http_requests as f64 * 1000.0) } else { None },
"failure_count": round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_FAILURES)?) as u64,
"measurement_seconds": measurement_seconds,
"missing_metrics": missing_metrics,
}))
}
async fn poll_otlp_visibility(
client: &Client,
port: u16,
table_name: &str,
database: &str,
expected_rows: u64,
timeout_seconds: u64,
) -> Result<Value> {
let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name));
let deadline = Instant::now() + Duration::from_secs(timeout_seconds);
let mut attempts = 0;
loop {
attempts += 1;
let mut result = http_post_sql(client, port, &sql, database).await;
let observed_rows = extract_count_value(&result);
let row_count_ok =
result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows);
result
.as_object_mut()
.ok_or("count result must be an object")?
.extend([
("expected_rows".to_string(), json!(expected_rows)),
("observed_rows".to_string(), json!(observed_rows)),
("attempts".to_string(), json!(attempts)),
("row_count_ok".to_string(), json!(row_count_ok)),
]);
if row_count_ok {
return Ok(result);
}
if Instant::now() >= deadline {
result["ok"] = json!(false);
result["row_count_ok"] = json!(false);
result["error"] = json!(format!(
"expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts"
));
return Ok(result);
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
pub(super) async fn run_finalize_otlp(args: FinalizeOtlpArgs) -> Result<()> {
let (case_path, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?;
let raw_case: toml::Value = toml::from_str(&fs::read_to_string(&case_path)?)?;
let case_metadata = serde_json::to_value(
raw_case
.get("case")
.cloned()
.unwrap_or(toml::Value::Table(toml::map::Map::new())),
)?;
let base: Value = serde_json::from_slice(&fs::read(&args.base_result)?)?;
let candidate: Value = serde_json::from_slice(&fs::read(&args.candidate_result)?)?;
let thresholds = enforce_otlp_thresholds(
&load.thresholds,
base.get("metrics").unwrap_or(&Value::Null),
candidate.get("metrics").unwrap_or(&Value::Null),
);
let failed = thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
|| [base.get("status"), candidate.get("status")]
.into_iter()
.any(|status| status.and_then(Value::as_str) == Some("failed"));
let report = json!({
"case_path": case_path,
"case": case_metadata,
"scenario": { "kind": "otlp_trace_load", "load": load },
"targets": [base, candidate],
"thresholds": thresholds,
"status": if failed { "failed" } else { "ok" },
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
fs::write(&args.output, &text)?;
print!("{text}");
if failed {
std::process::exit(1);
}
Ok(())
}
fn enforce_otlp_thresholds(
thresholds: &OtlpThresholds,
base: &Value,
candidate: &Value,
) -> Vec<Value> {
let mut results = Vec::new();
for (target, metrics) in [("base", base), ("candidate", candidate)] {
let failures = value_u64(metrics.get("failure_count"));
results.push(json!({
"target": target,
"threshold": "max_failure_count",
"status": if failures.is_some_and(|failures| failures <= thresholds.max_failure_count) { "passed" } else { "failed" },
"actual": failures,
"limit": thresholds.max_failure_count,
}));
}
for (name, base_value, candidate_value, limit, inverse) in [
(
"max_candidate_throughput_regression_pct",
value_f64(base.get("accepted_spans_per_second")),
value_f64(candidate.get("accepted_spans_per_second")),
thresholds.max_candidate_throughput_regression_pct,
true,
),
(
"max_candidate_mean_latency_regression_pct",
value_f64(base.get("mean_http_latency_ms")),
value_f64(candidate.get("mean_http_latency_ms")),
thresholds.max_candidate_mean_latency_regression_pct,
false,
),
] {
let (Some(base_value), Some(candidate_value)) =
(base_value.filter(|value| *value != 0.0), candidate_value)
else {
results.push(json!({
"threshold": name,
"status": "failed",
"reason": if inverse { "missing or zero throughput" } else { "missing or zero mean latency" },
"base": base_value,
"candidate": candidate_value,
}));
continue;
};
let actual = if inverse {
(base_value - candidate_value) / base_value * 100.0
} else {
(candidate_value - base_value) / base_value * 100.0
};
results.push(json!({
"threshold": name,
"status": if actual <= limit { "passed" } else { "failed" },
"actual_pct": actual,
"limit_pct": limit,
"base": base_value,
"candidate": candidate_value,
}));
}
results
}
#[cfg(test)]
mod tests {
use super::*;
fn otlp_load_for_test() -> OtlpLoad {
serde_json::from_value(json!({
"database": "public",
"table": "opentelemetry_traces",
"pipeline": "greptime_trace_v1",
"duration_seconds": 120,
"warmup_seconds": 60,
"rate": 50000,
"workers": 4,
"exporter_shards": 4,
"workload": "microservices",
"visibility_timeout_seconds": 30,
"thresholds": {
"max_candidate_throughput_regression_pct": 20.0,
"max_candidate_mean_latency_regression_pct": 20.0,
"max_failure_count": 0
}
}))
.unwrap()
}
#[test]
fn parses_labeled_otlp_metrics_and_rejects_non_finite_samples() {
let metrics = parse_prometheus_metrics(
"greptime_frontend_otlp_traces_rows 10\n\
greptime_frontend_otlp_traces_failure_count{kind=\"a\"} 1\n\
greptime_frontend_otlp_traces_failure_count{kind=\"b\"} 2\n",
)
.unwrap();
assert_eq!(metrics[OTLP_ROWS], 10.0);
assert_eq!(metrics[OTLP_FAILURES], 3.0);
assert!(parse_prometheus_metrics("greptime_frontend_otlp_traces_rows NaN").is_err());
let before = OtlpSnapshot {
captured_monotonic_seconds: 0.0,
values: HashMap::from([(OTLP_ROWS.to_string(), 2.0)]),
};
let after = OtlpSnapshot {
captured_monotonic_seconds: 1.0,
values: HashMap::from([(OTLP_ROWS.to_string(), 1.0)]),
};
assert!(metric_delta(&after, &before, OTLP_ROWS).is_err());
}
#[test]
fn summarizes_otlp_deltas_and_builds_exact_command() {
let snapshot = |captured: f64, values: Vec<(String, f64)>| OtlpSnapshot {
captured_monotonic_seconds: captured,
values: values.into_iter().collect(),
};
let initial = snapshot(0.0, vec![(OTLP_ROWS.to_string(), 10.0)]);
let warmed = snapshot(
5.0,
vec![
(OTLP_ROWS.to_string(), 110.0),
(OTLP_ELAPSED_SUM.to_string(), 1.0),
(OTLP_ELAPSED_COUNT.to_string(), 10.0),
],
);
let final_snapshot = snapshot(
15.0,
vec![
(OTLP_ROWS.to_string(), 310.0),
(OTLP_FAILURES.to_string(), 3.0),
(OTLP_ELAPSED_SUM.to_string(), 3.0),
(OTLP_ELAPSED_COUNT.to_string(), 30.0),
],
);
let metrics = summarize_otlp_metrics(&json!({
"snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot }
}))
.unwrap();
assert_eq!(metrics["accepted_spans"], 300);
assert_eq!(metrics["measurement_accepted_spans"], 200);
assert_eq!(metrics["accepted_spans_per_second"], 20.0);
assert_eq!(metrics["http_requests"], 20);
assert_eq!(metrics["mean_http_latency_ms"], 100.0);
assert_eq!(metrics["failure_count"], 3);
let missing = summarize_otlp_metrics(&json!({
"snapshots": {
"initial": snapshot(0.0, vec![(OTLP_ROWS.to_string(), 0.0)]),
"warmup": snapshot(1.0, vec![(OTLP_ROWS.to_string(), 1.0)]),
"final": snapshot(2.0, vec![(OTLP_ROWS.to_string(), 2.0)])
}
}))
.unwrap();
assert_eq!(
missing["missing_metrics"],
json!([OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT])
);
let command = otelgen_command(Path::new("/bin/otelgen"), 4000, &otlp_load_for_test());
assert_eq!(
command,
vec![
"/bin/otelgen",
"--protocol",
"http",
"--otel-exporter-otlp-endpoint",
"127.0.0.1:4000",
"--otel-exporter-otlp-url-path",
"/v1/otlp/v1/traces",
"--header",
"x-greptime-pipeline-name=greptime_trace_v1",
"--header",
"x-greptime-db-name=public",
"--header",
"x-greptime-trace-table-name=opentelemetry_traces",
"--log-level",
"error",
"--insecure",
"--duration",
"120",
"--rate",
"50000",
"traces",
"multi",
"--workers",
"4",
"--scenarios",
"microservices",
"--exporter-shards",
"4",
]
);
}
#[test]
fn otlp_thresholds_handle_zero_metrics_and_pass_fail_cases() {
let load = otlp_load_for_test();
let base = json!({ "failure_count": 0, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 100.0 });
let candidate = json!({ "failure_count": 0, "accepted_spans_per_second": 18.0, "mean_http_latency_ms": 110.0 });
let results = enforce_otlp_thresholds(&load.thresholds, &base, &candidate);
assert!(results.iter().all(|result| result["status"] == "passed"));
let failed = enforce_otlp_thresholds(
&load.thresholds,
&json!({ "failure_count": 0, "accepted_spans_per_second": 0.0, "mean_http_latency_ms": 100.0 }),
&json!({ "failure_count": 3, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 130.0 }),
);
assert!(
failed
.iter()
.any(|result| result["threshold"] == "max_failure_count"
&& result["target"] == "candidate"
&& result["status"] == "failed")
);
assert!(failed.iter().any(|result| result["threshold"]
== "max_candidate_throughput_regression_pct"
&& result["reason"] == "missing or zero throughput"));
assert!(failed.iter().any(|result| result["threshold"]
== "max_candidate_mean_latency_regression_pct"
&& result["status"] == "failed"));
}
}
src/cmd/src/bin/query_regression_runner/plan.rs
+122
View File
@@ -0,0 +1,122 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use std::process::Command;
use serde_json::Value;
use crate::query_regression_runner::Result;
use crate::query_regression_runner::model::{
Layout, OtlpLoad, Query, RemoteWrite, Scenario, Table,
};
pub(super) fn load_plan(generator: &PathBuf, case_path: &PathBuf) -> Result<Value> {
let output = Command::new(generator)
.args(["plan", "--case"])
.arg(case_path)
.output()?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
return Err(format!("query_perf_fixture plan failed: {:.2000}", stderr).into());
}
Ok(serde_json::from_slice(&output.stdout)?)
}
pub(super) fn normalize_scenario(scenario: Scenario) -> Result<(Vec<Table>, Vec<Query>)> {
match scenario {
Scenario::DirectReadableSst {
tables,
layout,
queries,
} => Ok((validate_direct_tables(tables, layout)?, queries)),
Scenario::PromRemoteWriteThenQuery {
remote_write,
queries,
} => Ok((
vec![Table {
database: remote_write.database,
name: remote_write.metric,
engine: "metric".to_string(),
columns: vec![],
primary_key: vec![],
time_index: None,
append_mode: None,
sst_format: None,
validate_show_create_engine: false,
}],
queries,
)),
Scenario::OtlpTraceLoad { .. } => {
Err("measure requires a query scenario, not otlp_trace_load".into())
}
}
}
pub(super) fn validate_direct_tables(tables: Vec<Table>, layout: Layout) -> Result<Vec<Table>> {
if tables.is_empty() || layout.regions != 1 {
return Err("runner supports one or more tables and exactly one region per table".into());
}
let mut pairs = HashSet::new();
let mut names = HashSet::new();
for table in &tables {
if !pairs.insert((&table.database, &table.name)) {
return Err("duplicate (database, name) table entries are not supported".into());
}
if !names.insert(&table.name) {
return Err("duplicate table names are not supported".into());
}
}
Ok(tables)
}
pub(super) fn normalized_remote_write(
generator: &PathBuf,
case: &Path,
) -> Result<(PathBuf, RemoteWrite)> {
let case_path = case.canonicalize()?;
let plan = load_plan(generator, &case_path)?;
let scenario = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
match serde_json::from_value(scenario)? {
Scenario::PromRemoteWriteThenQuery { remote_write, .. } => Ok((case_path, remote_write)),
Scenario::DirectReadableSst { .. } => {
Err("remote command requires scenario kind prom_remote_write_then_query".into())
}
Scenario::OtlpTraceLoad { .. } => {
Err("remote command requires scenario kind prom_remote_write_then_query".into())
}
}
}
pub(super) fn normalized_otlp_load(
generator: &PathBuf,
case: &Path,
) -> Result<(PathBuf, OtlpLoad)> {
let case_path = case.canonicalize()?;
let plan = load_plan(generator, &case_path)?;
let scenario = plan
.get("scenario")
.cloned()
.ok_or("fixture plan has no scenario")?;
match serde_json::from_value(scenario)? {
Scenario::OtlpTraceLoad { load } => Ok((case_path, load.load)),
Scenario::DirectReadableSst { .. } | Scenario::PromRemoteWriteThenQuery { .. } => {
Err("OTLP command requires scenario kind otlp_trace_load".into())
}
}
}
src/cmd/src/bin/query_regression_runner/remote/ingest.rs
+505
View File
@@ -0,0 +1,505 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::Path;
use std::process::Command;
use std::time::{Duration, Instant};
use reqwest::Client;
use serde_json::{Value, json};
use crate::query_regression_runner::model::{PromStore, RemoteWrite};
use crate::query_regression_runner::plan::normalized_remote_write;
use crate::query_regression_runner::sql::{
extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64,
};
use crate::query_regression_runner::{PrepareRemoteArgs, RenderRemoteConfigArgs, Result};
pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> {
let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
fs::write(args.output, frontend_prom_config(&remote.prom_store)?)?;
Ok(())
}
fn frontend_prom_config(prom: &PromStore) -> Result<String> {
Ok(format!(
"[prom_store]\nenable = true\nwith_metric_engine = true\npending_rows_flush_interval = {}\nmax_batch_rows = {}\nmax_concurrent_flushes = {}\nworker_channel_capacity = {}\nmax_inflight_requests = {}\n",
serde_json::to_string(&prom.pending_rows_flush_interval)?,
prom.max_batch_rows,
prom.max_concurrent_flushes,
prom.worker_channel_capacity,
prom.max_inflight_requests,
))
}
pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> {
if !args.http_timeout.is_finite() || args.http_timeout < 0.0 {
return Err("--http-timeout must be a non-negative finite number".into());
}
let (case_path, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
let client = Client::builder()
.timeout(Duration::from_secs_f64(args.http_timeout))
.build()?;
let base = prepare_remote_target(
"base",
args.base_http_port,
&args.fixture_generator,
&remote,
&client,
)
.await?;
let candidate = prepare_remote_target(
"candidate",
args.candidate_http_port,
&args.fixture_generator,
&remote,
&client,
)
.await?;
let report = json!({
"case_path": case_path,
"scenario": "prom_remote_write_then_query",
"base": base,
"candidate": candidate,
"status": "ok",
});
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
if let Some(path) = args.output {
fs::write(path, &text)?;
}
print!("{text}");
Ok(())
}
async fn prepare_remote_target(
name: &str,
port: u16,
generator: &Path,
remote: &RemoteWrite,
client: &Client,
) -> Result<Value> {
let create_database = http_post_sql(
client,
port,
&format!(
"CREATE DATABASE IF NOT EXISTS {}",
sql_ident(&remote.database)
),
"public",
)
.await;
if !create_database["ok"].as_bool().unwrap_or(false) {
return Err(format!(
"CREATE DATABASE {} failed for {name}: {create_database}",
remote.database
)
.into());
}
let (remote_write, flushes) = ingest_remote_write(generator, port, remote, client).await?;
let expected_rows = remote
.series_count
.checked_mul(remote.samples_per_series)
.ok_or("expected remote-write row count overflows u64")?;
let visibility = poll_expected_count(
client,
port,
&remote.metric,
&remote.database,
expected_rows,
remote.visibility_timeout_seconds,
)
.await?;
Ok(json!({
"name": name,
"create_database": create_database,
"remote_write": remote_write,
"flushes": flushes,
"visibility": visibility,
"status": "ok",
}))
}
#[derive(Clone, Debug, PartialEq, Eq)]
struct SampleChunk {
index: u64,
offset: u64,
samples_per_series: u64,
start_unix_millis: i64,
}
async fn ingest_remote_write(
generator: &Path,
port: u16,
remote: &RemoteWrite,
client: &Client,
) -> Result<(Value, Vec<Value>)> {
let Some(chunks) = sample_chunks(remote)? else {
let remote_write = run_remote_write(generator, port, remote, None)?;
let flush = flush_remote_table(client, port, remote, "final", None).await?;
return Ok((remote_write, vec![flush]));
};
let mut results = Vec::with_capacity(chunks.len());
let mut flushes = Vec::new();
let scheduled_flushes = scheduled_flushes(&chunks, remote.flush_every_sample_chunks);
for chunk in &chunks {
let mut result = run_remote_write(generator, port, remote, Some(chunk))?;
result
.as_object_mut()
.ok_or("remote-write result must be an object")?
.extend([
("sample_offset".to_string(), json!(chunk.offset)),
(
"samples_per_series".to_string(),
json!(chunk.samples_per_series),
),
("chunk_index".to_string(), json!(chunk.index)),
]);
results.push(result);
if let Some((_, reason)) = scheduled_flushes
.iter()
.find(|(index, _)| *index == chunk.index)
{
flushes
.push(flush_remote_table(client, port, remote, reason, Some(chunk.index)).await?);
}
}
Ok((
json!({
"status": "ok",
"mode": "sample-chunked",
"sample_chunk_size": remote.sample_chunk_size,
"flush_every_sample_chunks": remote.flush_every_sample_chunks,
"chunks": results,
"aggregate": summarize_remote_chunks(&results),
}),
flushes,
))
}
fn sample_chunks(remote: &RemoteWrite) -> Result<Option<Vec<SampleChunk>>> {
let Some(chunk_samples) = remote.sample_chunk_size else {
return Ok(None);
};
if chunk_samples == 0 {
return Err("scenario.remote_write.sample_chunk_size must be positive".into());
}
if remote.flush_every_sample_chunks == 0 {
return Err("scenario.remote_write.flush_every_sample_chunks must be positive".into());
}
let mut chunks = Vec::new();
let mut offset = 0;
while offset < remote.samples_per_series {
let samples_per_series = chunk_samples.min(remote.samples_per_series - offset);
chunks.push(SampleChunk {
index: chunks.len() as u64 + 1,
offset,
samples_per_series,
start_unix_millis: remote.start_unix_millis + offset as i64 * remote.step_millis,
});
offset += samples_per_series;
}
Ok(Some(chunks))
}
fn scheduled_flushes(chunks: &[SampleChunk], flush_every: u64) -> Vec<(u64, &'static str)> {
let mut scheduled = chunks
.iter()
.filter(|chunk| chunk.index % flush_every == 0)
.map(|chunk| (chunk.index, "periodic"))
.collect::<Vec<_>>();
if let Some(chunk) = chunks.last()
&& chunk.index % flush_every != 0
{
scheduled.push((chunk.index, "final"));
}
scheduled
}
fn summarize_remote_chunks(chunks: &[Value]) -> Value {
let mut rows = 0;
let mut samples_written = 0;
let mut batches = 0;
let mut elapsed_seconds = 0.0;
for chunk in chunks {
let summary = chunk.get("summary").unwrap_or(&Value::Null);
let chunk_rows = value_u64(summary.get("rows")).unwrap_or(0);
rows += chunk_rows;
samples_written += value_u64(summary.get("samples_written")).unwrap_or(chunk_rows);
batches += value_u64(summary.get("batches")).unwrap_or(0);
elapsed_seconds += value_f64(summary.get("elapsed_seconds"))
.or_else(|| value_f64(chunk.get("elapsed_seconds")))
.unwrap_or(0.0);
}
json!({ "rows": rows, "samples_written": samples_written, "batches": batches, "elapsed_seconds": elapsed_seconds })
}
fn run_remote_write(
generator: &Path,
port: u16,
remote: &RemoteWrite,
chunk: Option<&SampleChunk>,
) -> Result<Value> {
let command = remote_write_command(generator, port, remote, chunk);
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"remote-write generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let mut result = json!({
"status": "ok",
"command": command,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
});
match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) {
Ok(summary) => result["summary"] = summary,
Err(_) => result["summary_parse_error"] = result["stdout"].clone(),
}
Ok(result)
}
fn remote_write_command(
generator: &Path,
port: u16,
remote: &RemoteWrite,
chunk: Option<&SampleChunk>,
) -> Vec<String> {
let (samples_per_series, start_unix_millis, sample_offset, total_samples_per_series) = chunk
.map(|chunk| {
(
chunk.samples_per_series,
chunk.start_unix_millis,
Some(chunk.offset),
Some(remote.samples_per_series),
)
})
.unwrap_or((
remote.samples_per_series,
remote.start_unix_millis,
None,
None,
));
let mut command = vec![
generator.to_string_lossy().to_string(),
"prom-remote-write".to_string(),
"--endpoint".to_string(),
format!("http://127.0.0.1:{port}/v1/prometheus/write"),
"--database".to_string(),
remote.database.clone(),
"--metric".to_string(),
remote.metric.clone(),
"--physical-table".to_string(),
remote.physical_table.clone(),
"--series-count".to_string(),
remote.series_count.to_string(),
"--samples-per-series".to_string(),
samples_per_series.to_string(),
"--start-unix-millis".to_string(),
start_unix_millis.to_string(),
"--step-millis".to_string(),
remote.step_millis.to_string(),
"--chunk-series-count".to_string(),
remote.chunk_series_count.to_string(),
"--timeout-seconds".to_string(),
remote.timeout_seconds.to_string(),
"--value-pattern".to_string(),
remote.value.pattern.clone(),
"--value-base".to_string(),
json_number(remote.value.base),
"--value-step".to_string(),
json_number(remote.value.step),
"--value-cardinality".to_string(),
remote.value.cardinality.to_string(),
"--value-seed".to_string(),
remote.value.seed.to_string(),
"--value-run-length".to_string(),
remote.value.run_length.to_string(),
"--value-stall-every".to_string(),
remote.value.stall_every.to_string(),
"--value-stall-length".to_string(),
remote.value.stall_length.to_string(),
"--value-mixed-every".to_string(),
remote.value.mixed_every.to_string(),
];
if let Some(sample_offset) = sample_offset {
command.extend([
"--value-sample-offset".to_string(),
sample_offset.to_string(),
]);
}
if let Some(total_samples_per_series) = total_samples_per_series {
command.extend([
"--value-total-samples-per-series".to_string(),
total_samples_per_series.to_string(),
]);
}
command
}
async fn flush_remote_table(
client: &Client,
port: u16,
remote: &RemoteWrite,
reason: &str,
chunk_index: Option<u64>,
) -> Result<Value> {
let mut result = http_post_sql(
client,
port,
&format!("ADMIN FLUSH_TABLE({})", sql_string(&remote.physical_table)),
&remote.database,
)
.await;
if !result["ok"].as_bool().unwrap_or(false) {
return Err(format!(
"ADMIN FLUSH_TABLE {} failed: {result}",
remote.physical_table
)
.into());
}
result
.as_object_mut()
.ok_or("flush result must be an object")?
.extend([
("physical_table".to_string(), json!(remote.physical_table)),
("reason".to_string(), json!(reason)),
("chunk_index".to_string(), json!(chunk_index)),
]);
Ok(result)
}
async fn poll_expected_count(
client: &Client,
port: u16,
table_name: &str,
database: &str,
expected_rows: u64,
visibility_timeout_seconds: u64,
) -> Result<Value> {
let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name));
let deadline = Instant::now() + Duration::from_secs(visibility_timeout_seconds);
let mut attempts = 0;
loop {
attempts += 1;
let mut result = http_post_sql(client, port, &sql, database).await;
let observed_rows = extract_count_value(&result);
let row_count_ok =
result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows);
result
.as_object_mut()
.ok_or("count result must be an object")?
.extend([
("expected_rows".to_string(), json!(expected_rows)),
("observed_rows".to_string(), json!(observed_rows)),
("attempts".to_string(), json!(attempts)),
("row_count_ok".to_string(), json!(row_count_ok)),
]);
if row_count_ok {
return Ok(result);
}
if Instant::now() >= deadline {
return Err(format!(
"expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts"
)
.into());
}
tokio::time::sleep(Duration::from_millis(500)).await;
}
}
fn json_number(value: f64) -> String {
serde_json::to_string(&value).unwrap_or_else(|_| value.to_string())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::query_regression_runner::model::{PromStore, RemoteValue};
#[test]
fn schedules_remote_sample_chunks_and_flushes() {
let remote = RemoteWrite {
database: "public".to_string(),
metric: "metric".to_string(),
physical_table: "physical".to_string(),
series_count: 2,
samples_per_series: 5,
start_unix_millis: 100,
step_millis: 10,
chunk_series_count: 1,
timeout_seconds: 60,
sample_chunk_size: Some(2),
flush_every_sample_chunks: 2,
visibility_timeout_seconds: 30,
prom_store: PromStore {
pending_rows_flush_interval: "1s".to_string(),
max_batch_rows: 1,
max_concurrent_flushes: 1,
worker_channel_capacity: 1,
max_inflight_requests: 1,
},
value: RemoteValue {
pattern: "linear".to_string(),
base: 0.0,
step: 1.0,
cardinality: 1,
seed: 0,
run_length: 1,
stall_every: 0,
stall_length: 0,
mixed_every: 0,
},
storage: None,
read_bench: None,
};
let chunks = sample_chunks(&remote).unwrap().unwrap();
assert_eq!(
chunks,
vec![
SampleChunk {
index: 1,
offset: 0,
samples_per_series: 2,
start_unix_millis: 100,
},
SampleChunk {
index: 2,
offset: 2,
samples_per_series: 2,
start_unix_millis: 120,
},
SampleChunk {
index: 3,
offset: 4,
samples_per_series: 1,
start_unix_millis: 140,
},
]
);
assert_eq!(
scheduled_flushes(&chunks, remote.flush_every_sample_chunks),
vec![(2, "periodic"), (3, "final")]
);
}
}
src/cmd/src/bin/query_regression_runner/remote/mod.rs
+183
View File
@@ -0,0 +1,183 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use serde_json::{Value, json};
use self::read_bench::run_read_bench;
use self::storage::{enforce_storage_thresholds, run_storage_inspection};
use crate::query_regression_runner::model::DestinationConfig;
use crate::query_regression_runner::plan::normalized_remote_write;
use crate::query_regression_runner::{
FinalizeRemoteArgs, PrepareRemoteArgs, RenderRemoteConfigArgs, Result,
};
mod ingest;
mod read_bench;
mod storage;
pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> {
ingest::run_render_remote_config(args).await
}
pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> {
ingest::run_prepare_remote(args).await
}
pub(super) async fn run_finalize_remote(args: FinalizeRemoteArgs) -> Result<()> {
let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?;
let storage = remote.storage.as_ref().filter(|storage| storage.inspect);
let mut report: Value = serde_json::from_slice(&fs::read(&args.report)?)?;
let bench_root = args.report.parent().unwrap_or_else(|| Path::new("."));
let mut inspections = Vec::new();
let target_failed = {
let targets = report
.get_mut("targets")
.and_then(Value::as_array_mut)
.ok_or("report has no targets array")?;
for (name, data_home, destination) in [
(
"base",
args.base_data_home.as_deref(),
args.base_destination.as_deref(),
),
(
"candidate",
args.candidate_data_home.as_deref(),
args.candidate_destination.as_deref(),
),
] {
let (data_home, destination) = match (data_home, destination) {
(Some(data_home), None) => (data_home.to_path_buf(), None),
(None, Some(path)) => {
let destination: DestinationConfig =
toml::from_str(&fs::read_to_string(path)?)?;
(
PathBuf::from(destination.data_home),
Some(path.to_path_buf()),
)
}
(Some(_), Some(_)) => {
return Err(format!(
"{name}: --{name}-data-home and --{name}-destination are mutually exclusive"
)
.into());
}
(None, None) => {
return Err(format!(
"{name}: one of --{name}-data-home or --{name}-destination is required"
)
.into());
}
};
let target = targets
.iter_mut()
.find(|target| target.get("name").and_then(Value::as_str) == Some(name))
.ok_or_else(|| format!("report has no {name} target"))?;
let Some(storage) = storage else {
target
.as_object_mut()
.ok_or("report target must be an object")?
.insert(
"read_bench".to_string(),
json!({"status": "skipped", "reason": "storage inspection disabled"}),
);
continue;
};
let inspection = run_storage_inspection(
&args.fixture_generator,
&data_home,
destination.as_deref(),
storage,
)?;
let bench_dir = bench_root.join(name).join("read_bench");
let read_bench = run_read_bench(
&args.candidate_bin,
&data_home,
&bench_dir,
remote.read_bench.as_ref(),
&inspection,
)?;
let inspection_failed = inspection["status"] == "failed";
let bench_failed = read_bench["status"] == "failed";
let target = target
.as_object_mut()
.ok_or("report target must be an object")?;
target.insert("storage_inspection".to_string(), inspection.clone());
target.insert("read_bench".to_string(), read_bench);
if inspection_failed || bench_failed {
target.insert("status".to_string(), json!("failed"));
}
inspections.push(inspection);
}
targets.iter().any(|target| {
target["status"] == "failed"
|| target["storage_inspection"]["status"] == "failed"
|| target["read_bench"]["status"] == "failed"
})
};
let storage_thresholds = storage
.map(|storage| {
enforce_storage_thresholds(
storage,
inspections.first().unwrap_or(&Value::Null),
inspections.get(1).unwrap_or(&Value::Null),
)
})
.unwrap_or_default();
let threshold_failed = {
let thresholds = report
.get_mut("thresholds")
.and_then(Value::as_array_mut)
.ok_or("report has no thresholds array")?;
thresholds.retain(|threshold| !is_storage_threshold_entry(threshold));
thresholds.extend(storage_thresholds);
thresholds
.iter()
.any(|threshold| threshold["status"] == "failed")
};
let failed = target_failed || threshold_failed;
report["status"] = json!(if failed { "failed" } else { "ok" });
let text = format!("{}\n", serde_json::to_string_pretty(&report)?);
fs::write(&args.report, &text)?;
print!("{text}");
if failed {
std::process::exit(1);
}
Ok(())
}
fn is_storage_threshold_entry(threshold: &Value) -> bool {
let Some(name) = threshold.get("threshold").and_then(Value::as_str) else {
return false;
};
matches!(
name,
"max_candidate_total_file_size_regression_pct"
| "max_candidate_column_compressed_size_regression_pct"
| "max_candidate_column_uncompressed_size_regression_pct"
) || (threshold.get("target").is_some()
&& matches!(
name,
"min_files"
| "min_files_with_column"
| "max_total_file_size_bytes"
| "max_column_compressed_size_bytes"
| "max_column_uncompressed_size_bytes"
| "require_encodings"
| "forbid_encodings"
))
}
src/cmd/src/bin/query_regression_runner/remote/read_bench.rs
+363
View File
@@ -0,0 +1,363 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::fs;
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::Instant;
use regex::Regex;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
use crate::query_regression_runner::measure::median;
use crate::query_regression_runner::model::ReadBenchConfig;
#[derive(Clone, Debug)]
struct BenchTarget {
relative_path: String,
table_dir: String,
region_id: String,
path_type: String,
file_id: String,
}
pub(super) fn run_read_bench(
candidate_bin: &Path,
data_home: &Path,
bench_dir: &Path,
read_bench: Option<&ReadBenchConfig>,
inspection: &Value,
) -> Result<Value> {
let Some(read_bench) = read_bench.filter(|config| config.enabled) else {
return Ok(json!({ "status": "skipped", "reason": "read_bench disabled" }));
};
if !read_bench.parquetbench && !read_bench.scanbench {
return Ok(json!({ "status": "skipped", "reason": "parquetbench and scanbench disabled" }));
}
let report = inspection
.get("summary")
.filter(|summary| summary.is_object());
let files = report
.and_then(|report| report.get("files"))
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
let root = report
.and_then(|report| report.get("root"))
.and_then(Value::as_str)
.map(PathBuf::from)
.or_else(|| {
inspection
.get("root")
.and_then(Value::as_str)
.map(PathBuf::from)
})
.ok_or("storage inspection has no root")?;
let mut targets = files
.iter()
.filter(|file| {
file.get("relative_path")
.and_then(Value::as_str)
.is_some_and(|path| path.ends_with(".parquet"))
&& file
.get("columns")
.and_then(Value::as_array)
.is_some_and(|columns| !columns.is_empty())
})
.filter_map(|file| inspected_bench_target(data_home, &root, file).transpose())
.collect::<Result<Vec<_>>>()?;
if let Some(max_files) = read_bench.max_files {
targets.truncate(max_files);
}
if targets.is_empty() {
return Ok(
json!({ "status": "failed", "reason": "no inspected data SST files available for read_bench" }),
);
}
fs::create_dir_all(bench_dir)?;
let config_toml = bench_dir.join("bench.toml");
let scan_json = bench_dir.join("scan.json");
fs::write(
&config_toml,
format!(
"[storage]\ndata_home = \"{}\"\ntype = \"File\"\n\n[[region_engine]]\n[region_engine.mito]\n",
data_home.display()
),
)?;
fs::write(
&scan_json,
format!(
"{}\n",
serde_json::to_string_pretty(&json!({ "projection_names": read_bench.projection }))?
),
)?;
let mut parquet_runs = Vec::new();
if read_bench.parquetbench {
for target in &targets {
let command = vec![
candidate_bin.to_string_lossy().to_string(),
"datanode".to_string(),
"parquetbench".to_string(),
"--config".to_string(),
config_toml.to_string_lossy().to_string(),
"--region-id".to_string(),
target.region_id.clone(),
"--table-dir".to_string(),
target.table_dir.clone(),
"--file-id".to_string(),
target.file_id.clone(),
"--scan-config".to_string(),
scan_json.to_string_lossy().to_string(),
"--path-type".to_string(),
target.path_type.clone(),
"--iterations".to_string(),
read_bench.iterations.to_string(),
"--reader".to_string(),
read_bench.parquet_reader.clone(),
];
parquet_runs.push(run_bench_command(command, bench_target_value(target))?);
}
}
let mut scan_runs = Vec::new();
if read_bench.scanbench {
for (table_dir, region_id, path_type, files) in group_scan_paths(&targets) {
let command = vec![
candidate_bin.to_string_lossy().to_string(),
"datanode".to_string(),
"scanbench".to_string(),
"--config".to_string(),
config_toml.to_string_lossy().to_string(),
"--region-id".to_string(),
region_id.clone(),
"--table-dir".to_string(),
table_dir.clone(),
"--scan-config".to_string(),
scan_json.to_string_lossy().to_string(),
"--path-type".to_string(),
path_type.clone(),
"--scanner".to_string(),
read_bench.scan_scanner.clone(),
"--parallelism".to_string(),
read_bench.parallelism.to_string(),
"--iterations".to_string(),
read_bench.iterations.to_string(),
];
scan_runs.push(run_bench_command(command, json!({ "table_dir": table_dir, "region_id": region_id, "path_type": path_type, "files": files }))?);
}
}
let failed = parquet_runs
.iter()
.chain(&scan_runs)
.any(|run| run["status"] == "failed");
Ok(json!({
"status": if failed { "failed" } else { "ok" },
"config_path": config_toml,
"scan_config_path": scan_json,
"parquetbench": parquet_runs,
"scanbench": scan_runs,
"aggregate": {
"parquetbench_median_average_ms": bench_median(&parquet_runs),
"scanbench_median_average_ms": bench_median(&scan_runs),
},
}))
}
fn inspected_bench_target(
data_home: &Path,
root: &Path,
file: &Value,
) -> Result<Option<BenchTarget>> {
let relative_path = file
.get("relative_path")
.and_then(Value::as_str)
.unwrap_or_default();
if relative_path.is_empty() {
return Ok(None);
}
let root_suffix = root.strip_prefix(data_home).map_err(|_| {
format!(
"storage inspection root {} is not under datanode data home {}",
root.display(),
data_home.display()
)
})?;
let full_relative = root_suffix.join(relative_path);
let parts = full_relative
.components()
.map(|component| component.as_os_str().to_string_lossy().to_string())
.collect::<Vec<_>>();
if !parts.last().is_some_and(|part| part.ends_with(".parquet")) {
return Ok(None);
}
let Some((region_index, table_id, region_seq)) =
parts.iter().enumerate().find_map(|(index, part)| {
parse_region_dir(part).map(|(table_id, region_seq)| (index, table_id, region_seq))
})
else {
return Ok(None);
};
if region_index == 0 {
return Ok(None);
}
let file_index = parts.len() - 1;
let path_type = if region_index + 1 < file_index
&& matches!(parts[region_index + 1].as_str(), "data" | "metadata")
{
parts[region_index + 1].clone()
} else {
"bare".to_string()
};
if path_type == "metadata" {
return Ok(None);
}
let file_id = parts[file_index].trim_end_matches(".parquet").to_string();
Ok(Some(BenchTarget {
relative_path: full_relative.to_string_lossy().to_string(),
table_dir: format!("{}/", parts[..region_index].join("/")),
region_id: format!("{table_id}:{region_seq}"),
path_type,
file_id,
}))
}
fn parse_region_dir(part: &str) -> Option<(u64, u64)> {
let (table_id, region_seq) = part.split_once('_')?;
(region_seq.len() == 10
&& table_id.chars().all(|character| character.is_ascii_digit())
&& region_seq
.chars()
.all(|character| character.is_ascii_digit()))
.then(|| Some((table_id.parse().ok()?, region_seq.parse().ok()?)))?
}
fn bench_target_value(target: &BenchTarget) -> Value {
json!({
"relative_path": target.relative_path,
"table_dir": target.table_dir,
"region_id": target.region_id,
"path_type": target.path_type,
"file_id": target.file_id,
})
}
fn group_scan_paths(targets: &[BenchTarget]) -> Vec<(String, String, String, Vec<String>)> {
let mut groups: Vec<(String, String, String, Vec<String>)> = Vec::new();
for target in targets {
if let Some((_, _, _, paths)) =
groups
.iter_mut()
.find(|(table_dir, region_id, path_type, _)| {
table_dir == &target.table_dir
&& region_id == &target.region_id
&& path_type == &target.path_type
})
{
paths.push(target.relative_path.clone());
} else {
groups.push((
target.table_dir.clone(),
target.region_id.clone(),
target.path_type.clone(),
vec![target.relative_path.clone()],
));
}
}
groups
}
fn run_bench_command(command: Vec<String>, mut run: Value) -> Result<Value> {
let started = Instant::now();
let output = Command::new(&command[0]).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
let run = run.as_object_mut().ok_or("bench run must be an object")?;
run.extend([
("command".to_string(), json!(command)),
("returncode".to_string(), json!(output.status.code())),
("elapsed_seconds".to_string(), json!(elapsed_seconds)),
("stdout".to_string(), json!(stdout)),
("stderr".to_string(), json!(stderr)),
(
"status".to_string(),
json!(if output.status.success() {
"ok"
} else {
"failed"
}),
),
]);
if let Some(average_ms) = parse_average_duration(run["stdout"].as_str().unwrap_or_default()) {
run.insert("average_ms".to_string(), json!(average_ms));
}
Ok(Value::Object(run.clone()))
}
fn parse_average_duration(stdout: &str) -> Option<f64> {
Regex::new(r"(?i)Average duration[^0-9]*([0-9.]+)\s*ms")
.ok()?
.captures(stdout)?
.get(1)?
.as_str()
.parse()
.ok()
}
fn bench_median(runs: &[Value]) -> Option<f64> {
let values = runs
.iter()
.filter_map(|run| run.get("average_ms").and_then(Value::as_f64))
.collect::<Vec<_>>();
(!values.is_empty()).then(|| median(&values))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parses_average_duration_and_groups_scan_paths() {
assert_eq!(
parse_average_duration("work\nAverage duration: 12.5 ms\n"),
Some(12.5)
);
assert_eq!(parse_average_duration("completed"), None);
let targets = vec![
BenchTarget {
relative_path: "data/a.parquet".to_string(),
table_dir: "data/".to_string(),
region_id: "1:2".to_string(),
path_type: "data".to_string(),
file_id: "a".to_string(),
},
BenchTarget {
relative_path: "data/b.parquet".to_string(),
table_dir: "data/".to_string(),
region_id: "1:2".to_string(),
path_type: "data".to_string(),
file_id: "b".to_string(),
},
];
assert_eq!(
group_scan_paths(&targets),
vec![(
"data/".to_string(),
"1:2".to_string(),
"data".to_string(),
vec!["data/a.parquet".to_string(), "data/b.parquet".to_string()],
)]
);
}
}
src/cmd/src/bin/query_regression_runner/remote/storage.rs
+241
View File
@@ -0,0 +1,241 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::path::Path;
use std::process::Command;
use std::time::Instant;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
use crate::query_regression_runner::model::StorageConfig;
use crate::query_regression_runner::sql::{value_f64, value_u64};
pub(super) fn run_storage_inspection(
generator: &Path,
data_home: &Path,
destination: Option<&Path>,
storage: &StorageConfig,
) -> Result<Value> {
let mut command = vec![
generator.to_string_lossy().to_string(),
"inspect-footer".to_string(),
"--column".to_string(),
storage.column.clone(),
];
let root = if let Some(destination) = destination {
command.push("--destination".to_string());
command.push(destination.to_string_lossy().to_string());
data_home.to_path_buf()
} else {
let root = storage
.root_suffix
.as_deref()
.map_or_else(|| data_home.to_path_buf(), |suffix| data_home.join(suffix));
command.push("--root".to_string());
command.push(root.to_string_lossy().to_string());
root
};
if storage.include_metadata_files {
command.push("--include-metadata-files".to_string());
}
let started = Instant::now();
let output = Command::new(generator).args(&command[1..]).output()?;
let elapsed_seconds = started.elapsed().as_secs_f64();
let stdout = String::from_utf8_lossy(&output.stdout).to_string();
let stderr = String::from_utf8_lossy(&output.stderr).to_string();
if !output.status.success() {
return Err(format!(
"storage inspector failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}",
output.status.code(), stderr
)
.into());
}
let mut result = json!({
"status": "ok",
"command": command,
"root": root,
"returncode": output.status.code(),
"elapsed_seconds": elapsed_seconds,
"stdout": stdout,
"stderr": stderr,
});
match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) {
Ok(summary) => result["summary"] = summary,
Err(_) => {
result["summary_parse_error"] = result["stdout"].clone();
result["status"] = json!("failed");
}
}
Ok(result)
}
fn storage_summary(inspection: &Value) -> Value {
inspection
.get("summary")
.and_then(|summary| summary.get("summary"))
.filter(|summary| summary.is_object())
.cloned()
.unwrap_or_else(|| json!({}))
}
pub(super) fn enforce_storage_thresholds(
storage: &StorageConfig,
base_inspection: &Value,
candidate_inspection: &Value,
) -> Vec<Value> {
let base = storage_summary(base_inspection);
let candidate = storage_summary(candidate_inspection);
let targets = [("base", &base), ("candidate", &candidate)];
let mut results = Vec::new();
for (threshold, field, limit) in [
("min_files", "file_count", storage.min_files),
(
"min_files_with_column",
"files_with_column",
storage.min_files_with_column,
),
] {
for (target, summary) in &targets {
let actual = summary.get(field).cloned().unwrap_or(Value::Null);
let ok = value_u64(Some(&actual)).is_some_and(|actual| actual >= limit);
results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit }));
}
}
for (threshold, field, limit) in [
(
"max_total_file_size_bytes",
"total_file_size",
storage.max_total_file_size_bytes,
),
(
"max_column_compressed_size_bytes",
"column_compressed_size",
storage.max_column_compressed_size_bytes,
),
(
"max_column_uncompressed_size_bytes",
"column_uncompressed_size",
storage.max_column_uncompressed_size_bytes,
),
] {
let Some(limit) = limit else { continue };
for (target, summary) in &targets {
let actual = summary.get(field).cloned().unwrap_or(Value::Null);
let ok = value_f64(Some(&actual)).is_some_and(|actual| actual <= limit as f64);
results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit }));
}
}
for (target, summary) in &targets {
let encodings = summary
.get("unique_encodings")
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
for encoding in &storage.require_encodings {
results.push(json!({ "target": target, "threshold": "require_encodings", "encoding": encoding, "status": if encodings.iter().any(|value| value.as_str() == Some(encoding)) { "passed" } else { "failed" } }));
}
for encoding in &storage.forbid_encodings {
results.push(json!({ "target": target, "threshold": "forbid_encodings", "encoding": encoding, "status": if encodings.iter().all(|value| value.as_str() != Some(encoding)) { "passed" } else { "failed" } }));
}
}
for (threshold, field, limit) in [
(
"max_candidate_total_file_size_regression_pct",
"total_file_size",
storage.max_candidate_total_file_size_regression_pct,
),
(
"max_candidate_column_compressed_size_regression_pct",
"column_compressed_size",
storage.max_candidate_column_compressed_size_regression_pct,
),
(
"max_candidate_column_uncompressed_size_regression_pct",
"column_uncompressed_size",
storage.max_candidate_column_uncompressed_size_regression_pct,
),
] {
let Some(limit) = limit else { continue };
let base_value = base.get(field).cloned().unwrap_or(Value::Null);
let candidate_value = candidate.get(field).cloned().unwrap_or(Value::Null);
let (Some(base_number), Some(candidate_number)) = (
value_f64(Some(&base_value)).filter(|value| *value != 0.0),
value_f64(Some(&candidate_value)),
) else {
results.push(json!({ "threshold": threshold, "status": "failed", "reason": "missing or zero base/candidate value", "base": base_value, "candidate": candidate_value }));
continue;
};
let actual = (candidate_number - base_number) / base_number * 100.0;
results.push(json!({ "threshold": threshold, "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit, "base": base_value, "candidate": candidate_value }));
}
results
}
#[cfg(test)]
mod tests {
use super::super::is_storage_threshold_entry;
use super::*;
#[test]
fn enforces_storage_thresholds_for_each_target_and_regression() {
let storage: StorageConfig = serde_json::from_value(json!({
"inspect": true,
"column": "value",
"root_suffix": null,
"include_metadata_files": false,
"min_files": 2,
"min_files_with_column": 1,
"require_encodings": ["PLAIN"],
"forbid_encodings": ["DELTA"],
"max_total_file_size_bytes": 105,
"max_column_compressed_size_bytes": null,
"max_column_uncompressed_size_bytes": null,
"max_candidate_total_file_size_regression_pct": 5.0,
"max_candidate_column_compressed_size_regression_pct": null,
"max_candidate_column_uncompressed_size_regression_pct": null
}))
.unwrap();
let base = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 100, "unique_encodings": ["PLAIN"]}}});
let candidate = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 110, "unique_encodings": ["PLAIN"]}}});
let results = enforce_storage_thresholds(&storage, &base, &candidate);
assert!(
results
.iter()
.any(|result| result["threshold"] == "max_total_file_size_bytes"
&& result["target"] == "candidate"
&& result["status"] == "failed")
);
assert!(results.iter().any(|result| result["threshold"]
== "max_candidate_total_file_size_regression_pct"
&& result["actual_pct"] == 10.0
&& result["status"] == "failed"));
assert!(results.iter().any(
|result| result["threshold"] == "require_encodings" && result["status"] == "passed"
));
}
#[test]
fn identifies_prior_storage_threshold_entries() {
assert!(is_storage_threshold_entry(
&json!({"target": "base", "threshold": "min_files"})
));
assert!(is_storage_threshold_entry(&json!({
"threshold": "max_candidate_total_file_size_regression_pct"
})));
assert!(!is_storage_threshold_entry(&json!({
"threshold": "max_candidate_latency_regression_pct"
})));
}
}
src/cmd/src/bin/query_regression_runner/sql.rs
+238
View File
@@ -0,0 +1,238 @@
// Copyright 2023 Greptime Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use std::time::Instant;
use reqwest::Client;
use serde_json::{Value, json};
use crate::query_regression_runner::Result;
pub(super) fn sql_string(value: &str) -> String {
format!("'{}'", value.replace('\'', "''"))
}
pub(super) fn extract_rows(body: &Value) -> Vec<Value> {
fn visit(body: &Value, rows: &mut Vec<Value>) {
match body {
Value::Object(object) => {
for key in ["data", "rows", "records", "output"] {
let Some(value) = object.get(key) else {
continue;
};
if matches!(key, "data" | "rows") && value.is_array() {
rows.extend(value.as_array().unwrap().iter().cloned());
} else {
visit(value, rows);
}
}
}
Value::Array(values) => {
if !values.is_empty()
&& values
.iter()
.all(|value| !value.is_object() && !value.is_array())
{
rows.push(body.clone());
} else {
for value in values {
visit(value, rows);
}
}
}
_ => {}
}
}
let mut rows = Vec::new();
visit(body, &mut rows);
rows
}
pub(super) fn row_value<'a>(row: &'a Value, index: usize, name: &str) -> Option<&'a Value> {
match row {
Value::Object(values) => [
name.to_string(),
name.to_ascii_uppercase(),
name.to_ascii_lowercase(),
]
.into_iter()
.find_map(|key| values.get(&key)),
Value::Array(values) => values.get(index),
_ => Some(row),
}
}
pub(super) fn row_u64(row: &Value, index: usize, name: &str) -> Result<u64> {
let value =
row_value(row, index, name).ok_or_else(|| format!("missing {name} in row {row}"))?;
value
.as_u64()
.or_else(|| value.as_str().and_then(|value| value.parse().ok()))
.ok_or_else(|| format!("invalid {name} in row {row}").into())
}
pub(super) fn value_text(value: &Value) -> String {
match value {
Value::String(value) => value.clone(),
Value::Bool(value) => value.to_string(),
Value::Number(value) => value.to_string(),
Value::Null => "None".to_string(),
_ => value.to_string(),
}
}
pub(super) fn extract_count_value(result: &Value) -> Option<u64> {
let row = result
.get("response")?
.get("data")?
.as_array()?
.first()?
.as_object()?;
row.iter()
.find(|(key, _)| {
let key = key.to_ascii_lowercase();
key == "count(*)" || key.starts_with("count(")
})
.and_then(|(_, value)| value_u64(Some(value)))
}
pub(super) fn value_u64(value: Option<&Value>) -> Option<u64> {
value?.as_u64().or_else(|| value?.as_str()?.parse().ok())
}
pub(super) fn value_f64(value: Option<&Value>) -> Option<f64> {
value?.as_f64().or_else(|| value?.as_str()?.parse().ok())
}
pub(super) async fn http_post_sql(client: &Client, port: u16, sql: &str, db: &str) -> Value {
let started = Instant::now();
let request = client
.post(format!("http://127.0.0.1:{port}/v1/sql"))
.form(&[("sql", sql), ("db", db), ("format", "json")]);
match request.send().await {
Ok(response) => {
let status = response.status().as_u16();
match response.text().await {
Ok(raw) => {
let body = serde_json::from_str(&raw).unwrap_or_else(|_| json!({"raw": raw}));
let ok = status < 400 && !response_has_error(&body);
let mut sample = json!({
"ok": ok,
"status": status,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"response": body,
"sql": sql,
});
if status >= 400 {
sample
.as_object_mut()
.expect("HTTP samples are objects")
.insert("error".to_string(), Value::String(format!("HTTP {status}")));
}
sample
}
Err(error) => json!({
"ok": false,
"status": status,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"error": error.to_string(),
"sql": sql,
}),
}
}
Err(error) => json!({
"ok": false,
"status": Value::Null,
"latency_ms": started.elapsed().as_secs_f64() * 1000.0,
"error": error.to_string(),
"sql": sql,
}),
}
}
fn response_has_error(body: &Value) -> bool {
let Some(body) = body.as_object() else {
return false;
};
["error", "err_msg", "error_msg"]
.into_iter()
.any(|key| body.get(key).is_some_and(is_truthy))
|| body
.get("error_code")
.is_some_and(|value| !is_success_code(value))
|| (!body.contains_key("output")
&& body
.get("code")
.is_some_and(|value| !is_success_code(value)))
}
fn is_truthy(value: &Value) -> bool {
match value {
Value::Null => false,
Value::Bool(value) => *value,
Value::Number(value) => value.as_f64().is_none_or(|value| value != 0.0),
Value::String(value) => !value.is_empty(),
Value::Array(value) => !value.is_empty(),
Value::Object(value) => !value.is_empty(),
}
}
fn is_success_code(value: &Value) -> bool {
let code = match value {
Value::String(value) => value.clone(),
Value::Bool(value) => value.to_string(),
Value::Number(value) => value.to_string(),
Value::Null => "None".to_string(),
_ => value.to_string(),
};
matches!(code.to_lowercase().as_str(), "" | "0" | "success")
}
pub(super) fn sql_ident(name: &str) -> String {
format!("\"{}\"", name.replace('"', "\"\""))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn top_level_errors_do_not_inspect_rows() {
assert!(response_has_error(&json!({"error_code": 7})));
assert!(response_has_error(&json!({"code": "bad"})));
assert!(!response_has_error(
&json!({"output": [{"code": 7, "error": "row value"}]})
));
assert!(!response_has_error(
&json!({"error_code": "success", "output": []})
));
}
#[test]
fn extracts_remote_write_count_from_data_map() {
assert_eq!(
extract_count_value(&json!({"response": {"data": [{"COUNT(*)": "12"}]}})),
Some(12)
);
assert_eq!(
extract_count_value(&json!({"response": {"data": [{"count(value)": 7}]}})),
Some(7)
);
assert_eq!(
extract_count_value(&json!({"response": {"data": [[]]}})),
None
);
}
}
tests/perf/AGENTS.md
+23 -6
View File
@@ -3,22 +3,39 @@
- Keep GitHub Actions YAML thin. Put non-trivial control flow, case expansion,
report generation, and metadata writing in scripts under `.github/scripts/`;
workflow steps should mostly invoke those scripts.
- Runner lifecycle: the default path provisions one ephemeral Aliyun ECS
instance per run via `.github/scripts/aliyun-ecs-runner-provision.py` and
always releases it via `aliyun-ecs-runner-teardown.py`; a scheduled janitor
workflow sweeps leftovers. Build caches live on that instance's 50 GiB system
disk and are discarded with the VM. Runs do not share a workflow concurrency
group. The ECS custom image is built from the runner Dockerfile by
`.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py`;
keep the Dockerfile the single source of the tool contract. Dispatching with
any other `runner` value treats it as a literal self-hosted runner label
(see `ecs-image/bootstrap-runner-host.sh` for preparing such a host).
- Scheduled nightly comparison lives in `query-regression-nightly.yml`: it
waits for a successful Nightly Build, then calls `query-regression.yml`
with the previous vs current nightly SHAs. Keep SHA selection in
`.github/scripts/query-regression-nightly-refs.py`.
- Query regression PR runs should build base/candidate binaries once, then run
the default case set. Do not hard-code a single case such as
`promql_pushdown_7913` into the workflow path.
- The case DSL is not required to keep compatibility inside this PR. When the
DSL changes, update TOML cases, the Python runner, Rust fixture generator, and
DSL changes, update TOML cases, the outer lifecycle script, Rust helpers, and
docs together.
- `[case]` is report metadata only. `[scenario]` is the executable regression
configuration and must include `kind`, data layout, tables, queries, and
thresholds. Rust owns case schema, defaults, validation, and normalized plan
output through `query_perf_fixture plan`; `tests/perf/query_regression_runner.py`
should only orchestrate from that normalized JSON.
output through `query_perf_fixture plan`. `.github/scripts/query-regression-run.py`
owns process lifecycle; `query_regression_runner` consumes normalized plans,
frontend endpoints, direct-SST materialization requests, and OTLP target/finalize
requests.
- Keep the direct-SST generator generic. Issue-specific behavior belongs in case
files and thresholds, not in Rust generator logic.
- Before pushing perf harness changes, run at least:
- `uv run --no-project python -m py_compile .github/scripts/query-regression-run.py .github/scripts/query-regression-summary.py .github/scripts/query-regression-pr-metadata.py tests/perf/query_regression_runner.py`
- `uv run --no-project python -m py_compile .github/scripts/query-regression-run.py .github/scripts/query-regression-summary.py .github/scripts/query-regression-pr-metadata.py tests/perf/test_query_regression_runner_compaction_toctou.py tests/perf/test_query_regression_runner_otlp_trace_load.py`
- `uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py && uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py`
- `cargo fmt --all -- --check`
- `cargo build -p cmd --bin query_perf_fixture --features dev-tools`
- dry-run the Python runner and Rust fixture generator against all built-in
cases when the DSL or workflow case selection changes.
- exercise the outer lifecycle script and Rust fixture generator against all
built-in cases when the DSL or workflow case selection changes.
tests/perf/README.md
+64 -142
View File
@@ -41,7 +41,7 @@ release-level realism.
## Prometheus remote-write scenario
The runner also supports `scenario.kind = "prom_remote_write_then_query"` for a
The outer driver supports `scenario.kind = "prom_remote_write_then_query"` for a
bounded write-path smoke/regression flow. This path is explicit and separate from
the direct-SST fixture path: it starts the base and candidate distributed clusters,
writes deterministic Prometheus remote-write v1 samples through
@@ -111,11 +111,10 @@ helper so non-linear value patterns use a stable global/per-series ordinal acros
chunks.
Case schema, value distribution defaults, storage defaults, and read-bench
defaults are owned by Rust. The Python runner calls
`query_perf_fixture plan --case <case.toml>` and orchestrates the normalized JSON.
The same helper exposes `direct-sst`, `prom-remote-write`, and `inspect-footer`
subcommands; the old direct invocation (`query_perf_fixture --case ... --out-dir
...`) remains compatible.
defaults are owned by Rust. The outer CI driver calls
`query_perf_fixture plan --case <case.toml>` once, then uses the normalized plan
to select the Rust runner lifecycle. The fixture helper exposes `direct-sst`,
`prom-remote-write`, and `inspect-footer` subcommands.
Remote-write cases that need to validate storage output can add
`[scenario.remote_write.storage]`. When present, the scenario body becomes:
@@ -163,17 +162,15 @@ measures region scan cost, and query measurements still exercise the SQL/TQL
frontend path. Treat all performance conclusions as release-only; debug builds
are suitable only for command wiring and correctness checks.
The runner creates the configured database if needed, writes a per-target
`prepare-remote` creates the configured database if needed. The outer driver writes a per-target
frontend config enabling `[prom_store]` with metric engine storage and a non-zero
`pending_rows_flush_interval`, and validates that the logical metric table reaches
`series_count * samples_per_series` rows before trusting the query measurements.
Use `--fixture-generator /path/to/query_perf_fixture` to provide the Rust helper.
Deprecated `--remote-write-generator` and `--storage-inspector` options are kept
only for CLI compatibility. `--fixture-only` is rejected for remote-write cases;
use `--dry-run` for planning.
Use `--fixture-generator /path/to/query_perf_fixture` to provide the Rust helper
to the outer driver.
Large manual remote-write cases can set `sample_chunk_size` to split ingestion by
time. For each chunk, the runner invokes `query_perf_fixture prom-remote-write` with the
time. For each chunk, `prepare-remote` invokes `query_perf_fixture prom-remote-write` with the
same series cardinality but a shorter `--samples-per-series` and an advanced
`--start-unix-millis`. `flush_every_sample_chunks` controls periodic
`ADMIN FLUSH_TABLE('<physical_table>')` calls; with `flush_every_sample_chunks = 1`,
@@ -213,34 +210,27 @@ workflow dispatch accepts the `heavy` token to select this case.
## OTLP trace load scenario
`scenario.kind = "otlp_trace_load"` runs a bounded native `otelgen` process
against each local distributed cluster. The runner excludes the configured
warmup window, derives throughput and mean request latency from GreptimeDB's
OTLP-specific metrics, flushes the trace table, and verifies its row count
against the accepted-span counter. The case is intentionally outside the
default set until its variance is known.
against each local distributed cluster. The outer driver runs the base target
alone with Rust `run-otlp-target`, fully stops it, then runs the candidate alone
and calls Rust `finalize-otlp` to aggregate metrics, thresholds, and the final
report. The case is intentionally outside the default set until its variance is
known.
Build base/candidate `greptime` binaries with the same profile, build the
candidate `query_perf_fixture`, then run the case explicitly:
Build base/candidate `greptime` binaries with the same profile and build the
candidate `query_perf_fixture` and `query_regression_runner`, then run the outer
driver explicitly:
```bash
WORK_DIR="$(mktemp -d /tmp/query-perf-otlp.XXXXXX)"
REPORT="$WORK_DIR/trace-report.json"
python3 tests/perf/query_regression_runner.py \
--case tests/perf/query_cases/otlp_trace_load/case.toml \
uv run --no-project python .github/scripts/query-regression-run.py \
--cases tests/perf/query_cases/otlp_trace_load/case.toml \
--base-bin /path/to/base/target/nightly/greptime \
--candidate-bin /path/to/candidate/target/nightly/greptime \
--fixture-generator /path/to/candidate/target/nightly/query_perf_fixture \
--runner /path/to/candidate/target/nightly/query_regression_runner \
--otelgen-bin /path/to/otelgen \
--work-dir "$WORK_DIR" \
--output "$REPORT"
```
Install [YouPlot](https://github.com/red-data-tools/YouPlot) once, then render
all comparison metrics and threshold results in the terminal:
```bash
brew install youplot
tests/perf/plot_otlp_trace_report.sh "$REPORT"
--work-dir "$WORK_DIR"
REPORT="$WORK_DIR/otlp_trace_load/query-regression-report.json"
```
For each target, `accepted_spans` should equal `table_rows`, and `failures`
@@ -323,8 +313,8 @@ seed = 12345
# query, warmups, iterations, thresholds
```
The runner currently supports `direct_readable_sst` and
`prom_remote_write_then_query`.
The outer driver currently supports `direct_readable_sst`,
`prom_remote_write_then_query`, and `otlp_trace_load`.
## Metrics
@@ -340,120 +330,49 @@ Primary gates should compare query work rather than plan text:
Plan details such as pushed filters are useful diagnostics, but should not be the
main pass/fail signal.
## Runner MVP
## Runner lifecycle
`query_regression_runner.py` is the base-vs-candidate orchestration layer. The
current MVP parses a case, creates per-target work directories, and in real query
mode starts a local distributed cluster for each target: metasrv (memory-store,
region failover disabled), one datanode (`node_id=0`), and one frontend. It
creates the configured Mito table(s) through frontend HTTP SQL, discovers the
real one-region-per-table metadata via `information_schema`, stops only the
owning datanode, generates one shared direct-SST fixture per table using the
discovered `--region-id`, `--table-dir`, and `--table`, injects those region
subtrees into the datanode data home, restarts the datanode, then validates and
measures through frontend. Reports are written as JSON under the work directory.
`.github/scripts/query-regression-run.py` is the outer CI driver. It resolves the
base and candidate `greptime` binaries, candidate `query_perf_fixture`, and
candidate `query_regression_runner`; allocates disjoint localhost
meta/datanode/frontend HTTP, gRPC, MySQL, and Postgres ports for each target; and
writes component logs below `<work-dir>/<target>/logs/`.
The runner intentionally keeps metasrv alive for the whole target run because
memory-store metadata would otherwise be lost. It replaces only the discovered
datanode region directory under `data/greptime/<schema>/<table_id>/...` with
generated SST files and a manifest checkpoint. For multi-table cases this is
repeated per table, enabling true JOIN fixtures while still requiring exactly one
region per table. Base and candidate must discover identical per-table
`table_dir` and `region_id`; otherwise the run fails.
The Rust runner consumes the normalized plan and endpoint ports. For
`direct_readable_sst`, the driver starts both clusters, runs `prepare-direct`,
stops only both datanodes, writes File-object-store destination TOMLs, invokes
`materialize` for every fixture and target, restarts datanodes, then runs
`measure` (with one frontend restart retry). `materialize` uses OpenDAL and only
replaces the fixture's exact region prefix in each target data home.
Multi-table direct-SST cases must use unique table names as well as unique
`(database, name)` pairs because the generator currently selects a table with
`--table <name>`. Per-table fixture directories are derived from table index,
database, and table name with path-unsafe characters sanitized.
For `prom_remote_write_then_query`, it renders a target frontend configuration
with `render-remote-config`, starts both clusters, runs `prepare-remote` and
`measure`, stops both datanodes, then runs `finalize-remote`. Finalization owns
storage inspection and read-bench against the quiescent data homes. Both paths
write `query-regression-report.json` in the case work directory and always clean
up the remaining components.
Currently enforced threshold:
- `max_candidate_latency_regression_pct`, based on client-side median latency.
Server-side scan thresholds such as file ranges and scanned rows are planned for
a follow-up PR that extracts them from structured `EXPLAIN ANALYZE VERBOSE`
output. Do not add those threshold keys until the runner enforces them.
Dry-run example:
For local orchestration after building the three binaries, invoke the outer
driver directly:
```bash
uv run --no-project python tests/perf/query_regression_runner.py \
--case tests/perf/query_cases/promql_pushdown_7913/case.toml \
--base-bin /path/to/base/greptime \
--candidate-bin /path/to/candidate/greptime \
--work-dir /tmp/query-perf-work \
--dry-run
```
With a fixture generator:
```bash
uv run --no-project python tests/perf/query_regression_runner.py \
--case tests/perf/query_cases/promql_pushdown_7913/case.toml \
uv run --no-project python .github/scripts/query-regression-run.py \
--cases tests/perf/query_cases/smoke_direct_sst/case.toml \
--base-bin /path/to/base/greptime \
--candidate-bin /path/to/candidate/greptime \
--fixture-generator /path/to/query_perf_fixture \
--fixture-cache-dir /mnt/query-regression-fixtures \
--allow-large-fixture \
--work-dir /tmp/query-perf-work
--runner /path/to/query_regression_runner \
--work-dir /tmp/query-regression-work
```
This mode launches metasrv, datanode, and frontend for each target with explicit
localhost HTTP/gRPC/MySQL/Postgres ports and writes component stdout/stderr under
each target's `logs/` directory.
By default query mode requires fresh base/candidate work directories and fails if
either target directory already exists with contents. Use `--reuse-work-dir` only
when intentionally debugging an existing run directory. SQL HTTP requests default
to a 120 second timeout; override with `--http-timeout <seconds>` for slow lab
runs.
For large direct-SST fixtures, pass `--fixture-cache-dir <dir>` to store generated
fixtures in a persistent content-addressed cache keyed by case name and fixture
data configuration. Query and threshold edits reuse the same cached data as long
as the scenario layout and table definitions do not change. Cached fixtures are
reused automatically when their `summary.json`
matches the discovered table/region metadata; incompatible entries are
regenerated instead of reused. Fixture materialization keeps base and candidate
data directories isolated, but copies files efficiently by trying filesystem
reflinks first, hardlinks for immutable SST/object files next, and normal copies
as a fallback. Manifest files are reflinked or copied, not hardlinked.
Fixture generator smoke test:
The Rust runner subcommands are also useful for focused diagnostics:
```bash
cargo run -p cmd --bin query_perf_fixture --features dev-tools -- \
direct-sst \
--case tests/perf/query_cases/smoke_direct_sst/case.toml \
--out-dir /tmp/query-perf-smoke
```
Runner smoke test with fixture generation only:
```bash
uv run --no-project python tests/perf/query_regression_runner.py \
--case tests/perf/query_cases/smoke_direct_sst/case.toml \
--base-bin /path/to/query_perf_fixture \
--candidate-bin /path/to/query_perf_fixture \
--fixture-generator /path/to/query_perf_fixture \
--work-dir /tmp/query-perf-runner-smoke \
--fixture-only
```
`--fixture-only` preserves the earlier smoke behavior: it does not start
standalone servers, and it materializes the generated fixture into base and
candidate data directories for plumbing validation.
Remote-write runner dry-run:
```bash
uv run --no-project python tests/perf/query_regression_runner.py \
--case tests/perf/query_cases/prom_remote_write_seeded_random/case.toml \
--base-bin /path/to/base/greptime \
--candidate-bin /path/to/candidate/greptime \
--fixture-generator /path/to/query_perf_fixture \
--work-dir /tmp/query-perf-remote-write \
--dry-run
query_regression_runner prepare-direct --case <case.toml> --fixture-generator <fixture> \
--base-http-port <port> --candidate-http-port <port> --fixture-dir <dir> --output <json>
query_regression_runner materialize --fixture-dir <dir> --destination <toml>
query_regression_runner measure --case <case.toml> --fixture-generator <fixture> \
--base-http-port <port> --candidate-http-port <port> --output <report.json>
```
## GitHub Actions
@@ -462,14 +381,17 @@ uv run --no-project python tests/perf/query_regression_runner.py \
query regression runs. It builds its own binaries for now:
- base `greptime` from the PR base commit, or `workflow_dispatch` `base_ref`
- candidate `greptime` and `query_perf_fixture` from the PR merge ref/current
candidate checkout
- runner and summary formatter from the candidate checkout
- candidate `greptime`, `query_perf_fixture`, and `query_regression_runner` from
the PR merge ref/current candidate checkout
- outer driver and summary formatter from the candidate checkout
The workflow builds base and candidate `greptime` as normal release-equivalent
binaries. Candidate `query_perf_fixture` is the extra head-side helper binary;
the runner uses candidate `greptime datanode parquetbench/scanbench` as the
read-bench tool against each target's data directory.
The default `aliyun-ecs` runner provisions one ephemeral 50 GiB-system-disk
Aliyun ECS instance per run and tears it down afterwards; the scheduled janitor
reclaims leftovers. The workflow builds base and candidate `greptime` as normal
release-equivalent binaries. Candidate `query_perf_fixture` and
`query_regression_runner` are the extra head-side helpers; `finalize-remote`
uses candidate `greptime datanode parquetbench/scanbench` as the read-bench tool
against each target's data directory.
The workflow runs automatically only when `query-regression` or `heavy-regression`
is added to a non-draft PR; it does not rerun on pushes, ready-for-review, or
tests/perf/plot_otlp_trace_report.sh
-80
View File
@@ -1,80 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
if [[ $# -ne 1 ]]; then
printf 'Usage: %s REPORT.json\n' "${0##*/}" >&2
exit 2
fi
report="$1"
[[ -f "${report}" ]] || {
printf 'Report does not exist: %s\n' "${report}" >&2
exit 1
}
command -v jq >/dev/null || {
printf 'jq is required\n' >&2
exit 1
}
command -v uplot >/dev/null || {
printf 'YouPlot is required; install it with: brew install youplot\n' >&2
exit 1
}
jq -e '
(.status | type == "string") and
(.targets | type == "array" and length > 0) and
all(.targets[];
(.name | type == "string") and
(.metrics.accepted_spans | type == "number") and
(.visibility.observed_rows | type == "number") and
(.metrics.accepted_spans_per_second | type == "number") and
(.metrics.mean_http_latency_ms | type == "number") and
(.metrics.failure_count | type == "number")
)
' "${report}" >/dev/null || {
printf 'Report is not a completed OTLP trace comparison: %s\n' "${report}" >&2
exit 1
}
plot_metric() {
local title="$1"
local filter="$2"
local color="$3"
printf '\n'
jq -r ".targets[] | [.name, ${filter}] | @tsv" "${report}" |
uplot bar -t "${title}" -w 80 -c "${color}" -o -
}
jq -r '"Status: \(.status)"' "${report}"
row_count_status=0
jq -r '
"\nAccepted spans vs visible table rows",
(.targets[] |
" [\(if .metrics.accepted_spans == .visibility.observed_rows then "passed" else "failed" end)] \(.name): accepted=\(.metrics.accepted_spans), visible=\(.visibility.observed_rows)"
)
' "${report}"
jq -e 'all(.targets[]; .metrics.accepted_spans == .visibility.observed_rows)' "${report}" >/dev/null || row_count_status=1
plot_metric "Accepted spans" ".metrics.accepted_spans" blue
plot_metric "Visible table rows" ".visibility.observed_rows" green
plot_metric "Throughput (spans/s, higher is better)" ".metrics.accepted_spans_per_second" cyan
plot_metric "Mean HTTP latency (ms, lower is better)" ".metrics.mean_http_latency_ms" yellow
plot_metric "Failures (lower is better)" ".metrics.failure_count" red
jq -r '
"\nThresholds",
(.thresholds[] |
if has("actual_pct") then
" [\(.status)] \(.threshold): \(.actual_pct)% (limit \(.limit_pct)%; base \(.base), candidate \(.candidate))"
elif has("actual") then
" [\(.status)] \(.threshold) [\(.target)]: \(.actual) (limit \(.limit))"
else
" [\(.status)] \(.threshold): \(.reason // "no measured value")"
end
)
' "${report}"
exit "${row_count_status}"
tests/perf/query_regression_runner.py
-1663
View File
File diff suppressed because it is too large Load Diff
tests/perf/test_aliyun_ecs_runner_scripts.py
+122
View File
@@ -0,0 +1,122 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Coverage for the pure parts of the Aliyun ECS runner provision/teardown scripts."""
import base64
import importlib.util
import sys
import unittest
from datetime import datetime, timedelta, timezone
from pathlib import Path
SCRIPTS_DIR = Path(__file__).parents[2] / ".github/scripts"
def load_module(name: str, filename: str):
spec = importlib.util.spec_from_file_location(name, SCRIPTS_DIR / filename)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
provision = load_module("aliyun_ecs_runner_provision_under_test", "aliyun-ecs-runner-provision.py")
teardown = load_module("aliyun_ecs_runner_teardown_under_test", "aliyun-ecs-runner-teardown.py")
class ProvisionNamingTest(unittest.TestCase):
def test_runner_name_and_label_derive_from_run_id(self) -> None:
self.assertEqual(provision.runner_name_for_run("12345"), "qreg-ecs-12345")
self.assertEqual(provision.runner_label_for_run("12345"), "query-regression-ecs-12345")
class ProvisionUserDataTest(unittest.TestCase):
def render(self) -> str:
return provision.render_user_data(
runner_name="qreg-ecs-12345",
runner_label="query-regression-ecs-12345",
runner_token="TOKEN",
repo="GreptimeTeam/greptimedb",
)
def test_user_data_creates_cache_paths_on_the_system_disk(self) -> None:
script = self.render()
self.assertNotIn("DISK_SERIAL", script)
self.assertNotIn("mount --bind", script)
self.assertNotIn("mkfs.ext4", script)
for destination in provision.CACHE_PATHS:
self.assertIn(f'"{destination}"', script)
def test_user_data_wires_runner_registration(self) -> None:
script = self.render()
self.assertIn("RUNNER_NAME=qreg-ecs-12345", script)
self.assertIn("RUNNER_LABELS=query-regression-ecs-12345", script)
self.assertIn("RUNNER_TOKEN=TOKEN", script)
self.assertIn("REPO_URL=https://github.com/GreptimeTeam/greptimedb", script)
self.assertIn("PATH=/opt/cargo/bin:", script)
self.assertIn("systemctl restart --no-block ephemeral-github-runner.service", script)
def test_encode_user_data_round_trips(self) -> None:
script = self.render()
self.assertEqual(
base64.b64decode(provision.encode_user_data(script)).decode("utf-8"), script
)
def test_user_data_enables_swap_and_masks_oomd(self) -> None:
script = self.render()
self.assertIn("systemctl mask systemd-oomd.socket systemd-oomd.service", script)
self.assertIn("systemctl mask unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer", script)
self.assertIn('APT::Periodic::Unattended-Upgrade "0"', script)
self.assertIn(f'fallocate --length {provision.SWAP_SIZE_GIB}G "{provision.SWAP_FILE}"', script)
self.assertIn(f'swapon "{provision.SWAP_FILE}"', script)
self.assertIn("sysctl --write vm.swappiness=10", script)
self.assertIn("OOMPolicy=continue", script)
self.assertNotIn("OOMScoreAdjust", script)
self.assertLess(script.index("swapon"), script.index("systemctl restart --no-block ephemeral-github-runner.service"))
class TeardownExpiryTest(unittest.TestCase):
NOW = datetime(2026, 8, 17, 6, 0, tzinfo=timezone.utc)
TTL = timedelta(hours=4)
def test_parse_creation_time_formats(self) -> None:
self.assertEqual(
teardown.parse_creation_time("2026-08-17T01:02:03Z"),
datetime(2026, 8, 17, 1, 2, 3, tzinfo=timezone.utc),
)
self.assertEqual(
teardown.parse_creation_time("2026-08-17T01:02Z"),
datetime(2026, 8, 17, 1, 2, tzinfo=timezone.utc),
)
with self.assertRaises(ValueError):
teardown.parse_creation_time("not-a-time")
def test_expired_instance_names_selects_only_old_instances(self) -> None:
instances = [
("i-old", "qreg-ecs-1", "2026-08-17T01:00Z"), # 5h old: expired
("i-edge", "qreg-ecs-2", "2026-08-17T02:00Z"), # exactly TTL: expired
("i-fresh", "qreg-ecs-3", "2026-08-17T05:30Z"), # 30m old: kept
]
self.assertEqual(
teardown.expired_instance_names(instances, self.NOW, self.TTL),
[("i-old", "qreg-ecs-1"), ("i-edge", "qreg-ecs-2")],
)
if __name__ == "__main__":
unittest.main()
tests/perf/test_query_regression_nightly_refs.py
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
# Copyright 2023 Greptime Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Unit tests for consecutive Nightly Build SHA selection."""
import importlib.util
import sys
import unittest
from pathlib import Path
SCRIPTS_DIR = Path(__file__).parents[2] / ".github/scripts"
def load_module():
spec = importlib.util.spec_from_file_location(
"query_regression_nightly_refs_under_test",
SCRIPTS_DIR / "query-regression-nightly-refs.py",
)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
refs = load_module()
def run(*, run_id: int, sha: str, branch: str = "main", created_at: str = "2026-08-24T00:00:00Z"):
return refs.WorkflowRun(
id=run_id,
head_sha=sha,
head_branch=branch,
html_url=f"https://github.com/example/run/{run_id}",
created_at=created_at,
conclusion="success",
event="schedule",
)
class SelectNightlyRefsTest(unittest.TestCase):
def test_picks_newest_and_previous_on_same_branch(self) -> None:
pair = refs.select_base_and_candidate(
[
run(run_id=3, sha="ccc", created_at="2026-08-22T00:00:00Z"),
run(run_id=2, sha="bbb", created_at="2026-08-21T00:00:00Z"),
run(run_id=1, sha="aaa", created_at="2026-08-20T00:00:00Z"),
]
)
self.assertFalse(pair.skip)
assert pair.candidate is not None and pair.base is not None
self.assertEqual(pair.candidate.head_sha, "ccc")
self.assertEqual(pair.base.head_sha, "bbb")
def test_candidate_run_id_uses_that_run_and_the_next_older(self) -> None:
pair = refs.select_base_and_candidate(
[
run(run_id=3, sha="ccc", created_at="2026-08-22T00:00:00Z"),
run(run_id=2, sha="bbb", created_at="2026-08-21T00:00:00Z"),
run(run_id=1, sha="aaa", created_at="2026-08-20T00:00:00Z"),
],
candidate_run_id=2,
)
self.assertFalse(pair.skip)
assert pair.candidate is not None and pair.base is not None
self.assertEqual(pair.candidate.id, 2)
self.assertEqual(pair.base.id, 1)
def test_skips_when_consecutive_nightlies_share_a_sha(self) -> None:
pair = refs.select_base_and_candidate(
[
run(run_id=2, sha="same", created_at="2026-08-22T00:00:00Z"),
run(run_id=1, sha="same", created_at="2026-08-21T00:00:00Z"),
]
)
self.assertTrue(pair.skip)
self.assertIn("matches candidate", pair.reason)
def test_skips_other_branches_when_picking_previous(self) -> None:
pair = refs.select_base_and_candidate(
[
run(run_id=3, sha="ccc", branch="main", created_at="2026-08-22T00:00:00Z"),
run(run_id=2, sha="other", branch="feat", created_at="2026-08-21T12:00:00Z"),
run(run_id=1, sha="aaa", branch="main", created_at="2026-08-21T00:00:00Z"),
]
)
self.assertFalse(pair.skip)
assert pair.base is not None
self.assertEqual(pair.base.head_sha, "aaa")
def test_skips_when_only_one_nightly_exists(self) -> None:
pair = refs.select_base_and_candidate(
[run(run_id=1, sha="aaa")],
)
self.assertTrue(pair.skip)
self.assertIn("no previous", pair.reason)
def test_explicit_refs_bypass_github(self) -> None:
pair = refs.override_pair("base-sha", "cand-sha")
self.assertFalse(pair.skip)
assert pair.base is not None and pair.candidate is not None
self.assertEqual(pair.base.head_sha, "base-sha")
self.assertEqual(pair.candidate.head_sha, "cand-sha")
if __name__ == "__main__":
unittest.main()
tests/perf/test_query_regression_runner_compaction_toctou.py
+67 -129
View File
@@ -13,9 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Regression coverage for quiescing remote-write storage before read benches."""
"""Regression coverage for the outer remote-write lifecycle."""
import argparse
import importlib.util
import json
import sys
@@ -25,151 +24,90 @@ from pathlib import Path
from unittest.mock import patch
RUNNER_PATH = Path(__file__).with_name("query_regression_runner.py")
SPEC = importlib.util.spec_from_file_location("query_regression_runner_under_test", RUNNER_PATH)
RUNNER_PATH = Path(__file__).resolve().parents[2] / ".github/scripts/query-regression-run.py"
SPEC = importlib.util.spec_from_file_location("query_regression_outer_under_test", RUNNER_PATH)
assert SPEC is not None and SPEC.loader is not None
runner = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = runner
SPEC.loader.exec_module(runner)
class ReportOutputTest(unittest.TestCase):
def test_final_report_output_defaults_to_stdout_or_writes_to_file(self) -> None:
report = {"status": "ok"}
expected = json.dumps(report, indent=2, sort_keys=True)
class RemoteWriteLifecycleTest(unittest.TestCase):
def setUp(self) -> None:
self.args = type("Args", (), {"http_timeout": "1"})()
self.ports = list(range(10000, 10016))
with patch("builtins.print") as print_mock:
runner.output_report(report, None)
print_mock.assert_called_once_with(expected)
def test_measure_report_is_finalized_after_datanodes_stop(self) -> None:
events: list[str] = []
with tempfile.TemporaryDirectory() as tmpdir:
output = Path(tmpdir) / "nested" / "report.json"
with patch("builtins.print") as print_mock:
runner.output_report(report, output)
print_mock.assert_not_called()
self.assertEqual(output.read_text(), expected + "\n")
def run(command, **_kwargs):
phase = command[1]
events.append(phase)
if phase == "measure":
output = Path(command[command.index("--output") + 1])
output.write_text(json.dumps({"targets": [], "thresholds": [], "status": "ok"}))
return type("Result", (), {"returncode": 0})()
def start(target, component, _procs):
events.append(f"start:{target.name}:{component}")
class RemoteWriteCompactionToctouTest(unittest.TestCase):
def test_quiesces_each_target_before_storage_inspection_and_read_bench(self) -> None:
lifecycle: dict[str, list[str]] = {"base": [], "candidate": []}
clusters = []
def stop(target, component, _procs):
events.append(f"stop:{target.name}:{component}")
class RecordingCluster:
def __init__(self, target):
self.target = target
self.active = set()
self.stop_all_calls = 0
clusters.append(self)
with (
tempfile.TemporaryDirectory() as tmpdir,
patch.object(runner, "allocate_ports", return_value=self.ports),
patch.object(runner, "start_component", side_effect=start),
patch.object(runner, "stop_component", side_effect=stop),
patch.object(runner.subprocess, "run", side_effect=run),
):
self.assertEqual(
runner.run_remote_case(
self.args, Path("case.toml"), Path(tmpdir), Path("base"), Path("candidate"), Path("fixture"), Path("runner")
),
0,
)
self.assertTrue((Path(tmpdir) / "query-regression-report.json").exists())
def start_all(self, _config_path):
self.active.update(("metasrv", "datanode", "frontend"))
measure = events.index("measure")
finalize = events.index("finalize-remote")
self.assertLess(measure, events.index("stop:base:datanode"))
self.assertLess(events.index("stop:base:datanode"), finalize)
self.assertLess(events.index("stop:candidate:datanode"), finalize)
self.assertIn("stop:base:frontend", events[finalize + 1:])
self.assertIn("stop:candidate:metasrv", events[finalize + 1:])
def stop_component(self, name):
if name in self.active:
lifecycle[self.target.name].append(f'cluster.stop_component("{name}")')
self.active.remove(name)
def test_phase_exception_still_cleans_up_without_processes_or_network(self) -> None:
events: list[str] = []
def stop_all(self):
self.stop_all_calls += 1
for name in ("frontend", "datanode", "metasrv"):
self.stop_component(name)
def run(command, **_kwargs):
phase = command[1]
events.append(phase)
if phase == "prepare-remote":
raise RuntimeError("prepare exploded")
return type("Result", (), {"returncode": 0})()
def component_report(self):
return {}
def start(target, component, _procs):
events.append(f"start:{target.name}:{component}")
def run_queries(target, _case, _tables, _http_timeout):
lifecycle[target.name].append("query measurement completes")
return {"validation": [], "validation_errors": [], "measurements": [], "status": "ok"}
def stop(target, component, _procs):
events.append(f"stop:{target.name}:{component}")
def inspect_storage(_helper, target, _storage, *, dry_run):
self.assertFalse(dry_run)
lifecycle[target.name].append("storage inspection")
return {"status": "ok", "summary": {"summary": {}}}
with (
tempfile.TemporaryDirectory() as tmpdir,
patch.object(runner, "allocate_ports", return_value=self.ports),
patch.object(runner, "start_component", side_effect=start),
patch.object(runner, "stop_component", side_effect=stop),
patch.object(runner.subprocess, "run", side_effect=run),
):
with self.assertRaisesRegex(RuntimeError, "prepare exploded"):
runner.run_remote_case(
self.args, Path("case.toml"), Path(tmpdir), Path("base"), Path("candidate"), Path("fixture"), Path("runner")
)
def read_bench(_binary, target, _read_bench, _inspection, *, dry_run):
self.assertFalse(dry_run)
lifecycle[target.name].append("read bench")
return {"status": "ok"}
remote_write = {
"database": "public",
"metric": "cpu_usage",
"physical_table": "cpu_usage_physical",
"series_count": 1,
"samples_per_series": 1,
"visibility_timeout_seconds": 1,
"storage": {
"inspect": True,
"min_files": None,
"min_files_with_column": None,
"max_total_file_size_bytes": None,
"max_column_compressed_size_bytes": None,
"max_column_uncompressed_size_bytes": None,
"require_encodings": [],
"forbid_encodings": [],
},
"read_bench": {"enabled": True},
"prom_store": {
"pending_rows_flush_interval": "1s",
"max_batch_rows": 1,
"max_concurrent_flushes": 1,
"worker_channel_capacity": 1,
"max_inflight_requests": 1,
},
}
case = {
"scenario": {
"kind": "prom_remote_write_then_query",
"remote_write": remote_write,
"queries": [],
}
}
args = argparse.Namespace(
fixture_only=False,
fixture_generator=Path("/bin/true"),
remote_write_generator=None,
storage_inspector=None,
candidate_bin=Path("/bin/true"),
dry_run=False,
http_timeout=1.0,
)
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
targets = [
runner.make_target("base", Path("/bin/true"), root, list(range(10000, 10008))),
runner.make_target("candidate", Path("/bin/true"), root, list(range(10008, 10016))),
]
report = {"targets": [], "thresholds": []}
with (
patch.object(runner, "DistributedCluster", RecordingCluster),
patch.object(runner, "run_remote_write_ingestion", return_value=({"status": "ok"}, [{"ok": True}])),
patch.object(runner, "http_post_sql", return_value={"ok": True}),
patch.object(runner, "poll_expected_count", return_value={"ok": True, "row_count_ok": True}),
patch.object(runner, "run_queries", side_effect=run_queries),
patch.object(runner, "run_storage_inspection", side_effect=inspect_storage),
patch.object(runner, "run_read_bench", side_effect=read_bench),
):
runner.run_remote_write_scenario(args, case, root / "case.toml", targets, report)
for cluster in clusters:
self.assertEqual(cluster.stop_all_calls, 1)
self.assertEqual(cluster.active, set())
cluster.stop_all()
self.assertEqual(cluster.stop_all_calls, 2)
self.assertEqual(cluster.active, set())
expected = [
"query measurement completes",
'cluster.stop_component("datanode")',
"storage inspection",
"read bench",
]
relevant = {name: [event for event in events if event in expected] for name, events in lifecycle.items()}
for target_name in ("base", "candidate"):
with self.subTest(target=target_name):
self.assertEqual(relevant[target_name], expected)
self.assertIn("prepare-remote", events)
self.assertIn("stop:base:metasrv", events)
self.assertIn("stop:candidate:frontend", events)
if __name__ == "__main__":
tests/perf/test_query_regression_runner_otlp_trace_load.py
+54 -189
View File
@@ -13,215 +13,80 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""Regression coverage for the local OTLP trace load lifecycle and metrics."""
"""Regression coverage for the outer OTLP lifecycle."""
import importlib.util
import json
import os
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import patch
RUNNER_PATH = Path(__file__).with_name("query_regression_runner.py")
PLOTTER_PATH = Path(__file__).with_name("plot_otlp_trace_report.sh")
SPEC = importlib.util.spec_from_file_location("query_regression_runner_otlp_under_test", RUNNER_PATH)
RUNNER_PATH = Path(__file__).resolve().parents[2] / ".github/scripts/query-regression-run.py"
SPEC = importlib.util.spec_from_file_location("query_regression_outer_otlp_under_test", RUNNER_PATH)
assert SPEC is not None and SPEC.loader is not None
runner = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = runner
SPEC.loader.exec_module(runner)
class OtlpTraceReportPlotTest(unittest.TestCase):
def test_plots_all_comparison_metrics_and_thresholds(self) -> None:
report = {
"status": "ok",
"targets": [
{
"name": name,
"metrics": {
"accepted_spans": spans,
"accepted_spans_per_second": rate,
"mean_http_latency_ms": latency,
"failure_count": 0,
},
"visibility": {"observed_rows": spans},
}
for name, spans, rate, latency in (("base", 100, 10.0, 3.0), ("candidate", 120, 12.0, 2.5))
],
"thresholds": [
{"target": "base", "threshold": "max_failure_count", "status": "passed", "actual": 0, "limit": 0},
{
"threshold": "max_candidate_throughput_regression_pct",
"status": "passed",
"actual_pct": -20.0,
"limit_pct": 20.0,
"base": 10.0,
"candidate": 12.0,
},
],
}
class OtlpTraceOuterLifecycleTest(unittest.TestCase):
def test_stops_base_before_candidate_and_wires_rust_commands(self) -> None:
events: list[str] = []
commands: list[list[str]] = []
with tempfile.TemporaryDirectory() as tmpdir:
def run(command, **_kwargs):
commands.append(command)
phase = command[1]
events.append(phase)
output = Path(command[command.index("--output") + 1])
output.write_text(json.dumps({"status": "ok", "targets": []}))
return SimpleNamespace(returncode=0)
def start(target, component, _procs):
events.append(f"start:{target.name}:{component}")
def stop(target, component, _procs):
events.append(f"stop:{target.name}:{component}")
args = SimpleNamespace(http_timeout="1", otelgen_bin=Path("otelgen"))
ports = list(range(10000, 10016))
with (
tempfile.TemporaryDirectory() as tmpdir,
patch.object(runner, "allocate_ports", return_value=ports),
patch.object(runner, "start_component", side_effect=start),
patch.object(runner, "stop_component", side_effect=stop),
patch.object(runner.subprocess, "run", side_effect=run),
):
root = Path(tmpdir)
report_path = root / "report.json"
report_path.write_text(json.dumps(report))
fake_bin = root / "bin"
fake_bin.mkdir()
uplot = fake_bin / "uplot"
uplot.write_text('#!/usr/bin/env bash\nprintf "CALL %s\\n" "$*" >>"$UPLOT_LOG"\ncat >>"$UPLOT_LOG"\n')
uplot.chmod(0o755)
log_path = root / "uplot.log"
env = os.environ.copy()
env["PATH"] = f"{fake_bin}{os.pathsep}{env['PATH']}"
env["UPLOT_LOG"] = str(log_path)
result = subprocess.run([str(PLOTTER_PATH), str(report_path)], env=env, text=True, capture_output=True, check=False)
log = log_path.read_text()
report["targets"][1]["visibility"]["observed_rows"] = 119
report_path.write_text(json.dumps(report))
mismatch = subprocess.run([str(PLOTTER_PATH), str(report_path)], env=env, text=True, capture_output=True, check=False)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(log.count("CALL "), 5)
for metric in ("Accepted spans", "Visible table rows", "Throughput", "Mean HTTP latency", "Failures"):
self.assertIn(metric, log)
self.assertIn("base\t100", log)
self.assertIn("candidate\t12", log)
self.assertIn("Status: ok", result.stdout)
self.assertIn("[passed] candidate: accepted=120, visible=120", result.stdout)
self.assertIn("max_failure_count", result.stdout)
self.assertIn("max_candidate_throughput_regression_pct", result.stdout)
self.assertEqual(mismatch.returncode, 1, mismatch.stderr)
self.assertIn("[failed] candidate: accepted=120, visible=119", mismatch.stdout)
class OtlpTraceLoadTest(unittest.TestCase):
def test_stops_base_cluster_before_creating_candidate_cluster(self) -> None:
events = []
class FakeCluster:
def __init__(self, target):
self.target = target
self.stopped = False
events.append(f"create:{target.name}")
def component_report(self):
return {}
def stop_all(self):
if not self.stopped:
self.stopped = True
events.append(f"stop:{self.target.name}")
load = {
"database": "public",
"table": "opentelemetry_traces",
"pipeline": "greptime_trace_v1",
"duration_seconds": 120,
"warmup_seconds": 60,
"rate": 50_000,
"workers": 4,
"workload": "microservices",
"exporter_shards": 4,
"visibility_timeout_seconds": 1,
"thresholds": {
"max_candidate_throughput_regression_pct": 20,
"max_candidate_mean_latency_regression_pct": 20,
"max_failure_count": 0,
},
}
args = runner.argparse.Namespace(fixture_only=False, otelgen_bin=None, dry_run=True, http_timeout=1.0)
with tempfile.TemporaryDirectory() as tmpdir:
root = Path(tmpdir)
targets = [
runner.make_target("base", Path("/bin/true"), root, list(range(10_000, 10_008))),
runner.make_target("candidate", Path("/bin/true"), root, list(range(10_008, 10_016))),
self.assertEqual(
runner.run_otlp_case(args, Path("case.toml"), root, Path("base-bin"), Path("candidate-bin"), Path("fixture"), Path("runner")),
0,
)
expected = [
[
"runner", "run-otlp-target", "--case", "case.toml", "--fixture-generator", "fixture",
"--otelgen-bin", "otelgen", "--http-port", "10004", "--target-name", "base",
"--work-dir", str(root / "base"), "--output", str(root / "base" / "report.json"), "--http-timeout", "1",
],
[
"runner", "run-otlp-target", "--case", "case.toml", "--fixture-generator", "fixture",
"--otelgen-bin", "otelgen", "--http-port", "10012", "--target-name", "candidate",
"--work-dir", str(root / "candidate"), "--output", str(root / "candidate" / "report.json"), "--http-timeout", "1",
],
[
"runner", "finalize-otlp", "--case", "case.toml", "--fixture-generator", "fixture",
"--base-result", str(root / "base" / "report.json"),
"--candidate-result", str(root / "candidate" / "report.json"),
"--output", str(root / "query-regression-report.json"),
],
]
with patch.object(runner, "DistributedCluster", FakeCluster):
runner.run_otlp_trace_load_scenario(
args,
{"scenario": {"kind": "otlp_trace_load", "load": load}},
targets,
{"targets": []},
)
self.assertEqual(commands, expected)
self.assertEqual(events, ["create:base", "stop:base", "create:candidate", "stop:candidate"])
def test_warmup_lifecycle_and_labeled_metric_deltas(self) -> None:
class FakeProcess:
def __init__(self, *_args, **_kwargs):
self.returncode = None
self.wait_timeouts = []
def wait(self, timeout):
self.wait_timeouts.append(timeout)
if len(self.wait_timeouts) == 1:
raise subprocess.TimeoutExpired("otelgen", timeout)
self.returncode = 0
return 0
def poll(self):
return self.returncode
def kill(self):
self.returncode = -9
def snapshot(text: str, captured: float):
return {"captured_monotonic_seconds": captured, "values": runner.parse_prometheus_metrics(text)}
snapshots = [
snapshot("greptime_frontend_otlp_traces_rows 10\n", 0.0),
snapshot(
'greptime_frontend_otlp_traces_rows 110\n'
'greptime_servers_http_otlp_traces_elapsed_sum{db="public"} 1\n'
'greptime_servers_http_otlp_traces_elapsed_count{db="public"} 10\n',
5.0,
),
snapshot(
'greptime_frontend_otlp_traces_rows 310\n'
'greptime_frontend_otlp_traces_failure_count{label="decode"} 1\n'
'greptime_frontend_otlp_traces_failure_count{label="write"} 2\n'
'greptime_servers_http_otlp_traces_elapsed_sum{db="public"} 3\n'
'greptime_servers_http_otlp_traces_elapsed_count{db="public"} 30\n',
15.0,
),
]
load = {
"database": "public",
"table": "opentelemetry_traces",
"pipeline": "greptime_trace_v1",
"duration_seconds": 120,
"warmup_seconds": 60,
"rate": 50_000,
"workers": 4,
"workload": "microservices",
"exporter_shards": 4,
}
with tempfile.TemporaryDirectory() as tmpdir:
target = runner.make_target("base", Path("/bin/true"), Path(tmpdir), list(range(10_000, 10_008)))
process = FakeProcess()
with (
patch.object(runner, "fetch_otlp_metrics", side_effect=snapshots),
patch.object(runner.subprocess, "Popen", return_value=process),
patch.object(runner.time, "monotonic", side_effect=[0.0, 120.0]),
):
result = runner.run_otelgen_load(Path("/bin/otelgen"), target, load, 1.0, dry_run=False)
self.assertEqual(result["status"], "ok")
self.assertEqual(process.wait_timeouts, [60, 120])
metrics = runner.summarize_otlp_metrics(result)
self.assertEqual(metrics["accepted_spans"], 300)
self.assertEqual(metrics["measurement_accepted_spans"], 200)
self.assertEqual(metrics["accepted_spans_per_second"], 20.0)
self.assertEqual(metrics["http_requests"], 20)
self.assertEqual(metrics["mean_http_latency_ms"], 100.0)
self.assertEqual(metrics["failure_count"], 3)
self.assertLess(events.index("stop:base:metasrv"), events.index("start:candidate:metasrv"))
if __name__ == "__main__":