diff --git a/.github/runner-scale-sets/query-regression/README.md b/.github/runner-scale-sets/query-regression/README.md index d145771989..77b7d604f1 100644 --- a/.github/runner-scale-sets/query-regression/README.md +++ b/.github/runner-scale-sets/query-regression/README.md @@ -1,321 +1,184 @@ # Query regression self-hosted runners -The `Query Regression` workflow uses the dedicated ARC runner scale set -`perf-regression-8-cores`. ARC runner Pods run in the target Kubernetes cluster -and connect outbound to GitHub. The live scale set is currently **paused**: -`minRunners=0`, `maxRunners=0`, and no runner Pods. Do not resume it without -explicit approval. +The `Query Regression` workflow runs on **Aliyun ECS ephemeral runners** +(`aliyun-ecs`, the default and only automated path): a `provision` job on +`ubuntu-latest` creates one pay-as-you-go ECS instance per run, and the +instance registers itself as an ephemeral GitHub runner with a per-run +label. A `teardown` job (`if: always()`) deletes the instance; +`query-regression-janitor.yml` sweeps tagged leftovers older than 4 hours +daily. -## Prerequisites and trust admission +Build caches live on the instance's system disk, so **every run compiles +cold**; only the within-run reuse (base build warms the candidate build +through the shared target dir and sccache) applies. There is no retained +data disk. -Install the ARC scale set controller if it is not already installed: +Dispatching with any other `runner` value uses it as a literal self-hosted +runner label, which is how a manually prepared host (see +`ecs-image/bootstrap-runner-host.sh`) runs the workflow. For PR labels, set +the repository variable `QUERY_REGRESSION_PR_RUNNER` to such a label to +redirect PR runs away from ECS. + +The office ARC scale set `perf-regression-8-cores` that previously ran this +workflow is retired; see git history for its values files and pause/deploy +procedures. Tearing down the office cluster (helm release, runner namespace, +and the `query-regression-build-cache` PVC) is a manual operator action +outside this repository. This directory keeps its historical +`runner-scale-sets` name for path stability. + +## Nightly vs previous nightly + +`query-regression-nightly.yml` runs after a successful `GreptimeDB Nightly +Build` (`workflow_run`). It resolves that run's `head_sha` as the candidate +and the previous successful nightly (same branch, typically Friday when +Monday's nightly fires) as the base, then calls `query-regression.yml` with +those immutable SHAs. Both binaries are still compiled on the ECS runner; +this is not an artifact-download path. `workflow_dispatch` can pass explicit +`base_ref` / `candidate_ref` or a nightly run id. If there is no previous +nightly, or both nightlies built the same commit, the comparison is skipped. + +## Aliyun ECS path + +Configuration lives in repository variables/secrets: + +| Kind | Name | Purpose | +| --- | --- | --- | +| secret | `ALICLOUD_ECS_ACCESS_KEY_ID` / `ALICLOUD_ECS_ACCESS_KEY_SECRET` | RAM user scoped to ECS RunInstances/DeleteInstances/Describe*/CreateImage/RunCommand. Used only by provision/teardown jobs on `ubuntu-latest`; never reaches the ECS instance. | +| secret | `GH_PERSONAL_ACCESS_TOKEN` | Creates the short-lived runner registration token (shared with the jsonbench EC2 path). | +| vars | `ALIYUN_ECS_REGION_ID` / `ALIYUN_ECS_VSWITCH_ID` / `ALIYUN_ECS_SECURITY_GROUP_ID` | Network placement. The security group should allow egress only; no inbound rules are needed. The vSwitch pins the zone. | +| vars | `ALIYUN_ECS_INSTANCE_TYPE` | Dedicated (non-burstable, non-shared) instance family. Prefer 32 GiB (e.g. `ecs.g8i.2xlarge`); `ecs.c9i.2xlarge` is 8c16g and nightly thin-LTO of greptime peaks above that. Both base and candidate clusters run on the same machine, so noisy neighbors break thresholds. | +| vars | `QUERY_REGRESSION_ECS_IMAGE_ID` | Custom image built by `ecs-image/build-ecs-image.py`. | + +The system disk is 50 GiB, which covers the image, a 16 GiB swapfile, the +checkout, and cold build caches (target dir, cargo registry, sccache). ENOSPC +stops the runner itself from writing logs, which GitHub reports as `The +operation was canceled` with no telemetry, indistinguishable from a +platform-side cancellation. +cloud-init masks `systemd-oomd`, disables `unattended-upgrades` / +`apt-daily-upgrade`, creates `/swapfile`, and sets `OOMPolicy=continue` +on the runner unit. Ubuntu 24.04 defaults to `DefaultOOMPolicy=stop`, +which SIGTERM-s the whole unit when rustc is OOM-killed and GitHub +reports `The operation was canceled` with no telemetry. Unattended +upgrades can do the same via `systemctl restart` of the runner after a +library update; GitHub then records `UserCancelled` even though the job +was still valid. Swap is a safety net for 16 GiB types, not a substitute +for 32 GiB; linking on swap is slow. + +When a run dies with an unexplained `The operation was canceled` (no +"Canceled by" banner, healthy machine), re-dispatch with `keep_instance` +checked: the teardown job is skipped and the instance survives for +post-mortem inspection. The security group is egress-only, so inspect via +Cloud Assistant (`RunCommand`) or VNC: the runner's `_diag` logs under the +runner home record reconnects and worker crashes, `journalctl -u +ephemeral-github-runner.service` mirrors the console stream, `dmesg -T` +shows kernel OOM kills, and `machine-telemetry.log` in the job workspace +has the 30 s sampler history. The janitor sweep still deletes the instance +after its TTL, so finish the inspection within that window. + +Trust model on the ECS path: the instance receives only the one-hour runner +registration token via user data and holds no cloud credentials; the Aliyun +AK/SK exist only in the control-plane jobs. Instance tags +(`managed-by=query-regression-ci`, `query-regression-run-id`) feed the janitor +sweep. + +### Building and updating the ECS image + +The runner `Dockerfile` in the parent directory stays the single source of the +tool contract. Build a new ECS image from it: ```bash -helm upgrade --install arc \ - oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \ - --namespace arc-systems \ - --create-namespace \ - --version 0.14.2 +ALIBABA_CLOUD_ACCESS_KEY_ID=... ALIBABA_CLOUD_ACCESS_KEY_SECRET=... \ +uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \ + --region-id --vswitch-id --security-group-id \ + --base-image-id ``` -Create the GitHub App secret in the runner namespace. Prefer an App limited to -`GreptimeTeam/greptimedb`: +The script boots a temporary builder instance, `docker build`s the runner +image, materializes `/opt/rustup`, `/opt/cargo`, `/usr/local/bin` tools, and +`/home/runner` (actions-runner) onto the host, installs the ephemeral-runner +systemd unit from `ecs-image/`, and snapshots a custom image. It prints the +image id; set it as `QUERY_REGRESSION_ECS_IMAGE_ID`, and bump +`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time so the target +cache invalidates. Builder sentinel polling requires the Cloud Assistant +agent, which Aliyun public Ubuntu images include. -```bash -kubectl -n arc-runners create secret generic greptimedb-arc-github-app \ - --from-literal=github_app_id= \ - --from-literal=github_app_installation_id= \ - --from-file=github_app_private_key= -``` - -The values files here reference that secret by name. +## Trust admission for PR runs A maintainer applying the `query-regression` or `heavy-regression` label is -**trust admission for that exact PR revision**. `query-regression` runs the five -routine default cases; `heavy-regression` runs only the high-cardinality -`prom_remote_write_7913` remote-write case. The admitted job may use this scale -set's dedicated, writable persistent cache. `pull_request: labeled` is the only -PR trigger: the label event snapshots its merge, head, and base SHAs. A queued -job fetches that immutable event merge SHA directly, verifies it is a two-parent -merge whose parents include the snapshotted head exactly once, and uses its -other parent as the actual base build revision. The snapshotted event base is -retained for audit only, so a difference from the merge's non-head parent is -not a failure. The job never follows a newer mutable PR merge ref. An -unavailable event merge, or one that does not contain exactly one snapshotted -head parent, fails closed. A later PR head change does not retarget an already -queued run: it may execute only its previously trusted event revision if that -revision remains fetchable. To run the new revision, the maintainer must review -it, remove the label, and re-add the desired regression label; cancel the old -run if it is no longer wanted. An existing label does not automatically rerun -the benchmark. +**trust admission for that exact PR revision**. `query-regression` runs the +five routine default cases; `heavy-regression` runs only the high-cardinality +`prom_remote_write_7913` remote-write case. `pull_request: labeled` is the only PR +trigger: the label event snapshots its merge, head, and base SHAs. A queued +job fetches that immutable event merge SHA directly, verifies it is a +two-parent merge whose parents include the snapshotted head exactly once, and +uses its other parent as the actual base build revision. The snapshotted +event base is retained for audit only, so a difference from the merge's +non-head parent is not a failure. The job never follows a newer mutable PR +merge ref. An unavailable event merge, or one that does not contain exactly +one snapshotted head parent, fails closed. A later PR head change does not +retarget an already queued run: it may execute only its previously trusted +event revision if that revision remains fetchable. To run the new revision, +the maintainer must review it, remove the label, and re-add the desired +regression label; cancel the old run if it is no longer wanted. An existing +label does not automatically rerun the benchmark. -Admission does not relax runner hardening or GitHub permissions. Keep -service-account token mounting disabled; do not mount host paths, the Docker -socket, kubeconfig, or long-lived credentials. The runner and cache initializer -use UID/GID 1001, disallow privilege escalation, drop all capabilities, and use -the RuntimeDefault seccomp profile. Keep GitHub tokens least-privilege and -review workflow changes before admission. Where the CNI supports it, restrict -egress to required GitHub Actions, artifact/cache, Rust/crate/toolchain, DNS, -and image-registry endpoints; block unrelated cluster services, private ranges, -and metadata endpoints unless a case requires them. - -### Network routing prerequisite - -Required split routing is an **external environment-specific prerequisite**. The -responsible network operator must route GitHub Actions, GitHub content, -artifact/cache, crates.io, Rust toolchain, and image-registry traffic through -the approved path rather than the VPN where applicable. Neither this repository -nor Kubernetes configures that route. Verify it with the responsible network -operator before any canary. +Admission does not relax runner hardening or GitHub permissions. Keep the ECS +instance free of cloud credentials and long-lived tokens, keep the security +group egress-only, keep GitHub tokens least-privilege, and review workflow +changes before admission. ## Runner image and workflow tools -Build and push the derived runner image; it preserves the official -`/home/runner/run.sh` entrypoint and supplies CI tools needed at runtime. The -image builds `otelgen` from +The runner `Dockerfile` builds `otelgen` from [`WenyXu/otelgen`](https://github.com/WenyXu/otelgen) commit [`863a3f395d062c7322cc1de08a38774b7fdaa6c8`](https://github.com/WenyXu/otelgen/commit/863a3f395d062c7322cc1de08a38774b7fdaa6c8) -so trace cases do not download or compile tools during a benchmark run: +so trace cases do not download or compile tools during a benchmark run. It is +built only as a toolchain factory: `build-ecs-image.py` and +`bootstrap-runner-host.sh` both materialize its contents onto a host, and the +benchmark itself runs host-native. -```bash -docker build \ - --platform linux/amd64 \ - -f .github/runner-scale-sets/query-regression/Dockerfile \ - -t greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest \ - .github/runner-scale-sets/query-regression - -docker push greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner:latest -``` - -Deploy by digest, not mutable tag, by updating both image references in -`values-8-cores.yaml` after a rebuild. Update `RUNNER_IMAGE_DIGEST` and bump -`RUNNER_IMAGE_EPOCH` in `query-regression.yml` at the same time. If the registry -is private, use a dedicated read-only pull secret only as `imagePullSecrets`; -never expose registry credentials to runner containers. -Both digest-pinned init and runner containers use `IfNotPresent`: the immutable -digest makes a cached image safe and avoids adding a registry dependency to every -runner startup. - -The runner optionally imports only the non-sensitive `HTTP_PROXY`, -`HTTPS_PROXY`, and `NO_PROXY` variables from the -`query-regression-runner-local-env` ConfigMap. Manage that ConfigMap locally in -the target namespace; private endpoint configuration must not be committed, and -credentials or secrets must never be placed in a ConfigMap. - -Before builds, the workflow asserts UID/GID 1001 and exact image tool versions: -`libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.30.0`, `Python 3.12.3`, `sccache -0.16.0`, `otelgen` commit `863a3f395d062c7322cc1de08a38774b7fdaa6c8`, -root-owned `rustup 1.29.0`, and the image-baked -`nightly-2026-03-21` Rust toolchain. Rustup, Cargo, and Rustc must resolve from -`/opt/cargo/bin`; the runner cannot write `/opt/rustup` or `/opt/cargo/bin`. -Protobuf well-known includes, including `google/protobuf/any.proto` and -`google/protobuf/empty.proto`, are an image contract and must compile with -`protoc`. +Before builds, the workflow asserts the runner UID/GID (1001 in the ECS image; +overridable via `QUERY_REGRESSION_RUNNER_UID`/`QUERY_REGRESSION_RUNNER_GID`) +and exact tool versions: `libprotoc 3.21.12`, `uv 0.11.26`, `mold 2.40.4`, +`Python 3.14.4`, `sccache 0.16.0`, `otelgen` commit +`863a3f395d062c7322cc1de08a38774b7fdaa6c8`, root-owned `rustup 1.29.0`, and +the image-baked `nightly-2026-03-21` Rust toolchain. `mold` and `python3` +come from apt at image-build time (not Ubuntu 24.04's default 3.12); bump +the Verify pins together with `QUERY_REGRESSION_ECS_IMAGE_ID` when the +image is rebuilt. Rustup, Cargo, and Rustc +must resolve from `/opt/cargo/bin`; the runner cannot write `/opt/rustup` or +`/opt/cargo/bin`. Protobuf well-known includes, including +`google/protobuf/any.proto` and `google/protobuf/empty.proto`, are an image +contract and must compile with `protoc`. `actions-rust-lang/setup-rust-toolchain@v1` is intentionally removed. The -workflow sets its warning-denying mold `RUSTFLAGS` directly, disables automatic -Rustup installation, and performs no runtime toolchain downloads. +workflow sets its warning-denying mold `RUSTFLAGS` directly, disables +automatic Rustup installation, and performs no runtime toolchain downloads. -The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`, or -runtime Rust setup: the image establishes immutable executable state and the PVC -supplies only reusable Cargo data. Do not reintroduce those actions unless the -corresponding cache or image contract changes. +The workflow no longer uses GitHub `rust-cache`, `setup-protoc`, `setup-uv`, +or runtime Rust setup: the image establishes immutable executable state and +each instance's system disk holds only that run's Cargo data. Do not +reintroduce those actions unless the image contract changes. -## Capacity and persistent cache +## Capacity -`values-8-cores.yaml` is normal operation: `minRunners=0`, `maxRunners=1`. -`values-paused.yaml` is the mandatory pause overlay: `minRunners=0`, -`maxRunners=0`. The job uses group `query-regression-persistent-cache-v1`, -`queue: max`, and `cancel-in-progress: false`; admitted jobs queue rather than -replacing older pending jobs. During maintenance, cancel admitted queued runs as -well as pausing ARC. Runner Pods have `activeDeadlineSeconds=12600`. -The runner requests 6 CPU and limits at 8 CPU to preserve `minipc-3` -allocatable-capacity scheduling headroom; do not reset the request to 8 CPU -without revalidating scheduling capacity. +Each run provisions its own ECS instance, so overlapping dispatches proceed +in parallel. There is no workflow `concurrency` group. All Cargo state +(`CARGO_HOME` including registry/git, `CARGO_TARGET_DIR`, sccache, cache +metadata) lives on the 50 GiB system disk and is discarded with the VM. +`RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are image-owned. The runner +sets `RUSTC_WRAPPER=/usr/local/bin/sccache`, +`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=10G`, and +`CARGO_INCREMENTAL=0`. sccache uses its local disk backend and self-evicts +at 10G. Base and candidate builds share the target on that disk; Cargo +fingerprints invalidate source and dependency changes. -The cache claim `query-regression-build-cache` is a nominal 600Gi `local-path` -PVC in `arc-runners`. It is `ReadWriteOnce`; `local-path` uses -WaitForFirstConsumer binding and Delete reclaim behavior, produces a -node-affine local PV, is non-expandable, and the 600Gi request is not a hard -storage quota. The runner's `minipc-3` selector is its only consumer candidate. - -The initializer mounts the PVC root at `/cache`, creates and write-tests these -versioned subpaths as non-root UID/GID 1001, and the runner mounts them as: - -| Persistent state | PVC subpath | Runner mount | -| --- | --- | --- | -| Ephemeral Cargo home | `emptyDir` | `/home/runner/.cargo` | -| Cargo registry data | `cargo-registry-v1` | `/home/runner/.cargo/registry` | -| Cargo Git data | `cargo-git-v1` | `/home/runner/.cargo/git` | -| Cargo target | `query-regression-target-v1` | `/home/runner/query-regression-target` | -| Cache metadata | `meta-v1` | `/home/runner/query-regression-cache-meta` | -| sccache local disk cache | `sccache-v1` | `/home/runner/.cache/sccache` | -| Immutable Rust toolchain | image-owned | `/opt/rustup`, `/opt/cargo/bin` | - -The Pod security context uses UID/GID and `fsGroup` 1001 with -`fsGroupChangePolicy: OnRootMismatch`; no privileged `chown` or raw `hostPath` -is used. `CARGO_HOME` is a per-Pod `emptyDir`; only its nested `registry` and -`git` mounts are persistent. `RUSTUP_HOME=/opt/rustup` and `/opt/cargo/bin` are -image-owned immutable paths, while `CARGO_TARGET_DIR`, cache metadata, and -`SCCACHE_DIR` are persistent absolute paths. The runner sets -`RUSTC_WRAPPER=/usr/local/bin/sccache`, -`SCCACHE_DIR=/home/runner/.cache/sccache`, `SCCACHE_CACHE_SIZE=40G`, and -`CARGO_INCREMENTAL=0`. sccache uses its local PVC disk backend and self-evicts -at 40G; do not add runtime downloads, object storage, or a shared backend. - -The repository's `.cargo/config.toml` remains a trusted per-revision build input. -In contrast, `$CARGO_HOME/config*`, credentials, installed bins, and Cargo -metadata outside the persistent `registry` and `git` data mounts are ephemeral -and cannot survive to another Pod. - -The local disk backend has a one-server constraint. `maxRunners=1` and the -unchanged `query-regression-persistent-cache-v1` workflow concurrency group -serialize runs; do not increase runner capacity or relax that serialization -while this backend is in use. Base and candidate builds share the target; Cargo -fingerprints invalidate source and dependency changes. The workflow records the -sccache version and relevant environment in the target ABI marker, starts and -zeros sccache after cache and toolchain checks, shows initial/base/candidate -statistics, and resets statistics between base and candidate builds. - -### Disk preflight and cleanup contract - -Before applying or unpausing, verify the backing filesystem on `minipc-3` has -at least 900GiB free. The current local-path provisioner source is -`/opt/local-path-provisioner`; measure the filesystem containing it: - -```bash -df -PB1G /opt/local-path-provisioner -``` - -The workflow reports `du`, `df -P`, human-readable free space, and inode -availability before builds and in an always-run report. Its cleanup is narrow -and non-destructive: - -- warn at target size 400GiB; at 450GiB clear only the complete target root; -- warn at Cargo registry-plus-Git data size 60GiB; at 80GiB remove only - `registry/src` and `git/checkouts`, then abort if that persistent data remains - at least 80GiB; -- below 300GiB backing free space, clear the complete target root first, - remeasure, then remove only those Cargo extracted trees and checkouts; abort - if free space is still below 300GiB; -- never automatically remove Cargo registry cache/index, Git database, the - image-owned Cargo bin or Rustup toolchain, cache metadata, the self-evicting - sccache directory, or the PVC. - -The target clear uses fixed absolute roots and removes all entries, including -dotfiles. After migration validation, remove obsolete `cargo-home-v1` and -`rustup-home-v1` only in explicit maintenance while ARC is 0/0 and no runner Pod -exists; they are not mounted by the current configuration. - -## Deploy and pause safely - -First verify the configured external network route and the disk preflight. Apply the PVC; while -the scale set is paused, it is expected to remain `Pending` because -WaitForFirstConsumer has no scheduled runner: - -```bash -kubectl apply --dry-run=server \ - -f .github/runner-scale-sets/query-regression/cache-pvc.yaml -kubectl apply -f .github/runner-scale-sets/query-regression/cache-pvc.yaml -``` - -Render normal and paused configurations. Normal values are always first; the -pause overlay is always last: - -```bash -helm template perf-regression-8-cores \ - oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \ - --namespace arc-runners --version 0.14.2 \ - --set controllerServiceAccount.name=arc-gha-rs-controller \ - --set controllerServiceAccount.namespace=arc-systems \ - -f .github/runner-scale-sets/query-regression/values-8-cores.yaml - -helm template perf-regression-8-cores \ - oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \ - --namespace arc-runners --version 0.14.2 \ - --set controllerServiceAccount.name=arc-gha-rs-controller \ - --set controllerServiceAccount.namespace=arc-systems \ - -f .github/runner-scale-sets/query-regression/values-8-cores.yaml \ - -f .github/runner-scale-sets/query-regression/values-paused.yaml -``` - -The **first post-merge Helm deployment must reconcile the release in paused -mode**. Keep the pause overlay last: - -```bash -# First post-merge deployment and every return to paused mode: 0/0. -helm upgrade --install perf-regression-8-cores \ - oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \ - --namespace arc-runners --create-namespace --version 0.14.2 \ - --reset-values --wait \ - -f .github/runner-scale-sets/query-regression/values-8-cores.yaml \ - -f .github/runner-scale-sets/query-regression/values-paused.yaml - -# Expect 0/0 and no runner resources before considering normal mode. -kubectl -n arc-runners get autoscalingrunnerset perf-regression-8-cores \ - -o jsonpath='{.spec.minRunners}{"/"}{.spec.maxRunners}{"\n"}' -kubectl -n arc-runners get ephemeralrunners,pods \ - -l actions.github.com/scale-set-name=perf-regression-8-cores -``` - -Only after that verification and separate explicit approval, apply normal 0/1 -operation without the pause overlay: - -```bash -helm upgrade --install perf-regression-8-cores \ - oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \ - --namespace arc-runners --create-namespace --version 0.14.2 \ - --reset-values --wait \ - -f .github/runner-scale-sets/query-regression/values-8-cores.yaml -``` - -Do not use bare `helm rollback`, `--atomic`, or `--reuse-values`: a stored -revision can restore nonzero runner capacity. Inspect rendered manifests for -capacity, the `minipc-3` selector, cache claim and mounts, initializer, -security context, and resources. After approved normal mode receives its first -canary, the PVC binds to `minipc-3`. - -For local-PV node loss, cache recovery is intentionally disposable: return to -0/0, recreate the PVC on a healthy node, cold-fill it, and run a new canary. - -## Canary and rollback - -With explicit approval, run two identical `workflow_dispatch` canaries on -`perf-regression-8-cores`, using immutable full base and candidate commit SHAs -and `cargo_profile=nightly`. The first is the cold fill; the second verifies warm -reuse. Record the workflow's base/candidate build elapsed logs and cache -ABI-marker output, initial/base/candidate sccache statistics, and cache report. -Confirm the image tool contract (root-owned Rustup/Cargo paths, baked nightly, -and non-writable `/opt` roots) and that the ephemeral Cargo home contains only -the mounted registry/Git data before Cargo creates per-Pod state. -Obtain dependency and tool network byte counters from the configured -environment counter source, filtered to `minipc-3` and the dependency/tool -destinations. - -Accept the canary only when all of the following hold: - -- exactly one runner Pod runs on `minipc-3`, and both jobs use the same bound PV; -- UID/GID 1001 cache mounts are writable; the warm run does not invalidate the - target or bulk-redownload crates or toolchains; sccache reports separate base - and candidate build statistics without server or cache-path errors; immutable - Rustup/Cargo roots remain non-writable and only registry/Git data persists; -- warm base build time is at most 50% of cold base build time; -- warm dependency/tool network bytes are at most 10% of cold fill bytes; -- cache sizes remain below soft watermarks, node free space remains at least - 300GiB, and the benchmark is correct without TLS EOFs or timeouts; -- the configured environment counter source confirms this traffic is outside VPN - accounting. - -Immediately return to 0/0 after either canary unless ongoing normal operation -has been explicitly approved; return immediately on any traffic, cache, disk, -TLS, or correctness failure. To roll back, use the paused Helm upgrade above, -or another explicit `helm upgrade --install` with known-good values followed by -`values-paused.yaml`, `--reset-values`, and `--wait`. Do not delete the PVC -automatically; preserve it for diagnosis unless intentionally discarding cache. +The workflow reports `du` / `df` in telemetry. It does not try to reclaim +space across runs: a cold 50 GiB disk that fills up fails the build. ## Future optional phases -The current phase uses a digest-pinned image with sccache 0.16.0 and no shared -cache service. Optional follow-ups are an image additionally seeded with the -exact Rust toolchain and `cargo fetch --locked`; or an internal read/write -sccache backend or Cargo/Git mirror. Evaluate them only if persistent PVC reuse -is insufficient. +The current phase uses a materialized runner toolchain with sccache 0.16.0 and +no shared cache service. Optional follow-ups are an image additionally seeded +with `cargo fetch --locked` results, or an internal read/write sccache +backend or Cargo/Git mirror. Evaluate them only if cold compile time on the +system disk becomes the bottleneck. diff --git a/.github/runner-scale-sets/query-regression/cache-pvc.yaml b/.github/runner-scale-sets/query-regression/cache-pvc.yaml deleted file mode 100644 index 692f3a2d6e..0000000000 --- a/.github/runner-scale-sets/query-regression/cache-pvc.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: query-regression-build-cache - namespace: arc-runners - labels: - app.kubernetes.io/part-of: perf-regression-8-cores -spec: - volumeMode: Filesystem - accessModes: - - ReadWriteOnce - storageClassName: local-path - resources: - requests: - storage: 600Gi diff --git a/.github/runner-scale-sets/query-regression/ecs-image/bootstrap-runner-host.sh b/.github/runner-scale-sets/query-regression/ecs-image/bootstrap-runner-host.sh new file mode 100644 index 0000000000..883bbede09 --- /dev/null +++ b/.github/runner-scale-sets/query-regression/ecs-image/bootstrap-runner-host.sh @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Bootstrap an Ubuntu 24.04 host as a query-regression self-hosted runner. +# +# This is the host-native equivalent of the ECS image contract: it +# installs docker-ce from Docker's official repository, builds the runner +# image from the sibling Dockerfile (the single source of the tool contract), +# materializes the toolchain onto the host, and optionally registers the +# runner and installs its systemd service. +# +# Large state lives under --data-root (a data disk): docker's data-root, the +# runner home, and the rustup/cargo toolchain roots, symlinked back to the +# hard-coded contract paths (/home/runner, /opt/rustup, /opt/cargo). +# +# The script is idempotent: re-running refreshes the toolchain in place, and +# the runner registration (.runner/.credentials) and _work survive. A live +# runner service is stopped first and restarted at the end. +# +# Usage: +# sudo bash bootstrap-runner-host.sh --repo-dir /path/to/greptimedb +# sudo RUNNER_TOKEN= bash bootstrap-runner-host.sh --repo-dir . \ +# --register --repo-url https://github.com// +# +# Options: +# --repo-dir PATH greptimedb checkout containing the runner Dockerfile (required) +# --data-root PATH data disk mount point (default: /data) +# --uid / --gid N runner user id (default: 3141; the ECS image contract is 1001) +# --register also register the runner and install the systemd service +# --repo-url URL repository URL for registration (required with --register) +# --runner-name N runner name (default: qreg-host) +# --labels L runner labels (default: perf-regression-8-cores) +# +# With --register, provide a fresh registration token via RUNNER_TOKEN. + +set -euo pipefail + +REPO_DIR="" +DATA_ROOT="/data" +RUNNER_UID="3141" +RUNNER_GID="3141" +REGISTER="false" +REPO_URL="" +RUNNER_NAME="qreg-host" +RUNNER_LABELS="perf-regression-8-cores" +IMAGE_TAG="qreg-runner:manual" + +while [[ $# -gt 0 ]]; do + case "$1" in + --repo-dir) REPO_DIR="$2"; shift 2 ;; + --data-root) DATA_ROOT="$2"; shift 2 ;; + --uid) RUNNER_UID="$2"; shift 2 ;; + --gid) RUNNER_GID="$2"; shift 2 ;; + --register) REGISTER="true"; shift ;; + --repo-url) REPO_URL="$2"; shift 2 ;; + --runner-name) RUNNER_NAME="$2"; shift 2 ;; + --labels) RUNNER_LABELS="$2"; shift 2 ;; + -h | --help) sed -n '17,40p' "$0"; exit 0 ;; + *) echo "Unknown argument: $1" >&2; exit 2 ;; + esac +done + +[[ $EUID -eq 0 ]] || { echo "Run as root, e.g. sudo bash $0 ..." >&2; exit 1; } +[[ -n "${REPO_DIR}" ]] || { echo "--repo-dir is required" >&2; exit 2; } +DOCKERFILE="${REPO_DIR}/.github/runner-scale-sets/query-regression/Dockerfile" +[[ -f "${DOCKERFILE}" ]] || { echo "Dockerfile not found at ${DOCKERFILE}" >&2; exit 1; } +if [[ "${REGISTER}" == "true" ]]; then + [[ -n "${REPO_URL}" ]] || { echo "--repo-url is required with --register" >&2; exit 2; } + [[ -n "${RUNNER_TOKEN:-}" ]] || { echo "RUNNER_TOKEN env is required with --register" >&2; exit 2; } +fi + +step() { printf '\n==> %s\n' "$*"; } + +# A re-run refreshes files under the (possibly live) runner service, so stop +# it first and restart it at the end (unless --register re-creates it). +RUNNER_SERVICE="$(systemctl list-units --type=service --all --no-legend 'actions.runner.*' 2>/dev/null | awk '{print $1}' | head -n 1 || true)" +RUNNER_WAS_ACTIVE="false" +if [[ -n "${RUNNER_SERVICE}" ]] && systemctl is-active --quiet "${RUNNER_SERVICE}"; then + step "Stop running runner service ${RUNNER_SERVICE}" + systemctl stop "${RUNNER_SERVICE}" + RUNNER_WAS_ACTIVE="true" +fi + +step "Install docker-ce from Docker's official repository" +for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do + apt-get remove -y "${pkg}" 2>/dev/null || true +done +apt-get update +apt-get install -y ca-certificates curl +install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc +chmod a+r /etc/apt/keyrings/docker.asc +# shellcheck disable=SC1091 +. /etc/os-release +tee /etc/apt/sources.list.d/docker.sources > /dev/null <&2 + exit 1 +fi +echo "{ \"data-root\": \"${DATA_ROOT}/docker\" }" > /etc/docker/daemon.json +# containerd keeps its own content store (where build-time image data lands); +# move it off the system disk too. +mkdir -p /etc/containerd +containerd config default | sed "s|root = \"/var/lib/containerd\"|root = \"${DATA_ROOT}/containerd\"|" > /etc/containerd/config.toml +systemctl start containerd docker +docker info | grep "Docker Root Dir" + +step "Build the runner image" +docker build --platform linux/amd64 -f "${DOCKERFILE}" -t "${IMAGE_TAG}" "${REPO_DIR}" + +step "Materialize the toolchain onto the host" +mkdir -p "${DATA_ROOT}/opt" "${DATA_ROOT}/runner" /opt /home +container="$(docker create "${IMAGE_TAG}")" +trap 'docker rm -f "${container}" >/dev/null 2>&1 || true' EXIT +# The `/.` suffix copies directory *contents*, so re-running over an existing +# materialization refreshes it in place instead of nesting runner/runner. +# Files absent from the image (runner registration, _work) are preserved. +docker cp "${container}:/home/runner/." "${DATA_ROOT}/runner" +docker cp "${container}:/opt/rustup/." "${DATA_ROOT}/opt/rustup" +docker cp "${container}:/opt/cargo/." "${DATA_ROOT}/opt/cargo" +for tool in uv uvx otelgen sccache; do + docker cp "${container}:/usr/local/bin/${tool}" "/usr/local/bin/${tool}" +done +docker rm "${container}" > /dev/null +trap - EXIT +ln -sfn "${DATA_ROOT}/runner" /home/runner +ln -sfn "${DATA_ROOT}/opt/rustup" /opt/rustup +ln -sfn "${DATA_ROOT}/opt/cargo" /opt/cargo +docker image rm "${IMAGE_TAG}" > /dev/null + +step "Install system packages" +apt-get install -y --no-install-recommends \ + build-essential clang cmake git gzip jq libprotobuf-dev libssl-dev mold \ + openssh-client pkg-config protobuf-compiler python3 sudo tar unzip wget xz-utils zip zstd + +step "Create runner user (${RUNNER_UID}:${RUNNER_GID}) and environment" +getent group "${RUNNER_GID}" > /dev/null || groupadd -g "${RUNNER_GID}" runner +id -u runner > /dev/null 2>&1 || useradd -u "${RUNNER_UID}" -g "${RUNNER_GID}" -d /home/runner -s /bin/bash runner +chown -R "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner" +echo 'PATH=/opt/cargo/bin:/usr/local/bin:/usr/bin:/bin' > "${DATA_ROOT}/runner/.env" +chown "${RUNNER_UID}:${RUNNER_GID}" "${DATA_ROOT}/runner/.env" + +if [[ "${REGISTER}" == "true" ]]; then + step "Register runner ${RUNNER_NAME} (labels: ${RUNNER_LABELS})" + if [[ -n "${RUNNER_SERVICE}" ]]; then + (cd /home/runner && ./svc.sh uninstall) || true + fi + runuser -u runner -- bash -c "cd /home/runner && HOME=/home/runner ./config.sh \ + --url '${REPO_URL}' --token '${RUNNER_TOKEN}' --name '${RUNNER_NAME}' \ + --labels '${RUNNER_LABELS}' --unattended --replace --disableupdate" + + step "Install and start the runner service" + cd /home/runner + ./svc.sh install runner + ./svc.sh start + ./svc.sh status +else + step "Skipping registration (pass --register --repo-url ... with RUNNER_TOKEN to enable)" + if [[ "${RUNNER_WAS_ACTIVE}" == "true" ]]; then + step "Restart runner service ${RUNNER_SERVICE}" + systemctl start "${RUNNER_SERVICE}" + systemctl --no-pager status "${RUNNER_SERVICE}" || true + fi +fi + +step "Done" +echo "Runner home: /home/runner -> ${DATA_ROOT}/runner" +echo "Before each workflow run on this persistent host, clean transient cargo state:" +echo " sudo rm -f /home/runner/.cargo/.package-cache" +echo " sudo rm -rf /home/runner/.cargo/.global-cache" diff --git a/.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py b/.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py new file mode 100644 index 0000000000..7031917f91 --- /dev/null +++ b/.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py @@ -0,0 +1,392 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# PEP 723 inline metadata (see .github/scripts/aliyun-ecs-runner-provision.py +# for the convention). +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "alibabacloud_ecs20140526>=4.1.0,<6", +# "alibabacloud_tea_openapi>=0.3.12,<1", +# ] +# /// + +"""Build the query-regression ECS custom image (manual ops tool). + +Boots a temporary pay-as-you-go ECS instance from a public Ubuntu 24.04 image, +builds the existing runner container image (the Dockerfile in the parent +directory remains the single source of the tool contract), materializes the +tool directories onto the host filesystem so the workflow's "Verify runner +image tools" step holds unchanged, installs the ephemeral-runner systemd unit, +and snapshots the result as a custom image. The temporary instance is deleted +afterwards. + +Sentinel polling reads the instance's serial console output +(GetInstanceConsoleOutput) and looks for marker lines the user data writes to +/dev/console. This has no in-guest agent dependency. + +Usage: + uv run .github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py \ + --region-id cn-hangzhou --vswitch-id vsw-... --security-group-id sg-... \ + --base-image-id ubuntu_24_04_x64_20G_alibase_*.vhd +""" + +from __future__ import annotations + +import argparse +import base64 +import os +import time +from pathlib import Path + +ASSETS_DIR = Path(__file__).resolve().parent +DONE_MARKER = "QREG_IMAGE_BUILD_DONE" +FAILED_MARKER = "QREG_IMAGE_BUILD_FAILED" +POLL_INTERVAL_SECONDS = 15 +CONSOLE_POLL_INTERVAL_SECONDS = 30 +BUILD_TIMEOUT_SECONDS = 60 * 60 + +# Same apt package contract as the runner Dockerfile; the base +# actions-runner image is Ubuntu 24.04, so an Ubuntu 24.04 host resolves the +# same tool versions (protoc 3.21.12, mold 2.40.4, Python 3.14.4). +# Docker itself comes from Docker's official repository (docker-ce), not the +# distribution-packaged docker.io. +APT_PACKAGES = [ + "build-essential", + "ca-certificates", + "clang", + "cmake", + "curl", + "git", + "gpg", + "gzip", + "jq", + "libprotobuf-dev", + "libssl-dev", + "mold", + "openssh-client", + "pkg-config", + "protobuf-compiler", + "python3", + "sudo", + "tar", + "unzip", + "wget", + "xz-utils", + "zip", + "zstd", +] + +DOCKER_CE_PACKAGES = "docker-ce docker-ce-cli containerd.io docker-buildx-plugin" + + +def render_user_data(dockerfile: str, start_runner: str, unit: str) -> str: + dockerfile_b64 = base64.b64encode(dockerfile.encode()).decode() + start_runner_b64 = base64.b64encode(start_runner.encode()).decode() + unit_b64 = base64.b64encode(unit.encode()).decode() + packages = " ".join(APT_PACKAGES) + return f"""#!/bin/bash +set -euo pipefail +trap 'echo "{FAILED_MARKER} at line $LINENO" > /dev/console' ERR +# Stream the full build log to the serial console so the poller (and anyone +# watching GetInstanceConsoleOutput) sees real progress, not a silent login +# prompt for the whole build. +exec > >(tee -a /dev/console) 2>&1 + +apt-get update +DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {packages} + +# Docker from the official repository, not the distribution-packaged docker.io. +install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg +chmod a+r /etc/apt/keyrings/docker.gpg +. /etc/os-release +echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu ${{VERSION_CODENAME}} stable" \ + > /etc/apt/sources.list.d/docker.list +apt-get update +DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends {DOCKER_CE_PACKAGES} + +base64 -d > /tmp/Dockerfile <<'EOF' +{dockerfile_b64} +EOF +mkdir -p /tmp/image-context +docker build --platform linux/amd64 -f /tmp/Dockerfile -t qreg-runner:local /tmp/image-context + +# Materialize the tool contract onto the host filesystem. +container="$(docker create qreg-runner:local)" +trap 'docker rm -f "${{container}}" >/dev/null 2>&1 || true' EXIT +docker cp "${{container}}:/opt/rustup" /opt/rustup +docker cp "${{container}}:/opt/cargo" /opt/cargo +for tool in uv uvx otelgen sccache; do + docker cp "${{container}}:/usr/local/bin/${{tool}}" "/usr/local/bin/${{tool}}" +done +docker cp "${{container}}:/home/runner" /home/runner + +# Runner identity mirrors the container image (UID/GID 1001). +groupadd --gid 1001 runner 2>/dev/null || true +useradd --uid 1001 --gid 1001 --home-dir /home/runner --shell /bin/bash runner 2>/dev/null || true +chown -R 1001:1001 /home/runner + +mkdir -p /opt/ephemeral-github-runner +base64 -d > /opt/ephemeral-github-runner/start-runner.sh <<'EOF' +{start_runner_b64} +EOF +chmod 0755 /opt/ephemeral-github-runner/start-runner.sh +base64 -d > /etc/systemd/system/ephemeral-github-runner.service <<'EOF' +{unit_b64} +EOF +systemctl daemon-reload +systemctl enable ephemeral-github-runner.service + +# Keep the image free of the build-time docker state (also shrinks the +# snapshot: buildkit cache is several GB). +docker rm -f "${{container}}" >/dev/null +docker system prune -af >/dev/null +trap - EXIT + +echo "{DONE_MARKER}" > /dev/console +""" + + +def make_ecs_client(region_id: str): + from alibabacloud_ecs20140526.client import Client as EcsClient + from alibabacloud_tea_openapi.models import Config as OpenApiConfig + + return EcsClient( + OpenApiConfig( + access_key_id=os.environ["ALIBABA_CLOUD_ACCESS_KEY_ID"], + access_key_secret=os.environ["ALIBABA_CLOUD_ACCESS_KEY_SECRET"], + region_id=region_id, + endpoint=f"ecs.{region_id}.aliyuncs.com", + ) + ) + + +def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None: + from alibabacloud_ecs20140526 import models as ecs_models + + import json + + while time.monotonic() < deadline: + response = call_api_with_retry( + lambda: client.describe_instances( + ecs_models.DescribeInstancesRequest( + region_id=region_id, instance_ids=json.dumps([instance_id]) + ) + ), + "DescribeInstances", + ) + instances = response.body.instances.instance + if instances and instances[0].status == wanted: + return + time.sleep(POLL_INTERVAL_SECONDS) + raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time") + + +# Aliyun error codes worth retrying: throttling and server-side faults. Client +# errors (4xx: permissions, bad parameters) are configuration problems and must +# fail fast instead of being retried. +TRANSIENT_ERROR_CODES = {"Throttling", "Throttling.User", "InternalError", "ServiceUnavailable"} + + +def is_transient(error: Exception) -> bool: + # UnretryableException comes from the darabonba network layer: connection + # reset, read timeout, DNS blip. + if type(error).__name__ == "UnretryableException": + return True + code = getattr(error, "code", "") or "" + status = getattr(error, "statusCode", None) + return code in TRANSIENT_ERROR_CODES or (isinstance(status, int) and status >= 500) + + +def call_api_with_retry(fn, description: str, attempts: int = 5): + """Retry transient API/network failures; the build is too long to die on a blip. + + Only call this with idempotent or safely-repeatable requests (reads, + StopInstance, CreateImage). Never with RunInstances: if the request + succeeded but the response was lost, a retry double-creates instances. + """ + for attempt in range(1, attempts + 1): + try: + return fn() + except Exception as error: # noqa: BLE001 + if attempt == attempts or not is_transient(error): + raise + print(f"{description} failed (attempt {attempt}/{attempts}): {error}", flush=True) + time.sleep(POLL_INTERVAL_SECONDS) + raise RuntimeError("unreachable: retry loop exited without returning") + + +def read_console_output(client, region_id: str, instance_id: str) -> str: + """Fetch the instance's serial console output; no in-guest agent needed.""" + from alibabacloud_ecs20140526 import models as ecs_models + + response = call_api_with_retry( + lambda: client.get_instance_console_output( + ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id) + ), + "GetInstanceConsoleOutput", + ) + return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID")) + parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID")) + parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID")) + parser.add_argument("--base-image-id", default=os.environ.get("ALIYUN_ECS_BASE_IMAGE_ID")) + parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID")) + parser.add_argument("--instance-type", default="ecs.g7.xlarge") + parser.add_argument("--image-name", default=None, help="Defaults to a timestamped name.") + args = parser.parse_args() + + for name in ("region_id", "vswitch_id", "security_group_id", "base_image_id"): + if not getattr(args, name): + raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}") + + from alibabacloud_ecs20140526 import models as ecs_models + + client = make_ecs_client(args.region_id) + image_name = args.image_name or time.strftime( + "greptimedb-query-regression-runner-%Y%m%d%H%M%S", time.gmtime() + ) + + user_data = base64.b64encode( + render_user_data( + (ASSETS_DIR.parent / "Dockerfile").read_text(), + (ASSETS_DIR / "start-runner.sh").read_text(), + (ASSETS_DIR / "ephemeral-github-runner.service").read_text(), + ).encode() + ).decode() + + instance_id = None + try: + response = client.run_instances( + ecs_models.RunInstancesRequest( + region_id=args.region_id, + image_id=args.base_image_id, + resource_group_id=args.resource_group_id, + instance_type=args.instance_type, + v_switch_id=args.vswitch_id, + security_group_id=args.security_group_id, + instance_name=f"build-{image_name}", + description="Temporary builder for the query-regression ECS image", + amount=1, + instance_charge_type="PostPaid", + spot_strategy="NoSpot", + internet_charge_type="PayByTraffic", + internet_max_bandwidth_out=100, + # Peak usage is ~18G (OS+apt, docker image, and the materialized + # toolchain coexist briefly): deliberately tight, and a smaller + # disk makes the image snapshot faster. Instances created from + # the image get a larger system disk from the provision side. + system_disk=ecs_models.RunInstancesRequestSystemDisk( + category="cloud_essd", size="20" + ), + user_data=user_data, + tag=[ + ecs_models.RunInstancesRequestTag(key="managed-by", value="query-regression-ci"), + ecs_models.RunInstancesRequestTag(key="role", value="image-builder"), + ], + ) + ) + instance_id = response.body.instance_id_sets.instance_id_set[0] + print(f"Builder instance: {instance_id}", flush=True) + wait_for_instance_status(client, args.region_id, instance_id, "Running", time.monotonic() + 10 * 60) + + deadline = time.monotonic() + BUILD_TIMEOUT_SECONDS + while time.monotonic() < deadline: + console = read_console_output(client, args.region_id, instance_id) + if DONE_MARKER in console: + break + if FAILED_MARKER in console: + tail = "\n".join(console.splitlines()[-20:]) + raise RuntimeError(f"Image build failed on the builder; console tail:\n{tail}") + lines = console.splitlines() + print(f"Build in progress; last console line: {lines[-1] if lines else '(none yet)'}", flush=True) + time.sleep(CONSOLE_POLL_INTERVAL_SECONDS) + else: + raise TimeoutError("Image build did not finish in time") + + print("Stopping builder before image creation", flush=True) + try: + call_api_with_retry( + lambda: client.stop_instance( + ecs_models.StopInstanceRequest( + instance_id=instance_id, + # The instance is deleted right after the snapshot, but + # there is no reason to keep billing vCPU during the stop. + stopped_mode="StopCharging", + ) + ), + "StopInstance", + ) + except Exception as error: # noqa: BLE001 + # Instance families with local disks do not support StopCharging. + print(f"StopCharging unavailable ({error}); stopping with default mode", flush=True) + call_api_with_retry( + lambda: client.stop_instance(ecs_models.StopInstanceRequest(instance_id=instance_id)), + "StopInstance", + ) + wait_for_instance_status(client, args.region_id, instance_id, "Stopped", time.monotonic() + 10 * 60) + + image = call_api_with_retry( + lambda: client.create_image( + ecs_models.CreateImageRequest( + region_id=args.region_id, instance_id=instance_id, image_name=image_name + ) + ), + "CreateImage", + ) + image_id = image.body.image_id + deadline = time.monotonic() + 60 * 60 + while time.monotonic() < deadline: + description = call_api_with_retry( + lambda: client.describe_images( + ecs_models.DescribeImagesRequest(region_id=args.region_id, image_id=image_id) + ), + "DescribeImages", + ) + images = description.body.images.image + if images: + status = images[0].status + print(f"Image {image_id} status: {status} (progress {images[0].progress})", flush=True) + if status == "Available": + break + time.sleep(POLL_INTERVAL_SECONDS) + else: + raise TimeoutError( + f"Image {image_id} did not become Available in time. The snapshot " + f"continues server-side: check its status in the console and reuse it " + f"once Available instead of rebuilding." + ) + + print(f"Custom image ready: {image_id} ({image_name})", flush=True) + print(f"Set the repo variable QUERY_REGRESSION_ECS_IMAGE_ID={image_id}", flush=True) + return 0 + finally: + if instance_id: + try: + client.delete_instance( + ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True) + ) + print(f"Deleted builder instance {instance_id}", flush=True) + except Exception as error: # noqa: BLE001 + print(f"Failed to delete builder instance {instance_id}: {error}", flush=True) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/runner-scale-sets/query-regression/ecs-image/ephemeral-github-runner.service b/.github/runner-scale-sets/query-regression/ecs-image/ephemeral-github-runner.service new file mode 100644 index 0000000000..a1cbe6a450 --- /dev/null +++ b/.github/runner-scale-sets/query-regression/ecs-image/ephemeral-github-runner.service @@ -0,0 +1,45 @@ +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[Unit] +Description=Ephemeral GitHub Actions runner +After=network-online.target +Wants=network-online.target +# cloud-final runs the provision user data, which writes +# /etc/ephemeral-github-runner.env and the drop-ins below. Ordering after it +# makes the runner come up configured on first boot, no restart needed. +# (cloud-init's `systemctl restart` in user data remains as the bridge for +# images built before this ordering existed.) +After=cloud-final.service + +[Service] +Type=simple +# The toolchain lives in the image at /opt/cargo/bin (the Dockerfile's ENV +# PATH does not survive materialization onto the host). Publish it here so +# every runner job inherits it. /etc/ephemeral-github-runner.env currently +# carries the same PATH line as a bridge for images built before this +# directive existed; EnvironmentFile is applied after Environment=, and the +# two values are kept identical, so there is no conflict. +Environment=PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +EnvironmentFile=/etc/ephemeral-github-runner.env +ExecStart=/opt/ephemeral-github-runner/start-runner.sh +Restart=no +# If the kernel OOM killer kills a job child (rustc/cargo), keep the runner +# service alive so the job is reported as a normal failure with logs instead +# of the runner vanishing and GitHub reporting a cancellation. cloud-init's +# oom.conf drop-in carries the same directive for images built before this. +OOMPolicy=continue + +[Install] +WantedBy=multi-user.target diff --git a/.github/runner-scale-sets/query-regression/ecs-image/start-runner.sh b/.github/runner-scale-sets/query-regression/ecs-image/start-runner.sh new file mode 100644 index 0000000000..a6aff87ca2 --- /dev/null +++ b/.github/runner-scale-sets/query-regression/ecs-image/start-runner.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Configure and start the ephemeral GitHub Actions runner on an ECS instance. +# Baked into the custom image at /opt/ephemeral-github-runner/ by +# build-ecs-image.py; the per-run environment is written by cloud-init to +# /etc/ephemeral-github-runner.env. +set -euo pipefail + +# shellcheck disable=SC1091 +source /etc/ephemeral-github-runner.env + +export HOME=/home/runner +cd /home/runner + +if [[ ! -f /home/runner/.runner ]]; then + runuser -u runner -- ./config.sh \ + --url "${REPO_URL}" \ + --token "${RUNNER_TOKEN}" \ + --name "${RUNNER_NAME}" \ + --labels "${RUNNER_LABELS}" \ + --ephemeral --unattended --replace --disableupdate +fi + +exec runuser -u runner -- ./run.sh diff --git a/.github/runner-scale-sets/query-regression/values-8-cores.yaml b/.github/runner-scale-sets/query-regression/values-8-cores.yaml deleted file mode 100644 index cff1322680..0000000000 --- a/.github/runner-scale-sets/query-regression/values-8-cores.yaml +++ /dev/null @@ -1,146 +0,0 @@ -githubConfigUrl: "https://github.com/GreptimeTeam/greptimedb" -githubConfigSecret: greptimedb-arc-github-app - -runnerScaleSetName: "perf-regression-8-cores" -minRunners: 0 -maxRunners: 1 - -template: - spec: - automountServiceAccountToken: false - activeDeadlineSeconds: 12600 - nodeSelector: - kubernetes.io/hostname: minipc-3 - securityContext: - runAsNonRoot: true - runAsUser: 1001 - runAsGroup: 1001 - fsGroup: 1001 - fsGroupChangePolicy: OnRootMismatch - seccompProfile: - type: RuntimeDefault - volumes: - - name: cargo-home - emptyDir: {} - - name: build-cache - persistentVolumeClaim: - claimName: query-regression-build-cache - initContainers: - - name: initialize-build-cache - image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11 - imagePullPolicy: IfNotPresent - command: - - /bin/sh - - -ec - - | - umask 0002 - for directory in cargo-registry-v1 cargo-git-v1 query-regression-target-v1 meta-v1 sccache-v1; do - cache_directory="/cache/${directory}" - mkdir -p "${cache_directory}" - test -w "${cache_directory}" - probe_file="${cache_directory}/.write-probe" - : > "${probe_file}" - test -f "${probe_file}" - rm "${probe_file}" - done - securityContext: - runAsNonRoot: true - runAsUser: 1001 - runAsGroup: 1001 - allowPrivilegeEscalation: false - capabilities: - drop: - - ALL - resources: - requests: - cpu: 100m - memory: 128Mi - ephemeral-storage: 1Gi - limits: - cpu: 500m - memory: 512Mi - ephemeral-storage: 1Gi - volumeMounts: - - name: build-cache - mountPath: /cache - containers: - - name: runner - image: greptime-registry.cn-hangzhou.cr.aliyuncs.com/greptime/greptimedb-query-regression-runner@sha256:e713b294e23b7e15184e558866c90025e59930033e72c97650dbc7f1ca022d11 - imagePullPolicy: IfNotPresent - command: ["/home/runner/run.sh"] - env: - - name: CARGO_HOME - value: /home/runner/.cargo - - name: HTTP_PROXY - valueFrom: - configMapKeyRef: - name: query-regression-runner-local-env - key: HTTP_PROXY - optional: true - - name: HTTPS_PROXY - valueFrom: - configMapKeyRef: - name: query-regression-runner-local-env - key: HTTPS_PROXY - optional: true - - name: NO_PROXY - valueFrom: - configMapKeyRef: - name: query-regression-runner-local-env - key: NO_PROXY - optional: true - - name: UV_CACHE_DIR - value: /home/runner/.cargo/uv-cache - - name: RUSTUP_HOME - value: /opt/rustup - - name: RUSTUP_TOOLCHAIN - value: nightly-2026-03-21 - - name: RUSTUP_AUTO_INSTALL - value: "0" - - name: CARGO_TARGET_DIR - value: /home/runner/query-regression-target - - name: QUERY_REGRESSION_CACHE_META - value: /home/runner/query-regression-cache-meta - - name: RUSTC_WRAPPER - value: /usr/local/bin/sccache - - name: SCCACHE_DIR - value: /home/runner/.cache/sccache - - name: SCCACHE_CACHE_SIZE - value: 40G - - name: CARGO_INCREMENTAL - value: "0" - securityContext: - runAsNonRoot: true - runAsUser: 1001 - runAsGroup: 1001 - allowPrivilegeEscalation: false - capabilities: - drop: - - ALL - resources: - requests: - cpu: "6" - memory: 24Gi - ephemeral-storage: 80Gi - limits: - cpu: "8" - memory: 24Gi - ephemeral-storage: 80Gi - volumeMounts: - - name: cargo-home - mountPath: /home/runner/.cargo - - name: build-cache - mountPath: /home/runner/.cargo/registry - subPath: cargo-registry-v1 - - name: build-cache - mountPath: /home/runner/.cargo/git - subPath: cargo-git-v1 - - name: build-cache - mountPath: /home/runner/query-regression-target - subPath: query-regression-target-v1 - - name: build-cache - mountPath: /home/runner/query-regression-cache-meta - subPath: meta-v1 - - name: build-cache - mountPath: /home/runner/.cache/sccache - subPath: sccache-v1 diff --git a/.github/runner-scale-sets/query-regression/values-paused.yaml b/.github/runner-scale-sets/query-regression/values-paused.yaml deleted file mode 100644 index fe734ef761..0000000000 --- a/.github/runner-scale-sets/query-regression/values-paused.yaml +++ /dev/null @@ -1,2 +0,0 @@ -minRunners: 0 -maxRunners: 0 diff --git a/.github/scripts/aliyun-ecs-runner-provision.py b/.github/scripts/aliyun-ecs-runner-provision.py new file mode 100644 index 0000000000..307ee336eb --- /dev/null +++ b/.github/scripts/aliyun-ecs-runner-provision.py @@ -0,0 +1,488 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This is the first `.github/scripts/` user of PEP 723 inline script metadata: +# `uv run` installs the pinned Aliyun SDK before executing. The SDK import stays +# lazy (inside `make_ecs_client`) so unit tests can import this module with a +# plain stdlib interpreter. +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# # Pin exactly after the first live run against the Aliyun API. +# "alibabacloud_ecs20140526>=4.1.0,<6", +# "alibabacloud_tea_openapi>=0.3.12,<1", +# ] +# /// + +"""Provision an ephemeral Aliyun ECS query-regression runner. + +Creates one pay-as-you-go ECS instance from the query-regression custom image +and waits until the instance registers itself as an ephemeral GitHub Actions +runner with a per-run label. Build caches live on the instance system disk +and disappear with the VM. The instance id is written to job outputs as soon +as the VM exists so the teardown job is the single DeleteInstance caller (a +timeout here used to delete and then teardown deleted again). On runner +online timeout the console is dumped and the job fails; teardown releases +the instance. + +The Aliyun credentials are used only in this control-plane job (ubuntu-latest) +and are never passed to the ECS instance; the instance receives only a +short-lived GitHub runner registration token via user data. +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import os +import time +import urllib.error +import urllib.request +from dataclasses import dataclass + +RUNNER_NAME_PREFIX = "qreg-ecs" +RUNNER_LABEL_PREFIX = "query-regression-ecs" +MANAGED_BY_TAG_KEY = "managed-by" +MANAGED_BY_TAG_VALUE = "query-regression-ci" +RUN_TAG_KEY = "query-regression-run-id" + +# Runner cache paths on the instance system disk. They are created empty +# every provision and deleted with the VM. +CACHE_PATHS = ( + "/home/runner/.cargo/registry", + "/home/runner/.cargo/git", + "/home/runner/query-regression-target", + "/home/runner/query-regression-cache-meta", + "/home/runner/.cache/sccache", +) + +RUNNER_ONLINE_TIMEOUT_SECONDS = 10 * 60 +POLL_INTERVAL_SECONDS = 5 + +# Nightly thin-LTO of greptime peaks above ecs.c9i.2xlarge's 16 GiB. A swap +# file plus masking systemd-oomd lets the kernel reclaim rustc pages instead +# of SIGTERM-ing the runner service cgroup (GitHub then reports the step as +# "The operation was canceled"). Sized to match that 16 GiB instance; a +# 32 GiB type still benefits as a safety net. Lives on the system disk next +# to the cold build caches. +SWAP_FILE = "/swapfile" +SWAP_SIZE_GIB = 16 +# Image (~12G), 16G swap, checkout, base+candidate data homes, and cold +# build caches. Deleted with the instance. +SYSTEM_DISK_GIB = 50 + + +@dataclass(frozen=True) +class ProvisionConfig: + region_id: str + vswitch_id: str + security_group_id: str + image_id: str + instance_type: str + repo: str + run_id: str + github_token: str + # Optional resource group; required when the RAM grant is scoped to one. + resource_group_id: str | None = None + # Runner identity inside the image; the workflow asserts the same values. + runner_uid: str = "1001" + runner_gid: str = "1001" + + +def runner_name_for_run(run_id: str) -> str: + return f"{RUNNER_NAME_PREFIX}-{run_id}" + + +def runner_label_for_run(run_id: str) -> str: + return f"{RUNNER_LABEL_PREFIX}-{run_id}" + + +def render_user_data( + runner_name: str, + runner_label: str, + runner_token: str, + repo: str, + runner_uid: str = "1001", + runner_gid: str = "1001", +) -> str: + """Render the cloud-init shell script for the runner instance.""" + destinations = " ".join(f'"{dst}"' for dst in CACHE_PATHS) + cache_setup = f"""# Caches live on the system disk, are deleted with the instance, and every +# run compiles cold. Within-run reuse (base warming candidate via the shared +# target dir and sccache) still applies. +mkdir -p {destinations} +chown -R {runner_uid}:{runner_gid} /home/runner""" + swap_setup = f"""# Mask systemd-oomd before enabling swap: Ubuntu 24.04 kills the whole +# service cgroup on PSI pressure, and swap thrashing looks like pressure. +systemctl disable --now systemd-oomd.socket systemd-oomd.service || true +systemctl mask systemd-oomd.socket systemd-oomd.service || true +# apt-daily-upgrade / unattended-upgrades can `systemctl restart` services +# whose libraries were updated. That SIGTERM-s the runner mid-job; GitHub +# records UserCancelled (job still valid, child exit 143). These VMs live +# ~1h; security updates belong in the image, not at job time. +systemctl disable --now unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true +systemctl mask unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer apt-daily.service apt-daily-upgrade.service || true +systemctl stop unattended-upgrades.service || true +killall -9 unattended-upgr || true +cat > /etc/apt/apt.conf.d/99disable-auto-updates <<'APTEOF' +APT::Periodic::Update-Package-Lists "0"; +APT::Periodic::Unattended-Upgrade "0"; +APT::Periodic::Download-Upgradeable-Packages "0"; +APTEOF +if [[ ! -f "{SWAP_FILE}" ]]; then + fallocate --length {SWAP_SIZE_GIB}G "{SWAP_FILE}" + chmod 600 "{SWAP_FILE}" + mkswap "{SWAP_FILE}" +fi +swapon "{SWAP_FILE}" +sysctl --write vm.swappiness=10 +swapon --show +free --human""" + return f"""#!/bin/bash +set -euo pipefail + +{cache_setup} + +{swap_setup} + +cat > /etc/ephemeral-github-runner.env <<'ENVEOF' +RUNNER_NAME={runner_name} +RUNNER_LABELS={runner_label} +RUNNER_TOKEN={runner_token} +REPO_URL=https://github.com/{repo} +# The container image baked /opt/cargo/bin into PATH via Dockerfile ENV; on +# the host nothing inherits that, so publish it through the unit's +# EnvironmentFile: runner job processes inherit the runner's environment. +PATH=/opt/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin +ENVEOF + +# Stream the runner service's own logs (config output, job lifecycle, crash +# messages) to the serial console, next to the kernel's OOM-killer records. +# Teardown dumps the console tail before deleting the instance, so the +# witness of what killed the runner survives the machine. +mkdir -p /etc/systemd/system/ephemeral-github-runner.service.d +cat > /etc/systemd/system/ephemeral-github-runner.service.d/console.conf <<'CONFEOF' +[Service] +StandardOutput=journal+console +StandardError=journal+console +CONFEOF +# systemd's DefaultOOMPolicy=stop SIGTERM-s the whole unit when *any* +# process in the cgroup is OOM-killed (typically rustc during thin LTO). +# That is the "Session terminated, killing shell..." / job Canceled path: +# runuser dies, GitHub deletes the ephemeral registration, always() steps +# never run. continue keeps the runner up so cargo can report signal 9. +cat > /etc/systemd/system/ephemeral-github-runner.service.d/oom.conf <<'CONFEOF' +[Service] +OOMPolicy=continue +CONFEOF +systemctl daemon-reload +# restart (not start): the image enables this unit, so it may already +# have come up without the drop-ins if it raced cloud-init. restart +# applies OOMPolicy=continue to the running cgroup. --no-block matters: +# newer image units carry After=cloud-final.service, and this script IS +# cloud-final — a blocking restart would deadlock until TimeoutStartSec. +systemctl restart --no-block ephemeral-github-runner.service +""" + + +def encode_user_data(script: str) -> str: + return base64.b64encode(script.encode("utf-8")).decode("ascii") + + +def github_api(token: str, method: str, path: str, body: dict | None = None) -> dict: + data = json.dumps(body).encode("utf-8") if body is not None else None + request = urllib.request.Request( + f"https://api.github.com{path}", + data=data, + method=method, + headers={ + "Accept": "application/vnd.github+json", + "Authorization": f"Bearer {token}", + "X-GitHub-Api-Version": "2022-11-28", + }, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + # GitHub's error body says exactly why (e.g. "Must have admin rights to + # Repository" for a PAT without the required scope); surface it instead + # of a bare "HTTP Error 403". + body = error.read().decode("utf-8", "replace") + raise SystemExit( + f"GitHub API {method} {path} failed: HTTP {error.code}: {body}\n" + "The token comes from the GH_PERSONAL_ACCESS_TOKEN secret; it needs " + "'repo' scope (classic PAT) or 'Administration: write' on the " + "repository (fine-grained PAT)." + ) from error + + +def create_registration_token(github_token: str, repo: str) -> str: + response = github_api( + github_token, + "POST", + f"/repos/{repo}/actions/runners/registration-token", + body={}, + ) + return response["token"] + + +def find_runner_by_name(github_token: str, repo: str, name: str) -> dict | None: + page = 1 + while True: + response = github_api( + github_token, + "GET", + f"/repos/{repo}/actions/runners?per_page=100&page={page}", + ) + runners = response.get("runners", []) + for runner in runners: + if runner.get("name") == name: + return runner + if len(runners) < 100: + return None + page += 1 + + +def make_ecs_client(config: ProvisionConfig): + from alibabacloud_ecs20140526.client import Client as EcsClient + from alibabacloud_tea_openapi.models import Config as OpenApiConfig + + access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "") + access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "") + if not access_key_id or not access_key_secret: + raise SystemExit( + "ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the " + "ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or " + "misnamed secret expands to an empty string)." + ) + return EcsClient( + OpenApiConfig( + access_key_id=access_key_id, + access_key_secret=access_key_secret, + region_id=config.region_id, + endpoint=f"ecs.{config.region_id}.aliyuncs.com", + ) + ) + + +def wait_for_instance_status(client, region_id: str, instance_id: str, wanted: str, deadline: float) -> None: + from alibabacloud_ecs20140526 import models as ecs_models + + while time.monotonic() < deadline: + response = client.describe_instances( + ecs_models.DescribeInstancesRequest( + region_id=region_id, instance_ids=json.dumps([instance_id]) + ) + ) + instances = response.body.instances.instance + if instances and instances[0].status == wanted: + return + time.sleep(POLL_INTERVAL_SECONDS) + raise TimeoutError(f"Instance {instance_id} did not reach status {wanted} in time") + + +def fetch_console_output(client, region_id: str, instance_id: str) -> str: + from alibabacloud_ecs20140526 import models as ecs_models + + response = client.get_instance_console_output( + ecs_models.GetInstanceConsoleOutputRequest(region_id=region_id, instance_id=instance_id) + ) + return base64.b64decode(response.body.console_output or "").decode("utf-8", "replace") + + +class ConsoleTailer: + """Incrementally prints new serial-console lines so cloud-init progress is + visible in the CI log while waiting, not only after a failure dump.""" + + def __init__(self, client, region_id: str, instance_id: str) -> None: + self.client = client + self.region_id = region_id + self.instance_id = instance_id + self.printed_lines = 0 + + def poll(self) -> None: + try: + lines = fetch_console_output(self.client, self.region_id, self.instance_id).splitlines() + except Exception as error: # noqa: BLE001 + print(f"[console] unable to fetch console output: {error}", flush=True) + return + # The API returns a bounded tail; if it ever shrinks, restart from the + # beginning of the new buffer rather than skipping lines. + if len(lines) < self.printed_lines: + self.printed_lines = 0 + for line in lines[self.printed_lines :]: + print(f"[console] {line}", flush=True) + self.printed_lines = len(lines) + + +def dump_console_output(client, region_id: str, instance_id: str) -> None: + try: + output = fetch_console_output(client, region_id, instance_id) + except Exception as error: # noqa: BLE001 + print(f"Unable to fetch console output for {instance_id}: {error}", flush=True) + return + tail = "\n".join(output.splitlines()[-80:]) + print(f"::group::Console output tail for {instance_id}\n{tail}\n::endgroup::", flush=True) + summary_path = os.environ.get("GITHUB_STEP_SUMMARY") + if summary_path: + with open(summary_path, "a", encoding="utf-8") as summary: + summary.write(f"
ECS console output ({instance_id})\n\n```\n") + summary.write(tail) + summary.write("\n```\n
\n") + + +def append_github_output(name: str, value: str) -> None: + output_path = os.environ.get("GITHUB_OUTPUT") + if output_path: + with open(output_path, "a", encoding="utf-8") as output: + output.write(f"{name}={value}\n") + + +def run_instance(client, config: ProvisionConfig, user_data: str) -> str: + from alibabacloud_ecs20140526 import models as ecs_models + + request = ecs_models.RunInstancesRequest( + region_id=config.region_id, + image_id=config.image_id, + resource_group_id=config.resource_group_id, + instance_type=config.instance_type, + v_switch_id=config.vswitch_id, + security_group_id=config.security_group_id, + instance_name=runner_name_for_run(config.run_id), + host_name=runner_name_for_run(config.run_id), + description=f"Ephemeral query-regression runner for run {config.run_id}", + amount=1, + instance_charge_type="PostPaid", + spot_strategy="NoSpot", + internet_charge_type="PayByTraffic", + internet_max_bandwidth_out=100, + system_disk=ecs_models.RunInstancesRequestSystemDisk( + category="cloud_essd", + size=str(SYSTEM_DISK_GIB), + ), + user_data=user_data, + tag=[ + ecs_models.RunInstancesRequestTag(key=MANAGED_BY_TAG_KEY, value=MANAGED_BY_TAG_VALUE), + ecs_models.RunInstancesRequestTag(key=RUN_TAG_KEY, value=config.run_id), + ], + ) + response = client.run_instances(request) + instance_id = response.body.instance_id_sets.instance_id_set[0] + return instance_id + + +def provision(config: ProvisionConfig) -> int: + client = make_ecs_client(config) + runner_name = runner_name_for_run(config.run_id) + runner_label = runner_label_for_run(config.run_id) + + registration_token = create_registration_token(config.github_token, config.repo) + user_data = encode_user_data( + render_user_data( + runner_name, + runner_label, + registration_token, + config.repo, + config.runner_uid, + config.runner_gid, + ) + ) + + print(f"::group::Run instance {runner_name}", flush=True) + instance_id = run_instance(client, config, user_data) + print(f"Instance id: {instance_id}", flush=True) + # Teardown is the only DeleteInstance caller. Write outputs immediately + # so a later status/online failure still releases this VM. + append_github_output("label", runner_label) + append_github_output("instance_id", instance_id) + append_github_output("runner_name", runner_name) + wait_for_instance_status(client, config.region_id, instance_id, "Running", time.monotonic() + 5 * 60) + print("::endgroup::", flush=True) + + print("::group::Wait for runner online", flush=True) + deadline = time.monotonic() + RUNNER_ONLINE_TIMEOUT_SECONDS + tailer = ConsoleTailer(client, config.region_id, instance_id) + last_console_poll = 0.0 + while time.monotonic() < deadline: + runner = find_runner_by_name(config.github_token, config.repo, runner_name) + if runner is not None: + status = runner.get("status") + print(f"Runner {runner_name} status: {status}", flush=True) + if status == "online": + print("::endgroup::", flush=True) + print(f"Runner {runner_name} is online with label {runner_label}", flush=True) + return 0 + # The serial console lags the guest by a minute or so; 30s polling is + # enough to follow cloud-init without tripping API throttling. + if time.monotonic() - last_console_poll >= 30: + tailer.poll() + last_console_poll = time.monotonic() + time.sleep(POLL_INTERVAL_SECONDS) + + print("::endgroup::", flush=True) + print( + f"Runner {runner_name} did not come online in time; " + "leaving instance for the teardown job to delete", + flush=True, + ) + dump_console_output(client, config.region_id, instance_id) + return 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID")) + parser.add_argument("--vswitch-id", default=os.environ.get("ALIYUN_ECS_VSWITCH_ID")) + parser.add_argument("--security-group-id", default=os.environ.get("ALIYUN_ECS_SECURITY_GROUP_ID")) + parser.add_argument("--image-id", default=os.environ.get("QUERY_REGRESSION_ECS_IMAGE_ID")) + parser.add_argument("--instance-type", default=os.environ.get("ALIYUN_ECS_INSTANCE_TYPE")) + parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY")) + parser.add_argument("--run-id", default=os.environ.get("GITHUB_RUN_ID")) + parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN")) + parser.add_argument("--runner-uid", default=os.environ.get("QUERY_REGRESSION_RUNNER_UID", "1001")) + parser.add_argument("--runner-gid", default=os.environ.get("QUERY_REGRESSION_RUNNER_GID", "1001")) + parser.add_argument("--resource-group-id", default=os.environ.get("ALIYUN_ECS_RESOURCE_GROUP_ID")) + args = parser.parse_args() + + missing = [ + name + for name, value in vars(args).items() + if name not in ("resource_group_id",) + and (value is None or (isinstance(value, str) and not value)) + ] + if missing: + flags = ", ".join(f"--{name.replace('_', '-')}" for name in missing) + raise SystemExit(f"Missing required configuration: {flags} (or their env defaults)") + + config = ProvisionConfig( + region_id=args.region_id, + vswitch_id=args.vswitch_id, + security_group_id=args.security_group_id, + image_id=args.image_id, + instance_type=args.instance_type, + repo=args.repo, + run_id=args.run_id, + github_token=args.github_token, + resource_group_id=args.resource_group_id or None, + runner_uid=args.runner_uid, + runner_gid=args.runner_gid, + ) + return provision(config) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/scripts/aliyun-ecs-runner-teardown.py b/.github/scripts/aliyun-ecs-runner-teardown.py new file mode 100644 index 0000000000..36c80253b9 --- /dev/null +++ b/.github/scripts/aliyun-ecs-runner-teardown.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# PEP 723 inline metadata (see aliyun-ecs-runner-provision.py for the +# convention). The SDK import stays lazy so unit tests run on a plain +# stdlib interpreter. +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "alibabacloud_ecs20140526>=4.1.0,<6", +# "alibabacloud_tea_openapi>=0.3.12,<1", +# ] +# /// + +"""Tear down ephemeral Aliyun ECS query-regression runners. + +Two modes: + +- Targeted (per workflow run): delete one instance by id and deregister its + runner by name. Both steps are idempotent and best-effort; a missing + instance or runner is not an error. +- Sweep (scheduled janitor): delete every instance tagged as managed by + query-regression CI whose creation time is older than the given TTL, and + deregister the matching runners. This is the safety net for runs whose + teardown job never executed. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import os +import sys +from datetime import datetime, timedelta, timezone +from pathlib import Path + +# Reuse the GitHub API client, runner lookup, and tag constants from the +# provision script. +_PROVISION_SPEC = importlib.util.spec_from_file_location( + "aliyun_ecs_runner_provision", + Path(__file__).resolve().parent / "aliyun-ecs-runner-provision.py", +) +assert _PROVISION_SPEC is not None and _PROVISION_SPEC.loader is not None +provision = importlib.util.module_from_spec(_PROVISION_SPEC) +sys.modules[_PROVISION_SPEC.name] = provision +_PROVISION_SPEC.loader.exec_module(provision) + +SWEEP_TTL = timedelta(hours=4) +# ECS DescribeInstances creation_time is ISO8601 UTC, e.g. "2026-08-17T02:13Z". +CREATION_TIME_FORMATS = ("%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%dT%H:%MZ") + + +def parse_creation_time(value: str) -> datetime: + for fmt in CREATION_TIME_FORMATS: + try: + return datetime.strptime(value, fmt).replace(tzinfo=timezone.utc) + except ValueError: + continue + raise ValueError(f"Unparsable ECS creation time: {value}") + + +def expired_instance_names( + instances: list[tuple[str, str, str]], now: datetime, ttl: timedelta +) -> list[tuple[str, str]]: + """Pick (instance_id, instance_name) pairs whose creation is older than ttl. + + `instances` items are (instance_id, instance_name, creation_time). + """ + expired = [] + for instance_id, instance_name, creation_time in instances: + age = now - parse_creation_time(creation_time) + if age >= ttl: + expired.append((instance_id, instance_name)) + return expired + + +def make_ecs_client(region_id: str): + from alibabacloud_ecs20140526.client import Client as EcsClient + from alibabacloud_tea_openapi.models import Config as OpenApiConfig + + access_key_id = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_ID", "") + access_key_secret = os.environ.get("ALIBABA_CLOUD_ACCESS_KEY_SECRET", "") + if not access_key_id or not access_key_secret: + raise SystemExit( + "ALIBABA_CLOUD_ACCESS_KEY_ID/SECRET are empty; in CI they come from the " + "ALICLOUD_ECS_ACCESS_KEY_ID/SECRET repository secrets (a missing or " + "misnamed secret expands to an empty string)." + ) + return EcsClient( + OpenApiConfig( + access_key_id=access_key_id, + access_key_secret=access_key_secret, + region_id=region_id, + endpoint=f"ecs.{region_id}.aliyuncs.com", + ) + ) + + +def delete_instance(client, instance_id: str, region_id: str | None = None) -> bool: + import time + + from alibabacloud_ecs20140526 import models as ecs_models + + # Export the serial console before destroying the evidence: a cancelled + # run still runs this teardown job, and the console tail (cloud-init, + # runner service logs, kernel OOM records) is the only witness of what + # happened on the machine. + if region_id is not None: + provision.dump_console_output(client, region_id, instance_id) + + # DeleteInstance rejects instances that are still Initializing (or in + # another transitional status). Teardown runs right after a cancelled + # run, when the instance may be only seconds old, so retry the transient + # status errors for a few minutes instead of leaking the instance. + deadline = time.monotonic() + 5 * 60 + while True: + try: + client.delete_instance(ecs_models.DeleteInstanceRequest(instance_id=instance_id, force=True)) + print(f"Deleted instance {instance_id}", flush=True) + return True + except Exception as error: # noqa: BLE001 + message = str(error) + if "InvalidInstanceId.NotFound" in message: + print(f"Instance {instance_id} already gone", flush=True) + return True + if "IncorrectInstanceStatus" in message and time.monotonic() < deadline: + print(f"Instance {instance_id} is in a transitional status; retrying delete", flush=True) + time.sleep(15) + continue + print(f"Failed to delete instance {instance_id}: {error}", flush=True) + return False + + +def deregister_runner(token: str, repo: str, runner_name: str) -> bool: + runner = provision.find_runner_by_name(token, repo, runner_name) + if runner is None: + print(f"Runner {runner_name} is not registered", flush=True) + return True + try: + provision.github_api(token, "DELETE", f"/repos/{repo}/actions/runners/{runner['id']}") + print(f"Deregistered runner {runner_name} (id {runner['id']})", flush=True) + return True + except Exception as error: # noqa: BLE001 + print(f"Failed to deregister runner {runner_name}: {error}", flush=True) + return False + + +def list_managed_instances(client, region_id: str) -> list[tuple[str, str, str]]: + from alibabacloud_ecs20140526 import models as ecs_models + + result: list[tuple[str, str, str]] = [] + next_token = None + while True: + request = ecs_models.DescribeInstancesRequest( + region_id=region_id, + tag=[ + ecs_models.DescribeInstancesRequestTag( + key=provision.MANAGED_BY_TAG_KEY, value=provision.MANAGED_BY_TAG_VALUE + ) + ], + max_results=100, + next_token=next_token, + ) + response = client.describe_instances(request) + for instance in response.body.instances.instance: + result.append((instance.instance_id, instance.instance_name, instance.creation_time)) + next_token = response.body.next_token + if not next_token: + return result + + +def sweep(client, region_id: str, repo: str, github_token: str, ttl: timedelta) -> int: + instances = list_managed_instances(client, region_id) + print(f"Found {len(instances)} managed instance(s) in {region_id}", flush=True) + expired = expired_instance_names(instances, datetime.now(timezone.utc), ttl) + ok = True + for instance_id, instance_name in expired: + print(f"Instance {instance_id} ({instance_name}) exceeds TTL {ttl}; deleting", flush=True) + # Runner names mirror instance names by construction in the provision + # script (both are qreg-ecs-). + ok &= delete_instance(client, instance_id, region_id) + ok &= deregister_runner(github_token, repo, instance_name) + return 0 if ok else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--region-id", default=os.environ.get("ALIYUN_ECS_REGION_ID")) + parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY")) + parser.add_argument("--github-token", default=os.environ.get("GH_PERSONAL_ACCESS_TOKEN")) + parser.add_argument("--instance-id", default=os.environ.get("QUERY_REGRESSION_ECS_INSTANCE_ID")) + parser.add_argument("--runner-name", default=os.environ.get("QUERY_REGRESSION_ECS_RUNNER_NAME")) + parser.add_argument( + "--sweep", + action="store_true", + help="Janitor mode: delete all managed instances older than --sweep-ttl-hours.", + ) + parser.add_argument("--sweep-ttl-hours", type=float, default=SWEEP_TTL.total_seconds() / 3600) + args = parser.parse_args() + + for name in ("region_id", "repo", "github_token"): + if not getattr(args, name): + raise SystemExit(f"Missing required configuration: --{name.replace('_', '-')}") + + client = make_ecs_client(args.region_id) + + if args.sweep: + return sweep( + client, args.region_id, args.repo, args.github_token, timedelta(hours=args.sweep_ttl_hours) + ) + + ok = True + if args.instance_id: + ok &= delete_instance(client, args.instance_id, args.region_id) + else: + print("No instance id given; skipping instance deletion", flush=True) + if args.runner_name: + ok &= deregister_runner(args.github_token, args.repo, args.runner_name) + else: + print("No runner name given; skipping runner deregistration", flush=True) + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/scripts/query-regression-nightly-refs.py b/.github/scripts/query-regression-nightly-refs.py new file mode 100644 index 0000000000..6e9f54682d --- /dev/null +++ b/.github/scripts/query-regression-nightly-refs.py @@ -0,0 +1,270 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Resolve consecutive successful Nightly Build SHAs for query-regression. + +Query regression builds both binaries from git; this script only picks the +head SHAs of two Nightly Build workflow runs (today vs the previous success). +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from typing import Any + + +NIGHTLY_WORKFLOW = "nightly-build.yml" +NIGHTLY_EVENTS = frozenset({"schedule", "workflow_dispatch"}) + + +@dataclass(frozen=True) +class WorkflowRun: + id: int + head_sha: str + head_branch: str + html_url: str + created_at: str + conclusion: str + event: str + + +@dataclass(frozen=True) +class RefPair: + base: WorkflowRun | None + candidate: WorkflowRun | None + skip: bool + reason: str + + +def parse_run(payload: dict[str, Any]) -> WorkflowRun: + return WorkflowRun( + id=int(payload["id"]), + head_sha=str(payload["head_sha"]), + head_branch=str(payload.get("head_branch") or ""), + html_url=str(payload.get("html_url") or ""), + created_at=str(payload.get("created_at") or ""), + conclusion=str(payload.get("conclusion") or ""), + event=str(payload.get("event") or ""), + ) + + +def github_get(token: str, path: str, query: dict[str, str] | None = None) -> dict[str, Any]: + encoded = f"?{urllib.parse.urlencode(query)}" if query else "" + request = urllib.request.Request( + f"https://api.github.com{path}{encoded}", + method="GET", + headers={ + "Accept": "application/vnd.github+json", + "Authorization": f"Bearer {token}", + "X-GitHub-Api-Version": "2022-11-28", + }, + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + body = error.read().decode("utf-8", "replace") + raise SystemExit( + f"GitHub API GET {path} failed: HTTP {error.code}: {body}" + ) from error + + +def list_successful_nightly_runs( + token: str, + repo: str, + *, + branch: str | None = None, + per_page: int = 30, +) -> list[WorkflowRun]: + query = { + "status": "success", + "per_page": str(per_page), + } + if branch: + query["branch"] = branch + payload = github_get( + token, + f"/repos/{repo}/actions/workflows/{NIGHTLY_WORKFLOW}/runs", + query, + ) + runs = [parse_run(item) for item in payload.get("workflow_runs") or []] + return [ + run + for run in runs + if run.conclusion == "success" and run.event in NIGHTLY_EVENTS and run.head_sha + ] + + +def fetch_run(token: str, repo: str, run_id: int) -> WorkflowRun: + payload = github_get(token, f"/repos/{repo}/actions/runs/{run_id}") + return parse_run(payload) + + +def select_base_and_candidate( + runs: list[WorkflowRun], + *, + candidate_run_id: int | None = None, + candidate: WorkflowRun | None = None, +) -> RefPair: + """Pick candidate (newest / requested) and the previous successful run. + + `runs` must be newest-first. Same-SHA consecutive nightlies are skipped + (no commits that day). + """ + if candidate is None: + if candidate_run_id is not None: + candidate = next((run for run in runs if run.id == candidate_run_id), None) + if candidate is None: + return RefPair( + None, + None, + True, + f"candidate nightly run {candidate_run_id} was not a successful " + f"{NIGHTLY_WORKFLOW} run", + ) + elif runs: + candidate = runs[0] + else: + return RefPair(None, None, True, "no successful Nightly Build runs") + + branch = candidate.head_branch + older = [ + run + for run in runs + if run.id != candidate.id + and (not branch or not run.head_branch or run.head_branch == branch) + and (not candidate.created_at or run.created_at <= candidate.created_at) + ] + if not older: + return RefPair( + None, + candidate, + True, + "no previous successful Nightly Build to compare against", + ) + base = older[0] + if base.head_sha.lower() == candidate.head_sha.lower(): + return RefPair( + base, + candidate, + True, + f"previous nightly SHA {base.head_sha} matches candidate; nothing to compare", + ) + return RefPair(base, candidate, False, "") + + +def _override_run(sha: str) -> WorkflowRun: + return WorkflowRun( + id=0, + head_sha=sha, + head_branch="", + html_url="", + created_at="", + conclusion="success", + event="workflow_dispatch", + ) + + +def override_pair(base_ref: str, candidate_ref: str) -> RefPair: + return RefPair(_override_run(base_ref), _override_run(candidate_ref), False, "explicit refs") + + +def write_outputs(pair: RefPair) -> None: + values = { + "skip": "true" if pair.skip else "false", + "reason": pair.reason, + "base_sha": pair.base.head_sha if pair.base else "", + "candidate_sha": pair.candidate.head_sha if pair.candidate else "", + "base_run_url": pair.base.html_url if pair.base else "", + "candidate_run_url": pair.candidate.html_url if pair.candidate else "", + "base_run_id": str(pair.base.id) if pair.base and pair.base.id else "", + "candidate_run_id": str(pair.candidate.id) if pair.candidate and pair.candidate.id else "", + } + output_path = os.environ.get("GITHUB_OUTPUT") + if output_path: + with open(output_path, "a", encoding="utf-8") as handle: + for key, value in values.items(): + handle.write(f"{key}={value}\n") + for key, value in values.items(): + print(f"{key}={value}") + + +def parse_args(argv: list[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--repo", default=os.environ.get("GITHUB_REPOSITORY")) + parser.add_argument( + "--token", + default=os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN"), + ) + parser.add_argument("--branch", default=os.environ.get("NIGHTLY_BRANCH") or "") + parser.add_argument( + "--candidate-run-id", + default=os.environ.get("CANDIDATE_RUN_ID") or "", + ) + parser.add_argument("--base-ref", default=os.environ.get("BASE_REF") or "") + parser.add_argument("--candidate-ref", default=os.environ.get("CANDIDATE_REF") or "") + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = parse_args(argv) + base_ref = args.base_ref.strip() + candidate_ref = args.candidate_ref.strip() + if base_ref and candidate_ref: + write_outputs(override_pair(base_ref, candidate_ref)) + return 0 + if bool(base_ref) != bool(candidate_ref): + print("Provide both --base-ref and --candidate-ref, or neither.", file=sys.stderr) + return 2 + if not args.repo or not args.token: + print("--repo and --token (or GITHUB_REPOSITORY / GITHUB_TOKEN) are required.", file=sys.stderr) + return 2 + + candidate_run_id = int(args.candidate_run_id) if str(args.candidate_run_id).strip() else None + candidate: WorkflowRun | None = None + branch = args.branch.strip() or None + if candidate_run_id is not None: + candidate = fetch_run(args.token, args.repo, candidate_run_id) + if candidate.conclusion != "success": + write_outputs( + RefPair( + None, + candidate, + True, + f"nightly run {candidate_run_id} conclusion is {candidate.conclusion}", + ) + ) + return 0 + branch = branch or candidate.head_branch or None + + runs = list_successful_nightly_runs(args.token, args.repo, branch=branch) + pair = select_base_and_candidate( + runs, + candidate_run_id=candidate_run_id if candidate is None else None, + candidate=candidate, + ) + write_outputs(pair) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.github/scripts/query-regression-run.py b/.github/scripts/query-regression-run.py index 8bf78ca2f0..3148381dbe 100644 --- a/.github/scripts/query-regression-run.py +++ b/.github/scripts/query-regression-run.py @@ -18,10 +18,17 @@ from __future__ import annotations import argparse +import json import os import re +import signal +import socket import subprocess +import time +import urllib.request +from dataclasses import dataclass from pathlib import Path +from typing import Any DEFAULT_CASES = [ @@ -99,38 +106,479 @@ def append_step_summary(summary: Path) -> None: out.write(summary.read_text()) +@dataclass(frozen=True) +class RunTarget: + name: str + binary: Path + work_dir: Path + http_port: int + grpc_port: int + mysql_port: int + postgres_port: int + metasrv_rpc_port: int + metasrv_http_port: int + datanode_rpc_port: int + datanode_http_port: int + datanode_data_dir: Path + frontend_config: Path | None = None + + +def allocate_ports(n: int) -> list[int]: + socks = [] + try: + for _ in range(n): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.bind(("127.0.0.1", 0)) + socks.append(sock) + return [sock.getsockname()[1] for sock in socks] + finally: + for sock in socks: + sock.close() + + +def make_target( + name: str, + binary: Path, + root: Path, + ports: list[int], + frontend_config: Path | None = None, +) -> RunTarget: + work_dir = root / name + return RunTarget( + name=name, + binary=binary, + work_dir=work_dir, + http_port=ports[4], + grpc_port=ports[5], + mysql_port=ports[6], + postgres_port=ports[7], + metasrv_rpc_port=ports[0], + metasrv_http_port=ports[1], + datanode_rpc_port=ports[2], + datanode_http_port=ports[3], + datanode_data_dir=work_dir / "datanode-0" / "data", + frontend_config=frontend_config, + ) + + +def wait_health(port: int, timeout_s: float = 60.0) -> None: + deadline = time.monotonic() + timeout_s + last: Exception | None = None + while time.monotonic() < deadline: + try: + with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=2) as response: + if response.status < 500: + return + except Exception as err: # noqa: BLE001 - retain the last health diagnostic + last = err + time.sleep(0.5) + raise TimeoutError(f"health check timed out on port {port}: {last}") + + +def component_log_dir(target: RunTarget, name: str) -> Path: + return target.work_dir / "logs" / ("datanode-0" if name == "datanode" else name) + + +def component_command(target: RunTarget, name: str) -> list[str]: + log_dir = component_log_dir(target, name) + if name == "metasrv": + return [ + str(target.binary), "metasrv", "start", + "--grpc-bind-addr", f"127.0.0.1:{target.metasrv_rpc_port}", + "--grpc-server-addr", f"127.0.0.1:{target.metasrv_rpc_port}", + "--http-addr", f"127.0.0.1:{target.metasrv_http_port}", + "--backend", "memory-store", "--enable-region-failover", "false", + "--log-dir", str(log_dir), + ] + if name == "datanode": + return [ + str(target.binary), "datanode", "start", + "--grpc-bind-addr", f"127.0.0.1:{target.datanode_rpc_port}", + "--grpc-server-addr", f"127.0.0.1:{target.datanode_rpc_port}", + "--http-addr", f"127.0.0.1:{target.datanode_http_port}", + "--data-home", str(target.datanode_data_dir), "--log-dir", str(log_dir), + "--node-id", "0", "--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}", + ] + if name == "frontend": + command = [str(target.binary), "frontend", "start"] + if target.frontend_config is not None: + command.extend(["--config-file", str(target.frontend_config)]) + command.extend([ + "--metasrv-addrs", f"127.0.0.1:{target.metasrv_rpc_port}", + "--http-addr", f"127.0.0.1:{target.http_port}", + "--grpc-bind-addr", f"127.0.0.1:{target.grpc_port}", + "--grpc-server-addr", f"127.0.0.1:{target.grpc_port}", + "--mysql-addr", f"127.0.0.1:{target.mysql_port}", + "--postgres-addr", f"127.0.0.1:{target.postgres_port}", + "--log-dir", str(log_dir), + ]) + return command + raise ValueError(f"unknown component: {name}") + + +def start_component( + target: RunTarget, + name: str, + procs: dict[tuple[str, str], subprocess.Popen[bytes]], +) -> None: + if name != "metasrv": + metasrv = procs.get((target.name, "metasrv")) + if metasrv is None or metasrv.poll() is not None: + raise RuntimeError("metasrv exited; memory-store metadata is no longer valid") + if name == "datanode": + target.datanode_data_dir.mkdir(parents=True, exist_ok=True) + logs = component_log_dir(target, name) + logs.mkdir(parents=True, exist_ok=True) + with (logs / "stdout.log").open("ab") as out, (logs / "stderr.log").open("ab") as err: + procs[(target.name, name)] = subprocess.Popen( + component_command(target, name), + stdout=out, + stderr=err, + start_new_session=True, + ) + wait_health({ + "metasrv": target.metasrv_http_port, + "datanode": target.datanode_http_port, + "frontend": target.http_port, + }[name]) + + +def stop_component( + target: RunTarget, + name: str, + procs: dict[tuple[str, str], subprocess.Popen[bytes]], +) -> None: + proc = procs.pop((target.name, name), None) + if proc is None or proc.poll() is not None: + return + try: + os.killpg(proc.pid, signal.SIGTERM) + except ProcessLookupError: + return + try: + proc.wait(timeout=20) + except subprocess.TimeoutExpired: + try: + os.killpg(proc.pid, signal.SIGKILL) + except ProcessLookupError: + return + proc.wait(timeout=20) + + +def restart_component( + target: RunTarget, + name: str, + procs: dict[tuple[str, str], subprocess.Popen[bytes]], +) -> None: + stop_component(target, name, procs) + start_component(target, name, procs) + + +def stop_all(targets: list[RunTarget], procs: dict[tuple[str, str], subprocess.Popen[bytes]]) -> None: + for target in reversed(targets): + for name in ("frontend", "datanode", "metasrv"): + stop_component(target, name, procs) + + +def load_plan(fixture_generator: Path, case_path: Path) -> dict[str, Any]: + result = subprocess.run( + [str(fixture_generator), "plan", "--case", str(case_path)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + check=False, + ) + if result.returncode != 0: + raise RuntimeError(f"query_perf_fixture plan failed: {result.stderr[:2000]}") + return json.loads(result.stdout) + + +def run_direct_case( + args: argparse.Namespace, + case_path: Path, + work_dir: Path, + base_bin: Path, + candidate_bin: Path, + fixture_generator: Path, + runner: Path, +) -> int: + ports = allocate_ports(16) + targets = [ + make_target("base", base_bin, work_dir, ports[:8]), + make_target("candidate", candidate_bin, work_dir, ports[8:]), + ] + for target in targets: + if target.work_dir.exists() and any(target.work_dir.iterdir()): + raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}") + target.work_dir.mkdir(parents=True, exist_ok=True) + + procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {} + try: + for target in targets: + start_component(target, "metasrv", procs) + start_component(target, "datanode", procs) + start_component(target, "frontend", procs) + + prepare = [ + str(runner), "prepare-direct", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--base-http-port", str(targets[0].http_port), + "--candidate-http-port", str(targets[1].http_port), + "--fixture-dir", str(work_dir / "fixture"), + "--output", str(work_dir / "prepare-direct.json"), + "--http-timeout", str(args.http_timeout), + ] + if parse_bool(args.allow_large_fixture): + prepare.append("--allow-large-fixture") + prepare_status = subprocess.run(prepare, check=False).returncode + if prepare_status != 0: + return prepare_status + + for target in targets: + stop_component(target, "datanode", procs) + + prepared = json.loads((work_dir / "prepare-direct.json").read_text(encoding="utf-8")) + fixtures = prepared.get("fixtures") + if not isinstance(fixtures, list): + raise RuntimeError("prepare-direct report has no fixtures array") + for target in targets: + destination = target.work_dir / "materialize-destination.toml" + destination.write_text( + f"data_home = {json.dumps(str(target.datanode_data_dir))}\n" + "object_store = { type = \"File\" }\n", + encoding="utf-8", + ) + for fixture in fixtures: + fixture_dir = fixture.get("fixture_dir") if isinstance(fixture, dict) else None + if not isinstance(fixture_dir, str): + raise RuntimeError("prepare-direct fixture record has no fixture_dir") + materialize = [ + str(runner), "materialize", "--fixture-dir", fixture_dir, + "--destination", str(destination), + ] + materialize_status = subprocess.run(materialize, check=False).returncode + if materialize_status != 0: + return materialize_status + + for target in targets: + start_component(target, "datanode", procs) + + measure = [ + str(runner), "measure", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--base-http-port", str(targets[0].http_port), + "--candidate-http-port", str(targets[1].http_port), + "--output", str(work_dir / "query-regression-report.json"), + "--http-timeout", str(args.http_timeout), + ] + status = subprocess.run(measure, check=False).returncode + if status != 0: + for target in targets: + restart_component(target, "frontend", procs) + status = subprocess.run(measure, check=False).returncode + return status + finally: + stop_all(targets, procs) + + +def run_remote_case( + args: argparse.Namespace, + case_path: Path, + work_dir: Path, + base_bin: Path, + candidate_bin: Path, + fixture_generator: Path, + runner: Path, +) -> int: + ports = allocate_ports(16) + targets = [ + make_target( + "base", + base_bin, + work_dir, + ports[:8], + work_dir / "base" / "frontend-prom-store.toml", + ), + make_target( + "candidate", + candidate_bin, + work_dir, + ports[8:], + work_dir / "candidate" / "frontend-prom-store.toml", + ), + ] + for target in targets: + if target.work_dir.exists() and any(target.work_dir.iterdir()): + raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}") + target.work_dir.mkdir(parents=True, exist_ok=True) + if target.frontend_config is None: + raise RuntimeError(f"remote target has no frontend config path: {target.name}") + render = [ + str(runner), "render-remote-config", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--output", str(target.frontend_config), + ] + render_status = subprocess.run(render, check=False).returncode + if render_status != 0: + return render_status + + procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {} + report = work_dir / "query-regression-report.json" + try: + for target in targets: + start_component(target, "metasrv", procs) + start_component(target, "datanode", procs) + start_component(target, "frontend", procs) + + prepare = [ + str(runner), "prepare-remote", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--base-http-port", str(targets[0].http_port), + "--candidate-http-port", str(targets[1].http_port), + "--output", str(work_dir / "prepare-remote.json"), + "--http-timeout", str(args.http_timeout), + ] + prepare_status = subprocess.run(prepare, check=False).returncode + if prepare_status != 0: + return prepare_status + + measure = [ + str(runner), "measure", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--base-http-port", str(targets[0].http_port), + "--candidate-http-port", str(targets[1].http_port), + "--output", str(report), "--http-timeout", str(args.http_timeout), + ] + measure_status = subprocess.run(measure, check=False).returncode + if measure_status != 0 and not report.exists(): + return measure_status + + for target in targets: + stop_component(target, "datanode", procs) + + finalize = [ + str(runner), "finalize-remote", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--candidate-bin", str(candidate_bin), + "--base-data-home", str(targets[0].datanode_data_dir), + "--candidate-data-home", str(targets[1].datanode_data_dir), + "--report", str(report), + ] + finalize_status = subprocess.run(finalize, check=False).returncode + if finalize_status != 0: + return finalize_status + final_report = json.loads(report.read_text(encoding="utf-8")) + return 1 if measure_status != 0 or final_report.get("status") == "failed" else 0 + finally: + stop_all(targets, procs) + + +def run_otlp_case( + args: argparse.Namespace, + case_path: Path, + work_dir: Path, + base_bin: Path, + candidate_bin: Path, + fixture_generator: Path, + runner: Path, +) -> int: + if args.otelgen_bin is None: + raise ValueError("--otelgen-bin is required for otlp_trace_load") + ports = allocate_ports(16) + targets = [ + make_target("base", base_bin, work_dir, ports[:8]), + make_target("candidate", candidate_bin, work_dir, ports[8:]), + ] + for target in targets: + if target.work_dir.exists() and any(target.work_dir.iterdir()): + raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}") + target.work_dir.mkdir(parents=True, exist_ok=True) + + procs: dict[tuple[str, str], subprocess.Popen[bytes]] = {} + try: + for target in targets: + try: + start_component(target, "metasrv", procs) + start_component(target, "datanode", procs) + start_component(target, "frontend", procs) + command = [ + str(runner), "run-otlp-target", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--otelgen-bin", str(args.otelgen_bin), + "--http-port", str(target.http_port), + "--target-name", target.name, + "--work-dir", str(target.work_dir), + "--output", str(target.work_dir / "report.json"), + "--http-timeout", str(args.http_timeout), + ] + status = subprocess.run(command, check=False).returncode + finally: + stop_all([target], procs) + if status != 0: + return status + + output = work_dir / "query-regression-report.json" + finalize = [ + str(runner), "finalize-otlp", "--case", str(case_path), + "--fixture-generator", str(fixture_generator), + "--base-result", str(targets[0].work_dir / "report.json"), + "--candidate-result", str(targets[1].work_dir / "report.json"), + "--output", str(output), + ] + status = subprocess.run(finalize, check=False).returncode + if status != 0: + return status + report = json.loads(output.read_text(encoding="utf-8")) + return 1 if report.get("status") == "failed" else 0 + finally: + stop_all(targets, procs) + + def run_case(args: argparse.Namespace, case_path: Path, work_dir: Path) -> int: target_dir = profile_dir(args.cargo_profile) base_bin = args.base_bin or args.base_src / "target" / target_dir / "greptime" candidate_bin = args.candidate_bin or args.candidate_src / "target" / target_dir / "greptime" fixture_generator = args.fixture_generator or args.candidate_src / "target" / target_dir / "query_perf_fixture" - cmd = [ - "uv", - "run", - "--no-project", - "python", - str(args.candidate_src / "tests/perf/query_regression_runner.py"), - "--case", - str(case_path), - "--base-bin", - str(base_bin), - "--candidate-bin", - str(candidate_bin), - "--fixture-generator", - str(fixture_generator), - "--work-dir", - str(work_dir), - "--http-timeout", - str(args.http_timeout), - ] - if parse_bool(args.allow_large_fixture): - cmd.append("--allow-large-fixture") - if args.otelgen_bin is not None: - cmd.extend(["--otelgen-bin", str(args.otelgen_bin)]) - + runner = args.runner or args.candidate_src / "target" / target_dir / "query_regression_runner" + plan = load_plan(fixture_generator, case_path) + scenario = plan.get("scenario") + kind = scenario.get("kind") if isinstance(scenario, dict) else None print(f"::group::Query regression case: {case_path}", flush=True) try: - return subprocess.run(cmd, check=False).returncode + if kind == "direct_readable_sst": + return run_direct_case( + args, + case_path, + work_dir, + base_bin, + candidate_bin, + fixture_generator, + runner, + ) + if kind == "prom_remote_write_then_query": + return run_remote_case( + args, + case_path, + work_dir, + base_bin, + candidate_bin, + fixture_generator, + runner, + ) + if kind == "otlp_trace_load": + return run_otlp_case( + args, + case_path, + work_dir, + base_bin, + candidate_bin, + fixture_generator, + runner, + ) + raise ValueError( + f"unsupported scenario kind {kind!r}; supported: " + "'direct_readable_sst', 'prom_remote_write_then_query', 'otlp_trace_load'" + ) finally: print("::endgroup::", flush=True) @@ -156,6 +604,27 @@ def write_summary(args: argparse.Namespace, reports: list[Path]) -> int: return subprocess.run(cmd, check=False).returncode +def write_failed_report(work_dir: Path, case_path: Path, error: Exception) -> None: + work_dir.mkdir(parents=True, exist_ok=True) + path = work_dir / "query-regression-report.json" + try: + report = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(report, dict) or not isinstance(report.get("targets"), list) or not isinstance(report.get("thresholds"), list): + raise ValueError("existing report is malformed") + except (OSError, ValueError, json.JSONDecodeError): + report = { + "case_path": str(case_path), + "targets": [], + "thresholds": [], + } + report["status"] = "failed" + report["error"] = repr(error) + path.write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + def main() -> int: parser = argparse.ArgumentParser() parser.add_argument("--cases", action="append", help="'all', 'heavy', or comma/space separated case paths") @@ -169,6 +638,7 @@ def main() -> int: default=configured_path(os.environ.get("FIXTURE_GENERATOR")), ) parser.add_argument("--otelgen-bin", type=Path, default=configured_path(os.environ.get("OTELGEN_BIN"))) + parser.add_argument("--runner", type=Path, default=configured_path(os.environ.get("QUERY_REGRESSION_RUNNER"))) parser.add_argument("--cargo-profile", default=os.environ.get("CARGO_PROFILE", "nightly")) parser.add_argument("--work-dir", default=Path("query-regression-work"), type=Path) parser.add_argument("--http-timeout", default=os.environ.get("HTTP_TIMEOUT", "300")) @@ -198,7 +668,12 @@ def main() -> int: case_path = resolve_case_path(args.candidate_src, case) work_dir = args.work_dir / case_slug(case_path) reports.append(work_dir / "query-regression-report.json") - case_status = run_case(args, case_path, work_dir) + try: + case_status = run_case(args, case_path, work_dir) + except Exception as err: # noqa: BLE001 - preserve an aggregate report for the summary step + print(f"error: query regression case {case_path}: {err}", flush=True) + write_failed_report(work_dir, case_path, err) + case_status = 1 if case_status != 0: status = case_status or 1 diff --git a/.github/workflows/develop.yml b/.github/workflows/develop.yml index 1f679137b0..df027a5282 100644 --- a/.github/workflows/develop.yml +++ b/.github/workflows/develop.yml @@ -71,6 +71,14 @@ jobs: version: "latest" - name: Run check-version script tests run: uv run --no-project python .github/scripts/check-version-test.py + - name: Run query-regression Python tooling tests + run: | + uv run --no-project --python 3.12 python tests/perf/test_query_regression_runner_compaction_toctou.py + uv run --no-project --python 3.12 python tests/perf/test_query_regression_runner_otlp_trace_load.py + uv run --no-project --python 3.12 python tests/perf/test_query_regression_summary_otlp.py + uv run --no-project --python 3.12 python tests/perf/test_query_regression_case_selection.py + uv run --no-project --python 3.12 python tests/perf/test_query_regression_nightly_refs.py + uv run --no-project --python 3.12 python tests/perf/test_aliyun_ecs_runner_scripts.py check: if: ${{ github.repository == 'GreptimeTeam/greptimedb' }} diff --git a/.github/workflows/query-regression-janitor.yml b/.github/workflows/query-regression-janitor.yml new file mode 100644 index 0000000000..ce277ce2f0 --- /dev/null +++ b/.github/workflows/query-regression-janitor.yml @@ -0,0 +1,36 @@ +name: Query Regression ECS Janitor + +# Safety net for the aliyun-ecs query-regression path: deletes managed ECS +# instances (and their runner registrations) that outlived their workflow run, +# so a failed teardown never leaks pay-as-you-go spend. + +on: + schedule: + - cron: "23 3 * * *" + workflow_dispatch: + +permissions: + contents: read + +jobs: + sweep: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout teardown script + uses: actions/checkout@v4 + with: + persist-credentials: false + + - name: Install uv + uses: astral-sh/setup-uv@v6 + + - name: Sweep expired ECS runners + env: + ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }} + ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }} + GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }} + run: >- + uv run .github/scripts/aliyun-ecs-runner-teardown.py + --sweep --sweep-ttl-hours 4 diff --git a/.github/workflows/query-regression-nightly.yml b/.github/workflows/query-regression-nightly.yml new file mode 100644 index 0000000000..027fcd1698 --- /dev/null +++ b/.github/workflows/query-regression-nightly.yml @@ -0,0 +1,104 @@ +name: Query Regression Nightly + +# After a successful GreptimeDB Nightly Build, compare that commit against +# the previous successful nightly. The reusable Query Regression workflow +# still compiles both SHAs; this wrapper only resolves which two SHAs. + +on: + workflow_run: + workflows: + - GreptimeDB Nightly Build + types: + - completed + workflow_dispatch: + inputs: + base_ref: + description: Base ref/SHA (empty = previous successful nightly) + required: false + type: string + default: "" + candidate_ref: + description: Candidate ref/SHA (empty = latest successful nightly) + required: false + type: string + default: "" + candidate_run_id: + description: Nightly Build run id to treat as candidate (empty = latest) + required: false + type: string + default: "" + case: + description: Query perf case path(s), all, or heavy + required: false + type: string + default: all + +permissions: + contents: read + actions: read + +jobs: + resolve-refs: + name: Resolve previous vs current nightly SHAs + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 5 + outputs: + skip: ${{ steps.refs.outputs.skip }} + reason: ${{ steps.refs.outputs.reason }} + base_sha: ${{ steps.refs.outputs.base_sha }} + candidate_sha: ${{ steps.refs.outputs.candidate_sha }} + steps: + - name: Checkout ref resolver + uses: actions/checkout@v4 + with: + persist-credentials: false + + - name: Resolve nightly SHAs + id: refs + env: + GITHUB_TOKEN: ${{ github.token }} + GITHUB_REPOSITORY: ${{ github.repository }} + CANDIDATE_RUN_ID: ${{ github.event_name == 'workflow_run' && github.event.workflow_run.id || inputs.candidate_run_id || '' }} + BASE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.base_ref || '' }} + CANDIDATE_REF: ${{ github.event_name == 'workflow_dispatch' && inputs.candidate_ref || '' }} + run: python3 .github/scripts/query-regression-nightly-refs.py + + - name: Summarize comparison + if: always() + env: + SKIP: ${{ steps.refs.outputs.skip }} + REASON: ${{ steps.refs.outputs.reason }} + BASE_SHA: ${{ steps.refs.outputs.base_sha }} + CANDIDATE_SHA: ${{ steps.refs.outputs.candidate_sha }} + BASE_RUN_URL: ${{ steps.refs.outputs.base_run_url }} + CANDIDATE_RUN_URL: ${{ steps.refs.outputs.candidate_run_url }} + run: | + set -euo pipefail + { + if [[ "${SKIP}" == "true" ]]; then + printf 'Skipping query-regression nightly: %s\n' "${REASON}" + else + printf 'Comparing previous nightly `%s` -> current nightly `%s`\n' \ + "${BASE_SHA}" "${CANDIDATE_SHA}" + if [[ -n "${BASE_RUN_URL}" ]]; then + printf -- '- Previous Nightly Build: %s\n' "${BASE_RUN_URL}" + fi + if [[ -n "${CANDIDATE_RUN_URL}" ]]; then + printf -- '- Current Nightly Build: %s\n' "${CANDIDATE_RUN_URL}" + fi + fi + } | tee -a "${GITHUB_STEP_SUMMARY}" + + query-regression: + name: Query regression nightly + needs: [resolve-refs] + if: ${{ needs.resolve-refs.outputs.skip != 'true' && needs.resolve-refs.outputs.base_sha != '' && needs.resolve-refs.outputs.candidate_sha != '' }} + uses: ./.github/workflows/query-regression.yml + secrets: inherit + with: + case: ${{ github.event_name == 'workflow_dispatch' && inputs.case || 'all' }} + base_ref: ${{ needs.resolve-refs.outputs.base_sha }} + candidate_ref: ${{ needs.resolve-refs.outputs.candidate_sha }} + cargo_profile: nightly + runner: aliyun-ecs diff --git a/.github/workflows/query-regression.yml b/.github/workflows/query-regression.yml index fd15e65960..e94cd1f95d 100644 --- a/.github/workflows/query-regression.yml +++ b/.github/workflows/query-regression.yml @@ -32,10 +32,10 @@ on: type: string default: nightly runner: - description: Self-hosted runner label or ARC runner scale set for this query regression run + description: Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance per run required: false type: string - default: perf-regression-8-cores + default: aliyun-ecs workflow_dispatch: inputs: case: @@ -68,12 +68,20 @@ on: - release - dev runner: - description: Self-hosted runner label or ARC runner scale set for this query regression run + description: >- + Self-hosted runner label; aliyun-ecs provisions a fresh ECS instance + per run, any other value is used as a literal runner label required: true - type: choice - default: perf-regression-8-cores - options: - - perf-regression-8-cores + type: string + default: aliyun-ecs + keep_instance: + description: >- + Debug: keep the ECS instance after the run (skip teardown) so its + runner _diag logs, journal, and telemetry can be inspected; the + janitor still sweeps it after the TTL + required: false + type: boolean + default: false pull_request: types: [labeled] @@ -81,19 +89,99 @@ permissions: contents: read jobs: - query-regression: + test-tooling: + # Stdlib unittests for the workflow Python (case selection, report + # helpers, nightly SHA picking, and rendered ECS user-data). They do + # not talk to Aliyun or the Actions runner process; running them on + # ubuntu-latest fails fast before any ECS spend. Not gated on the + # regression labels: those labels boot a VM, these tests should not. + # Ordinary PRs also run the same tests from checks.yml, because this + # workflow only starts on `labeled` (or dispatch / workflow_call). + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + persist-credentials: false + + - name: Test query regression tooling + run: | + python3 tests/perf/test_query_regression_runner_compaction_toctou.py + python3 tests/perf/test_query_regression_runner_otlp_trace_load.py + python3 tests/perf/test_query_regression_summary_otlp.py + python3 tests/perf/test_query_regression_case_selection.py + python3 tests/perf/test_query_regression_nightly_refs.py + python3 tests/perf/test_aliyun_ecs_runner_scripts.py + + provision: + # Runs when the aliyun-ecs path is selected: explicitly via the runner + # input, or for PR labels by default (a QUERY_REGRESSION_PR_RUNNER + # repository variable set to another value redirects PRs to that literal + # runner label instead). Uses trusted scripts from the PR base (or the + # dispatched ref), never from candidate code. Waits for test-tooling so + # a broken user-data template does not still create a VM. + needs: [test-tooling] if: >- - ${{ github.event_name != 'pull_request' || - (github.event_name == 'pull_request' && - !github.event.pull_request.draft && - (github.event.label.name == 'query-regression' || - github.event.label.name == 'heavy-regression')) }} - runs-on: ${{ github.event_name != 'pull_request' && inputs.runner || 'perf-regression-8-cores' }} + ${{ !failure() && !cancelled() && + ((github.event_name != 'pull_request' && inputs.runner == 'aliyun-ecs') || + (github.event_name == 'pull_request' && + !github.event.pull_request.draft && + (github.event.label.name == 'query-regression' || + github.event.label.name == 'heavy-regression') && + (vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs') == 'aliyun-ecs')) }} + runs-on: ubuntu-latest + timeout-minutes: 45 + outputs: + label: ${{ steps.provision.outputs.label }} + instance_id: ${{ steps.provision.outputs.instance_id }} + runner_name: ${{ steps.provision.outputs.runner_name }} + steps: + - name: Checkout trusted provisioning scripts + uses: actions/checkout@v4 + with: + repository: ${{ github.repository }} + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }} + persist-credentials: false + + - name: Install uv + uses: astral-sh/setup-uv@v6 + + - name: Provision ECS runner + id: provision + env: + ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }} + ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }} + GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }} + ALIYUN_ECS_VSWITCH_ID: ${{ vars.ALIYUN_ECS_VSWITCH_ID }} + ALIYUN_ECS_SECURITY_GROUP_ID: ${{ vars.ALIYUN_ECS_SECURITY_GROUP_ID }} + ALIYUN_ECS_INSTANCE_TYPE: ${{ vars.ALIYUN_ECS_INSTANCE_TYPE }} + ALIYUN_ECS_RESOURCE_GROUP_ID: ${{ vars.ALIYUN_ECS_RESOURCE_GROUP_ID }} + QUERY_REGRESSION_ECS_IMAGE_ID: ${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }} + QUERY_REGRESSION_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }} + QUERY_REGRESSION_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }} + run: >- + uv run .github/scripts/aliyun-ecs-runner-provision.py + + query-regression: + needs: [provision, test-tooling] + # `!failure() && !cancelled()` both suppresses the implicit success() and + # lets the job run when provision was intentionally skipped because a + # literal runner label was selected; a failed or cancelled provision still + # blocks the run because no ECS runner would be waiting. + if: >- + ${{ !failure() && !cancelled() && + (github.event_name != 'pull_request' || + (github.event_name == 'pull_request' && + !github.event.pull_request.draft && + (github.event.label.name == 'query-regression' || + github.event.label.name == 'heavy-regression'))) }} + runs-on: >- + ${{ needs.provision.outputs.label || + (github.event_name != 'pull_request' && inputs.runner || + (vars.QUERY_REGRESSION_PR_RUNNER || 'aliyun-ecs')) }} timeout-minutes: 180 - concurrency: - group: query-regression-persistent-cache-v1 - queue: max - cancel-in-progress: false env: CARGO_PROFILE: ${{ github.event_name == 'pull_request' && 'nightly' || inputs.cargo_profile }} CARGO_HOME: /home/runner/.cargo @@ -105,7 +193,8 @@ jobs: QUERY_REGRESSION_CACHE_META: /home/runner/query-regression-cache-meta RUSTC_WRAPPER: /usr/local/bin/sccache SCCACHE_DIR: /home/runner/.cache/sccache - SCCACHE_CACHE_SIZE: 40G + # Caps the local sccache on the system disk next to target and cargo. + SCCACHE_CACHE_SIZE: 10G CARGO_INCREMENTAL: "0" RUSTFLAGS: -D warnings -C link-arg=-fuse-ld=mold QUERY_REGRESSION_CACHE_EPOCH: "1" @@ -114,7 +203,60 @@ jobs: EVENT_MERGE_SHA: ${{ github.event_name == 'pull_request' && github.sha || '' }} EVENT_HEAD_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || '' }} EVENT_BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }} + # Runner identity contract. The ECS image uses 1001; a manually prepared + # host may override via repo variables when 1001 is already taken. + EXPECTED_RUNNER_UID: ${{ vars.QUERY_REGRESSION_RUNNER_UID || '1001' }} + EXPECTED_RUNNER_GID: ${{ vars.QUERY_REGRESSION_RUNNER_GID || '1001' }} steps: + - name: Report provisioned ECS runner + if: ${{ needs.provision.outputs.instance_id != '' }} + shell: bash + run: | + set -euo pipefail + { + printf -- '- ECS instance: `%s` (type `%s`, image `%s`, region `%s`)\n' \ + "${{ needs.provision.outputs.instance_id }}" \ + "${{ vars.ALIYUN_ECS_INSTANCE_TYPE }}" \ + "${{ vars.QUERY_REGRESSION_ECS_IMAGE_ID }}" \ + "${{ vars.ALIYUN_ECS_REGION_ID }}" + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Start machine telemetry sampler + shell: bash + # The ECS instance is deleted when the run ends, so machine state must + # be captured while the job runs. A background sampler appends load, + # memory, disk, and top-memory processes to a log every 30s; the + # "Dump machine telemetry" step (if: always()) prints and uploads it. + # Note: when the job is *cancelled*, even always() steps are killed — + # telemetry covers failures, not cancellations. + run: | + set -euo pipefail + log="${GITHUB_WORKSPACE}/machine-telemetry.log" + { + echo "== baseline $(date -u +%Y-%m-%dT%H:%M:%SZ) ==" + nproc + free -h + df -h / /home/runner + } >> "${log}" + { + printf '## Disk (start)\n\n```\n' + df -hP / /home/runner + printf '```\n' + } >> "${GITHUB_STEP_SUMMARY}" + nohup bash -c " + while true; do + { + date -u '+== %Y-%m-%dT%H:%M:%SZ ==' + uptime + free -m + df -h / /home/runner + ps -eo pid,comm,%mem,%cpu --sort=-%mem | head -6 + } >> '${log}' 2>&1 + sleep 30 + done + " >/dev/null 2>&1 & + echo "Telemetry sampler started (pid $!), logging to ${log}" + - name: Checkout base source uses: actions/checkout@v4 with: @@ -263,14 +405,64 @@ jobs: shell: bash run: | set -euo pipefail - [[ "$(id -u)" == "1001" ]] - [[ "$(id -g)" == "1001" ]] - [[ "${UV_CACHE_DIR}" == "/home/runner/.cargo/uv-cache" ]] - [[ "$(protoc --version)" == "libprotoc 3.21.12" ]] + errors=() + record() { + printf 'FAIL %s\n' "$*" + errors+=("$*") + } + require_eq() { + local name="$1" actual="$2" expected="$3" + printf 'check %s: %s\n' "${name}" "${actual}" + [[ "${actual}" == "${expected}" ]] || record "${name}: expected '${expected}', got '${actual}'" + } + require_match() { + local name="$1" actual="$2" pattern="$3" + printf 'check %s: %s\n' "${name}" "${actual}" + [[ "${actual}" =~ $pattern ]] || record "${name}: expected to match ${pattern}, got '${actual}'" + } + require() { + local name="$1" + shift + printf 'check %s\n' "${name}" + "$@" || record "${name}" + } + capture() { + local out + if out="$("$@" 2>&1)"; then + printf '%s' "${out}" + else + printf '' "$*" + fi + } + + printf 'uid=%s gid=%s PATH=%s CARGO_HOME=%s\n' "$(id -u)" "$(id -g)" "${PATH}" "${CARGO_HOME}" + ls -la "${CARGO_HOME}" 2>&1 || printf '(CARGO_HOME missing)\n' + + # Image/host hygiene: CARGO_HOME must be empty of config and + # install-state before cargo/rustup run. Those tools create + # .package-cache / bin on first use, so this cannot come after. + cargo_home_dirty=() + for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do + if [[ -e "${CARGO_HOME}/${entry}" ]]; then + cargo_home_dirty+=("${entry}") + record "CARGO_HOME must not contain ${entry}" + fi + done + if (( ${#cargo_home_dirty[@]} > 0 )); then + printf 'CARGO_HOME contents:\n' >&2 + ls -la "${CARGO_HOME}" >&2 || true + fi + + require_eq uid "$(id -u)" "${EXPECTED_RUNNER_UID}" + require_eq gid "$(id -g)" "${EXPECTED_RUNNER_GID}" + require_eq UV_CACHE_DIR "${UV_CACHE_DIR}" "/home/runner/.cargo/uv-cache" + require_eq protoc "$(capture protoc --version)" "libprotoc 3.21.12" temporary_proto_dir="$(mktemp --directory)" trap 'rm -rf "${temporary_proto_dir}"' EXIT - test -r /usr/include/google/protobuf/any.proto - test -r /usr/include/google/protobuf/empty.proto + require "readable /usr/include/google/protobuf/any.proto" \ + test -r /usr/include/google/protobuf/any.proto + require "readable /usr/include/google/protobuf/empty.proto" \ + test -r /usr/include/google/protobuf/empty.proto printf '%s\n' \ 'syntax = "proto3";' \ 'package smoke;' \ @@ -278,47 +470,51 @@ jobs: 'import "google/protobuf/empty.proto";' \ 'message Smoke { google.protobuf.Any any = 1; google.protobuf.Empty empty = 2; }' \ > "${temporary_proto_dir}/smoke.proto" - protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \ + if protoc --proto_path="${temporary_proto_dir}" --proto_path=/usr/include \ --descriptor_set_out="${temporary_proto_dir}/smoke.pb" \ - "${temporary_proto_dir}/smoke.proto" - test -s "${temporary_proto_dir}/smoke.pb" - [[ "$(uv --version)" =~ ^uv[[:space:]]0\.11\.26([[:space:]]|$) ]] - mold_version="$(mold --version)" - [[ "${mold_version}" =~ ^mold[[:space:]]2\.30\.0([[:space:]]|$) ]] - [[ "$(python3 --version)" == "Python 3.12.3" ]] + "${temporary_proto_dir}/smoke.proto"; then + require "protoc smoke descriptor is non-empty" test -s "${temporary_proto_dir}/smoke.pb" + else + record "protoc smoke compile failed" + fi + require_match uv "$(capture uv --version)" '^uv[[:space:]]0\.11\.26([[:space:]]|$)' + require_match mold "$(capture mold --version)" '^mold[[:space:]]2\.40\.4([[:space:]]|$)' + require_eq python3 "$(capture python3 --version)" "Python 3.14.4" otelgen_path="$(command -v otelgen || true)" otelgen_version="" if [[ -n "${otelgen_path}" ]]; then - otelgen_version="$("${otelgen_path}" --version 2>&1 || true)" + otelgen_version="$(capture "${otelgen_path}" --version)" fi - printf 'otelgen path: %s\n' "${otelgen_path:-}" - printf 'otelgen version: %s\n' "${otelgen_version}" - [[ "${otelgen_path}" == "/usr/local/bin/otelgen" ]] - [[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]] - sccache_version="$(sccache --version)" - [[ "${sccache_version}" =~ ^sccache[[:space:]]0\.16\.0([[:space:]]|$) ]] - [[ "$(command -v rustup)" == "/opt/cargo/bin/rustup" ]] - [[ "$(command -v cargo)" == "/opt/cargo/bin/cargo" ]] - [[ "$(command -v rustc)" == "/opt/cargo/bin/rustc" ]] - [[ "$(rustup --version)" =~ ^rustup[[:space:]]1\.29\.0([[:space:]]|$) ]] - cargo_version="$(cargo --version)" - [[ "${cargo_version}" =~ ^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]] - rustc_version="$(rustc --version)" - [[ "${rustc_version}" =~ ^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$ ]] - active_toolchain="$(rustup show active-toolchain)" - [[ "${active_toolchain}" =~ ^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$) ]] - [[ "${RUSTUP_HOME}" == "/opt/rustup" ]] - [[ "${RUSTUP_TOOLCHAIN}" == "nightly-2026-03-21" ]] - [[ "${RUSTUP_AUTO_INSTALL}" == "0" ]] - test -r /opt/rustup && test -x /opt/rustup - test ! -w /opt/rustup - test ! -w /opt/cargo/bin - for entry in config config.toml credentials credentials.toml bin .crates.toml .crates2.json .global-cache .package-cache; do - test ! -e "${CARGO_HOME}/${entry}" - done + require_eq otelgen_path "${otelgen_path}" "/usr/local/bin/otelgen" + printf 'check otelgen_version: %s\n' "${otelgen_version}" + [[ "${otelgen_version}" == *"863a3f395d062c7322cc1de08a38774b7fdaa6c8"* ]] \ + || record "otelgen_version: expected commit 863a3f395d062c7322cc1de08a38774b7fdaa6c8, got '${otelgen_version}'" + require_match sccache "$(capture sccache --version)" '^sccache[[:space:]]0\.16\.0([[:space:]]|$)' + require_eq rustup_path "$(command -v rustup || true)" "/opt/cargo/bin/rustup" + require_eq cargo_path "$(command -v cargo || true)" "/opt/cargo/bin/cargo" + require_eq rustc_path "$(command -v rustc || true)" "/opt/cargo/bin/rustc" + require_match rustup "$(capture rustup --version)" '^rustup[[:space:]]1\.29\.0([[:space:]]|$)' + require_match cargo "$(capture cargo --version)" \ + '^cargo[[:space:]]1\.96\.0-nightly[[:space:]]\(cbb9bb8bd[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$' + require_match rustc "$(capture rustc --version)" \ + '^rustc[[:space:]]1\.96\.0-nightly[[:space:]]\(ac7f9ec7d[[:space:]][0-9]{4}-[0-9]{2}-[0-9]{2}\)$' + require_match active_toolchain "$(capture rustup show active-toolchain)" \ + '^nightly-2026-03-21-x86_64-unknown-linux-gnu([[:space:]]|$)' + require_eq RUSTUP_HOME "${RUSTUP_HOME}" "/opt/rustup" + require_eq RUSTUP_TOOLCHAIN "${RUSTUP_TOOLCHAIN}" "nightly-2026-03-21" + require_eq RUSTUP_AUTO_INSTALL "${RUSTUP_AUTO_INSTALL}" "0" + require "readable /opt/rustup" test -r /opt/rustup + require "executable /opt/rustup" test -x /opt/rustup + require "runner cannot write /opt/rustup" test ! -w /opt/rustup + require "runner cannot write /opt/cargo/bin" test ! -w /opt/cargo/bin mkdir -p "${CARGO_HOME}/registry" "${CARGO_HOME}/git" + if (( ${#errors[@]} > 0 )); then + printf '\nVerify runner image tools failed (%d checks):\n' "${#errors[@]}" >&2 + printf ' - %s\n' "${errors[@]}" >&2 + exit 1 + fi - - name: Prepare persistent query regression cache + - name: Prepare query regression cache shell: bash working-directory: src run: | @@ -362,24 +558,6 @@ jobs: find "${root}" -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + } - clear_cargo_extracted_trees() { - require_expected_root CARGO_REGISTRY "${CARGO_HOME}/registry" "${EXPECTED_CARGO_REGISTRY}" - require_expected_root CARGO_GIT "${CARGO_HOME}/git" "${EXPECTED_CARGO_GIT}" - rm -rf -- "${CARGO_HOME}/registry/src" "${CARGO_HOME}/git/checkouts" - } - - cargo_size_kib() { - du -sk -- "${CARGO_HOME}/registry" "${CARGO_HOME}/git" | awk '{ total += $1 } END { print total }' - } - - target_size_kib() { - du -sk -- "${CARGO_TARGET_DIR}" | cut -f1 - } - - free_kib() { - df -Pk "${CARGO_TARGET_DIR}" | awk 'NR == 2 { print $4 }' - } - report_cache_usage() { du -sh -- "${CARGO_HOME}" "${CARGO_HOME}/registry" "${CARGO_HOME}/git" \ "${RUSTUP_HOME}" "${CARGO_TARGET_DIR}" "${QUERY_REGRESSION_CACHE_META}" "${SCCACHE_DIR}" @@ -400,7 +578,7 @@ jobs: printf 'Refusing unexpected RUSTC_WRAPPER: %s\n' "${RUSTC_WRAPPER}" >&2 exit 1 } - [[ "${SCCACHE_CACHE_SIZE}" == "40G" ]] || { + [[ "${SCCACHE_CACHE_SIZE}" == "10G" ]] || { printf 'Refusing unexpected SCCACHE_CACHE_SIZE: %s\n' "${SCCACHE_CACHE_SIZE}" >&2 exit 1 } @@ -487,46 +665,10 @@ jobs: report_cache_usage - target_size="$(target_size_kib)" - if (( target_size >= 400 * 1024 * 1024 )); then - printf 'Warning: target cache is at least 400 GiB (%s KiB)\n' "${target_size}" >&2 - fi - if (( target_size >= 450 * 1024 * 1024 )); then - printf 'Target cache is at least 450 GiB; clearing complete target root\n' >&2 - clear_directory "${CARGO_TARGET_DIR}" - fi - - cargo_size="$(cargo_size_kib)" - if (( cargo_size >= 60 * 1024 * 1024 )); then - printf 'Warning: Cargo cache is at least 60 GiB (%s KiB)\n' "${cargo_size}" >&2 - fi - if (( cargo_size >= 80 * 1024 * 1024 )); then - printf 'Cargo cache is at least 80 GiB; removing extracted sources and checkouts\n' >&2 - clear_cargo_extracted_trees - cargo_size="$(cargo_size_kib)" - if (( cargo_size >= 80 * 1024 * 1024 )); then - printf 'Cargo cache remains at least 80 GiB after cleanup (%s KiB)\n' "${cargo_size}" >&2 - exit 1 - fi - fi - - free_space="$(free_kib)" - if (( free_space < 300 * 1024 * 1024 )); then - printf 'Free space is below 300 GiB; clearing complete target root\n' >&2 - clear_directory "${CARGO_TARGET_DIR}" - free_space="$(free_kib)" - if (( free_space < 300 * 1024 * 1024 )); then - printf 'Free space remains below 300 GiB; removing Cargo extracted sources and checkouts\n' >&2 - clear_cargo_extracted_trees - free_space="$(free_kib)" - if (( free_space < 300 * 1024 * 1024 )); then - printf 'Free space remains below 300 GiB after cleanup (%s KiB)\n' "${free_space}" >&2 - exit 1 - fi - fi - fi - - report_cache_usage + # Every run starts from a fresh system disk, so size/free-space + # watermarks from the retired retained-disk era are not restored. + # A run that overflows the disk fails the build outright, which the + # telemetry step makes diagnosable. sccache --start-server sccache --zero-stats @@ -567,24 +709,17 @@ jobs: git reset --hard "${VERIFIED_CANDIDATE_SHA}" git clean -ffdx - - name: Test query regression tooling - working-directory: src - run: | - uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py - uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py - uv run --no-project python tests/perf/test_query_regression_summary_otlp.py - uv run --no-project python tests/perf/test_query_regression_case_selection.py - - - name: Build candidate greptime and fixture generators + - name: Build candidate greptime and query regression helpers working-directory: src run: | set -euo pipefail SECONDS=0 cargo build --profile "${CARGO_PROFILE}" -p cmd --bin greptime cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_perf_fixture --features dev-tools + cargo build --profile "${CARGO_PROFILE}" -p cmd --bin query_regression_runner --features dev-tools candidate_build_elapsed="${SECONDS}" - printf 'Candidate greptime and fixture generator cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}" - printf -- '- Candidate greptime and fixture generator cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}" + printf 'Candidate greptime and query regression helper cargo builds elapsed: %s seconds\n' "${candidate_build_elapsed}" + printf -- '- Candidate greptime and query regression helper cargo builds: %s seconds\n' "${candidate_build_elapsed}" >> "${GITHUB_STEP_SUMMARY}" sccache --show-stats target_dir="${CARGO_PROFILE}" if [[ "${CARGO_PROFILE}" == "dev" ]]; then @@ -595,6 +730,8 @@ jobs: "${GITHUB_WORKSPACE}/query-regression-bins/candidate/greptime" cp "${CARGO_TARGET_DIR}/${target_dir}/query_perf_fixture" \ "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_perf_fixture" + cp "${CARGO_TARGET_DIR}/${target_dir}/query_regression_runner" \ + "${GITHUB_WORKSPACE}/query-regression-bins/candidate/query_regression_runner" - name: Run query regression id: run @@ -609,6 +746,7 @@ jobs: FIXTURE_GENERATOR: ${{ github.workspace }}/query-regression-bins/candidate/query_perf_fixture OTELGEN_BIN: /usr/local/bin/otelgen SUMMARY_SCRIPT: ${{ github.event_name == 'pull_request' && 'query-regression-trusted-scripts/query-regression-summary.py' || 'src/.github/scripts/query-regression-summary.py' }} + QUERY_REGRESSION_RUNNER: ${{ github.workspace }}/query-regression-bins/candidate/query_regression_runner run: >- uv run --no-project python src/.github/scripts/query-regression-run.py --base-src src @@ -639,6 +777,7 @@ jobs: query-regression-work/**/logs/** query-regression-work/**/otelgen/** query-regression-summary.md + machine-telemetry.log if-no-files-found: warn retention-days: 7 @@ -653,7 +792,7 @@ jobs: if-no-files-found: warn retention-days: 7 - - name: Report persistent cache usage + - name: Report cache usage if: ${{ always() }} shell: bash run: | @@ -687,12 +826,76 @@ jobs: df -P "${report_root}" df -hP "${report_root}" df -Pi "${report_root}" + { + printf '## Disk (end)\n\n```\n' + df -hP / /home/runner "${report_root}" + printf '\n' + du -sh -- \ + /home/runner/.cargo \ + /home/runner/.cargo/registry \ + /home/runner/.cargo/git \ + /opt/rustup \ + /home/runner/query-regression-target \ + /home/runner/query-regression-cache-meta \ + /home/runner/.cache/sccache \ + 2>/dev/null || true + printf '```\n' + } >> "${GITHUB_STEP_SUMMARY}" if command -v sccache >/dev/null 2>&1; then sccache --show-stats || printf 'Unable to show sccache statistics (report only)\n' >&2 else printf 'sccache is unavailable (report only)\n' >&2 fi + - name: Dump machine telemetry + if: ${{ always() }} + shell: bash + run: | + log="${GITHUB_WORKSPACE}/machine-telemetry.log" + if [[ -f "${log}" ]]; then + echo "::group::Machine telemetry (last 200 lines)" + tail -n 200 "${log}" + echo "::endgroup::" + else + echo "No telemetry log found (sampler never started?)" + fi + echo "::group::dmesg tail (OOM killer records)" + sudo dmesg -T 2>/dev/null | tail -n 50 || dmesg -T 2>/dev/null | tail -n 50 || \ + echo "dmesg unavailable without root" + echo "::endgroup::" + - name: Fail on regression failure if: ${{ steps.run.outputs.status != '0' }} run: exit 1 + + teardown: + # Releases the dynamically provisioned ECS runner. Runs even when the + # benchmark job fails or is cancelled; skipped when a literal runner label + # was selected because the provision outputs are empty, or when the + # dispatch set keep_instance to preserve the machine for post-mortem + # debugging (the janitor sweep still reclaims it after the TTL). + if: ${{ always() && needs.provision.outputs.instance_id != '' && !inputs.keep_instance }} + needs: [provision, query-regression] + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout trusted teardown scripts + uses: actions/checkout@v4 + with: + repository: ${{ github.repository }} + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.sha }} + persist-credentials: false + + - name: Install uv + uses: astral-sh/setup-uv@v6 + + - name: Teardown ECS runner + env: + ALIBABA_CLOUD_ACCESS_KEY_ID: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_ID }} + ALIBABA_CLOUD_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUD_ECS_ACCESS_KEY_SECRET }} + GH_PERSONAL_ACCESS_TOKEN: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + ALIYUN_ECS_REGION_ID: ${{ vars.ALIYUN_ECS_REGION_ID }} + QUERY_REGRESSION_ECS_INSTANCE_ID: ${{ needs.provision.outputs.instance_id }} + QUERY_REGRESSION_ECS_RUNNER_NAME: ${{ needs.provision.outputs.runner_name }} + run: >- + uv run .github/scripts/aliyun-ecs-runner-teardown.py diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index ee1754dd6b..7c907ebc34 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -464,7 +464,7 @@ jobs: base_ref: ${{ needs.prepare-release-validation.outputs.previous-release-tag }} candidate_ref: ${{ needs.prepare-release-validation.outputs.candidate-ref }} cargo_profile: nightly - runner: perf-regression-8-cores + runner: aliyun-ecs release-images-to-dockerhub: name: Build and push images to DockerHub diff --git a/Cargo.lock b/Cargo.lock index 327c34d588..756e93a011 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2195,6 +2195,8 @@ dependencies = [ "mito2", "moka", "object-store", + "object_store", + "object_store_opendal", "parquet", "plugins", "pprof", diff --git a/src/cmd/Cargo.toml b/src/cmd/Cargo.toml index 24edcd5633..10865b48a1 100644 --- a/src/cmd/Cargo.toml +++ b/src/cmd/Cargo.toml @@ -14,6 +14,11 @@ name = "query_perf_fixture" path = "src/bin/query_perf_fixture.rs" required-features = ["dev-tools"] +[[bin]] +name = "query_regression_runner" +path = "src/bin/query_regression_runner.rs" +required-features = ["dev-tools"] + [features] default = [ "servers/pprof", @@ -68,6 +73,7 @@ common-wal.workspace = true datafusion.workspace = true datafusion-common.workspace = true datafusion-physical-plan.workspace = true +datafusion_object_store.workspace = true datanode.workspace = true datatypes.workspace = true either = "1.15" @@ -87,6 +93,7 @@ mito-codec.workspace = true mito2.workspace = true moka = { workspace = true, features = ["future"] } object-store.workspace = true +object_store_opendal.workspace = true parquet = { workspace = true, features = ["object_store"] } plugins.workspace = true prometheus.workspace = true diff --git a/src/cmd/src/bin/query_perf_fixture/inspect_footer.rs b/src/cmd/src/bin/query_perf_fixture/inspect_footer.rs index a52005c4e3..4bc9845671 100644 --- a/src/cmd/src/bin/query_perf_fixture/inspect_footer.rs +++ b/src/cmd/src/bin/query_perf_fixture/inspect_footer.rs @@ -13,17 +13,39 @@ // limitations under the License. use std::collections::BTreeSet; -use std::fs::{self, File}; +use std::fs; use std::path::{Path, PathBuf}; +use std::sync::Arc; use clap::Args as ClapArgs; -use parquet::file::reader::{FileReader, SerializedFileReader}; -use serde::Serialize; +use datafusion_object_store::path::Path as StorePath; +use datafusion_object_store::{ObjectMeta, ObjectStore}; +use futures::{StreamExt, TryStreamExt}; +use object_store::config::ObjectStoreConfig; +use object_store::factory::new_raw_object_store; +use object_store::services::Fs; +use parquet::arrow::async_reader::ParquetObjectReader; +use parquet::file::metadata::ParquetMetaDataReader; +use serde::{Deserialize, Serialize}; + +/// Same shape as `query_regression_runner::model::DestinationConfig`. The two +/// binaries are separate crate roots, so the serde-compatible struct is defined +/// here instead of being shared; it reuses the `object-store` crate's +/// `ObjectStoreConfig` deserialization. +#[derive(Debug, Deserialize)] +struct DestinationConfig { + data_home: String, + object_store: ObjectStoreConfig, +} #[derive(Debug, ClapArgs)] pub(super) struct InspectFooterArgs { + /// Local data home (convenience shortcut for a File destination). #[arg(long)] - root: PathBuf, + root: Option, + /// TOML file: data_home = "..." and object_store = { type = "File" | "S3" | ... }. + #[arg(long)] + destination: Option, #[arg(long, default_value = "greptime_value")] column: String, #[arg(long)] @@ -68,64 +90,140 @@ struct FooterColumnChunkReport { num_values: i64, } -pub(super) fn run_inspect_footer( +/// A parquet data file discovered by listing the store, with the metadata needed +/// to read its footer without any extra stat/head call. +#[derive(Debug)] +struct ListedFile { + location: StorePath, + size: u64, + path: String, + relative_path: String, +} + +pub(super) async fn run_inspect_footer( args: InspectFooterArgs, -) -> Result<(), Box> { - let files = collect_parquet_files(&args.root, args.include_metadata_files)?; - let reports = files - .iter() - .map(|p| inspect_file(&args.root, p, &args.column)) - .collect::, _>>()?; +) -> Result<(), Box> { + let (destination, root) = match (args.destination, args.root) { + (Some(_), Some(_)) => { + return Err("--destination and --root are mutually exclusive; pass exactly one".into()); + } + (None, None) => return Err("one of --destination or --root is required".into()), + (Some(path), None) => { + let destination: DestinationConfig = toml::from_str(&fs::read_to_string(path)?)?; + (Some(destination), None) + } + (None, root) => (None, root), + }; + let store = build_store(destination.as_ref(), root.as_deref()).await?; + let root_display = root + .as_deref() + .map(|root| root.display().to_string()) + .unwrap_or_else(|| { + destination + .as_ref() + .expect("destination is set when root is not") + .data_home + .clone() + }); + let files = + collect_parquet_files(store.as_ref(), root.as_deref(), args.include_metadata_files).await?; + let mut reports = futures::stream::iter(files.iter()) + .map(|file| inspect_file(Arc::clone(&store), file, &args.column)) + .buffer_unordered(8) + .try_collect::>() + .await?; + reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path)); println!( "{}", serde_json::to_string_pretty(&FooterReport { - root: args.root.display().to_string(), + root: root_display, summary: summarize_footer(&args.column, &reports), files: reports })? ); Ok(()) } -fn collect_parquet_files( - root: &Path, + +/// Builds an arrow `object_store::ObjectStore` for the given destination. The +/// File backend (and the `--root` shortcut) uses the opendal `Fs` builder +/// directly to avoid the create_dir_all / clean_temp_dir side effects of +/// `object_store::factory::new_fs_object_store`; every other backend goes +/// through `object_store::factory::new_raw_object_store`. +async fn build_store( + destination: Option<&DestinationConfig>, + root: Option<&Path>, +) -> Result, Box> { + let operator = match destination { + Some(destination) => match &destination.object_store { + ObjectStoreConfig::File(_) => { + object_store::ObjectStore::new(Fs::default().root(&destination.data_home))?.finish() + } + _ => new_raw_object_store(&destination.object_store, &destination.data_home).await?, + }, + None => { + let root = root.expect("root must be set when destination is None"); + object_store::ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish() + } + }; + Ok(Arc::new(object_store_opendal::OpendalStore::new(operator))) +} + +async fn collect_parquet_files( + store: &dyn ObjectStore, + root: Option<&Path>, include_metadata_files: bool, -) -> Result, Box> { +) -> Result, Box> { + let metas = store.list(None).try_collect::>().await?; let mut files = Vec::new(); - collect_parquet_files_inner(root, root, include_metadata_files, &mut files)?; - files.sort(); + for meta in metas { + if !is_parquet_data_file(&meta) { + continue; + } + let is_metadata = meta + .location + .parts() + .any(|part| part.as_ref() == "metadata"); + if !include_metadata_files && is_metadata { + continue; + } + let relative_path = meta.location.to_string(); + let path = match root { + Some(root) => root.join(&relative_path).display().to_string(), + None => relative_path.clone(), + }; + files.push(ListedFile { + location: meta.location, + size: meta.size, + path, + relative_path, + }); + } + files.sort_by(|a, b| a.relative_path.cmp(&b.relative_path)); Ok(files) } -fn collect_parquet_files_inner( - root: &Path, - path: &Path, - include_metadata_files: bool, - files: &mut Vec, -) -> Result<(), Box> { - if path.is_dir() { - for entry in fs::read_dir(path)? { - collect_parquet_files_inner(root, &entry?.path(), include_metadata_files, files)?; - } - } else if path.extension().is_some_and(|ext| ext == "parquet") - && (include_metadata_files - || !path - .strip_prefix(root) - .unwrap_or(path) - .components() - .any(|c| c.as_os_str() == "metadata")) - { - files.push(path.to_path_buf()); - } - Ok(()) + +/// Keeps only real parquet data files: `.parquet` keys with a non-zero size. +/// Zero-size keys (and keys ending in `/`) are directory markers emitted by +/// some backends and are skipped even when they happen to end in `.parquet`, +/// since an empty parquet file has no readable footer anyway. +fn is_parquet_data_file(meta: &ObjectMeta) -> bool { + meta.location.extension() == Some("parquet") + && meta.size > 0 + && !meta.location.as_ref().ends_with('/') } -fn inspect_file( - root: &Path, - path: &Path, + +async fn inspect_file( + store: Arc, + file: &ListedFile, column: &str, -) -> Result> { - let file_size = fs::metadata(path)?.len(); - let reader = SerializedFileReader::new(File::open(path)?)?; - let metadata = reader.metadata().file_metadata(); - let row_groups = reader.metadata().row_groups(); +) -> Result> { + let mut reader = + ParquetObjectReader::new(store, file.location.clone()).with_file_size(file.size); + let metadata = ParquetMetaDataReader::new() + .load_and_finish(&mut reader, file.size) + .await?; + let file_metadata = metadata.file_metadata(); + let row_groups = metadata.row_groups(); let mut columns = Vec::new(); for (row_group_index, rg) in row_groups.iter().enumerate() { for (column_index, chunk) in rg.columns().iter().enumerate() { @@ -145,14 +243,10 @@ fn inspect_file( } } Ok(FooterFileReport { - path: path.display().to_string(), - relative_path: path - .strip_prefix(root) - .unwrap_or(path) - .display() - .to_string(), - file_size, - num_rows: metadata.num_rows(), + path: file.path.clone(), + relative_path: file.relative_path.clone(), + file_size: file.size, + num_rows: file_metadata.num_rows(), num_row_groups: row_groups.len(), columns, }) @@ -179,3 +273,199 @@ fn summarize_footer(column: &str, files: &[FooterFileReport]) -> FooterSummary { s.unique_encodings = encodings.into_iter().collect(); s } + +#[cfg(test)] +mod tests { + use std::fs::{self, File}; + + use parquet::column::writer::ColumnWriter; + use parquet::file::properties::WriterProperties; + use parquet::file::writer::SerializedFileWriter; + use parquet::schema::parser::parse_message_type; + use tempfile::TempDir; + + use super::*; + + /// Writes a minimal valid parquet file with a single `value` Int64 column + /// and `rows` values, so the footer can be read back through the object + /// store path. + fn write_parquet_file(path: &Path, rows: i64) { + let schema = Arc::new( + parse_message_type("message schema { REQUIRED INT64 value; }").expect("valid schema"), + ); + let props = Arc::new(WriterProperties::builder().build()); + let file = File::create(path).expect("create parquet file"); + let mut writer = SerializedFileWriter::new(file, schema, props).expect("init writer"); + let mut row_group = writer.next_row_group().expect("next row group"); + let mut column = row_group + .next_column() + .expect("next column") + .expect("has column"); + match column.untyped() { + ColumnWriter::Int64ColumnWriter(typed) => { + typed + .write_batch(&(1..=rows).collect::>(), None, None) + .expect("write batch"); + } + _ => panic!("unexpected column writer"), + } + column.close().expect("close column"); + row_group.close().expect("close row group"); + writer.close().expect("close writer"); + } + + fn fixture_tree(dir: &Path) { + // Nested data tree with a `metadata` subdirectory, a non-parquet file + // and a zero-size fake file. + let table = dir.join("data/table"); + fs::create_dir_all(&table).expect("create table dir"); + fs::create_dir_all(table.join("metadata")).expect("create metadata dir"); + write_parquet_file(&table.join("0001.parquet"), 3); + write_parquet_file(&table.join("0002.parquet"), 5); + write_parquet_file(&table.join("metadata/0001.parquet"), 7); + fs::write(table.join("notes.txt"), b"not parquet").expect("write notes"); + File::create(table.join("empty.parquet")).expect("create empty file"); + } + + /// Runs the full list -> filter -> footer-read path against a File backend + /// rooted at `dir` and returns the collected file reports. + async fn inspect_fixture(dir: &Path, include_metadata_files: bool) -> Vec { + let root = Some(dir.to_path_buf()); + let store = build_store(None, root.as_deref()) + .await + .expect("build store"); + let files = collect_parquet_files(store.as_ref(), root.as_deref(), include_metadata_files) + .await + .expect("collect files"); + let mut reports = futures::stream::iter(files.iter()) + .map(|file| inspect_file(Arc::clone(&store), file, "value")) + .buffer_unordered(8) + .try_collect::>() + .await + .expect("inspect files"); + reports.sort_by(|a, b| a.relative_path.cmp(&b.relative_path)); + reports + } + + #[tokio::test] + async fn lists_and_inspects_footer_files_without_metadata_by_default() { + let dir = TempDir::new().expect("tempdir"); + fixture_tree(dir.path()); + + let reports = inspect_fixture(dir.path(), false).await; + + let relative_paths = reports + .iter() + .map(|report| report.relative_path.as_str()) + .collect::>(); + // metadata/0001.parquet excluded, notes.txt and empty.parquet excluded. + assert_eq!( + relative_paths, + vec!["data/table/0001.parquet", "data/table/0002.parquet"] + ); + assert!(reports.iter().all(|report| report.file_size > 0)); + assert!(reports.iter().all(|report| report.num_row_groups == 1)); + assert!(reports.iter().all(|report| report.num_rows > 0)); + assert!( + reports + .iter() + .all(|report| report.columns.len() == 1 && report.columns[0].column_path == "value") + ); + // Default writer properties use dictionary encoding, so the column chunk + // reports PLAIN (dictionary page) plus RLE_DICTIONARY (data pages). + assert!( + reports + .iter() + .all(|report| report.columns[0].encodings.contains(&"PLAIN".to_string())) + ); + assert!( + reports + .iter() + .all(|report| report.columns[0].compression == "UNCOMPRESSED") + ); + } + + #[tokio::test] + async fn includes_metadata_files_when_requested() { + let dir = TempDir::new().expect("tempdir"); + fixture_tree(dir.path()); + + let reports = inspect_fixture(dir.path(), true).await; + + let relative_paths = reports + .iter() + .map(|report| report.relative_path.as_str()) + .collect::>(); + assert_eq!( + relative_paths, + vec![ + "data/table/0001.parquet", + "data/table/0002.parquet", + "data/table/metadata/0001.parquet" + ] + ); + let metadata_report = reports + .iter() + .find(|report| report.relative_path == "data/table/metadata/0001.parquet") + .expect("metadata report"); + assert_eq!(metadata_report.num_rows, 7); + } + + #[tokio::test] + async fn accepts_a_file_destination_toml() { + let dir = TempDir::new().expect("tempdir"); + fixture_tree(dir.path()); + let destination = dir.path().join("destination.toml"); + fs::write( + &destination, + format!( + "data_home = \"{}\"\n[object_store]\ntype = \"File\"\n", + dir.path().display() + ), + ) + .expect("write destination toml"); + + let destination: DestinationConfig = + toml::from_str(&fs::read_to_string(&destination).expect("read toml")) + .expect("parse toml"); + let store = build_store(Some(&destination), None) + .await + .expect("build store"); + let files = collect_parquet_files(store.as_ref(), None, false) + .await + .expect("collect files"); + + // Destination mode has no root prefix: paths are store-relative keys. + assert_eq!( + files + .iter() + .map(|file| file.relative_path.as_str()) + .collect::>(), + vec!["data/table/0001.parquet", "data/table/0002.parquet"] + ); + assert!(files.iter().all(|file| file.path == file.relative_path)); + } + + #[test] + fn requires_exactly_one_of_root_or_destination() { + use futures::FutureExt; + + let both = run_inspect_footer(InspectFooterArgs { + root: Some(PathBuf::from("/tmp/root")), + destination: Some(PathBuf::from("/tmp/destination.toml")), + column: "value".to_string(), + include_metadata_files: false, + }) + .now_or_never(); + assert!(matches!(both, Some(Err(_)))); + + let neither = run_inspect_footer(InspectFooterArgs { + root: None, + destination: None, + column: "value".to_string(), + include_metadata_files: false, + }) + .now_or_never(); + assert!(matches!(neither, Some(Err(_)))); + } +} diff --git a/src/cmd/src/bin/query_perf_fixture/mod.rs b/src/cmd/src/bin/query_perf_fixture/mod.rs index 1a794c0c3b..17f639bb54 100644 --- a/src/cmd/src/bin/query_perf_fixture/mod.rs +++ b/src/cmd/src/bin/query_perf_fixture/mod.rs @@ -95,9 +95,9 @@ pub async fn run() { Some(Command::PromRemoteWrite(rw)) => run_prom_remote_write(rw) .await .expect("prom remote write failed"), - Some(Command::InspectFooter(inspect)) => { - run_inspect_footer(inspect).expect("inspect footer failed") - } + Some(Command::InspectFooter(inspect)) => run_inspect_footer(inspect) + .await + .expect("inspect footer failed"), Some(Command::Plan(plan)) => run_plan(plan).expect("plan failed"), None => run_direct_sst(args.legacy).await, } diff --git a/src/cmd/src/bin/query_regression_runner.rs b/src/cmd/src/bin/query_regression_runner.rs new file mode 100644 index 0000000000..adc7a2496d --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner.rs @@ -0,0 +1,23 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(clippy::print_stderr, clippy::print_stdout)] + +#[path = "query_regression_runner/mod.rs"] +mod query_regression_runner; + +#[tokio::main] +async fn main() { + query_regression_runner::run().await; +} diff --git a/src/cmd/src/bin/query_regression_runner/direct.rs b/src/cmd/src/bin/query_regression_runner/direct.rs new file mode 100644 index 0000000000..04c191c8c5 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/direct.rs @@ -0,0 +1,501 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::time::{Duration, Instant}; + +use reqwest::Client; +use serde::Serialize; +use serde_json::{Value, json}; + +use crate::query_regression_runner::model::{Scenario, Table}; +use crate::query_regression_runner::plan::{load_plan, validate_direct_tables}; +use crate::query_regression_runner::sql::{ + extract_rows, http_post_sql, row_u64, row_value, sql_ident, sql_string, value_text, +}; +use crate::query_regression_runner::{PrepareDirectArgs, Result}; + +#[derive(Clone, Debug, Serialize)] +struct Discovery { + table: String, + catalog: String, + schema: String, + table_id: u64, + region_id: u64, + region_seq: u64, + table_dir: String, + region_dir: String, + peer_id: u64, + peer_addr: Value, + is_leader: Value, + status: Value, + discovery_queries: Value, +} + +#[derive(Debug)] +struct PreparedTarget { + creates: Vec, + discoveries: Vec, +} + +pub(super) async fn run_prepare_direct(args: PrepareDirectArgs) -> Result<()> { + if !args.http_timeout.is_finite() || args.http_timeout < 0.0 { + return Err("--http-timeout must be a non-negative finite number".into()); + } + let case_path = args.case.canonicalize()?; + let plan = load_plan(&args.fixture_generator, &case_path)?; + let scenario_value = plan + .get("scenario") + .cloned() + .ok_or("fixture plan has no scenario")?; + let scenario: Scenario = serde_json::from_value(scenario_value)?; + let tables = direct_tables(scenario)?; + let client = Client::builder() + .timeout(Duration::from_secs_f64(args.http_timeout)) + .build()?; + let base = prepare_target("base", args.base_http_port, &tables, &client).await?; + let candidate = prepare_target("candidate", args.candidate_http_port, &tables, &client).await?; + if base.discoveries.len() != candidate.discoveries.len() + || base + .discoveries + .iter() + .zip(&candidate.discoveries) + .any(|(base, candidate)| { + base.table != candidate.table + || base.region_id != candidate.region_id + || base.table_dir != candidate.table_dir + }) + { + return Err(format!( + "base/candidate metadata mismatch: base={:?}, candidate={:?}", + base.discoveries, candidate.discoveries + ) + .into()); + } + + let mut fixtures = Vec::with_capacity(tables.len()); + for (index, (table, discovery)) in tables.iter().zip(&base.discoveries).enumerate() { + let fixture_dir = table_fixture_dir(&args.fixture_dir, &tables, table, index); + fixtures.push(generate_direct_fixture( + &args.fixture_generator, + &case_path, + &fixture_dir, + table, + discovery, + tables.len() > 1, + args.allow_large_fixture, + )?); + } + + let report = json!({ + "case_path": case_path, + "scenario": "direct_readable_sst", + "base": { "create_table": base.creates, "discovery": base.discoveries }, + "candidate": { "create_table": candidate.creates, "discovery": candidate.discoveries }, + "fixtures": fixtures, + }); + let text = format!("{}\n", serde_json::to_string_pretty(&report)?); + if let Some(path) = args.output { + fs::write(path, &text)?; + } + print!("{text}"); + Ok(()) +} + +fn direct_tables(scenario: Scenario) -> Result> { + match scenario { + Scenario::DirectReadableSst { tables, layout, .. } => { + validate_direct_tables(tables, layout) + } + Scenario::PromRemoteWriteThenQuery { .. } => { + Err("prepare-direct requires scenario kind direct_readable_sst".into()) + } + Scenario::OtlpTraceLoad { .. } => { + Err("prepare-direct requires scenario kind direct_readable_sst".into()) + } + } +} + +async fn prepare_target( + name: &str, + port: u16, + tables: &[Table], + client: &Client, +) -> Result { + let mut creates = Vec::with_capacity(tables.len()); + let mut discoveries = Vec::with_capacity(tables.len()); + for table in tables { + let sql = create_table_sql(table)?; + let result = http_post_sql(client, port, &sql, &table.database).await; + if !result["ok"].as_bool().unwrap_or(false) { + return Err(format!("CREATE TABLE {} failed for {name}: {result}", table.name).into()); + } + creates.push(json!({ "table": table.name, "sql": sql, "result": result })); + discoveries.push(discover_region(client, port, table).await?); + } + Ok(PreparedTarget { + creates, + discoveries, + }) +} + +fn create_table_sql(table: &Table) -> Result { + if table.engine != "mito" { + return Err("prepare-direct requires table engine mito".into()); + } + let time_index = table + .time_index + .as_deref() + .ok_or("direct-SST table requires time_index")?; + if table.columns.is_empty() { + return Err("direct-SST table requires columns".into()); + } + let columns = table + .columns + .iter() + .map(|column| format!("{} {}", sql_ident(&column.name), column.data_type)) + .collect::>() + .join(",\n "); + let primary_key = table + .primary_key + .iter() + .map(|column| sql_ident(column)) + .collect::>() + .join(", "); + let mut options = Vec::new(); + if let Some(append_mode) = table.append_mode { + options.push(("append_mode", append_mode.to_string())); + } + if let Some(sst_format) = table + .sst_format + .as_deref() + .filter(|value| !value.is_empty()) + { + options.push(("sst_format", sst_format.to_string())); + } + let with = (!options.is_empty()).then(|| { + format!( + "\nWITH ({})", + options + .iter() + .map(|(key, value)| format!("'{key}'='{value}'")) + .collect::>() + .join(", ") + ) + }); + Ok(format!( + "CREATE TABLE {} (\n {columns},\n TIME INDEX ({}),\n PRIMARY KEY ({primary_key})\n) ENGINE=mito{};", + sql_ident(&table.name), + sql_ident(time_index), + with.unwrap_or_default() + )) +} + +async fn discover_region(client: &Client, port: u16, table: &Table) -> Result { + let schema = sql_string(&table.database); + let table_name = sql_string(&table.name); + let table_sql = format!( + "SELECT table_id FROM information_schema.tables WHERE table_schema = {schema} AND table_name = {table_name}" + ); + let table_result = http_post_sql(client, port, &table_sql, &table.database).await; + if !table_result["ok"].as_bool().unwrap_or(false) { + return Err(format!("table_id discovery failed: {table_result}").into()); + } + let table_rows = extract_rows(table_result.get("response").unwrap_or(&Value::Null)); + if table_rows.len() != 1 { + return Err(format!( + "expected one information_schema.tables row, got {}: {table_result}", + table_rows.len() + ) + .into()); + } + let table_id = row_u64(&table_rows[0], 0, "table_id")?; + + let region_sql = format!( + "SELECT region_id, peer_id, peer_addr, is_leader, status FROM information_schema.region_peers WHERE table_schema = {schema} AND table_name = {table_name}" + ); + let region_result = http_post_sql(client, port, ®ion_sql, &table.database).await; + if !region_result["ok"].as_bool().unwrap_or(false) { + return Err(format!("region_peers discovery failed: {region_result}").into()); + } + let region_rows = extract_rows(region_result.get("response").unwrap_or(&Value::Null)); + if region_rows.len() != 1 { + return Err(format!( + "expected one information_schema.region_peers row, got {}: {region_result}", + region_rows.len() + ) + .into()); + } + let row = ®ion_rows[0]; + let region_id = row_u64(row, 0, "region_id")?; + let peer_id = row_u64(row, 1, "peer_id")?; + let peer_addr = row_value(row, 2, "peer_addr") + .cloned() + .unwrap_or(Value::Null); + let is_leader = row_value(row, 3, "is_leader") + .cloned() + .unwrap_or(Value::Null); + let status = row_value(row, 4, "status").cloned().unwrap_or(Value::Null); + if !matches!( + value_text(&is_leader).to_ascii_lowercase().as_str(), + "yes" | "true" + ) { + return Err(format!( + "expected leader region peer, got is_leader={is_leader}: {region_result}" + ) + .into()); + } + if !value_text(&status).eq_ignore_ascii_case("ALIVE") { + return Err( + format!("expected ALIVE region peer, got status={status}: {region_result}").into(), + ); + } + if peer_id != 0 { + return Err(format!("expected region leader on datanode peer_id=0, got {peer_id}").into()); + } + if region_id >> 32 != table_id { + return Err(format!( + "table_id mismatch: tables={table_id}, region_id-derived={}", + region_id >> 32 + ) + .into()); + } + let region_seq = region_id & u32::MAX as u64; + let (table_dir, region_dir) = storage_paths(&table.database, table_id, region_seq); + Ok(Discovery { + table: table.name.clone(), + catalog: "greptime".to_string(), + schema: table.database.clone(), + table_id, + region_id, + region_seq, + table_dir: table_dir.clone(), + region_dir, + peer_id, + peer_addr, + is_leader, + status, + discovery_queries: json!({ "table": table_result, "region": region_result }), + }) +} + +fn storage_paths(database: &str, table_id: u64, region_seq: u64) -> (String, String) { + let table_dir = format!("data/greptime/{database}/{table_id}/"); + let region_dir = format!("{table_dir}{table_id}_{region_seq:010}"); + (table_dir, region_dir) +} + +fn table_fixture_dir(root: &Path, tables: &[Table], table: &Table, index: usize) -> PathBuf { + if tables.len() == 1 { + root.to_path_buf() + } else { + root.join(fixture_subdir(table, index)) + } +} + +fn fixture_subdir(table: &Table, index: usize) -> String { + let raw = format!("{index:02}_{}_{}", table.database, table.name); + let mut safe = String::new(); + let mut replaced = false; + for character in raw.chars() { + if character.is_ascii_alphanumeric() || matches!(character, '_' | '.' | '-') { + safe.push(character); + replaced = false; + } else if !replaced { + safe.push('_'); + replaced = true; + } + } + let safe = safe.trim_matches(['.', '_', '-']); + if safe.is_empty() { + format!("table_{index:02}") + } else { + safe.to_string() + } +} + +fn generate_direct_fixture( + generator: &Path, + case_path: &Path, + fixture_dir: &Path, + table: &Table, + discovery: &Discovery, + multi_table: bool, + allow_large_fixture: bool, +) -> Result { + if fixture_dir.exists() { + fs::remove_dir_all(fixture_dir)?; + } + fs::create_dir_all(fixture_dir)?; + let mut command = vec![ + generator.to_string_lossy().to_string(), + "direct-sst".to_string(), + "--case".to_string(), + case_path.to_string_lossy().to_string(), + "--out-dir".to_string(), + fixture_dir.to_string_lossy().to_string(), + ]; + if multi_table { + command.extend(["--table".to_string(), table.name.clone()]); + } + command.extend([ + "--region-id".to_string(), + discovery.region_id.to_string(), + "--table-dir".to_string(), + discovery.table_dir.clone(), + ]); + if allow_large_fixture { + command.push("--allow-large".to_string()); + } + let started = Instant::now(); + let output = Command::new(generator).args(&command[1..]).output()?; + let elapsed_seconds = started.elapsed().as_secs_f64(); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + if !output.status.success() { + return Err(format!( + "fixture generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}", + output.status.code(), stderr + ) + .into()); + } + let summary: Value = serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?; + assert_fixture_summary(&summary, table, discovery)?; + Ok(json!({ + "status": "ok", + "fixture_dir": fixture_dir, + "command": command, + "returncode": output.status.code(), + "elapsed_seconds": elapsed_seconds, + "stdout": stdout, + "stderr": stderr, + "summary": summary, + })) +} + +fn assert_fixture_summary(summary: &Value, table: &Table, discovery: &Discovery) -> Result<()> { + if summary.get("table").and_then(Value::as_str) != Some(&table.name) { + return Err(format!( + "fixture table mismatch: {:?} != {}", + summary.get("table"), + table.name + ) + .into()); + } + if summary.get("database").and_then(Value::as_str) != Some(&table.database) { + return Err(format!( + "fixture database mismatch: {:?} != {}", + summary.get("database"), + table.database + ) + .into()); + } + if summary.get("region_id").and_then(|value| { + value + .as_u64() + .or_else(|| value.as_str().and_then(|value| value.parse().ok())) + }) != Some(discovery.region_id) + { + return Err(format!( + "fixture region_id mismatch: {:?} != {}", + summary.get("region_id"), + discovery.region_id + ) + .into()); + } + if summary.get("table_dir").and_then(Value::as_str) != Some(&discovery.table_dir) { + return Err(format!( + "fixture table_dir mismatch: {:?} != {}", + summary.get("table_dir"), + discovery.table_dir + ) + .into()); + } + if summary + .get("region_dir") + .and_then(Value::as_str) + .map(|value| value.trim_matches('/')) + != Some(discovery.region_dir.trim_matches('/')) + { + return Err(format!( + "fixture region_dir mismatch: {:?} != {}", + summary.get("region_dir"), + discovery.region_dir + ) + .into()); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::query_regression_runner::model::Column; + + #[test] + fn creates_exact_direct_sst_table_sql() { + let table = Table { + database: "public".to_string(), + name: "metric\"name".to_string(), + engine: "mito".to_string(), + columns: vec![ + Column { + name: "host".to_string(), + data_type: "STRING".to_string(), + }, + Column { + name: "ts".to_string(), + data_type: "TIMESTAMP(9)".to_string(), + }, + ], + primary_key: vec!["host".to_string()], + time_index: Some("ts".to_string()), + append_mode: Some(true), + sst_format: Some("flat".to_string()), + validate_show_create_engine: true, + }; + assert_eq!( + create_table_sql(&table).unwrap(), + "CREATE TABLE \"metric\"\"name\" (\n \"host\" STRING,\n \"ts\" TIMESTAMP(9),\n TIME INDEX (\"ts\"),\n PRIMARY KEY (\"host\")\n) ENGINE=mito\nWITH ('append_mode'='true', 'sst_format'='flat');" + ); + } + + #[test] + fn extracts_named_and_positional_discovery_rows() { + let rows = extract_rows(&json!({"output": [{"data": [{"TABLE_ID": "7"}]}]})); + assert_eq!(rows, vec![json!({"TABLE_ID": "7"})]); + assert_eq!(row_u64(&rows[0], 0, "table_id").unwrap(), 7); + let rows = extract_rows(&json!({"data": [[30064771073_u64, 0, "addr", "YES", "ALIVE"]]})); + assert_eq!(rows.len(), 1); + assert_eq!(row_u64(&rows[0], 0, "region_id").unwrap(), 30_064_771_073); + assert_eq!(row_u64(&rows[0], 1, "peer_id").unwrap(), 0); + assert_eq!( + value_text(row_value(&rows[0], 3, "is_leader").unwrap()), + "YES" + ); + } + + #[test] + fn storage_paths_include_the_table_directory() { + assert_eq!( + storage_paths("public", 1024, 0), + ( + "data/greptime/public/1024/".to_string(), + "data/greptime/public/1024/1024_0000000000".to_string(), + ) + ); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/materialize.rs b/src/cmd/src/bin/query_regression_runner/materialize.rs new file mode 100644 index 0000000000..3cc9e1aa07 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/materialize.rs @@ -0,0 +1,225 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::path::Path; + +use futures::{AsyncWriteExt as _, TryStreamExt}; +use object_store::ObjectStore; +use object_store::factory::new_raw_object_store; +use object_store::services::Fs; + +use crate::query_regression_runner::model::{ + CopyCounts, DestinationConfig, FixtureSummary, MaterializeResult, +}; +use crate::query_regression_runner::{MaterializeArgs, Result}; + +pub(super) async fn run_materialize(args: MaterializeArgs) -> Result<()> { + let destination: DestinationConfig = toml::from_str(&fs::read_to_string(args.destination)?)?; + let result = materialize(&args.fixture_dir, destination).await?; + println!("{}", serde_json::to_string(&result)?); + Ok(()) +} + +async fn materialize( + fixture_dir: &Path, + destination: DestinationConfig, +) -> Result { + let summary: FixtureSummary = + serde_json::from_slice(&fs::read(fixture_dir.join("summary.json"))?)?; + let region_dir = validate_region_dir(&summary.region_dir)?; + let object_source = fs_operator(&fixture_dir.join("object-store"))?; + let manifest_source = fs_operator(&fixture_dir.join("manifest"))?; + let destination = + new_raw_object_store(&destination.object_store, &destination.data_home).await?; + + let region_prefix = format!("{region_dir}/"); + destination + .delete_with(®ion_prefix) + .recursive(true) + .await?; + + let object_store = copy_tree(&object_source, &destination, "/", "").await?; + let manifest = copy_tree( + &manifest_source, + &destination, + "/", + &format!("{region_dir}/manifest/"), + ) + .await?; + Ok(MaterializeResult { + region_dir, + object_store, + manifest, + }) +} + +fn fs_operator(root: &Path) -> Result { + Ok(ObjectStore::new(Fs::default().root(&root.to_string_lossy()))?.finish()) +} + +fn validate_region_dir(region_dir: &str) -> Result { + let region_dir = region_dir.trim_end_matches('/'); + if region_dir.is_empty() + || region_dir.starts_with('/') + || region_dir.contains('\\') + || region_dir + .split('/') + .any(|component| component.is_empty() || matches!(component, "." | "..")) + { + return Err( + "region_dir must be a non-empty relative OpenDAL key without dot components".into(), + ); + } + Ok(region_dir.to_string()) +} + +async fn copy_tree( + source: &ObjectStore, + destination: &ObjectStore, + source_prefix: &str, + destination_prefix: &str, +) -> Result { + let mut lister = source.lister_with(source_prefix).recursive(true).await?; + let mut counts = CopyCounts::default(); + while let Some(entry) = lister.try_next().await? { + if entry.metadata().is_dir() { + continue; + } + let source_path = entry.path().to_string(); + let reader = source + .reader(&source_path) + .await? + .into_futures_async_read(0..entry.metadata().content_length()) + .await?; + let mut writer = destination + .writer(&format!("{destination_prefix}{source_path}")) + .await? + .into_futures_async_write(); + let bytes = futures::io::copy(reader, &mut writer).await?; + writer.close().await?; + counts.files += 1; + counts.bytes += bytes; + } + Ok(counts) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use object_store::config::ObjectStoreConfig; + + use super::*; + + #[test] + fn region_dir_must_be_a_relative_opendal_key() { + assert_eq!( + validate_region_dir("data/db/region/").unwrap(), + "data/db/region" + ); + for region_dir in [ + "", + "/region", + ".", + "a/../region", + "a/./region", + "a\\region", + "a//region", + ] { + assert!(validate_region_dir(region_dir).is_err(), "{region_dir}"); + } + } + + #[tokio::test] + async fn materializes_fixture_from_fs_to_fs() { + let fixture = tempfile::tempdir().unwrap(); + let destination = tempfile::tempdir().unwrap(); + let region_dir = "data/public/metrics/00000000000000000001"; + let sst = fixture + .path() + .join("object-store") + .join(region_dir) + .join("00000000000000000001.parquet"); + fs::create_dir_all(sst.parent().unwrap()).unwrap(); + fs::write(&sst, b"sst").unwrap(); + let checkpoint = fixture + .path() + .join("manifest/00000000000000000001.checkpoint"); + fs::create_dir_all(checkpoint.parent().unwrap()).unwrap(); + fs::write(&checkpoint, b"checkpoint").unwrap(); + fs::write(fixture.path().join("manifest/_last_checkpoint"), b"last").unwrap(); + fs::write( + fixture.path().join("summary.json"), + format!(r#"{{"region_dir":"{region_dir}"}}"#), + ) + .unwrap(); + fs::write(fixture.path().join("files.jsonl"), "metadata").unwrap(); + fs::create_dir_all(destination.path().join(region_dir)).unwrap(); + fs::write(destination.path().join(region_dir).join("stale"), "stale").unwrap(); + fs::write(destination.path().join("unrelated"), "keep").unwrap(); + + let result = materialize( + fixture.path(), + DestinationConfig { + data_home: destination.path().to_string_lossy().to_string(), + object_store: ObjectStoreConfig::default(), + }, + ) + .await + .unwrap(); + + assert_eq!(result.object_store.files, 1); + assert_eq!(result.object_store.bytes, 3); + assert_eq!(result.manifest.files, 2); + assert_eq!(result.manifest.bytes, 14); + assert_eq!( + fs::read( + destination + .path() + .join(region_dir) + .join("00000000000000000001.parquet") + ) + .unwrap(), + b"sst" + ); + assert_eq!( + fs::read( + destination + .path() + .join(region_dir) + .join("manifest/00000000000000000001.checkpoint") + ) + .unwrap(), + b"checkpoint" + ); + assert_eq!( + fs::read( + destination + .path() + .join(region_dir) + .join("manifest/_last_checkpoint") + ) + .unwrap(), + b"last" + ); + assert!(!destination.path().join(region_dir).join("stale").exists()); + assert_eq!( + fs::read(destination.path().join("unrelated")).unwrap(), + b"keep" + ); + assert!(!destination.path().join("summary.json").exists()); + assert!(!destination.path().join("files.jsonl").exists()); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/measure.rs b/src/cmd/src/bin/query_regression_runner/measure.rs new file mode 100644 index 0000000000..2b389f7d78 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/measure.rs @@ -0,0 +1,423 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::time::Duration; +use std::{fs, io}; + +use reqwest::Client; +use serde_json::{Map, Value, json}; + +use crate::query_regression_runner::model::{Measurement, Query, QueryResult, Scenario, Table}; +use crate::query_regression_runner::plan::{load_plan, normalize_scenario}; +use crate::query_regression_runner::sql::{http_post_sql, sql_ident}; +use crate::query_regression_runner::{MeasureArgs, Result}; + +pub(super) async fn run_measure(args: MeasureArgs) -> Result<()> { + if !args.http_timeout.is_finite() || args.http_timeout < 0.0 { + return Err("--http-timeout must be a non-negative finite number".into()); + } + + let case_path = args.case.canonicalize()?; + let plan = load_plan(&args.fixture_generator, &case_path)?; + let case_text = fs::read_to_string(&case_path)?; + let raw_case: toml::Value = toml::from_str(&case_text)?; + let case_metadata = raw_case + .get("case") + .cloned() + .unwrap_or(toml::Value::Table(toml::map::Map::new())); + let case_metadata = serde_json::to_value(case_metadata)?; + + let scenario_value = plan + .get("scenario") + .cloned() + .ok_or("fixture plan has no scenario")?; + let scenario: Scenario = serde_json::from_value(scenario_value.clone())?; + let (tables, configured_queries) = normalize_scenario(scenario)?; + + let client = Client::builder() + .timeout(Duration::from_secs_f64(args.http_timeout)) + .build()?; + let base = run_target(args.base_http_port, &tables, &configured_queries, &client).await; + let candidate = run_target( + args.candidate_http_port, + &tables, + &configured_queries, + &client, + ) + .await; + let thresholds = enforce_thresholds(&configured_queries, &base, &candidate)?; + let status = if base.status == "failed" + || candidate.status == "failed" + || thresholds + .iter() + .any(|threshold| threshold["status"] == "failed") + { + "failed" + } else { + "ok" + }; + let report = json!({ + "case_path": case_path, + "case": case_metadata, + "scenario": scenario_value, + "queries": configured_queries, + "query_mode": "endpoint", + "http_timeout": args.http_timeout, + "targets": [target_report("base", args.base_http_port, base), target_report("candidate", args.candidate_http_port, candidate)], + "thresholds": thresholds, + "status": status, + }); + let text = format!("{}\n", serde_json::to_string_pretty(&report)?); + if let Some(path) = args.output { + fs::write(path, &text)?; + } + print!("{text}"); + if status == "failed" { + std::process::exit(1); + } + Ok(()) +} + +fn target_report(name: &str, http_port: u16, result: QueryResult) -> Value { + let status = if result.status == "ok" { + "measured" + } else { + "failed" + }; + json!({ + "name": name, + "http_port": http_port, + "validation": result.validation, + "validation_errors": result.validation_errors, + "measurements": result.measurements, + "status": status, + }) +} + +async fn run_target( + port: u16, + tables: &[Table], + configured_queries: &[Query], + client: &Client, +) -> QueryResult { + let mut queries = configured_queries.to_vec(); + if queries.is_empty() { + queries.push(Query { + name: Some("count_all".to_string()), + kind: Some("sql".to_string()), + query: format!("SELECT count(*) FROM {}", sql_ident(&tables[0].name)), + warmup: 0, + iterations: 1, + thresholds: Map::new(), + }); + } + + let db = &tables[0].database; + let mut validation = Vec::new(); + let mut validation_errors = Vec::new(); + for table in tables { + let sql = format!("SHOW CREATE TABLE {}", sql_ident(&table.name)); + let sample = http_post_sql(client, port, &sql, &table.database).await; + if !sample["ok"].as_bool().unwrap_or(false) { + validation_errors.push(json!({ + "sql": sql, + "error": sample.get("error"), + "response": sample.get("response"), + })); + } else { + for error in validate_show_create(&sample, table) { + validation_errors.push(json!({ + "sql": sql, + "error": error, + "response": sample.get("response"), + })); + } + } + validation.push(sample); + } + let first = http_post_sql(client, port, &queries[0].query, db).await; + if !first["ok"].as_bool().unwrap_or(false) { + validation_errors.push(json!({ + "sql": queries[0].query, + "error": first.get("error"), + "response": first.get("response"), + })); + } + validation.push(first); + + let mut measurements = Vec::with_capacity(queries.len()); + for query in &queries { + for _ in 0..query.warmup { + let warmup = http_post_sql(client, port, &query.query, db).await; + if !warmup["ok"].as_bool().unwrap_or(false) { + validation_errors.push(json!({ + "sql": query.query, + "phase": "warmup", + "error": warmup.get("error"), + "response": warmup.get("response"), + })); + } + } + let mut samples = Vec::with_capacity(query.iterations); + let mut good_latencies = Vec::with_capacity(query.iterations); + for _ in 0..query.iterations { + let mut sample = http_post_sql(client, port, &query.query, db).await; + let execution_time = sample + .get("response") + .and_then(extract_execution_time) + .cloned() + .unwrap_or(Value::Null); + sample + .as_object_mut() + .expect("HTTP samples are objects") + .insert("execution_time_ms".to_string(), execution_time); + if sample["ok"].as_bool().unwrap_or(false) { + good_latencies.push(sample["latency_ms"].as_f64().unwrap_or_default()); + } + samples.push(sample); + } + let median = (!good_latencies.is_empty()).then(|| median(&good_latencies)); + let p95 = (!good_latencies.is_empty()).then(|| percentile(&good_latencies, 95.0)); + let status = if good_latencies.len() == samples.len() { + "ok" + } else { + "failed" + }; + measurements.push(Measurement { + name: query.name.clone(), + kind: query.kind.clone(), + iterations: samples.len(), + samples, + latency_ms_median: median, + latency_ms_p95: p95, + status: status.to_string(), + }); + } + let failed = !validation_errors.is_empty() || measurements.iter().any(|m| m.status == "failed"); + QueryResult { + validation, + validation_errors, + measurements, + status: if failed { "failed" } else { "ok" }.to_string(), + } +} + +fn validate_show_create(result: &Value, table: &Table) -> Vec<&'static str> { + let text = result + .get("response") + .map(response_text) + .unwrap_or_default() + .to_lowercase(); + let mut errors = Vec::new(); + if !text.contains(&table.name.to_lowercase()) { + errors.push("SHOW CREATE output does not contain table name"); + } + if table.validate_show_create_engine && (!text.contains("engine") || !text.contains("mito")) { + errors.push("SHOW CREATE output does not mention ENGINE=mito"); + } + if table.append_mode.is_some() && !text.contains("append_mode") { + errors.push("SHOW CREATE output does not mention append_mode"); + } + if table.sst_format.is_some() && !text.contains("sst_format") { + errors.push("SHOW CREATE output does not mention sst_format"); + } + errors +} + +fn response_text(body: &Value) -> String { + body.as_str() + .map(ToOwned::to_owned) + .unwrap_or_else(|| serde_json::to_string(body).unwrap_or_default()) +} + +fn extract_execution_time(body: &Value) -> Option<&Value> { + match body { + Value::Object(map) => { + for key in ["execution_time_ms", "execution_time", "elapsed"] { + if let Some(value) = map.get(key) { + return Some(value); + } + } + map.values().find_map(extract_execution_time) + } + Value::Array(values) => values.iter().find_map(extract_execution_time), + _ => None, + } +} + +pub(super) fn median(values: &[f64]) -> f64 { + let mut ordered = values.to_vec(); + ordered.sort_by(f64::total_cmp); + let middle = ordered.len() / 2; + if ordered.len().is_multiple_of(2) { + (ordered[middle - 1] + ordered[middle]) / 2.0 + } else { + ordered[middle] + } +} + +fn percentile(values: &[f64], pct: f64) -> f64 { + if values.is_empty() { + return 0.0; + } + let mut ordered = values.to_vec(); + ordered.sort_by(f64::total_cmp); + let index = round_ties_even((pct / 100.0) * (ordered.len() - 1) as f64) + .clamp(0, ordered.len() as isize - 1) as usize; + ordered[index] +} + +pub(super) fn round_ties_even(value: f64) -> isize { + let floor = value.floor(); + let fraction = value - floor; + if fraction < 0.5 { + floor as isize + } else if fraction > 0.5 { + floor as isize + 1 + } else if (floor as isize) % 2 == 0 { + floor as isize + } else { + floor as isize + 1 + } +} + +fn enforce_thresholds( + queries: &[Query], + base: &QueryResult, + candidate: &QueryResult, +) -> Result> { + let base_by_name: HashMap<_, _> = base + .measurements + .iter() + .map(|measurement| (measurement.name.as_deref(), measurement)) + .collect(); + let mut results = Vec::new(); + for candidate_measurement in &candidate.measurements { + let query = queries + .iter() + .find(|query| query.name == candidate_measurement.name); + let thresholds = query.map_or_else(Map::new, |query| query.thresholds.clone()); + let base_measurement = base_by_name.get(&candidate_measurement.name.as_deref()); + if let Some(limit) = thresholds + .get("max_candidate_latency_regression_pct") + .filter(|value| !value.is_null()) + .map(value_as_f64) + .transpose()? + { + let result = match base_measurement { + None => { + json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing base measurement"}) + } + Some(base) if matches!(base.latency_ms_median, None | Some(0.0)) => { + json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "base median latency is missing or zero", "base_latency_ms_median": base.latency_ms_median}) + } + Some(_) if candidate_measurement.latency_ms_median.is_none() => { + json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing candidate measurement"}) + } + Some(base) => { + let actual = (candidate_measurement.latency_ms_median.unwrap() + - base.latency_ms_median.unwrap()) + / base.latency_ms_median.unwrap() + * 100.0; + json!({"query": candidate_measurement.name, "threshold": "max_candidate_latency_regression_pct", "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit}) + } + }; + results.push(result); + } + for key in thresholds + .keys() + .filter(|key| *key != "max_candidate_latency_regression_pct") + { + results.push(json!({"query": candidate_measurement.name, "threshold": key, "status": "failed", "reason": "unsupported threshold"})); + } + } + Ok(results) +} + +fn value_as_f64(value: &Value) -> Result { + value.as_f64().ok_or_else(|| { + io::Error::new(io::ErrorKind::InvalidData, "threshold must be numeric").into() + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn median_and_p95_match_python() { + assert_eq!(median(&[1.0, 8.0, 3.0, 4.0]), 3.5); + assert_eq!( + percentile( + &[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0], + 95.0 + ), + 10.0 + ); + assert_eq!(percentile(&[], 95.0), 0.0); + } + + #[test] + fn finds_nested_execution_time_in_priority_order() { + let body = json!({"output": [{"elapsed": 3}], "execution_time": 2}); + assert_eq!(extract_execution_time(&body), Some(&json!(2))); + assert_eq!( + extract_execution_time(&json!({"output": [{"elapsed": 3}]})), + Some(&json!(3)) + ); + } + + #[test] + fn threshold_rejects_unknown_keys_and_zero_base() { + let query = Query { + name: Some("q".to_string()), + kind: None, + query: "SELECT 1".to_string(), + warmup: 0, + iterations: 1, + thresholds: Map::from_iter([ + ("max_candidate_latency_regression_pct".to_string(), json!(0)), + ("other".to_string(), json!(1)), + ]), + }; + let measurement = |median| Measurement { + name: Some("q".to_string()), + kind: None, + iterations: 1, + samples: vec![], + latency_ms_median: median, + latency_ms_p95: median, + status: "ok".to_string(), + }; + let base = QueryResult { + validation: vec![], + validation_errors: vec![], + measurements: vec![measurement(Some(0.0))], + status: "ok".to_string(), + }; + let candidate = QueryResult { + validation: vec![], + validation_errors: vec![], + measurements: vec![measurement(Some(1.0))], + status: "ok".to_string(), + }; + let results = enforce_thresholds(&[query], &base, &candidate).unwrap(); + assert_eq!( + results[0]["reason"], + "base median latency is missing or zero" + ); + assert_eq!(results[1]["reason"], "unsupported threshold"); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/mod.rs b/src/cmd/src/bin/query_regression_runner/mod.rs new file mode 100644 index 0000000000..9ed903f5e0 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/mod.rs @@ -0,0 +1,204 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::error::Error; +use std::path::PathBuf; + +use clap::{Parser, Subcommand}; + +mod direct; +mod materialize; +mod measure; +mod model; +mod otlp; +mod plan; +mod remote; +mod sql; + +pub(super) type Result = std::result::Result>; + +#[derive(Debug, Parser)] +#[command(about = "Measure query endpoints or materialize a direct-SST fixture")] +struct Cli { + #[command(subcommand)] + command: RunnerCommand, +} + +#[derive(Debug, Subcommand)] +enum RunnerCommand { + /// Run normalized query regression cases against two HTTP endpoints. + Measure(MeasureArgs), + /// Create direct-SST tables, discover their regions, and generate fixtures. + PrepareDirect(PrepareDirectArgs), + /// Render the frontend Prometheus store configuration for a remote-write case. + RenderRemoteConfig(RenderRemoteConfigArgs), + /// Create a database and ingest a normalized Prometheus remote-write case. + PrepareRemote(PrepareRemoteArgs), + /// Inspect stopped remote-write storage and finalize the aggregate report. + FinalizeRemote(FinalizeRemoteArgs), + /// Run one externally managed OTLP trace-load target. + RunOtlpTarget(RunOtlpTargetArgs), + /// Combine externally managed OTLP trace-load target results. + FinalizeOtlp(FinalizeOtlpArgs), + /// Copy a direct-SST fixture into an object store destination. + Materialize(MaterializeArgs), +} + +#[derive(Debug, Parser)] +struct MeasureArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long)] + base_http_port: u16, + #[arg(long)] + candidate_http_port: u16, + #[arg(long, value_name = "PATH")] + output: Option, + #[arg(long, default_value_t = 120.0)] + http_timeout: f64, +} + +#[derive(Debug, Parser)] +struct PrepareDirectArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long)] + base_http_port: u16, + #[arg(long)] + candidate_http_port: u16, + #[arg(long, value_name = "PATH")] + fixture_dir: PathBuf, + #[arg(long, value_name = "PATH")] + output: Option, + #[arg(long, default_value_t = 120.0)] + http_timeout: f64, + #[arg(long)] + allow_large_fixture: bool, +} + +#[derive(Debug, Parser)] +struct RenderRemoteConfigArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long, value_name = "PATH")] + output: PathBuf, +} + +#[derive(Debug, Parser)] +struct PrepareRemoteArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long)] + base_http_port: u16, + #[arg(long)] + candidate_http_port: u16, + #[arg(long, value_name = "PATH")] + output: Option, + #[arg(long, default_value_t = 120.0)] + http_timeout: f64, +} + +#[derive(Debug, Parser)] +struct FinalizeRemoteArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long, value_name = "PATH")] + candidate_bin: PathBuf, + #[arg(long, value_name = "PATH")] + base_data_home: Option, + #[arg(long, value_name = "PATH")] + candidate_data_home: Option, + #[arg(long, value_name = "PATH")] + base_destination: Option, + #[arg(long, value_name = "PATH")] + candidate_destination: Option, + #[arg(long, value_name = "PATH")] + report: PathBuf, +} + +#[derive(Debug, Parser)] +struct RunOtlpTargetArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long, value_name = "PATH")] + otelgen_bin: PathBuf, + #[arg(long)] + http_port: u16, + #[arg(long)] + target_name: String, + #[arg(long, value_name = "PATH")] + work_dir: PathBuf, + #[arg(long, value_name = "PATH")] + output: PathBuf, + #[arg(long, default_value_t = 120.0)] + http_timeout: f64, +} + +#[derive(Debug, Parser)] +struct FinalizeOtlpArgs { + #[arg(long, value_name = "PATH")] + case: PathBuf, + #[arg(long, value_name = "PATH")] + fixture_generator: PathBuf, + #[arg(long, value_name = "PATH")] + base_result: PathBuf, + #[arg(long, value_name = "PATH")] + candidate_result: PathBuf, + #[arg(long, value_name = "PATH")] + output: PathBuf, +} + +#[derive(Debug, Parser)] +struct MaterializeArgs { + #[arg(long, value_name = "PATH")] + fixture_dir: PathBuf, + #[arg( + long, + value_name = "PATH", + help = "TOML file: data_home = \"...\" and object_store = { type = \"File\" }; object_store uses object_store::config::ObjectStoreConfig" + )] + destination: PathBuf, +} + +pub(super) async fn run() { + if let Err(error) = run_inner().await { + eprintln!("query_regression_runner: {error}"); + std::process::exit(1); + } +} + +async fn run_inner() -> Result<()> { + match Cli::parse().command { + RunnerCommand::Measure(args) => measure::run_measure(args).await, + RunnerCommand::PrepareDirect(args) => direct::run_prepare_direct(args).await, + RunnerCommand::RenderRemoteConfig(args) => remote::run_render_remote_config(args).await, + RunnerCommand::PrepareRemote(args) => remote::run_prepare_remote(args).await, + RunnerCommand::FinalizeRemote(args) => remote::run_finalize_remote(args).await, + RunnerCommand::RunOtlpTarget(args) => otlp::run_otlp_target(args).await, + RunnerCommand::FinalizeOtlp(args) => otlp::run_finalize_otlp(args).await, + RunnerCommand::Materialize(args) => materialize::run_materialize(args).await, + } +} diff --git a/src/cmd/src/bin/query_regression_runner/model.rs b/src/cmd/src/bin/query_regression_runner/model.rs new file mode 100644 index 0000000000..50d41b3f85 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/model.rs @@ -0,0 +1,235 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use object_store::config::ObjectStoreConfig; +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value}; + +#[derive(Debug, Deserialize)] +pub(super) struct DestinationConfig { + pub(super) data_home: String, + pub(super) object_store: ObjectStoreConfig, +} + +#[derive(Debug, Deserialize)] +pub(super) struct FixtureSummary { + pub(super) region_dir: String, +} + +#[derive(Debug, Default, Serialize)] +pub(super) struct CopyCounts { + pub(super) files: u64, + pub(super) bytes: u64, +} + +#[derive(Debug, Serialize)] +pub(super) struct MaterializeResult { + pub(super) region_dir: String, + pub(super) object_store: CopyCounts, + pub(super) manifest: CopyCounts, +} + +#[derive(Debug, Deserialize)] +#[serde(tag = "kind")] +pub(super) enum Scenario { + #[serde(rename = "direct_readable_sst")] + DirectReadableSst { + tables: Vec, + layout: Layout, + #[serde(default)] + queries: Vec, + }, + #[serde(rename = "prom_remote_write_then_query")] + PromRemoteWriteThenQuery { + remote_write: RemoteWrite, + #[serde(default)] + queries: Vec, + }, + #[serde(rename = "otlp_trace_load")] + OtlpTraceLoad { load: OtlpTraceLoad }, +} + +#[derive(Debug, Deserialize)] +pub(super) struct Layout { + pub(super) regions: usize, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct Table { + pub(super) database: String, + pub(super) name: String, + #[serde(default)] + pub(super) engine: String, + #[serde(default)] + pub(super) columns: Vec, + #[serde(default)] + pub(super) primary_key: Vec, + #[serde(default)] + pub(super) time_index: Option, + #[serde(default)] + pub(super) append_mode: Option, + #[serde(default)] + pub(super) sst_format: Option, + #[serde(default = "default_show_create_engine")] + pub(super) validate_show_create_engine: bool, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct Column { + pub(super) name: String, + #[serde(rename = "type")] + pub(super) data_type: String, +} + +const fn default_show_create_engine() -> bool { + true +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct RemoteWrite { + pub(super) database: String, + pub(super) metric: String, + pub(super) physical_table: String, + pub(super) series_count: u64, + pub(super) samples_per_series: u64, + pub(super) start_unix_millis: i64, + pub(super) step_millis: i64, + pub(super) chunk_series_count: u64, + pub(super) timeout_seconds: u64, + pub(super) sample_chunk_size: Option, + pub(super) flush_every_sample_chunks: u64, + pub(super) visibility_timeout_seconds: u64, + pub(super) prom_store: PromStore, + pub(super) value: RemoteValue, + pub(super) storage: Option, + pub(super) read_bench: Option, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct PromStore { + pub(super) pending_rows_flush_interval: String, + pub(super) max_batch_rows: u64, + pub(super) max_concurrent_flushes: u64, + pub(super) worker_channel_capacity: u64, + pub(super) max_inflight_requests: u64, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct RemoteValue { + pub(super) pattern: String, + pub(super) base: f64, + pub(super) step: f64, + pub(super) cardinality: u64, + pub(super) seed: u64, + pub(super) run_length: u64, + pub(super) stall_every: u64, + pub(super) stall_length: u64, + pub(super) mixed_every: u64, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +pub(super) struct OtlpTraceLoad { + pub(super) load: OtlpLoad, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +pub(super) struct OtlpLoad { + pub(super) database: String, + pub(super) table: String, + pub(super) pipeline: String, + pub(super) duration_seconds: u64, + pub(super) warmup_seconds: u64, + pub(super) rate: u64, + pub(super) workers: usize, + pub(super) exporter_shards: usize, + pub(super) workload: String, + pub(super) visibility_timeout_seconds: u64, + pub(super) thresholds: OtlpThresholds, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +pub(super) struct OtlpThresholds { + pub(super) max_candidate_throughput_regression_pct: f64, + pub(super) max_candidate_mean_latency_regression_pct: f64, + pub(super) max_failure_count: u64, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct StorageConfig { + pub(super) inspect: bool, + pub(super) column: String, + pub(super) root_suffix: Option, + pub(super) include_metadata_files: bool, + pub(super) min_files: u64, + pub(super) min_files_with_column: u64, + pub(super) require_encodings: Vec, + pub(super) forbid_encodings: Vec, + pub(super) max_total_file_size_bytes: Option, + pub(super) max_column_compressed_size_bytes: Option, + pub(super) max_column_uncompressed_size_bytes: Option, + pub(super) max_candidate_total_file_size_regression_pct: Option, + pub(super) max_candidate_column_compressed_size_regression_pct: Option, + pub(super) max_candidate_column_uncompressed_size_regression_pct: Option, +} + +#[derive(Clone, Debug, Deserialize)] +pub(super) struct ReadBenchConfig { + pub(super) enabled: bool, + pub(super) parquetbench: bool, + pub(super) scanbench: bool, + pub(super) iterations: u64, + pub(super) projection: Vec, + pub(super) parquet_reader: String, + pub(super) scan_scanner: String, + pub(super) parallelism: u64, + pub(super) max_files: Option, +} + +#[derive(Clone, Debug, Deserialize, Serialize)] +pub(super) struct Query { + #[serde(default)] + pub(super) name: Option, + #[serde(default)] + pub(super) kind: Option, + pub(super) query: String, + #[serde(default)] + pub(super) warmup: usize, + #[serde(default = "one")] + pub(super) iterations: usize, + #[serde(default)] + pub(super) thresholds: Map, +} + +const fn one() -> usize { + 1 +} + +#[derive(Debug, Serialize)] +pub(super) struct QueryResult { + pub(super) validation: Vec, + pub(super) validation_errors: Vec, + pub(super) measurements: Vec, + pub(super) status: String, +} + +#[derive(Debug, Serialize)] +pub(super) struct Measurement { + pub(super) name: Option, + pub(super) kind: Option, + pub(super) iterations: usize, + pub(super) samples: Vec, + pub(super) latency_ms_median: Option, + pub(super) latency_ms_p95: Option, + pub(super) status: String, +} diff --git a/src/cmd/src/bin/query_regression_runner/otlp.rs b/src/cmd/src/bin/query_regression_runner/otlp.rs new file mode 100644 index 0000000000..ceccad0ffc --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/otlp.rs @@ -0,0 +1,618 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::fs; +use std::path::Path; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +use regex::Regex; +use reqwest::Client; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; + +use crate::query_regression_runner::measure::round_ties_even; +use crate::query_regression_runner::model::{OtlpLoad, OtlpThresholds}; +use crate::query_regression_runner::plan::normalized_otlp_load; +use crate::query_regression_runner::sql::{ + extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64, +}; +use crate::query_regression_runner::{FinalizeOtlpArgs, Result, RunOtlpTargetArgs}; + +const OTLP_ROWS: &str = "greptime_frontend_otlp_traces_rows"; +const OTLP_FAILURES: &str = "greptime_frontend_otlp_traces_failure_count"; +const OTLP_ELAPSED_SUM: &str = "greptime_servers_http_otlp_traces_elapsed_sum"; +const OTLP_ELAPSED_COUNT: &str = "greptime_servers_http_otlp_traces_elapsed_count"; + +#[derive(Clone, Debug, Deserialize, Serialize)] +struct OtlpSnapshot { + captured_monotonic_seconds: f64, + values: HashMap, +} + +pub(super) async fn run_otlp_target(args: RunOtlpTargetArgs) -> Result<()> { + if !args.http_timeout.is_finite() || args.http_timeout < 0.0 { + return Err("--http-timeout must be a non-negative finite number".into()); + } + let (_, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?; + let client = Client::builder() + .timeout(Duration::from_secs_f64(args.http_timeout)) + .build()?; + let result = match run_otlp_target_inner(&args, &load, &client).await { + Ok(result) => result, + Err(error) => { + json!({ "name": args.target_name, "status": "failed", "error": error.to_string() }) + } + }; + let text = format!("{}\n", serde_json::to_string_pretty(&result)?); + fs::write(&args.output, &text)?; + print!("{text}"); + if result["status"] == "failed" { + std::process::exit(1); + } + Ok(()) +} + +async fn run_otlp_target_inner( + args: &RunOtlpTargetArgs, + load: &OtlpLoad, + client: &Client, +) -> Result { + let create_database = http_post_sql( + client, + args.http_port, + &format!( + "CREATE DATABASE IF NOT EXISTS {}", + sql_ident(&load.database) + ), + "public", + ) + .await; + if !create_database["ok"].as_bool().unwrap_or(false) { + return Ok( + json!({ "name": args.target_name, "create_database": create_database, "status": "failed" }), + ); + } + let otelgen = run_otelgen_load( + &args.otelgen_bin, + args.http_port, + &args.work_dir, + load, + client, + ) + .await?; + let metrics = summarize_otlp_metrics(&otelgen)?; + let flush = http_post_sql( + client, + args.http_port, + &format!("ADMIN FLUSH_TABLE({})", sql_string(&load.table)), + &load.database, + ) + .await; + let visibility = poll_otlp_visibility( + client, + args.http_port, + &load.table, + &load.database, + metrics["accepted_spans"].as_u64().unwrap_or_default(), + load.visibility_timeout_seconds, + ) + .await?; + let checks_ok = otelgen["status"] == "ok" + && metrics["missing_metrics"] + .as_array() + .is_some_and(Vec::is_empty) + && metrics["accepted_spans"].as_u64().unwrap_or_default() > 0 + && metrics["http_requests"].as_u64().unwrap_or_default() > 0 + && flush["ok"].as_bool().unwrap_or(false) + && visibility["ok"].as_bool().unwrap_or(false) + && visibility["row_count_ok"].as_bool().unwrap_or(false); + Ok(json!({ + "name": args.target_name, + "otelgen": otelgen, + "metrics": metrics, + "create_database": create_database, + "flush": flush, + "visibility": visibility, + "status": if checks_ok { "measured" } else { "failed" }, + })) +} + +fn otelgen_command(otelgen_bin: &Path, http_port: u16, load: &OtlpLoad) -> Vec { + vec![ + otelgen_bin.to_string_lossy().to_string(), + "--protocol".to_string(), + "http".to_string(), + "--otel-exporter-otlp-endpoint".to_string(), + format!("127.0.0.1:{http_port}"), + "--otel-exporter-otlp-url-path".to_string(), + "/v1/otlp/v1/traces".to_string(), + "--header".to_string(), + format!("x-greptime-pipeline-name={}", load.pipeline), + "--header".to_string(), + format!("x-greptime-db-name={}", load.database), + "--header".to_string(), + format!("x-greptime-trace-table-name={}", load.table), + "--log-level".to_string(), + "error".to_string(), + "--insecure".to_string(), + "--duration".to_string(), + load.duration_seconds.to_string(), + "--rate".to_string(), + load.rate.to_string(), + "traces".to_string(), + "multi".to_string(), + "--workers".to_string(), + load.workers.to_string(), + "--scenarios".to_string(), + load.workload.clone(), + "--exporter-shards".to_string(), + load.exporter_shards.to_string(), + ] +} + +async fn run_otelgen_load( + otelgen_bin: &Path, + http_port: u16, + work_dir: &Path, + load: &OtlpLoad, + client: &Client, +) -> Result { + let command = otelgen_command(otelgen_bin, http_port, load); + let clock = Instant::now(); + let initial = fetch_otlp_metrics(client, http_port, &clock).await?; + let log_dir = work_dir.join("otelgen"); + fs::create_dir_all(&log_dir)?; + let stdout_path = log_dir.join("stdout.log"); + let stderr_path = log_dir.join("stderr.log"); + let mut child = Command::new(otelgen_bin) + .args(&command[1..]) + .stdout(Stdio::from(fs::File::create(&stdout_path)?)) + .stderr(Stdio::from(fs::File::create(&stderr_path)?)) + .spawn()?; + let started = Instant::now(); + if load.warmup_seconds > 0 { + let _ = wait_for_child(&mut child, Duration::from_secs(load.warmup_seconds)).await?; + } + let warmed = fetch_otlp_metrics(client, http_port, &clock).await?; + let mut timed_out = false; + if child.try_wait()?.is_none() { + let remaining = load + .duration_seconds + .saturating_sub(load.warmup_seconds) + .saturating_add(60) + .max(60); + if !wait_for_child(&mut child, Duration::from_secs(remaining)).await? { + timed_out = true; + child.kill()?; + let _ = child.wait()?; + } + } + let final_snapshot = fetch_otlp_metrics(client, http_port, &clock).await?; + let returncode = match child.try_wait()? { + Some(status) => status.code(), + None => { + child.kill()?; + child.wait()?.code() + } + }; + let elapsed_seconds = started.elapsed().as_secs_f64(); + Ok(json!({ + "status": if returncode == Some(0) && !timed_out && elapsed_seconds + 1.0 >= load.duration_seconds as f64 { "ok" } else { "failed" }, + "cmd": command, + "returncode": returncode, + "timed_out": timed_out, + "elapsed_seconds": elapsed_seconds, + "stdout_path": stdout_path, + "stderr_path": stderr_path, + "snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot }, + })) +} + +async fn wait_for_child(child: &mut std::process::Child, timeout: Duration) -> Result { + let deadline = Instant::now() + timeout; + loop { + if child.try_wait()?.is_some() { + return Ok(true); + } + if Instant::now() >= deadline { + return Ok(false); + } + tokio::time::sleep(Duration::from_millis(100)).await; + } +} + +async fn fetch_otlp_metrics( + client: &Client, + http_port: u16, + clock: &Instant, +) -> Result { + let text = client + .get(format!("http://127.0.0.1:{http_port}/metrics")) + .send() + .await? + .error_for_status()? + .text() + .await?; + Ok(OtlpSnapshot { + captured_monotonic_seconds: clock.elapsed().as_secs_f64(), + values: parse_prometheus_metrics(&text)?, + }) +} + +fn parse_prometheus_metrics(text: &str) -> Result> { + let sample = Regex::new(r"^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{[^}]*\})?\s+(\S+)")?; + let mut values = HashMap::new(); + for line in text.lines() { + let Some(captures) = sample.captures(line.trim()) else { + continue; + }; + let (Some(name), Some(value)) = (captures.get(1), captures.get(2)) else { + continue; + }; + let name = name.as_str(); + if !matches!( + name, + OTLP_ROWS | OTLP_FAILURES | OTLP_ELAPSED_SUM | OTLP_ELAPSED_COUNT + ) { + continue; + } + let value: f64 = value.as_str().parse()?; + if !value.is_finite() { + return Err(format!("non-finite Prometheus sample for {name}").into()); + } + *values.entry(name.to_string()).or_default() += value; + } + Ok(values) +} + +fn metric_delta(after: &OtlpSnapshot, before: &OtlpSnapshot, name: &str) -> Result { + let delta = after.values.get(name).copied().unwrap_or_default() + - before.values.get(name).copied().unwrap_or_default(); + if delta < 0.0 { + return Err(format!("metric {name} decreased by {}", -delta).into()); + } + Ok(delta) +} + +fn summarize_otlp_metrics(run: &Value) -> Result { + let snapshots = run + .get("snapshots") + .ok_or("otelgen result has no snapshots")?; + let initial: OtlpSnapshot = serde_json::from_value(snapshots["initial"].clone())?; + let warmed: OtlpSnapshot = serde_json::from_value(snapshots["warmup"].clone())?; + let final_snapshot: OtlpSnapshot = serde_json::from_value(snapshots["final"].clone())?; + let missing_metrics = [OTLP_ROWS, OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT] + .into_iter() + .filter(|name| !final_snapshot.values.contains_key(*name)) + .collect::>(); + let accepted_spans = + round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_ROWS)?) as u64; + let measurement_accepted_spans = + round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ROWS)?) as u64; + let http_requests = + round_ties_even(metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_COUNT)?) as u64; + let latency_seconds = metric_delta(&final_snapshot, &warmed, OTLP_ELAPSED_SUM)?; + let measurement_seconds = + final_snapshot.captured_monotonic_seconds - warmed.captured_monotonic_seconds; + Ok(json!({ + "accepted_spans": accepted_spans, + "measurement_accepted_spans": measurement_accepted_spans, + "accepted_spans_per_second": if measurement_seconds > 0.0 { Some(measurement_accepted_spans as f64 / measurement_seconds) } else { None }, + "http_requests": http_requests, + "mean_http_latency_ms": if http_requests > 0 { Some(latency_seconds / http_requests as f64 * 1000.0) } else { None }, + "failure_count": round_ties_even(metric_delta(&final_snapshot, &initial, OTLP_FAILURES)?) as u64, + "measurement_seconds": measurement_seconds, + "missing_metrics": missing_metrics, + })) +} + +async fn poll_otlp_visibility( + client: &Client, + port: u16, + table_name: &str, + database: &str, + expected_rows: u64, + timeout_seconds: u64, +) -> Result { + let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name)); + let deadline = Instant::now() + Duration::from_secs(timeout_seconds); + let mut attempts = 0; + loop { + attempts += 1; + let mut result = http_post_sql(client, port, &sql, database).await; + let observed_rows = extract_count_value(&result); + let row_count_ok = + result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows); + result + .as_object_mut() + .ok_or("count result must be an object")? + .extend([ + ("expected_rows".to_string(), json!(expected_rows)), + ("observed_rows".to_string(), json!(observed_rows)), + ("attempts".to_string(), json!(attempts)), + ("row_count_ok".to_string(), json!(row_count_ok)), + ]); + if row_count_ok { + return Ok(result); + } + if Instant::now() >= deadline { + result["ok"] = json!(false); + result["row_count_ok"] = json!(false); + result["error"] = json!(format!( + "expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts" + )); + return Ok(result); + } + tokio::time::sleep(Duration::from_millis(500)).await; + } +} + +pub(super) async fn run_finalize_otlp(args: FinalizeOtlpArgs) -> Result<()> { + let (case_path, load) = normalized_otlp_load(&args.fixture_generator, &args.case)?; + let raw_case: toml::Value = toml::from_str(&fs::read_to_string(&case_path)?)?; + let case_metadata = serde_json::to_value( + raw_case + .get("case") + .cloned() + .unwrap_or(toml::Value::Table(toml::map::Map::new())), + )?; + let base: Value = serde_json::from_slice(&fs::read(&args.base_result)?)?; + let candidate: Value = serde_json::from_slice(&fs::read(&args.candidate_result)?)?; + let thresholds = enforce_otlp_thresholds( + &load.thresholds, + base.get("metrics").unwrap_or(&Value::Null), + candidate.get("metrics").unwrap_or(&Value::Null), + ); + let failed = thresholds + .iter() + .any(|threshold| threshold["status"] == "failed") + || [base.get("status"), candidate.get("status")] + .into_iter() + .any(|status| status.and_then(Value::as_str) == Some("failed")); + let report = json!({ + "case_path": case_path, + "case": case_metadata, + "scenario": { "kind": "otlp_trace_load", "load": load }, + "targets": [base, candidate], + "thresholds": thresholds, + "status": if failed { "failed" } else { "ok" }, + }); + let text = format!("{}\n", serde_json::to_string_pretty(&report)?); + fs::write(&args.output, &text)?; + print!("{text}"); + if failed { + std::process::exit(1); + } + Ok(()) +} + +fn enforce_otlp_thresholds( + thresholds: &OtlpThresholds, + base: &Value, + candidate: &Value, +) -> Vec { + let mut results = Vec::new(); + for (target, metrics) in [("base", base), ("candidate", candidate)] { + let failures = value_u64(metrics.get("failure_count")); + results.push(json!({ + "target": target, + "threshold": "max_failure_count", + "status": if failures.is_some_and(|failures| failures <= thresholds.max_failure_count) { "passed" } else { "failed" }, + "actual": failures, + "limit": thresholds.max_failure_count, + })); + } + for (name, base_value, candidate_value, limit, inverse) in [ + ( + "max_candidate_throughput_regression_pct", + value_f64(base.get("accepted_spans_per_second")), + value_f64(candidate.get("accepted_spans_per_second")), + thresholds.max_candidate_throughput_regression_pct, + true, + ), + ( + "max_candidate_mean_latency_regression_pct", + value_f64(base.get("mean_http_latency_ms")), + value_f64(candidate.get("mean_http_latency_ms")), + thresholds.max_candidate_mean_latency_regression_pct, + false, + ), + ] { + let (Some(base_value), Some(candidate_value)) = + (base_value.filter(|value| *value != 0.0), candidate_value) + else { + results.push(json!({ + "threshold": name, + "status": "failed", + "reason": if inverse { "missing or zero throughput" } else { "missing or zero mean latency" }, + "base": base_value, + "candidate": candidate_value, + })); + continue; + }; + let actual = if inverse { + (base_value - candidate_value) / base_value * 100.0 + } else { + (candidate_value - base_value) / base_value * 100.0 + }; + results.push(json!({ + "threshold": name, + "status": if actual <= limit { "passed" } else { "failed" }, + "actual_pct": actual, + "limit_pct": limit, + "base": base_value, + "candidate": candidate_value, + })); + } + results +} + +#[cfg(test)] +mod tests { + use super::*; + + fn otlp_load_for_test() -> OtlpLoad { + serde_json::from_value(json!({ + "database": "public", + "table": "opentelemetry_traces", + "pipeline": "greptime_trace_v1", + "duration_seconds": 120, + "warmup_seconds": 60, + "rate": 50000, + "workers": 4, + "exporter_shards": 4, + "workload": "microservices", + "visibility_timeout_seconds": 30, + "thresholds": { + "max_candidate_throughput_regression_pct": 20.0, + "max_candidate_mean_latency_regression_pct": 20.0, + "max_failure_count": 0 + } + })) + .unwrap() + } + + #[test] + fn parses_labeled_otlp_metrics_and_rejects_non_finite_samples() { + let metrics = parse_prometheus_metrics( + "greptime_frontend_otlp_traces_rows 10\n\ + greptime_frontend_otlp_traces_failure_count{kind=\"a\"} 1\n\ + greptime_frontend_otlp_traces_failure_count{kind=\"b\"} 2\n", + ) + .unwrap(); + assert_eq!(metrics[OTLP_ROWS], 10.0); + assert_eq!(metrics[OTLP_FAILURES], 3.0); + assert!(parse_prometheus_metrics("greptime_frontend_otlp_traces_rows NaN").is_err()); + let before = OtlpSnapshot { + captured_monotonic_seconds: 0.0, + values: HashMap::from([(OTLP_ROWS.to_string(), 2.0)]), + }; + let after = OtlpSnapshot { + captured_monotonic_seconds: 1.0, + values: HashMap::from([(OTLP_ROWS.to_string(), 1.0)]), + }; + assert!(metric_delta(&after, &before, OTLP_ROWS).is_err()); + } + + #[test] + fn summarizes_otlp_deltas_and_builds_exact_command() { + let snapshot = |captured: f64, values: Vec<(String, f64)>| OtlpSnapshot { + captured_monotonic_seconds: captured, + values: values.into_iter().collect(), + }; + let initial = snapshot(0.0, vec![(OTLP_ROWS.to_string(), 10.0)]); + let warmed = snapshot( + 5.0, + vec![ + (OTLP_ROWS.to_string(), 110.0), + (OTLP_ELAPSED_SUM.to_string(), 1.0), + (OTLP_ELAPSED_COUNT.to_string(), 10.0), + ], + ); + let final_snapshot = snapshot( + 15.0, + vec![ + (OTLP_ROWS.to_string(), 310.0), + (OTLP_FAILURES.to_string(), 3.0), + (OTLP_ELAPSED_SUM.to_string(), 3.0), + (OTLP_ELAPSED_COUNT.to_string(), 30.0), + ], + ); + let metrics = summarize_otlp_metrics(&json!({ + "snapshots": { "initial": initial, "warmup": warmed, "final": final_snapshot } + })) + .unwrap(); + assert_eq!(metrics["accepted_spans"], 300); + assert_eq!(metrics["measurement_accepted_spans"], 200); + assert_eq!(metrics["accepted_spans_per_second"], 20.0); + assert_eq!(metrics["http_requests"], 20); + assert_eq!(metrics["mean_http_latency_ms"], 100.0); + assert_eq!(metrics["failure_count"], 3); + let missing = summarize_otlp_metrics(&json!({ + "snapshots": { + "initial": snapshot(0.0, vec![(OTLP_ROWS.to_string(), 0.0)]), + "warmup": snapshot(1.0, vec![(OTLP_ROWS.to_string(), 1.0)]), + "final": snapshot(2.0, vec![(OTLP_ROWS.to_string(), 2.0)]) + } + })) + .unwrap(); + assert_eq!( + missing["missing_metrics"], + json!([OTLP_ELAPSED_SUM, OTLP_ELAPSED_COUNT]) + ); + let command = otelgen_command(Path::new("/bin/otelgen"), 4000, &otlp_load_for_test()); + assert_eq!( + command, + vec![ + "/bin/otelgen", + "--protocol", + "http", + "--otel-exporter-otlp-endpoint", + "127.0.0.1:4000", + "--otel-exporter-otlp-url-path", + "/v1/otlp/v1/traces", + "--header", + "x-greptime-pipeline-name=greptime_trace_v1", + "--header", + "x-greptime-db-name=public", + "--header", + "x-greptime-trace-table-name=opentelemetry_traces", + "--log-level", + "error", + "--insecure", + "--duration", + "120", + "--rate", + "50000", + "traces", + "multi", + "--workers", + "4", + "--scenarios", + "microservices", + "--exporter-shards", + "4", + ] + ); + } + + #[test] + fn otlp_thresholds_handle_zero_metrics_and_pass_fail_cases() { + let load = otlp_load_for_test(); + let base = json!({ "failure_count": 0, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 100.0 }); + let candidate = json!({ "failure_count": 0, "accepted_spans_per_second": 18.0, "mean_http_latency_ms": 110.0 }); + let results = enforce_otlp_thresholds(&load.thresholds, &base, &candidate); + assert!(results.iter().all(|result| result["status"] == "passed")); + let failed = enforce_otlp_thresholds( + &load.thresholds, + &json!({ "failure_count": 0, "accepted_spans_per_second": 0.0, "mean_http_latency_ms": 100.0 }), + &json!({ "failure_count": 3, "accepted_spans_per_second": 20.0, "mean_http_latency_ms": 130.0 }), + ); + assert!( + failed + .iter() + .any(|result| result["threshold"] == "max_failure_count" + && result["target"] == "candidate" + && result["status"] == "failed") + ); + assert!(failed.iter().any(|result| result["threshold"] + == "max_candidate_throughput_regression_pct" + && result["reason"] == "missing or zero throughput")); + assert!(failed.iter().any(|result| result["threshold"] + == "max_candidate_mean_latency_regression_pct" + && result["status"] == "failed")); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/plan.rs b/src/cmd/src/bin/query_regression_runner/plan.rs new file mode 100644 index 0000000000..4fcc5135cd --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/plan.rs @@ -0,0 +1,122 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashSet; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use serde_json::Value; + +use crate::query_regression_runner::Result; +use crate::query_regression_runner::model::{ + Layout, OtlpLoad, Query, RemoteWrite, Scenario, Table, +}; + +pub(super) fn load_plan(generator: &PathBuf, case_path: &PathBuf) -> Result { + let output = Command::new(generator) + .args(["plan", "--case"]) + .arg(case_path) + .output()?; + if !output.status.success() { + let stderr = String::from_utf8_lossy(&output.stderr); + return Err(format!("query_perf_fixture plan failed: {:.2000}", stderr).into()); + } + Ok(serde_json::from_slice(&output.stdout)?) +} + +pub(super) fn normalize_scenario(scenario: Scenario) -> Result<(Vec
, Vec)> { + match scenario { + Scenario::DirectReadableSst { + tables, + layout, + queries, + } => Ok((validate_direct_tables(tables, layout)?, queries)), + Scenario::PromRemoteWriteThenQuery { + remote_write, + queries, + } => Ok(( + vec![Table { + database: remote_write.database, + name: remote_write.metric, + engine: "metric".to_string(), + columns: vec![], + primary_key: vec![], + time_index: None, + append_mode: None, + sst_format: None, + validate_show_create_engine: false, + }], + queries, + )), + Scenario::OtlpTraceLoad { .. } => { + Err("measure requires a query scenario, not otlp_trace_load".into()) + } + } +} + +pub(super) fn validate_direct_tables(tables: Vec
, layout: Layout) -> Result> { + if tables.is_empty() || layout.regions != 1 { + return Err("runner supports one or more tables and exactly one region per table".into()); + } + let mut pairs = HashSet::new(); + let mut names = HashSet::new(); + for table in &tables { + if !pairs.insert((&table.database, &table.name)) { + return Err("duplicate (database, name) table entries are not supported".into()); + } + if !names.insert(&table.name) { + return Err("duplicate table names are not supported".into()); + } + } + Ok(tables) +} + +pub(super) fn normalized_remote_write( + generator: &PathBuf, + case: &Path, +) -> Result<(PathBuf, RemoteWrite)> { + let case_path = case.canonicalize()?; + let plan = load_plan(generator, &case_path)?; + let scenario = plan + .get("scenario") + .cloned() + .ok_or("fixture plan has no scenario")?; + match serde_json::from_value(scenario)? { + Scenario::PromRemoteWriteThenQuery { remote_write, .. } => Ok((case_path, remote_write)), + Scenario::DirectReadableSst { .. } => { + Err("remote command requires scenario kind prom_remote_write_then_query".into()) + } + Scenario::OtlpTraceLoad { .. } => { + Err("remote command requires scenario kind prom_remote_write_then_query".into()) + } + } +} + +pub(super) fn normalized_otlp_load( + generator: &PathBuf, + case: &Path, +) -> Result<(PathBuf, OtlpLoad)> { + let case_path = case.canonicalize()?; + let plan = load_plan(generator, &case_path)?; + let scenario = plan + .get("scenario") + .cloned() + .ok_or("fixture plan has no scenario")?; + match serde_json::from_value(scenario)? { + Scenario::OtlpTraceLoad { load } => Ok((case_path, load.load)), + Scenario::DirectReadableSst { .. } | Scenario::PromRemoteWriteThenQuery { .. } => { + Err("OTLP command requires scenario kind otlp_trace_load".into()) + } + } +} diff --git a/src/cmd/src/bin/query_regression_runner/remote/ingest.rs b/src/cmd/src/bin/query_regression_runner/remote/ingest.rs new file mode 100644 index 0000000000..f563402afc --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/remote/ingest.rs @@ -0,0 +1,505 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::path::Path; +use std::process::Command; +use std::time::{Duration, Instant}; + +use reqwest::Client; +use serde_json::{Value, json}; + +use crate::query_regression_runner::model::{PromStore, RemoteWrite}; +use crate::query_regression_runner::plan::normalized_remote_write; +use crate::query_regression_runner::sql::{ + extract_count_value, http_post_sql, sql_ident, sql_string, value_f64, value_u64, +}; +use crate::query_regression_runner::{PrepareRemoteArgs, RenderRemoteConfigArgs, Result}; + +pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> { + let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?; + fs::write(args.output, frontend_prom_config(&remote.prom_store)?)?; + Ok(()) +} + +fn frontend_prom_config(prom: &PromStore) -> Result { + Ok(format!( + "[prom_store]\nenable = true\nwith_metric_engine = true\npending_rows_flush_interval = {}\nmax_batch_rows = {}\nmax_concurrent_flushes = {}\nworker_channel_capacity = {}\nmax_inflight_requests = {}\n", + serde_json::to_string(&prom.pending_rows_flush_interval)?, + prom.max_batch_rows, + prom.max_concurrent_flushes, + prom.worker_channel_capacity, + prom.max_inflight_requests, + )) +} + +pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> { + if !args.http_timeout.is_finite() || args.http_timeout < 0.0 { + return Err("--http-timeout must be a non-negative finite number".into()); + } + let (case_path, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?; + let client = Client::builder() + .timeout(Duration::from_secs_f64(args.http_timeout)) + .build()?; + let base = prepare_remote_target( + "base", + args.base_http_port, + &args.fixture_generator, + &remote, + &client, + ) + .await?; + let candidate = prepare_remote_target( + "candidate", + args.candidate_http_port, + &args.fixture_generator, + &remote, + &client, + ) + .await?; + let report = json!({ + "case_path": case_path, + "scenario": "prom_remote_write_then_query", + "base": base, + "candidate": candidate, + "status": "ok", + }); + let text = format!("{}\n", serde_json::to_string_pretty(&report)?); + if let Some(path) = args.output { + fs::write(path, &text)?; + } + print!("{text}"); + Ok(()) +} + +async fn prepare_remote_target( + name: &str, + port: u16, + generator: &Path, + remote: &RemoteWrite, + client: &Client, +) -> Result { + let create_database = http_post_sql( + client, + port, + &format!( + "CREATE DATABASE IF NOT EXISTS {}", + sql_ident(&remote.database) + ), + "public", + ) + .await; + if !create_database["ok"].as_bool().unwrap_or(false) { + return Err(format!( + "CREATE DATABASE {} failed for {name}: {create_database}", + remote.database + ) + .into()); + } + let (remote_write, flushes) = ingest_remote_write(generator, port, remote, client).await?; + let expected_rows = remote + .series_count + .checked_mul(remote.samples_per_series) + .ok_or("expected remote-write row count overflows u64")?; + let visibility = poll_expected_count( + client, + port, + &remote.metric, + &remote.database, + expected_rows, + remote.visibility_timeout_seconds, + ) + .await?; + Ok(json!({ + "name": name, + "create_database": create_database, + "remote_write": remote_write, + "flushes": flushes, + "visibility": visibility, + "status": "ok", + })) +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct SampleChunk { + index: u64, + offset: u64, + samples_per_series: u64, + start_unix_millis: i64, +} + +async fn ingest_remote_write( + generator: &Path, + port: u16, + remote: &RemoteWrite, + client: &Client, +) -> Result<(Value, Vec)> { + let Some(chunks) = sample_chunks(remote)? else { + let remote_write = run_remote_write(generator, port, remote, None)?; + let flush = flush_remote_table(client, port, remote, "final", None).await?; + return Ok((remote_write, vec![flush])); + }; + let mut results = Vec::with_capacity(chunks.len()); + let mut flushes = Vec::new(); + let scheduled_flushes = scheduled_flushes(&chunks, remote.flush_every_sample_chunks); + for chunk in &chunks { + let mut result = run_remote_write(generator, port, remote, Some(chunk))?; + result + .as_object_mut() + .ok_or("remote-write result must be an object")? + .extend([ + ("sample_offset".to_string(), json!(chunk.offset)), + ( + "samples_per_series".to_string(), + json!(chunk.samples_per_series), + ), + ("chunk_index".to_string(), json!(chunk.index)), + ]); + results.push(result); + if let Some((_, reason)) = scheduled_flushes + .iter() + .find(|(index, _)| *index == chunk.index) + { + flushes + .push(flush_remote_table(client, port, remote, reason, Some(chunk.index)).await?); + } + } + Ok(( + json!({ + "status": "ok", + "mode": "sample-chunked", + "sample_chunk_size": remote.sample_chunk_size, + "flush_every_sample_chunks": remote.flush_every_sample_chunks, + "chunks": results, + "aggregate": summarize_remote_chunks(&results), + }), + flushes, + )) +} + +fn sample_chunks(remote: &RemoteWrite) -> Result>> { + let Some(chunk_samples) = remote.sample_chunk_size else { + return Ok(None); + }; + if chunk_samples == 0 { + return Err("scenario.remote_write.sample_chunk_size must be positive".into()); + } + if remote.flush_every_sample_chunks == 0 { + return Err("scenario.remote_write.flush_every_sample_chunks must be positive".into()); + } + let mut chunks = Vec::new(); + let mut offset = 0; + while offset < remote.samples_per_series { + let samples_per_series = chunk_samples.min(remote.samples_per_series - offset); + chunks.push(SampleChunk { + index: chunks.len() as u64 + 1, + offset, + samples_per_series, + start_unix_millis: remote.start_unix_millis + offset as i64 * remote.step_millis, + }); + offset += samples_per_series; + } + Ok(Some(chunks)) +} + +fn scheduled_flushes(chunks: &[SampleChunk], flush_every: u64) -> Vec<(u64, &'static str)> { + let mut scheduled = chunks + .iter() + .filter(|chunk| chunk.index % flush_every == 0) + .map(|chunk| (chunk.index, "periodic")) + .collect::>(); + if let Some(chunk) = chunks.last() + && chunk.index % flush_every != 0 + { + scheduled.push((chunk.index, "final")); + } + scheduled +} + +fn summarize_remote_chunks(chunks: &[Value]) -> Value { + let mut rows = 0; + let mut samples_written = 0; + let mut batches = 0; + let mut elapsed_seconds = 0.0; + for chunk in chunks { + let summary = chunk.get("summary").unwrap_or(&Value::Null); + let chunk_rows = value_u64(summary.get("rows")).unwrap_or(0); + rows += chunk_rows; + samples_written += value_u64(summary.get("samples_written")).unwrap_or(chunk_rows); + batches += value_u64(summary.get("batches")).unwrap_or(0); + elapsed_seconds += value_f64(summary.get("elapsed_seconds")) + .or_else(|| value_f64(chunk.get("elapsed_seconds"))) + .unwrap_or(0.0); + } + json!({ "rows": rows, "samples_written": samples_written, "batches": batches, "elapsed_seconds": elapsed_seconds }) +} + +fn run_remote_write( + generator: &Path, + port: u16, + remote: &RemoteWrite, + chunk: Option<&SampleChunk>, +) -> Result { + let command = remote_write_command(generator, port, remote, chunk); + let started = Instant::now(); + let output = Command::new(generator).args(&command[1..]).output()?; + let elapsed_seconds = started.elapsed().as_secs_f64(); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + if !output.status.success() { + return Err(format!( + "remote-write generator failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}", + output.status.code(), stderr + ) + .into()); + } + let mut result = json!({ + "status": "ok", + "command": command, + "returncode": output.status.code(), + "elapsed_seconds": elapsed_seconds, + "stdout": stdout, + "stderr": stderr, + }); + match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) { + Ok(summary) => result["summary"] = summary, + Err(_) => result["summary_parse_error"] = result["stdout"].clone(), + } + Ok(result) +} + +fn remote_write_command( + generator: &Path, + port: u16, + remote: &RemoteWrite, + chunk: Option<&SampleChunk>, +) -> Vec { + let (samples_per_series, start_unix_millis, sample_offset, total_samples_per_series) = chunk + .map(|chunk| { + ( + chunk.samples_per_series, + chunk.start_unix_millis, + Some(chunk.offset), + Some(remote.samples_per_series), + ) + }) + .unwrap_or(( + remote.samples_per_series, + remote.start_unix_millis, + None, + None, + )); + let mut command = vec![ + generator.to_string_lossy().to_string(), + "prom-remote-write".to_string(), + "--endpoint".to_string(), + format!("http://127.0.0.1:{port}/v1/prometheus/write"), + "--database".to_string(), + remote.database.clone(), + "--metric".to_string(), + remote.metric.clone(), + "--physical-table".to_string(), + remote.physical_table.clone(), + "--series-count".to_string(), + remote.series_count.to_string(), + "--samples-per-series".to_string(), + samples_per_series.to_string(), + "--start-unix-millis".to_string(), + start_unix_millis.to_string(), + "--step-millis".to_string(), + remote.step_millis.to_string(), + "--chunk-series-count".to_string(), + remote.chunk_series_count.to_string(), + "--timeout-seconds".to_string(), + remote.timeout_seconds.to_string(), + "--value-pattern".to_string(), + remote.value.pattern.clone(), + "--value-base".to_string(), + json_number(remote.value.base), + "--value-step".to_string(), + json_number(remote.value.step), + "--value-cardinality".to_string(), + remote.value.cardinality.to_string(), + "--value-seed".to_string(), + remote.value.seed.to_string(), + "--value-run-length".to_string(), + remote.value.run_length.to_string(), + "--value-stall-every".to_string(), + remote.value.stall_every.to_string(), + "--value-stall-length".to_string(), + remote.value.stall_length.to_string(), + "--value-mixed-every".to_string(), + remote.value.mixed_every.to_string(), + ]; + if let Some(sample_offset) = sample_offset { + command.extend([ + "--value-sample-offset".to_string(), + sample_offset.to_string(), + ]); + } + if let Some(total_samples_per_series) = total_samples_per_series { + command.extend([ + "--value-total-samples-per-series".to_string(), + total_samples_per_series.to_string(), + ]); + } + command +} + +async fn flush_remote_table( + client: &Client, + port: u16, + remote: &RemoteWrite, + reason: &str, + chunk_index: Option, +) -> Result { + let mut result = http_post_sql( + client, + port, + &format!("ADMIN FLUSH_TABLE({})", sql_string(&remote.physical_table)), + &remote.database, + ) + .await; + if !result["ok"].as_bool().unwrap_or(false) { + return Err(format!( + "ADMIN FLUSH_TABLE {} failed: {result}", + remote.physical_table + ) + .into()); + } + result + .as_object_mut() + .ok_or("flush result must be an object")? + .extend([ + ("physical_table".to_string(), json!(remote.physical_table)), + ("reason".to_string(), json!(reason)), + ("chunk_index".to_string(), json!(chunk_index)), + ]); + Ok(result) +} + +async fn poll_expected_count( + client: &Client, + port: u16, + table_name: &str, + database: &str, + expected_rows: u64, + visibility_timeout_seconds: u64, +) -> Result { + let sql = format!("SELECT count(*) FROM {}", sql_ident(table_name)); + let deadline = Instant::now() + Duration::from_secs(visibility_timeout_seconds); + let mut attempts = 0; + loop { + attempts += 1; + let mut result = http_post_sql(client, port, &sql, database).await; + let observed_rows = extract_count_value(&result); + let row_count_ok = + result["ok"].as_bool().unwrap_or(false) && observed_rows == Some(expected_rows); + result + .as_object_mut() + .ok_or("count result must be an object")? + .extend([ + ("expected_rows".to_string(), json!(expected_rows)), + ("observed_rows".to_string(), json!(observed_rows)), + ("attempts".to_string(), json!(attempts)), + ("row_count_ok".to_string(), json!(row_count_ok)), + ]); + if row_count_ok { + return Ok(result); + } + if Instant::now() >= deadline { + return Err(format!( + "expected {expected_rows} rows but observed {observed_rows:?} after {attempts} attempts" + ) + .into()); + } + tokio::time::sleep(Duration::from_millis(500)).await; + } +} + +fn json_number(value: f64) -> String { + serde_json::to_string(&value).unwrap_or_else(|_| value.to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::query_regression_runner::model::{PromStore, RemoteValue}; + + #[test] + fn schedules_remote_sample_chunks_and_flushes() { + let remote = RemoteWrite { + database: "public".to_string(), + metric: "metric".to_string(), + physical_table: "physical".to_string(), + series_count: 2, + samples_per_series: 5, + start_unix_millis: 100, + step_millis: 10, + chunk_series_count: 1, + timeout_seconds: 60, + sample_chunk_size: Some(2), + flush_every_sample_chunks: 2, + visibility_timeout_seconds: 30, + prom_store: PromStore { + pending_rows_flush_interval: "1s".to_string(), + max_batch_rows: 1, + max_concurrent_flushes: 1, + worker_channel_capacity: 1, + max_inflight_requests: 1, + }, + value: RemoteValue { + pattern: "linear".to_string(), + base: 0.0, + step: 1.0, + cardinality: 1, + seed: 0, + run_length: 1, + stall_every: 0, + stall_length: 0, + mixed_every: 0, + }, + storage: None, + read_bench: None, + }; + let chunks = sample_chunks(&remote).unwrap().unwrap(); + assert_eq!( + chunks, + vec![ + SampleChunk { + index: 1, + offset: 0, + samples_per_series: 2, + start_unix_millis: 100, + }, + SampleChunk { + index: 2, + offset: 2, + samples_per_series: 2, + start_unix_millis: 120, + }, + SampleChunk { + index: 3, + offset: 4, + samples_per_series: 1, + start_unix_millis: 140, + }, + ] + ); + assert_eq!( + scheduled_flushes(&chunks, remote.flush_every_sample_chunks), + vec![(2, "periodic"), (3, "final")] + ); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/remote/mod.rs b/src/cmd/src/bin/query_regression_runner/remote/mod.rs new file mode 100644 index 0000000000..946b00b61d --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/remote/mod.rs @@ -0,0 +1,183 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::path::{Path, PathBuf}; + +use serde_json::{Value, json}; + +use self::read_bench::run_read_bench; +use self::storage::{enforce_storage_thresholds, run_storage_inspection}; +use crate::query_regression_runner::model::DestinationConfig; +use crate::query_regression_runner::plan::normalized_remote_write; +use crate::query_regression_runner::{ + FinalizeRemoteArgs, PrepareRemoteArgs, RenderRemoteConfigArgs, Result, +}; + +mod ingest; +mod read_bench; +mod storage; + +pub(super) async fn run_render_remote_config(args: RenderRemoteConfigArgs) -> Result<()> { + ingest::run_render_remote_config(args).await +} + +pub(super) async fn run_prepare_remote(args: PrepareRemoteArgs) -> Result<()> { + ingest::run_prepare_remote(args).await +} + +pub(super) async fn run_finalize_remote(args: FinalizeRemoteArgs) -> Result<()> { + let (_, remote) = normalized_remote_write(&args.fixture_generator, &args.case)?; + let storage = remote.storage.as_ref().filter(|storage| storage.inspect); + let mut report: Value = serde_json::from_slice(&fs::read(&args.report)?)?; + let bench_root = args.report.parent().unwrap_or_else(|| Path::new(".")); + let mut inspections = Vec::new(); + let target_failed = { + let targets = report + .get_mut("targets") + .and_then(Value::as_array_mut) + .ok_or("report has no targets array")?; + for (name, data_home, destination) in [ + ( + "base", + args.base_data_home.as_deref(), + args.base_destination.as_deref(), + ), + ( + "candidate", + args.candidate_data_home.as_deref(), + args.candidate_destination.as_deref(), + ), + ] { + let (data_home, destination) = match (data_home, destination) { + (Some(data_home), None) => (data_home.to_path_buf(), None), + (None, Some(path)) => { + let destination: DestinationConfig = + toml::from_str(&fs::read_to_string(path)?)?; + ( + PathBuf::from(destination.data_home), + Some(path.to_path_buf()), + ) + } + (Some(_), Some(_)) => { + return Err(format!( + "{name}: --{name}-data-home and --{name}-destination are mutually exclusive" + ) + .into()); + } + (None, None) => { + return Err(format!( + "{name}: one of --{name}-data-home or --{name}-destination is required" + ) + .into()); + } + }; + let target = targets + .iter_mut() + .find(|target| target.get("name").and_then(Value::as_str) == Some(name)) + .ok_or_else(|| format!("report has no {name} target"))?; + let Some(storage) = storage else { + target + .as_object_mut() + .ok_or("report target must be an object")? + .insert( + "read_bench".to_string(), + json!({"status": "skipped", "reason": "storage inspection disabled"}), + ); + continue; + }; + let inspection = run_storage_inspection( + &args.fixture_generator, + &data_home, + destination.as_deref(), + storage, + )?; + let bench_dir = bench_root.join(name).join("read_bench"); + let read_bench = run_read_bench( + &args.candidate_bin, + &data_home, + &bench_dir, + remote.read_bench.as_ref(), + &inspection, + )?; + let inspection_failed = inspection["status"] == "failed"; + let bench_failed = read_bench["status"] == "failed"; + let target = target + .as_object_mut() + .ok_or("report target must be an object")?; + target.insert("storage_inspection".to_string(), inspection.clone()); + target.insert("read_bench".to_string(), read_bench); + if inspection_failed || bench_failed { + target.insert("status".to_string(), json!("failed")); + } + inspections.push(inspection); + } + targets.iter().any(|target| { + target["status"] == "failed" + || target["storage_inspection"]["status"] == "failed" + || target["read_bench"]["status"] == "failed" + }) + }; + let storage_thresholds = storage + .map(|storage| { + enforce_storage_thresholds( + storage, + inspections.first().unwrap_or(&Value::Null), + inspections.get(1).unwrap_or(&Value::Null), + ) + }) + .unwrap_or_default(); + let threshold_failed = { + let thresholds = report + .get_mut("thresholds") + .and_then(Value::as_array_mut) + .ok_or("report has no thresholds array")?; + thresholds.retain(|threshold| !is_storage_threshold_entry(threshold)); + thresholds.extend(storage_thresholds); + thresholds + .iter() + .any(|threshold| threshold["status"] == "failed") + }; + let failed = target_failed || threshold_failed; + report["status"] = json!(if failed { "failed" } else { "ok" }); + let text = format!("{}\n", serde_json::to_string_pretty(&report)?); + fs::write(&args.report, &text)?; + print!("{text}"); + if failed { + std::process::exit(1); + } + Ok(()) +} + +fn is_storage_threshold_entry(threshold: &Value) -> bool { + let Some(name) = threshold.get("threshold").and_then(Value::as_str) else { + return false; + }; + matches!( + name, + "max_candidate_total_file_size_regression_pct" + | "max_candidate_column_compressed_size_regression_pct" + | "max_candidate_column_uncompressed_size_regression_pct" + ) || (threshold.get("target").is_some() + && matches!( + name, + "min_files" + | "min_files_with_column" + | "max_total_file_size_bytes" + | "max_column_compressed_size_bytes" + | "max_column_uncompressed_size_bytes" + | "require_encodings" + | "forbid_encodings" + )) +} diff --git a/src/cmd/src/bin/query_regression_runner/remote/read_bench.rs b/src/cmd/src/bin/query_regression_runner/remote/read_bench.rs new file mode 100644 index 0000000000..6b20a31c18 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/remote/read_bench.rs @@ -0,0 +1,363 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; +use std::time::Instant; + +use regex::Regex; +use serde_json::{Value, json}; + +use crate::query_regression_runner::Result; +use crate::query_regression_runner::measure::median; +use crate::query_regression_runner::model::ReadBenchConfig; + +#[derive(Clone, Debug)] +struct BenchTarget { + relative_path: String, + table_dir: String, + region_id: String, + path_type: String, + file_id: String, +} + +pub(super) fn run_read_bench( + candidate_bin: &Path, + data_home: &Path, + bench_dir: &Path, + read_bench: Option<&ReadBenchConfig>, + inspection: &Value, +) -> Result { + let Some(read_bench) = read_bench.filter(|config| config.enabled) else { + return Ok(json!({ "status": "skipped", "reason": "read_bench disabled" })); + }; + if !read_bench.parquetbench && !read_bench.scanbench { + return Ok(json!({ "status": "skipped", "reason": "parquetbench and scanbench disabled" })); + } + let report = inspection + .get("summary") + .filter(|summary| summary.is_object()); + let files = report + .and_then(|report| report.get("files")) + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + let root = report + .and_then(|report| report.get("root")) + .and_then(Value::as_str) + .map(PathBuf::from) + .or_else(|| { + inspection + .get("root") + .and_then(Value::as_str) + .map(PathBuf::from) + }) + .ok_or("storage inspection has no root")?; + let mut targets = files + .iter() + .filter(|file| { + file.get("relative_path") + .and_then(Value::as_str) + .is_some_and(|path| path.ends_with(".parquet")) + && file + .get("columns") + .and_then(Value::as_array) + .is_some_and(|columns| !columns.is_empty()) + }) + .filter_map(|file| inspected_bench_target(data_home, &root, file).transpose()) + .collect::>>()?; + if let Some(max_files) = read_bench.max_files { + targets.truncate(max_files); + } + if targets.is_empty() { + return Ok( + json!({ "status": "failed", "reason": "no inspected data SST files available for read_bench" }), + ); + } + fs::create_dir_all(bench_dir)?; + let config_toml = bench_dir.join("bench.toml"); + let scan_json = bench_dir.join("scan.json"); + fs::write( + &config_toml, + format!( + "[storage]\ndata_home = \"{}\"\ntype = \"File\"\n\n[[region_engine]]\n[region_engine.mito]\n", + data_home.display() + ), + )?; + fs::write( + &scan_json, + format!( + "{}\n", + serde_json::to_string_pretty(&json!({ "projection_names": read_bench.projection }))? + ), + )?; + let mut parquet_runs = Vec::new(); + if read_bench.parquetbench { + for target in &targets { + let command = vec![ + candidate_bin.to_string_lossy().to_string(), + "datanode".to_string(), + "parquetbench".to_string(), + "--config".to_string(), + config_toml.to_string_lossy().to_string(), + "--region-id".to_string(), + target.region_id.clone(), + "--table-dir".to_string(), + target.table_dir.clone(), + "--file-id".to_string(), + target.file_id.clone(), + "--scan-config".to_string(), + scan_json.to_string_lossy().to_string(), + "--path-type".to_string(), + target.path_type.clone(), + "--iterations".to_string(), + read_bench.iterations.to_string(), + "--reader".to_string(), + read_bench.parquet_reader.clone(), + ]; + parquet_runs.push(run_bench_command(command, bench_target_value(target))?); + } + } + let mut scan_runs = Vec::new(); + if read_bench.scanbench { + for (table_dir, region_id, path_type, files) in group_scan_paths(&targets) { + let command = vec![ + candidate_bin.to_string_lossy().to_string(), + "datanode".to_string(), + "scanbench".to_string(), + "--config".to_string(), + config_toml.to_string_lossy().to_string(), + "--region-id".to_string(), + region_id.clone(), + "--table-dir".to_string(), + table_dir.clone(), + "--scan-config".to_string(), + scan_json.to_string_lossy().to_string(), + "--path-type".to_string(), + path_type.clone(), + "--scanner".to_string(), + read_bench.scan_scanner.clone(), + "--parallelism".to_string(), + read_bench.parallelism.to_string(), + "--iterations".to_string(), + read_bench.iterations.to_string(), + ]; + scan_runs.push(run_bench_command(command, json!({ "table_dir": table_dir, "region_id": region_id, "path_type": path_type, "files": files }))?); + } + } + let failed = parquet_runs + .iter() + .chain(&scan_runs) + .any(|run| run["status"] == "failed"); + Ok(json!({ + "status": if failed { "failed" } else { "ok" }, + "config_path": config_toml, + "scan_config_path": scan_json, + "parquetbench": parquet_runs, + "scanbench": scan_runs, + "aggregate": { + "parquetbench_median_average_ms": bench_median(&parquet_runs), + "scanbench_median_average_ms": bench_median(&scan_runs), + }, + })) +} + +fn inspected_bench_target( + data_home: &Path, + root: &Path, + file: &Value, +) -> Result> { + let relative_path = file + .get("relative_path") + .and_then(Value::as_str) + .unwrap_or_default(); + if relative_path.is_empty() { + return Ok(None); + } + let root_suffix = root.strip_prefix(data_home).map_err(|_| { + format!( + "storage inspection root {} is not under datanode data home {}", + root.display(), + data_home.display() + ) + })?; + let full_relative = root_suffix.join(relative_path); + let parts = full_relative + .components() + .map(|component| component.as_os_str().to_string_lossy().to_string()) + .collect::>(); + if !parts.last().is_some_and(|part| part.ends_with(".parquet")) { + return Ok(None); + } + let Some((region_index, table_id, region_seq)) = + parts.iter().enumerate().find_map(|(index, part)| { + parse_region_dir(part).map(|(table_id, region_seq)| (index, table_id, region_seq)) + }) + else { + return Ok(None); + }; + if region_index == 0 { + return Ok(None); + } + let file_index = parts.len() - 1; + let path_type = if region_index + 1 < file_index + && matches!(parts[region_index + 1].as_str(), "data" | "metadata") + { + parts[region_index + 1].clone() + } else { + "bare".to_string() + }; + if path_type == "metadata" { + return Ok(None); + } + let file_id = parts[file_index].trim_end_matches(".parquet").to_string(); + Ok(Some(BenchTarget { + relative_path: full_relative.to_string_lossy().to_string(), + table_dir: format!("{}/", parts[..region_index].join("/")), + region_id: format!("{table_id}:{region_seq}"), + path_type, + file_id, + })) +} + +fn parse_region_dir(part: &str) -> Option<(u64, u64)> { + let (table_id, region_seq) = part.split_once('_')?; + (region_seq.len() == 10 + && table_id.chars().all(|character| character.is_ascii_digit()) + && region_seq + .chars() + .all(|character| character.is_ascii_digit())) + .then(|| Some((table_id.parse().ok()?, region_seq.parse().ok()?)))? +} + +fn bench_target_value(target: &BenchTarget) -> Value { + json!({ + "relative_path": target.relative_path, + "table_dir": target.table_dir, + "region_id": target.region_id, + "path_type": target.path_type, + "file_id": target.file_id, + }) +} + +fn group_scan_paths(targets: &[BenchTarget]) -> Vec<(String, String, String, Vec)> { + let mut groups: Vec<(String, String, String, Vec)> = Vec::new(); + for target in targets { + if let Some((_, _, _, paths)) = + groups + .iter_mut() + .find(|(table_dir, region_id, path_type, _)| { + table_dir == &target.table_dir + && region_id == &target.region_id + && path_type == &target.path_type + }) + { + paths.push(target.relative_path.clone()); + } else { + groups.push(( + target.table_dir.clone(), + target.region_id.clone(), + target.path_type.clone(), + vec![target.relative_path.clone()], + )); + } + } + groups +} + +fn run_bench_command(command: Vec, mut run: Value) -> Result { + let started = Instant::now(); + let output = Command::new(&command[0]).args(&command[1..]).output()?; + let elapsed_seconds = started.elapsed().as_secs_f64(); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + let run = run.as_object_mut().ok_or("bench run must be an object")?; + run.extend([ + ("command".to_string(), json!(command)), + ("returncode".to_string(), json!(output.status.code())), + ("elapsed_seconds".to_string(), json!(elapsed_seconds)), + ("stdout".to_string(), json!(stdout)), + ("stderr".to_string(), json!(stderr)), + ( + "status".to_string(), + json!(if output.status.success() { + "ok" + } else { + "failed" + }), + ), + ]); + if let Some(average_ms) = parse_average_duration(run["stdout"].as_str().unwrap_or_default()) { + run.insert("average_ms".to_string(), json!(average_ms)); + } + Ok(Value::Object(run.clone())) +} + +fn parse_average_duration(stdout: &str) -> Option { + Regex::new(r"(?i)Average duration[^0-9]*([0-9.]+)\s*ms") + .ok()? + .captures(stdout)? + .get(1)? + .as_str() + .parse() + .ok() +} + +fn bench_median(runs: &[Value]) -> Option { + let values = runs + .iter() + .filter_map(|run| run.get("average_ms").and_then(Value::as_f64)) + .collect::>(); + (!values.is_empty()).then(|| median(&values)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_average_duration_and_groups_scan_paths() { + assert_eq!( + parse_average_duration("work\nAverage duration: 12.5 ms\n"), + Some(12.5) + ); + assert_eq!(parse_average_duration("completed"), None); + let targets = vec![ + BenchTarget { + relative_path: "data/a.parquet".to_string(), + table_dir: "data/".to_string(), + region_id: "1:2".to_string(), + path_type: "data".to_string(), + file_id: "a".to_string(), + }, + BenchTarget { + relative_path: "data/b.parquet".to_string(), + table_dir: "data/".to_string(), + region_id: "1:2".to_string(), + path_type: "data".to_string(), + file_id: "b".to_string(), + }, + ]; + assert_eq!( + group_scan_paths(&targets), + vec![( + "data/".to_string(), + "1:2".to_string(), + "data".to_string(), + vec!["data/a.parquet".to_string(), "data/b.parquet".to_string()], + )] + ); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/remote/storage.rs b/src/cmd/src/bin/query_regression_runner/remote/storage.rs new file mode 100644 index 0000000000..f6bf920b78 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/remote/storage.rs @@ -0,0 +1,241 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::path::Path; +use std::process::Command; +use std::time::Instant; + +use serde_json::{Value, json}; + +use crate::query_regression_runner::Result; +use crate::query_regression_runner::model::StorageConfig; +use crate::query_regression_runner::sql::{value_f64, value_u64}; + +pub(super) fn run_storage_inspection( + generator: &Path, + data_home: &Path, + destination: Option<&Path>, + storage: &StorageConfig, +) -> Result { + let mut command = vec![ + generator.to_string_lossy().to_string(), + "inspect-footer".to_string(), + "--column".to_string(), + storage.column.clone(), + ]; + let root = if let Some(destination) = destination { + command.push("--destination".to_string()); + command.push(destination.to_string_lossy().to_string()); + data_home.to_path_buf() + } else { + let root = storage + .root_suffix + .as_deref() + .map_or_else(|| data_home.to_path_buf(), |suffix| data_home.join(suffix)); + command.push("--root".to_string()); + command.push(root.to_string_lossy().to_string()); + root + }; + if storage.include_metadata_files { + command.push("--include-metadata-files".to_string()); + } + let started = Instant::now(); + let output = Command::new(generator).args(&command[1..]).output()?; + let elapsed_seconds = started.elapsed().as_secs_f64(); + let stdout = String::from_utf8_lossy(&output.stdout).to_string(); + let stderr = String::from_utf8_lossy(&output.stderr).to_string(); + if !output.status.success() { + return Err(format!( + "storage inspector failed: command={command:?}, returncode={:?}, elapsed_seconds={elapsed_seconds:.3}, stderr={:.2000}", + output.status.code(), stderr + ) + .into()); + } + let mut result = json!({ + "status": "ok", + "command": command, + "root": root, + "returncode": output.status.code(), + "elapsed_seconds": elapsed_seconds, + "stdout": stdout, + "stderr": stderr, + }); + match serde_json::from_str(result["stdout"].as_str().unwrap_or_default()) { + Ok(summary) => result["summary"] = summary, + Err(_) => { + result["summary_parse_error"] = result["stdout"].clone(); + result["status"] = json!("failed"); + } + } + Ok(result) +} + +fn storage_summary(inspection: &Value) -> Value { + inspection + .get("summary") + .and_then(|summary| summary.get("summary")) + .filter(|summary| summary.is_object()) + .cloned() + .unwrap_or_else(|| json!({})) +} + +pub(super) fn enforce_storage_thresholds( + storage: &StorageConfig, + base_inspection: &Value, + candidate_inspection: &Value, +) -> Vec { + let base = storage_summary(base_inspection); + let candidate = storage_summary(candidate_inspection); + let targets = [("base", &base), ("candidate", &candidate)]; + let mut results = Vec::new(); + for (threshold, field, limit) in [ + ("min_files", "file_count", storage.min_files), + ( + "min_files_with_column", + "files_with_column", + storage.min_files_with_column, + ), + ] { + for (target, summary) in &targets { + let actual = summary.get(field).cloned().unwrap_or(Value::Null); + let ok = value_u64(Some(&actual)).is_some_and(|actual| actual >= limit); + results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit })); + } + } + for (threshold, field, limit) in [ + ( + "max_total_file_size_bytes", + "total_file_size", + storage.max_total_file_size_bytes, + ), + ( + "max_column_compressed_size_bytes", + "column_compressed_size", + storage.max_column_compressed_size_bytes, + ), + ( + "max_column_uncompressed_size_bytes", + "column_uncompressed_size", + storage.max_column_uncompressed_size_bytes, + ), + ] { + let Some(limit) = limit else { continue }; + for (target, summary) in &targets { + let actual = summary.get(field).cloned().unwrap_or(Value::Null); + let ok = value_f64(Some(&actual)).is_some_and(|actual| actual <= limit as f64); + results.push(json!({ "target": target, "threshold": threshold, "status": if ok { "passed" } else { "failed" }, "actual": actual, "limit": limit })); + } + } + for (target, summary) in &targets { + let encodings = summary + .get("unique_encodings") + .and_then(Value::as_array) + .cloned() + .unwrap_or_default(); + for encoding in &storage.require_encodings { + results.push(json!({ "target": target, "threshold": "require_encodings", "encoding": encoding, "status": if encodings.iter().any(|value| value.as_str() == Some(encoding)) { "passed" } else { "failed" } })); + } + for encoding in &storage.forbid_encodings { + results.push(json!({ "target": target, "threshold": "forbid_encodings", "encoding": encoding, "status": if encodings.iter().all(|value| value.as_str() != Some(encoding)) { "passed" } else { "failed" } })); + } + } + for (threshold, field, limit) in [ + ( + "max_candidate_total_file_size_regression_pct", + "total_file_size", + storage.max_candidate_total_file_size_regression_pct, + ), + ( + "max_candidate_column_compressed_size_regression_pct", + "column_compressed_size", + storage.max_candidate_column_compressed_size_regression_pct, + ), + ( + "max_candidate_column_uncompressed_size_regression_pct", + "column_uncompressed_size", + storage.max_candidate_column_uncompressed_size_regression_pct, + ), + ] { + let Some(limit) = limit else { continue }; + let base_value = base.get(field).cloned().unwrap_or(Value::Null); + let candidate_value = candidate.get(field).cloned().unwrap_or(Value::Null); + let (Some(base_number), Some(candidate_number)) = ( + value_f64(Some(&base_value)).filter(|value| *value != 0.0), + value_f64(Some(&candidate_value)), + ) else { + results.push(json!({ "threshold": threshold, "status": "failed", "reason": "missing or zero base/candidate value", "base": base_value, "candidate": candidate_value })); + continue; + }; + let actual = (candidate_number - base_number) / base_number * 100.0; + results.push(json!({ "threshold": threshold, "status": if actual <= limit { "passed" } else { "failed" }, "actual_pct": actual, "limit_pct": limit, "base": base_value, "candidate": candidate_value })); + } + results +} + +#[cfg(test)] +mod tests { + use super::super::is_storage_threshold_entry; + use super::*; + + #[test] + fn enforces_storage_thresholds_for_each_target_and_regression() { + let storage: StorageConfig = serde_json::from_value(json!({ + "inspect": true, + "column": "value", + "root_suffix": null, + "include_metadata_files": false, + "min_files": 2, + "min_files_with_column": 1, + "require_encodings": ["PLAIN"], + "forbid_encodings": ["DELTA"], + "max_total_file_size_bytes": 105, + "max_column_compressed_size_bytes": null, + "max_column_uncompressed_size_bytes": null, + "max_candidate_total_file_size_regression_pct": 5.0, + "max_candidate_column_compressed_size_regression_pct": null, + "max_candidate_column_uncompressed_size_regression_pct": null + })) + .unwrap(); + let base = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 100, "unique_encodings": ["PLAIN"]}}}); + let candidate = json!({"summary": {"summary": {"file_count": 2, "files_with_column": 1, "total_file_size": 110, "unique_encodings": ["PLAIN"]}}}); + let results = enforce_storage_thresholds(&storage, &base, &candidate); + assert!( + results + .iter() + .any(|result| result["threshold"] == "max_total_file_size_bytes" + && result["target"] == "candidate" + && result["status"] == "failed") + ); + assert!(results.iter().any(|result| result["threshold"] + == "max_candidate_total_file_size_regression_pct" + && result["actual_pct"] == 10.0 + && result["status"] == "failed")); + assert!(results.iter().any( + |result| result["threshold"] == "require_encodings" && result["status"] == "passed" + )); + } + + #[test] + fn identifies_prior_storage_threshold_entries() { + assert!(is_storage_threshold_entry( + &json!({"target": "base", "threshold": "min_files"}) + )); + assert!(is_storage_threshold_entry(&json!({ + "threshold": "max_candidate_total_file_size_regression_pct" + }))); + assert!(!is_storage_threshold_entry(&json!({ + "threshold": "max_candidate_latency_regression_pct" + }))); + } +} diff --git a/src/cmd/src/bin/query_regression_runner/sql.rs b/src/cmd/src/bin/query_regression_runner/sql.rs new file mode 100644 index 0000000000..4d8baab2c8 --- /dev/null +++ b/src/cmd/src/bin/query_regression_runner/sql.rs @@ -0,0 +1,238 @@ +// Copyright 2023 Greptime Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::time::Instant; + +use reqwest::Client; +use serde_json::{Value, json}; + +use crate::query_regression_runner::Result; + +pub(super) fn sql_string(value: &str) -> String { + format!("'{}'", value.replace('\'', "''")) +} + +pub(super) fn extract_rows(body: &Value) -> Vec { + fn visit(body: &Value, rows: &mut Vec) { + match body { + Value::Object(object) => { + for key in ["data", "rows", "records", "output"] { + let Some(value) = object.get(key) else { + continue; + }; + if matches!(key, "data" | "rows") && value.is_array() { + rows.extend(value.as_array().unwrap().iter().cloned()); + } else { + visit(value, rows); + } + } + } + Value::Array(values) => { + if !values.is_empty() + && values + .iter() + .all(|value| !value.is_object() && !value.is_array()) + { + rows.push(body.clone()); + } else { + for value in values { + visit(value, rows); + } + } + } + _ => {} + } + } + + let mut rows = Vec::new(); + visit(body, &mut rows); + rows +} + +pub(super) fn row_value<'a>(row: &'a Value, index: usize, name: &str) -> Option<&'a Value> { + match row { + Value::Object(values) => [ + name.to_string(), + name.to_ascii_uppercase(), + name.to_ascii_lowercase(), + ] + .into_iter() + .find_map(|key| values.get(&key)), + Value::Array(values) => values.get(index), + _ => Some(row), + } +} + +pub(super) fn row_u64(row: &Value, index: usize, name: &str) -> Result { + let value = + row_value(row, index, name).ok_or_else(|| format!("missing {name} in row {row}"))?; + value + .as_u64() + .or_else(|| value.as_str().and_then(|value| value.parse().ok())) + .ok_or_else(|| format!("invalid {name} in row {row}").into()) +} + +pub(super) fn value_text(value: &Value) -> String { + match value { + Value::String(value) => value.clone(), + Value::Bool(value) => value.to_string(), + Value::Number(value) => value.to_string(), + Value::Null => "None".to_string(), + _ => value.to_string(), + } +} + +pub(super) fn extract_count_value(result: &Value) -> Option { + let row = result + .get("response")? + .get("data")? + .as_array()? + .first()? + .as_object()?; + row.iter() + .find(|(key, _)| { + let key = key.to_ascii_lowercase(); + key == "count(*)" || key.starts_with("count(") + }) + .and_then(|(_, value)| value_u64(Some(value))) +} + +pub(super) fn value_u64(value: Option<&Value>) -> Option { + value?.as_u64().or_else(|| value?.as_str()?.parse().ok()) +} + +pub(super) fn value_f64(value: Option<&Value>) -> Option { + value?.as_f64().or_else(|| value?.as_str()?.parse().ok()) +} + +pub(super) async fn http_post_sql(client: &Client, port: u16, sql: &str, db: &str) -> Value { + let started = Instant::now(); + let request = client + .post(format!("http://127.0.0.1:{port}/v1/sql")) + .form(&[("sql", sql), ("db", db), ("format", "json")]); + match request.send().await { + Ok(response) => { + let status = response.status().as_u16(); + match response.text().await { + Ok(raw) => { + let body = serde_json::from_str(&raw).unwrap_or_else(|_| json!({"raw": raw})); + let ok = status < 400 && !response_has_error(&body); + let mut sample = json!({ + "ok": ok, + "status": status, + "latency_ms": started.elapsed().as_secs_f64() * 1000.0, + "response": body, + "sql": sql, + }); + if status >= 400 { + sample + .as_object_mut() + .expect("HTTP samples are objects") + .insert("error".to_string(), Value::String(format!("HTTP {status}"))); + } + sample + } + Err(error) => json!({ + "ok": false, + "status": status, + "latency_ms": started.elapsed().as_secs_f64() * 1000.0, + "error": error.to_string(), + "sql": sql, + }), + } + } + Err(error) => json!({ + "ok": false, + "status": Value::Null, + "latency_ms": started.elapsed().as_secs_f64() * 1000.0, + "error": error.to_string(), + "sql": sql, + }), + } +} + +fn response_has_error(body: &Value) -> bool { + let Some(body) = body.as_object() else { + return false; + }; + ["error", "err_msg", "error_msg"] + .into_iter() + .any(|key| body.get(key).is_some_and(is_truthy)) + || body + .get("error_code") + .is_some_and(|value| !is_success_code(value)) + || (!body.contains_key("output") + && body + .get("code") + .is_some_and(|value| !is_success_code(value))) +} + +fn is_truthy(value: &Value) -> bool { + match value { + Value::Null => false, + Value::Bool(value) => *value, + Value::Number(value) => value.as_f64().is_none_or(|value| value != 0.0), + Value::String(value) => !value.is_empty(), + Value::Array(value) => !value.is_empty(), + Value::Object(value) => !value.is_empty(), + } +} + +fn is_success_code(value: &Value) -> bool { + let code = match value { + Value::String(value) => value.clone(), + Value::Bool(value) => value.to_string(), + Value::Number(value) => value.to_string(), + Value::Null => "None".to_string(), + _ => value.to_string(), + }; + matches!(code.to_lowercase().as_str(), "" | "0" | "success") +} + +pub(super) fn sql_ident(name: &str) -> String { + format!("\"{}\"", name.replace('"', "\"\"")) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn top_level_errors_do_not_inspect_rows() { + assert!(response_has_error(&json!({"error_code": 7}))); + assert!(response_has_error(&json!({"code": "bad"}))); + assert!(!response_has_error( + &json!({"output": [{"code": 7, "error": "row value"}]}) + )); + assert!(!response_has_error( + &json!({"error_code": "success", "output": []}) + )); + } + + #[test] + fn extracts_remote_write_count_from_data_map() { + assert_eq!( + extract_count_value(&json!({"response": {"data": [{"COUNT(*)": "12"}]}})), + Some(12) + ); + assert_eq!( + extract_count_value(&json!({"response": {"data": [{"count(value)": 7}]}})), + Some(7) + ); + assert_eq!( + extract_count_value(&json!({"response": {"data": [[]]}})), + None + ); + } +} diff --git a/tests/perf/AGENTS.md b/tests/perf/AGENTS.md index 7f64b283bd..45027eb1d0 100644 --- a/tests/perf/AGENTS.md +++ b/tests/perf/AGENTS.md @@ -3,22 +3,39 @@ - Keep GitHub Actions YAML thin. Put non-trivial control flow, case expansion, report generation, and metadata writing in scripts under `.github/scripts/`; workflow steps should mostly invoke those scripts. +- Runner lifecycle: the default path provisions one ephemeral Aliyun ECS + instance per run via `.github/scripts/aliyun-ecs-runner-provision.py` and + always releases it via `aliyun-ecs-runner-teardown.py`; a scheduled janitor + workflow sweeps leftovers. Build caches live on that instance's 50 GiB system + disk and are discarded with the VM. Runs do not share a workflow concurrency + group. The ECS custom image is built from the runner Dockerfile by + `.github/runner-scale-sets/query-regression/ecs-image/build-ecs-image.py`; + keep the Dockerfile the single source of the tool contract. Dispatching with + any other `runner` value treats it as a literal self-hosted runner label + (see `ecs-image/bootstrap-runner-host.sh` for preparing such a host). +- Scheduled nightly comparison lives in `query-regression-nightly.yml`: it + waits for a successful Nightly Build, then calls `query-regression.yml` + with the previous vs current nightly SHAs. Keep SHA selection in + `.github/scripts/query-regression-nightly-refs.py`. - Query regression PR runs should build base/candidate binaries once, then run the default case set. Do not hard-code a single case such as `promql_pushdown_7913` into the workflow path. - The case DSL is not required to keep compatibility inside this PR. When the - DSL changes, update TOML cases, the Python runner, Rust fixture generator, and + DSL changes, update TOML cases, the outer lifecycle script, Rust helpers, and docs together. - `[case]` is report metadata only. `[scenario]` is the executable regression configuration and must include `kind`, data layout, tables, queries, and thresholds. Rust owns case schema, defaults, validation, and normalized plan - output through `query_perf_fixture plan`; `tests/perf/query_regression_runner.py` - should only orchestrate from that normalized JSON. + output through `query_perf_fixture plan`. `.github/scripts/query-regression-run.py` + owns process lifecycle; `query_regression_runner` consumes normalized plans, + frontend endpoints, direct-SST materialization requests, and OTLP target/finalize + requests. - Keep the direct-SST generator generic. Issue-specific behavior belongs in case files and thresholds, not in Rust generator logic. - Before pushing perf harness changes, run at least: - - `uv run --no-project python -m py_compile .github/scripts/query-regression-run.py .github/scripts/query-regression-summary.py .github/scripts/query-regression-pr-metadata.py tests/perf/query_regression_runner.py` + - `uv run --no-project python -m py_compile .github/scripts/query-regression-run.py .github/scripts/query-regression-summary.py .github/scripts/query-regression-pr-metadata.py tests/perf/test_query_regression_runner_compaction_toctou.py tests/perf/test_query_regression_runner_otlp_trace_load.py` + - `uv run --no-project python tests/perf/test_query_regression_runner_compaction_toctou.py && uv run --no-project python tests/perf/test_query_regression_runner_otlp_trace_load.py` - `cargo fmt --all -- --check` - `cargo build -p cmd --bin query_perf_fixture --features dev-tools` - - dry-run the Python runner and Rust fixture generator against all built-in - cases when the DSL or workflow case selection changes. + - exercise the outer lifecycle script and Rust fixture generator against all + built-in cases when the DSL or workflow case selection changes. diff --git a/tests/perf/README.md b/tests/perf/README.md index c335f0a43b..a8869204ad 100644 --- a/tests/perf/README.md +++ b/tests/perf/README.md @@ -41,7 +41,7 @@ release-level realism. ## Prometheus remote-write scenario -The runner also supports `scenario.kind = "prom_remote_write_then_query"` for a +The outer driver supports `scenario.kind = "prom_remote_write_then_query"` for a bounded write-path smoke/regression flow. This path is explicit and separate from the direct-SST fixture path: it starts the base and candidate distributed clusters, writes deterministic Prometheus remote-write v1 samples through @@ -111,11 +111,10 @@ helper so non-linear value patterns use a stable global/per-series ordinal acros chunks. Case schema, value distribution defaults, storage defaults, and read-bench -defaults are owned by Rust. The Python runner calls -`query_perf_fixture plan --case ` and orchestrates the normalized JSON. -The same helper exposes `direct-sst`, `prom-remote-write`, and `inspect-footer` -subcommands; the old direct invocation (`query_perf_fixture --case ... --out-dir -...`) remains compatible. +defaults are owned by Rust. The outer CI driver calls +`query_perf_fixture plan --case ` once, then uses the normalized plan +to select the Rust runner lifecycle. The fixture helper exposes `direct-sst`, +`prom-remote-write`, and `inspect-footer` subcommands. Remote-write cases that need to validate storage output can add `[scenario.remote_write.storage]`. When present, the scenario body becomes: @@ -163,17 +162,15 @@ measures region scan cost, and query measurements still exercise the SQL/TQL frontend path. Treat all performance conclusions as release-only; debug builds are suitable only for command wiring and correctness checks. -The runner creates the configured database if needed, writes a per-target +`prepare-remote` creates the configured database if needed. The outer driver writes a per-target frontend config enabling `[prom_store]` with metric engine storage and a non-zero `pending_rows_flush_interval`, and validates that the logical metric table reaches `series_count * samples_per_series` rows before trusting the query measurements. -Use `--fixture-generator /path/to/query_perf_fixture` to provide the Rust helper. -Deprecated `--remote-write-generator` and `--storage-inspector` options are kept -only for CLI compatibility. `--fixture-only` is rejected for remote-write cases; -use `--dry-run` for planning. +Use `--fixture-generator /path/to/query_perf_fixture` to provide the Rust helper +to the outer driver. Large manual remote-write cases can set `sample_chunk_size` to split ingestion by -time. For each chunk, the runner invokes `query_perf_fixture prom-remote-write` with the +time. For each chunk, `prepare-remote` invokes `query_perf_fixture prom-remote-write` with the same series cardinality but a shorter `--samples-per-series` and an advanced `--start-unix-millis`. `flush_every_sample_chunks` controls periodic `ADMIN FLUSH_TABLE('')` calls; with `flush_every_sample_chunks = 1`, @@ -213,34 +210,27 @@ workflow dispatch accepts the `heavy` token to select this case. ## OTLP trace load scenario `scenario.kind = "otlp_trace_load"` runs a bounded native `otelgen` process -against each local distributed cluster. The runner excludes the configured -warmup window, derives throughput and mean request latency from GreptimeDB's -OTLP-specific metrics, flushes the trace table, and verifies its row count -against the accepted-span counter. The case is intentionally outside the -default set until its variance is known. +against each local distributed cluster. The outer driver runs the base target +alone with Rust `run-otlp-target`, fully stops it, then runs the candidate alone +and calls Rust `finalize-otlp` to aggregate metrics, thresholds, and the final +report. The case is intentionally outside the default set until its variance is +known. -Build base/candidate `greptime` binaries with the same profile, build the -candidate `query_perf_fixture`, then run the case explicitly: +Build base/candidate `greptime` binaries with the same profile and build the +candidate `query_perf_fixture` and `query_regression_runner`, then run the outer +driver explicitly: ```bash WORK_DIR="$(mktemp -d /tmp/query-perf-otlp.XXXXXX)" -REPORT="$WORK_DIR/trace-report.json" -python3 tests/perf/query_regression_runner.py \ - --case tests/perf/query_cases/otlp_trace_load/case.toml \ +uv run --no-project python .github/scripts/query-regression-run.py \ + --cases tests/perf/query_cases/otlp_trace_load/case.toml \ --base-bin /path/to/base/target/nightly/greptime \ --candidate-bin /path/to/candidate/target/nightly/greptime \ --fixture-generator /path/to/candidate/target/nightly/query_perf_fixture \ + --runner /path/to/candidate/target/nightly/query_regression_runner \ --otelgen-bin /path/to/otelgen \ - --work-dir "$WORK_DIR" \ - --output "$REPORT" -``` - -Install [YouPlot](https://github.com/red-data-tools/YouPlot) once, then render -all comparison metrics and threshold results in the terminal: - -```bash -brew install youplot -tests/perf/plot_otlp_trace_report.sh "$REPORT" + --work-dir "$WORK_DIR" +REPORT="$WORK_DIR/otlp_trace_load/query-regression-report.json" ``` For each target, `accepted_spans` should equal `table_rows`, and `failures` @@ -323,8 +313,8 @@ seed = 12345 # query, warmups, iterations, thresholds ``` -The runner currently supports `direct_readable_sst` and -`prom_remote_write_then_query`. +The outer driver currently supports `direct_readable_sst`, +`prom_remote_write_then_query`, and `otlp_trace_load`. ## Metrics @@ -340,120 +330,49 @@ Primary gates should compare query work rather than plan text: Plan details such as pushed filters are useful diagnostics, but should not be the main pass/fail signal. -## Runner MVP +## Runner lifecycle -`query_regression_runner.py` is the base-vs-candidate orchestration layer. The -current MVP parses a case, creates per-target work directories, and in real query -mode starts a local distributed cluster for each target: metasrv (memory-store, -region failover disabled), one datanode (`node_id=0`), and one frontend. It -creates the configured Mito table(s) through frontend HTTP SQL, discovers the -real one-region-per-table metadata via `information_schema`, stops only the -owning datanode, generates one shared direct-SST fixture per table using the -discovered `--region-id`, `--table-dir`, and `--table`, injects those region -subtrees into the datanode data home, restarts the datanode, then validates and -measures through frontend. Reports are written as JSON under the work directory. +`.github/scripts/query-regression-run.py` is the outer CI driver. It resolves the +base and candidate `greptime` binaries, candidate `query_perf_fixture`, and +candidate `query_regression_runner`; allocates disjoint localhost +meta/datanode/frontend HTTP, gRPC, MySQL, and Postgres ports for each target; and +writes component logs below `//logs/`. -The runner intentionally keeps metasrv alive for the whole target run because -memory-store metadata would otherwise be lost. It replaces only the discovered -datanode region directory under `data/greptime///...` with -generated SST files and a manifest checkpoint. For multi-table cases this is -repeated per table, enabling true JOIN fixtures while still requiring exactly one -region per table. Base and candidate must discover identical per-table -`table_dir` and `region_id`; otherwise the run fails. +The Rust runner consumes the normalized plan and endpoint ports. For +`direct_readable_sst`, the driver starts both clusters, runs `prepare-direct`, +stops only both datanodes, writes File-object-store destination TOMLs, invokes +`materialize` for every fixture and target, restarts datanodes, then runs +`measure` (with one frontend restart retry). `materialize` uses OpenDAL and only +replaces the fixture's exact region prefix in each target data home. -Multi-table direct-SST cases must use unique table names as well as unique -`(database, name)` pairs because the generator currently selects a table with -`--table `. Per-table fixture directories are derived from table index, -database, and table name with path-unsafe characters sanitized. +For `prom_remote_write_then_query`, it renders a target frontend configuration +with `render-remote-config`, starts both clusters, runs `prepare-remote` and +`measure`, stops both datanodes, then runs `finalize-remote`. Finalization owns +storage inspection and read-bench against the quiescent data homes. Both paths +write `query-regression-report.json` in the case work directory and always clean +up the remaining components. -Currently enforced threshold: - -- `max_candidate_latency_regression_pct`, based on client-side median latency. - -Server-side scan thresholds such as file ranges and scanned rows are planned for -a follow-up PR that extracts them from structured `EXPLAIN ANALYZE VERBOSE` -output. Do not add those threshold keys until the runner enforces them. - -Dry-run example: +For local orchestration after building the three binaries, invoke the outer +driver directly: ```bash -uv run --no-project python tests/perf/query_regression_runner.py \ - --case tests/perf/query_cases/promql_pushdown_7913/case.toml \ - --base-bin /path/to/base/greptime \ - --candidate-bin /path/to/candidate/greptime \ - --work-dir /tmp/query-perf-work \ - --dry-run -``` - -With a fixture generator: - -```bash -uv run --no-project python tests/perf/query_regression_runner.py \ - --case tests/perf/query_cases/promql_pushdown_7913/case.toml \ +uv run --no-project python .github/scripts/query-regression-run.py \ + --cases tests/perf/query_cases/smoke_direct_sst/case.toml \ --base-bin /path/to/base/greptime \ --candidate-bin /path/to/candidate/greptime \ --fixture-generator /path/to/query_perf_fixture \ - --fixture-cache-dir /mnt/query-regression-fixtures \ - --allow-large-fixture \ - --work-dir /tmp/query-perf-work + --runner /path/to/query_regression_runner \ + --work-dir /tmp/query-regression-work ``` -This mode launches metasrv, datanode, and frontend for each target with explicit -localhost HTTP/gRPC/MySQL/Postgres ports and writes component stdout/stderr under -each target's `logs/` directory. - -By default query mode requires fresh base/candidate work directories and fails if -either target directory already exists with contents. Use `--reuse-work-dir` only -when intentionally debugging an existing run directory. SQL HTTP requests default -to a 120 second timeout; override with `--http-timeout ` for slow lab -runs. - -For large direct-SST fixtures, pass `--fixture-cache-dir ` to store generated -fixtures in a persistent content-addressed cache keyed by case name and fixture -data configuration. Query and threshold edits reuse the same cached data as long -as the scenario layout and table definitions do not change. Cached fixtures are -reused automatically when their `summary.json` -matches the discovered table/region metadata; incompatible entries are -regenerated instead of reused. Fixture materialization keeps base and candidate -data directories isolated, but copies files efficiently by trying filesystem -reflinks first, hardlinks for immutable SST/object files next, and normal copies -as a fallback. Manifest files are reflinked or copied, not hardlinked. - -Fixture generator smoke test: +The Rust runner subcommands are also useful for focused diagnostics: ```bash -cargo run -p cmd --bin query_perf_fixture --features dev-tools -- \ - direct-sst \ - --case tests/perf/query_cases/smoke_direct_sst/case.toml \ - --out-dir /tmp/query-perf-smoke -``` - -Runner smoke test with fixture generation only: - -```bash -uv run --no-project python tests/perf/query_regression_runner.py \ - --case tests/perf/query_cases/smoke_direct_sst/case.toml \ - --base-bin /path/to/query_perf_fixture \ - --candidate-bin /path/to/query_perf_fixture \ - --fixture-generator /path/to/query_perf_fixture \ - --work-dir /tmp/query-perf-runner-smoke \ - --fixture-only -``` - -`--fixture-only` preserves the earlier smoke behavior: it does not start -standalone servers, and it materializes the generated fixture into base and -candidate data directories for plumbing validation. - -Remote-write runner dry-run: - -```bash -uv run --no-project python tests/perf/query_regression_runner.py \ - --case tests/perf/query_cases/prom_remote_write_seeded_random/case.toml \ - --base-bin /path/to/base/greptime \ - --candidate-bin /path/to/candidate/greptime \ - --fixture-generator /path/to/query_perf_fixture \ - --work-dir /tmp/query-perf-remote-write \ - --dry-run +query_regression_runner prepare-direct --case --fixture-generator \ + --base-http-port --candidate-http-port --fixture-dir --output +query_regression_runner materialize --fixture-dir --destination +query_regression_runner measure --case --fixture-generator \ + --base-http-port --candidate-http-port --output ``` ## GitHub Actions @@ -462,14 +381,17 @@ uv run --no-project python tests/perf/query_regression_runner.py \ query regression runs. It builds its own binaries for now: - base `greptime` from the PR base commit, or `workflow_dispatch` `base_ref` -- candidate `greptime` and `query_perf_fixture` from the PR merge ref/current - candidate checkout -- runner and summary formatter from the candidate checkout +- candidate `greptime`, `query_perf_fixture`, and `query_regression_runner` from + the PR merge ref/current candidate checkout +- outer driver and summary formatter from the candidate checkout -The workflow builds base and candidate `greptime` as normal release-equivalent -binaries. Candidate `query_perf_fixture` is the extra head-side helper binary; -the runner uses candidate `greptime datanode parquetbench/scanbench` as the -read-bench tool against each target's data directory. +The default `aliyun-ecs` runner provisions one ephemeral 50 GiB-system-disk +Aliyun ECS instance per run and tears it down afterwards; the scheduled janitor +reclaims leftovers. The workflow builds base and candidate `greptime` as normal +release-equivalent binaries. Candidate `query_perf_fixture` and +`query_regression_runner` are the extra head-side helpers; `finalize-remote` +uses candidate `greptime datanode parquetbench/scanbench` as the read-bench tool +against each target's data directory. The workflow runs automatically only when `query-regression` or `heavy-regression` is added to a non-draft PR; it does not rerun on pushes, ready-for-review, or diff --git a/tests/perf/plot_otlp_trace_report.sh b/tests/perf/plot_otlp_trace_report.sh deleted file mode 100755 index fa1e5ed796..0000000000 --- a/tests/perf/plot_otlp_trace_report.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/usr/bin/env bash - -set -euo pipefail - -if [[ $# -ne 1 ]]; then - printf 'Usage: %s REPORT.json\n' "${0##*/}" >&2 - exit 2 -fi - -report="$1" - -[[ -f "${report}" ]] || { - printf 'Report does not exist: %s\n' "${report}" >&2 - exit 1 -} -command -v jq >/dev/null || { - printf 'jq is required\n' >&2 - exit 1 -} -command -v uplot >/dev/null || { - printf 'YouPlot is required; install it with: brew install youplot\n' >&2 - exit 1 -} - -jq -e ' - (.status | type == "string") and - (.targets | type == "array" and length > 0) and - all(.targets[]; - (.name | type == "string") and - (.metrics.accepted_spans | type == "number") and - (.visibility.observed_rows | type == "number") and - (.metrics.accepted_spans_per_second | type == "number") and - (.metrics.mean_http_latency_ms | type == "number") and - (.metrics.failure_count | type == "number") - ) -' "${report}" >/dev/null || { - printf 'Report is not a completed OTLP trace comparison: %s\n' "${report}" >&2 - exit 1 -} - -plot_metric() { - local title="$1" - local filter="$2" - local color="$3" - - printf '\n' - jq -r ".targets[] | [.name, ${filter}] | @tsv" "${report}" | - uplot bar -t "${title}" -w 80 -c "${color}" -o - -} - -jq -r '"Status: \(.status)"' "${report}" -row_count_status=0 -jq -r ' - "\nAccepted spans vs visible table rows", - (.targets[] | - " [\(if .metrics.accepted_spans == .visibility.observed_rows then "passed" else "failed" end)] \(.name): accepted=\(.metrics.accepted_spans), visible=\(.visibility.observed_rows)" - ) -' "${report}" -jq -e 'all(.targets[]; .metrics.accepted_spans == .visibility.observed_rows)' "${report}" >/dev/null || row_count_status=1 - -plot_metric "Accepted spans" ".metrics.accepted_spans" blue -plot_metric "Visible table rows" ".visibility.observed_rows" green -plot_metric "Throughput (spans/s, higher is better)" ".metrics.accepted_spans_per_second" cyan -plot_metric "Mean HTTP latency (ms, lower is better)" ".metrics.mean_http_latency_ms" yellow -plot_metric "Failures (lower is better)" ".metrics.failure_count" red - -jq -r ' - "\nThresholds", - (.thresholds[] | - if has("actual_pct") then - " [\(.status)] \(.threshold): \(.actual_pct)% (limit \(.limit_pct)%; base \(.base), candidate \(.candidate))" - elif has("actual") then - " [\(.status)] \(.threshold) [\(.target)]: \(.actual) (limit \(.limit))" - else - " [\(.status)] \(.threshold): \(.reason // "no measured value")" - end - ) -' "${report}" - -exit "${row_count_status}" diff --git a/tests/perf/query_regression_runner.py b/tests/perf/query_regression_runner.py deleted file mode 100644 index 724c38ebf1..0000000000 --- a/tests/perf/query_regression_runner.py +++ /dev/null @@ -1,1663 +0,0 @@ -#!/usr/bin/env python3 -# Copyright 2023 Greptime Team -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Base-vs-candidate query performance regression runner.""" - -from __future__ import annotations - -import argparse -import fcntl -import hashlib -import json -import math -import os -import re -import shutil -import socket -import statistics -import subprocess -import sys -import time -import tomllib -import urllib.error -import urllib.parse -import urllib.request -from dataclasses import dataclass -from pathlib import Path -from typing import Any - - -FICLONE = 0x40049409 -OTLP_TRACE_METRICS = { - "greptime_frontend_otlp_traces_rows", - "greptime_frontend_otlp_traces_failure_count", - "greptime_servers_http_otlp_traces_elapsed_sum", - "greptime_servers_http_otlp_traces_elapsed_count", -} -PROMETHEUS_SAMPLE_RE = re.compile(r"^([A-Za-z_:][A-Za-z0-9_:]*)(?:\{.*\})?\s+([^\s]+)(?:\s+.*)?$") - - -@dataclass(frozen=True) -class RunTarget: - name: str - binary: Path - work_dir: Path - data_dir: Path - fixture_dir: Path - report_path: Path - http_port: int - grpc_port: int - mysql_port: int - postgres_port: int - metasrv_rpc_port: int - metasrv_http_port: int - datanode_rpc_port: int - datanode_http_port: int - datanode_data_dir: Path - - -def parse_args() -> argparse.Namespace: - p = argparse.ArgumentParser(description="Run a query perf case against base and candidate binaries.") - p.add_argument("--case", required=True, type=Path) - p.add_argument("--base-bin", required=True, type=Path) - p.add_argument("--candidate-bin", required=True, type=Path) - p.add_argument("--fixture-generator", type=Path) - p.add_argument("--otelgen-bin", type=Path) - p.add_argument("--remote-write-generator", type=Path, help="deprecated: use --fixture-generator query_perf_fixture") - p.add_argument("--storage-inspector", type=Path, help="deprecated: use --fixture-generator query_perf_fixture") - p.add_argument("--work-dir", required=True, type=Path) - p.add_argument("--output", type=Path, help="write the final JSON report to this file instead of stdout") - p.add_argument("--fixture-cache-dir", type=Path, help="persistent directory for generated fixtures, keyed by case content") - p.add_argument("--reuse-fixture", action="store_true") - p.add_argument("--allow-large-fixture", action="store_true") - p.add_argument("--dry-run", action="store_true") - p.add_argument("--fixture-only", action="store_true", help="old smoke mode: generate/materialize fixture only") - p.add_argument("--reuse-work-dir", action="store_true", help="allow non-empty base/candidate work dirs") - p.add_argument("--http-timeout", type=float, default=120.0, help="HTTP SQL timeout seconds") - return p.parse_args() - - -def load_case(path: Path) -> dict[str, Any]: - with path.open("rb") as f: - return tomllib.load(f) - - -def load_normalized_case(case_path: Path, fixture_generator: Path | None) -> dict[str, Any]: - if fixture_generator is None: - raise ValueError("--fixture-generator query_perf_fixture is required for Rust-owned case planning") - result = run_command([str(fixture_generator), "plan", "--case", str(case_path)]) - if result["returncode"] != 0: - raise RuntimeError(f"query_perf_fixture plan failed: {result['stderr'][:2000]}") - plan = json.loads(result["stdout"]) - return {"case": load_case(case_path).get("case", {}), "scenario": plan["scenario"], "schema_version": plan.get("schema_version")} - - -def require_binary(path: Path, name: str, *, dry_run: bool) -> None: - if dry_run: - return - if not path.exists(): - raise FileNotFoundError(f"{name} binary does not exist: {path}") - if not os.access(path, os.X_OK): - raise PermissionError(f"{name} binary is not executable: {path}") - - -def allocate_ports(n: int) -> list[int]: - socks = [] - try: - for _ in range(n): - s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - s.bind(("127.0.0.1", 0)) - socks.append(s) - return [s.getsockname()[1] for s in socks] - finally: - for s in socks: - s.close() - - -def make_target(name: str, binary: Path, root: Path, ports: list[int], fixture_dir: Path | None = None) -> RunTarget: - work_dir = root / name - return RunTarget( - name, - binary, - work_dir, - work_dir / "cluster_data", - fixture_dir or root / "fixture", - work_dir / "report.json", - ports[4], - ports[5], - ports[6], - ports[7], - ports[0], - ports[1], - ports[2], - ports[3], - work_dir / "datanode-0" / "data", - ) - - -def run_command(cmd: list[str], cwd: Path | None = None) -> dict[str, Any]: - started = time.monotonic() - proc = subprocess.run(cmd, cwd=cwd, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False) - return {"cmd": cmd, "cwd": str(cwd) if cwd else None, "returncode": proc.returncode, "elapsed_seconds": time.monotonic() - started, "stdout": proc.stdout, "stderr": proc.stderr} - - -def sql_ident(name: str) -> str: - return '"' + name.replace('"', '""') + '"' - - -def sql_string(value: str) -> str: - return "'" + value.replace("'", "''") + "'" - - -def column_sql(col: dict[str, Any]) -> str: - return f"{sql_ident(col['name'])} {col['type']}" - - -def scenario(case: dict[str, Any]) -> dict[str, Any]: - value = case.get("scenario") - if not isinstance(value, dict): - raise ValueError("case requires [scenario] with a supported kind") - kind = value.get("kind") - supported = ("direct_readable_sst", "prom_remote_write_then_query", "otlp_trace_load") - if kind not in supported: - raise ValueError(f"unsupported scenario kind {kind!r}; supported: {', '.join(supported)}") - return value - - -def case_tables(case: dict[str, Any]) -> list[dict[str, Any]]: - value = scenario(case) - if value.get("kind") == "otlp_trace_load": - load = value["load"] - return [{"database": load["database"], "name": load["table"], "engine": "trace", "validate_show_create_engine": False}] - if value.get("kind") == "prom_remote_write_then_query": - remote = value["remote_write"] - metric = remote["metric"] - database = remote["database"] - return [{"database": database, "name": metric, "engine": "metric", "validate_show_create_engine": False}] - tables = value.get("tables") or [] - if not tables or value.get("layout", {}).get("regions") != 1: - raise ValueError("runner supports one or more tables and exactly one region per table") - pairs = [(table.get("database"), table.get("name")) for table in tables] - if len(set(pairs)) != len(pairs): - raise ValueError("duplicate (database, name) table entries are not supported") - names = [table.get("name") for table in tables] - if len(set(names)) != len(names): - raise ValueError("duplicate table names are not supported because fixture generator --table selects by name") - return list(tables) - - -def fixture_subdir(table: dict[str, Any], index: int) -> str: - raw = f"{index:02d}_{table['database']}_{table['name']}" - safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", raw).strip("._-") - return safe or f"table_{index:02d}" - - -def safe_path_component(raw: str) -> str: - safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", raw).strip("._-") - return safe or "query_perf_case" - - -def fixture_root(work_root: Path, case_path: Path, case: dict[str, Any], fixture_cache_dir: Path | None) -> Path: - if fixture_cache_dir is None: - return work_root / "fixture" - case_name = case.get("case", {}).get("name") or case_path.parent.name or case_path.stem - fixture_config = dict(scenario(case)) - fixture_config.pop("queries", None) - digest = hashlib.sha256(json.dumps(fixture_config, sort_keys=True).encode()).hexdigest()[:16] - return fixture_cache_dir.resolve() / f"{safe_path_component(case_name)}-{digest}" - - -def table_fixture_dir(root: Path, tables: list[dict[str, Any]], table: dict[str, Any], index: int) -> Path: - return root if len(tables) == 1 else root / fixture_subdir(table, index) - - -def create_table_sql(table: dict[str, Any]) -> str: - cols = ",\n ".join(column_sql(c) for c in table["columns"]) - pk = ", ".join(sql_ident(c) for c in table.get("primary_key", [])) - opts = [] - if "append_mode" in table: - opts.append(("append_mode", str(table["append_mode"]).lower())) - if table.get("sst_format"): - opts.append(("sst_format", table["sst_format"])) - with_sql = "" - if opts: - with_sql = "\nWITH (" + ", ".join(f"'{k}'='{v}'" for k, v in opts) + ")" - return f"CREATE TABLE {sql_ident(table['name'])} (\n {cols},\n TIME INDEX ({sql_ident(table['time_index'])}),\n PRIMARY KEY ({pk})\n) ENGINE=mito{with_sql};" - - -def planned_queries(case: dict[str, Any]) -> list[dict[str, Any]]: - queries = scenario(case).get("queries", []) - if isinstance(queries, dict): - return [{"name": name, **value} for name, value in queries.items()] - return list(queries) - - -def http_post_sql(port: int, sql: str, db: str, timeout: float) -> dict[str, Any]: - data = urllib.parse.urlencode({"sql": sql, "db": db, "format": "json"}).encode() - req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/sql", data=data, method="POST") - started = time.monotonic() - elapsed_ms = (time.monotonic() - started) * 1000.0 - try: - with urllib.request.urlopen(req, timeout=timeout) as resp: - raw = resp.read().decode() - status = resp.status - elapsed_ms = (time.monotonic() - started) * 1000.0 - try: - body: Any = json.loads(raw) - except json.JSONDecodeError: - body = {"raw": raw} - ok = status < 400 and not response_has_error(body) - return {"ok": ok, "status": status, "latency_ms": elapsed_ms, "response": body, "sql": sql} - except urllib.error.HTTPError as e: - elapsed_ms = (time.monotonic() - started) * 1000.0 - raw = e.read().decode(errors="replace") - try: - body = json.loads(raw) - except json.JSONDecodeError: - body = {"raw": raw} - return { - "ok": False, - "status": e.code, - "latency_ms": elapsed_ms, - "response": body, - "error": repr(e), - "sql": sql, - } - except Exception as e: # noqa: BLE001 - report HTTP/query failures as JSON - elapsed_ms = (time.monotonic() - started) * 1000.0 - return {"ok": False, "status": None, "latency_ms": elapsed_ms, "error": repr(e), "sql": sql} - - -def response_has_error(body: Any) -> bool: - """Detect top-level GreptimeDB HTTP error envelopes without inspecting rows. - - Query outputs may legitimately contain columns named `code` or `error`, so - this intentionally avoids recursive checks through result rows. - """ - if isinstance(body, dict): - if body.get("error") or body.get("err_msg") or body.get("error_msg"): - return True - if "error_code" in body and str(body.get("error_code", "")).lower() not in ("", "0", "success"): - return True - if "code" in body and "output" not in body and str(body.get("code", "")).lower() not in ("", "0", "success"): - return True - return False - - -def wait_health(port: int, timeout_s: float = 60.0) -> None: - deadline = time.monotonic() + timeout_s - last = None - while time.monotonic() < deadline: - try: - with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=2) as r: - if r.status < 500: - return - except Exception as e: # noqa: BLE001 - diagnostic loop - last = e - time.sleep(0.5) - raise TimeoutError(f"health check timed out on port {port}: {last}") - - -class DistributedCluster: - """Local metasrv + one datanode + frontend cluster for query-mode runs.""" - - def __init__(self, target: RunTarget): - self.target = target - self.procs: dict[str, subprocess.Popen[bytes]] = {} - - def component_report(self) -> dict[str, Any]: - return { - "metasrv": { - "grpc": f"127.0.0.1:{self.target.metasrv_rpc_port}", - "http": f"127.0.0.1:{self.target.metasrv_http_port}", - "logs": str(self.target.work_dir / "logs" / "metasrv"), - }, - "datanode_0": { - "node_id": 0, - "grpc": f"127.0.0.1:{self.target.datanode_rpc_port}", - "http": f"127.0.0.1:{self.target.datanode_http_port}", - "data_home": str(self.target.datanode_data_dir), - "logs": str(self.target.work_dir / "logs" / "datanode-0"), - }, - "frontend": { - "http": f"127.0.0.1:{self.target.http_port}", - "grpc": f"127.0.0.1:{self.target.grpc_port}", - "mysql": f"127.0.0.1:{self.target.mysql_port}", - "postgres": f"127.0.0.1:{self.target.postgres_port}", - "logs": str(self.target.work_dir / "logs" / "frontend"), - }, - } - - def _spawn(self, name: str, args: list[str]) -> None: - logs = self.target.work_dir / "logs" / name - logs.mkdir(parents=True, exist_ok=True) - with (logs / "stdout.log").open("ab") as out, (logs / "stderr.log").open("ab") as err: - self.procs[name] = subprocess.Popen(args, stdout=out, stderr=err) - - def _ensure_metasrv_alive(self) -> None: - proc = self.procs.get("metasrv") - if proc is None or proc.poll() is not None: - raise RuntimeError("metasrv exited; memory-store metadata is no longer valid") - - def start_metasrv(self) -> None: - log_dir = self.target.work_dir / "logs" / "metasrv" - self._spawn( - "metasrv", - [ - str(self.target.binary), - "metasrv", - "start", - "--grpc-bind-addr", - f"127.0.0.1:{self.target.metasrv_rpc_port}", - "--grpc-server-addr", - f"127.0.0.1:{self.target.metasrv_rpc_port}", - "--http-addr", - f"127.0.0.1:{self.target.metasrv_http_port}", - "--backend", - "memory-store", - "--enable-region-failover", - "false", - "--log-dir", - str(log_dir), - ], - ) - wait_health(self.target.metasrv_http_port) - - def start_datanode(self) -> None: - self._ensure_metasrv_alive() - log_dir = self.target.work_dir / "logs" / "datanode-0" - self.target.datanode_data_dir.mkdir(parents=True, exist_ok=True) - self._spawn( - "datanode", - [ - str(self.target.binary), - "datanode", - "start", - "--grpc-bind-addr", - f"127.0.0.1:{self.target.datanode_rpc_port}", - "--grpc-server-addr", - f"127.0.0.1:{self.target.datanode_rpc_port}", - "--http-addr", - f"127.0.0.1:{self.target.datanode_http_port}", - "--data-home", - str(self.target.datanode_data_dir), - "--log-dir", - str(log_dir), - "--node-id", - "0", - "--metasrv-addrs", - f"127.0.0.1:{self.target.metasrv_rpc_port}", - ], - ) - wait_health(self.target.datanode_http_port) - - def start_frontend(self, config_file: Path | None = None) -> None: - self._ensure_metasrv_alive() - log_dir = self.target.work_dir / "logs" / "frontend" - cmd = [ - str(self.target.binary), - "frontend", - "start", - ] - if config_file is not None: - cmd += ["--config-file", str(config_file)] - cmd += [ - "--metasrv-addrs", - f"127.0.0.1:{self.target.metasrv_rpc_port}", - "--http-addr", - f"127.0.0.1:{self.target.http_port}", - "--grpc-bind-addr", - f"127.0.0.1:{self.target.grpc_port}", - "--grpc-server-addr", - f"127.0.0.1:{self.target.grpc_port}", - "--mysql-addr", - f"127.0.0.1:{self.target.mysql_port}", - "--postgres-addr", - f"127.0.0.1:{self.target.postgres_port}", - "--log-dir", - str(log_dir), - ] - self._spawn("frontend", cmd) - wait_health(self.target.http_port) - - def start_all(self, frontend_config: Path | None = None) -> None: - self.start_metasrv() - self.start_datanode() - self.start_frontend(frontend_config) - - def stop_component(self, name: str) -> None: - proc = self.procs.pop(name, None) - if proc is None: - return - proc.terminate() - try: - proc.wait(timeout=20) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait(timeout=20) - - def stop_all(self) -> None: - for name in ("frontend", "datanode", "metasrv"): - self.stop_component(name) - - def restart_frontend(self) -> None: - self.stop_component("frontend") - self.start_frontend() - - -def extract_rows(body: Any) -> list[Any]: - """Extract rows from GreptimeDB HTTP JSON in a tolerant way.""" - rows: list[Any] = [] - if isinstance(body, dict): - for key in ("data", "rows", "records", "output"): - if key in body: - value = body[key] - if key in ("data", "rows") and isinstance(value, list): - rows.extend(value) - else: - rows.extend(extract_rows(value)) - elif isinstance(body, list): - if body and all(not isinstance(v, (dict, list)) for v in body): - rows.append(body) - else: - for item in body: - rows.extend(extract_rows(item)) - return rows - - -def row_value(row: Any, idx: int, name: str) -> Any: - if isinstance(row, dict): - for key in (name, name.upper(), name.lower()): - if key in row: - return row[key] - return None - if isinstance(row, list): - return row[idx] - return row - - -def discover_region_via_frontend(target: RunTarget, table: dict[str, Any], http_timeout: float) -> dict[str, Any]: - schema = table["database"] - table_name = table["name"].replace("'", "''") - schema_name = schema.replace("'", "''") - table_sql = f"SELECT table_id FROM information_schema.tables WHERE table_schema = '{schema_name}' AND table_name = '{table_name}'" - table_result = http_post_sql(target.http_port, table_sql, schema, http_timeout) - if not table_result["ok"]: - raise RuntimeError(f"table_id discovery failed: {table_result}") - table_rows = extract_rows(table_result.get("response")) - if len(table_rows) != 1: - raise RuntimeError(f"expected one information_schema.tables row, got {len(table_rows)}: {table_result}") - table_id = int(row_value(table_rows[0], 0, "table_id")) - - region_sql = f"SELECT region_id, peer_id, peer_addr, is_leader, status FROM information_schema.region_peers WHERE table_schema = '{schema_name}' AND table_name = '{table_name}'" - region_result = http_post_sql(target.http_port, region_sql, schema, http_timeout) - if not region_result["ok"]: - raise RuntimeError(f"region_peers discovery failed: {region_result}") - region_rows = extract_rows(region_result.get("response")) - if len(region_rows) != 1: - raise RuntimeError(f"expected one information_schema.region_peers row, got {len(region_rows)}: {region_result}") - row = region_rows[0] - region_id = int(row_value(row, 0, "region_id")) - peer_id = int(row_value(row, 1, "peer_id")) - peer_addr = row_value(row, 2, "peer_addr") - is_leader = row_value(row, 3, "is_leader") - status = row_value(row, 4, "status") - if str(is_leader).lower() not in ("yes", "true"): - raise RuntimeError(f"expected leader region peer, got is_leader={is_leader}: {region_result}") - if str(status).upper() != "ALIVE": - raise RuntimeError(f"expected ALIVE region peer, got status={status}: {region_result}") - if peer_id != 0: - raise RuntimeError(f"expected region leader on datanode peer_id=0, got {peer_id}") - computed_table_id = region_id >> 32 - region_seq = region_id & 0xFFFFFFFF - if computed_table_id != table_id: - raise RuntimeError(f"table_id mismatch: tables={table_id}, region_id-derived={computed_table_id}") - table_dir = f"data/greptime/{schema}/{table_id}/" - region_dir = f"data/greptime/{schema}/{table_id}/{table_id}_{region_seq:010}" - return { - "catalog": "greptime", - "schema": schema, - "table_id": table_id, - "region_id": region_id, - "region_seq": region_seq, - "table_dir": table_dir, - "region_dir": region_dir, - "peer_id": peer_id, - "peer_addr": peer_addr, - "is_leader": is_leader, - "status": status, - "discovery_queries": {"table": table_result, "region": region_result}, - } - - -def assert_fixture_summary(summary: dict[str, Any], *, region_id: int | None, table_dir: str | None, region_dir: str | None = None, table_name: str | None = None, database: str | None = None) -> None: - if table_name is not None and summary.get("table") != table_name: - raise RuntimeError(f"fixture table mismatch: {summary.get('table')} != {table_name}") - if database is not None and summary.get("database") != database: - raise RuntimeError(f"fixture database mismatch: {summary.get('database')} != {database}") - if region_id is not None and int(summary["region_id"]) != region_id: - raise RuntimeError(f"fixture region_id mismatch: {summary['region_id']} != {region_id}") - if table_dir is not None and summary["table_dir"] != table_dir: - raise RuntimeError(f"fixture table_dir mismatch: {summary['table_dir']} != {table_dir}") - if region_dir is not None and summary["region_dir"].strip("/") != region_dir.strip("/"): - raise RuntimeError(f"fixture region_dir mismatch: {summary['region_dir']} != {region_dir}") - - -def generate_fixture(generator: Path | None, case_path: Path, fixture_dir: Path, *, dry_run: bool, reuse_fixture: bool, allow_large_fixture: bool, table_name: str | None = None, database: str | None = None, region_id: int | None = None, table_dir: str | None = None, region_dir: str | None = None) -> dict[str, Any]: - if generator is None: - return {"status": "skipped", "reason": "no fixture generator provided"} - summary_path = fixture_dir / "summary.json" - reuse_miss: str | None = None - if reuse_fixture and summary_path.exists(): - summary = json.loads(summary_path.read_text()) - try: - assert_fixture_summary(summary, region_id=region_id, table_dir=table_dir, region_dir=region_dir, table_name=table_name, database=database) - return {"status": "reused", "fixture_dir": str(fixture_dir), "summary": summary} - except RuntimeError as e: - reuse_miss = str(e) - shutil.rmtree(fixture_dir) - if fixture_dir.exists() and not reuse_fixture: - shutil.rmtree(fixture_dir) - fixture_dir.mkdir(parents=True, exist_ok=True) - cmd = [str(generator), "direct-sst", "--case", str(case_path), "--out-dir", str(fixture_dir)] - if table_name is not None: - cmd += ["--table", table_name] - if region_id is not None: - cmd += ["--region-id", str(region_id)] - if table_dir is not None: - cmd += ["--table-dir", table_dir] - if allow_large_fixture: - cmd.append("--allow-large") - if dry_run: - return {"status": "dry-run", "cmd": cmd} - result = run_command(cmd) - result["status"] = "ok" if result["returncode"] == 0 else "failed" - if result["returncode"] != 0: - raise RuntimeError(f"fixture generator failed: {result['stderr'][:2000]}") - summary = json.loads(summary_path.read_text()) - assert_fixture_summary(summary, region_id=region_id, table_dir=table_dir, region_dir=region_dir, table_name=table_name, database=database) - result["summary"] = summary - if reuse_miss is not None: - result["reuse_miss"] = reuse_miss - return result - - -def copy_stats() -> dict[str, int]: - return {"files": 0, "dirs": 0, "reflinked": 0, "hardlinked": 0, "copied": 0} - - -def merge_copy_stats(left: dict[str, int], right: dict[str, int]) -> None: - for key, value in right.items(): - left[key] = left.get(key, 0) + value - - -def reflink_file(src: Path, dst: Path) -> bool: - try: - dst.parent.mkdir(parents=True, exist_ok=True) - sfd = os.open(src, os.O_RDONLY) - try: - dfd = os.open(dst, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, src.stat().st_mode & 0o777) - try: - fcntl.ioctl(dfd, FICLONE, sfd) - shutil.copystat(src, dst, follow_symlinks=True) - return True - finally: - os.close(dfd) - finally: - os.close(sfd) - except OSError: - try: - dst.unlink() - except OSError: - pass - return False - - -def materialize_file(src: Path, dst: Path, *, allow_hardlink: bool) -> str: - if reflink_file(src, dst): - return "reflinked" - if allow_hardlink: - try: - dst.parent.mkdir(parents=True, exist_ok=True) - try: - dst.unlink() - except FileNotFoundError: - pass - os.link(src, dst) - return "hardlinked" - except OSError: - pass - dst.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(src, dst) - return "copied" - - -def copy_tree_contents(src: Path, dst: Path, *, allow_hardlinks: bool = True) -> dict[str, int]: - if not src.exists(): - raise FileNotFoundError(f"fixture path does not exist: {src}") - stats = copy_stats() - dst.mkdir(parents=True, exist_ok=True) - for child in src.iterdir(): - target = dst / child.name - if child.is_dir(): - stats["dirs"] += 1 - merge_copy_stats(stats, copy_tree_contents(child, target, allow_hardlinks=allow_hardlinks)) - else: - mode = materialize_file(child, target, allow_hardlink=allow_hardlinks) - stats["files"] += 1 - stats[mode] += 1 - return stats - - -def materialize_fixture(target: RunTarget, *, dry_run: bool, preserve_state: bool, expected_region_dir: str | None = None, fixture_dir: Path | None = None, reset_data: bool = True) -> dict[str, Any]: - fixture_dir = fixture_dir or target.fixture_dir - summary_path = fixture_dir / "summary.json" - materialize_root = target.datanode_data_dir if preserve_state else target.data_dir - if dry_run: - return {"status": "dry-run", "summary_path": str(summary_path), "data_dir": str(materialize_root)} - summary = json.loads(summary_path.read_text()) - object_store_dir = fixture_dir / "object-store" - manifest_dir = fixture_dir / "manifest" - region_dir = summary["region_dir"].strip("/") - target_region_dir = materialize_root / region_dir - data_root = materialize_root.resolve() - resolved_region_dir = target_region_dir.resolve(strict=False) - if data_root != resolved_region_dir and data_root not in resolved_region_dir.parents: - raise RuntimeError(f"unsafe fixture region_dir escapes data_dir: {region_dir}") - if expected_region_dir is not None and region_dir != expected_region_dir.strip("/"): - raise RuntimeError(f"fixture region_dir {region_dir} does not equal discovered {expected_region_dir}") - target_manifest_dir = target_region_dir / "manifest" - if not preserve_state and reset_data and materialize_root.exists(): - shutil.rmtree(materialize_root) - materialize_root.mkdir(parents=True, exist_ok=True) - if preserve_state and target_region_dir.exists(): - shutil.rmtree(target_region_dir) - object_stats: dict[str, int] - if preserve_state: - fixture_region_dir = object_store_dir / region_dir - object_stats = copy_tree_contents(fixture_region_dir, target_region_dir, allow_hardlinks=True) - else: - object_stats = copy_tree_contents(object_store_dir, materialize_root, allow_hardlinks=True) - if target_manifest_dir.exists(): - shutil.rmtree(target_manifest_dir) - manifest_stats = copy_tree_contents(manifest_dir, target_manifest_dir, allow_hardlinks=False) - return {"status": "ok", "summary_path": str(summary_path), "data_dir": str(materialize_root), "region_dir": region_dir, "manifest_dir": str(target_manifest_dir), "preserve_state": preserve_state, "copy_stats": {"object_store": object_stats, "manifest": manifest_stats}} - - -def response_text(body: Any) -> str: - return json.dumps(body, sort_keys=True) if not isinstance(body, str) else body - - -def validate_show_create(result: dict[str, Any], table: dict[str, Any]) -> list[str]: - text = response_text(result.get("response", {})).lower() - errors = [] - if table["name"].lower() not in text: - errors.append("SHOW CREATE output does not contain table name") - if table.get("validate_show_create_engine", True) and ("engine" not in text or "mito" not in text): - errors.append("SHOW CREATE output does not mention ENGINE=mito") - if "append_mode" in table and "append_mode" not in text: - errors.append("SHOW CREATE output does not mention append_mode") - if table.get("sst_format") and "sst_format" not in text: - errors.append("SHOW CREATE output does not mention sst_format") - return errors - - -def run_queries(target: RunTarget, case: dict[str, Any], tables: list[dict[str, Any]], http_timeout: float) -> dict[str, Any]: - table = tables[0] - db = table["database"] - queries = planned_queries(case) - if not queries: - queries = [{"name": "count_all", "kind": "sql", "query": f"SELECT count(*) FROM {sql_ident(table['name'])}", "warmup": 0, "iterations": 1}] - validations = [] - validation_errors = [] - for table in tables: - for sql in [f"SHOW CREATE TABLE {sql_ident(table['name'])}"]: - result = http_post_sql(target.http_port, sql, table["database"], http_timeout) - if not result["ok"]: - validation_errors.append({"sql": sql, "error": result.get("error"), "response": result.get("response")}) - else: - for error in validate_show_create(result, table): - validation_errors.append({"sql": sql, "error": error, "response": result.get("response")}) - validations.append(result) - if queries: - result = http_post_sql(target.http_port, queries[0]["query"], db, http_timeout) - if not result["ok"]: - validation_errors.append({"sql": queries[0]["query"], "error": result.get("error"), "response": result.get("response")}) - validations.append(result) - measurements = [] - for q in queries: - for _ in range(int(q.get("warmup", 0))): - warmup = http_post_sql(target.http_port, q["query"], db, http_timeout) - if not warmup["ok"]: - validation_errors.append({"sql": q["query"], "phase": "warmup", "error": warmup.get("error"), "response": warmup.get("response")}) - samples = [] - for _ in range(int(q.get("iterations", 1))): - result = http_post_sql(target.http_port, q["query"], db, http_timeout) - result["execution_time_ms"] = extract_execution_time(result.get("response")) - samples.append(result) - good_lats = [s["latency_ms"] for s in samples if s["ok"]] - measurements.append({"name": q.get("name"), "kind": q.get("kind"), "iterations": len(samples), "samples": samples, "latency_ms_median": statistics.median(good_lats) if good_lats else None, "latency_ms_p95": percentile(good_lats, 95) if good_lats else None, "status": "ok" if len(good_lats) == len(samples) else "failed"}) - return {"validation": validations, "validation_errors": validation_errors, "measurements": measurements, "status": "failed" if validation_errors or any(m["status"] == "failed" for m in measurements) else "ok"} - - -def extract_execution_time(body: Any) -> Any: - if isinstance(body, dict): - for key in ("execution_time_ms", "execution_time", "elapsed"): - if key in body: - return body[key] - for value in body.values(): - found = extract_execution_time(value) - if found is not None: - return found - if isinstance(body, list): - for value in body: - found = extract_execution_time(value) - if found is not None: - return found - return None - - -def percentile(values: list[float], pct: float) -> float: - if not values: - return 0.0 - ordered = sorted(values) - idx = min(len(ordered) - 1, max(0, round((pct / 100.0) * (len(ordered) - 1)))) - return ordered[idx] - - -def enforce_thresholds(case: dict[str, Any], base: dict[str, Any], candidate: dict[str, Any]) -> list[dict[str, Any]]: - results = [] - base_by_name = {m["name"]: m for m in base.get("measurements", [])} - for cm in candidate.get("measurements", []): - bm = base_by_name.get(cm["name"]) - qcfg = next((q for q in planned_queries(case) if q.get("name") == cm["name"]), {}) - th = qcfg.get("thresholds") or {} - max_reg = th.get("max_candidate_latency_regression_pct") - if max_reg is not None and not bm: - results.append({"query": cm["name"], "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing base measurement"}) - elif max_reg is not None and bm and bm.get("latency_ms_median") in (None, 0): - results.append({"query": cm["name"], "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "base median latency is missing or zero", "base_latency_ms_median": bm.get("latency_ms_median")}) - elif max_reg is not None and bm and cm.get("latency_ms_median") is None: - results.append({"query": cm["name"], "threshold": "max_candidate_latency_regression_pct", "status": "failed", "reason": "missing candidate measurement"}) - elif bm and max_reg is not None: - ratio = (cm["latency_ms_median"] - bm["latency_ms_median"]) / bm["latency_ms_median"] * 100.0 - results.append({"query": cm["name"], "threshold": "max_candidate_latency_regression_pct", "status": "passed" if ratio <= max_reg else "failed", "actual_pct": ratio, "limit_pct": max_reg}) - for key in th: - if key != "max_candidate_latency_regression_pct": - results.append({"query": cm["name"], "threshold": key, "status": "failed", "reason": "unsupported threshold"}) - return results - - -def storage_config(remote: dict[str, Any]) -> dict[str, Any] | None: - value = remote["storage"] - if not isinstance(value, dict): - return None - if value["inspect"] is False: - return None - return value - - -def run_storage_inspection(helper: Path | None, target: RunTarget, storage: dict[str, Any], *, dry_run: bool) -> dict[str, Any]: - if helper is None: - if not dry_run: - raise ValueError("--fixture-generator query_perf_fixture is required when storage inspection is enabled") - helper = Path("query_perf_fixture") - root = target.datanode_data_dir - if storage.get("root_suffix"): - root = root / str(storage["root_suffix"]) - cmd = [ - str(helper), - "inspect-footer", - "--root", str(root), - "--column", str(storage["column"]), - ] - if storage["include_metadata_files"]: - cmd.append("--include-metadata-files") - if dry_run: - return {"status": "dry-run", "cmd": cmd, "root": str(root)} - result = run_command(cmd) - result["status"] = "ok" if result["returncode"] == 0 else "failed" - if result["returncode"] != 0: - raise RuntimeError(f"storage inspector failed for {target.name}: {result['stderr'][:2000]}") - try: - result["summary"] = json.loads(result["stdout"]) - except json.JSONDecodeError: - result["summary_parse_error"] = result["stdout"] - result["status"] = "failed" - return result - - -def storage_summary(inspection: dict[str, Any]) -> dict[str, Any]: - summary = inspection.get("summary") - if isinstance(summary, dict): - nested = summary.get("summary") - if isinstance(nested, dict): - return nested - return {} - - -def enforce_storage_thresholds(storage: dict[str, Any] | None, base: dict[str, Any] | None, candidate: dict[str, Any] | None) -> list[dict[str, Any]]: - if not storage: - return [] - results: list[dict[str, Any]] = [] - target_summaries = [("base", storage_summary(base or {})), ("candidate", storage_summary(candidate or {}))] - cand = target_summaries[1][1] - base_summary = storage_summary(base or {}) - - def add_abs(name: str, field: str) -> None: - limit = storage.get(name) - if limit is None: - return - for target_name, summary in target_summaries: - actual = summary.get(field) - ok = actual is not None and float(actual) <= float(limit) - results.append({"target": target_name, "threshold": name, "status": "passed" if ok else "failed", "actual": actual, "limit": limit}) - - def add_min(name: str, field: str) -> None: - limit = storage[name] - if limit is None: - return - for target_name, summary in target_summaries: - actual = summary.get(field) - ok = actual is not None and int(actual) >= int(limit) - results.append({"target": target_name, "threshold": name, "status": "passed" if ok else "failed", "actual": actual, "limit": limit}) - - def add_cmp(name: str, field: str) -> None: - limit = storage.get(name) - if limit is None: - return - base_value = base_summary.get(field) - candidate_value = cand.get(field) - if base_value in (None, 0) or candidate_value is None: - results.append({"threshold": name, "status": "failed", "reason": "missing or zero base/candidate value", "base": base_value, "candidate": candidate_value}) - return - actual = (float(candidate_value) - float(base_value)) / float(base_value) * 100.0 - results.append({"threshold": name, "status": "passed" if actual <= float(limit) else "failed", "actual_pct": actual, "limit_pct": limit, "base": base_value, "candidate": candidate_value}) - - add_min("min_files", "file_count") - add_min("min_files_with_column", "files_with_column") - add_abs("max_total_file_size_bytes", "total_file_size") - add_abs("max_column_compressed_size_bytes", "column_compressed_size") - add_abs("max_column_uncompressed_size_bytes", "column_uncompressed_size") - for target_name, summary in target_summaries: - encodings = set(summary.get("unique_encodings") or []) - for required in storage["require_encodings"]: - results.append({"target": target_name, "threshold": "require_encodings", "encoding": required, "status": "passed" if required in encodings else "failed"}) - for forbidden in storage["forbid_encodings"]: - results.append({"target": target_name, "threshold": "forbid_encodings", "encoding": forbidden, "status": "passed" if forbidden not in encodings else "failed"}) - add_cmp("max_candidate_total_file_size_regression_pct", "total_file_size") - add_cmp("max_candidate_column_compressed_size_regression_pct", "column_compressed_size") - add_cmp("max_candidate_column_uncompressed_size_regression_pct", "column_uncompressed_size") - return results - - -def planned_storage_thresholds(storage: dict[str, Any] | None) -> list[dict[str, Any]]: - if not storage: - return [] - return list(storage["planned_thresholds"]) - - -REGION_DIR_RE = re.compile(r"^(\d+)_(\d{10})$") - - -def inspection_report(inspection: dict[str, Any] | None) -> dict[str, Any]: - if not inspection: - return {} - report = inspection.get("summary") - return report if isinstance(report, dict) else {} - - -def inspected_relative_path(target: RunTarget, inspection: dict[str, Any], relative_path: str) -> Path: - report = inspection_report(inspection) - root_text = report.get("root") or inspection.get("root") - if not root_text: - return Path(relative_path) - root = Path(root_text).resolve(strict=False) - data_home = target.datanode_data_dir.resolve(strict=False) - try: - root_suffix = root.relative_to(data_home) - except ValueError as e: - raise RuntimeError(f"storage inspection root {root} is not under datanode data home {data_home}") from e - return root_suffix / relative_path - - -def parse_sst_bench_target(target: RunTarget, inspection: dict[str, Any], file: dict[str, Any], *, dry_run: bool) -> dict[str, str] | None: - relative_path = str(file.get("relative_path") or "") - if dry_run and (not relative_path or "<" in relative_path): - return { - "relative_path": relative_path or "/_/data/.parquet", - "table_dir": "/", - "region_id": ":", - "path_type": "data", - "file_id": "", - } - if not relative_path: - return None - full_relative = inspected_relative_path(target, inspection, relative_path) - parts = full_relative.parts - if not parts or not parts[-1].endswith(".parquet"): - return None - region_index = next((idx for idx, part in enumerate(parts) if REGION_DIR_RE.match(part)), None) - if region_index is None or region_index == 0: - if dry_run: - return { - "relative_path": relative_path, - "table_dir": "/", - "region_id": ":", - "path_type": "data", - "file_id": Path(parts[-1]).stem, - } - return None - region_match = REGION_DIR_RE.match(parts[region_index]) - assert region_match is not None - file_index = len(parts) - 1 - path_type = "bare" - if region_index + 1 < file_index and parts[region_index + 1] in ("data", "metadata"): - path_type = parts[region_index + 1] - if path_type == "metadata": - return None - table_dir = "/".join(parts[:region_index]).rstrip("/") + "/" - return { - "relative_path": str(full_relative), - "table_dir": table_dir, - "region_id": f"{int(region_match.group(1))}:{int(region_match.group(2))}", - "path_type": path_type, - "file_id": Path(parts[-1]).stem, - } - - -def run_read_bench(bench_binary: Path, target: RunTarget, read_bench: dict[str, Any] | None, storage_inspection: dict[str, Any] | None, *, dry_run: bool) -> dict[str, Any]: - if not read_bench or read_bench["enabled"] is False: - return {"status": "skipped", "reason": "read_bench disabled"} - run_parquetbench = read_bench["parquetbench"] - run_scanbench = read_bench["scanbench"] - if not run_parquetbench and not run_scanbench: - return {"status": "skipped", "reason": "parquetbench and scanbench disabled"} - if storage_inspection is None: - raise ValueError("read_bench requires storage inspection") - bench_dir = target.work_dir / "read_bench" - config_toml = bench_dir / "bench.toml" - scan_json = bench_dir / "scan.json" - files = inspection_report(storage_inspection).get("files") or [] - ssts = [f for f in files if f.get("relative_path", "").endswith(".parquet") and f.get("columns")] - if dry_run and not ssts: - ssts = [{"relative_path": ".parquet"}] - if read_bench["max_files"] is not None: - ssts = ssts[: int(read_bench["max_files"])] - bench_targets = [parsed for f in ssts if (parsed := parse_sst_bench_target(target, storage_inspection, f, dry_run=dry_run)) is not None] - if not bench_targets: - return {"status": "failed", "reason": "no inspected data SST files available for read_bench"} - config_text = f'[storage]\ndata_home = "{target.datanode_data_dir}"\ntype = "File"\n\n[[region_engine]]\n[region_engine.mito]\n' - commands = [] - parquet_runs = [] - scan_runs = [] - iterations = str(int(read_bench["iterations"])) - if run_parquetbench: - for bench_target in bench_targets: - cmd = [str(bench_binary), "datanode", "parquetbench", "--config", str(config_toml), "--region-id", bench_target["region_id"], "--table-dir", bench_target["table_dir"], "--file-id", bench_target["file_id"], "--scan-config", str(scan_json), "--path-type", bench_target["path_type"], "--iterations", iterations, "--reader", str(read_bench["parquet_reader"])] - commands.append(cmd) - parquet_runs.append({**bench_target, "cmd": cmd, "status": "dry-run" if dry_run else "planned"}) - regions: dict[tuple[str, str, str], list[str]] = {} - if run_scanbench: - for bench_target in bench_targets: - regions.setdefault((bench_target["table_dir"], bench_target["region_id"], bench_target["path_type"]), []).append(bench_target["relative_path"]) - for (table_dir, region_id, path_type), region_files in regions.items(): - cmd = [str(bench_binary), "datanode", "scanbench", "--config", str(config_toml), "--region-id", region_id, "--table-dir", table_dir, "--scan-config", str(scan_json), "--path-type", path_type, "--scanner", str(read_bench["scan_scanner"]), "--parallelism", str(int(read_bench["parallelism"])), "--iterations", iterations] - commands.append(cmd) - scan_runs.append({"table_dir": table_dir, "region_id": region_id, "path_type": path_type, "files": region_files, "cmd": cmd, "status": "dry-run" if dry_run else "planned"}) - if dry_run: - return {"status": "dry-run", "config_path": str(config_toml), "scan_config_path": str(scan_json), "commands": commands, "parquetbench": parquet_runs, "scanbench": scan_runs} - bench_dir.mkdir(parents=True, exist_ok=True) - config_toml.write_text(config_text) - scan_json.write_text(json.dumps({"projection_names": read_bench["projection"]}, indent=2) + "\n") - avg_re = re.compile(r"Average duration[^0-9]*([0-9.]+)\s*ms", re.I) - for runs in (parquet_runs, scan_runs): - for run in runs: - result = run_command(run["cmd"]) - run.update(result) - run["status"] = "ok" if result["returncode"] == 0 else "failed" - m = avg_re.search(result.get("stdout", "")) - if m: - run["average_ms"] = float(m.group(1)) - def median(runs: list[dict[str, Any]]) -> float | None: - vals = [r["average_ms"] for r in runs if "average_ms" in r] - return statistics.median(vals) if vals else None - return {"status": "ok" if all(r.get("status") == "ok" for r in parquet_runs + scan_runs) else "failed", "config_path": str(config_toml), "scan_config_path": str(scan_json), "parquetbench": parquet_runs, "scanbench": scan_runs, "aggregate": {"parquetbench_median_average_ms": median(parquet_runs), "scanbench_median_average_ms": median(scan_runs)}} - - -def write_json(path: Path, payload: dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") - - -def output_report(report: dict[str, Any], output: Path | None) -> None: - if output is None: - print(json.dumps(report, indent=2, sort_keys=True)) - else: - write_json(output, report) - - -def parse_prometheus_metrics(text: str, names: set[str] = OTLP_TRACE_METRICS) -> dict[str, float]: - values: dict[str, float] = {} - for line in text.splitlines(): - match = PROMETHEUS_SAMPLE_RE.match(line.strip()) - if not match or match.group(1) not in names: - continue - value = float(match.group(2)) - if not math.isfinite(value): - raise ValueError(f"non-finite Prometheus sample for {match.group(1)}") - values[match.group(1)] = values.get(match.group(1), 0.0) + value - return values - - -def fetch_otlp_metrics(target: RunTarget, http_timeout: float) -> dict[str, Any]: - with urllib.request.urlopen(f"http://127.0.0.1:{target.http_port}/metrics", timeout=http_timeout) as response: - text = response.read().decode() - return {"captured_monotonic_seconds": time.monotonic(), "values": parse_prometheus_metrics(text)} - - -def metric_delta(after: dict[str, Any], before: dict[str, Any], name: str) -> float: - delta = float(after["values"].get(name, 0.0)) - float(before["values"].get(name, 0.0)) - if delta < 0: - raise RuntimeError(f"metric {name} decreased by {-delta}") - return delta - - -def otelgen_command(otelgen_bin: Path, target: RunTarget, load: dict[str, Any]) -> list[str]: - return [ - str(otelgen_bin), - "--protocol", "http", - "--otel-exporter-otlp-endpoint", f"127.0.0.1:{target.http_port}", - "--otel-exporter-otlp-url-path", "/v1/otlp/v1/traces", - "--header", f"x-greptime-pipeline-name={load['pipeline']}", - "--header", f"x-greptime-db-name={load['database']}", - "--header", f"x-greptime-trace-table-name={load['table']}", - "--log-level", "error", - "--insecure", - "--duration", str(int(load["duration_seconds"])), - "--rate", str(int(load["rate"])), - "traces", "multi", - "--workers", str(int(load["workers"])), - "--scenarios", str(load["workload"]), - "--exporter-shards", str(int(load["exporter_shards"])), - ] - - -def run_otelgen_load(otelgen_bin: Path | None, target: RunTarget, load: dict[str, Any], http_timeout: float, *, dry_run: bool) -> dict[str, Any]: - binary = otelgen_bin or Path("otelgen") - cmd = otelgen_command(binary, target, load) - if dry_run: - return {"status": "dry-run", "cmd": cmd} - - initial = fetch_otlp_metrics(target, http_timeout) - log_dir = target.work_dir / "otelgen" - log_dir.mkdir(parents=True, exist_ok=True) - stdout_path = log_dir / "stdout.log" - stderr_path = log_dir / "stderr.log" - duration = int(load["duration_seconds"]) - warmup = int(load["warmup_seconds"]) - timed_out = False - started = time.monotonic() - with stdout_path.open("wb") as stdout, stderr_path.open("wb") as stderr: - proc = subprocess.Popen(cmd, stdout=stdout, stderr=stderr) - try: - if warmup: - try: - proc.wait(timeout=warmup) - except subprocess.TimeoutExpired: - pass - warmed = fetch_otlp_metrics(target, http_timeout) - if proc.poll() is None: - try: - proc.wait(timeout=max(60, duration - warmup + 60)) - except subprocess.TimeoutExpired: - timed_out = True - proc.kill() - proc.wait(timeout=20) - final = fetch_otlp_metrics(target, http_timeout) - finally: - if proc.poll() is None: - proc.kill() - proc.wait(timeout=20) - elapsed = time.monotonic() - started - return { - "status": "ok" if proc.returncode == 0 and not timed_out and elapsed + 1 >= duration else "failed", - "cmd": cmd, - "returncode": proc.returncode, - "timed_out": timed_out, - "elapsed_seconds": elapsed, - "stdout_path": str(stdout_path), - "stderr_path": str(stderr_path), - "snapshots": {"initial": initial, "warmup": warmed, "final": final}, - } - - -def summarize_otlp_metrics(run: dict[str, Any]) -> dict[str, Any]: - initial = run["snapshots"]["initial"] - warmed = run["snapshots"]["warmup"] - final = run["snapshots"]["final"] - rows = "greptime_frontend_otlp_traces_rows" - failures = "greptime_frontend_otlp_traces_failure_count" - elapsed_sum = "greptime_servers_http_otlp_traces_elapsed_sum" - elapsed_count = "greptime_servers_http_otlp_traces_elapsed_count" - missing = sorted(name for name in (rows, elapsed_sum, elapsed_count) if name not in final["values"]) - accepted_spans = int(round(metric_delta(final, initial, rows))) - measurement_accepted_spans = int(round(metric_delta(final, warmed, rows))) - http_requests = int(round(metric_delta(final, warmed, elapsed_count))) - latency_seconds = metric_delta(final, warmed, elapsed_sum) - measurement_seconds = final["captured_monotonic_seconds"] - warmed["captured_monotonic_seconds"] - return { - "accepted_spans": accepted_spans, - "measurement_accepted_spans": measurement_accepted_spans, - "accepted_spans_per_second": measurement_accepted_spans / measurement_seconds if measurement_seconds > 0 else None, - "http_requests": http_requests, - "mean_http_latency_ms": latency_seconds / http_requests * 1000.0 if http_requests else None, - "failure_count": int(round(metric_delta(final, initial, failures))), - "measurement_seconds": measurement_seconds, - "missing_metrics": missing, - } - - -def planned_otlp_thresholds(load: dict[str, Any]) -> list[dict[str, Any]]: - thresholds = load["thresholds"] - return [ - {"threshold": "max_candidate_throughput_regression_pct", "status": "planned", "limit_pct": thresholds["max_candidate_throughput_regression_pct"]}, - {"threshold": "max_candidate_mean_latency_regression_pct", "status": "planned", "limit_pct": thresholds["max_candidate_mean_latency_regression_pct"]}, - {"target": "each", "threshold": "max_failure_count", "status": "planned", "limit": thresholds["max_failure_count"]}, - ] - - -def enforce_otlp_thresholds(load: dict[str, Any], base: dict[str, Any], candidate: dict[str, Any]) -> list[dict[str, Any]]: - thresholds = load["thresholds"] - results: list[dict[str, Any]] = [] - for target_name, metrics in (("base", base), ("candidate", candidate)): - failures = metrics.get("failure_count") - limit = int(thresholds["max_failure_count"]) - results.append({"target": target_name, "threshold": "max_failure_count", "status": "passed" if failures is not None and failures <= limit else "failed", "actual": failures, "limit": limit}) - - base_rate = base.get("accepted_spans_per_second") - candidate_rate = candidate.get("accepted_spans_per_second") - throughput_limit = float(thresholds["max_candidate_throughput_regression_pct"]) - if base_rate in (None, 0) or candidate_rate is None: - results.append({"threshold": "max_candidate_throughput_regression_pct", "status": "failed", "reason": "missing or zero throughput", "base": base_rate, "candidate": candidate_rate}) - else: - actual = (base_rate - candidate_rate) / base_rate * 100.0 - results.append({"threshold": "max_candidate_throughput_regression_pct", "status": "passed" if actual <= throughput_limit else "failed", "actual_pct": actual, "limit_pct": throughput_limit, "base": base_rate, "candidate": candidate_rate}) - - base_latency = base.get("mean_http_latency_ms") - candidate_latency = candidate.get("mean_http_latency_ms") - latency_limit = float(thresholds["max_candidate_mean_latency_regression_pct"]) - if base_latency in (None, 0) or candidate_latency is None: - results.append({"threshold": "max_candidate_mean_latency_regression_pct", "status": "failed", "reason": "missing or zero mean latency", "base": base_latency, "candidate": candidate_latency}) - else: - actual = (candidate_latency - base_latency) / base_latency * 100.0 - results.append({"threshold": "max_candidate_mean_latency_regression_pct", "status": "passed" if actual <= latency_limit else "failed", "actual_pct": actual, "limit_pct": latency_limit, "base": base_latency, "candidate": candidate_latency}) - return results - - -def require_fresh_work_dirs(targets: list[RunTarget], *, reuse_work_dir: bool, dry_run: bool, fixture_only: bool) -> None: - if dry_run or reuse_work_dir or fixture_only: - return - for target in targets: - if target.work_dir.exists() and any(target.work_dir.iterdir()): - raise RuntimeError(f"target work_dir exists and is non-empty: {target.work_dir}; pass --reuse-work-dir to override") - - -def write_frontend_prom_config(target: RunTarget, remote: dict[str, Any]) -> Path: - prom = remote["prom_store"] - path = target.work_dir / "frontend-prom-store.toml" - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - "[prom_store]\n" - "enable = true\n" - "with_metric_engine = true\n" - f"pending_rows_flush_interval = {json.dumps(str(prom['pending_rows_flush_interval']))}\n" - f"max_batch_rows = {int(prom['max_batch_rows'])}\n" - f"max_concurrent_flushes = {int(prom['max_concurrent_flushes'])}\n" - f"worker_channel_capacity = {int(prom['worker_channel_capacity'])}\n" - f"max_inflight_requests = {int(prom['max_inflight_requests'])}\n" - ) - return path - - -def run_remote_write(generator: Path | None, target: RunTarget, remote: dict[str, Any], *, dry_run: bool) -> dict[str, Any]: - if generator is None: - if not dry_run: - raise ValueError("--fixture-generator query_perf_fixture is required for prom_remote_write_then_query") - generator = Path("query_perf_fixture") - metric = remote["metric"] - physical_table = remote["physical_table"] - cmd = [ - str(generator), - "prom-remote-write", - "--endpoint", f"http://127.0.0.1:{target.http_port}/v1/prometheus/write", - "--database", remote["database"], - "--metric", metric, - "--physical-table", physical_table, - "--series-count", str(int(remote["series_count"])), - "--samples-per-series", str(int(remote["samples_per_series"])), - "--start-unix-millis", str(int(remote["start_unix_millis"])), - "--step-millis", str(int(remote["step_millis"])), - "--chunk-series-count", str(int(remote["chunk_series_count"])), - "--timeout-seconds", str(int(remote["timeout_seconds"])), - ] - value = remote["value"] - cmd.extend(["--value-pattern", str(value["pattern"])]) - cmd.extend(["--value-base", str(value["base"])]) - cmd.extend(["--value-step", str(value["step"])]) - cmd.extend(["--value-cardinality", str(int(value["cardinality"]))]) - cmd.extend(["--value-seed", str(int(value["seed"]))]) - cmd.extend(["--value-run-length", str(int(value["run_length"]))]) - cmd.extend(["--value-stall-every", str(int(value["stall_every"]))]) - cmd.extend(["--value-stall-length", str(int(value["stall_length"]))]) - cmd.extend(["--value-mixed-every", str(int(value["mixed_every"]))]) - if "sample_offset" in remote: - cmd.extend(["--value-sample-offset", str(int(remote["sample_offset"]))]) - if "total_samples_per_series" in remote: - cmd.extend(["--value-total-samples-per-series", str(int(remote["total_samples_per_series"]))]) - if dry_run: - return {"status": "dry-run", "cmd": cmd} - result = run_command(cmd) - result["status"] = "ok" if result["returncode"] == 0 else "failed" - if result["returncode"] != 0: - raise RuntimeError(f"remote-write generator failed for {target.name}: {result['stderr'][:2000]}") - try: - result["summary"] = json.loads(result["stdout"]) - except json.JSONDecodeError: - result["summary_parse_error"] = result["stdout"] - return result - - -def summarize_remote_write_chunks(chunks: list[dict[str, Any]]) -> dict[str, Any]: - summary = {"rows": 0, "samples_written": 0, "batches": 0, "elapsed_seconds": 0.0} - for chunk in chunks: - chunk_summary = chunk.get("summary") or {} - summary["rows"] += int(chunk_summary.get("rows", 0)) - summary["samples_written"] += int(chunk_summary.get("samples_written", chunk_summary.get("rows", 0))) - summary["batches"] += int(chunk_summary.get("batches", 0)) - summary["elapsed_seconds"] += float(chunk_summary.get("elapsed_seconds", chunk.get("elapsed_seconds", 0.0))) - return summary - - -def flush_remote_physical_table(target: RunTarget, db: str, physical_table: str, args: argparse.Namespace, *, dry_run: bool, reason: str, chunk_index: int | None = None) -> dict[str, Any]: - if dry_run: - return {"status": "dry-run", "physical_table": physical_table, "reason": reason, "chunk_index": chunk_index} - result = http_post_sql(target.http_port, f"ADMIN FLUSH_TABLE({sql_string(physical_table)})", db, args.http_timeout) - result["physical_table"] = physical_table - result["reason"] = reason - result["chunk_index"] = chunk_index - return result - - -def run_remote_write_ingestion(generator: Path | None, target: RunTarget, remote: dict[str, Any], args: argparse.Namespace, *, dry_run: bool) -> tuple[dict[str, Any], list[dict[str, Any]]]: - sample_chunk_size = remote["sample_chunk_size"] - db = remote["database"] - physical_table = remote["physical_table"] - if sample_chunk_size is None: - rw = run_remote_write(generator, target, remote, dry_run=dry_run) - flush = flush_remote_physical_table(target, db, physical_table, args, dry_run=dry_run, reason="final") - return rw, [flush] - - total_samples = int(remote["samples_per_series"]) - chunk_samples = int(sample_chunk_size) - if chunk_samples <= 0: - raise ValueError("scenario.remote_write.sample_chunk_size must be positive") - flush_every = int(remote["flush_every_sample_chunks"]) - if flush_every <= 0: - raise ValueError("scenario.remote_write.flush_every_sample_chunks must be positive") - start = int(remote["start_unix_millis"]) - step = int(remote["step_millis"]) - chunks: list[dict[str, Any]] = [] - flushes: list[dict[str, Any]] = [] - chunk_index = 0 - last_flushed_chunk = 0 - for offset in range(0, total_samples, chunk_samples): - current = min(chunk_samples, total_samples - offset) - chunk_index += 1 - chunk_remote = dict(remote) - chunk_remote["samples_per_series"] = current - chunk_remote["start_unix_millis"] = start + offset * step - chunk_remote["sample_offset"] = offset - chunk_remote["total_samples_per_series"] = total_samples - result = run_remote_write(generator, target, chunk_remote, dry_run=dry_run) - result["sample_offset"] = offset - result["samples_per_series"] = current - result["chunk_index"] = chunk_index - chunks.append(result) - if chunk_index % flush_every == 0: - flushes.append(flush_remote_physical_table(target, db, physical_table, args, dry_run=dry_run, reason="periodic", chunk_index=chunk_index)) - last_flushed_chunk = chunk_index - if last_flushed_chunk != chunk_index: - flushes.append(flush_remote_physical_table(target, db, physical_table, args, dry_run=dry_run, reason="final", chunk_index=chunk_index)) - aggregate = summarize_remote_write_chunks(chunks) - if dry_run: - series_count = int(remote["series_count"]) - chunk_series_count = int(remote["chunk_series_count"]) - aggregate["rows"] = series_count * total_samples - aggregate["samples_written"] = aggregate["rows"] - aggregate["batches"] = chunk_index * ((series_count + chunk_series_count - 1) // chunk_series_count) - return { - "status": "dry-run" if dry_run else ("ok" if all(chunk.get("status") == "ok" for chunk in chunks) else "failed"), - "mode": "sample-chunked", - "sample_chunk_size": chunk_samples, - "flush_every_sample_chunks": flush_every, - "chunks": chunks, - "aggregate": aggregate, - }, flushes - - -def expected_remote_write_rows(remote: dict[str, Any]) -> int: - return int(remote["series_count"]) * int(remote["samples_per_series"]) - - -def extract_count_value(result: dict[str, Any]) -> int | None: - body = result.get("response") - if not isinstance(body, dict): - return None - data = body.get("data") - if not isinstance(data, list) or not data: - return None - row = data[0] - if not isinstance(row, dict): - return None - for key, value in row.items(): - if key.lower() == "count(*)" or key.lower().startswith("count("): - try: - return int(value) - except (TypeError, ValueError): - return None - return None - - -def poll_expected_count(target: RunTarget, table_name: str, db: str, expected_rows: int, timeout_s: float, http_timeout: float) -> dict[str, Any]: - sql = f"SELECT count(*) FROM {sql_ident(table_name)}" - deadline = time.monotonic() + timeout_s - attempts = 0 - last: dict[str, Any] | None = None - while True: - attempts += 1 - result = http_post_sql(target.http_port, sql, db, http_timeout) - observed_rows = extract_count_value(result) - result["expected_rows"] = expected_rows - result["observed_rows"] = observed_rows - result["attempts"] = attempts - result["row_count_ok"] = result.get("ok") and observed_rows == expected_rows - if result["row_count_ok"]: - return result - last = result - if time.monotonic() >= deadline: - break - time.sleep(0.5) - assert last is not None - last["ok"] = False - last["row_count_ok"] = False - last["error"] = f"expected {expected_rows} rows but observed {last.get('observed_rows')} after {attempts} attempts" - return last - - -def run_remote_write_scenario(args: argparse.Namespace, case: dict[str, Any], case_path: Path, targets: list[RunTarget], report: dict[str, Any]) -> None: - remote = scenario(case)["remote_write"] - tables = case_tables(case) - if args.fixture_only: - raise ValueError("--fixture-only is not supported for prom_remote_write_then_query; use --dry-run for planning") - helper = args.fixture_generator or args.remote_write_generator - if helper is not None: - require_binary(helper, "query_perf_fixture", dry_run=args.dry_run) - elif not args.dry_run: - raise ValueError("--fixture-generator is required for prom_remote_write_then_query") - storage = storage_config(remote) - if storage and helper is None and not args.dry_run: - raise ValueError("--fixture-generator is required when scenario.remote_write.storage.inspect is enabled") - clusters: list[DistributedCluster] = [] - target_results = [] - storage_results = [] - try: - for target in targets: - target.work_dir.mkdir(parents=True, exist_ok=True) - config_path = write_frontend_prom_config(target, remote) - cluster = DistributedCluster(target) - clusters.append(cluster) - if not args.dry_run: - cluster.start_all(config_path) - db = remote["database"] - create_database = {"status": "dry-run", "database": db} - if not args.dry_run: - create_database = http_post_sql(target.http_port, f"CREATE DATABASE IF NOT EXISTS {sql_ident(db)}", "public", args.http_timeout) - rw, flushes = run_remote_write_ingestion(helper, target, remote, args, dry_run=args.dry_run) - visibility = {"status": "dry-run"} - query_result = {"validation": [], "validation_errors": [], "measurements": [], "status": "planned"} - if not args.dry_run: - visibility = poll_expected_count(target, tables[0]["name"], db, expected_remote_write_rows(remote), float(remote["visibility_timeout_seconds"]), args.http_timeout) - query_result = run_queries(target, case, tables, args.http_timeout) - cluster.stop_component("datanode") - storage_inspection = None - if storage: - storage_inspection = run_storage_inspection(helper or args.storage_inspector, target, storage, dry_run=args.dry_run) - storage_results.append(storage_inspection) - read_bench_result = run_read_bench(args.candidate_bin, target, remote["read_bench"], storage_inspection, dry_run=args.dry_run) - tr = {"name": target.name, "binary": str(target.binary), "work_dir": str(target.work_dir), "components": cluster.component_report(), "frontend_config": str(config_path), "create_database": create_database, "remote_write": rw, "flushes": flushes, "flush": flushes[-1] if flushes else None, "visibility": visibility, **query_result} - if storage_inspection is not None: - tr["storage_inspection"] = storage_inspection - tr["read_bench"] = read_bench_result - flushes_ok = all(flush.get("ok") for flush in flushes) - storage_ok = storage_inspection is None or args.dry_run or storage_inspection.get("status") == "ok" - read_bench_ok = args.dry_run or read_bench_result.get("status") in ("ok", "skipped") - remote_checks_ok = args.dry_run or (create_database.get("ok") and flushes_ok and visibility.get("ok") and visibility.get("row_count_ok") and storage_ok and read_bench_ok) - if not remote_checks_ok: - tr.setdefault("validation_errors", []).append({"phase": "remote_write_visibility_storage", "create_database_ok": create_database.get("ok"), "flushes_ok": flushes_ok, "visibility_ok": visibility.get("ok"), "row_count_ok": visibility.get("row_count_ok"), "storage_ok": storage_ok, "read_bench_ok": read_bench_ok, "create_database": create_database, "flushes": flushes, "visibility": visibility, "read_bench": read_bench_result}) - tr["status"] = "planned" if args.dry_run else ("measured" if remote_checks_ok and query_result["status"] == "ok" else "failed") - write_json(target.report_path, tr) - report["targets"].append(tr) - target_results.append(query_result) - if not args.dry_run: - report["thresholds"] = enforce_thresholds(case, target_results[0], target_results[1]) + enforce_storage_thresholds(storage, storage_results[0] if storage_results else None, storage_results[1] if len(storage_results) > 1 else None) - elif storage: - report["thresholds"] = planned_storage_thresholds(storage) - report["status"] = "planned" if args.dry_run else ("failed" if any(t["status"] == "failed" for t in report["thresholds"]) or any(t.get("status") == "failed" for t in report["targets"]) else "ok") - finally: - for cluster in reversed(clusters): - cluster.stop_all() - - -def run_otlp_trace_load_scenario(args: argparse.Namespace, case: dict[str, Any], targets: list[RunTarget], report: dict[str, Any]) -> None: - load = scenario(case)["load"] - if args.fixture_only: - raise ValueError("--fixture-only is not supported for otlp_trace_load; use --dry-run for planning") - if args.otelgen_bin is not None: - require_binary(args.otelgen_bin, "otelgen", dry_run=args.dry_run) - elif not args.dry_run: - raise ValueError("--otelgen-bin is required for otlp_trace_load") - - clusters: list[DistributedCluster] = [] - metrics_results: list[dict[str, Any]] = [] - try: - for target in targets: - target.work_dir.mkdir(parents=True, exist_ok=True) - cluster = DistributedCluster(target) - clusters.append(cluster) - if not args.dry_run: - cluster.start_all() - create_database: dict[str, Any] = {"status": "dry-run", "database": load["database"]} - if not args.dry_run: - create_database = http_post_sql(target.http_port, f"CREATE DATABASE IF NOT EXISTS {sql_ident(load['database'])}", "public", args.http_timeout) - otelgen = run_otelgen_load(args.otelgen_bin, target, load, args.http_timeout, dry_run=args.dry_run) - metrics: dict[str, Any] = {"status": "dry-run"} - flush: dict[str, Any] = {"status": "dry-run", "table": load["table"]} - visibility: dict[str, Any] = {"status": "dry-run"} - if not args.dry_run: - metrics = summarize_otlp_metrics(otelgen) - flush = http_post_sql(target.http_port, f"ADMIN FLUSH_TABLE({sql_string(load['table'])})", load["database"], args.http_timeout) - visibility = poll_expected_count(target, load["table"], load["database"], int(metrics["accepted_spans"]), float(load["visibility_timeout_seconds"]), args.http_timeout) - tr = { - "name": target.name, - "binary": str(target.binary), - "work_dir": str(target.work_dir), - "components": cluster.component_report(), - "create_database": create_database, - "otelgen": otelgen, - "metrics": metrics, - "flush": flush, - "visibility": visibility, - } - checks_ok = args.dry_run or ( - create_database.get("ok") - and otelgen.get("status") == "ok" - and not metrics.get("missing_metrics") - and metrics.get("accepted_spans", 0) > 0 - and metrics.get("http_requests", 0) > 0 - and flush.get("ok") - and visibility.get("ok") - and visibility.get("row_count_ok") - ) - tr["status"] = "planned" if args.dry_run else ("measured" if checks_ok else "failed") - write_json(target.report_path, tr) - report["targets"].append(tr) - metrics_results.append(metrics) - cluster.stop_all() - report["thresholds"] = planned_otlp_thresholds(load) if args.dry_run else enforce_otlp_thresholds(load, metrics_results[0], metrics_results[1]) - report["status"] = "planned" if args.dry_run else ("failed" if any(t["status"] == "failed" for t in report["thresholds"]) or any(t["status"] == "failed" for t in report["targets"]) else "ok") - finally: - for cluster in reversed(clusters): - cluster.stop_all() - - -def main() -> int: - args = parse_args() - case_path = args.case.resolve() - if args.fixture_generator is not None: - require_binary(args.fixture_generator, "fixture generator", dry_run=False) - case = load_normalized_case(case_path, args.fixture_generator) - scenario_config = scenario(case) - scenario_kind = scenario_config.get("kind") - tables = case_tables(case) - work_root = args.work_dir.resolve() - work_root.mkdir(parents=True, exist_ok=True) - require_binary(args.base_bin, "base", dry_run=args.dry_run or args.fixture_only) - require_binary(args.candidate_bin, "candidate", dry_run=args.dry_run or args.fixture_only) - if scenario_kind == "direct_readable_sst" and not args.dry_run and args.fixture_generator is None: - raise ValueError("--fixture-generator is required unless --dry-run is set") - ports = allocate_ports(16) - targets = [make_target("base", args.base_bin.resolve(), work_root, ports[:8]), make_target("candidate", args.candidate_bin.resolve(), work_root, ports[8:])] - require_fresh_work_dirs(targets, reuse_work_dir=args.reuse_work_dir, dry_run=args.dry_run, fixture_only=args.fixture_only) - fixture_dir = fixture_root(work_root, case_path, case, args.fixture_cache_dir) - reuse_fixture = args.reuse_fixture or args.fixture_cache_dir is not None - report: dict[str, Any] = {"case_path": str(case_path), "case": case.get("case", {}), "scenario": scenario_config, "queries": planned_queries(case), "dry_run": args.dry_run, "fixture_only": args.fixture_only, "query_mode": "fixture-only" if args.fixture_only else "distributed", "reuse_work_dir": args.reuse_work_dir, "reuse_fixture": reuse_fixture, "fixture_cache_dir": str(args.fixture_cache_dir.resolve()) if args.fixture_cache_dir is not None else None, "fixture_dir": str(fixture_dir), "http_timeout": args.http_timeout, "targets": [], "thresholds": [], "status": "planned" if args.dry_run else "running"} - - if scenario_kind == "otlp_trace_load": - try: - run_otlp_trace_load_scenario(args, case, targets, report) - except Exception as e: # noqa: BLE001 - write machine-readable failure report - report["status"] = "failed" - report["error"] = repr(e) - write_json(work_root / "query-regression-report.json", report) - output_report(report, args.output) - return 1 if report["status"] == "failed" else 0 - - if scenario_kind == "prom_remote_write_then_query": - try: - run_remote_write_scenario(args, case, case_path, targets, report) - except Exception as e: # noqa: BLE001 - write machine-readable failure report - report["status"] = "failed" - report["error"] = repr(e) - write_json(work_root / "query-regression-report.json", report) - output_report(report, args.output) - return 1 if report["status"] == "failed" else 0 - - if args.fixture_only or args.dry_run: - generations = [] - for table_idx, table in enumerate(tables): - per_table_fixture_dir = table_fixture_dir(fixture_dir, tables, table, table_idx) - generations.append(generate_fixture(args.fixture_generator, case_path, per_table_fixture_dir, dry_run=args.dry_run, reuse_fixture=reuse_fixture, allow_large_fixture=args.allow_large_fixture, table_name=table["name"] if len(tables) > 1 else None, database=table["database"] if len(tables) > 1 else None)) - report["fixture_generation"] = generations[0] if len(generations) == 1 else generations - for target in targets: - target.work_dir.mkdir(parents=True, exist_ok=True) - mats = [] - for table_idx, table in enumerate(tables): - per_table_fixture_dir = table_fixture_dir(fixture_dir, tables, table, table_idx) - mats.append(materialize_fixture(target, dry_run=args.dry_run, preserve_state=False, fixture_dir=per_table_fixture_dir, reset_data=not mats)) - tr = {"name": target.name, "binary": str(target.binary), "work_dir": str(target.work_dir), "data_dir": str(target.data_dir), "fixture_dir": str(fixture_dir), "fixture_materialization": mats[0] if len(mats) == 1 else mats, "measurements": [], "status": "planned" if args.dry_run else "fixture-ready"} - write_json(target.report_path, tr) - report["targets"].append(tr) - report["status"] = "planned" if args.dry_run else "fixture-ready" - write_json(work_root / "query-regression-report.json", report) - output_report(report, args.output) - return 0 - - clusters: list[DistributedCluster] = [] - try: - discovered = [] - for target in targets: - target.work_dir.mkdir(parents=True, exist_ok=True) - cluster = DistributedCluster(target) - clusters.append(cluster) - cluster.start_all() - create_results = [] - metas = [] - for table in tables: - create_result = http_post_sql(target.http_port, create_table_sql(table), table["database"], args.http_timeout) - if not create_result["ok"]: - raise RuntimeError(f"CREATE TABLE {table['name']} failed for {target.name}: {create_result}") - create_results.append({"table": table["name"], "result": create_result}) - metas.append({"table": table["name"], **discover_region_via_frontend(target, table, args.http_timeout)}) - cluster.stop_component("datanode") - cluster._ensure_metasrv_alive() - discovered.append(metas) - report["targets"].append({"name": target.name, "binary": str(target.binary), "work_dir": str(target.work_dir), "data_dir": str(target.data_dir), "datanode_data_home": str(target.datanode_data_dir), "components": cluster.component_report(), "create_table": create_results[0]["result"] if len(create_results) == 1 else create_results, "discovered": metas[0] if len(metas) == 1 else metas, "discovered_tables": metas}) - - if len(discovered[0]) != len(discovered[1]) or any(a["table"] != b["table"] or a["region_id"] != b["region_id"] or a["table_dir"] != b["table_dir"] for a, b in zip(discovered[0], discovered[1])): - raise RuntimeError(f"base/candidate metadata mismatch: {discovered}") - generations = [] - for table_idx, meta in enumerate(discovered[0]): - per_table_fixture_dir = table_fixture_dir(fixture_dir, tables, tables[table_idx], table_idx) - generations.append(generate_fixture(args.fixture_generator, case_path, per_table_fixture_dir, dry_run=False, reuse_fixture=reuse_fixture, allow_large_fixture=args.allow_large_fixture, table_name=meta["table"] if len(tables) > 1 else None, database=meta["schema"] if len(tables) > 1 else None, region_id=meta["region_id"], table_dir=meta["table_dir"], region_dir=meta["region_dir"])) - report["fixture_generation"] = generations[0] if len(generations) == 1 else generations - - target_results = [] - for idx, target in enumerate(targets): - cluster = clusters[idx] - materialize = [] - for table_idx, meta in enumerate(discovered[idx]): - per_table_fixture_dir = table_fixture_dir(fixture_dir, tables, tables[table_idx], table_idx) - materialize.append(materialize_fixture(target, dry_run=False, preserve_state=True, expected_region_dir=meta["region_dir"], fixture_dir=per_table_fixture_dir)) - cluster.start_datanode() - query_result = run_queries(target, case, tables, args.http_timeout) - if query_result["status"] == "failed": - cluster.restart_frontend() - retry_result = run_queries(target, case, tables, args.http_timeout) - query_result["frontend_restart_retry"] = retry_result - if retry_result["status"] == "ok": - query_result = retry_result - report["targets"][idx]["fixture_dir"] = str(fixture_dir) - report["targets"][idx]["fixture_materialization"] = materialize[0] if len(materialize) == 1 else materialize - report["targets"][idx].update(query_result) - report["targets"][idx]["status"] = "measured" if query_result["status"] == "ok" else "failed" - write_json(target.report_path, report["targets"][idx]) - target_results.append(query_result) - - report["thresholds"] = enforce_thresholds(case, target_results[0], target_results[1]) - report["status"] = "failed" if any(t["status"] == "failed" for t in report["thresholds"]) or any(r["status"] == "failed" for r in target_results) else "ok" - except Exception as e: # noqa: BLE001 - write machine-readable failure report - report["status"] = "failed" - report["error"] = repr(e) - finally: - for cluster in reversed(clusters): - cluster.stop_all() - write_json(work_root / "query-regression-report.json", report) - output_report(report, args.output) - return 1 if report["status"] == "failed" else 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/perf/test_aliyun_ecs_runner_scripts.py b/tests/perf/test_aliyun_ecs_runner_scripts.py new file mode 100644 index 0000000000..581fc88abd --- /dev/null +++ b/tests/perf/test_aliyun_ecs_runner_scripts.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Coverage for the pure parts of the Aliyun ECS runner provision/teardown scripts.""" + +import base64 +import importlib.util +import sys +import unittest +from datetime import datetime, timedelta, timezone +from pathlib import Path + + +SCRIPTS_DIR = Path(__file__).parents[2] / ".github/scripts" + + +def load_module(name: str, filename: str): + spec = importlib.util.spec_from_file_location(name, SCRIPTS_DIR / filename) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +provision = load_module("aliyun_ecs_runner_provision_under_test", "aliyun-ecs-runner-provision.py") +teardown = load_module("aliyun_ecs_runner_teardown_under_test", "aliyun-ecs-runner-teardown.py") + + +class ProvisionNamingTest(unittest.TestCase): + def test_runner_name_and_label_derive_from_run_id(self) -> None: + self.assertEqual(provision.runner_name_for_run("12345"), "qreg-ecs-12345") + self.assertEqual(provision.runner_label_for_run("12345"), "query-regression-ecs-12345") + + +class ProvisionUserDataTest(unittest.TestCase): + def render(self) -> str: + return provision.render_user_data( + runner_name="qreg-ecs-12345", + runner_label="query-regression-ecs-12345", + runner_token="TOKEN", + repo="GreptimeTeam/greptimedb", + ) + + def test_user_data_creates_cache_paths_on_the_system_disk(self) -> None: + script = self.render() + self.assertNotIn("DISK_SERIAL", script) + self.assertNotIn("mount --bind", script) + self.assertNotIn("mkfs.ext4", script) + for destination in provision.CACHE_PATHS: + self.assertIn(f'"{destination}"', script) + + def test_user_data_wires_runner_registration(self) -> None: + script = self.render() + self.assertIn("RUNNER_NAME=qreg-ecs-12345", script) + self.assertIn("RUNNER_LABELS=query-regression-ecs-12345", script) + self.assertIn("RUNNER_TOKEN=TOKEN", script) + self.assertIn("REPO_URL=https://github.com/GreptimeTeam/greptimedb", script) + self.assertIn("PATH=/opt/cargo/bin:", script) + self.assertIn("systemctl restart --no-block ephemeral-github-runner.service", script) + + def test_encode_user_data_round_trips(self) -> None: + script = self.render() + self.assertEqual( + base64.b64decode(provision.encode_user_data(script)).decode("utf-8"), script + ) + + def test_user_data_enables_swap_and_masks_oomd(self) -> None: + script = self.render() + self.assertIn("systemctl mask systemd-oomd.socket systemd-oomd.service", script) + self.assertIn("systemctl mask unattended-upgrades.service apt-daily.timer apt-daily-upgrade.timer", script) + self.assertIn('APT::Periodic::Unattended-Upgrade "0"', script) + self.assertIn(f'fallocate --length {provision.SWAP_SIZE_GIB}G "{provision.SWAP_FILE}"', script) + self.assertIn(f'swapon "{provision.SWAP_FILE}"', script) + self.assertIn("sysctl --write vm.swappiness=10", script) + self.assertIn("OOMPolicy=continue", script) + self.assertNotIn("OOMScoreAdjust", script) + self.assertLess(script.index("swapon"), script.index("systemctl restart --no-block ephemeral-github-runner.service")) + + +class TeardownExpiryTest(unittest.TestCase): + NOW = datetime(2026, 8, 17, 6, 0, tzinfo=timezone.utc) + TTL = timedelta(hours=4) + + def test_parse_creation_time_formats(self) -> None: + self.assertEqual( + teardown.parse_creation_time("2026-08-17T01:02:03Z"), + datetime(2026, 8, 17, 1, 2, 3, tzinfo=timezone.utc), + ) + self.assertEqual( + teardown.parse_creation_time("2026-08-17T01:02Z"), + datetime(2026, 8, 17, 1, 2, tzinfo=timezone.utc), + ) + with self.assertRaises(ValueError): + teardown.parse_creation_time("not-a-time") + + def test_expired_instance_names_selects_only_old_instances(self) -> None: + instances = [ + ("i-old", "qreg-ecs-1", "2026-08-17T01:00Z"), # 5h old: expired + ("i-edge", "qreg-ecs-2", "2026-08-17T02:00Z"), # exactly TTL: expired + ("i-fresh", "qreg-ecs-3", "2026-08-17T05:30Z"), # 30m old: kept + ] + self.assertEqual( + teardown.expired_instance_names(instances, self.NOW, self.TTL), + [("i-old", "qreg-ecs-1"), ("i-edge", "qreg-ecs-2")], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/perf/test_query_regression_nightly_refs.py b/tests/perf/test_query_regression_nightly_refs.py new file mode 100644 index 0000000000..74b168bb46 --- /dev/null +++ b/tests/perf/test_query_regression_nightly_refs.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +# Copyright 2023 Greptime Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Unit tests for consecutive Nightly Build SHA selection.""" + +import importlib.util +import sys +import unittest +from pathlib import Path + + +SCRIPTS_DIR = Path(__file__).parents[2] / ".github/scripts" + + +def load_module(): + spec = importlib.util.spec_from_file_location( + "query_regression_nightly_refs_under_test", + SCRIPTS_DIR / "query-regression-nightly-refs.py", + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +refs = load_module() + + +def run(*, run_id: int, sha: str, branch: str = "main", created_at: str = "2026-08-24T00:00:00Z"): + return refs.WorkflowRun( + id=run_id, + head_sha=sha, + head_branch=branch, + html_url=f"https://github.com/example/run/{run_id}", + created_at=created_at, + conclusion="success", + event="schedule", + ) + + +class SelectNightlyRefsTest(unittest.TestCase): + def test_picks_newest_and_previous_on_same_branch(self) -> None: + pair = refs.select_base_and_candidate( + [ + run(run_id=3, sha="ccc", created_at="2026-08-22T00:00:00Z"), + run(run_id=2, sha="bbb", created_at="2026-08-21T00:00:00Z"), + run(run_id=1, sha="aaa", created_at="2026-08-20T00:00:00Z"), + ] + ) + self.assertFalse(pair.skip) + assert pair.candidate is not None and pair.base is not None + self.assertEqual(pair.candidate.head_sha, "ccc") + self.assertEqual(pair.base.head_sha, "bbb") + + def test_candidate_run_id_uses_that_run_and_the_next_older(self) -> None: + pair = refs.select_base_and_candidate( + [ + run(run_id=3, sha="ccc", created_at="2026-08-22T00:00:00Z"), + run(run_id=2, sha="bbb", created_at="2026-08-21T00:00:00Z"), + run(run_id=1, sha="aaa", created_at="2026-08-20T00:00:00Z"), + ], + candidate_run_id=2, + ) + self.assertFalse(pair.skip) + assert pair.candidate is not None and pair.base is not None + self.assertEqual(pair.candidate.id, 2) + self.assertEqual(pair.base.id, 1) + + def test_skips_when_consecutive_nightlies_share_a_sha(self) -> None: + pair = refs.select_base_and_candidate( + [ + run(run_id=2, sha="same", created_at="2026-08-22T00:00:00Z"), + run(run_id=1, sha="same", created_at="2026-08-21T00:00:00Z"), + ] + ) + self.assertTrue(pair.skip) + self.assertIn("matches candidate", pair.reason) + + def test_skips_other_branches_when_picking_previous(self) -> None: + pair = refs.select_base_and_candidate( + [ + run(run_id=3, sha="ccc", branch="main", created_at="2026-08-22T00:00:00Z"), + run(run_id=2, sha="other", branch="feat", created_at="2026-08-21T12:00:00Z"), + run(run_id=1, sha="aaa", branch="main", created_at="2026-08-21T00:00:00Z"), + ] + ) + self.assertFalse(pair.skip) + assert pair.base is not None + self.assertEqual(pair.base.head_sha, "aaa") + + def test_skips_when_only_one_nightly_exists(self) -> None: + pair = refs.select_base_and_candidate( + [run(run_id=1, sha="aaa")], + ) + self.assertTrue(pair.skip) + self.assertIn("no previous", pair.reason) + + def test_explicit_refs_bypass_github(self) -> None: + pair = refs.override_pair("base-sha", "cand-sha") + self.assertFalse(pair.skip) + assert pair.base is not None and pair.candidate is not None + self.assertEqual(pair.base.head_sha, "base-sha") + self.assertEqual(pair.candidate.head_sha, "cand-sha") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/perf/test_query_regression_runner_compaction_toctou.py b/tests/perf/test_query_regression_runner_compaction_toctou.py index 0e79cf245d..58f7f31b7f 100644 --- a/tests/perf/test_query_regression_runner_compaction_toctou.py +++ b/tests/perf/test_query_regression_runner_compaction_toctou.py @@ -13,9 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Regression coverage for quiescing remote-write storage before read benches.""" +"""Regression coverage for the outer remote-write lifecycle.""" -import argparse import importlib.util import json import sys @@ -25,151 +24,90 @@ from pathlib import Path from unittest.mock import patch -RUNNER_PATH = Path(__file__).with_name("query_regression_runner.py") -SPEC = importlib.util.spec_from_file_location("query_regression_runner_under_test", RUNNER_PATH) +RUNNER_PATH = Path(__file__).resolve().parents[2] / ".github/scripts/query-regression-run.py" +SPEC = importlib.util.spec_from_file_location("query_regression_outer_under_test", RUNNER_PATH) assert SPEC is not None and SPEC.loader is not None runner = importlib.util.module_from_spec(SPEC) sys.modules[SPEC.name] = runner SPEC.loader.exec_module(runner) -class ReportOutputTest(unittest.TestCase): - def test_final_report_output_defaults_to_stdout_or_writes_to_file(self) -> None: - report = {"status": "ok"} - expected = json.dumps(report, indent=2, sort_keys=True) +class RemoteWriteLifecycleTest(unittest.TestCase): + def setUp(self) -> None: + self.args = type("Args", (), {"http_timeout": "1"})() + self.ports = list(range(10000, 10016)) - with patch("builtins.print") as print_mock: - runner.output_report(report, None) - print_mock.assert_called_once_with(expected) + def test_measure_report_is_finalized_after_datanodes_stop(self) -> None: + events: list[str] = [] - with tempfile.TemporaryDirectory() as tmpdir: - output = Path(tmpdir) / "nested" / "report.json" - with patch("builtins.print") as print_mock: - runner.output_report(report, output) - print_mock.assert_not_called() - self.assertEqual(output.read_text(), expected + "\n") + def run(command, **_kwargs): + phase = command[1] + events.append(phase) + if phase == "measure": + output = Path(command[command.index("--output") + 1]) + output.write_text(json.dumps({"targets": [], "thresholds": [], "status": "ok"})) + return type("Result", (), {"returncode": 0})() + def start(target, component, _procs): + events.append(f"start:{target.name}:{component}") -class RemoteWriteCompactionToctouTest(unittest.TestCase): - def test_quiesces_each_target_before_storage_inspection_and_read_bench(self) -> None: - lifecycle: dict[str, list[str]] = {"base": [], "candidate": []} - clusters = [] + def stop(target, component, _procs): + events.append(f"stop:{target.name}:{component}") - class RecordingCluster: - def __init__(self, target): - self.target = target - self.active = set() - self.stop_all_calls = 0 - clusters.append(self) + with ( + tempfile.TemporaryDirectory() as tmpdir, + patch.object(runner, "allocate_ports", return_value=self.ports), + patch.object(runner, "start_component", side_effect=start), + patch.object(runner, "stop_component", side_effect=stop), + patch.object(runner.subprocess, "run", side_effect=run), + ): + self.assertEqual( + runner.run_remote_case( + self.args, Path("case.toml"), Path(tmpdir), Path("base"), Path("candidate"), Path("fixture"), Path("runner") + ), + 0, + ) + self.assertTrue((Path(tmpdir) / "query-regression-report.json").exists()) - def start_all(self, _config_path): - self.active.update(("metasrv", "datanode", "frontend")) + measure = events.index("measure") + finalize = events.index("finalize-remote") + self.assertLess(measure, events.index("stop:base:datanode")) + self.assertLess(events.index("stop:base:datanode"), finalize) + self.assertLess(events.index("stop:candidate:datanode"), finalize) + self.assertIn("stop:base:frontend", events[finalize + 1:]) + self.assertIn("stop:candidate:metasrv", events[finalize + 1:]) - def stop_component(self, name): - if name in self.active: - lifecycle[self.target.name].append(f'cluster.stop_component("{name}")') - self.active.remove(name) + def test_phase_exception_still_cleans_up_without_processes_or_network(self) -> None: + events: list[str] = [] - def stop_all(self): - self.stop_all_calls += 1 - for name in ("frontend", "datanode", "metasrv"): - self.stop_component(name) + def run(command, **_kwargs): + phase = command[1] + events.append(phase) + if phase == "prepare-remote": + raise RuntimeError("prepare exploded") + return type("Result", (), {"returncode": 0})() - def component_report(self): - return {} + def start(target, component, _procs): + events.append(f"start:{target.name}:{component}") - def run_queries(target, _case, _tables, _http_timeout): - lifecycle[target.name].append("query measurement completes") - return {"validation": [], "validation_errors": [], "measurements": [], "status": "ok"} + def stop(target, component, _procs): + events.append(f"stop:{target.name}:{component}") - def inspect_storage(_helper, target, _storage, *, dry_run): - self.assertFalse(dry_run) - lifecycle[target.name].append("storage inspection") - return {"status": "ok", "summary": {"summary": {}}} + with ( + tempfile.TemporaryDirectory() as tmpdir, + patch.object(runner, "allocate_ports", return_value=self.ports), + patch.object(runner, "start_component", side_effect=start), + patch.object(runner, "stop_component", side_effect=stop), + patch.object(runner.subprocess, "run", side_effect=run), + ): + with self.assertRaisesRegex(RuntimeError, "prepare exploded"): + runner.run_remote_case( + self.args, Path("case.toml"), Path(tmpdir), Path("base"), Path("candidate"), Path("fixture"), Path("runner") + ) - def read_bench(_binary, target, _read_bench, _inspection, *, dry_run): - self.assertFalse(dry_run) - lifecycle[target.name].append("read bench") - return {"status": "ok"} - - remote_write = { - "database": "public", - "metric": "cpu_usage", - "physical_table": "cpu_usage_physical", - "series_count": 1, - "samples_per_series": 1, - "visibility_timeout_seconds": 1, - "storage": { - "inspect": True, - "min_files": None, - "min_files_with_column": None, - "max_total_file_size_bytes": None, - "max_column_compressed_size_bytes": None, - "max_column_uncompressed_size_bytes": None, - "require_encodings": [], - "forbid_encodings": [], - }, - "read_bench": {"enabled": True}, - "prom_store": { - "pending_rows_flush_interval": "1s", - "max_batch_rows": 1, - "max_concurrent_flushes": 1, - "worker_channel_capacity": 1, - "max_inflight_requests": 1, - }, - } - case = { - "scenario": { - "kind": "prom_remote_write_then_query", - "remote_write": remote_write, - "queries": [], - } - } - args = argparse.Namespace( - fixture_only=False, - fixture_generator=Path("/bin/true"), - remote_write_generator=None, - storage_inspector=None, - candidate_bin=Path("/bin/true"), - dry_run=False, - http_timeout=1.0, - ) - - with tempfile.TemporaryDirectory() as tmpdir: - root = Path(tmpdir) - targets = [ - runner.make_target("base", Path("/bin/true"), root, list(range(10000, 10008))), - runner.make_target("candidate", Path("/bin/true"), root, list(range(10008, 10016))), - ] - report = {"targets": [], "thresholds": []} - with ( - patch.object(runner, "DistributedCluster", RecordingCluster), - patch.object(runner, "run_remote_write_ingestion", return_value=({"status": "ok"}, [{"ok": True}])), - patch.object(runner, "http_post_sql", return_value={"ok": True}), - patch.object(runner, "poll_expected_count", return_value={"ok": True, "row_count_ok": True}), - patch.object(runner, "run_queries", side_effect=run_queries), - patch.object(runner, "run_storage_inspection", side_effect=inspect_storage), - patch.object(runner, "run_read_bench", side_effect=read_bench), - ): - runner.run_remote_write_scenario(args, case, root / "case.toml", targets, report) - - for cluster in clusters: - self.assertEqual(cluster.stop_all_calls, 1) - self.assertEqual(cluster.active, set()) - cluster.stop_all() - self.assertEqual(cluster.stop_all_calls, 2) - self.assertEqual(cluster.active, set()) - - expected = [ - "query measurement completes", - 'cluster.stop_component("datanode")', - "storage inspection", - "read bench", - ] - relevant = {name: [event for event in events if event in expected] for name, events in lifecycle.items()} - for target_name in ("base", "candidate"): - with self.subTest(target=target_name): - self.assertEqual(relevant[target_name], expected) + self.assertIn("prepare-remote", events) + self.assertIn("stop:base:metasrv", events) + self.assertIn("stop:candidate:frontend", events) if __name__ == "__main__": diff --git a/tests/perf/test_query_regression_runner_otlp_trace_load.py b/tests/perf/test_query_regression_runner_otlp_trace_load.py index ab14ceef24..8b2ca0a0ba 100644 --- a/tests/perf/test_query_regression_runner_otlp_trace_load.py +++ b/tests/perf/test_query_regression_runner_otlp_trace_load.py @@ -13,215 +13,80 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Regression coverage for the local OTLP trace load lifecycle and metrics.""" +"""Regression coverage for the outer OTLP lifecycle.""" import importlib.util import json -import os -import subprocess import sys import tempfile import unittest from pathlib import Path +from types import SimpleNamespace from unittest.mock import patch -RUNNER_PATH = Path(__file__).with_name("query_regression_runner.py") -PLOTTER_PATH = Path(__file__).with_name("plot_otlp_trace_report.sh") -SPEC = importlib.util.spec_from_file_location("query_regression_runner_otlp_under_test", RUNNER_PATH) +RUNNER_PATH = Path(__file__).resolve().parents[2] / ".github/scripts/query-regression-run.py" +SPEC = importlib.util.spec_from_file_location("query_regression_outer_otlp_under_test", RUNNER_PATH) assert SPEC is not None and SPEC.loader is not None runner = importlib.util.module_from_spec(SPEC) sys.modules[SPEC.name] = runner SPEC.loader.exec_module(runner) -class OtlpTraceReportPlotTest(unittest.TestCase): - def test_plots_all_comparison_metrics_and_thresholds(self) -> None: - report = { - "status": "ok", - "targets": [ - { - "name": name, - "metrics": { - "accepted_spans": spans, - "accepted_spans_per_second": rate, - "mean_http_latency_ms": latency, - "failure_count": 0, - }, - "visibility": {"observed_rows": spans}, - } - for name, spans, rate, latency in (("base", 100, 10.0, 3.0), ("candidate", 120, 12.0, 2.5)) - ], - "thresholds": [ - {"target": "base", "threshold": "max_failure_count", "status": "passed", "actual": 0, "limit": 0}, - { - "threshold": "max_candidate_throughput_regression_pct", - "status": "passed", - "actual_pct": -20.0, - "limit_pct": 20.0, - "base": 10.0, - "candidate": 12.0, - }, - ], - } +class OtlpTraceOuterLifecycleTest(unittest.TestCase): + def test_stops_base_before_candidate_and_wires_rust_commands(self) -> None: + events: list[str] = [] + commands: list[list[str]] = [] - with tempfile.TemporaryDirectory() as tmpdir: + def run(command, **_kwargs): + commands.append(command) + phase = command[1] + events.append(phase) + output = Path(command[command.index("--output") + 1]) + output.write_text(json.dumps({"status": "ok", "targets": []})) + return SimpleNamespace(returncode=0) + + def start(target, component, _procs): + events.append(f"start:{target.name}:{component}") + + def stop(target, component, _procs): + events.append(f"stop:{target.name}:{component}") + + args = SimpleNamespace(http_timeout="1", otelgen_bin=Path("otelgen")) + ports = list(range(10000, 10016)) + with ( + tempfile.TemporaryDirectory() as tmpdir, + patch.object(runner, "allocate_ports", return_value=ports), + patch.object(runner, "start_component", side_effect=start), + patch.object(runner, "stop_component", side_effect=stop), + patch.object(runner.subprocess, "run", side_effect=run), + ): root = Path(tmpdir) - report_path = root / "report.json" - report_path.write_text(json.dumps(report)) - fake_bin = root / "bin" - fake_bin.mkdir() - uplot = fake_bin / "uplot" - uplot.write_text('#!/usr/bin/env bash\nprintf "CALL %s\\n" "$*" >>"$UPLOT_LOG"\ncat >>"$UPLOT_LOG"\n') - uplot.chmod(0o755) - log_path = root / "uplot.log" - env = os.environ.copy() - env["PATH"] = f"{fake_bin}{os.pathsep}{env['PATH']}" - env["UPLOT_LOG"] = str(log_path) - - result = subprocess.run([str(PLOTTER_PATH), str(report_path)], env=env, text=True, capture_output=True, check=False) - log = log_path.read_text() - report["targets"][1]["visibility"]["observed_rows"] = 119 - report_path.write_text(json.dumps(report)) - mismatch = subprocess.run([str(PLOTTER_PATH), str(report_path)], env=env, text=True, capture_output=True, check=False) - - self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual(log.count("CALL "), 5) - for metric in ("Accepted spans", "Visible table rows", "Throughput", "Mean HTTP latency", "Failures"): - self.assertIn(metric, log) - self.assertIn("base\t100", log) - self.assertIn("candidate\t12", log) - self.assertIn("Status: ok", result.stdout) - self.assertIn("[passed] candidate: accepted=120, visible=120", result.stdout) - self.assertIn("max_failure_count", result.stdout) - self.assertIn("max_candidate_throughput_regression_pct", result.stdout) - self.assertEqual(mismatch.returncode, 1, mismatch.stderr) - self.assertIn("[failed] candidate: accepted=120, visible=119", mismatch.stdout) - - -class OtlpTraceLoadTest(unittest.TestCase): - def test_stops_base_cluster_before_creating_candidate_cluster(self) -> None: - events = [] - - class FakeCluster: - def __init__(self, target): - self.target = target - self.stopped = False - events.append(f"create:{target.name}") - - def component_report(self): - return {} - - def stop_all(self): - if not self.stopped: - self.stopped = True - events.append(f"stop:{self.target.name}") - - load = { - "database": "public", - "table": "opentelemetry_traces", - "pipeline": "greptime_trace_v1", - "duration_seconds": 120, - "warmup_seconds": 60, - "rate": 50_000, - "workers": 4, - "workload": "microservices", - "exporter_shards": 4, - "visibility_timeout_seconds": 1, - "thresholds": { - "max_candidate_throughput_regression_pct": 20, - "max_candidate_mean_latency_regression_pct": 20, - "max_failure_count": 0, - }, - } - args = runner.argparse.Namespace(fixture_only=False, otelgen_bin=None, dry_run=True, http_timeout=1.0) - - with tempfile.TemporaryDirectory() as tmpdir: - root = Path(tmpdir) - targets = [ - runner.make_target("base", Path("/bin/true"), root, list(range(10_000, 10_008))), - runner.make_target("candidate", Path("/bin/true"), root, list(range(10_008, 10_016))), + self.assertEqual( + runner.run_otlp_case(args, Path("case.toml"), root, Path("base-bin"), Path("candidate-bin"), Path("fixture"), Path("runner")), + 0, + ) + expected = [ + [ + "runner", "run-otlp-target", "--case", "case.toml", "--fixture-generator", "fixture", + "--otelgen-bin", "otelgen", "--http-port", "10004", "--target-name", "base", + "--work-dir", str(root / "base"), "--output", str(root / "base" / "report.json"), "--http-timeout", "1", + ], + [ + "runner", "run-otlp-target", "--case", "case.toml", "--fixture-generator", "fixture", + "--otelgen-bin", "otelgen", "--http-port", "10012", "--target-name", "candidate", + "--work-dir", str(root / "candidate"), "--output", str(root / "candidate" / "report.json"), "--http-timeout", "1", + ], + [ + "runner", "finalize-otlp", "--case", "case.toml", "--fixture-generator", "fixture", + "--base-result", str(root / "base" / "report.json"), + "--candidate-result", str(root / "candidate" / "report.json"), + "--output", str(root / "query-regression-report.json"), + ], ] - with patch.object(runner, "DistributedCluster", FakeCluster): - runner.run_otlp_trace_load_scenario( - args, - {"scenario": {"kind": "otlp_trace_load", "load": load}}, - targets, - {"targets": []}, - ) + self.assertEqual(commands, expected) - self.assertEqual(events, ["create:base", "stop:base", "create:candidate", "stop:candidate"]) - - def test_warmup_lifecycle_and_labeled_metric_deltas(self) -> None: - class FakeProcess: - def __init__(self, *_args, **_kwargs): - self.returncode = None - self.wait_timeouts = [] - - def wait(self, timeout): - self.wait_timeouts.append(timeout) - if len(self.wait_timeouts) == 1: - raise subprocess.TimeoutExpired("otelgen", timeout) - self.returncode = 0 - return 0 - - def poll(self): - return self.returncode - - def kill(self): - self.returncode = -9 - - def snapshot(text: str, captured: float): - return {"captured_monotonic_seconds": captured, "values": runner.parse_prometheus_metrics(text)} - - snapshots = [ - snapshot("greptime_frontend_otlp_traces_rows 10\n", 0.0), - snapshot( - 'greptime_frontend_otlp_traces_rows 110\n' - 'greptime_servers_http_otlp_traces_elapsed_sum{db="public"} 1\n' - 'greptime_servers_http_otlp_traces_elapsed_count{db="public"} 10\n', - 5.0, - ), - snapshot( - 'greptime_frontend_otlp_traces_rows 310\n' - 'greptime_frontend_otlp_traces_failure_count{label="decode"} 1\n' - 'greptime_frontend_otlp_traces_failure_count{label="write"} 2\n' - 'greptime_servers_http_otlp_traces_elapsed_sum{db="public"} 3\n' - 'greptime_servers_http_otlp_traces_elapsed_count{db="public"} 30\n', - 15.0, - ), - ] - load = { - "database": "public", - "table": "opentelemetry_traces", - "pipeline": "greptime_trace_v1", - "duration_seconds": 120, - "warmup_seconds": 60, - "rate": 50_000, - "workers": 4, - "workload": "microservices", - "exporter_shards": 4, - } - - with tempfile.TemporaryDirectory() as tmpdir: - target = runner.make_target("base", Path("/bin/true"), Path(tmpdir), list(range(10_000, 10_008))) - process = FakeProcess() - with ( - patch.object(runner, "fetch_otlp_metrics", side_effect=snapshots), - patch.object(runner.subprocess, "Popen", return_value=process), - patch.object(runner.time, "monotonic", side_effect=[0.0, 120.0]), - ): - result = runner.run_otelgen_load(Path("/bin/otelgen"), target, load, 1.0, dry_run=False) - - self.assertEqual(result["status"], "ok") - self.assertEqual(process.wait_timeouts, [60, 120]) - metrics = runner.summarize_otlp_metrics(result) - self.assertEqual(metrics["accepted_spans"], 300) - self.assertEqual(metrics["measurement_accepted_spans"], 200) - self.assertEqual(metrics["accepted_spans_per_second"], 20.0) - self.assertEqual(metrics["http_requests"], 20) - self.assertEqual(metrics["mean_http_latency_ms"], 100.0) - self.assertEqual(metrics["failure_count"], 3) + self.assertLess(events.index("stop:base:metasrv"), events.index("start:candidate:metasrv")) if __name__ == "__main__":