diff --git a/benchmarks/README.md b/benchmarks/README.md index fc758c006f..17574920a7 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -69,3 +69,40 @@ Generates SVG graphs from `*_benchmark.json` data files. ## CI The GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs hourly with 1/4/8 worker configurations plus WAC benchmarks. Results are committed to the `benchmarks` branch. + +## Cluster benchmarks — sim mode + +Provisioning + measuring throughput on a real multi-node Kubernetes cluster +(minikube + helm + per-pod cgroup sampling + PG analysis) is a separate +workflow on top of `main.ts`. See [`sim/README.md`](sim/README.md) for: + +- bringing up the cluster (`wm_sim up`) +- firing a phased or flood workload from `workloads/` +- the consolidated dashboard with throughput, queue depth, per-node CPU + + oversaturation, PG latency / conn counts, OOM events, restart events +- the JSONL poller pipeline + reliability fixes (sampler host-log, rollout- + complete readiness, procs_running oversaturation) + +Quick path: + +```sh +# Provision cluster + deploy Windmill (foreground; Ctrl-C tears down): +wm_sim up \ + --topology sim/topologies/k8s-4node.json \ + --helm ../windmill-helm-charts/charts/windmill \ + --helm-values sim/values/smoke.yaml \ + --helm-values sim/values/local.yaml + +# In another shell — port-forward + run bench: +kubectl --context wm-sim-k8s-4node port-forward --address 0.0.0.0 \ + -n default svc/windmill-app 33405:8000 & + +deno run -A main.ts \ + --host http://127.0.0.1:33405 \ + --token \ + --workload-config workloads/io_150ms_flood.json \ + --minikube-profile wm-sim-k8s-4node \ + --wait-ready 60 + +# Report appears at reports//dashboard.svg +``` diff --git a/benchmarks/sim/README.md b/benchmarks/sim/README.md index adb2cd3c88..b62336d320 100644 --- a/benchmarks/sim/README.md +++ b/benchmarks/sim/README.md @@ -1,160 +1,263 @@ # Windmill benchmark sim -Provisioning layer on top of the existing benchmark harness. Reuses everything -in `benchmarks/` (workload kinds, `runBenchmark`, graph primitives) and adds: -declarative topology, per-node resource limits, optional per-node DB latency, -optional CPU pinning for benchmark accuracy, and per-worker metrics collection. +Provisions a minikube-backed multi-node Kubernetes cluster, deploys Windmill +on it via Helm, runs workload benches against it, and produces a single SVG +dashboard with throughput, queue depth, per-node CPU + memory + oversaturation, +per-pod CPU/memory peaks, PG response latency + connection counts, OOM and +restart events, and pod inventory. -## Three node modes +The goal: realistic, reproducible benches at scales where running Windmill +on bare containers (`docker run`) hides the production failure modes +(scheduler saturation, cgroup_mutex contention, PG backend thrash, kubelet +log rotation, OOM cascades). Everything here exists to make those +production-shape problems observable in 3 minutes from a clean checkout. -Every node in a topology runs in one of three modes: +## What you actually get -| Mode | What it is | How it's provisioned | -|------|-----------|----------------------| -| `native` | Windmill `MODE=standalone` — API server + workers in one process. The harness pushes jobs at its API. | One docker (or podman) container per node, `--cpus` / `--memory` / optional `--cpuset-cpus`. | -| `agent` | Windmill `MODE=worker` — worker-only, no API. Models a worker fleet that scales separately from the API. | Same as `native`, minus the API port mapping. Needs a peer `native` node so the harness has somewhere to push. | -| `k8s` | A k3d cluster with Windmill deployed via the helm chart (upstream or a locally-cloned copy). | `k3d cluster create` provisions k3s in containers, then `helm install` deploys Windmill onto it. Planned for M5 — schema and design are in place. | - -The provisioner is a single class that handles all three. The container -runtime is configurable via `SIM_CONTAINER_CMD` (defaults to `docker`, -swap to `podman` if you prefer). For k8s mode, `k3d` and `helm` need to be -on `PATH`. - -## Layered design - -``` -sim.ts (this layer) - → loads suite + topology - → SimProvisioner: PG + toxiproxy (if needed) + per-node provisioning - → calls existing runBenchmark() against the chosen API node - → Collector queries v2_job_completed / v2_job_queue from outside the cluster - → writes JSON + markdown report - → teardown - │ - ▼ -benchmarks/benchmark_oneoff.ts (existing, unchanged) - → login, createBenchScript, push jobs, poll for completion - │ - ▼ -Windmill instance(s) (containers or k8s pods) - → BENCHMARK_KIND env loads fixtures via backend bench.rs - → workers run normal pull loop - │ - ▼ -Postgres container (shared by every node) -``` - -## Two config files per run - -1. **Topology** (e.g. `topologies/single_node.json`) — pure infra description. -2. **Suite** (e.g. `configs/baseline.json`) — workload, optionally references - a topology by relative path. - -Suites without a `topology` reference are first-class and continue to run via -`benchmark_suite.ts` against any existing Windmill at `--host`. Suites that -declare a topology run via `sim.ts` (this directory), which provisions infra -before delegating to the same underlying runner. - -## Example topologies - -- `topologies/single_node.json` — 1 PG + 1 native node. Baseline smoke test. -- `topologies/heterogeneous_latency.json` — 1 native + 1 agent at 50ms DB RTT. - The "does the pull race self-level?" experiment. -- `topologies/k8s_3node.json` — k3d cluster shape: 3 worker pods. Helm chart - selection is passed on the CLI (see below); the topology only describes the - cluster. - -## k8s mode: chart selection via CLI - -The topology specifies cluster shape (pods, workers_per_pod, cluster_agents). -The helm chart itself is selected at the sim CLI: +Single CLI to bring up the cluster + deploy Windmill: ```sh -# Upstream chart -deno run -A sim.ts -c configs/k8s.json \ - --helm-chart windmill/windmill \ - --helm-repo https://windmill-labs.github.io/windmill-helm-charts - -# Locally-cloned chart (relative to cwd) -deno run -A sim.ts -c configs/k8s.json \ - --helm-chart ../windmill-helm-charts/charts/windmill - -# Inline value overrides -deno run -A sim.ts -c configs/k8s.json --helm-chart windmill/windmill \ - --helm-set worker.replicaCount=5 \ - --helm-set windmill.numWorkers=2 \ - --helm-values-file ./extra-values.yaml +wm_sim up \ + --topology sim/topologies/k8s-4node.json \ + --helm ../windmill-helm-charts/charts/windmill \ + --helm-values sim/values/smoke.yaml \ + --helm-values sim/values/local.yaml # gitignored; carries the EE license ``` +Then in a second shell, port-forward the API + fire a bench: + +```sh +kubectl --context wm-sim-k8s-4node port-forward --address 0.0.0.0 \ + -n default svc/windmill-app 33405:8000 & + +cd benchmarks && deno run -A main.ts \ + --host http://127.0.0.1:33405 \ + --token \ + --workload-config workloads/io_150ms_flood.json \ + --minikube-profile wm-sim-k8s-4node \ + --wait-ready 60 +``` + +Bench runs ~3 min. Report lands at `benchmarks/reports//`, with +`dashboard.svg` as the entry point. Open it in a browser. + ## Prerequisites -- A container runtime: `docker` (default) or `podman` (set - `SIM_CONTAINER_CMD=podman`). -- A Windmill image built with the `benchmark` Cargo feature, tagged - `windmill-bench:latest`. Override the tag via `SIM_WM_IMAGE`. The image - must honour `DATABASE_URL`, `NUM_WORKERS`, `MODE` (`standalone` or - `worker`), `BENCHMARK_KIND`, and `WORKER_TAGS`. -- Postgres 17 image (default `postgres:17`, override via `SIM_PG_IMAGE`). -- For k8s mode only: `k3d` and `helm` on `PATH`. +- `nix develop` from the repo flake — provisions `minikube`, `kubectl`, `helm`, + `pgbadger`, `deno`, and the `wm_sim` wrapper. **No host config required + beyond the flake** with one exception: if you want multi-node KVM clusters + on NixOS, libvirtd needs to be enabled at the system level — see + `sim/nix/kvm2-driver.nix`. +- Root + libvirtd for the kvm2 driver. Single-node clusters work with the + qemu2 driver without root, but multi-node needs kvm2. +- A clone of `windmill-helm-charts` next to this repo (`../windmill-helm-charts`). + The vendored chart was deleted in favor of using the upstream chart with + bench-specific overlays in `values/smoke.yaml`. The chart depends on a few + knobs (PriorityClass support, `oomImmune`, `maxConnections`) that may not + be in upstream yet — see the windmill-helm-charts PR. -## Run +## Topology + values layout -```sh -cd benchmarks/sim -deno run -A sim.ts -c configs/baseline.json +``` +sim/ + topologies/ + k8s-4node.json # 1 control plane + 3 workers (4 vCPU, 20 GiB each) + k8s-3node.json # control plane + 2 workers + ... # several other shapes + values/ + smoke.yaml # bench-tuned helm overlay (worker reqs, PG sizing, + # priorityClass=wm-critical, maxConnections, etc.) + local.example.yaml # template — copy to local.yaml, fill in your + # EE license key. local.yaml is gitignored. ``` -Useful flags: +`smoke.yaml` is the source of truth for chart config across benches. Tune +worker replica counts, PG memory, max_connections, etc. there. -- `--factor N` — multiply every benchmark's `jobs` count by N. -- `--keep-infra` — leave containers running after the run. -- `--snapshot-interval ` — queue-depth sampling cadence (default 1000). +## Architecture -## Outputs +``` +wm_sim up + │ + ▼ provision + ┌────────────────────────────────────────────────────────────────┐ + │ k8s_provisioner: minikube start + add-nodes + per-node sizing │ + │ image_cache: pre-load images so bringup is offline-safe │ + │ helm_deploy: helm upgrade --install windmill via smoke.yaml│ + │ toxiproxy_k8s: per-node toxiproxy DaemonSet for latency inj │ + │ cpu_sampler_k8s: per-node sampler DaemonSet (10Hz cpu.stat) │ + └────────────────────────────────────────────────────────────────┘ + │ + ▼ benchmarks/main.ts (separate shell) + ┌────────────────────────────────────────────────────────────────┐ + │ readiness: wait for samplers stable, workers Ready, │ + │ PG responsive, queue empty, │ + │ *deployment rollout complete* — mid-rollout │ + │ pod churn starves the sampler under │ + │ cgroup_mutex contention │ + │ start pollers: pod_timeline, oom, pg_latency, pg_conn, │ + │ node_load (all write JSONL into report dir) │ + │ truncate sampler /var/log/wm-sim-cpu-sampler/sampler.tsv on │ + │ host log: each node so this run gets a clean file │ + │ push jobs: pushers → API → workers pull → PG queue │ + │ stop pollers: finalize JSONL │ + │ collect samples: scp host log from each node (fallback: │ + │ kubectl logs) │ + │ collect pg.log: kubectl logs PG pod → pgBadger HTML report │ + │ render_report: JSONL + samples → dashboard.svg + report.md │ + └────────────────────────────────────────────────────────────────┘ + │ + ▼ output: benchmarks/reports// +``` -Each run writes two files in the working directory: +## Measurement subsystem -- `sim_results_.json` — raw structured data: workload results, - per-worker stats, queue snapshots, derived throughput. -- `sim_results_.md` — human-readable summary: throughput table, - per-worker table sorted by jobs handled, **utilization spread** headline, - ASCII sparkline of queue depth over time. +| Signal | Source | Cadence | File in report | +|---|---|---|---| +| Per-pod CPU + memory | DS sampler reading cgroup cpu.stat / memory.current | 10 Hz | `cpu_samples.tsv` | +| Workers Ready per node | `kubectl get pods` | 1 Hz | `pod_timeline.jsonl` | +| OOM events | `kubectl get events` + node-kernel dmesg | live | `oom_events_live.jsonl` | +| PG query latency | `psql -c "\timing on" -c "SELECT 1"` | 4 Hz | `pg_latency.jsonl` | +| PG connections by state | `SELECT state, count(*) FROM pg_stat_activity` | 1 Hz | `pg_connections.jsonl` | +| Node loadavg + procs_running | ssh `/proc/loadavg`, `/proc/stat` | 0.5 Hz | `node_load.jsonl` | +| Throughput (jobs/s sent + processed) | bench worker `postMessage` | sub-second | `throughput_samples.json` | +| Failed jobs | `/api/w/{ws}/jobs/completed/list` | post-bench | `failed_jobs.jsonl` | +| Pod inventory + node placement | `kubectl get pods -o json` | post-bench | `pods.json` | +| PG slow-query analysis | PG log → pgBadger | post-bench, background | `pgbadger.html` | -The headline metric for the question this sim is built to answer ("are -workers balanced?") is the utilization spread line. A small spread (e.g. -±5pp) means the queue is self-balancing. A large spread (e.g. close -worker 95%, far worker 40%) is the smoking gun for uneven pull dynamics. +Key reliability fixes (don't undo them): -## Resource validation +- **Sampler host file**: the sampler dual-writes to stdout AND a hostPath log + file. `kubectl logs --since-time=X` loses early data when kubelet rotates + log files under heavy benches (m04 was missing the first 90-120s of every + busy run). Collector now scp's the host file as primary source. +- **Readiness rollout check**: `kubectl get deploy windmill-workers-default` + is checked for `status.updatedReplicas == spec.replicas`. Mid-rolling-update + pod churn floods the kernel's `cgroup_mutex`, starving every other process + on the node — including the sampler. Pod-readiness alone isn't enough. +- **Oversaturation uses procs_running**, not load1. loadavg includes D-state + procs (PG backends waiting on disk I/O, kernel mutex contention) which has + nothing to do with CPU starvation but inflates "oversaturation" by 5-10x. + `procs_running` from `/proc/stat` is the runnable count. -`sim.ts` refuses to run if the topology's total CPU or memory exceeds the -host's available budget after the `reserved` buffer. `host_resources` is -optional in the topology — when omitted, the validator falls back to the -live host's logical core count and `MemTotal`. Declare it explicitly when -you want the topology to be portable to a specific target host. +## Dashboard panels -Set `pin_cpus: true` on the topology to assign each container a disjoint -`--cpuset-cpus` range. This eliminates CFS time-slicing between containers -entirely; useful when two nodes' throughput numbers must not be -artificially correlated. +Rendered into `dashboard.svg`. Layout is roughly: + +| Section | Panels | +|---|---| +| **Run context** | Topology, workload, helm values, bench cmd, report dir | +| **Throughput** | Jobs sent + processed over time, with push-window shading and phase boundaries | +| **Queue depth** | Pending jobs in `v2_job_queue` | +| **Node memory** | Per-node memory time-series + per-phase average bar chart | +| **PG latency** | Pure SQL time (psql `\timing`) + kubectl-exec roundtrip (separate axes) | +| **PG connections** | active / idle / idle_in_xact over time | +| **Util group** | One panel per node, 2 per row. Solid blue CPU util (capped 100%) on top of translucent orange oversaturation. 100% reference line. PG-host node flagged `[PG]` | +| **Node CPU** | Multi-line per-node CPU, with PG-host node tinted via stroke + label | +| **Pod inventory** | Bar chart of pod count by node and pod-type donut | +| **Failed jobs** | Cumulative + categorized bars | +| **Restart events** | Pod restarts detected from `pod_timeline.jsonl` | +| **OOM events** | L0 scheduler preemptions, L1 kubelet evictions, L2 cgroup + node-kernel kills | + +All time-series share an x-axis origin (`bench_start_ms` written into +`meta.json`) so the same x-coordinate on every panel means the same wall-clock +moment. Tick labels are relative seconds ("0s, 30s, …"), not wall-clock. + +## Workloads + +`benchmarks/workloads/*.json`. Each is a phased JSON spec the bench reads at +startup. Phases run sequentially, each with its own pusher count, mode mix, +duration range, and ram_mb distribution. + +Examples: + +| Workload | Shape | Use case | +|---|---|---| +| `io_4phase.json` | idle → 2.5s → 500ms → 150ms IO jobs | Steady-state vs saturation in one bench | +| `io_150ms_flood.json` | 60 pushers, 150ms IO, 120s | Worker-host CFS saturation regime | +| `io_300ms_flood.json` | 60 pushers, 300ms IO, 120s | Lower context-switch rate | +| `io_1s_flood.json`, `io_2s_flood.json` | Same shape, longer jobs | Near-theoretical worker scaling | +| `ops_day.json` | Mixed phases simulating a day | Realistic load profile | +| `cpu_heavy.json`, `mixed.json`, `burst.json` | Non-IO and bursty shapes | CPU-bound and spikiness | + +## Agent workers (HTTP+JWT path) + +`smoke.yaml` declares a second worker group `agent` with `mode: agent`. Agent +workers reach the windmill API over HTTP+JWT instead of holding a direct +sqlx pool to PG. Useful for measuring how much of the bench cap is PG +contention vs. the HTTP-mediated path. + +The JWT lives in a K8s Secret `windmill-agent-token` (not in any committed +yaml). Create it once: + +```sh +kubectl --context wm-sim-k8s-4node create secret generic windmill-agent-token \ + --from-literal=token='jwt_agent_…' +``` ## Tests ```sh -nix develop --command bash -c "cd benchmarks && deno test -A sim/*_test.ts" +cd benchmarks && deno test --allow-import --no-check \ + sim/util_metrics_test.ts sim/util_panel_snapshot_test.ts ``` -38 unit tests cover the validator, config loader, proxy planner, schema -modes, and report writer. The provisioner and Postgres collector are -intentionally not unit-tested — they need real infrastructure to exercise. +- `sim/util_metrics_test.ts` — 8 tests for `computeOversatPct`, the formula + the util panel uses. Specifically guards the procs_running > load1 fallback, + the clamp-at-zero, invalid-ncpu inputs. +- `sim/util_panel_snapshot_test.ts` — 5 assertions on the rendered util-panel + SVG: orange-behind-blue z-order, 100% reference line, relative-time ticks + (no wall-clock leak), phase-boundary dashed verticals, shared-origin override. +- `sim/topology_test.ts`, `sim/toxiproxy_test.ts` — earlier coverage of the + validator + proxy planner. -## Roadmap +Pre-existing typecheck errors in `graph.ts` (untyped d3 callback args, +implicit `any`) trip `deno test` without `--no-check`. The functional tests +pass regardless. -- **M1 ✅**: scaffold + single-node baseline + container provisioner + - resource validator. -- **M2 ✅**: multi-node + per-node toxiproxy latency. -- **M3 ✅**: metrics collection from Postgres (per-worker utilization, - queue depth over time). -- **M4 ✅**: markdown + JSON report. SVG graphs deferred until needed. -- **M5**: k3d-based k8s mode with helm install (upstream or local chart). -- **M6+**: chaos timeline, assertion engine, resilience test corpus. +## Operational notes + +- Cluster name / minikube profile is `wm-sim-k8s-4node` by default. Pass + `--minikube-profile ` to `main.ts` if you renamed it. +- Default kubectl context comes from minikube. Use + `kubectl --context wm-sim-k8s-4node …` to be explicit. +- The port-forward dies easily under heavy bench load (kubelet drops the + connection). If a bench fails partway with "connection refused", restart + PF and re-fire. +- `minikube stop` is safe — VM disks persist, etcd survives, helm releases + + Secrets all come back when you `minikube start`. `wm_sim up` does + `minikube delete` first, so it's destructive — only use it for clean reprovisions. + +## Code layout + +``` +sim/ + sim.ts # wm_sim CLI entry (cliffy) + k8s_provisioner.ts # minikube up + node sizing + helm_deploy.ts # helm upgrade --install + image_cache.ts # pre-load images so bringup is offline-safe + cpu_sampler_k8s.ts # sampler DaemonSet + collector + toxiproxy_k8s.ts # per-node toxiproxy DaemonSet + pg_logging.ts # ALTER SYSTEM + SIGHUP for verbose PG logs + pgbadger.ts # post-bench PG log → HTML + readiness.ts # pre-bench cluster health check + pod_timeline.ts # pod readiness poller + pod_inventory.ts # post-bench pod inventory snapshot + oom_poller.ts # live OOM event capture + oom_events.ts # post-bench OOM event parsing + pg_latency_poller.ts # 4Hz PG \timing on SELECT 1 + pg_conn_poller.ts # pg_stat_activity by state + node_load_poller.ts # /proc/loadavg + /proc/stat procs_running + failed_jobs.ts # post-bench failed-jobs fetch + util_metrics.ts # pure helpers (oversat formula) + render_report.ts # JSONL → SVG dashboard + report.md + dashboard.ts # SVG layout primitives (grid, sections) + svg_to_pdf.ts # dashboard.svg → dashboard.pdf (best-effort) + topology.ts # topology JSON loader + validator + values/ # helm overlay yamls + topologies/ # topology JSON files + nix/kvm2-driver.nix # kvm2 driver overlay for nixpkgs +``` + +The bench runner itself is `benchmarks/main.ts` (one level up). It calls +into `sim/` for measurement + report rendering.