mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-28 00:02:42 +00:00
docs(benchmarks): rewrite sim/README + link from benchmarks/README
sim/README.md was last touched May 27 and described the pre-minikube docker/podman-based "three node modes" design with k8s mode listed as "planned for M5". The reality is now k8s-only via minikube + helm, and the workflow / config-files / outputs all changed. New sim/README.md covers: - The actual `wm_sim up` CLI form against the external windmill-helm-charts chart and smoke.yaml / local.yaml overlays. - Two-shell workflow: wm_sim up in one, port-forward + main.ts in the other, report at reports/<timestamp>/dashboard.svg. - Measurement subsystem table: per-pod CPU/mem sampler DS, JSONL pollers (pod_timeline, oom, pg_latency, pg_conn, node_load), throughput capture, failed-jobs fetch, pgBadger. - The three reliability fixes we leaned on this session and why they matter: sampler host-log file (kubelet log rotation), readiness rollout-complete check (cgroup_mutex starvation), procs_running for oversaturation (not load1 / D-state overcount). - Dashboard panel index + the shared bench_start_ms x-axis origin. - Workload catalog (io_4phase + flood variants + ops_day + cpu). - Agent worker setup with the Secret-based JWT. - Test entry points (util_metrics_test, util_panel_snapshot_test). - Operational notes (port-forward fragility, minikube stop safety). - File-by-file code layout under sim/. benchmarks/README.md gains a "Cluster benchmarks — sim mode" section linking to sim/README.md plus a quick-start snippet so the path from the legacy benchmark suite docs into the cluster workflow is obvious. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
pyranota
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
74b662d8de
commit
4356e60eff
@@ -69,3 +69,40 @@ Generates SVG graphs from `*_benchmark.json` data files.
|
||||
## CI
|
||||
|
||||
The GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs hourly with 1/4/8 worker configurations plus WAC benchmarks. Results are committed to the `benchmarks` branch.
|
||||
|
||||
## Cluster benchmarks — sim mode
|
||||
|
||||
Provisioning + measuring throughput on a real multi-node Kubernetes cluster
|
||||
(minikube + helm + per-pod cgroup sampling + PG analysis) is a separate
|
||||
workflow on top of `main.ts`. See [`sim/README.md`](sim/README.md) for:
|
||||
|
||||
- bringing up the cluster (`wm_sim up`)
|
||||
- firing a phased or flood workload from `workloads/`
|
||||
- the consolidated dashboard with throughput, queue depth, per-node CPU +
|
||||
oversaturation, PG latency / conn counts, OOM events, restart events
|
||||
- the JSONL poller pipeline + reliability fixes (sampler host-log, rollout-
|
||||
complete readiness, procs_running oversaturation)
|
||||
|
||||
Quick path:
|
||||
|
||||
```sh
|
||||
# Provision cluster + deploy Windmill (foreground; Ctrl-C tears down):
|
||||
wm_sim up \
|
||||
--topology sim/topologies/k8s-4node.json \
|
||||
--helm ../windmill-helm-charts/charts/windmill \
|
||||
--helm-values sim/values/smoke.yaml \
|
||||
--helm-values sim/values/local.yaml
|
||||
|
||||
# In another shell — port-forward + run bench:
|
||||
kubectl --context wm-sim-k8s-4node port-forward --address 0.0.0.0 \
|
||||
-n default svc/windmill-app 33405:8000 &
|
||||
|
||||
deno run -A main.ts \
|
||||
--host http://127.0.0.1:33405 \
|
||||
--token <admin-token> \
|
||||
--workload-config workloads/io_150ms_flood.json \
|
||||
--minikube-profile wm-sim-k8s-4node \
|
||||
--wait-ready 60
|
||||
|
||||
# Report appears at reports/<timestamp>/dashboard.svg
|
||||
```
|
||||
|
||||
+232
-129
@@ -1,160 +1,263 @@
|
||||
# Windmill benchmark sim
|
||||
|
||||
Provisioning layer on top of the existing benchmark harness. Reuses everything
|
||||
in `benchmarks/` (workload kinds, `runBenchmark`, graph primitives) and adds:
|
||||
declarative topology, per-node resource limits, optional per-node DB latency,
|
||||
optional CPU pinning for benchmark accuracy, and per-worker metrics collection.
|
||||
Provisions a minikube-backed multi-node Kubernetes cluster, deploys Windmill
|
||||
on it via Helm, runs workload benches against it, and produces a single SVG
|
||||
dashboard with throughput, queue depth, per-node CPU + memory + oversaturation,
|
||||
per-pod CPU/memory peaks, PG response latency + connection counts, OOM and
|
||||
restart events, and pod inventory.
|
||||
|
||||
## Three node modes
|
||||
The goal: realistic, reproducible benches at scales where running Windmill
|
||||
on bare containers (`docker run`) hides the production failure modes
|
||||
(scheduler saturation, cgroup_mutex contention, PG backend thrash, kubelet
|
||||
log rotation, OOM cascades). Everything here exists to make those
|
||||
production-shape problems observable in 3 minutes from a clean checkout.
|
||||
|
||||
Every node in a topology runs in one of three modes:
|
||||
## What you actually get
|
||||
|
||||
| Mode | What it is | How it's provisioned |
|
||||
|------|-----------|----------------------|
|
||||
| `native` | Windmill `MODE=standalone` — API server + workers in one process. The harness pushes jobs at its API. | One docker (or podman) container per node, `--cpus` / `--memory` / optional `--cpuset-cpus`. |
|
||||
| `agent` | Windmill `MODE=worker` — worker-only, no API. Models a worker fleet that scales separately from the API. | Same as `native`, minus the API port mapping. Needs a peer `native` node so the harness has somewhere to push. |
|
||||
| `k8s` | A k3d cluster with Windmill deployed via the helm chart (upstream or a locally-cloned copy). | `k3d cluster create` provisions k3s in containers, then `helm install` deploys Windmill onto it. Planned for M5 — schema and design are in place. |
|
||||
|
||||
The provisioner is a single class that handles all three. The container
|
||||
runtime is configurable via `SIM_CONTAINER_CMD` (defaults to `docker`,
|
||||
swap to `podman` if you prefer). For k8s mode, `k3d` and `helm` need to be
|
||||
on `PATH`.
|
||||
|
||||
## Layered design
|
||||
|
||||
```
|
||||
sim.ts (this layer)
|
||||
→ loads suite + topology
|
||||
→ SimProvisioner: PG + toxiproxy (if needed) + per-node provisioning
|
||||
→ calls existing runBenchmark() against the chosen API node
|
||||
→ Collector queries v2_job_completed / v2_job_queue from outside the cluster
|
||||
→ writes JSON + markdown report
|
||||
→ teardown
|
||||
│
|
||||
▼
|
||||
benchmarks/benchmark_oneoff.ts (existing, unchanged)
|
||||
→ login, createBenchScript, push jobs, poll for completion
|
||||
│
|
||||
▼
|
||||
Windmill instance(s) (containers or k8s pods)
|
||||
→ BENCHMARK_KIND env loads fixtures via backend bench.rs
|
||||
→ workers run normal pull loop
|
||||
│
|
||||
▼
|
||||
Postgres container (shared by every node)
|
||||
```
|
||||
|
||||
## Two config files per run
|
||||
|
||||
1. **Topology** (e.g. `topologies/single_node.json`) — pure infra description.
|
||||
2. **Suite** (e.g. `configs/baseline.json`) — workload, optionally references
|
||||
a topology by relative path.
|
||||
|
||||
Suites without a `topology` reference are first-class and continue to run via
|
||||
`benchmark_suite.ts` against any existing Windmill at `--host`. Suites that
|
||||
declare a topology run via `sim.ts` (this directory), which provisions infra
|
||||
before delegating to the same underlying runner.
|
||||
|
||||
## Example topologies
|
||||
|
||||
- `topologies/single_node.json` — 1 PG + 1 native node. Baseline smoke test.
|
||||
- `topologies/heterogeneous_latency.json` — 1 native + 1 agent at 50ms DB RTT.
|
||||
The "does the pull race self-level?" experiment.
|
||||
- `topologies/k8s_3node.json` — k3d cluster shape: 3 worker pods. Helm chart
|
||||
selection is passed on the CLI (see below); the topology only describes the
|
||||
cluster.
|
||||
|
||||
## k8s mode: chart selection via CLI
|
||||
|
||||
The topology specifies cluster shape (pods, workers_per_pod, cluster_agents).
|
||||
The helm chart itself is selected at the sim CLI:
|
||||
Single CLI to bring up the cluster + deploy Windmill:
|
||||
|
||||
```sh
|
||||
# Upstream chart
|
||||
deno run -A sim.ts -c configs/k8s.json \
|
||||
--helm-chart windmill/windmill \
|
||||
--helm-repo https://windmill-labs.github.io/windmill-helm-charts
|
||||
|
||||
# Locally-cloned chart (relative to cwd)
|
||||
deno run -A sim.ts -c configs/k8s.json \
|
||||
--helm-chart ../windmill-helm-charts/charts/windmill
|
||||
|
||||
# Inline value overrides
|
||||
deno run -A sim.ts -c configs/k8s.json --helm-chart windmill/windmill \
|
||||
--helm-set worker.replicaCount=5 \
|
||||
--helm-set windmill.numWorkers=2 \
|
||||
--helm-values-file ./extra-values.yaml
|
||||
wm_sim up \
|
||||
--topology sim/topologies/k8s-4node.json \
|
||||
--helm ../windmill-helm-charts/charts/windmill \
|
||||
--helm-values sim/values/smoke.yaml \
|
||||
--helm-values sim/values/local.yaml # gitignored; carries the EE license
|
||||
```
|
||||
|
||||
Then in a second shell, port-forward the API + fire a bench:
|
||||
|
||||
```sh
|
||||
kubectl --context wm-sim-k8s-4node port-forward --address 0.0.0.0 \
|
||||
-n default svc/windmill-app 33405:8000 &
|
||||
|
||||
cd benchmarks && deno run -A main.ts \
|
||||
--host http://127.0.0.1:33405 \
|
||||
--token <admin-token> \
|
||||
--workload-config workloads/io_150ms_flood.json \
|
||||
--minikube-profile wm-sim-k8s-4node \
|
||||
--wait-ready 60
|
||||
```
|
||||
|
||||
Bench runs ~3 min. Report lands at `benchmarks/reports/<timestamp>/`, with
|
||||
`dashboard.svg` as the entry point. Open it in a browser.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A container runtime: `docker` (default) or `podman` (set
|
||||
`SIM_CONTAINER_CMD=podman`).
|
||||
- A Windmill image built with the `benchmark` Cargo feature, tagged
|
||||
`windmill-bench:latest`. Override the tag via `SIM_WM_IMAGE`. The image
|
||||
must honour `DATABASE_URL`, `NUM_WORKERS`, `MODE` (`standalone` or
|
||||
`worker`), `BENCHMARK_KIND`, and `WORKER_TAGS`.
|
||||
- Postgres 17 image (default `postgres:17`, override via `SIM_PG_IMAGE`).
|
||||
- For k8s mode only: `k3d` and `helm` on `PATH`.
|
||||
- `nix develop` from the repo flake — provisions `minikube`, `kubectl`, `helm`,
|
||||
`pgbadger`, `deno`, and the `wm_sim` wrapper. **No host config required
|
||||
beyond the flake** with one exception: if you want multi-node KVM clusters
|
||||
on NixOS, libvirtd needs to be enabled at the system level — see
|
||||
`sim/nix/kvm2-driver.nix`.
|
||||
- Root + libvirtd for the kvm2 driver. Single-node clusters work with the
|
||||
qemu2 driver without root, but multi-node needs kvm2.
|
||||
- A clone of `windmill-helm-charts` next to this repo (`../windmill-helm-charts`).
|
||||
The vendored chart was deleted in favor of using the upstream chart with
|
||||
bench-specific overlays in `values/smoke.yaml`. The chart depends on a few
|
||||
knobs (PriorityClass support, `oomImmune`, `maxConnections`) that may not
|
||||
be in upstream yet — see the windmill-helm-charts PR.
|
||||
|
||||
## Run
|
||||
## Topology + values layout
|
||||
|
||||
```sh
|
||||
cd benchmarks/sim
|
||||
deno run -A sim.ts -c configs/baseline.json
|
||||
```
|
||||
sim/
|
||||
topologies/
|
||||
k8s-4node.json # 1 control plane + 3 workers (4 vCPU, 20 GiB each)
|
||||
k8s-3node.json # control plane + 2 workers
|
||||
... # several other shapes
|
||||
values/
|
||||
smoke.yaml # bench-tuned helm overlay (worker reqs, PG sizing,
|
||||
# priorityClass=wm-critical, maxConnections, etc.)
|
||||
local.example.yaml # template — copy to local.yaml, fill in your
|
||||
# EE license key. local.yaml is gitignored.
|
||||
```
|
||||
|
||||
Useful flags:
|
||||
`smoke.yaml` is the source of truth for chart config across benches. Tune
|
||||
worker replica counts, PG memory, max_connections, etc. there.
|
||||
|
||||
- `--factor N` — multiply every benchmark's `jobs` count by N.
|
||||
- `--keep-infra` — leave containers running after the run.
|
||||
- `--snapshot-interval <ms>` — queue-depth sampling cadence (default 1000).
|
||||
## Architecture
|
||||
|
||||
## Outputs
|
||||
```
|
||||
wm_sim up
|
||||
│
|
||||
▼ provision
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ k8s_provisioner: minikube start + add-nodes + per-node sizing │
|
||||
│ image_cache: pre-load images so bringup is offline-safe │
|
||||
│ helm_deploy: helm upgrade --install windmill via smoke.yaml│
|
||||
│ toxiproxy_k8s: per-node toxiproxy DaemonSet for latency inj │
|
||||
│ cpu_sampler_k8s: per-node sampler DaemonSet (10Hz cpu.stat) │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ benchmarks/main.ts (separate shell)
|
||||
┌────────────────────────────────────────────────────────────────┐
|
||||
│ readiness: wait for samplers stable, workers Ready, │
|
||||
│ PG responsive, queue empty, │
|
||||
│ *deployment rollout complete* — mid-rollout │
|
||||
│ pod churn starves the sampler under │
|
||||
│ cgroup_mutex contention │
|
||||
│ start pollers: pod_timeline, oom, pg_latency, pg_conn, │
|
||||
│ node_load (all write JSONL into report dir) │
|
||||
│ truncate sampler /var/log/wm-sim-cpu-sampler/sampler.tsv on │
|
||||
│ host log: each node so this run gets a clean file │
|
||||
│ push jobs: pushers → API → workers pull → PG queue │
|
||||
│ stop pollers: finalize JSONL │
|
||||
│ collect samples: scp host log from each node (fallback: │
|
||||
│ kubectl logs) │
|
||||
│ collect pg.log: kubectl logs PG pod → pgBadger HTML report │
|
||||
│ render_report: JSONL + samples → dashboard.svg + report.md │
|
||||
└────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ output: benchmarks/reports/<ISO timestamp>/
|
||||
```
|
||||
|
||||
Each run writes two files in the working directory:
|
||||
## Measurement subsystem
|
||||
|
||||
- `sim_results_<ts>.json` — raw structured data: workload results,
|
||||
per-worker stats, queue snapshots, derived throughput.
|
||||
- `sim_results_<ts>.md` — human-readable summary: throughput table,
|
||||
per-worker table sorted by jobs handled, **utilization spread** headline,
|
||||
ASCII sparkline of queue depth over time.
|
||||
| Signal | Source | Cadence | File in report |
|
||||
|---|---|---|---|
|
||||
| Per-pod CPU + memory | DS sampler reading cgroup cpu.stat / memory.current | 10 Hz | `cpu_samples.tsv` |
|
||||
| Workers Ready per node | `kubectl get pods` | 1 Hz | `pod_timeline.jsonl` |
|
||||
| OOM events | `kubectl get events` + node-kernel dmesg | live | `oom_events_live.jsonl` |
|
||||
| PG query latency | `psql -c "\timing on" -c "SELECT 1"` | 4 Hz | `pg_latency.jsonl` |
|
||||
| PG connections by state | `SELECT state, count(*) FROM pg_stat_activity` | 1 Hz | `pg_connections.jsonl` |
|
||||
| Node loadavg + procs_running | ssh `/proc/loadavg`, `/proc/stat` | 0.5 Hz | `node_load.jsonl` |
|
||||
| Throughput (jobs/s sent + processed) | bench worker `postMessage` | sub-second | `throughput_samples.json` |
|
||||
| Failed jobs | `/api/w/{ws}/jobs/completed/list` | post-bench | `failed_jobs.jsonl` |
|
||||
| Pod inventory + node placement | `kubectl get pods -o json` | post-bench | `pods.json` |
|
||||
| PG slow-query analysis | PG log → pgBadger | post-bench, background | `pgbadger.html` |
|
||||
|
||||
The headline metric for the question this sim is built to answer ("are
|
||||
workers balanced?") is the utilization spread line. A small spread (e.g.
|
||||
±5pp) means the queue is self-balancing. A large spread (e.g. close
|
||||
worker 95%, far worker 40%) is the smoking gun for uneven pull dynamics.
|
||||
Key reliability fixes (don't undo them):
|
||||
|
||||
## Resource validation
|
||||
- **Sampler host file**: the sampler dual-writes to stdout AND a hostPath log
|
||||
file. `kubectl logs --since-time=X` loses early data when kubelet rotates
|
||||
log files under heavy benches (m04 was missing the first 90-120s of every
|
||||
busy run). Collector now scp's the host file as primary source.
|
||||
- **Readiness rollout check**: `kubectl get deploy windmill-workers-default`
|
||||
is checked for `status.updatedReplicas == spec.replicas`. Mid-rolling-update
|
||||
pod churn floods the kernel's `cgroup_mutex`, starving every other process
|
||||
on the node — including the sampler. Pod-readiness alone isn't enough.
|
||||
- **Oversaturation uses procs_running**, not load1. loadavg includes D-state
|
||||
procs (PG backends waiting on disk I/O, kernel mutex contention) which has
|
||||
nothing to do with CPU starvation but inflates "oversaturation" by 5-10x.
|
||||
`procs_running` from `/proc/stat` is the runnable count.
|
||||
|
||||
`sim.ts` refuses to run if the topology's total CPU or memory exceeds the
|
||||
host's available budget after the `reserved` buffer. `host_resources` is
|
||||
optional in the topology — when omitted, the validator falls back to the
|
||||
live host's logical core count and `MemTotal`. Declare it explicitly when
|
||||
you want the topology to be portable to a specific target host.
|
||||
## Dashboard panels
|
||||
|
||||
Set `pin_cpus: true` on the topology to assign each container a disjoint
|
||||
`--cpuset-cpus` range. This eliminates CFS time-slicing between containers
|
||||
entirely; useful when two nodes' throughput numbers must not be
|
||||
artificially correlated.
|
||||
Rendered into `dashboard.svg`. Layout is roughly:
|
||||
|
||||
| Section | Panels |
|
||||
|---|---|
|
||||
| **Run context** | Topology, workload, helm values, bench cmd, report dir |
|
||||
| **Throughput** | Jobs sent + processed over time, with push-window shading and phase boundaries |
|
||||
| **Queue depth** | Pending jobs in `v2_job_queue` |
|
||||
| **Node memory** | Per-node memory time-series + per-phase average bar chart |
|
||||
| **PG latency** | Pure SQL time (psql `\timing`) + kubectl-exec roundtrip (separate axes) |
|
||||
| **PG connections** | active / idle / idle_in_xact over time |
|
||||
| **Util group** | One panel per node, 2 per row. Solid blue CPU util (capped 100%) on top of translucent orange oversaturation. 100% reference line. PG-host node flagged `[PG]` |
|
||||
| **Node CPU** | Multi-line per-node CPU, with PG-host node tinted via stroke + label |
|
||||
| **Pod inventory** | Bar chart of pod count by node and pod-type donut |
|
||||
| **Failed jobs** | Cumulative + categorized bars |
|
||||
| **Restart events** | Pod restarts detected from `pod_timeline.jsonl` |
|
||||
| **OOM events** | L0 scheduler preemptions, L1 kubelet evictions, L2 cgroup + node-kernel kills |
|
||||
|
||||
All time-series share an x-axis origin (`bench_start_ms` written into
|
||||
`meta.json`) so the same x-coordinate on every panel means the same wall-clock
|
||||
moment. Tick labels are relative seconds ("0s, 30s, …"), not wall-clock.
|
||||
|
||||
## Workloads
|
||||
|
||||
`benchmarks/workloads/*.json`. Each is a phased JSON spec the bench reads at
|
||||
startup. Phases run sequentially, each with its own pusher count, mode mix,
|
||||
duration range, and ram_mb distribution.
|
||||
|
||||
Examples:
|
||||
|
||||
| Workload | Shape | Use case |
|
||||
|---|---|---|
|
||||
| `io_4phase.json` | idle → 2.5s → 500ms → 150ms IO jobs | Steady-state vs saturation in one bench |
|
||||
| `io_150ms_flood.json` | 60 pushers, 150ms IO, 120s | Worker-host CFS saturation regime |
|
||||
| `io_300ms_flood.json` | 60 pushers, 300ms IO, 120s | Lower context-switch rate |
|
||||
| `io_1s_flood.json`, `io_2s_flood.json` | Same shape, longer jobs | Near-theoretical worker scaling |
|
||||
| `ops_day.json` | Mixed phases simulating a day | Realistic load profile |
|
||||
| `cpu_heavy.json`, `mixed.json`, `burst.json` | Non-IO and bursty shapes | CPU-bound and spikiness |
|
||||
|
||||
## Agent workers (HTTP+JWT path)
|
||||
|
||||
`smoke.yaml` declares a second worker group `agent` with `mode: agent`. Agent
|
||||
workers reach the windmill API over HTTP+JWT instead of holding a direct
|
||||
sqlx pool to PG. Useful for measuring how much of the bench cap is PG
|
||||
contention vs. the HTTP-mediated path.
|
||||
|
||||
The JWT lives in a K8s Secret `windmill-agent-token` (not in any committed
|
||||
yaml). Create it once:
|
||||
|
||||
```sh
|
||||
kubectl --context wm-sim-k8s-4node create secret generic windmill-agent-token \
|
||||
--from-literal=token='jwt_agent_…'
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
nix develop --command bash -c "cd benchmarks && deno test -A sim/*_test.ts"
|
||||
cd benchmarks && deno test --allow-import --no-check \
|
||||
sim/util_metrics_test.ts sim/util_panel_snapshot_test.ts
|
||||
```
|
||||
|
||||
38 unit tests cover the validator, config loader, proxy planner, schema
|
||||
modes, and report writer. The provisioner and Postgres collector are
|
||||
intentionally not unit-tested — they need real infrastructure to exercise.
|
||||
- `sim/util_metrics_test.ts` — 8 tests for `computeOversatPct`, the formula
|
||||
the util panel uses. Specifically guards the procs_running > load1 fallback,
|
||||
the clamp-at-zero, invalid-ncpu inputs.
|
||||
- `sim/util_panel_snapshot_test.ts` — 5 assertions on the rendered util-panel
|
||||
SVG: orange-behind-blue z-order, 100% reference line, relative-time ticks
|
||||
(no wall-clock leak), phase-boundary dashed verticals, shared-origin override.
|
||||
- `sim/topology_test.ts`, `sim/toxiproxy_test.ts` — earlier coverage of the
|
||||
validator + proxy planner.
|
||||
|
||||
## Roadmap
|
||||
Pre-existing typecheck errors in `graph.ts` (untyped d3 callback args,
|
||||
implicit `any`) trip `deno test` without `--no-check`. The functional tests
|
||||
pass regardless.
|
||||
|
||||
- **M1 ✅**: scaffold + single-node baseline + container provisioner +
|
||||
resource validator.
|
||||
- **M2 ✅**: multi-node + per-node toxiproxy latency.
|
||||
- **M3 ✅**: metrics collection from Postgres (per-worker utilization,
|
||||
queue depth over time).
|
||||
- **M4 ✅**: markdown + JSON report. SVG graphs deferred until needed.
|
||||
- **M5**: k3d-based k8s mode with helm install (upstream or local chart).
|
||||
- **M6+**: chaos timeline, assertion engine, resilience test corpus.
|
||||
## Operational notes
|
||||
|
||||
- Cluster name / minikube profile is `wm-sim-k8s-4node` by default. Pass
|
||||
`--minikube-profile <name>` to `main.ts` if you renamed it.
|
||||
- Default kubectl context comes from minikube. Use
|
||||
`kubectl --context wm-sim-k8s-4node …` to be explicit.
|
||||
- The port-forward dies easily under heavy bench load (kubelet drops the
|
||||
connection). If a bench fails partway with "connection refused", restart
|
||||
PF and re-fire.
|
||||
- `minikube stop` is safe — VM disks persist, etcd survives, helm releases
|
||||
+ Secrets all come back when you `minikube start`. `wm_sim up` does
|
||||
`minikube delete` first, so it's destructive — only use it for clean reprovisions.
|
||||
|
||||
## Code layout
|
||||
|
||||
```
|
||||
sim/
|
||||
sim.ts # wm_sim CLI entry (cliffy)
|
||||
k8s_provisioner.ts # minikube up + node sizing
|
||||
helm_deploy.ts # helm upgrade --install
|
||||
image_cache.ts # pre-load images so bringup is offline-safe
|
||||
cpu_sampler_k8s.ts # sampler DaemonSet + collector
|
||||
toxiproxy_k8s.ts # per-node toxiproxy DaemonSet
|
||||
pg_logging.ts # ALTER SYSTEM + SIGHUP for verbose PG logs
|
||||
pgbadger.ts # post-bench PG log → HTML
|
||||
readiness.ts # pre-bench cluster health check
|
||||
pod_timeline.ts # pod readiness poller
|
||||
pod_inventory.ts # post-bench pod inventory snapshot
|
||||
oom_poller.ts # live OOM event capture
|
||||
oom_events.ts # post-bench OOM event parsing
|
||||
pg_latency_poller.ts # 4Hz PG \timing on SELECT 1
|
||||
pg_conn_poller.ts # pg_stat_activity by state
|
||||
node_load_poller.ts # /proc/loadavg + /proc/stat procs_running
|
||||
failed_jobs.ts # post-bench failed-jobs fetch
|
||||
util_metrics.ts # pure helpers (oversat formula)
|
||||
render_report.ts # JSONL → SVG dashboard + report.md
|
||||
dashboard.ts # SVG layout primitives (grid, sections)
|
||||
svg_to_pdf.ts # dashboard.svg → dashboard.pdf (best-effort)
|
||||
topology.ts # topology JSON loader + validator
|
||||
values/ # helm overlay yamls
|
||||
topologies/ # topology JSON files
|
||||
nix/kvm2-driver.nix # kvm2 driver overlay for nixpkgs
|
||||
```
|
||||
|
||||
The bench runner itself is `benchmarks/main.ts` (one level up). It calls
|
||||
into `sim/` for measurement + report rendering.
|
||||
|
||||
Reference in New Issue
Block a user