mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-19 08:03:16 +00:00
pyranotaandClaude Opus 4.7
4356e60eff
sim/README.md was last touched May 27 and described the pre-minikube docker/podman-based "three node modes" design with k8s mode listed as "planned for M5". The reality is now k8s-only via minikube + helm, and the workflow / config-files / outputs all changed. New sim/README.md covers: - The actual `wm_sim up` CLI form against the external windmill-helm-charts chart and smoke.yaml / local.yaml overlays. - Two-shell workflow: wm_sim up in one, port-forward + main.ts in the other, report at reports/<timestamp>/dashboard.svg. - Measurement subsystem table: per-pod CPU/mem sampler DS, JSONL pollers (pod_timeline, oom, pg_latency, pg_conn, node_load), throughput capture, failed-jobs fetch, pgBadger. - The three reliability fixes we leaned on this session and why they matter: sampler host-log file (kubelet log rotation), readiness rollout-complete check (cgroup_mutex starvation), procs_running for oversaturation (not load1 / D-state overcount). - Dashboard panel index + the shared bench_start_ms x-axis origin. - Workload catalog (io_4phase + flood variants + ops_day + cpu). - Agent worker setup with the Secret-based JWT. - Test entry points (util_metrics_test, util_panel_snapshot_test). - Operational notes (port-forward fragility, minikube stop safety). - File-by-file code layout under sim/. benchmarks/README.md gains a "Cluster benchmarks — sim mode" section linking to sim/README.md plus a quick-start snippet so the path from the legacy benchmark suite docs into the cluster workflow is obvious. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Benchmarks
Deno/TS benchmark suite for measuring Windmill job and flow execution throughput.
Quick Start
# Install Deno
curl -fsSL https://deno.land/install.sh | sh
# Run a single benchmark
deno run -A benchmark_oneoff.ts --kind noop --jobs 10000
# Run the full suite
deno run -A benchmark_suite.ts -c suite_config.json
# Run WAC v2 benchmarks (workflow-as-code vs flow comparison)
deno run -A benchmark_suite.ts -c suite_wac.json
Benchmark Kinds
Script benchmarks
noop— Empty jobs (measures pure scheduling overhead)deno,bun,python,go,bash— Language runtimesnativets— BunNative (no isolation)dedicated,dedicated_nativets— Dedicated worker mode
Flow benchmarks
2steps— 2-step flow (deno + identity)bigscriptinflow— Flow with large raw bash scriptflow_seq_2_bun— 2 sequential bun stepsflow_par_2_bun— 2 parallel bun steps (branchall)flow_seq_3_bun— 3 sequential bun stepsflow:<path>— Custom flow by pathscript:<path>— Custom script by path
WAC v2 benchmarks (workflow-as-code)
wac_seq_2— 2 sequential taskswac_par_2— 2 parallel tasks (Promise.all)wac_seq_3— 3 sequential taskswac_inline_2— 2 inline steps (no child jobs)
Suite Configs
| File | Description |
|---|---|
suite_config.json |
Main benchmark suite (noop, languages, flows) |
suite_dedicated.json |
Dedicated worker benchmarks |
suite_dedicated_nativets.json |
Dedicated NativeTS benchmarks |
suite_wac.json |
WAC v2 vs flow comparison benchmarks |
Interactive Benchmark Tool
deno run -A main.ts -e admin@windmill.dev -p changeme --host http://localhost:8000
Options: --workers, --seconds, --maximum-throughput, --use-flows, --script-pattern, --export-json, --export-csv
Graph Generation
deno run -A benchmark_graphs.ts -c graphs_config.json
Generates SVG graphs from *_benchmark.json data files.
CI
The GitHub Actions workflow (.github/workflows/benchmark.yml) runs hourly with 1/4/8 worker configurations plus WAC benchmarks. Results are committed to the benchmarks branch.
Cluster benchmarks — sim mode
Provisioning + measuring throughput on a real multi-node Kubernetes cluster
(minikube + helm + per-pod cgroup sampling + PG analysis) is a separate
workflow on top of main.ts. See sim/README.md for:
- bringing up the cluster (
wm_sim up) - firing a phased or flood workload from
workloads/ - the consolidated dashboard with throughput, queue depth, per-node CPU + oversaturation, PG latency / conn counts, OOM events, restart events
- the JSONL poller pipeline + reliability fixes (sampler host-log, rollout- complete readiness, procs_running oversaturation)
Quick path:
# Provision cluster + deploy Windmill (foreground; Ctrl-C tears down):
wm_sim up \
--topology sim/topologies/k8s-4node.json \
--helm ../windmill-helm-charts/charts/windmill \
--helm-values sim/values/smoke.yaml \
--helm-values sim/values/local.yaml
# In another shell — port-forward + run bench:
kubectl --context wm-sim-k8s-4node port-forward --address 0.0.0.0 \
-n default svc/windmill-app 33405:8000 &
deno run -A main.ts \
--host http://127.0.0.1:33405 \
--token <admin-token> \
--workload-config workloads/io_150ms_flood.json \
--minikube-profile wm-sim-k8s-4node \
--wait-ready 60
# Report appears at reports/<timestamp>/dashboard.svg