mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-19 00:02:03 +00:00

Files

T

pyranota c5f4028a2c docs(benchmarks): fix README — port-forward is wm_sim's job

Earlier README told users to run a second `kubectl port-forward` and
hard-coded port 33405. Both wrong: wm_sim up calls portForwardApi() in
helm_deploy.ts:213, picks a free local port, prints `[helm] API reachable
at http://127.0.0.1:<port>`, and keeps the forward alive as a child of
the wm_sim process.

Updated workflow: read the URL out of wm_sim's output, pass it to main.ts
as --host. Kept a note about restarting the forward manually when the
kubelet drops it under heavy bench load (real failure mode we hit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-06-08 11:58:18 +02:00

3.5 KiB

Raw Permalink Blame History

Benchmarks

Deno/TS benchmark suite for measuring Windmill job and flow execution throughput.

Quick Start

# Install Deno
curl -fsSL https://deno.land/install.sh | sh

# Run a single benchmark
deno run -A benchmark_oneoff.ts --kind noop --jobs 10000

# Run the full suite
deno run -A benchmark_suite.ts -c suite_config.json

# Run WAC v2 benchmarks (workflow-as-code vs flow comparison)
deno run -A benchmark_suite.ts -c suite_wac.json

Benchmark Kinds

Script benchmarks

noop — Empty jobs (measures pure scheduling overhead)
deno, bun, python, go, bash — Language runtimes
nativets — BunNative (no isolation)
dedicated, dedicated_nativets — Dedicated worker mode

Flow benchmarks

2steps — 2-step flow (deno + identity)
bigscriptinflow — Flow with large raw bash script
flow_seq_2_bun — 2 sequential bun steps
flow_par_2_bun — 2 parallel bun steps (branchall)
flow_seq_3_bun — 3 sequential bun steps
flow:<path> — Custom flow by path
script:<path> — Custom script by path

WAC v2 benchmarks (workflow-as-code)

wac_seq_2 — 2 sequential tasks
wac_par_2 — 2 parallel tasks (Promise.all)
wac_seq_3 — 3 sequential tasks
wac_inline_2 — 2 inline steps (no child jobs)

Suite Configs

File	Description
`suite_config.json`	Main benchmark suite (noop, languages, flows)
`suite_dedicated.json`	Dedicated worker benchmarks
`suite_dedicated_nativets.json`	Dedicated NativeTS benchmarks
`suite_wac.json`	WAC v2 vs flow comparison benchmarks

Interactive Benchmark Tool

deno run -A main.ts -e admin@windmill.dev -p changeme --host http://localhost:8000

Options: --workers, --seconds, --maximum-throughput, --use-flows, --script-pattern, --export-json, --export-csv

Graph Generation

deno run -A benchmark_graphs.ts -c graphs_config.json

Generates SVG graphs from *_benchmark.json data files.

CI

The GitHub Actions workflow (.github/workflows/benchmark.yml) runs hourly with 1/4/8 worker configurations plus WAC benchmarks. Results are committed to the benchmarks branch.

Cluster benchmarks — sim mode

Provisioning + measuring throughput on a real multi-node Kubernetes cluster (minikube + helm + per-pod cgroup sampling + PG analysis) is a separate workflow on top of main.ts. See sim/README.md for:

bringing up the cluster (wm_sim up)
firing a phased or flood workload from workloads/
the consolidated dashboard with throughput, queue depth, per-node CPU + oversaturation, PG latency / conn counts, OOM events, restart events
the JSONL poller pipeline + reliability fixes (sampler host-log, rollout- complete readiness, procs_running oversaturation)

Quick path:

# Provision cluster + deploy Windmill (foreground; Ctrl-C tears down).
# Prints `[helm] API reachable at http://127.0.0.1:<port>` — note the port.
# Port-forward stays alive as a child of wm_sim, so keep this terminal open.
wm_sim up \
  --topology    sim/topologies/k8s-4node.json \
  --helm        ../windmill-helm-charts/charts/windmill \
  --helm-values sim/values/smoke.yaml \
  --helm-values sim/values/local.yaml

# In another shell — fire bench against the wm_sim-managed port:
deno run -A main.ts \
  --host http://127.0.0.1:<port> \
  --token <admin-token> \
  --workload-config workloads/io_150ms_flood.json \
  --minikube-profile wm-sim-k8s-4node \
  --wait-ready 60

# Report appears at reports/<timestamp>/dashboard.svg

3.5 KiB Raw Permalink Blame History