Stand up a minikube-backed simulation subsystem for benching Windmill under
realistic multi-node load, with a per-bench measurement pipeline and a
dashboard renderer that consolidates throughput, queue depth, per-node CPU,
PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into
one SVG report.
Sim infrastructure (sim/):
- k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON
- helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays
- image_cache: pre-load required images so bench bringup is offline-safe
- toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection
- cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual-
writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/
sampler.tsv) so heavy benches no longer lose early samples to kubelet log
rotation
- pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart
- pgbadger: post-bench PG log analysis HTML report
- readiness: pre-bench cluster health check (samplers stable ≥30s, workers
ready, PG responsive, queue empty, **deploy.status rollout-complete**) —
the rollout-complete check catches mid-rolling-update fires that previously
starved m04's sampler under cgroup_mutex contention
Per-bench JSONL pollers, started/finalized alongside the bench loop:
- pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel)
- oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup)
- pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip
- pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact)
- node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node
Dashboard renderer (sim/render_report.ts + graph.ts):
- Util group: one panel per node with translucent orange oversaturation area
BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals.
cols:2 grid wraps after 2 panels per row.
- PG node tinted with [PG] flag in legend across the dashboard.
- Phase-boundary verticals + push-window shaded zones layered consistently.
- All x-axes switched from wall-clock HH:MM to relative seconds-from-bench-
start. Shared origin sourced from meta.json's bench_start_ms so 0s on every
panel = the same wall-clock moment (previously each chart picked its own
earliest sample as origin, causing drift between panels).
Oversaturation metric, with explicit fallback:
- Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure.
- Fallback to load1 when procs_running is missing (older reports).
- load1 overcounted previously because it includes uninterruptible D-state
procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation"
by 5-10x under load.
- Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the
procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases.
Sampler reliability:
- HostPath log file in addition to stdout so the bench's scp-based collector
bypasses kubelet log rotation entirely.
- main.ts truncates the host log file on every node before pushers start
(parallel ssh, best-effort) so it doesn't grow unbounded across runs.
- Collector falls back to kubectl-logs when scp fails for any node.
Workloads (workloads/):
- io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs)
- io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase
flood configs to isolate the worker-host CFS context-switch storm vs PG
contention regime
- burst, ops_day, cpu_*, etc. for other scenarios
Tests:
- sim/util_metrics_test.ts — 8 cases for computeOversatPct
- sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG
invariants (orange behind blue, 100% ref line, relative-time ticks NOT
wall-clock, phase-boundary verticals, shared-origin override)
Helm values:
- sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem
request, PG w/ 3-core request + wm-critical priorityClass + oomImmune +
maxConnections, app w/ wm-critical + oomImmune + no resource limits.
- sim/values/local.example.yaml — template for the gitignored local.yaml
that carries the EE license key.
- Depends on the wm-critical PriorityClass + oomImmune + maxConnections
knobs landing in windmill-helm-charts (separate PR).
graph.ts additions:
- areaFills param: ordered list of per-kind translucent area fills drawn
before lines, used by the util panel for orange-behind-blue layering
- lineColorOverrides: pin per-kind line colors so oversaturation reliably
renders orange regardless of d3 ordinal-color insertion order
- highlightKindToken: substring-match flag for the PG-node tint in Node CPU
- xRelativeOriginMs: shared bench-start origin for the relative-time x-axis
- DataPointMulti is now exported for downstream tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In > 7.10.0 regression was introduced breaking our build process.
This commit reverts the change and fixes tests
Signed-off-by: pyranota <pyra@duck.com>
* feat: add nu (nushell) support
* add worker tests
* deactivate tables and non-any types below top-level
full support will come in V1
for V0 it's better to keep things minimal and simple
* add syntax highlighting
used python's grammar, since nushell isn't supported by monaco nor svelte-highlights
for V1 nu will get it`s own grammar
* add logo
* partially implement plugin support
* change logo + ability to deploy + nsjail draft
* static variables + get_resource + get_variable
* lsp/dev.nu + initial nu lsp (not working yet)
* make it work with nsjail
* nullguard
* Much more flexible signature parsing and better error-messages
* add init script
* rename nulsp to nu
* install nu to dockerfile
* fix merge
* implement Default for MainArgSignature
* stage NU_CACHE_DIR
* improve dockerfiles
* dev.nu for parser-wasm + flake.nix
* update code for windows
* add nushell to flake
* upload Cargo.lock
* make build.sh work on nixos
* build wasm cli parsers
* add docs to README_DEV.md
* add helper script docker/dev.nu
* improve docker/dev.nu
* fix windows
* commit frontend/package(lock).json
* update cargo.lock
* correctly update cargo.lock
* remove lsp
* update flake.nix to include svelte server and nushell
* Revert base.sql to main
* remove PLUGIN_USE_RE
* make CARGO_PATH private
* add nu to cli
* Change flags to build wasm-nu-parser
* remove flake.nix from parser-wasm
* update wasm-build target
* remove unused import
* add cli support for nu
* update github workflows
* wasm-build 0.17 -> 0.19
* update build script
* update cargo.lock
* Fix typographical error
* update Cargo.lock
* update ENV_SETTINGS
* use published nu parser
* update package.lock
* rewrite parser in tree-sitter
* implement parser from scratch
* polishing
* change init script to match new parser
* fix imports
* fix cli build
* fix cli build
* merge
* update wasm
* use MiniPulledJob
* update cli
* change cli wasm schema
* change cli
* update deno.json
* make wasm modules load lazily
* regenerate parsers
* remove leftover
* update cargo.lock
* clean up dnt.ts
* add docs to cli/test.nu
* add schema validation option
* add Nu to try_validate_schema
* reference frontend to new parser version
pyranota
centdix
wendrul
pyranota
Lucas Abel