mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-19 08:01:25 +00:00
pyranota
74b662d8de
Stand up a minikube-backed simulation subsystem for benching Windmill under realistic multi-node load, with a per-bench measurement pipeline and a dashboard renderer that consolidates throughput, queue depth, per-node CPU, PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into one SVG report. Sim infrastructure (sim/): - k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON - helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays - image_cache: pre-load required images so bench bringup is offline-safe - toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection - cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual- writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/ sampler.tsv) so heavy benches no longer lose early samples to kubelet log rotation - pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart - pgbadger: post-bench PG log analysis HTML report - readiness: pre-bench cluster health check (samplers stable ≥30s, workers ready, PG responsive, queue empty, **deploy.status rollout-complete**) — the rollout-complete check catches mid-rolling-update fires that previously starved m04's sampler under cgroup_mutex contention Per-bench JSONL pollers, started/finalized alongside the bench loop: - pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel) - oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup) - pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip - pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact) - node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node Dashboard renderer (sim/render_report.ts + graph.ts): - Util group: one panel per node with translucent orange oversaturation area BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals. cols:2 grid wraps after 2 panels per row. - PG node tinted with [PG] flag in legend across the dashboard. - Phase-boundary verticals + push-window shaded zones layered consistently. - All x-axes switched from wall-clock HH:MM to relative seconds-from-bench- start. Shared origin sourced from meta.json's bench_start_ms so 0s on every panel = the same wall-clock moment (previously each chart picked its own earliest sample as origin, causing drift between panels). Oversaturation metric, with explicit fallback: - Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure. - Fallback to load1 when procs_running is missing (older reports). - load1 overcounted previously because it includes uninterruptible D-state procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation" by 5-10x under load. - Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases. Sampler reliability: - HostPath log file in addition to stdout so the bench's scp-based collector bypasses kubelet log rotation entirely. - main.ts truncates the host log file on every node before pushers start (parallel ssh, best-effort) so it doesn't grow unbounded across runs. - Collector falls back to kubectl-logs when scp fails for any node. Workloads (workloads/): - io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs) - io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase flood configs to isolate the worker-host CFS context-switch storm vs PG contention regime - burst, ops_day, cpu_*, etc. for other scenarios Tests: - sim/util_metrics_test.ts — 8 cases for computeOversatPct - sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG invariants (orange behind blue, 100% ref line, relative-time ticks NOT wall-clock, phase-boundary verticals, shared-origin override) Helm values: - sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem request, PG w/ 3-core request + wm-critical priorityClass + oomImmune + maxConnections, app w/ wm-critical + oomImmune + no resource limits. - sim/values/local.example.yaml — template for the gitignored local.yaml that carries the EE license key. - Depends on the wm-critical PriorityClass + oomImmune + maxConnections knobs landing in windmill-helm-charts (separate PR). graph.ts additions: - areaFills param: ordered list of per-kind translucent area fills drawn before lines, used by the util panel for orange-behind-blue layering - lineColorOverrides: pin per-kind line colors so oversaturation reliably renders orange regardless of d3 ordinal-color insertion order - highlightKindToken: substring-match flag for the PG-node tint in Node CPU - xRelativeOriginMs: shared bench-start origin for the relative-time x-axis - DataPointMulti is now exported for downstream tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
128 lines
3.3 KiB
JSON
128 lines
3.3 KiB
JSON
{
|
|
"nodes": {
|
|
"flake-utils": {
|
|
"inputs": {
|
|
"systems": "systems"
|
|
},
|
|
"locked": {
|
|
"lastModified": 1731533236,
|
|
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
|
|
"owner": "numtide",
|
|
"repo": "flake-utils",
|
|
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "numtide",
|
|
"repo": "flake-utils",
|
|
"type": "github"
|
|
}
|
|
},
|
|
"nixpkgs": {
|
|
"locked": {
|
|
"lastModified": 1764517877,
|
|
"narHash": "sha256-pp3uT4hHijIC8JUK5MEqeAWmParJrgBVzHLNfJDZxg4=",
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "2d293cbfa5a793b4c50d17c05ef9e385b90edf6c",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"id": "nixpkgs",
|
|
"ref": "nixos-unstable",
|
|
"type": "indirect"
|
|
}
|
|
},
|
|
"nixpkgs-oapi-gen": {
|
|
"locked": {
|
|
"lastModified": 1740303746,
|
|
"narHash": "sha256-XcdiWLEhjJkMxDLKQJ0CCivmYYCvA5MDxu9pMybM5kM=",
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "2d068ae5c6516b2d04562de50a58c682540de9bf",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"id": "nixpkgs",
|
|
"rev": "2d068ae5c6516b2d04562de50a58c682540de9bf",
|
|
"type": "indirect"
|
|
}
|
|
},
|
|
"nixpkgs-sim": {
|
|
"locked": {
|
|
"lastModified": 1779560665,
|
|
"narHash": "sha256-tpyBcxPpcQb8ukyNF7DoCwfSY3VPsxHoYwj00Cayv5o=",
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "64c08a7ca051951c8eae34e3e3cb1e202fe36786",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"id": "nixpkgs",
|
|
"rev": "64c08a7ca051951c8eae34e3e3cb1e202fe36786",
|
|
"type": "indirect"
|
|
}
|
|
},
|
|
"nixpkgs_2": {
|
|
"locked": {
|
|
"lastModified": 1744536153,
|
|
"narHash": "sha256-awS2zRgF4uTwrOKwwiJcByDzDOdo3Q1rPZbiHQg/N38=",
|
|
"owner": "NixOS",
|
|
"repo": "nixpkgs",
|
|
"rev": "18dd725c29603f582cf1900e0d25f9f1063dbf11",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "NixOS",
|
|
"ref": "nixpkgs-unstable",
|
|
"repo": "nixpkgs",
|
|
"type": "github"
|
|
}
|
|
},
|
|
"root": {
|
|
"inputs": {
|
|
"flake-utils": "flake-utils",
|
|
"nixpkgs": "nixpkgs",
|
|
"nixpkgs-oapi-gen": "nixpkgs-oapi-gen",
|
|
"nixpkgs-sim": "nixpkgs-sim",
|
|
"rust-overlay": "rust-overlay"
|
|
}
|
|
},
|
|
"rust-overlay": {
|
|
"inputs": {
|
|
"nixpkgs": "nixpkgs_2"
|
|
},
|
|
"locked": {
|
|
"lastModified": 1764643237,
|
|
"narHash": "sha256-6Ezx9DqVv5UZ7DBK9rcNwBuQUENFyWPS7M09I+FvNao=",
|
|
"owner": "oxalica",
|
|
"repo": "rust-overlay",
|
|
"rev": "e66d6b924ac59e6c722f69332f6540ea57c69233",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "oxalica",
|
|
"repo": "rust-overlay",
|
|
"type": "github"
|
|
}
|
|
},
|
|
"systems": {
|
|
"locked": {
|
|
"lastModified": 1681028828,
|
|
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
|
|
"owner": "nix-systems",
|
|
"repo": "default",
|
|
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
|
|
"type": "github"
|
|
},
|
|
"original": {
|
|
"owner": "nix-systems",
|
|
"repo": "default",
|
|
"type": "github"
|
|
}
|
|
}
|
|
},
|
|
"root": "root",
|
|
"version": 7
|
|
}
|