Files
windmill/benchmarks/sim/pgbadger.ts
pyranota 74b662d8de feat(benchmarks): k8s sim mode + util-group dashboard + reliability fixes
Stand up a minikube-backed simulation subsystem for benching Windmill under
realistic multi-node load, with a per-bench measurement pipeline and a
dashboard renderer that consolidates throughput, queue depth, per-node CPU,
PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into
one SVG report.

Sim infrastructure (sim/):
- k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON
- helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays
- image_cache: pre-load required images so bench bringup is offline-safe
- toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection
- cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual-
  writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/
  sampler.tsv) so heavy benches no longer lose early samples to kubelet log
  rotation
- pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart
- pgbadger: post-bench PG log analysis HTML report
- readiness: pre-bench cluster health check (samplers stable ≥30s, workers
  ready, PG responsive, queue empty, **deploy.status rollout-complete**) —
  the rollout-complete check catches mid-rolling-update fires that previously
  starved m04's sampler under cgroup_mutex contention

Per-bench JSONL pollers, started/finalized alongside the bench loop:
- pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel)
- oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup)
- pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip
- pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact)
- node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node

Dashboard renderer (sim/render_report.ts + graph.ts):
- Util group: one panel per node with translucent orange oversaturation area
  BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals.
  cols:2 grid wraps after 2 panels per row.
- PG node tinted with [PG] flag in legend across the dashboard.
- Phase-boundary verticals + push-window shaded zones layered consistently.
- All x-axes switched from wall-clock HH:MM to relative seconds-from-bench-
  start. Shared origin sourced from meta.json's bench_start_ms so 0s on every
  panel = the same wall-clock moment (previously each chart picked its own
  earliest sample as origin, causing drift between panels).

Oversaturation metric, with explicit fallback:
- Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure.
- Fallback to load1 when procs_running is missing (older reports).
- load1 overcounted previously because it includes uninterruptible D-state
  procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation"
  by 5-10x under load.
- Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the
  procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases.

Sampler reliability:
- HostPath log file in addition to stdout so the bench's scp-based collector
  bypasses kubelet log rotation entirely.
- main.ts truncates the host log file on every node before pushers start
  (parallel ssh, best-effort) so it doesn't grow unbounded across runs.
- Collector falls back to kubectl-logs when scp fails for any node.

Workloads (workloads/):
- io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs)
- io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase
  flood configs to isolate the worker-host CFS context-switch storm vs PG
  contention regime
- burst, ops_day, cpu_*, etc. for other scenarios

Tests:
- sim/util_metrics_test.ts — 8 cases for computeOversatPct
- sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG
  invariants (orange behind blue, 100% ref line, relative-time ticks NOT
  wall-clock, phase-boundary verticals, shared-origin override)

Helm values:
- sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem
  request, PG w/ 3-core request + wm-critical priorityClass + oomImmune +
  maxConnections, app w/ wm-critical + oomImmune + no resource limits.
- sim/values/local.example.yaml — template for the gitignored local.yaml
  that carries the EE license key.
- Depends on the wm-critical PriorityClass + oomImmune + maxConnections
  knobs landing in windmill-helm-charts (separate PR).

graph.ts additions:
- areaFills param: ordered list of per-kind translucent area fills drawn
  before lines, used by the util panel for orange-behind-blue layering
- lineColorOverrides: pin per-kind line colors so oversaturation reliably
  renders orange regardless of d3 ordinal-color insertion order
- highlightKindToken: substring-match flag for the PG-node tint in Node CPU
- xRelativeOriginMs: shared bench-start origin for the relative-time x-axis
- DataPointMulti is now exported for downstream tests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-08 11:43:47 +02:00

65 lines
2.5 KiB
TypeScript

// Reusable pgBadger integration: turn a PostgreSQL stderr log into a rich HTML
// report (top queries by time + call counts, lock waits, temp files,
// checkpoints, connection/session timeline, query-type distribution).
//
// Used two ways:
// 1. The sim (`sim.ts --pgbadger`) sets these PG log settings on the
// provisioned Postgres, captures its log after the run, and renders a
// per-run `pgbadger.html` into the results folder.
// 2. Standalone, against ANY PostgreSQL log (e.g. from a manual benchmark
// run against an external Windmill): enable PGBADGER_PG_SETTINGS on that
// PG, capture its log, then:
// deno run -A pgbadger.ts <pg.log> <out.html>
//
// pgBadger must be on PATH (it's in the flake devshell / `nix run nixpkgs#pgbadger`).
// Must stay in sync with PGBADGER_PG_SETTINGS' log_line_prefix below.
export const PGBADGER_LOG_PREFIX = "%t [%p]: user=%u,db=%d,app=%a,client=%h ";
// PostgreSQL settings that make the log pgBadger-parseable and rich. Pass each
// as a `-c name=value` to postgres. log_min_duration_statement=0 logs EVERY
// query with its duration (verbose — only enable when you want the report).
export const PGBADGER_PG_SETTINGS: string[] = [
"log_min_duration_statement=0",
"log_checkpoints=on",
"log_connections=on",
"log_disconnections=on",
"log_lock_waits=on",
"log_temp_files=0",
"log_autovacuum_min_duration=0",
"lc_messages=C", // pgBadger needs English log messages to parse them
`log_line_prefix=${PGBADGER_LOG_PREFIX}`,
];
// Render a pgBadger HTML report from a PostgreSQL log file. Best-effort: logs a
// warning and returns false on failure rather than throwing.
export async function runPgbadger(logPath: string, outHtml: string): Promise<boolean> {
try {
const p = new Deno.Command("pgbadger", {
args: ["--prefix", PGBADGER_LOG_PREFIX, "-f", "stderr", "-o", outHtml, logPath],
stdout: "piped",
stderr: "piped",
});
const { code, stderr } = await p.output();
if (code !== 0) {
console.warn(`[pgbadger] failed (${code}): ${new TextDecoder().decode(stderr).slice(0, 500)}`);
return false;
}
return true;
} catch (e) {
console.warn(`[pgbadger] could not run (is it on PATH?): ${(e as Error).message}`);
return false;
}
}
if (import.meta.main) {
const [log, out] = Deno.args;
if (!log || !out) {
console.error("usage: deno run -A pgbadger.ts <pg.log> <out.html>");
Deno.exit(2);
}
const ok = await runPgbadger(log, out);
if (ok) console.log(`pgBadger report written to ${out}`);
Deno.exit(ok ? 0 : 1);
}