Files
windmill/benchmarks/sim/pod_inventory.ts
pyranota 74b662d8de feat(benchmarks): k8s sim mode + util-group dashboard + reliability fixes
Stand up a minikube-backed simulation subsystem for benching Windmill under
realistic multi-node load, with a per-bench measurement pipeline and a
dashboard renderer that consolidates throughput, queue depth, per-node CPU,
PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into
one SVG report.

Sim infrastructure (sim/):
- k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON
- helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays
- image_cache: pre-load required images so bench bringup is offline-safe
- toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection
- cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual-
  writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/
  sampler.tsv) so heavy benches no longer lose early samples to kubelet log
  rotation
- pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart
- pgbadger: post-bench PG log analysis HTML report
- readiness: pre-bench cluster health check (samplers stable ≥30s, workers
  ready, PG responsive, queue empty, **deploy.status rollout-complete**) —
  the rollout-complete check catches mid-rolling-update fires that previously
  starved m04's sampler under cgroup_mutex contention

Per-bench JSONL pollers, started/finalized alongside the bench loop:
- pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel)
- oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup)
- pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip
- pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact)
- node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node

Dashboard renderer (sim/render_report.ts + graph.ts):
- Util group: one panel per node with translucent orange oversaturation area
  BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals.
  cols:2 grid wraps after 2 panels per row.
- PG node tinted with [PG] flag in legend across the dashboard.
- Phase-boundary verticals + push-window shaded zones layered consistently.
- All x-axes switched from wall-clock HH:MM to relative seconds-from-bench-
  start. Shared origin sourced from meta.json's bench_start_ms so 0s on every
  panel = the same wall-clock moment (previously each chart picked its own
  earliest sample as origin, causing drift between panels).

Oversaturation metric, with explicit fallback:
- Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure.
- Fallback to load1 when procs_running is missing (older reports).
- load1 overcounted previously because it includes uninterruptible D-state
  procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation"
  by 5-10x under load.
- Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the
  procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases.

Sampler reliability:
- HostPath log file in addition to stdout so the bench's scp-based collector
  bypasses kubelet log rotation entirely.
- main.ts truncates the host log file on every node before pushers start
  (parallel ssh, best-effort) so it doesn't grow unbounded across runs.
- Collector falls back to kubectl-logs when scp fails for any node.

Workloads (workloads/):
- io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs)
- io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase
  flood configs to isolate the worker-host CFS context-switch storm vs PG
  contention regime
- burst, ops_day, cpu_*, etc. for other scenarios

Tests:
- sim/util_metrics_test.ts — 8 cases for computeOversatPct
- sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG
  invariants (orange behind blue, 100% ref line, relative-time ticks NOT
  wall-clock, phase-boundary verticals, shared-origin override)

Helm values:
- sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem
  request, PG w/ 3-core request + wm-critical priorityClass + oomImmune +
  maxConnections, app w/ wm-critical + oomImmune + no resource limits.
- sim/values/local.example.yaml — template for the gitignored local.yaml
  that carries the EE license key.
- Depends on the wm-critical PriorityClass + oomImmune + maxConnections
  knobs landing in windmill-helm-charts (separate PR).

graph.ts additions:
- areaFills param: ordered list of per-kind translucent area fills drawn
  before lines, used by the util panel for orange-behind-blue layering
- lineColorOverrides: pin per-kind line colors so oversaturation reliably
  renders orange regardless of d3 ordinal-color insertion order
- highlightKindToken: substring-match flag for the PG-node tint in Node CPU
- xRelativeOriginMs: shared bench-start origin for the relative-time x-axis
- DataPointMulti is now exported for downstream tests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-08 11:43:47 +02:00

87 lines
2.8 KiB
TypeScript

// Snapshot of pods alive on the cluster at teardown — written as JSON so the
// report renderer can map pod UIDs (the only thing the cgroup-based CPU
// sampler can extract) to human-meaningful names + nodes + the worker-group
// label.
//
// Shape (one entry per pod):
// { uid, name, namespace, node, labels: {...} }
import type { MinikubeProvisioner } from "./k8s_provisioner.ts";
export type PodEntry = {
uid: string;
name: string;
namespace: string;
node: string;
labels: Record<string, string>;
// Captured so the renderer can show "what died during the bench" without
// relying on the event-based oom_events.json (which misses containerd's
// mislabeled "Error" exit 137 kills, dmesg-source flakiness, etc.).
containers?: Array<{
name: string;
restartCount: number;
lastState?: {
terminated?: {
reason?: string;
exitCode?: number;
finishedAt?: string;
};
};
}>;
};
export async function capturePodInventory(
prov: MinikubeProvisioner,
outPath: string,
): Promise<PodEntry[]> {
const res = await prov.kubectl([
"get", "pods", "-A",
"-o", "json",
]);
if (res.code !== 0) {
console.warn(`[inventory] kubectl get pods failed`);
return [];
}
let parsed: { items?: Array<Record<string, unknown>> };
try {
parsed = JSON.parse(res.stdout);
} catch (e) {
console.warn(`[inventory] JSON parse failed: ${(e as Error).message}`);
return [];
}
const entries: PodEntry[] = (parsed.items ?? []).map((p) => {
const meta = (p.metadata as Record<string, unknown> | undefined) ?? {};
const spec = (p.spec as Record<string, unknown> | undefined) ?? {};
const status = (p.status as Record<string, unknown> | undefined) ?? {};
const rawCs = (status.containerStatuses as Array<Record<string, unknown>> | undefined) ?? [];
const containers = rawCs.map((cs) => {
const ls = (cs.lastState as Record<string, unknown> | undefined) ?? {};
const term = (ls.terminated as Record<string, unknown> | undefined);
return {
name: String(cs.name ?? ""),
restartCount: Number(cs.restartCount ?? 0),
lastState: term
? {
terminated: {
reason: term.reason as string | undefined,
exitCode: term.exitCode as number | undefined,
finishedAt: term.finishedAt as string | undefined,
},
}
: undefined,
};
});
return {
uid: String(meta.uid ?? ""),
name: String(meta.name ?? ""),
namespace: String(meta.namespace ?? ""),
node: String(spec.nodeName ?? ""),
labels: (meta.labels as Record<string, string> | undefined) ?? {},
containers,
};
});
await Deno.writeTextFile(outPath, JSON.stringify(entries, null, 2));
console.log(`[inventory] ${entries.length} pods captured -> ${outPath}`);
return entries;
}