Files
windmill/benchmarks/sim/dashboard.ts
pyranota 74b662d8de feat(benchmarks): k8s sim mode + util-group dashboard + reliability fixes
Stand up a minikube-backed simulation subsystem for benching Windmill under
realistic multi-node load, with a per-bench measurement pipeline and a
dashboard renderer that consolidates throughput, queue depth, per-node CPU,
PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into
one SVG report.

Sim infrastructure (sim/):
- k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON
- helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays
- image_cache: pre-load required images so bench bringup is offline-safe
- toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection
- cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual-
  writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/
  sampler.tsv) so heavy benches no longer lose early samples to kubelet log
  rotation
- pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart
- pgbadger: post-bench PG log analysis HTML report
- readiness: pre-bench cluster health check (samplers stable ≥30s, workers
  ready, PG responsive, queue empty, **deploy.status rollout-complete**) —
  the rollout-complete check catches mid-rolling-update fires that previously
  starved m04's sampler under cgroup_mutex contention

Per-bench JSONL pollers, started/finalized alongside the bench loop:
- pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel)
- oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup)
- pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip
- pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact)
- node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node

Dashboard renderer (sim/render_report.ts + graph.ts):
- Util group: one panel per node with translucent orange oversaturation area
  BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals.
  cols:2 grid wraps after 2 panels per row.
- PG node tinted with [PG] flag in legend across the dashboard.
- Phase-boundary verticals + push-window shaded zones layered consistently.
- All x-axes switched from wall-clock HH:MM to relative seconds-from-bench-
  start. Shared origin sourced from meta.json's bench_start_ms so 0s on every
  panel = the same wall-clock moment (previously each chart picked its own
  earliest sample as origin, causing drift between panels).

Oversaturation metric, with explicit fallback:
- Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure.
- Fallback to load1 when procs_running is missing (older reports).
- load1 overcounted previously because it includes uninterruptible D-state
  procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation"
  by 5-10x under load.
- Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the
  procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases.

Sampler reliability:
- HostPath log file in addition to stdout so the bench's scp-based collector
  bypasses kubelet log rotation entirely.
- main.ts truncates the host log file on every node before pushers start
  (parallel ssh, best-effort) so it doesn't grow unbounded across runs.
- Collector falls back to kubectl-logs when scp fails for any node.

Workloads (workloads/):
- io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs)
- io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase
  flood configs to isolate the worker-host CFS context-switch storm vs PG
  contention regime
- burst, ops_day, cpu_*, etc. for other scenarios

Tests:
- sim/util_metrics_test.ts — 8 cases for computeOversatPct
- sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG
  invariants (orange behind blue, 100% ref line, relative-time ticks NOT
  wall-clock, phase-boundary verticals, shared-origin override)

Helm values:
- sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem
  request, PG w/ 3-core request + wm-critical priorityClass + oomImmune +
  maxConnections, app w/ wm-critical + oomImmune + no resource limits.
- sim/values/local.example.yaml — template for the gitignored local.yaml
  that carries the EE license key.
- Depends on the wm-critical PriorityClass + oomImmune + maxConnections
  knobs landing in windmill-helm-charts (separate PR).

graph.ts additions:
- areaFills param: ordered list of per-kind translucent area fills drawn
  before lines, used by the util panel for orange-behind-blue layering
- lineColorOverrides: pin per-kind line colors so oversaturation reliably
  renders orange regardless of d3 ordinal-color insertion order
- highlightKindToken: substring-match flag for the PG-node tint in Node CPU
- xRelativeOriginMs: shared bench-start origin for the relative-time x-axis
- DataPointMulti is now exported for downstream tests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-08 11:43:47 +02:00

357 lines
16 KiB
TypeScript

// Stitches per-metric chart SVGs (rendered by graph.ts:drawGraphMulti) into a
// single dashboard SVG with a Windmill-branded header. No new chart rendering
// — we just position the existing SVG outputs inside one outer document.
//
// Brand applied to the wrapper only (background, fonts, header colors). The
// embedded charts keep their existing line-color cycle for now.
import { drawGraphMulti } from "../graph.ts";
// Multi-series data point — matches the shape graph.ts:drawGraphMulti consumes.
// Each entry is one (series, sample) point — `kind` groups points into lines.
export type DataPointMulti = {
value: number;
date: Date;
kind: string;
};
// Brand tokens, light mode. Extracted from frontend/brand-guidelines.md so we
// don't have to import the frontend Tailwind config.
const BRAND = {
surface_primary: "#fbfbfd",
surface_tertiary: "#ffffff",
border_light: "#e5e7eb",
text_emphasis: "#1d2430",
text_primary: "#3d4758",
text_secondary: "#718096",
text_hint: "#8d93a1",
font: "Inter, ui-sans-serif, system-ui, -apple-system, Segoe UI, sans-serif",
font_mono: "ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace",
};
export type DashboardMeta = {
topology: string;
suite: string;
generated: string; // ISO timestamp
walltime_s: number;
jobs_completed: number;
throughput_per_s: number;
// Free-form summary rows shown in the header. Use sentence case.
summary?: Array<{ label: string; value: string }>;
};
export type DashboardPanel = {
title: string;
yLabel: string;
// Time-series data — rendered via drawGraphMulti (one line per `kind`).
// Pass `[]` if you're providing a pre-rendered svg instead.
data: DataPointMulti[];
// Optional pre-rendered SVG (e.g. from drawBars for distributions). When
// present, used as-is instead of calling drawGraphMulti.
svg?: string;
// Optional dashed vertical lines drawn on the chart. Used by phased benches
// to mark phase boundaries on every time-series panel. Either Date[] (lines
// + labels) or { dates, hideLabels: true } to draw lines without P1>P2
// text — only the primary chart (Throughput) labels them; other charts
// get bare lines so the dashboard isn't visually noisy.
verticalLines?: Date[] | { dates: Date[]; hideLabels?: boolean };
// Optional dashed horizontal reference lines (e.g. CPU 100% ceiling).
horizontalLines?: { y: number; label: string }[];
// Optional translucent shaded rectangles spanning [from, to] on the x-axis
// (e.g. "push window"). Drawn UNDER the data lines.
shadedZones?: { from: Date; to: Date; fill?: string; label?: string }[];
// Optional phase grouping. Panels with the same `phaseGroup.index` are
// pulled out of the general 2-col grid and rendered together inside a
// boxed section labeled with `phaseGroup.label`. `cols` overrides the
// default 3-per-row layout for this section (e.g. Node memory uses 2).
phaseGroup?: { index: number; label: string; cols?: number };
// How many grid columns this panel spans. Default 1; the Node memory
// time-series uses 2 so it gets the full first row of its section.
colSpan?: number;
// Optional list of area fills drawn behind the lines (back→front order).
// Each entry tints the area under a series whose `kind` matches exactly.
// Used by the per-node util panels to layer oversaturation behind CPU util.
areaFills?: { kind: string; color: string; opacity?: number }[];
// Optional y-axis cap. When set, fixes the chart y-axis to [0, yMax]
// instead of letting drawGraphMulti auto-scale.
yMax?: number;
// Optional per-kind line color override (matches drawGraphMulti's param).
lineColorOverrides?: Record<string, string>;
};
// drawGraphMulti returns a full <svg>...</svg> document. To embed it inside
// our outer SVG we strip the outer tag and keep the inner content. The inner
// content is already wrapped in a transformed <g>, so we just need the
// dimensions to lay panels out.
function extractSvgInner(svg: string): { inner: string; width: number; height: number } {
const m = svg.match(/<svg[^>]*\swidth="(\d+)"[^>]*\sheight="(\d+)"[^>]*>([\s\S]*)<\/svg>\s*$/);
if (!m) {
return { inner: svg, width: 530, height: 250 };
}
return { inner: m[3], width: parseInt(m[1], 10), height: parseInt(m[2], 10) };
}
// Shared relative-time origin for the whole dashboard (epoch ms). Read from
// meta.json's bench_start_ms — used as the "0s" reference on every panel's
// x-axis so the same x position means the same moment across charts.
export type DashboardOptions = { xRelativeOriginMs?: number };
export function renderDashboard(
meta: DashboardMeta,
panels: DashboardPanel[],
opts: DashboardOptions = {},
): string {
const panelsWithData = panels.filter((p) => p.data.length > 0 || p.svg);
const PADDING = 40; // outer margin
const COLS = 2;
const COL_GAP = 48;
const ROW_GAP = 40;
const PHASE_BOX_PAD = 40; // inner padding inside a phase box — bumped from 20
// so rotated x-axis labels on bar charts (drawBars
// uses text-anchor:end transform:rotate(-45) which
// can extend past the panel SVG's declared height/width)
// don't touch the box border.
const PHASE_LABEL_H = 56; // height reserved for phase header text — two
// lines (heading + subtitle) when the label
// contains " — "; single line otherwise.
const PHASE_BOX_GAP = 32; // vertical gap between phase boxes
// Header height grows with the summary list. Base block (title + suite +
// generated + headline metrics) takes ~80px; each summary row adds 18px.
const summaryCount = (meta.summary ?? []).length;
const HEADER_H = Math.max(130, 80 + summaryCount * 18 + 24);
// Split panels: general 2-col grid above, phase-grouped boxes below. Both
// share the same panelW/panelH so all charts align visually.
const generalPanels = panelsWithData.filter((p) => !p.phaseGroup);
const phasedPanels = panelsWithData.filter((p) => !!p.phaseGroup);
// Pre-render all panels (general + phased) up front — we need their widths
// to compute the dashboard total width before we can lay them out.
const allRendered = panelsWithData.map((p) => {
const svg = p.svg ?? drawGraphMulti(p.data, p.title, p.yLabel, p.yMax, p.verticalLines, p.horizontalLines, p.shadedZones, undefined, p.areaFills, p.lineColorOverrides, opts.xRelativeOriginMs);
return { ...extractSvgInner(svg), title: p.title, phaseGroup: p.phaseGroup };
});
const renderedGeneral = allRendered.filter((r) => !r.phaseGroup);
const renderedPhased = allRendered.filter((r) => r.phaseGroup);
// Two separate "slot widths" — the general grid (with throughput/CPU/etc.
// time-series at 1060px) inflates panelW for the top section, but per-phase
// distribution panels (bars/donut, 640-690px) shouldn't be forced to reserve
// 1060 each just because some OTHER panel is that wide. Computing each
// separately keeps the dashboard from being absurdly wider than its content.
const generalPanelW = Math.max(530, ...renderedGeneral.map((r) => r.width));
const panelH = Math.max(250, ...allRendered.map((r) => r.height));
// Group phased panels by index → ordered phase sections (each section will
// render as a single boxed row containing N side-by-side panels). The slot
// width for a section is the MAX width of its own panels — bar charts get
// bar-chart slot widths, not the global time-series slot width.
const phaseByIndex = new Map<number, { label: string; panels: typeof renderedPhased; slotW: number }>();
for (const r of renderedPhased) {
const pg = r.phaseGroup!;
let entry = phaseByIndex.get(pg.index);
if (!entry) {
entry = { label: pg.label, panels: [], slotW: 0 };
phaseByIndex.set(pg.index, entry);
}
entry.panels.push(r);
if (r.width > entry.slotW) entry.slotW = r.width;
}
const phaseSections = [...phaseByIndex.entries()]
.sort(([a], [b]) => a - b)
.map(([_, v]) => v);
const generalRows = Math.ceil(renderedGeneral.length / COLS);
// Per-section column count (default 3, overridable via phaseGroup.cols)
// + per-panel colSpan support so a section can be e.g. 1 wide hero +
// 2-per-row beneath (Node memory section uses this).
const DEFAULT_COLS = 3;
const sectionColsOf = (s: typeof phaseSections[number]): number => {
for (const r of s.panels) {
const pg = r.phaseGroup;
if (pg?.cols !== undefined) return pg.cols;
}
return Math.min(s.panels.length, DEFAULT_COLS);
};
const colSpansOf = (s: typeof phaseSections[number]): number[] =>
s.panels.map((p) => Math.max(1, p.colSpan ?? 1));
// Compute grid row count taking colSpan into account.
const sectionRowCountOf = (s: typeof phaseSections[number]): number => {
const cols = sectionColsOf(s);
let row = 0;
let usedInRow = 0;
for (const span of colSpansOf(s)) {
if (usedInRow + span > cols) {
row++;
usedInRow = span;
} else {
usedInRow += span;
}
}
return row + 1;
};
const sectionRowWs = phaseSections.map((s) => {
const cols = sectionColsOf(s);
return cols * s.slotW + (cols - 1) * COL_GAP + PHASE_BOX_PAD * 2;
});
const generalGridW = COLS * generalPanelW + (COLS - 1) * COL_GAP;
const contentW = Math.max(generalGridW, ...sectionRowWs, 0);
const totalW = PADDING * 2 + contentW;
const sectionHeights = phaseSections.map((s) => {
const rows = sectionRowCountOf(s);
return PHASE_LABEL_H + rows * panelH + (rows - 1) * ROW_GAP + PHASE_BOX_PAD * 2;
});
const phaseSectionH = phaseSections.length > 0
? sectionHeights.reduce((a, h) => a + h, 0)
+ (phaseSections.length - 1) * PHASE_BOX_GAP
+ (renderedGeneral.length > 0 ? ROW_GAP : 0)
: 0;
const totalH = PADDING * 2 + HEADER_H + 16
+ (generalRows > 0 ? generalRows * panelH + (generalRows - 1) * ROW_GAP : 0)
+ phaseSectionH;
// --- Header ---
const summaryRows = meta.summary ?? [];
const summaryX0 = PADDING;
const summaryY0 = PADDING + 64; // below the title block
const summaryLineH = 18;
// Title + suite path + generated timestamp on the left,
// headline metrics on the right.
const headerSvg = `
<rect x="0" y="0" width="${totalW}" height="${totalH}" fill="${BRAND.surface_primary}"/>
<rect x="${PADDING}" y="${PADDING}" width="${totalW - PADDING * 2}" height="${HEADER_H}"
fill="${BRAND.surface_tertiary}" stroke="${BRAND.border_light}" stroke-width="1"/>
<text x="${PADDING + 20}" y="${PADDING + 32}"
font-family="${BRAND.font}" font-size="18" font-weight="600"
fill="${BRAND.text_emphasis}">Sim run: ${escapeText(meta.topology)}</text>
<text x="${PADDING + 20}" y="${PADDING + 52}"
font-family="${BRAND.font_mono}" font-size="11"
fill="${BRAND.text_secondary}">${escapeText(meta.suite)}</text>
<text x="${PADDING + 20}" y="${PADDING + 68}"
font-family="${BRAND.font}" font-size="11"
fill="${BRAND.text_hint}">${escapeText(meta.generated)}</text>
${headlineMetric(totalW - PADDING - 320, PADDING + 32, "Wall time", `${meta.walltime_s.toFixed(2)}s`)}
${headlineMetric(totalW - PADDING - 220, PADDING + 32, "Jobs done", String(meta.jobs_completed))}
${headlineMetric(totalW - PADDING - 100, PADDING + 32, "Throughput", `${meta.throughput_per_s.toFixed(1)}/s`)}
${summaryRows.map((row, i) => `
<text x="${summaryX0 + 20}" y="${summaryY0 + 18 + i * summaryLineH}"
font-family="${BRAND.font}" font-size="12" font-weight="600"
fill="${BRAND.text_emphasis}">${escapeText(row.label)}:</text>
<text x="${summaryX0 + 140}" y="${summaryY0 + 18 + i * summaryLineH}"
font-family="${BRAND.font_mono}" font-size="11"
fill="${BRAND.text_primary}">${escapeText(row.value)}</text>
`).join("\n")}
`;
// --- General panels (2-col grid, using time-series slot width) ---
const generalY0 = PADDING + HEADER_H + 16;
const generalSvg = renderedGeneral.map((p, i) => {
const col = i % COLS;
const row = Math.floor(i / COLS);
const x = PADDING + col * (generalPanelW + COL_GAP);
const y = generalY0 + row * (panelH + ROW_GAP);
return `<g transform="translate(${x}, ${y})">${p.inner}</g>`;
}).join("\n");
// --- Phase sections (one boxed row per phase, no wrap) ---
// Cycle through subtle pastel backgrounds so adjacent phases are
// visually distinguishable; user explicitly asked for "different
// background / outlines" so the eye can compare phases at a glance.
const PHASE_PALETTE = [
{ fill: "#f0f6ff", stroke: "#bcd5ff" }, // soft blue
{ fill: "#fff7ed", stroke: "#fdba74" }, // soft orange
{ fill: "#f0fdf4", stroke: "#86efac" }, // soft green
{ fill: "#fdf4ff", stroke: "#e9d5ff" }, // soft violet
{ fill: "#fff1f2", stroke: "#fda4af" }, // soft rose
];
const phasesY0 = generalY0
+ (generalRows > 0 ? generalRows * panelH + (generalRows - 1) * ROW_GAP + ROW_GAP : 0);
const phasesSvg = phaseSections.map((section, si) => {
const palette = PHASE_PALETTE[si % PHASE_PALETTE.length];
const boxY = phasesY0
+ sectionHeights.slice(0, si).reduce((a, h) => a + h + PHASE_BOX_GAP, 0);
const phaseBoxH = sectionHeights[si];
const slotW = section.slotW;
const cols = sectionColsOf(section);
const rowW = cols * slotW + (cols - 1) * COL_GAP;
const boxX = PADDING;
const boxW = contentW;
const innerX0 = boxX + Math.max(PHASE_BOX_PAD, (boxW - rowW) / 2);
// Lay out with colSpan awareness: panel with colSpan=2 takes two grid
// slots (slotW + COL_GAP + slotW) and a single-span panel takes one.
// Wrap to next row when current row's used cols would overflow.
const spans = colSpansOf(section);
let row = 0;
let usedInRow = 0;
const panelsRow = section.panels.map((p, i) => {
const span = spans[i];
if (usedInRow + span > cols) {
row++;
usedInRow = 0;
}
const col = usedInRow;
const x = innerX0 + col * (slotW + COL_GAP);
const y = boxY + PHASE_LABEL_H + PHASE_BOX_PAD + row * (panelH + ROW_GAP);
usedInRow += span;
return `<g transform="translate(${x}, ${y})">${p.inner}</g>`;
}).join("\n");
return `
<rect x="${boxX}" y="${boxY}" width="${boxW}" height="${phaseBoxH}"
rx="10" ry="10"
fill="${palette.fill}" stroke="${palette.stroke}" stroke-width="1.5"/>
${renderSectionLabel(section.label, boxX + PHASE_BOX_PAD, boxY + 22)}
${panelsRow}
`;
}).join("\n");
const panelsSvg = `${generalSvg}\n${phasesSvg}`;
return `<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" width="${totalW}" height="${totalH}" viewBox="0 0 ${totalW} ${totalH}">
${headerSvg}
${panelsSvg}
</svg>`;
}
// Section header that breaks a "Foo — bar baz qux" label into two lines
// (heading + a slightly smaller subtitle). Single-line labels render
// unchanged. Wrapping keeps long section labels from overflowing the box
// width without needing dynamic measurement.
function renderSectionLabel(label: string, x: number, y: number): string {
const sep = " — ";
const idx = label.indexOf(sep);
if (idx < 0) {
return `<text x="${x}" y="${y}" font-family="${BRAND.font}" font-size="14"
font-weight="600" fill="${BRAND.text_emphasis}">${escapeText(label)}</text>`;
}
const heading = label.slice(0, idx);
const subtitle = label.slice(idx + sep.length);
return `
<text x="${x}" y="${y}" font-family="${BRAND.font}" font-size="14"
font-weight="600" fill="${BRAND.text_emphasis}">${escapeText(heading)}</text>
<text x="${x}" y="${y + 18}" font-family="${BRAND.font}" font-size="12"
font-weight="500" fill="${BRAND.text_primary}">${escapeText(subtitle)}</text>
`;
}
function headlineMetric(x: number, y: number, label: string, value: string): string {
return `
<text x="${x}" y="${y}" font-family="${BRAND.font}" font-size="11"
fill="${BRAND.text_secondary}">${escapeText(label)}</text>
<text x="${x}" y="${y + 22}" font-family="${BRAND.font_mono}" font-size="18" font-weight="600"
fill="${BRAND.text_emphasis}">${escapeText(value)}</text>
`;
}
function escapeText(s: string): string {
return String(s)
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;");
}