mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-13 00:03:57 +00:00
Ruben Fiszel
54217846a9
Existing tests only verified the algorithm's outputs on fabricated activity tables. This adds a real end-to-end simulation that drives 50 concurrent mock workers (async tasks doing the actual pull → mark running → sleep → complete cycle over real v2_job_queue rows) against sustained diverse traffic from one noisy + five victim workspaces. Each scenario runs twice — once with the fairness algorithm disabled (control) and once enabled (treatment) — and asserts the QoL improvement on the victim workspaces is substantial. Setup: - Pool: dedicated 80-connection pool (sqlx::test's 10-conn default serialised the workers and masked the algorithm's effect). - Backlog: 1500 pre-populated noisy queue rows + 200 noisy completions in the rolling window, so workers start saturated and the first fairness refresh already has a dominant-workspace signal to act on. - Sustained load: 4 noisy pushers running flat out (no inter-insert sleep), 3 medium victims (10 jobs/s) + 2 quiet victims (4 jobs/s), each with 60-100 ms job durations, for 5 seconds. - Refresh: 250 ms cadence, mirroring `maybe_refresh_overloaded`. Observed results (stable across 3 consecutive runs): - Control: victim completion rate ~28%, p95 latency ~15s - Treatment: victim completion rate 100%, p95 latency ~200ms - ~65x p95 improvement, ~3.5x completion-rate improvement Headline assertion accepts either ≥5x p95 improvement OR ≥1.5x completion-rate improvement, so the test stays green across CI machines whose relative speeds shift the balance between the two signals. Total runtime ~23s, marked `#[ignore]` to keep the default `cargo test` profile sub-second.