mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-07 08:02:40 +00:00
Ruben Fiszel
42dd49aacd
Two new ignored simulations to characterise the algorithm's behavior
under conditions where the cap is not stable:
1. `fairness_oscillation_long_run` — 25 s sustained load with the
production-default 10 s window. Records per-second victim p95
latency and per-second noisy completion count to expose the
cap/uncap cycle. Asserts the oscillation peaks are bounded.
2. `fairness_burst_then_stop` — addresses the specific concern that
when noisy enqueues a large burst at t=0 (lowest scheduled_for in
the queue), workers fully switch to noisy on every uncap. Pushes
10 000 noisy jobs in one shot, no sustained noisy push, then runs
3 victim workspaces at 20 jobs/s each for 30 s. Records per-second
metrics for both workspaces.
Empirical findings (10 s window, ACTIVE_REFRESH_SECS=2 s,
refresh_loop driven every 250 ms):
- cap holds the bulk of the time; uncap windows are bounded by
refresh interval, not the duration window
- in the burst scenario the cycle is ~3 s (not 10 s) because
victim activity in the window dilutes noisy's stale completions
well before they age out
- victim p95 stays at ~85 ms baseline, with periodic blips to
~250-300 ms during uncap bursts — worst-second p95 285 ms
- victim completion rate remains 100 % throughout
- noisy is throttled, not excluded: drains ~22 % of its burst over
the 30 s simulation while victims see negligible degradation
Also: parameterised `run_scenario` on the fairness window so the
oscillation test can use the production-default 10 s instead of the
3 s short-window used by the other tests.