mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-19 16:03:29 +00:00
Ruben Fiszel
60e0b6e2fc
The refresh_loop helper had a rollback hack that forced re-aggregation
every 250 ms, masking the production reality: in the real cluster
`maybe_refresh_overloaded` is rate-limited to `ACTIVE_REFRESH_SECS`
(2 s when something is capped, 5 s when not). The hack was useful for
the short headline test but unfair to the long-running burst /
oscillation simulations, which need to expose what victims actually
experience between refreshes.
Plumbed a `force_refresh` flag through `refresh_loop` and
`run_scenario`. The headline `fairness_50_workers_diverse_workload`
keeps the fast refresh (its 5 s push window needs the cap to take
effect in time). The long simulations now run with the natural 2 s
claim guard.
Empirical impact of switching to prod cadence on the burst scenario:
- cycle becomes ~12 s (vs ~3 s with forced refresh): noisy uncaps
for ~3 s per cycle, then is re-capped on the next refresh
- during each ~3 s uncap window workers fully drain noisy at
capacity (~1000–1100 noisy jobs per window)
- worst-second victim p95 jumps from 285 ms to ~2 s
- avg per-second victim p95 still well under 1.5 s
- victim completion rate remains 100 %
The bumped assertions reflect this: avg p95 < 1500 ms (up from
1000 ms) and worst-second p95 < 4 s (up from 3 s). These bounds
still catch a meaningful regression while accommodating the real
oscillation amplitude.