mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-19 08:01:25 +00:00
Ruben Fiszel
c82056cfde
* fix(schedules): stop disabling schedules on transient push errors A scheduled flow whose next-occurrence push failed after retry exhaustion used to be disabled, killing a healthy schedule over a transient DB blip (pool contention, statement timeout). Now that the unarmed-schedule reconciler exists (#10174), transient failures no longer disable: the current occurrence runs to completion and the reconciler re-arms the next occurrence once this run leaves the queue. In the flow schedule-push path after retry exhaustion we now branch on the error: QuotaExceeded/NotFound still disable (the schedule's own fault, and rearm_schedule would otherwise leave them enabled-yet-unarmed forever), while transient errors are only reported and the flow continues. The previous iteration returned a SchedulePushZombieError to force a zombie restart; that is removed, because zombie detection cancels (does not restart) same-worker flows, so it would have lost the current run of a same-worker scheduled flow. The now-obsolete SchedulePushZombieError type and its catch in worker.rs are deleted. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(schedules): back off and surface repeated reconciler re-arm failures The unarmed-schedule reconciler retried a schedule that could not be re-armed on every pass, forever, logging only to the server. With the flow schedule-push path no longer disabling on non-transient errors, a persistently-broken push (bad stored cron/timezone/args, lapsed license key) now stays enabled and would spin in that loop silently. The reconciler now tracks consecutive re-arm failures per schedule: exponential back-off (2, 4, 8, … passes, capped) between retries so a broken schedule is not hammered, and after 3 consecutive failures it surfaces the cause once (records schedule.error + raises a critical alert) without disabling. Both reset the moment the schedule re-arms, which also clears the recorded error. Verified end-to-end on a running server: a flow schedule with a corrupted cron stays enabled, retries back off, the error is surfaced after the third failure, and it re-arms and clears the error once the cron is fixed. Fixes WIN-2198 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>