mirror of
https://github.com/warmbly/warmbly.git
synced 2026-09-05 08:01:23 +00:00
Matthew Meszaros
faaa3cded5
Three closed-loop background goroutines on the backend that manage the worker fleet without operator intervention. Rebalancer (default 5min): for each tier, drain hot workers (>80% utilization) onto cold workers (<50%, healthy). Safety rails: per- mailbox 24h cooldown to prevent thrashing, max 200 in-flight migrations, destination must be healthy or watch. Scaler (default 1h): compute fleet utilization per tier. At >=70% sustained, emit warning. At >=85% sustained, emit critical alert. If AUTO_PROVISION is allowed by provisioning_policy, snapshot the active auto-template for the tier into a new provisioning_jobs row — the state machine picks it up and provisions the box without admin click. QuarantineEvaluator (default 5min): inspect rolling 1h bounce/complaint rates, transition workers between health bands (healthy / watch / throttled / quarantined / blocked) using CLAUDE.md thresholds. Quarantined and blocked workers are auto-drained by the Rebalancer because ListCapacityCandidates excludes them. Every action is written to decision_log so the admin Decisions page can answer 'why did the system do X'.