mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
Jinwoo Hong
46d0a00548
* fix(relay): move director traffic in two steps so a deploy does not land on one instance Cloud Run routes a new director revision's floor instances their first request roughly 60-110 s after they start, while a traffic move reaches clients in 15-30 s. The one-step 100% move therefore put every assignment on the single instance started with the revision, which sits at its own admission limit at peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way. With a floor above one, the deploy now moves 20% first, waits until Cloud Run's request log shows every floor instance serving (six-minute deadline), then moves the rest. A timeout or failure at that stage returns all traffic to the previous serving revision, which still holds its warm floor. The staging deploy identity gains log read access for the same check. * fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery A hung gcloud log read could hold the 20/80 split past the six-minute deadline; each read now times out after 30 s and counts as an unread poll. A timeout logs the distinct instances seen, so a partial share packed onto fewer instances than the floor shows up. The runbook covers restoring a split left by a cancelled job and the missing selector-rollback tag after a timed-out deploy.