starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-26 00:02:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(relay): drain a same-cap cell over 5 minutes, not 2 (#22584)

Browse Source 
* fix(relay): drain a same-cap cell over 5 minutes, not 2

The 2026-09-23 c27 roll drained 2,145 hosts over the 2-minute window,
about 18 re-dials/s, while the director re-places roughly 8/s through
its single-slot sticky lane. The overflow queued behind slow
re-placements and timed out, so /v1/assign returned 503 fleet-wide for
about 5 minutes. 5 minutes is the cell's maximum pace window and keeps
the remaining 2,650-host cells near lane capacity. The transition wait
already outlasts a 5-minute window (17 min).

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* test(relay): pin the same-cap drain window contract at 5 minutes

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): keep the drain wait at lease plus the 5-minute window

The transition wait after a drain was set to the 15-minute migration
lease plus the pace window. Widening the window to 5 minutes without
moving the wait left 12 minutes for a migration that can hold for 15,
so a late-window migration would time the wave out into rollback.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
This commit is contained in:
Jinwoo Hong
2026-09-23 23:38:15 -04:00
committed by GitHub
parent 795b64b9a6
commit 128e97ffca
2 changed files with 12 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cloud/dev/scripts/relay-same-cap-script-census.test.mjs
+3 -2
View File
@@ -311,9 +311,10 @@ describe('same-cap roll scripts accept every same-cap cell', () => {
it('paces the drain it sends to the selected cell', () => {
const drain = workflow.split('--mode drain')[1] ?? ''
assert.match(drain.split('\n').slice(0, 2).join(' '), /--pace-window-ms "\$\{DRAIN_PACE_WINDOW_MS\}"/)
assert.match(workflow, /DRAIN_PACE_WINDOW_MS: '120000'/)
// 5 min is the cell's DRAIN_PACE_WINDOW_MAX_MS; a 2,700-host cell at 2 min overruns the director's sticky lane.
assert.match(workflow, /DRAIN_PACE_WINDOW_MS: '300000'/)
// The transition wait has to outlast the pacing window on top of the leases it waits on.
assert.match(workflow, /--activity restart-safe[\s\S]*?--timeout-ms 1020000/)
assert.match(workflow, /--activity restart-safe[\s\S]*?--timeout-ms 1200000/)
})
it('passes this cell\'s rehome protocol and pool on every plan validation the job runs', () => {