mirror of
https://github.com/stablyai/orca.git
synced 2026-10-02 16:02:15 +00:00
Jinwoo Hong
84f58a1fc7
* fix(relay): refuse a redial at once while the host's own release holds its row During an Asia drain the host whose socket closes is the one that redials. Its release on the draining cell locks its assignment row first, then waits on the cell's busy row for up to the lock timeout. The director's sticky and placement paths waited on that assignment row inside the single sticky slot, and the sticky path then locked the busy cell row itself before it checked isolation. The slot backed up and dials timed out fleet-wide. Both paths now take the host's assignment row NOWAIT and throw RelayAssignmentRowBusyError when it is held. /v1/assign answers that with 503, Retry-After 1 and error assignment_row_busy, logged with its own reason. The sticky path decides isolation before it touches the pinned cell row. An isolated retry keeps its own tier as its retry scope, so a busy lock inside it no longer falls to the all-rows path. The local drain arm drops its zero-release-failures bar, which the Asia arm never had. The drain harness gains a departing-host arm: each host releases its own lease, then redials after 150, 400 or 1000 ms on the desktop client's 5-5.5 s pacing. At 400 ms, main rejected 83 of 180 first dials by sticky wait timeout, placed 11.6/s with 3.1 director backends lock-waiting, and took 11.2 s at p95 from release to placed. Now: 16 fast refusals, 18/s, no lock waits, 5.7 s at p95. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): wait briefly for a calm host's row and keep the dead-cell sweep going The dead-cell sweep treated RelayAssignmentRowBusyError as fatal, so one busy host ended the sweep for every later host each tick. It now skips that host and carries on. The sticky path refused a busy row at once for every host. A calm host redialling after its own clean close often meets its own short release, and a refusal costs it the client's 5 s assign gate. When the pinned cell is general and live, the sticky path now waits up to 1 s for the row before refusing. A roll-isolated, parked or dead cell still gets the immediate refusal. Placement keeps NOWAIT, because it holds cell rows while it would wait. A resume refused for a busy row now carries Retry-After 1 as well. The departing-host harness arm now bounds the busy refusals at 20% of hosts and the p95 at 8 s, and counts unexpected errors apart from retryable refusals. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): never wait on a host's row while the sticky retry holds its cell row The inventory-first sticky retry takes the pinned cell row before the assignment row. With the calm-host bounded wait it could then wait up to 1 s on the assignment row while holding the cell row, the reverse of the ranked lock order, against this host's own release, which holds its row and wants the cell's. The bounded wait now applies only when no cell row is held; the retry stays NOWAIT. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010