Files
orca/cloud/apps/relay/src
Jinwoo Hong 61b09b7a02 fix(relay): abandon dead client accepts, jitter and lengthen the control lease, fail direct probes fast (#18959)
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease

The accept runs several serialized Postgres calls behind the contended
cell-inventory lock, and phones bound their dial. Finishing that work for a
phone that had already left acquired (and leaked for 90s) an activity lease and
then failed at bind with host_data_reservation_already_bound. Check the client
socket between the DB steps and unwind what was taken, reporting the stage on
orca_relay_client_accept_abandoned.

Jitter the control lease grant so a cohort that reconnected in the same minute
(a cell recreate dumps hundreds at once) walks apart instead of rebinding
together every cycle.

On the phone, treat a probe session that enters 'reconnecting' as a failed
probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting
it out held the supervisor's operation mutex for the full 12s bound.

* perf(relay): lengthen the control lease to 6h

The lease bounds how long a host lingers on a cell after a missed drain, and
rebinding it is the only passive rebalancing we have, so it stays finite. 6h
keeps both properties while cutting control-activation traffic on the contended
cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the
75s silence watchdog are enforced separately, so the longer grant authorizes
nothing extra. The jitter widens with it, to +/-30 min.

* fix(relay): let one flap recover the direct probe; correct the leak window

'reconnecting' is published on any socket close, so rejecting on it outright
turned a single access-point flap into a booked direct failure and a 60s
cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected'
transition still resolves; a dead LAN still fails in ~2s rather than holding the
supervisor's operation mutex for the 12s bound.

The abandoned accept held its activity lease for the 10s attach deadline, not
90s -- the attach timer is armed before bind throws and already unwinds it.

Also cover the assignment-stage check that guards reserveCredential, and drop a
spread assertion the two exact-value assertions above already imply.

* fix(relay): extend the probe grace once on a handshake; pin the lease band top

The redial fires at 500ms but 'connected' waits on the Noise handshake and a
capability RPC, so one 2s window is too tight for real work. A 'handshaking'
transition is evidence the peer answered, so extend the grace once; a stalled
handshake still fails at ~3.5s, far inside the 12s bound.

The longest-lease case only had an upper bound, which a jitter clamped to one
side would satisfy. Pin it to the exact top of the band instead, and assert the
assignment resolve ran so the third-guard test cannot pass vacuously.
2026-09-05 20:47:27 -04:00
..