On 2026-09-04 an auth outage signed ~21,600 desktops out of Orca Cloud and
every paired phone showed the generic "Can't reach desktop" for hours. The
desktop knew why, the cell watched it happen, and neither could say so.
The desktop now names auth loss on its control close reason; the cell
remembers that reason per (userId, relayHostId) and replays it as the close
reason of the 4404 it already sends a phone whose host is absent; the phone
turns it into "Desktop signed out — sign in to Orca on your desktop to
reconnect". Retry cadence, close codes and every message body are untouched.
The reason rides the WebSocket close reason because there is no additive JSON
channel to a shipped phone: RelayPhoneHelloSchema, RelayAuthSchema and the
director's ResolveResponseSchema are all zod .strict(), and /v1/connect
rejects any query string outright. A new close code was also rejected — an old
phone would fall out of mobileRelayRecoveryFor and off the 5-15s host-offline
backoff onto the faster transport backoff.
The cell keeps the reason in memory rather than Postgres: a phone reaches the
cell its host's assignment row already names, which is the cell that saw the
close, and losing it on a cell restart degrades to today's verdict rather than
a wrong one.
* fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths
activateControl, acquireActivity, changeActivity and
removeSupersededSameCellControls each adjust exactly one cell's
reservation, yet took SELECT * FROM relay_cells FOR UPDATE, so every
desktop rebind and phone reconnect in the fleet queued behind every
other one and behind placement. They now use the single-row atomic
update (or lock only their own cell row), leaving the inventory lock to
placement and sweeps.
Fleet-wide 55P03 retries ran p50 430 / p99 1320 per five minutes on
2026-09-03, every cell pinned sqlLatencyMsMax at the lock timeout, and
the old cell image crashed on the resulting pool timeouts ~every 15
minutes. A real-Postgres test holds another cell's row and asserts a
rebind proceeds; re-adding the inventory lock fails it.
* fix(relay): lock the touched cell rows in order on cross-cell activity moves
Review found that acquireActivity's existing-lease branch could lock the
old lease's cell row (via removeActivityLease) before the new cell's row,
which cycles with placement's ascending inventory lock; reproduced on
real Postgres as paired 55P03 retries. lockCellRows now takes the one or
two rows a per-connection path touches in cell_id order with the 500 ms
request bound, and the census fails on any inline relay_cells FOR UPDATE
outside the named lock helpers. A three-cell Postgres test moves an
activity from the highest cell to a lower one while the target row is
held and asserts the mover holds nothing else; five revert-mutants
(inventory lock on each path, dropped ordering, dropped ORDER BY) fail it.
* test(relay): make the inline relay_cells lock census scan whole statements
Review showed two evasions: a FOR UPDATE inside query() and a queryLocked
whose FROM relay_cells sat past a fixed line window. The guard now matches
every query()/queryLocked() template statement in full; both evasions
fail it. Also clears relay_cell_connection_snapshots in the connection-
headroom Postgres suite so an aborted run does not poison the next.
The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.
* fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar
The pre-drain dry-run froze at minute one on relayPostgresRetryExhausted: 0
in every run since #18521 reached the director, blocking the cell roll that
carries the same fix. #18521 made contended request-path waiters fail fast
(500 ms) instead of succeeding slowly, so exhaustion is now a steady
contention rate: 236/236 five-minute windows non-zero over 23 h; post-#18521
p50 42 / p90 147 / max 220 fleet-wide; the 2026-08-23 incident peaked at 467.
300 clears every measured healthy window and stays under the incident shape.
/v1/assign 503 share was unchanged by #18521 (13.9% vs 12.3%).
* test(cloud): pin the exhausted-retry freeze boundary at exactly 300
* docs(cloud): reword relay comments that still described the zero exhausted-retry bar
decodeURIComponent on the /v1/connect/ and /v1/host/data/ path segments threw
URIError out of the http 'upgrade' listener, which is uncaught and kills the
relay process. Any client that sends GET /v1/connect/% could take down a cell
(and every connection on it) or a director instance. Pre-existing since the
splice landed (orca-cloud #20); not introduced by the import.
A malformed escape now takes the existing 4xx reject branch. The blackbox test
sends three malformed connect targets and one host-data target to the real
server and asserts no uncaughtException fires and a well-formed upgrade still
gets 101 afterwards; reverting either site fails it.
* fix(cloud): retry the committed-winner collision codes in relay schema startup
`CREATE TABLE IF NOT EXISTS` only checks the name before the catalog inserts, so
the loser of a concurrent CREATE fails in one of two ways depending on timing:
on the catalog unique index (23505, which the startup retry already handled) or,
when the winner has committed by the time the loser reaches TypeCreate /
heap_create_with_catalog, on the name check those routines repeat (42710
duplicate type, 42P07 duplicate relation). The predicate treated the latter as
fatal, so a director could fail startup on a table it was about to find present.
This is what turned `postgres-schema-concurrency-postgres.test.ts` red on main
and on every relay PR (CI's shared runner loses the race more often than a dev
box): a throwaway diagnostic run in CI reported 42710 from TypeCreate and 42P07
from heap_create_with_catalog as the only rejection reasons.
Treat 42710/42P07 as retryable for `CREATE TABLE IF NOT EXISTS` and 42P07 for
`CREATE [UNIQUE] INDEX IF NOT EXISTS`; every other statement shape still fails
fast. The concurrency test now runs ten rounds and reports the loser's SQLSTATE
instead of a bare boolean.
* chore(cloud): allowlist the RFC 6455 example Sec-WebSocket-Key for upgrade tests
Cloud Verify's Secret scan runs gitleaks over --all refs, so the raw-socket
upgrade test on fix/relay-upgrade-malformed-uri (#18547) trips every cloud PR's
scan until its allowlist reaches main. Land the allowlist here first.
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/.
The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over
all 23 rows) is one global critical section shared by the assignment hot path
and every director sweep; with the pool's 1s lock_timeout a blocked waiter held
a pooled client for a full second, producing ~690 55P03 retries per 5 minutes
in production. Request paths now bound the wait at 500ms with a SET LOCAL that
is restored to the pool default before the next statement; director-only sweeps
take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is
exported as additive runtime-metrics fields so the bound can be tuned.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong