Incident 2026-09-04 ~01:05Z: after a background/foreground cycle the phone's
relay dial timed out inside the cell's acceptClient DB phase while the fleet
was in a cell-inventory lock storm (55P03 retries ~7.5k/h vs a ~1k/h floor).
Relay (cloud/apps/relay)
- acceptClient checks socket.readyState after each serialized Postgres call
and abandons the accept once the phone has hung up, releasing the capacity
reservation, failing the credential reservation, and releasing the activity
lease it just acquired instead of leaking it to expiry cleanup and then
throwing host_data_reservation_already_bound at bind.
- New structured event orca_relay_client_accept_abandoned {stage, elapsedMs}
and runtime-metric fields clientAcceptsAbandonedByStageDelta /
clientAcceptAbandonedMsMax so the "phone gave up behind the lock" rate is
quantifiable per cell.
- Control lease grants are jittered: 55 min minus [0, 10 min). Every host
that (re)connected in the same minute rebound as one cohort every ~54 min
(c27 autoheal recreate at 23:23Z re-homed ~420 controls; ~1.1k 1006 +
~1k 4408 "control rebound" closes landed in a 3 s window at 00:50:14Z),
and each rebind is an activateControl transaction on the inventory lock.
No wire change: leaseExpiresAt was always a server-chosen absolute time.
Desktop (src/main/runtime/relay)
- Control rotation rebinds 1-6 min early instead of 1-2 min, so a re-homed
cohort spreads across cycles rather than pinning one phase for the life of
the process.
Phone (mobile/src/transport)
- openAuthenticatedDirectEndpoint treats 'reconnecting' as a failed probe. On
a dead LAN the foreground direct dial dies with an instant 1006 and the
direct client enters its own 500/1000/2000 ms backoff; the probe used to
wait out its full 12 s bound holding the supervisor mutex, so relay recovery
queued behind three doomed redials. The stage-aware bound from #18518 is
unaffected.
Not done here: rolling the 23 GCE cells onto the post-#18521 image (500 ms
lock_timeout) is a deploy owned by cloud-deploy-relay-production-same-cap.
The five `node ... | tee` steps in cloud-operate-relay-production-rehome-job.yml
reported tee's exit code, so a thrown inspect or apply passed green. The Aug 28
21:25Z and Aug 29 inspects and today's first inspect all printed
"director returned an invalid regional rehome control" (the durable control had
moved to generation 12 when the Aug 28 rehome aborted) and still succeeded.
`shell: bash` adds `-o pipefail`. A test pins the default and the tee count.
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/.
The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over
all 23 rows) is one global critical section shared by the assignment hot path
and every director sweep; with the pool's 1s lock_timeout a blocked waiter held
a pooled client for a full second, producing ~690 55P03 retries per 5 minutes
in production. Request paths now bound the wait at 500ms with a SET LOCAL that
is restored to the pool default before the next statement; director-only sweeps
take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is
exported as additive runtime-metrics fields so the bound can be tuned.
Mirrors stablyai/orca-cloud#470. The private relay workflows are retired, so
the dual accept has one live arm left. Add `github_workflow_file_prefix` for
the primary repository's workflow filenames, point `github_repo`/
`github_repo_id` at `stablyai/orca` (`1183888342`), and empty
`github_accepted_repositories` in both environments. Every relay provider goes
back to a single arm naming `cloud-` prefixed workflow refs.
`cloud/infra/terraform` stays byte-identical to the private branch. The two
identity tests diverge here as they already did, so they take the same change
rather than the same bytes: both now render the trusted ref head from the
Terraform variable instead of this checkout's own workflow filenames, which is
what lets the length pin be the same 791 characters in either repository.
* chore(cloud): pin staging relay c3 to the director's image
Mirrors stablyai/orca-cloud#468. c3 stayed on sha-c91439af after the
director and c4 moved to sha-e3e92d95, so the staging capacity proof's
compatible-director-image check has failed since 2026-08-14.
* test(cloud): scope the launch-image pin to staging C4 now that C3 shares the digest
* test(cloud): keep the public workflow assertions; scope only the launch-image pin to C4
The relay root does not declare it (it belongs to the private apps root), and
Terraform rejects an undeclared -var, so the first public Deploy Relay Staging
run failed at the C4 image bind.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo-H
Jinwoo Hong