Jinwoo Hong
7ffcb7c7b4
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 |
| Prod | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$58 | $\color{#cf222e}{\Huge{\mathbf{−}}}$13 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
<!-- /orca-pr-loc -->
## ELI5
When we restart a relay server during a routine rollout, computers that were offline during the rollout still have that server written down as "their" server. If one of them comes back during the ~45 seconds while the replacement server is starting up, the director currently tells it "your server is unavailable, try again" over and over, instead of just sending it to another server. This change lets the director send it elsewhere, but only when it has proof that the old server process is really gone.
## What Changed
**Before (production, 2026-10-24 23:43Z, cell production-gce-c15):** during a same-capacity roll the cell is roll-isolated, drained, then its single VM is replaced. A desktop host that was offline during the drain reconnected in the gap between the old process stopping and the new one being ready. The director refused it for ~26 s (`assignment rejected route=assign lane=placement hinted=true reason=relay_home_cell_unavailable cause=not_ready`, x5, plus `route=resolve ... cause=unheard`, x2). It got a grant somewhere else only once the new process was live.
**After:** while the replacement process is starting up (`not_ready`), the same host is moved to another cell on its first try. A host on a roll-isolated cell can be moved off even though the cell is not live, as long as the process heartbeating for the cell *started after the roll isolated it* (`relay_cell_runtime.started_at > relay_cell_admission.roll_isolated_at`), its heartbeat is fresh, and (for connection-capped cells) its latest heartbeat reports zero connections. Both lanes that move an isolated host apply it: the narrowed isolated-replacement path (`isolatedIncumbentMayMove`) and the all-rows path it falls back to. Together these serve assign and resolve, because resolve hands a non-live cell to `assign`. "Roll-isolated" is the existing predicate: `migration-only` with a roll stamp younger than 2 h.
**Still refused (residual):** the `unheard` window, when the old process's heartbeat is past the 45 s TTL and the new process has not been heard yet. On c15 that was about 5-10 s. Nothing proves the drained sockets are gone there, so the refusal stays, and the client's retry recovers once the new process heartbeats. Also unchanged: an ordinary readiness dip, a `migration-only` cell without a roll stamp, a stale (over 2 h) stamp, and a new process that later goes unheard. Fence handling is untouched.
## Why
The refusal exists because moving a host off a cell we cannot fully trust could leave the host's old socket served by the old process while its clients go to the new cell. Until now, the only proof the code accepted was an operator-written fence. This adds a second proof that costs no operator step:
- **A new incarnation means the drained process is gone.** Each process mints a fresh incarnation UUID and `startedAt` at boot (`cell-heartbeat-client.ts`), and the director rejects a heartbeat with an older `startedAt` once a newer one is recorded. Each cell VM runs exactly one relay container (`docker run --name orca-relay --restart always`, with stale containers force-removed at boot). The cell MIG uses `replacement_method = RECREATE` with `max_surge = 0` (pinned by a Terraform assert), so the old VM is deleted before the new one is created. A same-VM container restart also produces a new incarnation and is just as safe: Docker restarts only after the old process exits, and that exit closes all of its sockets.
- **Whatever the new process holds, it can enforce.** It is heartbeating (fresh), and on capped cells it reports no connections at all. A host that attaches between two heartbeats carries the old epoch, so the epoch bump this move writes makes the cell refuse it or close it with 4409. That check is the same live `verifyCellAssignment` / control-renewal read the existing live-isolated path relies on.
- **Clock skew** between the cell's `startedAt` and the director's stamp is covered by the zero-connections check on capped cells only: a drained process that only *looks* post-stamp still holds its sockets, so it is refused. Uncapped cells report no counts and get no such guard; they already move off without a fence on the dead-cell path today, so this adds no new exposure there.
Alternatives considered: (a) adding an incarnation column to `relay_assignments`: a schema change for no extra proof. (b) Honouring any not-live roll-isolated cell: that drops the proof. (c) Having the roll write a fence for the `unheard` window: the only fence driver refuses enabled cells, a fence is a MIG resize to 0 that would fight the roll's own apply, and the window is a few seconds the client retry already absorbs.
## Linked Issue
N/A (internal production incident, 2026-10-24 c15 roll).
## Visual Proof
N/A: director placement logic only, no UI.
## Testing
- [x] Automated tests added: `cloud/apps/relay/src/assignment-roll-replaced-cell.test.ts` (18 cases; the redial cases run for both hinted assign and resolve):
- not_ready with an incarnation started after isolation: re-placed, epoch bumped.
- Refused as before: not_ready with the pre-isolation incarnation, with reported connections, after the new process goes unheard, with a stale stamp, on an ordinary unstamped readiness dip, and on an unstamped migration-only cell. Also refused: the unheard window between the old and new process.
- One case per lane. The narrowed path's cross-region move fails if only the narrowed-path change is reverted. The all-rows path's move, forced by units no lease backs, fails if only the all-rows change is reverted.
- [x] Full relay vitest suite, `--maxWorkers=2`: 130 files / 1344 tests passed, 39 files (201 tests) skipped. The Postgres-backed suites skip locally (no `ORCA_RELAY_TEST_POSTGRES_URL`); CI runs them.
- [x] `pnpm typecheck` in `cloud/apps/relay`.
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
Backwards compatibility: no schema, wire or RPC change; old directors keep refusing as today. Performance: the new query runs only when a host's home cell is roll-isolated and not live; the general sticky hot path is unchanged.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from cloud/, not the repository root. The source is covered by
the repository's root MIT license.
The 2026-10-21 security review records dependency/container scans, remaining upstream advisories, CI coverage, and the validation required before rolling out the updated images and host pins.
Packages
packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).apps/relay: the relay server. The same image runs as a director or a cell depending onORCA_RELAY_ROLE.apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.apps/pushandpackages/push-contract: the mobile push gateway that holds the APNs key and sends to phones through APNs and FCM, and its wire contract. Deployment and operations are owned by the privatestablyai/orca-cloudrepository.
Mobile push gateway
apps/push is a separate Cloud Run service from the relay. Phones never hold an
Orca credential for it: the desktop host authenticates with the same X25519
key it uses for the relay, answering an encrypted challenge to mint a 24 hour
session, then registers each paired phone's native push token and asks the
gateway to push. The gateway queues each event as its own notification,
enforces per-host quotas and request limits, and retires a
registration as soon as Apple or Google reports the token unregistered.
Provider push is the only ordinary mobile OS-banner path. The notification
socket is retained only for live dismissal and reconnect tray reconciliation;
it never creates or recovers banners. Desktop notification categories remain
authoritative.
Each delivery is persisted as one notification event. Before deploying an
incompatible queue format, stop all older push gateway revisions and clear only
unpublished push delivery fixtures; no queue preservation or migration is required.
FCM notification messages are inherently collapsible while offline and have a
small concurrent collapse-key budget, so every pending alert is not guaranteed.
Storage follows the relay pattern: PostgreSQL in production, SQLite for tests
and local development. Configure it with ORCA_PUSH_PUBLIC_URL, ORCA_PUSH_FCM_PROJECT_ID,
ORCA_PUSH_DATABASE_URL, the three APNs variables (ORCA_PUSH_APNS_KEY,
ORCA_PUSH_APNS_KEY_ID, ORCA_PUSH_APPLE_TEAM_ID, all three or none), and
optionally ORCA_PUSH_APNS_TOPIC. The FCM credential comes from
the runtime service account, so no key material is configured for Android. See
push gateway operations for deployment and recovery.
Logging is aggregate counters only. Tokens, notification titles, notification bodies, and full host fingerprints never reach a log line.
Deployment configuration
The relay runtime and wire contracts are public. Production and staging Terraform,
backend configuration, operator tools, runbooks, broker image assembly, and deployment
workflows live in the private stablyai/orca-cloud repository. That repository pins
an immutable public source revision and combines it with its deployment configuration.
Cloud Verify here builds and tests public runtime code, audits dependencies, scans
relay and push images, and checks for secrets without cloud credentials. Private CI
also validates Terraform, scans the SQL proxy and broker, and runs deployment contracts.
Local development
cd cloud
pnpm install
pnpm build
pnpm test
pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only
when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17
database, for example:
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
Configuration is read from environment variables validated in
apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32
bytes) is the only required value; everything else has a local default.