Jinwoo Hong
0ae42fe594
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$402 |
| Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$155 |
<!-- /orca-pr-loc -->
## ELI5
Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has.
## What Changed
- New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes.
- With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool).
- A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s.
- A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free.
- A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy.
- A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots.
- The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant.
- Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first.
What a held cell does during a director outage (unchanged code paths, now for longer than the window):
- Desktops already connected keep their sockets. Nothing is closed or re-registered.
- A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today.
- The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row.
- New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders.
## Why
The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells.
Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10):
| | Cells line | Operating | Spare |
|---|---:|---:|---:|
| Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 |
| Plus C36/C37 at 16 | 332 | 487 | 3 |
| Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 |
| Plus two more Asia cells at 12 | 332 | 487 | 3 |
So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest:
- The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set.
- Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes.
Alternatives considered:
- Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget.
- A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current.
- Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch.
Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two.
Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image.
## Linked Issue
N/A (internal relay capacity work).
## Visual Proof
N/A: relay server behaviour only, no UI.
## Testing
- `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped.
- `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not.
- `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before.
- `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test.
- Mutation checks: making the slot count always 0, or removing the lock, each fails a test.
- Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally.
- Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above).
- [x] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from cloud/, not the repository root. The source is covered by
the repository's root MIT license.
The 2026-10-21 security review records dependency/container scans, remaining upstream advisories, CI coverage, and the validation required before rolling out the updated images and host pins.
Packages
packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).apps/relay: the relay server. The same image runs as a director or a cell depending onORCA_RELAY_ROLE.apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.apps/pushandpackages/push-contract: the mobile push gateway that holds the APNs key and sends to phones through APNs and FCM, and its wire contract. Deployment and operations are owned by the privatestablyai/orca-cloudrepository.
Mobile push gateway
apps/push is a separate Cloud Run service from the relay. Phones never hold an
Orca credential for it: the desktop host authenticates with the same X25519
key it uses for the relay, answering an encrypted challenge to mint a 24 hour
session, then registers each paired phone's native push token and asks the
gateway to push. The gateway queues each event as its own notification,
enforces per-host quotas and request limits, and retires a
registration as soon as Apple or Google reports the token unregistered.
Provider push is the only ordinary mobile OS-banner path. The notification
socket is retained only for live dismissal and reconnect tray reconciliation;
it never creates or recovers banners. Desktop notification categories remain
authoritative.
Each delivery is persisted as one notification event. Before deploying an
incompatible queue format, stop all older push gateway revisions and clear only
unpublished push delivery fixtures; no queue preservation or migration is required.
FCM notification messages are inherently collapsible while offline and have a
small concurrent collapse-key budget, so every pending alert is not guaranteed.
Storage follows the relay pattern: PostgreSQL in production, SQLite for tests
and local development. Configure it with ORCA_PUSH_PUBLIC_URL, ORCA_PUSH_FCM_PROJECT_ID,
ORCA_PUSH_DATABASE_URL, the three APNs variables (ORCA_PUSH_APNS_KEY,
ORCA_PUSH_APNS_KEY_ID, ORCA_PUSH_APPLE_TEAM_ID, all three or none), and
optionally ORCA_PUSH_APNS_TOPIC. The FCM credential comes from
the runtime service account, so no key material is configured for Android. See
push gateway operations for deployment and recovery.
Logging is aggregate counters only. Tokens, notification titles, notification bodies, and full host fingerprints never reach a log line.
Deployment configuration
The relay runtime and wire contracts are public. Production and staging Terraform,
backend configuration, operator tools, runbooks, broker image assembly, and deployment
workflows live in the private stablyai/orca-cloud repository. That repository pins
an immutable public source revision and combines it with its deployment configuration.
Cloud Verify here builds and tests public runtime code, audits dependencies, scans
relay and push images, and checks for secrets without cloud credentials. Private CI
also validates Terraform, scans the SQL proxy and broker, and runs deployment contracts.
Local development
cd cloud
pnpm install
pnpm build
pnpm test
pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only
when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17
database, for example:
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
Configuration is read from environment variables validated in
apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32
bytes) is the only required value; everything else has a local default.