Files
orca/cloud
Jinwoo Hong 818e7829f6 feat(relay): split step-5 shadow compare and dry-run logs per cell (#28693)
* feat(relay): split step-5 shadow compare and dry-run logs per cell

The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet

- countByCell (was cells) counts a compare on every cell it involves: the
  database's cell, every seat's cell when there is no row, and both cells
  for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): count a duplicate-seat compare on every seat cell

An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): key per-cell counts on prototype-free records

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): explain a duplicate seat left by a just-superseded half-open control

When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): bound the superseded duplicate-seat rule on the move's age

A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): explain a superseded duplicate seat only from a prior sighting

compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): also bound a superseded duplicate seat on its join age

A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): count an ageless superseded duplicate seat as its own class

A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): limit superseded-unknown-age to an instance's first 100 s

Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* docs(relay): say where a missing sighting comes from after the first 100 s

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
2026-10-14 19:15:04 -04:00
..

Orca Relay

The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.

This directory is an independent pnpm workspace inside the Orca monorepo. Run its commands from cloud/, not the repository root. The source is covered by the repository's root MIT license.

Packages

  • packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).
  • apps/relay: the relay server. The same image runs as a director or a cell depending on ORCA_RELAY_ROLE.
  • apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.
  • apps/relay-ops: the relay operations console and the incident monitor behind pnpm ops:relay, pnpm incident:relay, and pnpm incident:relay-preflight.
  • apps/push and packages/push-contract: the mobile push gateway that holds the APNs key and sends to phones through APNs and FCM, and its wire contract. It is deployed and operated from here but is not part of the relay data path; see docs/push-gateway.md.

Mobile push gateway

apps/push is a separate Cloud Run service from the relay. Phones never hold an Orca credential for it: the desktop host authenticates with the same X25519 key it uses for the relay, answering an encrypted challenge to mint a 24 hour session, then registers each paired phone's native push token and asks the gateway to push. The gateway queues each event as its own notification, enforces per-host quotas and request limits, and retires a registration as soon as Apple or Google reports the token unregistered. Provider push is the only ordinary mobile OS-banner path. The notification socket is retained only for live dismissal and reconnect tray reconciliation; it never creates or recovers banners. Desktop notification categories remain authoritative. Each delivery is persisted as one notification event. Before deploying an incompatible queue format, stop all older push gateway revisions and clear only unpublished push delivery fixtures; no queue preservation or migration is required. FCM notification messages are inherently collapsible while offline and have a small concurrent collapse-key budget, so every pending alert is not guaranteed.

Storage follows the relay pattern: PostgreSQL in production, SQLite for tests and local development. Configure it with ORCA_PUSH_PUBLIC_URL, ORCA_PUSH_FCM_PROJECT_ID, ORCA_PUSH_DATABASE_URL, the three APNs variables (ORCA_PUSH_APNS_KEY, ORCA_PUSH_APNS_KEY_ID, ORCA_PUSH_APPLE_TEAM_ID, all three or none), and optionally ORCA_PUSH_APNS_TOPIC. The FCM credential comes from the runtime service account, so no key material is configured for Android. See push gateway operations for deployment and recovery.

Logging is aggregate counters only. Tokens, notification titles, notification bodies, and full host fingerprints never reach a log line.

Infrastructure and operations

  • infra/terraform: the relay Terraform root. It owns the cells, the director, the shared Cloud SQL instance, DNS, observability, and every GitHub Workload Identity provider the relay workflows authenticate through. backend/ holds the per-environment backend configuration and environments/ the tfvars. Drive it through pnpm infra:init, pnpm infra:plan, and pnpm infra:apply.
  • dev/scripts: the deploy, capacity, admission, rehome, monitoring, and load scripts the workflows call, plus the contract tests that pin each workflow and Terraform surface. Run them with pnpm test.
  • dev/contracts and dev/fixtures: the checked-in data those contract tests read, including the Terraform root partition.
  • docs/: the relay runbooks, capacity-testing guide, incident-monitor reference, the workflow variable reference in docs/relay-workflows.md, and the push gateway runbook in docs/push-gateway.md.

Workflows

The 25 .github/workflows/cloud-*.yml workflows are the deploy and operate surface: publish and deploy the director, roll GCE cell capacity, operate Asia admission and regional rehoming, prove staging capacity, monitor production, power staging up and down, and deploy the mobile push gateway. .github/actions/cloud-sql-rollout-lease is the compare-and-swap lease that serializes rollouts against the shared Cloud SQL instance. Push reuses that action with its own lease object and deployment concurrency group.

Every one of them is inert. Each top-level job is gated on vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true', a repository variable that is unset here, so the two scheduled triggers and every manual dispatch skip without running a step. Only the repository owner, holding the GCP identities these workflows authenticate as, can turn them on.

Cloud Verify is not gated. It builds, typechecks, lints, tests, secret-scans, and validates the relay Terraform on every change under cloud/, and it runs on fork pull requests, so it configures no backend and holds no credential.

What is not here

The terraform-foundation and terraform-apps roots and the API and auth services live in the private stablyai/orca-cloud repository. Scripts and tests that spanned both trees were narrowed to the relay side rather than carrying a dangling reference.

Local development

cd cloud
pnpm install
pnpm build
pnpm test

pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17 database, for example:

docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
  -e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
  pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg

Configuration is read from environment variables validated in apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32 bytes) is the only required value; everything else has a local default.