Jinwoo Hong
6e46d5ef33
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to 300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip - The flip back holds no concurrency group: GitHub keeps one pending run per group and a newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is re-read and retried (5 tries, backoff) while the switch file still says reserve. - The 503 background comes from the 10 minutes before a required flipped-at, or an explicit value; neither is refused. - A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the exact break-glass inputs. - Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable. - Seat-behind-row key adds the director instance and cell; six segments share one command. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): read the reserve watch's 503 publishing front from every response code A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy cell back within minutes. The 503 read now groups request_count by response code in one query; the newest minute any code has a point for is how far the series is published, so a 503 minute at or before it with no point is zero and minutes past it are left out. Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start in later segments, a pending pre-flip background no longer adds to the unverifiable streak, the flip back retries runtime-status before calling for break glass, and the concurrency rationale lives once in the docs. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read - A flip back whose write or plan never finished (no ID token, a failed login, a silent cell) raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does. - The first segment reads the Cloud SQL rollout lease through the lease action's own reader and refuses to start while any run holds it, or when it cannot be read. - A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the watch ends refused without a write. - The duplicate-seat read takes the per-minute ledger summary in the same query and window; with no fresh summary in it the sample is unverifiable, not zero duplicates. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): page on a refused reserve watch or a failed flip back A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and alert policy page on any such line, per cell, through relay_alert_notification_channels, and the deploy identity gets roles/logging.logWriter in production to write it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * docs(relay): note a force-cancel can skip the reserve-watch flip-back write
Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from cloud/, not the repository root. The source is covered by
the repository's root MIT license.
Packages
packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).apps/relay: the relay server. The same image runs as a director or a cell depending onORCA_RELAY_ROLE.apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.apps/relay-ops: the relay operations console and the incident monitor behindpnpm ops:relay,pnpm incident:relay, andpnpm incident:relay-preflight.apps/pushandpackages/push-contract: the mobile push gateway that holds the APNs key and sends to phones through APNs and FCM, and its wire contract. It is deployed and operated from here but is not part of the relay data path; see docs/push-gateway.md.
Mobile push gateway
apps/push is a separate Cloud Run service from the relay. Phones never hold an
Orca credential for it: the desktop host authenticates with the same X25519
key it uses for the relay, answering an encrypted challenge to mint a 24 hour
session, then registers each paired phone's native push token and asks the
gateway to push. The gateway queues each event as its own notification,
enforces per-host quotas and request limits, and retires a
registration as soon as Apple or Google reports the token unregistered.
Provider push is the only ordinary mobile OS-banner path. The notification
socket is retained only for live dismissal and reconnect tray reconciliation;
it never creates or recovers banners. Desktop notification categories remain
authoritative.
Each delivery is persisted as one notification event. Before deploying an
incompatible queue format, stop all older push gateway revisions and clear only
unpublished push delivery fixtures; no queue preservation or migration is required.
FCM notification messages are inherently collapsible while offline and have a
small concurrent collapse-key budget, so every pending alert is not guaranteed.
Storage follows the relay pattern: PostgreSQL in production, SQLite for tests
and local development. Configure it with ORCA_PUSH_PUBLIC_URL, ORCA_PUSH_FCM_PROJECT_ID,
ORCA_PUSH_DATABASE_URL, the three APNs variables (ORCA_PUSH_APNS_KEY,
ORCA_PUSH_APNS_KEY_ID, ORCA_PUSH_APPLE_TEAM_ID, all three or none), and
optionally ORCA_PUSH_APNS_TOPIC. The FCM credential comes from
the runtime service account, so no key material is configured for Android. See
push gateway operations for deployment and recovery.
Logging is aggregate counters only. Tokens, notification titles, notification bodies, and full host fingerprints never reach a log line.
Infrastructure and operations
infra/terraform: the relay Terraform root. It owns the cells, the director, the shared Cloud SQL instance, DNS, observability, and every GitHub Workload Identity provider the relay workflows authenticate through.backend/holds the per-environment backend configuration andenvironments/the tfvars. Drive it throughpnpm infra:init,pnpm infra:plan, andpnpm infra:apply.dev/scripts: the deploy, capacity, admission, rehome, monitoring, and load scripts the workflows call, plus the contract tests that pin each workflow and Terraform surface. Run them withpnpm test.dev/contractsanddev/fixtures: the checked-in data those contract tests read, including the Terraform root partition.docs/: the relay runbooks, capacity-testing guide, incident-monitor reference, the workflow variable reference indocs/relay-workflows.md, and the push gateway runbook indocs/push-gateway.md.
Workflows
The 25 .github/workflows/cloud-*.yml workflows are the deploy and operate
surface: publish and deploy the director, roll GCE cell capacity, operate Asia
admission and regional rehoming, prove staging capacity, monitor production,
power staging up and down, and deploy the mobile push gateway.
.github/actions/cloud-sql-rollout-lease is the compare-and-swap lease that
serializes rollouts against the shared Cloud SQL instance. Push reuses that
action with its own lease object and deployment concurrency group.
Every one of them is inert. Each top-level job is gated on
vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true', a repository variable that is
unset here, so the two scheduled triggers and every manual dispatch skip
without running a step. Only the repository owner, holding the GCP identities
these workflows authenticate as, can turn them on.
Cloud Verify is not gated. It builds, typechecks, lints, tests, secret-scans,
and validates the relay Terraform on every change under cloud/, and it runs
on fork pull requests, so it configures no backend and holds no credential.
What is not here
The terraform-foundation and terraform-apps roots and the API and auth
services live in the private stablyai/orca-cloud repository. Scripts and
tests that spanned both trees were narrowed to the relay side rather than
carrying a dangling reference.
Local development
cd cloud
pnpm install
pnpm build
pnpm test
pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only
when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17
database, for example:
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
Configuration is read from environment variables validated in
apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32
bytes) is the only required value; everything else has a local default.