Jinwoo Hong
407e9ebc33
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
| Prod | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 |
<!-- /orca-pr-loc -->
## ELI5
Some relay director instances talk to the database about twice as slowly as the rest. Those slow ones are where almost all the single-instance stalls happen. Today the logs can't tell us where an instance is running, so we can't work out why some are slow. This PR adds two numbers to the log line every director already writes every 30 seconds: the zone the instance runs in, and the instance's typical database round-trip time.
## What Changed
- **Before:** the director's `orca_relay_runtime_metrics` line had per-connection statement timings but nothing about where the instance runs. Getting one per-instance speed number meant pooling those connection timings by hand.
- **After:** two new fields on that line.
- `instanceZone`: read once at boot from the Cloud Run metadata server (`instance/zone`). It's only read when `K_SERVICE` is set, so GCE cells and local runs skip it. If the read fails, the field is simply left off.
- `stickyStatementMsP50`: the median duration of every sticky-lane database statement in the window, across all connections. It comes from a new window-wide sample of up to 1024 statements, the same kind already kept per connection.
- **Unchanged:** behaviour. The zone read is fire-and-forget with a 3 s timeout and never blocks or fails boot. No request path reads either field.
## Why
In 7 days of director logs (10-17 to 10-25), each instance's median per-statement time sat at either about 1.7 ms or about 3.6 ms. An instance sometimes switched between the two mid-life, with no deploy. 74 of the 81 single-instance stalls happened while the instance was at the slow speed: about 15x the rate at the fast speed.
- CPU, the event loop, the pg pool and database locks were all ruled out. That leaves the network path between the instance and Cloud SQL, and we can't confirm it without knowing each instance's zone.
- With zone and speed on the same line, a couple of days of logs will show whether "slow" follows the zone or the host. That decides between the next fixes: changing the Cloud Run sandbox generation, using a private IP, or cutting round trips per assignment.
- **Alternative considered:** a boot-time-only zone log line. It would force every analysis to join two log streams, and the field on each line costs a few bytes.
- **Alternative considered:** logging a "fast/slow" label instead of the raw median. That would bake today's thresholds into the image.
## Linked Issue
N/A (internal investigation of director single-instance stalls).
## Visual Proof
N/A: log-only change on a backend service, with no UI.
## Testing
- `pnpm typecheck` for `@orca-cloud/relay`.
- The full relay vitest suite: 1348 passed, 201 skipped (postgres-backed suites skip without a database).
- New tests cover:
- parsing the zone from the metadata answer, and returning nothing on a refused, unreachable or malformed answer;
- the zone appearing only once it is known;
- the pooled statement median.
- The floating-promise census lists the zone read as settling its own failures.
- [ ] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## AI Disclosure
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
- **Wire compatibility:** none affected. These are new optional fields on a log line, and the existing watchers read named fields only.
- **SSH, mobile, desktop:** not affected.
- **Rollout:** rides the next director image. No pin or deploy is part of this PR.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from cloud/, not the repository root. The source is covered by
the repository's root MIT license.
The 2026-10-21 security review records dependency/container scans, remaining upstream advisories, CI coverage, and the validation required before rolling out the updated images and host pins.
Packages
packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).apps/relay: the relay server. The same image runs as a director or a cell depending onORCA_RELAY_ROLE.apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.apps/pushandpackages/push-contract: the mobile push gateway that holds the APNs key and sends to phones through APNs and FCM, and its wire contract. Deployment and operations are owned by the privatestablyai/orca-cloudrepository.
Mobile push gateway
apps/push is a separate Cloud Run service from the relay. Phones never hold an
Orca credential for it: the desktop host authenticates with the same X25519
key it uses for the relay, answering an encrypted challenge to mint a 24 hour
session, then registers each paired phone's native push token and asks the
gateway to push. The gateway queues each event as its own notification,
enforces per-host quotas and request limits, and retires a
registration as soon as Apple or Google reports the token unregistered.
Provider push is the only ordinary mobile OS-banner path. The notification
socket is retained only for live dismissal and reconnect tray reconciliation;
it never creates or recovers banners. Desktop notification categories remain
authoritative.
Each delivery is persisted as one notification event. Before deploying an
incompatible queue format, stop all older push gateway revisions and clear only
unpublished push delivery fixtures; no queue preservation or migration is required.
FCM notification messages are inherently collapsible while offline and have a
small concurrent collapse-key budget, so every pending alert is not guaranteed.
Storage follows the relay pattern: PostgreSQL in production, SQLite for tests
and local development. Configure it with ORCA_PUSH_PUBLIC_URL, ORCA_PUSH_FCM_PROJECT_ID,
ORCA_PUSH_DATABASE_URL, the three APNs variables (ORCA_PUSH_APNS_KEY,
ORCA_PUSH_APNS_KEY_ID, ORCA_PUSH_APPLE_TEAM_ID, all three or none), and
optionally ORCA_PUSH_APNS_TOPIC. The FCM credential comes from
the runtime service account, so no key material is configured for Android. See
push gateway operations for deployment and recovery.
Logging is aggregate counters only. Tokens, notification titles, notification bodies, and full host fingerprints never reach a log line.
Deployment configuration
The relay runtime and wire contracts are public. Production and staging Terraform,
backend configuration, operator tools, runbooks, broker image assembly, and deployment
workflows live in the private stablyai/orca-cloud repository. That repository pins
an immutable public source revision and combines it with its deployment configuration.
Cloud Verify here builds and tests public runtime code, audits dependencies, scans
relay and push images, and checks for secrets without cloud credentials. Private CI
also validates Terraform, scans the SQL proxy and broker, and runs deployment contracts.
Local development
cd cloud
pnpm install
pnpm build
pnpm test
pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only
when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17
database, for example:
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
Configuration is read from environment variables validated in
apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32
bytes) is the only required value; everything else has a local default.