mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Jinwoo Hong
e068947d4c
* feat(relay): alert on far-cell placement and skewed region hints US desktops were homed on asia-east2 cells for weeks in 2026-08 with every existing relay alert green. Roughly 226 of 332 hosts on those cells were non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but nothing in Cloud Monitoring could see distance: the connection, queue, heap, and SQL bars all measure a cell's own health, which was fine. Three policies close that gap. Two read distance per cell, from the accept and control-RTT timing added in the parent commit: phone-accept p95 above 2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide, as the asia-east2 share of the region hints desktops send the director, so a mis-picking client probe is visible before it lands anyone on a far cell. All three are MQL rather than the metric filters the other relay policies use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can only align one with a percentile; each alert needs the sum of the extracted values as a volume floor so a sparse window cannot page. None of these metrics exists in the project yet, so what was checked against production is the query shape: the same MQL run over existing metrics of the same kind. The skew denominator needs one log-based metric per hint key, so `requestedRegionsDelta` now has one per relay region plus the unhinted bucket. Those ride the existing snapshot metric family, which adds map entries without touching the live metrics. A ratchet test pins the key list to relay-contract's RELAY_REGIONS: a region added there without a metric would shrink the denominator, so the test fails rather than letting the share quietly inflate. * fix(relay): compare hinted regions against placed ones, not a fixed share Review found the skew alert inverted at both ends. A fixed 40% bar on the asia-east2 share of region hints was silent through the exact broken state it was written for, and would page forever once the desktop probe is fixed and the genuine APAC share rises past it. An absolute share cannot separate those because it has no reference point. The hint share now has one: the share of assignments the director actually placed in that region during the same hour. Measured over twelve hours on 2026-09-07, while the probe was still mis-picking, asia-east2 was 33.8% of 33,800 hinted requests and 7.9% of 45,364 assignments. That is a 4.27x divergence and a 25.9-point gap, so the alert fires above 2x and 15 points, inside the broken state and outside a healthy one. Both bars must hold: the ratio alone blows up on tiny placement counts, the gap alone misses a proportionally large skew at low volume. The reviewer proposed either bar alone; requiring both keeps each one meaningful and still clears today's numbers with room. `unhinted` requests leave the denominator. They were 27% of all requests, so a client that always sends a hint would move the number from 21.9% to 35.0% with no behaviour change at all. The comparison needs per-region placement counters, so `selectedRegionsDelta` gets log-based metrics alongside the requested ones. Rather than extract four hyphenated map keys through quoted field paths, which nothing in the project does and which cannot be checked without applying, the relay now also publishes flat `requestedRegion<Region>Delta` and `selectedRegion<Region>Delta` fields next to the untouched maps. They are emitted as zeros in every interval, so no series can drop out of the alert's inner join in an hour with no asia placements, which is exactly the hour the skew is worst. Additive only: metricVersion is unchanged, the maps still carry anything outside the catalog, and the emitter's leak guard still passes. Two corrections to what the previous commit claimed. None of these metrics exist in the project yet, so the code, the doc and this message now say what was actually checked against production: the query shapes, run over existing metrics of the same kind. And the control-RTT policy records that EU desktops on us-central1 sit at 100-130 ms, so a European-heavy cell can approach the 150 ms bar while correctly homed. The skew alert will stay lit after a client fix until the backlog is rehomed. Sticky assignment never re-consults the hint, so a desktop already on an asia cell keeps landing there whatever it now asks for. The policy description and the doc both say so, so nobody reads a slow clear as a failed fix. * fix(relay): cross-multiply the skew bars so a zero placement share still fires `hint_share / placement_share` is undefined in the hour that matters most. When the director placed nobody in the region, MQL returns no rows for either 0/0 or x/0, so the series disappears before the gap and volume clauses run and the alert stays silent. That hour is not hypothetical: it is every desktop asking for a region while the director puts nobody there, which is what a drained, fenced, or full region looks like, and it is the most extreme skew the alert can see. The condition is now cross-multiplied, `hint_share > 2 * placement_share`, which is well defined at zero. Both forms were run read-only against production surrogates chosen so the placement denominator is exactly zero: the ratio form returned no rows, the cross-multiplied form returned the series with the condition true on every point. A second surrogate pass with a tiny hint share returned the series with the condition false, so the gap clause still suppresses the healthy shape rather than the query silently matching everything. The flat field names are no longer derived on either side. Terraform title cased each dash-separated part and the emitter upper cased each part's first character, so the ratchet had to pin two source expressions by regex, which a reformat would break and which never compared the actual rendered names. Both sides now declare a literal map, relay-contract's RELAY_REGION_METRIC_SEGMENTS and Terraform's relay_region_field_segments, and the test compares the two declarations against each other and against the expected names. `satisfies Record<RelayRegion, string>` makes a region added without a segment a compile error rather than a silent gap in the alert's denominators. Both ratchets were checked by mutation: a wrong Terraform segment, a contract region with no Terraform entry, and a revert to the ratio form each fail the node test, and the new region fails the contract build.
Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and desktops never talk to each other directly: each opens an outbound WebSocket to a relay cell, the relay pairs the two sessions, and it splices frames between them. A director assigns hosts to cells and coordinates migrations; cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from cloud/, not the repository root. The source is covered by
the repository's root MIT license.
Packages
packages/relay-contract: the wire contract shared by the relay, the desktop app, and the mobile app (frame shapes, close codes, admission budgets, splice state machine).apps/relay: the relay server. The same image runs as a director or a cell depending onORCA_RELAY_ROLE.apps/relay-fence-broker: a private, IAM-only service that owns the durable mutation lease, the Terraform checkout, and the narrow Compute mutation used when a registered target is superseded. The workflow that calls it holds read and invoke rights only, never those mutation permissions.apps/relay-ops: the relay operations console and the incident monitor behindpnpm ops:relay,pnpm incident:relay, andpnpm incident:relay-preflight.
Infrastructure and operations
infra/terraform: the relay Terraform root. It owns the cells, the director, the shared Cloud SQL instance, DNS, observability, and every GitHub Workload Identity provider the relay workflows authenticate through.backend/holds the per-environment backend configuration andenvironments/the tfvars. Drive it throughpnpm infra:init,pnpm infra:plan, andpnpm infra:apply.dev/scripts: the deploy, capacity, admission, rehome, monitoring, and load scripts the workflows call, plus the contract tests that pin each workflow and Terraform surface. Run them withpnpm test.dev/contractsanddev/fixtures: the checked-in data those contract tests read, including the Terraform root partition.docs/: the relay runbooks, capacity-testing guide, incident-monitor reference, and the workflow variable reference indocs/relay-workflows.md.
Workflows
The 24 .github/workflows/cloud-*.yml workflows are the relay's deploy and
operate surface: publish and deploy the director, roll GCE cell capacity,
operate Asia admission and regional rehoming, prove staging capacity, monitor
production, and power staging up and down. .github/actions/cloud-sql-rollout-lease
is the compare-and-swap lease that serializes every rollout against the shared
Cloud SQL instance.
Every one of them is inert. Each top-level job is gated on
vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true', a repository variable that is
unset here, so the two scheduled triggers and every manual dispatch skip
without running a step. Only the repository owner, holding the GCP identities
these workflows authenticate as, can turn them on.
Cloud Verify is not gated. It builds, typechecks, lints, tests, secret-scans,
and validates the relay Terraform on every change under cloud/, and it runs
on fork pull requests, so it configures no backend and holds no credential.
What is not here
The terraform-foundation and terraform-apps roots and the API and auth
services live in the private stablyai/orca-cloud repository. Scripts and
tests that spanned both trees were narrowed to the relay side rather than
carrying a dangling reference.
Local development
cd cloud
pnpm install
pnpm build
pnpm test
pnpm test runs the SQLite-backed suites. Tests that need PostgreSQL run only
when ORCA_RELAY_TEST_POSTGRES_URL points at a disposable PostgreSQL 16 or 17
database, for example:
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
Configuration is read from environment variables validated in
apps/relay/src/config.ts. ORCA_RELAY_ASSIGNMENT_SIGNING_KEY (at least 32
bytes) is the only required value; everything else has a local default.