* feat(cloud): add the mobile push gateway and its contract package (#8129)
A small open-source service that holds the APNs key and FCM credentials and
sends background push to paired phones on the desktop's behalf. Hosts
authenticate with a box challenge and HMAC proof on their pairing key, the
same shape the relay uses, so signed-in and accountless desktops share one
path. Tokens are stored; alert text is held only for the coalescing window.
The contract doc in docs/reference is the source of truth for every wire
shape. The interop test runs the real desktop answerer against a real
gateway-issued challenge so transcript drift fails in CI.
* feat(push): register phones and send background push from the desktop (#8129)
Adds the notifications.remote-push.v1 capability, the registerPush and
unregisterPush RPCs on the mobile allowlist, a gateway client with a cached
session and 401 re-auth, a durable unregister outbox, and a dispatcher that
offers every mobile notification to the gateway after the socket fan-out.
The dispatcher is fire-and-forget with one retry and drops registrations the
gateway reports dead.
Puts agentState on the mobile frame and fixes the #4375 wording so a working
agent is never announced as finished. The relay host-proof code moves onto a
shared envelope module with no behaviour change.
* feat(mobile): background push registration, receive, and settings (#8129)
Fetches the native APNs or FCM token, registers it with every paired host
that advertises the capability, and re-registers on token change. Foreground
pushes are suppressed inside handleNotification against the same seen set
the socket path uses, so nothing shows twice. Taps route by host fingerprint.
One Background notifications switch, off by default, with the disclaimer and
needs-input / finished sub-switches; hidden until a paired desktop is new
enough. Adds google-services.json and the expo-notifications plugin.
* chore(cloud): Terraform and deploy workflow for the push gateway (#8129)
Declares the Cloud Run service, runtime account, secrets, and orca_push
database behind push_gateway_enabled, true only in production. The deploy
workflow is gated like the relay's, deploys with no traffic, probes /ready
and a validate-only FCM send, then shifts traffic. It runs as the shared
production deploy account because the Cloud SQL rollout lease grant is
foundation-owned; its extra authority is three bindings on the push service.
docs/push-gateway.md carries the import commands for the resources created
by hand and the APNs key rotation procedure.
* docs: describe background notifications on the phone (#8129)
* docs: check in the mobile push contract (#8129)
Seven committed files cite it as the source of truth for every wire shape;
docs/reference is allowlisted per file, so add the entry.
* test(push): replay one checked-in host-proof vector on both sides (#8129)
Cloud Verify installs only the cloud workspace, so the gateway suite cannot
import the desktop answerer. Replace the cross-workspace import with a fixed
challenge vector generated from the contract package; the gateway fixture and
the desktop answerer each replay it and must produce the same HMAC. A
transcript drift on either side now fails in that side's own suite.
* fix(cloud): open the push gateway with invoker_iam_disabled, not an allUsers binding (#8129)
The production domain-restricted-sharing policy rejects an allUsers
run.invoker member, which the runbook anticipated. Opt the service out of
invoker IAM the way the relay director already does; the host proof is the
authentication either way.
* docs(cloud): the push.onorca.dev record exists and is hand-managed (#8129)
* fix(push): close review findings in the gateway (#8129)
- Quota reservation takes a per-host advisory lock; READ COMMITTED admitted
a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16).
- Challenge issuance no longer writes push_hosts; the row lands on proof
verification. Stale hosts prune after 30 days. Per-IP token bucket on the
two unauthenticated routes.
- Streaming body limit via hono bodyLimit; a chunked body bypassed the
Content-Length check.
- registrationIds deduped in the schema; per-host device cap of 64; list
bounded to its schema.
- Gateway-side challenge TTL is the specified 10 s, not 40 s.
- APNs stream settles on close as well as end/error.
* fix(push): close review findings in the desktop client (#8129)
- A gateway registration the registry cannot persist is enqueued for delete
instead of leaking a live token.
- Unregister outbox re-reads pending per pass, honours enqueues during a
drain, and retries with backoff instead of waiting for the next launch.
- Dispatcher batches registrations by 20 rather than starving the rest.
- 401 compare-and-clear; a 401 after re-auth is unreachable; refused
handshakes and 429s are cached briefly instead of re-handshaking per event.
- Service is stopped on quit.
* fix(mobile): close review findings in push registration and receive (#8129)
- Consent generation guards a register that finishes after the switch went
off; the host is re-queued for unregister instead of recorded live.
- Foreground pushes seed the watermark before adopting the epoch, so a push
on a never-connected session cannot wipe a valid watermark.
- aps-environment follows the build via app.config.js; the iOS release
workflow sets it to production. A bare plugin entry wrote development.
- Pushes the OS showed while closed are marked seen before catch-up replay.
- Token null result is not cached; failed capability probes are retried and
never block an unregister; coalesced summaries are shown but not marked.
- Unresolvable fingerprint routes nowhere and is suppressed in foreground.
- Android channel ensured at boot; capability hook diffs clients by identity.
* fix(cloud): harden the push deploy workflow and size the gateway to the budget (#8129)
- Roll traffic back on a failed post-shift check; delete a candidate that
never took traffic; retry the origin probe and the FCM probe.
- Assert Terraform-owned scaling instead of mutating it from the workflow.
- Build before taking the Cloud SQL rollout lease.
- Declare the database pool in Terraform (2 per instance, max 2 instances)
and add the gateway to the connection budget; the previous default put the
shared instance 65 connections over its ceiling.
- State plainly that the shared deploy identity's relay authority is inherited.
* fix(push): read the runtime from shared state at push startup (#8129)
Threading the runtime through launchDesktopMode put the launch module one
line over the 300-line lint budget after the rebase.
* fix(push): key the unauthenticated rate limit on the hop Cloud Run wrote (#8129)
Cloud Run appends the connecting peer to x-forwarded-for; the limiter read
the left-most value, which the caller controls, so a forged first hop earned
a fresh bucket per request.
* fix(push): close the final security review findings in the gateway and infra (#8129)
- app.onError logs only the error name and answers a bare 500; hono's default
handler printed the whole error, and a pg error carries the row in detail
- a second per-IP bucket (240/min) runs ahead of the bearer lookup on every
authenticated route, so forged bearers cannot spend the two-connection pool
- one live session per host: minting deletes the host's earlier row
- device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free
- notificationId is printable ASCII, since it becomes the APNs collapse header
- the impersonated FCM probe token is masked in the workflow log
- prevent_destroy on the Apple secrets and the orca_push database
* fix(push): close the final security review findings in the desktop client (#8129)
- fetch never follows a redirect: a 307 would replay the host proof and the
phone's token to whatever origin the redirect named
- registerPush params are strict and the paired identity is spread last
- a per-device bucket (10/min) bounds a phone looping registerPush, which
costs a gateway write and a synchronous registry write each time
* fix(mobile): close the final security review findings in push receive (#8129)
- a push with no epoch can no longer claim a seq-derived dedup key, in the
foreground or from the tray; a forged seq:N could otherwise swallow the
real bell at that seq
- a provider-delivered push with no host catalog, or no fingerprint at all,
stays unrouted instead of falling back to the hostId its raw data carries
* docs(push): record the ip buckets, session and host retention, and the token-ownership limit (#8129)
* fix(push): apply the schema on an untimed pool and retry statement-timeout aborts (#8129)
Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection
pool with statement_timeout 0 that is closed before the serving pool opens, and
SQLSTATE 57014 joins the bounded transaction retry path.
* fix: harden mobile push delivery and deployment recovery
* feat: align mobile notification preferences with desktop delivery
* fix: accept variable-length APNs device tokens
* fix: deduplicate native APNs and background socket notifications
All 19 general cells on 4916ed67, selector gen 148 -> 186, 0 serving-process
exits across the roll. Three waves used the no-restart mode=rollback resume
(c13 transient trust-probe 409; c26/c21 post-apply runtime-status 503 shed).
Checklist: 2.3, 4.1, 4.3 relay side deployed; status header 2026-09-06.
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease
The accept runs several serialized Postgres calls behind the contended
cell-inventory lock, and phones bound their dial. Finishing that work for a
phone that had already left acquired (and leaked for 90s) an activity lease and
then failed at bind with host_data_reservation_already_bound. Check the client
socket between the DB steps and unwind what was taken, reporting the stage on
orca_relay_client_accept_abandoned.
Jitter the control lease grant so a cohort that reconnected in the same minute
(a cell recreate dumps hundreds at once) walks apart instead of rebinding
together every cycle.
On the phone, treat a probe session that enters 'reconnecting' as a failed
probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting
it out held the supervisor's operation mutex for the full 12s bound.
* perf(relay): lengthen the control lease to 6h
The lease bounds how long a host lingers on a cell after a missed drain, and
rebinding it is the only passive rebalancing we have, so it stays finite. 6h
keeps both properties while cutting control-activation traffic on the contended
cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the
75s silence watchdog are enforced separately, so the longer grant authorizes
nothing extra. The jitter widens with it, to +/-30 min.
* fix(relay): let one flap recover the direct probe; correct the leak window
'reconnecting' is published on any socket close, so rejecting on it outright
turned a single access-point flap into a booked direct failure and a 60s
cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected'
transition still resolves; a dead LAN still fails in ~2s rather than holding the
supervisor's operation mutex for the 12s bound.
The abandoned accept held its activity lease for the 10s attach deadline, not
90s -- the attach timer is armed before bind throws and already unwinds it.
Also cover the assignment-stage check that guards reserveCredential, and drop a
spread assertion the two exact-value assertions above already imply.
* fix(relay): extend the probe grace once on a handshake; pin the lease band top
The redial fires at 500ms but 'connected' waits on the Noise handshake and a
capability RPC, so one 2s window is too tight for real work. A 'handshaking'
transition is evidence the peer answered, so extend the grace once; a stalled
handshake still fails at ~3.5s, far inside the 12s bound.
The longest-lease case only had an upper bound, which a jitter clamped to one
side would satisfy. Pin it to the exact top of the band instead, and assert the
assignment resolve ran so the third-guard test cannot pass vacuously.
Operator record for the 2026-09-04 relay reconnect incident and the Roll 1
same-cap cell image roll (complete 2026-09-05, selector gen 148), plus the
follow-up checklist, roadmap, and the Roll 2 implementation plan.
Docs only; split out of #18565 so the record merges independently of the code.
The incident monitor froze three healthy 15-minute production gates on
2026-09-05 because asia-east2 cells are judged against a bar calibrated
for us-central1. A cell's /ready fetches the auth JWKS and runs SELECT 1
against Cloud SQL, both in us-central1, so from the US GitHub runner the
asia-east2 round trip measures p50 0.88 s / max 2.7 s against 0.08-0.5 s
for us-central1 cells.
Give cell.<id>.latency_ms a per-region threshold (us-central1 2000,
asia-east2 4000) carried on IncidentCellExpectation from the tfvars
region. Director and auth latency rules keep the flat 2000 bar, and hard
faults are still caught by the .health/.ready equal-1 checks and the
probe's 8 s fetch timeout.
Also name the signal and its observed/threshold in the live preflight
failure message, keeping the source/code tokens other tooling matches on.
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells
c27-c29), but the canary script it drives hard-rejected anything outside the
16 US capacity cells, so the first Asia same-cap canary failed closed at
isolate. Give the canary an explicit --approved-cells switch that selects the
same-cap allowlist, and pass it from the four same-cap job invocations. With no
switch the behaviour is unchanged, so the US-only capacity workflow keeps its
scope.
The live metrics have role and cell_id only. A label change on a log metric
is delete+create, so applying the region label would replace all 21 metrics,
reset their history, and blank the relay alert policies during the swap.
Matching Terraform to live state makes the targeted plan create-only (8
renewal metrics never applied, plus the incident dashboard from #18717).
The sticky refresh path and reservation reconciliation both took the fleet-wide
`relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell
queued unrelated reconnects and migration completions behind it. Both now lock
only the rows they touch, in the same ascending cell_id order, and the sticky
grant moves its counter by a delta instead of writing back a snapshot value.
Placement keeps the ordered inventory lock: choosing the least-loaded cell is a
genuinely fleet-wide decision, and dynamically locking only the selected target
is what allowed cross-cell cycles before.
The pool's statement_timeout becomes env-configurable and a 57014 now reaches
the bounded transaction retry instead of surfacing as a terminal failure.
Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended
before the serving pool opens, so a slow CREATE INDEX cannot inherit a request
deadline it will never fit inside.
The active probe recorded a network-layer failure as `false`, so a single
thrown fetch on the runner froze the production monitor at `auth.health=0`
while the service answered 200 throughout. It also required `/ready` on the
auth endpoint, which serves no such path, forcing every auth sample onto the
11s retry.
A thrown fetch now means "no reading" and is re-asked once after 1s; only a
second throw, or a non-ok response, yields false. `requiresReady` is threaded
per endpoint (director and cells true, auth false). Latency now measures the
answering round trip rather than probe wall time, so retry delays are not
reported as serving latency.
Cells run cloud-sql-proxy against the auth database's public IP, so every
connection burns a Cloud NAT port on the relay gateway; that allocation filled
on 2026-09-04 and every cell's proxy dial timed out at once. Add --private-ip
behind relay_cloud_sql_private_ip so production can move the traffic onto the
VPC peering once the foundation root has applied it.
Default false, and the rendered startup script is byte-identical to main with
that default, so merging rolls nothing. --unix-socket is untouched: it selects
the listener, not the upstream address, so DATABASE_URL does not change.
The director is Cloud Run and egresses outside this VPC's NAT, so it is not
part of the problem; moving it would mean VPC egress plus a TCP DSN and its own
secret, which is a separate change.
201 relay cell process exits over 48h on 2026-09-04 paged nobody. Adds a log metric on the Docker `container die` event for the orca-relay container, an alert at >3 exits per instance per 15 min, and a four-chart incident dashboard covering the signals that had to be assembled by hand during the outage.
On 2026-09-04 an auth outage signed ~21,600 desktops out of Orca Cloud and
every paired phone showed the generic "Can't reach desktop" for hours. The
desktop knew why, the cell watched it happen, and neither could say so.
The desktop now names auth loss on its control close reason; the cell
remembers that reason per (userId, relayHostId) and replays it as the close
reason of the 4404 it already sends a phone whose host is absent; the phone
turns it into "Desktop signed out — sign in to Orca on your desktop to
reconnect". Retry cadence, close codes and every message body are untouched.
The reason rides the WebSocket close reason because there is no additive JSON
channel to a shipped phone: RelayPhoneHelloSchema, RelayAuthSchema and the
director's ResolveResponseSchema are all zod .strict(), and /v1/connect
rejects any query string outright. A new close code was also rejected — an old
phone would fall out of mobileRelayRecoveryFor and off the 5-15s host-offline
backoff onto the faster transport backoff.
The cell keeps the reason in memory rather than Postgres: a phone reaches the
cell its host's assignment row already names, which is the cell that saw the
close, and losing it on a cell restart degrades to today's verdict rather than
a wrong one.
* feat(relay-infra): dynamic NAT ports and alerts for the 2026-09-04 stall signals
Relay cells reach Cloud SQL's public IP through Cloud NAT. The static default
of 64 ports per VM filled on 2026-09-04 and every cell's proxy dial timed out
at once, which read as a fleet-wide SQL stall against a healthy database.
Switch both regional NATs to dynamic port allocation (64..4096 per VM).
Add the three alerts that would have paged inside the first ten minutes:
- Cloud SQL WAL-triggered checkpoint loop (log metric on
"checkpoint starting: wal", > 3 in 5 min)
- Cloud SQL disk utilization > 70%
- Cloud NAT OUT_OF_RESOURCES packet drops on the relay gateways
No workflow applies google_monitoring_* or the NAT resources today; every
relay workflow is target-scoped to cells. Apply is a reviewed targeted plan
(see PR body).
* test(cloud): register the three new relay alert policies in the root partition fixture
* fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths
activateControl, acquireActivity, changeActivity and
removeSupersededSameCellControls each adjust exactly one cell's
reservation, yet took SELECT * FROM relay_cells FOR UPDATE, so every
desktop rebind and phone reconnect in the fleet queued behind every
other one and behind placement. They now use the single-row atomic
update (or lock only their own cell row), leaving the inventory lock to
placement and sweeps.
Fleet-wide 55P03 retries ran p50 430 / p99 1320 per five minutes on
2026-09-03, every cell pinned sqlLatencyMsMax at the lock timeout, and
the old cell image crashed on the resulting pool timeouts ~every 15
minutes. A real-Postgres test holds another cell's row and asserts a
rebind proceeds; re-adding the inventory lock fails it.
* fix(relay): lock the touched cell rows in order on cross-cell activity moves
Review found that acquireActivity's existing-lease branch could lock the
old lease's cell row (via removeActivityLease) before the new cell's row,
which cycles with placement's ascending inventory lock; reproduced on
real Postgres as paired 55P03 retries. lockCellRows now takes the one or
two rows a per-connection path touches in cell_id order with the 500 ms
request bound, and the census fails on any inline relay_cells FOR UPDATE
outside the named lock helpers. A three-cell Postgres test moves an
activity from the highest cell to a lower one while the target row is
held and asserts the mover holds nothing else; five revert-mutants
(inventory lock on each path, dropped ordering, dropped ORDER BY) fail it.
* test(relay): make the inline relay_cells lock census scan whole statements
Review showed two evasions: a FOR UPDATE inside query() and a queryLocked
whose FROM relay_cells sat past a fixed line window. The guard now matches
every query()/queryLocked() template statement in full; both evasions
fail it. Also clears relay_cell_connection_snapshots in the connection-
headroom Postgres suite so an aborted run does not poison the next.
The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.
* fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar
The pre-drain dry-run froze at minute one on relayPostgresRetryExhausted: 0
in every run since #18521 reached the director, blocking the cell roll that
carries the same fix. #18521 made contended request-path waiters fail fast
(500 ms) instead of succeeding slowly, so exhaustion is now a steady
contention rate: 236/236 five-minute windows non-zero over 23 h; post-#18521
p50 42 / p90 147 / max 220 fleet-wide; the 2026-08-23 incident peaked at 467.
300 clears every measured healthy window and stays under the incident shape.
/v1/assign 503 share was unchanged by #18521 (13.9% vs 12.3%).
* test(cloud): pin the exhausted-retry freeze boundary at exactly 300
* docs(cloud): reword relay comments that still described the zero exhausted-retry bar
decodeURIComponent on the /v1/connect/ and /v1/host/data/ path segments threw
URIError out of the http 'upgrade' listener, which is uncaught and kills the
relay process. Any client that sends GET /v1/connect/% could take down a cell
(and every connection on it) or a director instance. Pre-existing since the
splice landed (orca-cloud #20); not introduced by the import.
A malformed escape now takes the existing 4xx reject branch. The blackbox test
sends three malformed connect targets and one host-data target to the real
server and asserts no uncaughtException fires and a well-formed upgrade still
gets 101 afterwards; reverting either site fails it.
* fix(cloud): retry the committed-winner collision codes in relay schema startup
`CREATE TABLE IF NOT EXISTS` only checks the name before the catalog inserts, so
the loser of a concurrent CREATE fails in one of two ways depending on timing:
on the catalog unique index (23505, which the startup retry already handled) or,
when the winner has committed by the time the loser reaches TypeCreate /
heap_create_with_catalog, on the name check those routines repeat (42710
duplicate type, 42P07 duplicate relation). The predicate treated the latter as
fatal, so a director could fail startup on a table it was about to find present.
This is what turned `postgres-schema-concurrency-postgres.test.ts` red on main
and on every relay PR (CI's shared runner loses the race more often than a dev
box): a throwaway diagnostic run in CI reported 42710 from TypeCreate and 42P07
from heap_create_with_catalog as the only rejection reasons.
Treat 42710/42P07 as retryable for `CREATE TABLE IF NOT EXISTS` and 42P07 for
`CREATE [UNIQUE] INDEX IF NOT EXISTS`; every other statement shape still fails
fast. The concurrency test now runs ten rounds and reports the loser's SQLSTATE
instead of a bare boolean.
* chore(cloud): allowlist the RFC 6455 example Sec-WebSocket-Key for upgrade tests
Cloud Verify's Secret scan runs gitleaks over --all refs, so the raw-socket
upgrade test on fix/relay-upgrade-malformed-uri (#18547) trips every cloud PR's
scan until its allowlist reaches main. Land the allowlist here first.
The five `node ... | tee` steps in cloud-operate-relay-production-rehome-job.yml
reported tee's exit code, so a thrown inspect or apply passed green. The Aug 28
21:25Z and Aug 29 inspects and today's first inspect all printed
"director returned an invalid regional rehome control" (the durable control had
moved to generation 12 when the Aug 28 rehome aborted) and still succeeded.
`shell: bash` adds `-o pipefail`. A test pins the default and the tee count.
Mirrors stablyai/orca-cloud#471 (squash c3354e8), byte-identical under cloud/.
The relay's cell-inventory lock (SELECT ... FROM relay_cells FOR UPDATE over
all 23 rows) is one global critical section shared by the assignment hot path
and every director sweep; with the pool's 1s lock_timeout a blocked waiter held
a pooled client for a full second, producing ~690 55P03 retries per 5 minutes
in production. Request paths now bound the wait at 500ms with a SET LOCAL that
is restored to the pool default before the next statement; director-only sweeps
take the lock NOWAIT and skip the tick; sweep timers are jittered; hold time is
exported as additive runtime-metrics fields so the bound can be tuned.
Mirrors stablyai/orca-cloud#470. The private relay workflows are retired, so
the dual accept has one live arm left. Add `github_workflow_file_prefix` for
the primary repository's workflow filenames, point `github_repo`/
`github_repo_id` at `stablyai/orca` (`1183888342`), and empty
`github_accepted_repositories` in both environments. Every relay provider goes
back to a single arm naming `cloud-` prefixed workflow refs.
`cloud/infra/terraform` stays byte-identical to the private branch. The two
identity tests diverge here as they already did, so they take the same change
rather than the same bytes: both now render the trusted ref head from the
Terraform variable instead of this checkout's own workflow filenames, which is
what lets the length pin be the same 791 characters in either repository.
* chore(cloud): pin staging relay c3 to the director's image
Mirrors stablyai/orca-cloud#468. c3 stayed on sha-c91439af after the
director and c4 moved to sha-e3e92d95, so the staging capacity proof's
compatible-director-image check has failed since 2026-08-14.
* test(cloud): scope the launch-image pin to staging C4 now that C3 shares the digest
* test(cloud): keep the public workflow assertions; scope only the launch-image pin to C4
The relay root does not declare it (it belongs to the private apps root), and
Terraform rejects an undeclared -var, so the first public Deploy Relay Staging
run failed at the C4 image bind.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong