orca/cloud/docs/push-gateway.md

Orca mobile push gateway

orca-cloud-push is a public Cloud Run service in onorca-cloud that turns a desktop notification into an APNs or FCM push for a paired phone. The desktop registers each phone's native token with it and calls POST /v1/send after the socket fan-out it already does; the phone treats APNs/FCM as the sole ordinary OS-banner path. The notification socket is retained only for live dismissal and reconnect tray reconciliation; it does not create or recover banners. Desktop notification categories remain authoritative. The service is the only place the Apple .p8 signing key is readable, which is the reason it exists as a service at all.

The request schemas live in packages/push-contract/src/. This document covers Terraform ownership, deployment, credential rotation, and recovery.

There is no staging push gateway. That is a decision, not an omission. push_gateway_enabled is false in environments/staging.tfvars and true in environments/production.tfvars, and every resource in infra/terraform/push-gateway.tf is behind it. A staging gateway would be a tfvars edit plus a second set of Apple credentials.

Shape

Setting	Value	Where
Cloud Run service	orca-cloud-push	push_cloud_run_service_name
Region	us-central1	region
Instances	min 1, max 2	push_min_instances, push_max_instances
Database pool	2 per instance	push_database_pool_max
Concurrency	80	push_concurrency
Ingress	all	INGRESS_TRAFFIC_ALL
Invoker	IAM disabled	invoker_iam_disabled = true on the service
Runtime identity	orca-cloud-push@onorca-cloud.iam.gserviceaccount.com	google_service_account.push_runtime
Database	orca_push on dedicated HA PostgreSQL 17	google_sql_database.push_dedicated
Hostname	push.onorca.dev	push_base_url

The minimum of one instance is deliberate and did not move when the ceiling came down to two. A cold start delays a notification past the point where it is worth showing, so the floor is what keeps a notification prompt. The ceiling is a different question, answered below.

Push uses its approved dedicated two-vCPU HA database. Two instances with a two-connection pool draw four connections; three simultaneous revision resources draw twelve. Tagged candidates can run outside the service-wide cap, so Terraform bounds instances × pool × 3 at 64 connections, leaving dedicated capacity for maintenance and operators. Increase pool sizes only after measuring contention. The shared Relay budget excludes push entirely.

Authentication is the host proof in POST /v1/host/challenge, not Cloud Run IAM, so the service opts out of invoker IAM with invoker_iam_disabled = true, exactly as the relay director does. The project's domain-restricted-sharing policy refuses an allUsers invoker binding, so that is the only way to reach an open service here.

Environment

Set on the container by Terraform:

Variable	Source
PORT	Cloud Run, container port 8080
ORCA_PUSH_PUBLIC_URL	push_base_url
ORCA_PUSH_FCM_PROJECT_ID	project_id (required for standalone runtime)
ORCA_PUSH_DATABASE_URL	Secret orca-cloud-push-dedicated-database-url, pinned version
ORCA_PUSH_DATABASE_POOL_MAX	push_database_pool_max, 2 per instance
ORCA_PUSH_APNS_KEY	Secret orca-cloud-push-apns-key, version latest
ORCA_PUSH_APNS_KEY_ID	Secret orca-cloud-push-apns-key-id, version latest
ORCA_PUSH_APPLE_TEAM_ID	Secret orca-cloud-push-apple-team-id, version latest

ORCA_PUSH_APNS_TOPIC is left to its application default (com.stably.orca.mobile). Add it here only when it has to differ from the code default, so that a code-side change stays visible rather than silently overridden.

Terraform owns the three Apple secret names, labels, and replication, and never a version. The .p8 is issued by the Apple developer portal, so a Terraform-managed version would put the private key in state and would fight the rotation below. The database URL secret is different: Terraform generates that password, so it owns that version, exactly as relay-database.tf does. That puts the generated password and the full database URL in the state bucket, which the shared deploy identity can read; the Apple key never appears there. The three Apple secrets and the orca_push database carry prevent_destroy, so disabling the gateway fails the plan instead of deleting the only copy of the signing key or every live device token.

Importing what already exists

The runtime account, the three Apple secrets, and their accessor bindings were created out of band alongside the Apple credentials. They are declared so a plan is clean, and imported once. Run these from cloud/ after pnpm infra:init --env production, review the resulting plan, and expect the imported resources to show no changes.

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_service_account.push_runtime[0]' \
  projects/onorca-cloud/serviceAccounts/orca-cloud-push@onorca-cloud.iam.gserviceaccount.com

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_project_iam_member.push_runtime_fcm_admin[0]' \
  'onorca-cloud roles/firebasecloudmessaging.admin serviceAccount:orca-cloud-push@onorca-cloud.iam.gserviceaccount.com'

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_project_iam_member.push_runtime_service_usage_consumer[0]' \
  'onorca-cloud roles/serviceusage.serviceUsageConsumer serviceAccount:orca-cloud-push@onorca-cloud.iam.gserviceaccount.com'

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret.push_provider["orca-cloud-push-apns-key"]' \
  projects/onorca-cloud/secrets/orca-cloud-push-apns-key

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret.push_provider["orca-cloud-push-apns-key-id"]' \
  projects/onorca-cloud/secrets/orca-cloud-push-apns-key-id

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret.push_provider["orca-cloud-push-apple-team-id"]' \
  projects/onorca-cloud/secrets/orca-cloud-push-apple-team-id

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret_iam_member.push_provider_runtime_accessor["orca-cloud-push-apns-key"]' \
  'projects/onorca-cloud/secrets/orca-cloud-push-apns-key roles/secretmanager.secretAccessor serviceAccount:orca-cloud-push@onorca-cloud.iam.gserviceaccount.com'

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret_iam_member.push_provider_runtime_accessor["orca-cloud-push-apns-key-id"]' \
  'projects/onorca-cloud/secrets/orca-cloud-push-apns-key-id roles/secretmanager.secretAccessor serviceAccount:orca-cloud-push@onorca-cloud.iam.gserviceaccount.com'

terraform -chdir=infra/terraform import -var-file=environments/production.tfvars \
  'google_secret_manager_secret_iam_member.push_provider_runtime_accessor["orca-cloud-push-apple-team-id"]' \
  'projects/onorca-cloud/secrets/orca-cloud-push-apple-team-id roles/secretmanager.secretAccessor serviceAccount:orca-cloud-push@onorca-cloud.iam.gserviceaccount.com'

The push resources already exist in production. Preserve their addresses, dedicated database and identities; review the database cleanup runbook before applying changes. This root has unrelated standing drift, so an untargeted apply is never automatic.

Two things this root does not declare, because the carve assigns them elsewhere. Neither affects whether this root's plan is clean, since an undeclared resource is invisible to it.

• firebase.googleapis.com and fcm.googleapis.com are project service enablement, which is google_project_service.required in the foundation root. They are already enabled; add them to the foundation root's list so a foundation plan stays clean.
• The Firebase attachment on onorca-cloud is project-level and belongs with foundation for the same reason. It exists already.

Deploying

Deploy Push Gateway Production (.github/workflows/cloud-push-deploy.yml) is the only supported path. Like every cloud-* workflow it does nothing until ORCA_CLOUD_OPERATIONS_ENABLED is true, it runs only on main, and it needs the confirmation string DEPLOY_PUSH_GATEWAY.

It authenticates as the dedicated orca-cloud-gha-push identity through PRODUCTION_GCP_PUSH_DEPLOY_WORKLOAD_IDENTITY_PROVIDER and PRODUCTION_GCP_PUSH_DEPLOY_SERVICE_ACCOUNT. push-deploy-identity.tf restricts Workload Identity to this exact dispatch workflow on main in the production environment. Its distinct principal attribute cannot assume the shared Relay deploy identity.

The account can write images to Artifact Registry, deploy the push service, impersonate only the push runtime account, and manage exactly terraform/state/push-rollout/production.lock in the production state bucket. The relay root owns that conditional lease grant. It grants no Terraform-state object access. Publish github_push_workload_identity_provider and github_push_deploy_service_account as the production-environment variables above.

The workflow uses the production-push-rollout concurrency group with cancellation disabled and the existing durable lease action on the push-specific object. Push and Relay deploy independently; two push deploys cannot race traffic changes. Finish every old shared-lock push run before enabling the new workflow and lease grant. See the cleanup runbook for the bounded IAM transition and removal of any obsolete foundation-owned push membership.

The run builds the reviewed source_sha while the workflow stays on main. Buildx returns its own pushed digest (no mutable-tag lookup); every subsequent check and deployment uses that same digest. Before any production boot, a network-isolated container checks that the image recognizes ORCA_PUSH_MODE=validation and rejects invalid modes. Older images that lack this capability are refused before they can connect to production.

Under the production push rollout lease, it records the serving rollback revision and asserts Terraform-owned scaling. It deploys a tagged, zero-traffic validation revision:

• Validation opens PostgreSQL with default_transaction_read_only=on and skips schema setup.
• No delivery worker or challenge, session, or delivery pruner starts.
• Only /health and /ready are available; all application routes return 503.
• /health attests mode: validation; /ready checks database connectivity only. It does not prove schema compatibility, provider delivery, or active-worker readiness. Container probes can still use /health without treating an inert process as unhealthy.

The build explicitly targets linux/amd64 with provenance disabled so build metadata records a single manifest digest, rather than an OCI index that Cloud Run resolves to a different digest. The workflow verifies the exact image and scaling, probes readiness and mode, and checks the runtime identity with a validate-only FCM request. Cloud Run rejects deletion of the latest created revision even when it has no tag or traffic. Activation therefore creates a successor before removing the validation tag and deleting validation. The dedicated 64-connection budget reserves three simultaneous revision pools: serving, validation/rejected, and active/recovery successor (12 configured pool connections at the current two-by-two shape). Revision deletion is not proof of physical SQL session drain; verify termination and SQL sessions in controlled rollout acceptance. There is no shutdown sleep used as a drain gate.

Activation deliberately starts production effects. The distinct active revision uses the exact validated digest with the validation override removed. Schema setup runs on its existing one-connection untimed pool, followed by workers and pruners, before HTTP promotion. The workflow checks digest, full runtime spec and secret-reference shape, scaling, readiness and active mode, then moves HTTP traffic and checks the public origin. Those checks commit the new serving revision; subsequent retirement failures do not trigger rollback to a possibly deleted previous revision. The previous consumer is retired and all tags are cleared. Retain the previous immutable image from the summary: later recovery redeploys that digest, because the previous revision is deleted.

Before any candidate creation, the workflow requires exactly one revision resource, the sole HTTP serving revision. Existing historical revisions or leftovers from interrupted runs require explicit operator review and cleanup under the lease first; the workflow does not blindly delete them. This gate and retirement after every successful rollout prevent repeated runs accumulating workers. Terraform still owns configuration and scaling; removing validation mode adds no ignored field.

On failure before public checks pass, any attempted traffic shift is first rolled back and verified. If partial activation created a successor, recovery retires non-latest validation first; deletion failure stops recovery before a fourth resource can be created. Recovery then deploys the captured known-good digest as a tagged, zero-traffic successor with normal mode. It verifies template shape, secret references and scaling, probes tagged readiness and active mode, promotes the recovery revision, verifies traffic and public health, and only then deletes rejected and previous revisions. The latest recovery revision remains serving. Known-good recovery schema and workers can execute before promotion; neither recovery nor traffic rollback undoes schema changes or sent notifications.

Partial creates record deterministic names before mutation. Failed recovery or deletion requires operator cleanup under the lease; the next automated run refuses leftover resources. A canceled runner can require the same intervention. Traffic restoration alone does not stop queue consumers.

Manual recovery must preserve the three-resource bound and keep the successor serving:

# Hold the rollout lease; inspect latest, traffic, tags and existing revisions first.
# If three resources remain after partial activation, retire non-latest inert validation first.
# Restore previous traffic if its revision still exists and a failed candidate took traffic.
gcloud run deploy orca-cloud-push \
  --project onorca-cloud --region us-central1 --image <known-good-image-at-digest> \
  --remove-env-vars ORCA_PUSH_MODE --no-traffic \
  --tag <unique-recovery-tag> --revision-suffix <unique-recovery-suffix>
# Verify exact digest, template spec/secret references/scaling, tagged /ready and active /health.
gcloud run services update-traffic orca-cloud-push \
  --project onorca-cloud --region us-central1 --to-revisions <recovery-revision>=100
# Verify traffic and public /ready and /health before retiring old consumers.
gcloud run services update-traffic orca-cloud-push \
  --project onorca-cloud --region us-central1 --clear-tags
gcloud run revisions delete <rejected-or-previous-revision> \
  --project onorca-cloud --region us-central1
# Repeat only for reviewed obsolete revisions; retain the latest serving recovery revision.

Never merely remove validation mode while the template still holds a rejected image. Terraform owns environment configuration but ignores the image, so that would activate rejected code. Remove a tag only if it remains present. Verify the recovery revision is serving, the template is safe, and obsolete revision deletion and connection drain completed; already accepted provider sends cannot be undone. Activation-time schema changes must be additive and compatible with the rollback image: rollback does not reverse migrations or queue mutations. The inert phase intentionally cannot validate a new schema by applying it to production. Review migrations and validate them against isolated PostgreSQL before dispatch. No actual Cloud Run rollout, provider delivery or physical-device acceptance is implied by local contract tests.

Incompatible queue rollout prerequisite

The queue stores one notification object per delivery. Before deploying a revision that changes this format, stop every older push gateway revision and clear only unpublished push delivery fixtures from the push database. This is an unpublished feature, so do not preserve or migrate queued fixtures; no production mutation is implied by this prerequisite.

Why the FCM probe impersonates the runtime account

A gateway that boots and answers /ready can still be unable to send: the FCM grant lives on the runtime service account, not on anything the readiness check touches. The probe therefore mints an access token for orca-cloud-push@onorca-cloud.iam.gserviceaccount.com and posts validate_only: true with a token that cannot exist. validate_only stops Google before any delivery, and a healthy credential answers INVALID_ARGUMENT because the device token is garbage. PERMISSION_DENIED, 401, and 403 are the failures the step exists to catch, and they fail the run immediately, before traffic moves. Those four answers are the only conclusive ones: a 429, a 5xx, or a transport failure says nothing about the credential, so the send is retried up to five times at five-second intervals rather than read as either verdict. Probing as the deploy identity instead would prove something true about the wrong account.

Rotating the APNs key

Apple keys do not expire, so this is for a suspected compromise or a routine rotation. Order matters: the new key must be serving before the old one is revoked, or every iOS push fails in the window between.

1. In the Apple developer portal, create a new APNs authentication key. Download the .p8 once; Apple will not show it again. Note the new key ID. A team may hold two APNs keys at a time, which is what makes this overlap possible.

2. Add a version to each changed secret, without printing the value:

   gcloud secrets versions add orca-cloud-push-apns-key \
     --project onorca-cloud --data-file /path/to/AuthKey_NEW.p8
   printf '%s' '<new key id>' | gcloud secrets versions add orca-cloud-push-apns-key-id \
     --project onorca-cloud --data-file=-
   

   The team ID does not change, so orca-cloud-push-apple-team-id is untouched.

3. Dispatch Deploy Push Gateway Production. The container reads latest at start, so only a new revision picks the key up; there is no in-place reload.

4. Verify from a real device that an iOS notification still arrives. The workflow's FCM probe covers Android only, and APNs has no validate-only equivalent.

5. Only then revoke the old key in the Apple portal, and disable the superseded secret versions:

   gcloud secrets versions disable <old-version> \
     --project onorca-cloud --secret orca-cloud-push-apns-key
   

   Disable rather than destroy, so a rollback to the previous revision still works. Destroy after the next clean deploy.

Delete the downloaded .p8 from disk when you are done. It is the whole credential.

Dead tokens

A push token stops working when the app is uninstalled, when the user restores to a new device, or when iOS reissues it. Both providers report this, and the shapes differ:

• APNs: HTTP 410, or 400 with BadDeviceToken or Unregistered. DeviceTokenNotForTopic is a provider configuration error and leaves the registration live. Check the APNs topic and environment; future notifications can resume after correction without phone re-registration. The failed notification is not retried for this non-transient error.
• FCM: UNREGISTERED, or INVALID_ARGUMENT whose message names the token.

The gateway marks the registration dead_at and returns status: "dead" for it, and the desktop drops the registration when it sees that. Nothing here retries a dead token. A phone that comes back re-registers the same host/device pair, retaining its registrationId and clearing dead_at. The per-minute delivery_dead counter measures delivery outcomes, not currently dead registrations. A spike across many hosts warrants checking credentials and topics.

Quotas

Two independent limits, both enforced in the gateway and both returning HTTP 200 with status: "rate_limited" per result rather than failing the request:

Limit	Scope
300 logical alerts per rolling 15 minutes	per hostFingerprint
300 logical dismissals per rolling 15 minutes	per hostFingerprint, separate budget
20 registrationIds	per request, hard cap, HTTP 400 over it

Fanout to several phones counts one logical event; there is no per-phone daily allowance. Unauthenticated handshakes and invalid bearer attempts have separate 30/minute IP buckets. Authenticated routes use a 600/minute host bucket and a shared 6,000/minute client-IP bucket per instance. The IP budget cannot be reset by generating another host key. It is shared by clients behind one NAT and is an abuse safeguard, not a global provider-spending cap. Auth database lookup concurrency and waiting work are bounded independently of HTTP concurrency.

push_events backs quota accounting. push_event_recipients deduplicates fanout and push_delivery_batches retains its historical name and persists individual deliveries, worker leases, retries and outcomes. Identity metadata is retained for 24 hours. Payloads expire within five minutes and are cleared on completion or by minute-level expiry cleanup. FCM project-level provider quotas remain independent of host limits.

Logging is aggregate counters only. Never log a token, a title, a body, or a full fingerprint; the first four characters of a fingerprint are the most that may appear.

DNS: one hand-managed record

The Cloud Run domain mapping is created here, and Google issues and renews the certificate. The onorca.dev zone is not in this root: it is a Cloudflare zone whose Terraform-managed records live in the apps root in stablyai/orca-cloud, and whose relay and auth records are managed by hand. The push record follows the relay's precedent and was created by hand on 2026-09-04:

push.onorca.dev.  CNAME  ghs.googlehosted.com.   (DNS only, not proxied)

terraform -chdir=infra/terraform output push_dns_record prints the same three fields. If the record is ever lost, recreate it exactly like that; Cloudflare proxying blocks certificate issuance and breaks Cloud Run host routing.

Recovery and delivery guarantees

Candidate tags and deterministic revision names are recorded before deployment. Promotion intent is recorded before changing traffic, so a failed verification or ambiguous mutation result still triggers rollback. A known-good successor must exist before the rejected latest revision can be deleted. After verified recovery promotion and public checks, rejected and previous consumers are retired; the recovery revision remains serving. Failed cleanup blocks subsequent rollout admission. The summary runs even if candidate discovery or traffic verification fails.

Push uses the relay's schema-startup retry implementation through @orca-cloud/postgres-schema. Session replacement is serialized per host and a unique host index upgrades older databases by retaining their newest session. Cloud Verify runs push concurrency tests against PostgreSQL.

Accepted sends commit quota and pending work together before returning queued. Workers resume unfinished deliveries after restarts without relying on desktop retries. The durable queue and expiring leases coordinate replicas. All provider attempts retain the original five-minute deadline and respect provider backoff; no retry extends alert life. Silent dismissal messages have their own quota and cancel matching unsent alerts. Mobile OS delivery/execution is not guaranteed.

Shutdown stops admission and new claims; unfinished leases remain recoverable. Provider acceptance and SQL completion cannot be atomic, so repeated transport delivery remains possible after a crash. Stable per-event replacement identities reduce duplicates without promising exactly-once visible delivery. FCM notification messages are inherently collapsible while offline and support only a small number of concurrent collapse keys per device, so excess pending messages may be discarded and every offline alert is not guaranteed to appear. Socket reconnect reconciles dismissals against the current native tray; it has no stored replay watermark and never recovers a missed OS banner.

Dedicated database operations

Push has one dedicated database attachment, with stable Terraform addresses and deletion protection. There is no switch to shared storage. Follow the database operations runbook for deployment prerequisites, legacy resource ownership, capacity and recovery.