The live metrics have role and cell_id only. A label change on a log metric
is delete+create, so applying the region label would replace all 21 metrics,
reset their history, and blank the relay alert policies during the swap.
Matching Terraform to live state makes the targeted plan create-only (8
renewal metrics never applied, plus the incident dashboard from #18717).
Cells run cloud-sql-proxy against the auth database's public IP, so every
connection burns a Cloud NAT port on the relay gateway; that allocation filled
on 2026-09-04 and every cell's proxy dial timed out at once. Add --private-ip
behind relay_cloud_sql_private_ip so production can move the traffic onto the
VPC peering once the foundation root has applied it.
Default false, and the rendered startup script is byte-identical to main with
that default, so merging rolls nothing. --unix-socket is untouched: it selects
the listener, not the upstream address, so DATABASE_URL does not change.
The director is Cloud Run and egresses outside this VPC's NAT, so it is not
part of the problem; moving it would mean VPC egress plus a TCP DSN and its own
secret, which is a separate change.
201 relay cell process exits over 48h on 2026-09-04 paged nobody. Adds a log metric on the Docker `container die` event for the orca-relay container, an alert at >3 exits per instance per 15 min, and a four-chart incident dashboard covering the signals that had to be assembled by hand during the outage.
* feat(relay-infra): dynamic NAT ports and alerts for the 2026-09-04 stall signals
Relay cells reach Cloud SQL's public IP through Cloud NAT. The static default
of 64 ports per VM filled on 2026-09-04 and every cell's proxy dial timed out
at once, which read as a fleet-wide SQL stall against a healthy database.
Switch both regional NATs to dynamic port allocation (64..4096 per VM).
Add the three alerts that would have paged inside the first ten minutes:
- Cloud SQL WAL-triggered checkpoint loop (log metric on
"checkpoint starting: wal", > 3 in 5 min)
- Cloud SQL disk utilization > 70%
- Cloud NAT OUT_OF_RESOURCES packet drops on the relay gateways
No workflow applies google_monitoring_* or the NAT resources today; every
relay workflow is target-scoped to cells. Apply is a reviewed targeted plan
(see PR body).
* test(cloud): register the three new relay alert policies in the root partition fixture
Mirrors stablyai/orca-cloud#470. The private relay workflows are retired, so
the dual accept has one live arm left. Add `github_workflow_file_prefix` for
the primary repository's workflow filenames, point `github_repo`/
`github_repo_id` at `stablyai/orca` (`1183888342`), and empty
`github_accepted_repositories` in both environments. Every relay provider goes
back to a single arm naming `cloud-` prefixed workflow refs.
`cloud/infra/terraform` stays byte-identical to the private branch. The two
identity tests diverge here as they already did, so they take the same change
rather than the same bytes: both now render the trusted ref head from the
Terraform variable instead of this checkout's own workflow filenames, which is
what lets the length pin be the same 791 characters in either repository.
* chore(cloud): pin staging relay c3 to the director's image
Mirrors stablyai/orca-cloud#468. c3 stayed on sha-c91439af after the
director and c4 moved to sha-e3e92d95, so the staging capacity proof's
compatible-director-image check has failed since 2026-08-14.
* test(cloud): scope the launch-image pin to staging C4 now that C3 shares the digest
* test(cloud): keep the public workflow assertions; scope only the launch-image pin to C4
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong