201 relay cell process exits over 48h on 2026-09-04 paged nobody. Adds a log metric on the Docker `container die` event for the orca-relay container, an alert at >3 exits per instance per 15 min, and a four-chart incident dashboard covering the signals that had to be assembled by hand during the outage.
* feat(relay-infra): dynamic NAT ports and alerts for the 2026-09-04 stall signals
Relay cells reach Cloud SQL's public IP through Cloud NAT. The static default
of 64 ports per VM filled on 2026-09-04 and every cell's proxy dial timed out
at once, which read as a fleet-wide SQL stall against a healthy database.
Switch both regional NATs to dynamic port allocation (64..4096 per VM).
Add the three alerts that would have paged inside the first ten minutes:
- Cloud SQL WAL-triggered checkpoint loop (log metric on
"checkpoint starting: wal", > 3 in 5 min)
- Cloud SQL disk utilization > 70%
- Cloud NAT OUT_OF_RESOURCES packet drops on the relay gateways
No workflow applies google_monitoring_* or the NAT resources today; every
relay workflow is target-scoped to cells. Apply is a reviewed targeted plan
(see PR body).
* test(cloud): register the three new relay alert policies in the root partition fixture
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong