From ea61d1c03e98618ed8aab79ce768602f1b19b3ac Mon Sep 17 00:00:00 2001 From: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com> Date: Sun, 11 Oct 2026 14:24:15 -0400 Subject: [PATCH] docs(relay): director Cloud SQL sidecar enablement and rollback (#27461) * docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's rollback candidate. A new log-based metric can take minutes to appear, so its alert policy can fail the first apply. Both observed on staging 2026-10-11. * docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease --- cloud/docs/orca-relay-operations.md | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/cloud/docs/orca-relay-operations.md b/cloud/docs/orca-relay-operations.md index bab64731cee..536d39a9f4b 100644 --- a/cloud/docs/orca-relay-operations.md +++ b/cloud/docs/orca-relay-operations.md @@ -466,6 +466,49 @@ SIGTERM is a degraded path. It may advertise zero grace, rejects new work, close After either path, verify close/`1006`/`504` observations, bounded reconnect arrival, subscription replay, deterministic ordinary in-flight rejection, and idempotent install/confirmation reconciliation. +## Director Cloud SQL transport (proxy sidecar) + +`relay_director_cloud_sql_proxy_sidecar` moves the director off Cloud Run's built-in `/cloudsql` +socket, whose `connect()` blocks the event loop for the whole upstream dial (10 s per attempt while +Cloud SQL starts or fails over), onto a Cloud SQL Auth Proxy sidecar on `127.0.0.1:5432`. Staging +runs it since 2026-10-11 (serving `orca-cloud-relay-staging-00177-qur`); production keeps the socket +until it is enabled through this sequence. + +1. In production, only after a reviewed `production.tfvars` PR sets the variable (and + `relay_event_loop_stall_alert`) and a staging Cloud SQL stop/start drill on the sidecar has passed. + Plan with exactly these targets and review it: `google_cloud_run_v2_service.relay`, the three + `relay_director_database_url` resources (secret, version, accessor), + `google_logging_metric.relay_event_loop_stall` and + `google_monitoring_alert_policy.relay_event_loop_stall`. Expect `5 to add, 1 to change`. The + director change also carries the template drift already on `main` (`ORCA_RELAY_IMAGE_DIGEST` + dropped, which the deploy restores; reordered or default-valued environment entries); stop on + anything else, such as concurrency, timeout or image. +2. Apply the saved plan; in production, only while holding the production Cloud SQL rollout lease + (`terraform/state/cloud-sql-rollout/production.lock` in `onorca-cloud-terraform-state`, the + `cloud-sql-rollout-lease` action). **The metric must exist before its alert policy**: a new log-based metric + takes minutes to become visible, and the first apply can fail on the policy with `404 Cannot + find metric(s)`. Re-plan and apply the policy alone a few minutes later, or apply the metric in + an earlier run. Staging hit this on 2026-10-11. +3. The apply creates a 0% revision (the service declares no traffic block). Cloud Run does **not** + start it: it reads `Ready` with reason `Retired` and logs nothing, so it proves the template + parses, not that the sidecar boots. Record its name; it is `status.latestCreatedRevisionName`. +4. Dispatch the director deploy workflow with the **serving** image digest, + `shape-baseline-revision=` and `confirmation=ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE`. + The deploy compares its candidates with that revision's shape instead of the serving one, and + refuses an image change, an identity bootstrap, a topology change or pruning in the same run. + The first real sidecar boot is the deploy's `selector-rollback` candidate, health-checked before + any traffic moves; the proxy logs `Listening on 127.0.0.1:5432` and its startup and liveness + probes pass. +5. Verify the serving revision has `relay` then `cloud-sql-proxy`, `DATABASE_URL` from + `orca-cloud-relay-director-database-url`, `ORCA_RELAY_IMAGE_DIGEST` restored, the warm floor + kept, and no `run.googleapis.com/cloudsql-instances` annotation. + +Rollback: the pre-sidecar revision survives (shape adoption refuses pruning) and keeps its own +`/cloudsql` connection and socket secret, so `gcloud run services update-traffic +--to-revisions==100` restores it. No workflow owns that move; in production it +needs approval and the rollout lease. The next deploy then refuses until a reviewed apply with the +variable `false` and a deploy that adopts that apply's revision. + ## Rollback Before target registration, an expired migration automatically releases target reservations and returns the assignment to the source with another strictly newer epoch. Verify: