fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), adoptable deploy shape, event-loop stall alert (#27406)

* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert

On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.

relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.

* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar

- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
  apply created; the deploy preserves that shape instead of the serving one. It must be the
  latest created, unserved revision on the serving image. Wired into the staging and
  production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
  with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
  adopted apply shape does not fall to the 80 / 30 s defaults.

* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar

parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
This commit is contained in:
Jinwoo Hong
2026-10-11 04:30:27 -04:00
committed by GitHub
parent 12a8fc0166
commit 2bdb55350d
10 changed files with 438 additions and 19 deletions
@@ -51,6 +51,10 @@ on:
description: Exact immutable serving predecessor digest for the one-time identity bootstrap
required: true
type: string
shape-baseline-revision:
description: Only after a Terraform apply changed the director template. The apply's 0% revision, whose shape the deploy adopts
required: false
type: string
permissions:
contents: read
@@ -93,6 +97,7 @@ jobs:
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
SHAPE_BASELINE_REVISION: ${{ inputs.shape-baseline-revision }}
steps:
- uses: actions/checkout@v4
@@ -140,7 +145,13 @@ jobs:
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test -z "${SHAPE_BASELINE_REVISION}"
test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY
elif test -n "${SHAPE_BASELINE_REVISION}"; then
[[ "${SHAPE_BASELINE_REVISION}" =~ ^orca-cloud-relay-[0-9]{5}-[a-z0-9]{3}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE
elif test "${PRUNE_INCOMPATIBLE_REVISIONS}" = true; then
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = PRUNE_INCOMPATIBLE_RELAY_DIRECTOR_REVISIONS
@@ -222,12 +233,17 @@ jobs:
fi
[[ "${target_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
shape_args=()
if test -n "${SHAPE_BASELINE_REVISION}"; then
shape_args=(--shape-baseline-revision "${SHAPE_BASELINE_REVISION}")
fi
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
"${shape_args[@]}" \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" \
--bootstrap-runtime-identity "${BOOTSTRAP_RUNTIME_IDENTITY}" \
@@ -39,7 +39,11 @@ on:
type: choice
options: [preserve, '0', '2']
confirmation:
description: BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY for a bootstrap
description: BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY for a bootstrap, ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE with a shape baseline
required: false
type: string
shape-baseline-revision:
description: Only after a Terraform apply changed the director template. The apply's 0% revision, whose shape the deploy adopts
required: false
type: string
shadow-seat-feed-cells:
@@ -86,6 +90,7 @@ jobs:
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }}
MIN_INSTANCES: ${{ inputs.min-instances }}
SHAPE_BASELINE_REVISION: ${{ inputs.shape-baseline-revision }}
steps:
- uses: actions/checkout@v4
@@ -106,6 +111,12 @@ jobs:
else
test -z "${PREDECESSOR_IMAGE_DIGEST}"
fi
if test -n "${SHAPE_BASELINE_REVISION}"; then
[[ "${SHAPE_BASELINE_REVISION}" =~ ^orca-cloud-relay-staging-[0-9]{5}-[a-z0-9]{3}$ ]]
test "${BOOTSTRAP_RUNTIME_IDENTITY}" = false
test "${CAPACITY_CELL_ID}" = none
test "${CONFIRMATION}" = ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE
fi
- uses: hashicorp/setup-terraform@v3
with:
@@ -186,6 +197,10 @@ jobs:
if test "${MIN_INSTANCES}" != preserve; then
floor_args=(--min-instances "${MIN_INSTANCES}")
fi
shape_args=()
if test -n "${SHAPE_BASELINE_REVISION}"; then
shape_args=(--shape-baseline-revision "${SHAPE_BASELINE_REVISION}")
fi
topology_args=()
if test "${CAPACITY_CELL_ID}" != none; then
topology_args=(
@@ -207,6 +222,7 @@ jobs:
"${floor_args[@]}" \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
"${identity_args[@]}" \
"${shape_args[@]}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience "${REHOME_AUDIENCE}" \
--rehome-control-origin https://relay-staging.onorca.dev \