fix(relay): keep the backend service out of the same-cap wave; the capacity role cannot update it (#22140)

The capacity role the same-cap wave authenticates as, orcaRelayProductionCapacity,
has no compute.backendServices.update. Since #21860 added
`google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to both of the
job's plan invocations, every wave has therefore created the new instance template,
modified the MIG, and then failed 403 on the backend, leaving the cell isolated with
its trust probe, admission restore, and shadow gate all skipped. Run 35684694704 on
production-gce-c7 is the first one that hit it in production.

Drop the backend target from both plans and restore the resume gate to exactly
`.changes == 2` (the template-and-MIG rollback-image drift) or a converged plan,
removing the backend-only resume apply #21865 added on top. A resume applies nothing
again, which is what a resume means.

The validator keeps its bound on a cell backend update, so it still reports one and
refuses anything wider, but a wave plan can no longer contain one. The drain timeout
from #21848 and the log_config from #21860 need a root apply by a principal that holds
the permission; granting the capacity role that permission is itself a root apply, so
it can follow as its own change rather than blocking every wave in the meantime.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
This commit is contained in:
Jinwoo Hong
2026-09-22 02:50:32 -04:00
committed by GitHub
parent 895f2cf477
commit bf8d63bc67
4 changed files with 41 additions and 75 deletions
@@ -598,16 +598,15 @@ jobs:
# Zero resource changes prove the prior run's apply completed and no
# restart will follow, keeping the incarnation check honest. Root
# outputs may lag a targeted apply, so judge resource_changes only.
# The backend service is targeted too, so its reviewed drain timeout
# and request logging can be the only thing left here; neither
# restarts an instance, so the validator below clears that on its
# own, without the template-and-MIG pair.
# The cell backend service is deliberately not targeted: the capacity role has no
# compute.backendServices.update, so naming it fails the apply after the MIG has
# rolled. Its drain and log_config settings ship in a root apply by a principal
# that holds the permission.
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
if ! terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
@@ -644,21 +643,7 @@ jobs:
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
"${POOL_ARGUMENTS[@]}")"
echo "${RESUME_REVIEW}"
jq -e '.changes == 2
or (.changes == 0 and ((.backendUpdate // []) | length) > 0)' \
<<< "${RESUME_REVIEW}" >/dev/null
# changes == 0 here means the template and MIG are converged and this cell's
# reviewed backend update is the only thing left, so the resume is not complete:
# apply it, or the cell silently keeps the 300-second drain and no request
# logging and the operator reads that as a finished roll. The plan holds nothing
# else (the validator bounded it to this cell's backend, and the template and MIG
# are no-ops in it), and neither attribute restarts an instance, so the
# incarnation check downstream stays honest. Template-and-MIG drift still applies
# nothing, which is what a resume means.
if test "$(jq -er '.changes' <<< "${RESUME_REVIEW}")" = 0; then
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
fi
jq -e '.changes == 2' <<< "${RESUME_REVIEW}" >/dev/null
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
@@ -681,7 +666,6 @@ jobs:
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
PLAN_REVIEW="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap.tfplan" \