mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
fix(relay): keep the backend service out of the same-cap wave; the capacity role cannot update it (#22140)
The capacity role the same-cap wave authenticates as, orcaRelayProductionCapacity, has no compute.backendServices.update. Since #21860 added `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to both of the job's plan invocations, every wave has therefore created the new instance template, modified the MIG, and then failed 403 on the backend, leaving the cell isolated with its trust probe, admission restore, and shadow gate all skipped. Run 35684694704 on production-gce-c7 is the first one that hit it in production. Drop the backend target from both plans and restore the resume gate to exactly `.changes == 2` (the template-and-MIG rollback-image drift) or a converged plan, removing the backend-only resume apply #21865 added on top. A resume applies nothing again, which is what a resume means. The validator keeps its bound on a cell backend update, so it still reports one and refuses anything wider, but a wave plan can no longer contain one. The drain timeout from #21848 and the log_config from #21860 need a root apply by a principal that holds the permission; granting the capacity role that permission is itself a root apply, so it can follow as its own change rather than blocking every wave in the meantime. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
This commit is contained in:
@@ -598,16 +598,15 @@ jobs:
|
||||
# Zero resource changes prove the prior run's apply completed and no
|
||||
# restart will follow, keeping the incarnation check honest. Root
|
||||
# outputs may lag a targeted apply, so judge resource_changes only.
|
||||
# The backend service is targeted too, so its reviewed drain timeout
|
||||
# and request logging can be the only thing left here; neither
|
||||
# restarts an instance, so the validator below clears that on its
|
||||
# own, without the template-and-MIG pair.
|
||||
# The cell backend service is deliberately not targeted: the capacity role has no
|
||||
# compute.backendServices.update, so naming it fails the apply after the MIG has
|
||||
# rolled. Its drain and log_config settings ship in a root apply by a principal
|
||||
# that holds the permission.
|
||||
terraform -chdir=infra/terraform plan \
|
||||
-var-file=environments/production.tfvars \
|
||||
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
|
||||
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
|
||||
if ! terraform -chdir=infra/terraform show -json \
|
||||
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
|
||||
@@ -644,21 +643,7 @@ jobs:
|
||||
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
|
||||
"${POOL_ARGUMENTS[@]}")"
|
||||
echo "${RESUME_REVIEW}"
|
||||
jq -e '.changes == 2
|
||||
or (.changes == 0 and ((.backendUpdate // []) | length) > 0)' \
|
||||
<<< "${RESUME_REVIEW}" >/dev/null
|
||||
# changes == 0 here means the template and MIG are converged and this cell's
|
||||
# reviewed backend update is the only thing left, so the resume is not complete:
|
||||
# apply it, or the cell silently keeps the 300-second drain and no request
|
||||
# logging and the operator reads that as a finished roll. The plan holds nothing
|
||||
# else (the validator bounded it to this cell's backend, and the template and MIG
|
||||
# are no-ops in it), and neither attribute restarts an instance, so the
|
||||
# incarnation check downstream stays honest. Template-and-MIG drift still applies
|
||||
# nothing, which is what a resume means.
|
||||
if test "$(jq -er '.changes' <<< "${RESUME_REVIEW}")" = 0; then
|
||||
terraform -chdir=infra/terraform apply -auto-approve \
|
||||
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
|
||||
fi
|
||||
jq -e '.changes == 2' <<< "${RESUME_REVIEW}" >/dev/null
|
||||
fi
|
||||
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
|
||||
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
|
||||
@@ -681,7 +666,6 @@ jobs:
|
||||
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
|
||||
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
||||
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
|
||||
PLAN_REVIEW="$(terraform -chdir=infra/terraform show -json \
|
||||
"${RUNNER_TEMP}/relay-same-cap.tfplan" \
|
||||
|
||||
Reference in New Issue
Block a user