starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-10-03 08:02:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG (#24373)

Browse Source 
* fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG

The stranded-rollback recovery ran a gcloud rolling action, which renames the
MIG version outside Terraform. Every later plan for that cell then reverted the
label, and the capacity-plan validator refused the revert as an unreviewed MIG
change, so the cell could be neither rolled nor rolled back.

The validator now accepts a MIG field moving back to what relay-gce-cells.tf
declares (version name and update policy), in every mode, and a test pins those
values to the Terraform file. The stranded branch recreates the cell's single
instance with recreate-instances, which leaves the MIG untouched.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): let a label-only MIG plan through and recreate on it in a stranded rollback

A stranded rollback whose template is already in place plans only the version
name revert. The validator still required the MIG template to move, so that
plan was refused, and the recreate gate (changes == 0) would have skipped a
plan of one change and left the drain flag set. Require the template move only
when no declared field reconciles, and recreate whenever the template was not
replaced (changes < 2).

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
This commit is contained in:
Jinwoo Hong
2026-10-01 08:12:49 -04:00
committed by GitHub
parent f9940d5354
commit 744e7722c2
5 changed files with 251 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
+13 -8
View File
@@ -709,15 +709,20 @@ jobs:
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
# A stranded cell already runs the reviewed template, so the apply above replaces
# no instance and the drain flag, which only a restart clears, would survive the
# whole wave. Roll the MIG explicitly on exactly the policy a template change uses.
# Every field is passed: gcloud persists these into the MIG's update policy, and it
# defaults the method to substitute on a group with no stateful config, so omitting
# one drifts the policy off the reviewed one and fails every later targeted plan.
# whole wave. Fewer than the template-and-MIG pair means the template stayed put,
# including a MIG-only reconciliation. Recreate its one instance from the MIG's
# current template; a rolling action would rewrite the MIG's version name outside
# Terraform, and every later targeted plan would carry that revert.
if test "${ROLLBACK_STAGE}" = stranded \
&& test "$(jq -er '.changes' <<< "${PLAN_REVIEW}")" = 0; then
gcloud compute instance-groups managed rolling-action replace "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--replacement-method recreate --max-surge 0 --max-unavailable 1
&& jq -e '.changes < 2' <<< "${PLAN_REVIEW}" >/dev/null; then
STRANDED_INSTANCE="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--format=json \
| jq -er 'if length == 1 then .[0].instance | split("/") | last
else error("stranded cell MIG does not have exactly one instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${STRANDED_INSTANCE}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
fi