mirror of
https://github.com/stablyai/orca.git
synced 2026-09-26 08:02:38 +00:00
Jinwoo Hong
c8a5580659
* fix(relay): re-place hosts off a cell isolated for a roll A roll isolates a cell by moving it out of the 'general' admission class; the cell then refuses every attach with 4503. The director never noticed, because the only liveness test it applies to a host's current cell reads `relay_cell_runtime.ready` and the heartbeat, and an isolated cell keeps heartbeating ready=1 for the whole drain. So every host on that cell was handed its own dead cell, closed, and handed it back — 500-1,900 hosts looping for 13-16 minutes per cell roll, at ~6 dials each per minute, with no neighbour absorbing anything. The sticky lane now treats a live incumbent whose admission is 'migration-only' — the state a roll's isolate step writes — the same way it treats a dead one: it returns null, which means "fall through to placement". The placement lane had the identical hole eleven lines further down, so it takes the same predicate; without that second swap the sticky change is inert, because placement would hand the pin straight back (a draining cell has more headroom than anyone). An isolated incumbent skips the dead-cell fence branch: that branch exists to prove an unreachable cell stopped serving a host, and this one is reachable and enforces the epoch itself. 'existing-only' is deliberately untouched — those cells serve the hosts they already hold, and only `assignmentStrandedOnUnservedCell` may release that pin. A host with an open `relay_assignment_migrations` row keeps its pin too, so this stays disjoint from the migration machinery. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): gate re-placement on a roll-isolation marker, not on admission Review of the first commit found the predicate wrong. `migration-only` is an admission class, not a drain signal: an Asia `--mode rollback`, an evacuation or forward-recovery target awaiting a separate promote dispatch, a failed same-cap wave's re-isolate, an abandoned migration retired on its target and a rehome settlement all park loaded cells there durably, with no migration lease and no open migration row. All five were indistinguishable from a roll's isolate, so the first commit would have converted `operate-relay-asia-admission --mode rollback` from a reversible admission flip into a mass move of ~4,000 hosts — and, because `leastLoadedCell` treated region as a preference, into us-central1. The signal is now an explicit stamp. `relay_cell_admission` gains a nullable `roll_isolated_at`, added through the shared schema runner's catalog pre-check so a migrated database takes no relation lock on boot and an un-migrated one gets a catalog-only rewrite. The same-cap isolate step is its only writer, via a new optional `rollIsolatedCells` on the selector apply; the same UPDATE that writes the state clears the stamp whenever a cell leaves 'migration-only', so a restore cannot leave one behind and a failed wave's re-isolate keeps the one it has. Every other admission writer omits the field, so its cells stay unmarked and their hosts stay pinned. Old directors ignore the field; old callers never send it. Region is now a constraint rather than a preference on this path only: a re-placement must find a general, live cell with connection headroom in the host's own region, or the pin is kept and one `orca_relay_sticky_replacement_deferred` event is logged. Cross-region spill is no longer reachable here. The fence bypass is narrowed to a live incumbent. It was always a no-op for the intended case, and for a stamped cell that stops heartbeating while still holding sockets it reopened split-brain; that cell now takes the dead-cell path unchanged. Also: the hot-path admission reader no longer throws on an unrecognised state — it sits on every sticky dial and the rule it feeds is "move the host", so an unreadable row has to mean "don't". And the sticky lane reads the admission row once for both the stranded rule and the stamp instead of twice. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): emit the re-placement events after the transaction commits CodeRabbit on assignment-store.ts:1075. Both events were written where they are decided, which is inside assignOnce's transaction. A reservation or lease write failing after that point rolls the placement back, but a line already on stdout cannot be rolled back with it — so the canary this PR asks an operator to read would count re-placements that never happened, and a Postgres transaction retry could leave a stale line behind as well. The transaction now returns its events alongside the RelayAssignment and the caller flushes them once it has resolved. Returning them rather than setting a variable in the enclosing scope is what makes the retry case safe too: only the attempt that committed can carry its events out. assign()'s signature is unchanged; the extra shape lives entirely inside assignOnce. orca_relay_sticky_replacement_deferred was moved the same way. It cost one more push into the array that already existed, and it is decided inside the same transaction, so leaving it behind would have been the odd case rather than the cheap one. The new test injects a failure on the first write after the decision, asserts no event is emitted, and asserts the assignment is still on its original cell — without that second assertion the absence would only prove the emit was early, not that it would have been wrong. A control dial with nothing injected emits exactly one event, so the case cannot pass on a broken harness. With the emit put back inside the transaction, it fails. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): expire the roll stamp, correct the wire note, assert the stamp landed Delta review findings B, D and E. A (the deferral path's cost) is deliberately not implemented; it is now written up under Follow-ups in the PR body as required before any Asia roll, because it cannot fire in a US canary. B, which also closes C: the stamp was written, carried and never compared to anything. A roll isolates and restores one cell inside ~15 minutes, so a stamp older than two hours is not a roll in progress. It is a failed wave whose failsafe re-isolated a possibly healthy cell and is waiting on an operator — the postmortem in this tree records gaps of hours — or an orphan left by a director rollback whose restore wrote 'general' without the clause that clears the stamp, which the selector's 'keep' branch would then preserve until some later park reactivated it. Both want the same answer and it is the pre-existing one: keep the pin. One comparison against a value already on the row. The bound takes the caller's `now` rather than reading the clock again, so one assign reasons about one instant; the stamp's age is now a thing that decides whether a host moves, and two clock reads could disagree across it. D: the comment beside the new request field claimed an updated caller reaching an older director "is simply ignored". The schema is .strict(), so it is a 400. That fails closed — the isolate aborts before MUTATION_STARTED is set and nothing is written — but it is a deploy ordering constraint, and it was undocumented. The comment now says so and the PR body's rollout notes carry it. E: nothing read the `rollIsolated` the script already prints, so an older script against a newer director would silently produce today's behaviour and the canary would read as "the fix did nothing" with no way to tell that from a wrong premise. Both isolate steps now assert it, beside the generation they already parse. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
927 lines
52 KiB
YAML
927 lines
52 KiB
YAML
name: Deploy Relay Production Same-Cap Job
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
mode: { required: true, type: string }
|
|
target-cell-id: { required: true, type: string }
|
|
target-image-digest: { required: true, type: string }
|
|
rollback-image-digest: { required: true, type: string }
|
|
target-rehome-protocol: { required: true, type: string }
|
|
rollback-rehome-protocol: { required: true, type: string }
|
|
expected-selector-generation: { required: true, type: string }
|
|
expected-existing-only-cells: { required: true, type: string }
|
|
expected-migration-only-cells: { required: true, type: string }
|
|
expected-general-cells: { required: true, type: string }
|
|
expected-rehome-generation: { required: true, type: string }
|
|
monitor-run-id: { required: true, type: string }
|
|
monitor-run-attempt: { required: true, type: string }
|
|
gate-override-reason: { required: false, type: string, default: '' }
|
|
gate-override-confirmation: { required: false, type: string, default: '' }
|
|
wave-index: { required: true, type: string }
|
|
|
|
permissions:
|
|
actions: read
|
|
contents: read
|
|
id-token: write
|
|
|
|
defaults:
|
|
run:
|
|
working-directory: cloud
|
|
|
|
jobs:
|
|
rollout:
|
|
if: ${{ github.ref == 'refs/heads/main' }}
|
|
runs-on: blacksmith-2vcpu-ubuntu-2204
|
|
timeout-minutes: 75
|
|
environment: production
|
|
env:
|
|
GCP_PROJECT_ID: onorca-cloud
|
|
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
|
|
DIRECTOR_ORIGIN: https://relay.onorca.dev
|
|
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
|
|
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
|
|
DEPLOY_MODE: ${{ inputs.mode }}
|
|
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
|
|
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
|
|
TARGET_REHOME_PROTOCOL: ${{ inputs.target-rehome-protocol }}
|
|
ROLLBACK_REHOME_PROTOCOL: ${{ inputs.rollback-rehome-protocol }}
|
|
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
|
|
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
|
|
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
|
|
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
|
|
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
|
|
WAVE_INDEX: ${{ inputs.wave-index }}
|
|
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
|
|
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
|
|
GATE_OVERRIDE_REASON: ${{ inputs.gate-override-reason }}
|
|
GATE_OVERRIDE_CONFIRMATION: ${{ inputs.gate-override-confirmation }}
|
|
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
|
|
# ~800 controls over 2 min is ~7 re-dials/s per cell, well under the director's
|
|
# 5 x 80 in-flight assign cap. A cell on an older image ignores it and drains at once.
|
|
DRAIN_PACE_WINDOW_MS: '120000'
|
|
steps:
|
|
- name: Require exact reusable-workflow configuration
|
|
working-directory: .
|
|
env:
|
|
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
|
|
CAPACITY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
|
|
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
|
|
run: |
|
|
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
|
|
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
|
|
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
|
|
test "${TARGET_IMAGE_DIGEST}" != "${ROLLBACK_IMAGE_DIGEST}"
|
|
[[ "${TARGET_REHOME_PROTOCOL}" =~ ^(0|1|3)$ ]]
|
|
[[ "${ROLLBACK_REHOME_PROTOCOL}" =~ ^(0|1|3)$ ]]
|
|
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
|
|
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
|
|
# cell_1..cell_10 in the calling wave; the chain is static, so this range is too.
|
|
[[ "${WAVE_INDEX}" =~ ^[0-9]$ ]]
|
|
# The caller validated this too; re-check here so a malformed override
|
|
# can never reach a mutation through this reusable workflow.
|
|
if test -n "${GATE_OVERRIDE_REASON}${GATE_OVERRIDE_CONFIRMATION}"; then
|
|
test "${DEPLOY_MODE}" != verify
|
|
test "${GATE_OVERRIDE_CONFIRMATION}" = \
|
|
"SKIP_RELAY_MONITOR_GATE ${TARGET_IMAGE_DIGEST}"
|
|
[[ "${GATE_OVERRIDE_REASON}" =~ ^[[:print:]]{12,500}$ ]]
|
|
fi
|
|
if test "${DEPLOY_MODE}" != verify && test "${GITHUB_RUN_ATTEMPT}" != 1; then
|
|
echo "mutations are single-dispatch: re-runs replay aged evidence," >&2
|
|
echo "so recover each remaining cell with its own fresh monitor" >&2
|
|
echo "dry-run and canary-apply dispatch instead" >&2
|
|
exit 1
|
|
fi
|
|
test -n "${GCP_REGION}"
|
|
test -n "${DEPLOY_WIF}"
|
|
test -n "${DEPLOY_SERVICE_ACCOUNT}"
|
|
test -n "${CAPACITY_WIF}"
|
|
test -n "${CAPACITY_SERVICE_ACCOUNT}"
|
|
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
|
|
|
|
# Full history: the monitor evidence this job verifies is sealed at an ancestor commit,
|
|
# and the provenance check fails closed on a commit a shallow clone left out.
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- uses: pnpm/action-setup@v4
|
|
with: { package_json_file: cloud/package.json }
|
|
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: 24
|
|
|
|
- run: pnpm install --frozen-lockfile
|
|
|
|
- uses: hashicorp/setup-terraform@v3
|
|
with: { terraform_wrapper: false }
|
|
|
|
# One approved-cell table, in the wave validator the dispatch gate already uses, so
|
|
# a cell's class and its wave's selector delta cannot drift apart between the two.
|
|
- name: Resolve this cell's admission class and wave selector delta
|
|
run: |
|
|
CELL_CLASS="$(node dev/scripts/relay-production-same-cap-wave.mjs cell-class \
|
|
--cell-id "${TARGET_CELL_ID}")"
|
|
ENTRY_ADMISSION="$(jq -er '.entryAdmission' <<< "${CELL_CLASS}")"
|
|
SELECTOR_WAVE_DELTA="$(jq -er '.selectorWaveDelta' <<< "${CELL_CLASS}")"
|
|
if test "${DEPLOY_MODE}" = verify; then
|
|
EFFECTIVE_SELECTOR_GENERATION="${EXPECTED_SELECTOR_GENERATION}"
|
|
else
|
|
EFFECTIVE_SELECTOR_GENERATION="$((EXPECTED_SELECTOR_GENERATION \
|
|
+ (SELECTOR_WAVE_DELTA * WAVE_INDEX)))"
|
|
fi
|
|
{
|
|
echo "ENTRY_ADMISSION=${ENTRY_ADMISSION}"
|
|
echo "SELECTOR_WAVE_DELTA=${SELECTOR_WAVE_DELTA}"
|
|
echo "EFFECTIVE_SELECTOR_GENERATION=${EFFECTIVE_SELECTOR_GENERATION}"
|
|
} >> "${GITHUB_ENV}"
|
|
|
|
- name: Require fresh aggregate monitor evidence reference
|
|
if: ${{ inputs.mode != 'verify' && inputs.gate-override-confirmation == '' }}
|
|
run: |
|
|
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
|
|
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
|
|
|
|
- name: Download private aggregate monitor evidence
|
|
if: ${{ inputs.mode != 'verify' && inputs.gate-override-confirmation == '' }}
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
|
|
path: ${{ github.workspace }}/relay-monitor-evidence
|
|
github-token: ${{ github.token }}
|
|
run-id: ${{ inputs.monitor-run-id }}
|
|
|
|
- name: Verify monitor evidence provenance
|
|
if: ${{ inputs.mode != 'verify' && inputs.gate-override-confirmation == '' }}
|
|
run: |
|
|
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
|
|
--directory "${OUTPUT_DIRECTORY}" \
|
|
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
|
|
--run-id "${MONITOR_RUN_ID}" \
|
|
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
|
|
--commit-sha "${GITHUB_SHA}" \
|
|
--mode dry-run \
|
|
--required-migration-policy strict \
|
|
--wave-index "${WAVE_INDEX}"
|
|
|
|
- name: Download this wave's single-use safety authority
|
|
if: ${{ inputs.mode != 'verify' && inputs.gate-override-confirmation == '' }}
|
|
uses: actions/download-artifact@v4
|
|
with:
|
|
name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
|
|
path: ${{ runner.temp }}/relay-same-cap-monitor-authority
|
|
github-token: ${{ github.token }}
|
|
run-id: ${{ github.run_id }}
|
|
|
|
- name: Require safety evidence consumed by this workflow
|
|
if: ${{ inputs.mode != 'verify' && inputs.gate-override-confirmation == '' }}
|
|
run: |
|
|
# Mutations are single-dispatch: a fresh dispatch cannot resume a
|
|
# partial batch (the canary authority binds the batch-entry selector
|
|
# generation), so each remaining cell is recovered by its own fresh
|
|
# monitor dry-run and canary-apply dispatch, never by re-running
|
|
# aged evidence.
|
|
test "${GITHUB_RUN_ATTEMPT}" = 1
|
|
MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
|
|
test "$(< "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}")" = \
|
|
"${GITHUB_RUN_ID}"
|
|
|
|
- id: deploy-auth
|
|
uses: google-github-actions/auth@v2
|
|
with:
|
|
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
|
|
token_format: id_token
|
|
id_token_audience: https://relay.onorca.dev/v1/admin/drain
|
|
id_token_include_email: true
|
|
|
|
- uses: google-github-actions/setup-gcloud@v2
|
|
|
|
- uses: ./.github/actions/cloud-sql-rollout-lease
|
|
with:
|
|
bucket: onorca-cloud-terraform-state
|
|
object: terraform/state/cloud-sql-rollout/production.lock
|
|
release: 'false'
|
|
|
|
- name: Recheck aggregate SQL, pool, reconnect, migration, and selector safety
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
|
|
run: |
|
|
# Freshness-only failures are publish lag, not health, on every wave
|
|
# including the first; the CLI still caps the retry at the wave's
|
|
# evidence-age budget, so this cannot mutate on aged evidence.
|
|
#
|
|
# This live recheck runs on every mutating wave, including a
|
|
# break-glass one. With the aggregate gate overridden there is no
|
|
# sealed state to read, so the expected selector comes from the
|
|
# dispatch inputs the rehome inspect below verifies against the live
|
|
# director; every threshold the sample is judged against is unchanged.
|
|
if test -n "${GATE_OVERRIDE_CONFIRMATION}"; then
|
|
jq -n \
|
|
--arg existingOnly "${EXPECTED_EXISTING_ONLY_CELLS/none/}" \
|
|
--arg migrationOnly "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
|
|
--arg general "${EXPECTED_GENERAL_CELLS/none/}" \
|
|
'{existingOnly:$existingOnly,migrationOnly:$migrationOnly,general:$general}
|
|
| map_values(split(",") | map(select(length > 0)))' \
|
|
> "${RUNNER_TEMP}/relay-same-cap-selector.json"
|
|
pnpm incident:relay-preflight -- \
|
|
--no-monitor-state \
|
|
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
|
|
--selector-membership-file "${RUNNER_TEMP}/relay-same-cap-selector.json" \
|
|
--wave-index "${WAVE_INDEX}" \
|
|
--selector-wave-delta "${SELECTOR_WAVE_DELTA}" --retry-freshness
|
|
else
|
|
pnpm incident:relay-preflight -- \
|
|
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
|
|
--wave-index "${WAVE_INDEX}" \
|
|
--selector-wave-delta "${SELECTOR_WAVE_DELTA}" --retry-freshness
|
|
fi
|
|
|
|
- name: Require durable rehome disabled and exact selector
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
|
|
run: |
|
|
node dev/scripts/operate-relay-regional-rehome.mjs \
|
|
--mode inspect \
|
|
--director-origin "${DIRECTOR_ORIGIN}" \
|
|
--expected-selector-generation "${EFFECTIVE_SELECTOR_GENERATION}" \
|
|
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
|
|
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
|
|
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
|
|
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
|
|
| jq -e '.control.enabled == false' >/dev/null
|
|
|
|
- name: Initialize the exact production backend
|
|
run: node dev/scripts/infra.mjs init --env production
|
|
|
|
- name: Resolve immutable same-cap cell configuration
|
|
shell: bash
|
|
run: |
|
|
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
|
|
case "${TARGET_HOSTNAME}" in
|
|
c7|c8|c9|c10|c13|c14|c15|c16|c19|c20|c21|c22|c23|c24|c25|c26)
|
|
EXPECTED_HARD_CAP=1000
|
|
EXPECTED_REGION=us-central1
|
|
EXPECTED_DATABASE_POOL_MAX=
|
|
;;
|
|
c17|c18)
|
|
EXPECTED_HARD_CAP=600
|
|
EXPECTED_REGION=us-central1
|
|
EXPECTED_DATABASE_POOL_MAX=
|
|
;;
|
|
c27|c28|c29)
|
|
EXPECTED_HARD_CAP=3000
|
|
EXPECTED_REGION=asia-east2
|
|
EXPECTED_DATABASE_POOL_MAX=16
|
|
;;
|
|
*) exit 1 ;;
|
|
esac
|
|
EXPECTED_UNOBSERVED_BOUND=60
|
|
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
|
|
CELLS_JSON="$(terraform -chdir=infra/terraform console \
|
|
-var-file=environments/production.tfvars \
|
|
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
|
|
SOURCE_CELLS="$(terraform -chdir=infra/terraform console \
|
|
-var-file=environments/production.tfvars \
|
|
<<< 'jsonencode(var.relay_region_rehome_source_cell_ids)' | jq -er '.')"
|
|
CURRENT_SHAPE="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${CELLS_JSON}")"
|
|
test "$(jq -r '.connection_hard_cap' <<< "${CURRENT_SHAPE}")" = "${EXPECTED_HARD_CAP}"
|
|
test "$(jq -r '.connection_unobserved_bound' <<< "${CURRENT_SHAPE}")" = \
|
|
"${EXPECTED_UNOBSERVED_BOUND}"
|
|
# The startup script emits a pool line only off the root default, so an unpinned cell
|
|
# must still be on that default or its plan would carry a line nothing reviews.
|
|
test "$(jq -r '.database_pool_max' <<< "${CURRENT_SHAPE}")" = \
|
|
"${EXPECTED_DATABASE_POOL_MAX:-10}"
|
|
TARGET_ZONE="$(jq -r '.zone' <<< "${CURRENT_SHAPE}")"
|
|
MIG_NAME="orca-cloud-relay-gce-${TARGET_HOSTNAME}"
|
|
if test "${DEPLOY_MODE}" = rollback; then
|
|
DESIRED_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
|
|
CURRENT_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
|
|
DESIRED_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
|
|
CURRENT_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
|
|
else
|
|
DESIRED_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
|
|
CURRENT_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
|
|
DESIRED_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
|
|
CURRENT_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
|
|
fi
|
|
# The startup template emits rehome trust lines only for a declared source cell, so
|
|
# require membership exactly when either side of this roll expects those lines.
|
|
if test "${EXPECTED_REGION}" = us-central1 && {
|
|
test "${DESIRED_REHOME_PROTOCOL}" != 0 || test "${CURRENT_REHOME_PROTOCOL}" != 0
|
|
}; then
|
|
jq -e --arg cell "${TARGET_CELL_ID}" 'index($cell) != null' \
|
|
<<< "${SOURCE_CELLS}" >/dev/null
|
|
fi
|
|
DESIRED_IMAGE="${IMAGE_REPOSITORY}@${DESIRED_IMAGE_DIGEST}"
|
|
OVERRIDE_CELLS_JSON="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
|
|
--arg image "${DESIRED_IMAGE}" '.[$cell].image = $image' <<< "${CELLS_JSON}")"
|
|
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
|
|
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-same-cap.tfvars.json"
|
|
SERVED_DIGEST="$(gcloud artifacts docker images describe "${DESIRED_IMAGE}" \
|
|
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
|
|
test "${SERVED_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"
|
|
{
|
|
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
|
|
echo "CELL_ORIGIN=${CELL_ORIGIN}"
|
|
echo "TARGET_ZONE=${TARGET_ZONE}"
|
|
echo "MIG_NAME=${MIG_NAME}"
|
|
echo "EXPECTED_HARD_CAP=${EXPECTED_HARD_CAP}"
|
|
echo "EXPECTED_UNOBSERVED_BOUND=${EXPECTED_UNOBSERVED_BOUND}"
|
|
echo "EXPECTED_DATABASE_POOL_MAX=${EXPECTED_DATABASE_POOL_MAX}"
|
|
echo "EXPECTED_REGION=${EXPECTED_REGION}"
|
|
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
|
|
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
|
|
echo "CURRENT_IMAGE_DIGEST=${CURRENT_IMAGE_DIGEST}"
|
|
echo "DESIRED_REHOME_PROTOCOL=${DESIRED_REHOME_PROTOCOL}"
|
|
echo "CURRENT_REHOME_PROTOCOL=${CURRENT_REHOME_PROTOCOL}"
|
|
} >> "${GITHUB_ENV}"
|
|
|
|
- name: Verify exact current generation, digest, cap, and rollback point
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
|
|
run: |
|
|
# A single transient 5xx (LB warm-up behind a fresh instance) must not
|
|
# fail a canary; 4xx (auth, generation mismatch) still fails fast.
|
|
admin_post() {
|
|
local out="${RUNNER_TEMP}/$1.json"
|
|
if ! curl --fail-with-body --max-time 30 \
|
|
--retry 3 --retry-delay 2 --retry-connrefused --output "${out}" \
|
|
--request POST "$2" \
|
|
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
|
|
--header 'Content-Type: application/json' --data "$3"; then
|
|
cat "${out}" >&2
|
|
return 1
|
|
fi
|
|
cat "${out}"
|
|
}
|
|
CURRENT_RUNTIME="$(admin_post current-runtime \
|
|
"${CELL_ORIGIN}/v1/admin/runtime-status" '{"v":1}')"
|
|
# Two different failures leave the cell on the rollback image, and the image
|
|
# alone cannot tell them apart. A rollback that failed between its template
|
|
# apply and its admission restore restarted the cell, so that cell is not
|
|
# draining and resumes. A wave that stopped after its drain and before its
|
|
# template apply never restarted anything, so its cell is still draining and
|
|
# is stranded: the drain flag only clears on a restart, so it has to be rolled.
|
|
LIVE_IMAGE_DIGEST="$(jq -r '.imageDigest' <<< "${CURRENT_RUNTIME}")"
|
|
LIVE_DRAINING="$(jq -r '.draining' <<< "${CURRENT_RUNTIME}")"
|
|
if test "${DEPLOY_MODE}" = rollback \
|
|
&& test "${LIVE_IMAGE_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"; then
|
|
if test "${LIVE_DRAINING}" = true; then
|
|
ROLLBACK_STAGE=stranded
|
|
else
|
|
ROLLBACK_STAGE=resume
|
|
fi
|
|
PREDECESSOR_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
|
|
PREDECESSOR_REHOME_PROTOCOL="${DESIRED_REHOME_PROTOCOL}"
|
|
else
|
|
ROLLBACK_STAGE=roll
|
|
PREDECESSOR_IMAGE_DIGEST="${CURRENT_IMAGE_DIGEST}"
|
|
PREDECESSOR_REHOME_PROTOCOL="${CURRENT_REHOME_PROTOCOL}"
|
|
fi
|
|
if test "${ROLLBACK_STAGE}" = resume; then
|
|
ROLLBACK_RESUME=true
|
|
else
|
|
ROLLBACK_RESUME=false
|
|
fi
|
|
# A stranded cell's template still carries the image the cell is serving, so that
|
|
# is the predecessor its plan is reviewed against. A template already moved on to
|
|
# the target is refused here rather than rolled backwards under a stale review.
|
|
if test "${ROLLBACK_STAGE}" = stranded; then
|
|
PLAN_ROLLBACK_IMAGE="${DESIRED_IMAGE}"
|
|
else
|
|
PLAN_ROLLBACK_IMAGE="${IMAGE_REPOSITORY}@${CURRENT_IMAGE_DIGEST}"
|
|
fi
|
|
# Rollback is the documented recovery from a failed canary, which
|
|
# leaves the cell migration-only (and possibly still marked
|
|
# draining); apply and verify still require the cell pristine in the
|
|
# class it is declared to serve in.
|
|
if test "${DEPLOY_MODE}" = rollback; then
|
|
PRECHECK_ADMISSION=general-or-migration-only
|
|
else
|
|
PRECHECK_ADMISSION="${ENTRY_ADMISSION}"
|
|
fi
|
|
# Draining sheds connections, and a migration-only cell holds none, so the flag
|
|
# carries no precondition there. It also outlives a failed wave, because the drain
|
|
# that set it is followed by no restart, which is the state a failed canary leaves.
|
|
if test "${DEPLOY_MODE}" = rollback \
|
|
|| test "${ENTRY_ADMISSION}" = migration-only; then
|
|
PRECHECK_DRAINING=either
|
|
else
|
|
PRECHECK_DRAINING=forbidden
|
|
fi
|
|
# A resumed rollback already restarted, so its cell has to come back not draining;
|
|
# that is what separates it from a wave that stopped before its template apply.
|
|
if test "${PRECHECK_DRAINING}" = either && test "${ROLLBACK_RESUME}" != true; then
|
|
PREDECESSOR_DRAINING_OK=true
|
|
else
|
|
PREDECESSOR_DRAINING_OK=false
|
|
fi
|
|
RESTORED_MIGRATION_CELLS="$(jq -rn \
|
|
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
|
|
--arg target "${TARGET_CELL_ID}" \
|
|
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
|
|
RESTORED_GENERAL_CELLS="$(jq -rn \
|
|
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
|
|
--arg target "${TARGET_CELL_ID}" \
|
|
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
|
|
test -n "${RESTORED_MIGRATION_CELLS}" || RESTORED_MIGRATION_CELLS=none
|
|
test -n "${RESTORED_GENERAL_CELLS}" || RESTORED_GENERAL_CELLS=none
|
|
ISOLATED_MIGRATION_CELLS="$(jq -rn \
|
|
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
|
|
--arg target "${TARGET_CELL_ID}" \
|
|
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
|
|
ISOLATED_GENERAL_CELLS="$(jq -rn \
|
|
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
|
|
--arg target "${TARGET_CELL_ID}" \
|
|
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
|
|
test -n "${ISOLATED_MIGRATION_CELLS}" || ISOLATED_MIGRATION_CELLS=none
|
|
test -n "${ISOLATED_GENERAL_CELLS}" || ISOLATED_GENERAL_CELLS=none
|
|
# A migration-only cell is already isolated and is handed back isolated, so both
|
|
# halves of its wave see exactly the membership it entered with.
|
|
if test "${ENTRY_ADMISSION}" = migration-only; then
|
|
RESTORED_MIGRATION_CELLS="${ISOLATED_MIGRATION_CELLS}"
|
|
RESTORED_GENERAL_CELLS="${ISOLATED_GENERAL_CELLS}"
|
|
fi
|
|
{
|
|
echo "ROLLBACK_RESUME=${ROLLBACK_RESUME}"
|
|
echo "ROLLBACK_STAGE=${ROLLBACK_STAGE}"
|
|
echo "PLAN_ROLLBACK_IMAGE=${PLAN_ROLLBACK_IMAGE}"
|
|
# The drain wait and the plan review both read the image this cell actually
|
|
# serves, which is the rollback image on a stranded cell and not the current one.
|
|
echo "PREDECESSOR_IMAGE_DIGEST=${PREDECESSOR_IMAGE_DIGEST}"
|
|
# The failsafe consumes these; deriving them here keeps them
|
|
# defined for a failure in any later step.
|
|
echo "ISOLATED_MIGRATION_CELLS=${ISOLATED_MIGRATION_CELLS}"
|
|
echo "ISOLATED_GENERAL_CELLS=${ISOLATED_GENERAL_CELLS}"
|
|
# No restart happens on resume, so isolate below is skipped and
|
|
# cannot advance the selector generation.
|
|
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${EFFECTIVE_SELECTOR_GENERATION}"
|
|
# A failed-canary rollback enters with the target migration-only,
|
|
# so the restore inspect cannot reuse the entry membership inputs.
|
|
echo "RESTORED_MIGRATION_CELLS=${RESTORED_MIGRATION_CELLS}"
|
|
echo "RESTORED_GENERAL_CELLS=${RESTORED_GENERAL_CELLS}"
|
|
} >> "${GITHUB_ENV}"
|
|
if ! jq -e --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
|
|
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
|
|
--arg region "${EXPECTED_REGION}" \
|
|
--argjson hardCap "${EXPECTED_HARD_CAP}" \
|
|
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
|
|
--argjson drainingOk "${PREDECESSOR_DRAINING_OK}" \
|
|
'.role == "cell" and .cellId == $cell and .cellUrl == $origin and
|
|
(.region == $region or
|
|
($region == "us-central1" and $protocol == 0 and .region == null)) and
|
|
.imageDigest == $digest and
|
|
.connectionCapacity.hardCap == $hardCap and
|
|
.connectionCapacity.unobservedBound == $unobservedBound and
|
|
(.draining == false or $drainingOk) and
|
|
(.regionalRehomeProtocol // 0) == $protocol' <<< "${CURRENT_RUNTIME}" >/dev/null
|
|
then
|
|
jq -r --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
|
|
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
|
|
--arg region "${EXPECTED_REGION}" \
|
|
--argjson hardCap "${EXPECTED_HARD_CAP}" \
|
|
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
|
|
--argjson drainingOk "${PREDECESSOR_DRAINING_OK}" \
|
|
'[
|
|
if .role != "cell" then "role" else empty end,
|
|
if .cellId != $cell then "cellId" else empty end,
|
|
if .cellUrl != $origin then "cellUrl" else empty end,
|
|
if (.region != $region and
|
|
($region != "us-central1" or $protocol != 0 or .region != null))
|
|
then "region" else empty end,
|
|
if .imageDigest != $digest then "imageDigest" else empty end,
|
|
if .connectionCapacity.hardCap != $hardCap then "hardCap" else empty end,
|
|
if .connectionCapacity.unobservedBound != $unobservedBound then "unobservedBound" else empty end,
|
|
if (.draining != false and ($drainingOk | not)) then "draining" else empty end,
|
|
if (.regionalRehomeProtocol // 0) != $protocol then "regionalRehomeProtocol" else empty end
|
|
] | "runtime predecessor mismatch fields=" + join(",")' \
|
|
<<< "${CURRENT_RUNTIME}" >&2
|
|
exit 1
|
|
fi
|
|
# The exact legacy digest binds omitted pre-region fields to US and protocol 0.
|
|
jq -r '[
|
|
if .region == null then "region" else empty end,
|
|
if .regionalRehomeProtocol == null then "regionalRehomeProtocol" else empty end
|
|
] | if length > 0 then "runtime predecessor normalized legacy fields=" + join(",") else empty end' \
|
|
<<< "${CURRENT_RUNTIME}"
|
|
CURRENT_DIRECTOR_STATUS="$(admin_post current-cell-status \
|
|
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
|
|
"$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
|
|
SOURCE_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
|
|
<<< "${CURRENT_DIRECTOR_STATUS}")"
|
|
if test "${ROLLBACK_RESUME}" = true && ! jq -e \
|
|
'.status.admissionState == "migration-only"' \
|
|
<<< "${CURRENT_DIRECTOR_STATUS}" >/dev/null; then
|
|
echo 'resume requires the isolated migration-only cell a failed rollback leaves' >&2
|
|
exit 1
|
|
fi
|
|
[[ "${SOURCE_INCARNATION}" =~ ^[0-9a-f-]{36}$ ]]
|
|
echo "SOURCE_INCARNATION=${SOURCE_INCARNATION}" >> "${GITHUB_ENV}"
|
|
node dev/scripts/verify-relay-capacity-transition.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--heartbeat fresh --admission "${PRECHECK_ADMISSION}" \
|
|
--draining "${PRECHECK_DRAINING}" --activity allowed \
|
|
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
|
|
|
|
- name: Finish read-only verification
|
|
if: ${{ inputs.mode == 'verify' }}
|
|
run: echo 'Exact same-cap rollback point verified.'
|
|
|
|
- name: Reversibly isolate and drain only the selected cell
|
|
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
|
|
id: drain
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
|
|
run: |
|
|
# Opens the window the report-only shadow health gate below judges this cell over.
|
|
echo "drain-started-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
|
|
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
|
|
# A cell isolated by a failed canary is already migration-only, so
|
|
# isolate is a no-op there that does not advance the selector; the
|
|
# result's generation is authoritative either way.
|
|
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode isolate)"
|
|
echo "${ISOLATE_RESULT}"
|
|
# Isolating a migration-only cell must be a read-only no-op; a change here would
|
|
# mean the live class is not the one this wave planned around.
|
|
if test "${ENTRY_ADMISSION}" = migration-only; then
|
|
jq -e '.changed == false' <<< "${ISOLATE_RESULT}" >/dev/null
|
|
fi
|
|
# The director re-places hosts off a cell only when the isolate stamped it, so a
|
|
# script too old to ask for the stamp produces today's behaviour and the canary
|
|
# reads as "the fix did nothing" with nothing to tell that from a wrong premise.
|
|
jq -e '.rollIsolated == true' <<< "${ISOLATE_RESULT}" >/dev/null
|
|
ISOLATE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
|
|
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${ISOLATE_GENERATION}" >> "${GITHUB_ENV}"
|
|
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode drain \
|
|
--pace-window-ms "${DRAIN_PACE_WINDOW_MS}"
|
|
# The wait has to outlast the pacing window as well as the leases it waits on.
|
|
node dev/scripts/verify-relay-capacity-transition.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--heartbeat either --admission migration-only --draining required \
|
|
--activity restart-safe --expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}" \
|
|
--timeout-ms 1020000
|
|
|
|
- id: capacity-auth
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
uses: google-github-actions/auth@v2
|
|
with:
|
|
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
|
|
|
|
- name: Require converged Terraform state and a stable MIG on resume
|
|
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME == 'true' }}
|
|
shell: bash
|
|
env:
|
|
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
|
|
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
|
|
run: |
|
|
# A cell on the root pool default emits no pool line, so pin one only where it exists.
|
|
POOL_ARGUMENTS=()
|
|
if test -n "${EXPECTED_DATABASE_POOL_MAX}"; then
|
|
POOL_ARGUMENTS=(--database-pool-max "${EXPECTED_DATABASE_POOL_MAX}")
|
|
fi
|
|
# Zero resource changes prove the prior run's apply completed and no
|
|
# restart will follow, keeping the incarnation check honest. Root
|
|
# outputs may lag a targeted apply, so judge resource_changes only.
|
|
# The backend service is targeted too, so its reviewed drain timeout
|
|
# and request logging can be the only thing left here; neither
|
|
# restarts an instance, so the validator below clears that on its
|
|
# own, without the template-and-MIG pair.
|
|
terraform -chdir=infra/terraform plan \
|
|
-var-file=environments/production.tfvars \
|
|
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
|
|
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
|
|
if ! terraform -chdir=infra/terraform show -json \
|
|
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
|
|
| jq -e '[.resource_changes[]?
|
|
| select(.change.actions | any(. != "no-op" and . != "read"))]
|
|
| length == 0' >/dev/null
|
|
then
|
|
# An apply that failed before its template apply also resumes here
|
|
# (the cell still serves the rollback image), and repo drift since
|
|
# the cell's last roll (for example newly added rehome trust
|
|
# config) then legitimately replaces the template. Nothing is
|
|
# applied on resume either way, so accept exactly the drift the
|
|
# reviewed validator would let a real apply ship for the image the
|
|
# cell already serves: the template leaves and re-enters the
|
|
# rollback image, as exactly the template-and-MIG change pair.
|
|
terraform -chdir=infra/terraform show -json \
|
|
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
|
|
| jq -r '"resume found unconverged resources: " +
|
|
([.resource_changes[]?
|
|
| select(.change.actions | any(. != "no-op" and . != "read"))
|
|
| .address] | join(","))'
|
|
echo 'requiring reviewed rollback-image drift'
|
|
RESUME_REVIEW="$(terraform -chdir=infra/terraform show -json \
|
|
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
|
|
| node dev/scripts/validate-relay-capacity-plan.mjs \
|
|
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
|
|
--hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--image "${DESIRED_IMAGE}" \
|
|
--rollback-image "${DESIRED_IMAGE}" \
|
|
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
|
|
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
|
|
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
|
|
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
|
|
"${POOL_ARGUMENTS[@]}")"
|
|
echo "${RESUME_REVIEW}"
|
|
jq -e '.changes == 2
|
|
or (.changes == 0 and ((.backendUpdate // []) | length) > 0)' \
|
|
<<< "${RESUME_REVIEW}" >/dev/null
|
|
# changes == 0 here means the template and MIG are converged and this cell's
|
|
# reviewed backend update is the only thing left, so the resume is not complete:
|
|
# apply it, or the cell silently keeps the 300-second drain and no request
|
|
# logging and the operator reads that as a finished roll. The plan holds nothing
|
|
# else (the validator bounded it to this cell's backend, and the template and MIG
|
|
# are no-ops in it), and neither attribute restarts an instance, so the
|
|
# incarnation check downstream stays honest. Template-and-MIG drift still applies
|
|
# nothing, which is what a resume means.
|
|
if test "$(jq -er '.changes' <<< "${RESUME_REVIEW}")" = 0; then
|
|
terraform -chdir=infra/terraform apply -auto-approve \
|
|
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
|
|
fi
|
|
fi
|
|
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
|
|
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
|
|
|
|
- name: Apply only the selected same-cap template and MIG
|
|
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
|
|
id: apply
|
|
shell: bash
|
|
env:
|
|
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
|
|
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
|
|
run: |
|
|
# A cell on the root pool default emits no pool line, so pin one only where it exists.
|
|
POOL_ARGUMENTS=()
|
|
if test -n "${EXPECTED_DATABASE_POOL_MAX}"; then
|
|
POOL_ARGUMENTS=(--database-pool-max "${EXPECTED_DATABASE_POOL_MAX}")
|
|
fi
|
|
terraform -chdir=infra/terraform plan \
|
|
-var-file=environments/production.tfvars \
|
|
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
|
|
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
"-target=google_compute_backend_service.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
|
|
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
|
|
PLAN_REVIEW="$(terraform -chdir=infra/terraform show -json \
|
|
"${RUNNER_TEMP}/relay-same-cap.tfplan" \
|
|
| node dev/scripts/validate-relay-capacity-plan.mjs \
|
|
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
|
|
--hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" --image "${DESIRED_IMAGE}" \
|
|
--rollback-image "${PLAN_ROLLBACK_IMAGE}" \
|
|
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
|
|
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
|
|
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
|
|
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
|
|
"${POOL_ARGUMENTS[@]}")"
|
|
echo "${PLAN_REVIEW}"
|
|
# Stamped before the apply, not after it: the new container announces its listener while
|
|
# the MIG is still converging, so a bound taken at the end of this step is already past
|
|
# the announcement the shadow gate looks for.
|
|
echo "apply-started-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
|
|
terraform -chdir=infra/terraform apply -auto-approve \
|
|
"${RUNNER_TEMP}/relay-same-cap.tfplan"
|
|
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
|
|
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
|
|
# A stranded cell already runs the reviewed template, so the apply above replaces
|
|
# no instance and the drain flag, which only a restart clears, would survive the
|
|
# whole wave. Roll the MIG explicitly on exactly the policy a template change uses.
|
|
# Every field is passed: gcloud persists these into the MIG's update policy, and it
|
|
# defaults the method to substitute on a group with no stateful config, so omitting
|
|
# one drifts the policy off the reviewed one and fails every later targeted plan.
|
|
if test "${ROLLBACK_STAGE}" = stranded \
|
|
&& test "$(jq -er '.changes' <<< "${PLAN_REVIEW}")" = 0; then
|
|
gcloud compute instance-groups managed rolling-action replace "${MIG_NAME}" \
|
|
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
|
|
--replacement-method recreate --max-surge 0 --max-unavailable 1
|
|
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
|
|
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
|
|
fi
|
|
# Recorded for the operator comparing verdicts; the gate's boot search opens at the
|
|
# apply-started-at stamp above, not here.
|
|
echo "apply-completed-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
|
|
|
|
- id: post-auth
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
uses: google-github-actions/auth@v2
|
|
with:
|
|
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
|
|
token_format: id_token
|
|
id_token_audience: https://relay.onorca.dev/v1/admin/drain
|
|
id_token_include_email: true
|
|
|
|
- name: Verify new incarnation, exact image, protocol, and durable safety
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
id: verify-target
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
|
|
run: |
|
|
# A single transient 5xx (LB warm-up behind a fresh instance) must not
|
|
# fail a canary; 4xx (auth, generation mismatch) still fails fast.
|
|
admin_post() {
|
|
local out="${RUNNER_TEMP}/$1.json"
|
|
if ! curl --fail-with-body --max-time 30 \
|
|
--retry 3 --retry-delay 2 --retry-connrefused --output "${out}" \
|
|
--request POST "$2" \
|
|
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
|
|
--header 'Content-Type: application/json' --data "$3"; then
|
|
cat "${out}" >&2
|
|
return 1
|
|
fi
|
|
cat "${out}"
|
|
}
|
|
node dev/scripts/verify-relay-capacity-transition.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--heartbeat fresh --admission migration-only --draining forbidden \
|
|
--activity allowed --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
|
|
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" --timeout-ms 900000
|
|
TARGET_RUNTIME="$(admin_post target-runtime \
|
|
"${CELL_ORIGIN}/v1/admin/runtime-status" '{"v":1}')"
|
|
jq -e --arg digest "${DESIRED_IMAGE_DIGEST}" \
|
|
--argjson protocol "${DESIRED_REHOME_PROTOCOL}" \
|
|
'.imageDigest == $digest and (.regionalRehomeProtocol // 0) == $protocol' \
|
|
<<< "${TARGET_RUNTIME}" >/dev/null
|
|
TARGET_DIRECTOR_STATUS="$(admin_post target-cell-status \
|
|
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
|
|
"$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
|
|
TARGET_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
|
|
<<< "${TARGET_DIRECTOR_STATUS}")"
|
|
if test "${ROLLBACK_RESUME}" = true; then
|
|
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
|
|
# No restart happened; the incarnation legitimately stays put.
|
|
test "${TARGET_INCARNATION}" = "${SOURCE_INCARNATION}"
|
|
else
|
|
test "${TARGET_INCARNATION}" != "${SOURCE_INCARNATION}"
|
|
fi
|
|
echo "TARGET_INCARNATION=${TARGET_INCARNATION}" >> "${GITHUB_ENV}"
|
|
node dev/scripts/operate-relay-regional-rehome.mjs \
|
|
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
|
|
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ISOLATE}" \
|
|
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
|
|
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
|
|
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
|
|
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
|
|
| jq -e '.control.enabled == false' >/dev/null
|
|
echo "verify-ended-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
|
|
|
|
- name: Prove exact per-host trust and idempotent no-neighbor behavior
|
|
if: ${{ inputs.mode != 'verify' && ((inputs.mode == 'rollback' && inputs.rollback-rehome-protocol != '0') || (inputs.mode != 'rollback' && inputs.target-rehome-protocol != '0')) }}
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
|
|
run: |
|
|
node dev/scripts/probe-relay-rehome-trust.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-id "${TARGET_CELL_ID}" \
|
|
--cell-incarnation "${TARGET_INCARNATION}"
|
|
|
|
- name: Restore only the verified selected cell to its entry admission
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
|
|
run: |
|
|
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
|
|
# Activating a migration-only cell would promote it to a serving cell for good, so
|
|
# restore it with the idempotent isolate that reports the authoritative generation.
|
|
if test "${ENTRY_ADMISSION}" = migration-only; then
|
|
RESTORE_MODE=isolate
|
|
else
|
|
RESTORE_MODE=activate
|
|
fi
|
|
RESTORE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode "${RESTORE_MODE}")"
|
|
echo "${RESTORE_RESULT}"
|
|
SELECTOR_GENERATION_AFTER_RESTORE="$(jq -er '.generation' \
|
|
<<< "${RESTORE_RESULT}")"
|
|
node dev/scripts/verify-relay-capacity-transition.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
|
|
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
|
|
--heartbeat fresh --admission "${ENTRY_ADMISSION}" \
|
|
--draining forbidden --activity allowed \
|
|
--expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
|
|
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}"
|
|
node dev/scripts/operate-relay-regional-rehome.mjs \
|
|
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
|
|
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_RESTORE}" \
|
|
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
|
|
--expected-migration-only-cells "${RESTORED_MIGRATION_CELLS}" \
|
|
--expected-general-cells "${RESTORED_GENERAL_CELLS}" \
|
|
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
|
|
| jq -e '.control.enabled == false' >/dev/null
|
|
|
|
# Report only: this evaluates the oracles an operator reads by hand after a cell and records
|
|
# what it would have called, so its verdicts can be compared with the operator's over a full
|
|
# roll before it is ever allowed to block. Two independent guarantees keep it inert: the
|
|
# script exits 0 on every verdict, and continue-on-error keeps even a crash off the job's
|
|
# outcome. The failsafe below therefore cannot fire on anything this step observes.
|
|
#
|
|
# It runs after the restore, not before it, for two reasons: the cell goes back into
|
|
# admission on exactly today's schedule rather than waiting out a minute of log reads, and
|
|
# the window it judges has closed by then, so Cloud Logging's ingestion lag is behind it.
|
|
# These are fleet oracles anyway; when this does gate, what it gates is the next cell.
|
|
- name: Shadow health gate (report only)
|
|
id: shadow-gate
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
continue-on-error: true
|
|
# continue-on-error bounds this step's contribution to the job outcome, not its clock,
|
|
# and its reads are serialised. A timed-out step is a failed step, which continue-on-error
|
|
# absorbs; without this bound a Logging 429 storm or an expired credential makes every
|
|
# read cost its full retry budget and can push the job past timeout-minutes, and a
|
|
# cancelled job takes the whole wave with it. A healthy gate is already minutes of
|
|
# serial reads, so both bounds sit above that: the script settles at seven minutes and
|
|
# reaching this eight is the pathological case. Eight on top of a ~14-minute cell still
|
|
# leaves the job's 75 minutes intact.
|
|
timeout-minutes: 8
|
|
env:
|
|
DRAIN_STARTED_AT: ${{ steps.drain.outputs.drain-started-at }}
|
|
APPLY_STARTED_AT: ${{ steps.apply.outputs.apply-started-at }}
|
|
APPLY_COMPLETED_AT: ${{ steps.apply.outputs.apply-completed-at }}
|
|
VERIFY_ENDED_AT: ${{ steps.verify-target.outputs.verify-ended-at }}
|
|
SHADOW_GATE_DIRECTORY: ${{ runner.temp }}/relay-same-cap-shadow-gate
|
|
SHADOW_GATE_NAME: relay-same-cap-shadow-gate-${{ inputs.target-cell-id }}-${{ github.run_id }}.json
|
|
run: |
|
|
mkdir -p "${SHADOW_GATE_DIRECTORY}"
|
|
node dev/scripts/relay-same-cap-shadow-gate.mjs \
|
|
--cell-id "${TARGET_CELL_ID}" \
|
|
--cell-host "${TARGET_HOSTNAME}.relay.onorca.dev" \
|
|
--project-id "${GCP_PROJECT_ID}" \
|
|
--director-service orca-cloud-relay \
|
|
--drain-started-at "${DRAIN_STARTED_AT}" \
|
|
--apply-started-at "${APPLY_STARTED_AT}" \
|
|
--apply-completed-at "${APPLY_COMPLETED_AT}" \
|
|
--verify-ended-at "${VERIFY_ENDED_AT}" \
|
|
--summary-file "${GITHUB_STEP_SUMMARY}" \
|
|
--output-file "${SHADOW_GATE_DIRECTORY}/${SHADOW_GATE_NAME}"
|
|
|
|
- name: Publish the shadow health gate verdict
|
|
if: ${{ inputs.mode != 'verify' }}
|
|
continue-on-error: true
|
|
# One small JSON file; a retrying upload must not spend the wave's remaining minutes either.
|
|
timeout-minutes: 2
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: relay-same-cap-shadow-gate-${{ inputs.target-cell-id }}-${{ github.run_id }}.json
|
|
path: ${{ runner.temp }}/relay-same-cap-shadow-gate
|
|
if-no-files-found: warn
|
|
retention-days: 14
|
|
overwrite: true
|
|
|
|
- id: cleanup-auth
|
|
if: ${{ failure() && inputs.mode != 'verify' }}
|
|
uses: google-github-actions/auth@v2
|
|
with:
|
|
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
|
|
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
|
|
token_format: id_token
|
|
id_token_audience: https://relay.onorca.dev/v1/admin/drain
|
|
id_token_include_email: true
|
|
|
|
- name: Keep a failed cell isolated and rehome disabled
|
|
if: ${{ failure() && inputs.mode != 'verify' }}
|
|
continue-on-error: true
|
|
env:
|
|
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
|
|
run: |
|
|
test "${MUTATION_STARTED:-false}" = true || exit 0
|
|
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
|
|
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
|
|
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode isolate)"
|
|
echo "${ISOLATE_RESULT}"
|
|
# Same reason as the isolate step above: a failed wave leaves this cell isolated
|
|
# deliberately, and the stamp is what lets its hosts leave.
|
|
jq -e '.rollIsolated == true' <<< "${ISOLATE_RESULT}" >/dev/null
|
|
# The isolate result carries the authoritative post-isolate generation;
|
|
# fixed offsets are wrong whenever an earlier isolate was a no-op.
|
|
FAILSAFE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
|
|
node dev/scripts/operate-relay-regional-rehome.mjs \
|
|
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
|
|
--expected-selector-generation "${FAILSAFE_GENERATION}" \
|
|
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
|
|
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
|
|
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
|
|
--expected-control-generation "${EXPECTED_REHOME_GENERATION}"
|