diff --git a/.github/actions/cloud-sql-rollout-lease/README.md b/.github/actions/cloud-sql-rollout-lease/README.md new file mode 100644 index 00000000000..e5042e11ad4 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/README.md @@ -0,0 +1,152 @@ +# Cloud SQL rollout lease + +A compare-and-swap lease on one Cloud Storage object, used to serialize Cloud SQL +**connection-budget** rollouts across two repositories. + +`concurrency.group: production-cloud-sql-rollout` only serializes runs inside a single repository. +Once the relay workflows live in `stablyai/orca` and the app workflows stay in +`stablyai/orca-cloud`, there are two independent queues pointed at one shared Cloud SQL instance. +`relay-cloud-sql-connection-budget.mjs` computes `rolloutOverlap` as a `Math.max` over the relay +director, api, auth and relay-cell candidates, which is only sound when exactly one rollout is in +flight. This lease is what keeps that assumption true. Keep the per-repo concurrency groups **and** +the lease; they solve different halves of the problem. + +## What it protects + +No workflow runs a Cloud SQL schema migration. Every locked workflow either deploys a Cloud Run +revision or applies a GCE instance template against the shared instance, so the lease must cover +**all rollouts**, not just migrations. + +## Usage + +The lease step must run **after** `google-github-actions/setup-gcloud`, and after +`actions/checkout` — `uses: ./.github/actions/...` resolves against the checked-out workspace. +It belongs in the first job of the workflow that holds a GCP credential, which is not always the +gate job: `deploy-relay-production-same-cap`'s gate runs no `gcloud`, so its first acquire happens +in the first cell job. + +```yaml +- uses: google-github-actions/auth@v2 + with: { workload_identity_provider: ..., service_account: ... } +- uses: google-github-actions/setup-gcloud@v2 +- uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock +``` + +Buckets and objects in use: + +| Environment | Bucket | Object | +| ----------- | ----------------------------------- | ------------------------------------------------- | +| production | `onorca-cloud-terraform-state` | `terraform/state/cloud-sql-rollout/production.lock` | +| staging | `onorca-cloud-staging-terraform-state` | `terraform/state/cloud-sql-rollout/staging.lock` | + +Workflows that serve both environments (`deploy-relay-asia-topology`, +`operate-relay-asia-admission`) select the pair with an `inputs.environment == 'production'` +ternary on both `bucket` and `object`. `deploy-staging` keeps its own `deploy-artifacts-staging` +concurrency group but takes the staging lease, because it rolls the staging API revision. + +The object sits beside `terraform/state/relay-fence-broker/.lock`. The IAM grant names both the +relay and app service accounts, so it is a **foundation-root** resource: `roles/storage.objectAdmin` +conditioned on the `terraform/state/cloud-sql-rollout/` prefix, **plus** an unconditioned +`roles/storage.legacyBucketReader`. Without the second role the generation-matched write fails in a +way that looks like a permissions flake. + +## One lease per run, not per job + +`deploy-relay-production-capacity` calls its reusable job six times and +`deploy-relay-production-same-cap` four times. Each call is a separate job on a separate runner, so +a naive per-job acquire/release would leave the object free between waves — for runs that have taken +up to 85 minutes. + +The lease is therefore keyed to the **run**, not the job. `holder-key` defaults to +`${{ github.repository }}/${{ github.run_id }}`, and a job that finds its own holder key on a live +lease **re-enters** it: the record is refreshed, not rejected. Every job in the chain acquires; only +the last one releases. + +```yaml +jobs: + gate: + steps: + - uses: ./.github/actions/cloud-sql-rollout-lease + with: { bucket: ..., object: ..., release: 'false' } # intermediate + + wave-1: # ... release: 'false' on every wave job + + release_lease: + needs: [gate, wave-1, wave-2, wave-3, wave-4] + if: always() + steps: + - uses: google-github-actions/auth@v2 + - uses: google-github-actions/setup-gcloud@v2 + - uses: ./.github/actions/cloud-sql-rollout-lease + with: { bucket: ..., object: ..., release: 'true' } # final +``` + +`release: 'false'` still acquires and still runs its `post` step; `post` only skips the delete. A +single-job workflow leaves `release` at its `true` default and needs no extra job. So does a +workflow whose several jobs can never hold the lease at once: `prove-relay-staging-capacity`'s two +lease-holding jobs are guarded by complementary `inputs.mode` conditions, and the contract test +checks that exclusivity rather than assuming it. + +If the final job never runs (runner killed, run cancelled hard), the lease expires on its TTL. + +## Timing + +- **TTL 35 minutes**, matching `apps/relay-fence-broker/src/mutation-lease.ts`. +- **Renewal every 5 minutes.** `main` spawns a detached background Node process that rewrites + `expires_at` on the same generation-matched path; `post` kills it by pid read back from + `$GITHUB_STATE`. The renewer stops on its own the moment the object stops being ours, and has a + six-hour backstop in case `post` never runs. Its log is written to + `$RUNNER_TEMP/cloud-sql-rollout-lease-renewer.log` and echoed by `post`. +- Renewal is mandatory, not optional: capacity runs have taken 85 minutes, well past any sane TTL. + +## Failure behaviour + +| Situation | Behaviour | +| ---------------------------------- | -------------------------------------------------------------------- | +| Object absent | Acquire with `ifGenerationMatch: 0`. | +| Live lease, our own holder key | Re-enter. Refresh `expires_at`, keep `acquired_at`. Never fails. | +| Live lease, another holder | **Fail the job immediately**, printing the holder's repository, workflow and run URL. Never queues, never steals. | +| Expired lease | Take over with the observed generation and emit `::warning::` naming the stale holder. | +| `412` on write | Someone raced us. Fail as a conflict. | +| Bucket unreachable, `403`, `5xx` | **Fail closed.** | +| Record present but unparseable | **Fail closed.** A record we cannot read is never treated as free; an operator must inspect and delete it. | +| Release finds a foreign holder | Warn and leave it alone. Our lease had already expired. | +| Release fails | Warn only. `post` never fails a job over a release; the TTL bounds the damage. | + +## Why `monitor-relay-production` must not use this + +`monitor-relay-production` is in the `production-cloud-sql-rollout` concurrency group but is +**read-only**: its identity holds only monitoring, logging, Cloud SQL and compute *viewer* roles, +and it runs `gcloud sql instances describe`, never a mutation. It consumes no connection budget. +Putting it on the durable lease would let a monitoring run block a real rollout, and a rollout block +monitoring exactly when an operator most needs it. Keep its same-repo concurrency group; keep it off +the lease. The lock census contract test records it in the not-a-candidate map with this reason. + +## Token acquisition + +`gcloud auth print-access-token`, not a hand-rolled exchange of the `external_account` credentials +file. Every consuming workflow already runs `setup-gcloud`, gcloud already handles every ADC flavour +including the service-account impersonation leg, and this action must stay zero-dependency because +it is duplicated by hand into the public repo. The GCE metadata server that +`apps/relay-fence-broker/src/google-metadata.ts` uses does **not** exist on GitHub or Blacksmith +runners; only the compare-and-swap algorithm is shared with the fence broker. + +## Duplication + +This directory is copied verbatim into `stablyai/orca`. It has no `package.json`, no +`node_modules`, and imports nothing outside itself — `action-contract.test.mjs` enforces all three. +Cross-repo consumption via `uses: stablyai/orca/.github/actions/...@` was rejected: it would +put public-repo code inside private app deploys that hold a production credential, and neither +repository protects `main` today. + +## Tests + +``` +node --test .github/actions/cloud-sql-rollout-lease/ +``` + +`storage-lease.test.mjs` drives the real compare-and-swap path against an in-memory Cloud Storage +fake that enforces generations. No network. diff --git a/.github/actions/cloud-sql-rollout-lease/action-contract.test.mjs b/.github/actions/cloud-sql-rollout-lease/action-contract.test.mjs new file mode 100644 index 00000000000..631daef7d3e --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/action-contract.test.mjs @@ -0,0 +1,52 @@ +import assert from 'node:assert/strict' +import { readFileSync, readdirSync } from 'node:fs' +import { test } from 'node:test' + +const here = new URL('./', import.meta.url) +const action = readFileSync(new URL('action.yml', here), 'utf8') +const modules = readdirSync(here).filter((name) => name.endsWith('.mjs')) +const shipped = modules.filter((name) => !name.endsWith('.test.mjs')) + +test('is a node24 JavaScript action with an always-run post step', () => { + // A composite action has no `post:`, so the lease could never be released on cancel or failure. + assert.match(action, /^ {2}using: node24$/m) + assert.doesNotMatch(action, /using: composite/) + assert.match(action, /^ {2}main: main\.mjs$/m) + assert.match(action, /^ {2}post: post\.mjs$/m) + assert.match(action, /^ {2}post-if: always\(\)$/m) +}) + +test('declares the inputs the wave-chain callers depend on', () => { + for (const input of ['bucket:', 'object:', 'holder-key:', 'release:']) { + assert.match(action, new RegExp(`^ {2}${input}$`, 'm'), input) + } + assert.match(action, /default: \$\{\{ github\.repository \}\}\/\$\{\{ github\.run_id \}\}/) + assert.match(action, /default: 'true'/) +}) + +test('stays self-contained so it can be duplicated into the public repo', () => { + assert.deepEqual( + readdirSync(here).filter((name) => name === 'package.json' || name === 'node_modules'), + [], + 'the action must run with zero installed dependencies' + ) + for (const name of modules) { + const source = readFileSync(new URL(name, here), 'utf8') + for (const match of source.matchAll(/^import\b[\s\S]*?from '([^']+)'/gm)) { + const specifier = match[1] + const local = specifier.startsWith('.') + assert.ok( + specifier.startsWith('node:') || (local && !specifier.includes('..')), + `${name} imports ${specifier}; only node: builtins and same-directory modules are allowed` + ) + } + } +}) + +test('never reaches for the GCE metadata server', () => { + // Runners have no metadata.google.internal; the fence broker's token path must not be copied. + for (const name of shipped) { + const source = readFileSync(new URL(name, here), 'utf8') + assert.doesNotMatch(source, /metadata\.google\.internal/, name) + } +}) diff --git a/.github/actions/cloud-sql-rollout-lease/action.yml b/.github/actions/cloud-sql-rollout-lease/action.yml new file mode 100644 index 00000000000..afd4b8efee0 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/action.yml @@ -0,0 +1,41 @@ +name: Cloud SQL rollout lease +description: >- + Serialize Cloud SQL connection-budget rollouts across repositories with a compare-and-swap lease + on a Cloud Storage object. Fails immediately when another run holds the lease; never queues, + never steals. + +inputs: + bucket: + description: Terraform state bucket that holds the lease object. + required: true + object: + description: Lease object name, e.g. terraform/state/cloud-sql-rollout/production.lock + required: true + holder-key: + description: >- + Identity that owns the lease. Every job in one run must pass the same value; a job that finds + its own holder key on a live lease re-enters it instead of failing. + required: false + default: ${{ github.repository }}/${{ github.run_id }} + release: + description: >- + Release the lease in the post step. Set to "false" on every job of a multi-job wave except the + final always() job, which sets "true". + required: false + default: 'true' + +outputs: + holder-key: + description: The holder key written to the lease object. + generation: + description: Cloud Storage generation of the lease object after acquisition. + expires-at: + description: ISO-8601 instant at which the lease expires without renewal. + reentrant: + description: '"true" when this job re-entered a lease its own run already held.' + +runs: + using: node24 + main: main.mjs + post: post.mjs + post-if: always() diff --git a/.github/actions/cloud-sql-rollout-lease/gcloud-access-token.mjs b/.github/actions/cloud-sql-rollout-lease/gcloud-access-token.mjs new file mode 100644 index 00000000000..ffc6aac4f95 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/gcloud-access-token.mjs @@ -0,0 +1,42 @@ +import { execFileSync } from 'node:child_process' + +// DESIGN CHOICE: shell out to `gcloud auth print-access-token` instead of exchanging the +// external_account credentials file that google-github-actions/auth writes. +// +// Every workflow on the Cloud SQL rollout lease already runs google-github-actions/setup-gcloud +// right after auth (verified across all 11 mutating members), so gcloud is on PATH and already +// bound to the federated identity. Doing the exchange ourselves would mean reimplementing the STS +// token swap plus the service-account impersonation leg, in an action that must stay +// zero-dependency and is duplicated by hand into a second repo. gcloud already handles every ADC +// flavour and refreshes on its own. The metadata server is not an option: it does not exist on +// GitHub or Blacksmith runners. +// +// Consequence, documented in the README: the lease step MUST come after setup-gcloud. + +const TOKEN_REUSE_MS = 40 * 60 * 1_000 // GCP access tokens live ~60 min; re-mint well before that. + +export function createAccessTokenSource({ run = runGcloud, now = Date.now } = {}) { + let cached = null + return () => { + if (cached && cached.mintedAt + TOKEN_REUSE_MS > now()) return cached.token + const token = run() + if (!token) throw new Error('gcloud auth print-access-token returned an empty token') + cached = { token, mintedAt: now() } + return token + } +} + +function runGcloud() { + const binary = process.platform === 'win32' ? 'gcloud.cmd' : 'gcloud' + try { + return execFileSync(binary, ['auth', 'print-access-token'], { + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'pipe'], + timeout: 60_000 + }).trim() + } catch (error) { + // Never surface stdout; it is the token on success and noise on failure. + const detail = String(error?.stderr ?? '').trim() || error?.message || 'unknown failure' + throw new Error(`could not mint a GCP access token via gcloud: ${detail}`) + } +} diff --git a/.github/actions/cloud-sql-rollout-lease/holder-identity.mjs b/.github/actions/cloud-sql-rollout-lease/holder-identity.mjs new file mode 100644 index 00000000000..563f678066d --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/holder-identity.mjs @@ -0,0 +1,18 @@ +// Who we claim to be on the lease object. Shared by main and the detached renewer so both agree +// on the holder key without re-deriving it from a different set of environment variables. + +export function holderIdentity(explicitHolderKey) { + const repository = process.env.GITHUB_REPOSITORY ?? 'unknown' + const runId = process.env.GITHUB_RUN_ID ?? 'unknown' + const server = process.env.GITHUB_SERVER_URL ?? 'https://github.com' + const holderKey = explicitHolderKey || `${repository}/${runId}` + if (/[\r\n]/.test(holderKey)) throw new Error('holder-key must be single-line') + return { + holderKey, + repository, + workflow: process.env.GITHUB_WORKFLOW ?? 'unknown', + runId, + runUrl: `${server}/${repository}/actions/runs/${runId}`, + runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? 'unknown' + } +} diff --git a/.github/actions/cloud-sql-rollout-lease/main.mjs b/.github/actions/cloud-sql-rollout-lease/main.mjs new file mode 100644 index 00000000000..0a429763a73 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/main.mjs @@ -0,0 +1,81 @@ +import { spawn } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import { createAccessTokenSource } from './gcloud-access-token.mjs' +import { holderIdentity } from './holder-identity.mjs' +import { fail, input, notice, renewerLogPath, saveState, setOutput, warn } from './runner-state.mjs' +import { CloudSqlRolloutLease, LeaseConflict, describeHolder } from './storage-lease.mjs' + +const bucket = input('bucket') +const objectName = input('object') +const release = input('release') !== 'false' + +if (!bucket || !objectName) { + fail('cloud-sql-rollout-lease requires both `bucket` and `object`') + process.exit(1) +} + +const holder = holderIdentity(input('holder-key')) +const lease = new CloudSqlRolloutLease({ + bucket, + objectName, + accessToken: createAccessTokenSource() +}) + +let claim +try { + claim = await lease.acquire(holder) +} catch (error) { + if (error instanceof LeaseConflict) { + fail( + `${error.message}. Cloud SQL rollouts are serialized across repositories; this run will not queue or steal the lease. Wait for the holder to finish, then re-run.` + ) + if (error.holder) { + console.log(`Lease holder repository: ${error.holder.repository}`) + console.log(`Lease holder workflow: ${error.holder.workflow}`) + console.log(`Lease holder run: ${error.holder.run_url}`) + } + } else { + // Bucket unreachable, permission denied, unreadable record: fail closed. + fail(`could not acquire ${lease.uri}: ${error.message}`) + } + process.exit(1) +} + +// Persist before anything else can throw, so `post` always releases what we hold. +saveState('acquired', 'true') +saveState('bucket', bucket) +saveState('object', objectName) +saveState('holder_key', holder.holderKey) +saveState('release', release ? 'true' : 'false') + +setOutput('holder-key', holder.holderKey) +setOutput('generation', claim.generation) +setOutput('expires-at', new Date(claim.record.expires_at).toISOString()) +setOutput('reentrant', claim.state === 'reentrant' ? 'true' : 'false') + +if (claim.state === 'reentrant') { + notice( + `Re-entered the Cloud SQL rollout lease on ${lease.uri} already held by this run; refreshed to ${new Date(claim.record.expires_at).toISOString()}.` + ) +} else if (claim.state === 'takeover') { + notice(`Took over ${lease.uri} from ${describeHolder(claim.previous)}.`) +} else { + notice( + `Acquired ${lease.uri} until ${new Date(claim.record.expires_at).toISOString()} (holder ${holder.holderKey}).` + ) +} + +try { + const renewer = spawn( + process.execPath, + [fileURLToPath(new URL('./renew.mjs', import.meta.url)), bucket, objectName, holder.holderKey], + { detached: true, stdio: 'ignore', env: process.env } + ) + renewer.unref() + saveState('renewer_pid', String(renewer.pid)) + const log = renewerLogPath() + notice(`Lease renewer running as pid ${renewer.pid}${log ? `, logging to ${log}` : ''}.`) +} catch (error) { + // A missing renewer is survivable for short jobs; the TTL still covers 35 minutes. + warn(`could not start the lease renewer: ${error.message}. The lease will expire on its TTL.`) +} diff --git a/.github/actions/cloud-sql-rollout-lease/post.mjs b/.github/actions/cloud-sql-rollout-lease/post.mjs new file mode 100644 index 00000000000..774db1323a7 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/post.mjs @@ -0,0 +1,73 @@ +import { readFileSync } from 'node:fs' +import { createAccessTokenSource } from './gcloud-access-token.mjs' +import { notice, renewerLogPath, savedState, warn } from './runner-state.mjs' +import { CloudSqlRolloutLease, describeHolder } from './storage-lease.mjs' + +stopRenewer() +printRenewerLog() + +if (savedState('acquired') !== 'true') { + notice('No Cloud SQL rollout lease was acquired by this step; nothing to release.') + process.exit(0) +} + +const bucket = savedState('bucket') +const objectName = savedState('object') +const holderKey = savedState('holder_key') + +if (savedState('release') !== 'true') { + notice( + `Holding gs://${bucket}/${objectName} for the rest of run ${holderKey}; a later job with release=true must free it.` + ) + process.exit(0) +} + +const lease = new CloudSqlRolloutLease({ + bucket, + objectName, + accessToken: createAccessTokenSource() +}) + +try { + const result = await lease.release(holderKey) + if (result.released) { + notice(`Released ${lease.uri} at generation ${result.generation}.`) + } else if (result.reason === 'absent') { + notice(`${lease.uri} was already gone; nothing to release.`) + } else if (result.reason === 'foreign') { + warn( + `${lease.uri} is now held by ${describeHolder(result.holder)}; leaving it alone. Our lease had already expired.` + ) + } else { + warn(`${lease.uri} changed while releasing it; leaving it to expire on its TTL.`) + } +} catch (error) { + // Never fail a job in post over a release; the TTL bounds the damage to 35 minutes. + warn(`could not release ${lease.uri}: ${error.message}. It will expire on its TTL.`) +} + +function stopRenewer() { + const pid = Number(savedState('renewer_pid')) + if (!Number.isInteger(pid) || pid <= 0) return + try { + process.kill(pid, 'SIGTERM') + notice(`Stopped the lease renewer (pid ${pid}).`) + } catch (error) { + if (error?.code !== 'ESRCH') warn(`could not stop the lease renewer ${pid}: ${error.message}`) + } +} + +function printRenewerLog() { + const path = renewerLogPath() + if (!path) return + let text = '' + try { + text = readFileSync(path, 'utf8') + } catch { + return + } + if (!text.trim()) return + console.log('::group::Cloud SQL rollout lease renewer log') + console.log(text.trimEnd()) + console.log('::endgroup::') +} diff --git a/.github/actions/cloud-sql-rollout-lease/renew.mjs b/.github/actions/cloud-sql-rollout-lease/renew.mjs new file mode 100644 index 00000000000..24cd1f3a210 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/renew.mjs @@ -0,0 +1,67 @@ +import { appendFileSync } from 'node:fs' +import { createAccessTokenSource } from './gcloud-access-token.mjs' +import { renewerLogPath } from './runner-state.mjs' +import { CloudSqlRolloutLease, RENEW_INTERVAL_MS } from './storage-lease.mjs' + +// Detached renewer. `main` spawns it, `post` kills it. It rewrites expires_at on the same +// generation-matched path as acquisition, and stops the moment the object stops being ours. + +const MAX_LIFETIME_MS = 6 * 60 * 60 * 1_000 // Backstop if post never runs (runner killed). + +const [bucket, objectName, holderKey] = process.argv.slice(2) +const logPath = renewerLogPath() +const startedAt = Date.now() + +function log(message) { + if (!logPath) return + try { + appendFileSync(logPath, `${new Date().toISOString()} ${message}\n`) + } catch { + // A renewer that cannot log must still renew. + } +} + +if (!bucket || !objectName || !holderKey) { + log('renewer started without bucket/object/holder-key; exiting') + process.exit(1) +} + +const lease = new CloudSqlRolloutLease({ + bucket, + objectName, + accessToken: createAccessTokenSource(), + warn: (message) => log(`warning ${message}`) +}) + +let stopping = false +for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) { + process.on(signal, () => { + stopping = true + log(`received ${signal}; stopping`) + process.exit(0) + }) +} + +log(`renewer started for ${lease.uri} holder=${holderKey} interval=${RENEW_INTERVAL_MS}ms`) + +while (!stopping) { + await new Promise((resolve) => { + setTimeout(resolve, RENEW_INTERVAL_MS) + }) + if (stopping) break + if (Date.now() - startedAt > MAX_LIFETIME_MS) { + log('renewer hit its maximum lifetime; stopping so the lease can expire') + break + } + try { + const result = await lease.renew(holderKey) + if (!result.renewed) { + log(`lease is no longer ours (${result.reason}); stopping`) + break + } + log(`renewed until ${new Date(result.record.expires_at).toISOString()} at generation ${result.generation}`) + } catch (error) { + // Transient GCS or token failures are retried on the next tick; the TTL covers 7 misses. + log(`renewal attempt failed: ${error.message}`) + } +} diff --git a/.github/actions/cloud-sql-rollout-lease/runner-state.mjs b/.github/actions/cloud-sql-rollout-lease/runner-state.mjs new file mode 100644 index 00000000000..d7cadbed663 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/runner-state.mjs @@ -0,0 +1,51 @@ +import { appendFileSync } from 'node:fs' + +// Action-state and output plumbing via the runner's file protocol, so the action needs no +// @actions/core dependency. Values are single-line by construction; anything else is rejected. + +/** The detached renewer's stdio is ignored, so it appends here instead and `post` echoes it. */ +export function renewerLogPath() { + const dir = process.env.RUNNER_TEMP + return dir ? `${dir}/cloud-sql-rollout-lease-renewer.log` : null +} + +export function input(name) { + return (process.env[`INPUT_${name.replace(/ /g, '_').toUpperCase()}`] ?? '').trim() +} + +export function savedState(name) { + return (process.env[`STATE_${name}`] ?? '').trim() +} + +export function saveState(name, value) { + appendToEnvFile('GITHUB_STATE', name, value) +} + +export function setOutput(name, value) { + appendToEnvFile('GITHUB_OUTPUT', name, value) +} + +export function notice(message) { + console.log(`::notice::${oneLine(message)}`) +} + +export function warn(message) { + console.log(`::warning::${oneLine(message)}`) +} + +export function fail(message) { + console.log(`::error::${oneLine(message)}`) + process.exitCode = 1 +} + +function appendToEnvFile(variable, name, value) { + const text = String(value) + if (/[\r\n]/.test(text)) throw new Error(`${name} must be single-line`) + const path = process.env[variable] + if (!path) return // Running outside a runner (local smoke run); nothing to persist. + appendFileSync(path, `${name}=${text}\n`) +} + +function oneLine(message) { + return String(message).replace(/\r?\n/g, ' ') +} diff --git a/.github/actions/cloud-sql-rollout-lease/storage-lease.mjs b/.github/actions/cloud-sql-rollout-lease/storage-lease.mjs new file mode 100644 index 00000000000..ad6c8bd4904 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/storage-lease.mjs @@ -0,0 +1,218 @@ +// Compare-and-swap lease over a single Cloud Storage object. +// +// Ported from apps/relay-fence-broker/src/mutation-lease.ts rather than imported: this action is +// duplicated verbatim into stablyai/orca, so it must carry no repo-local imports. Only the +// algorithm is shared (read metadata -> write with ifGenerationMatch -> 412 is a conflict -> +// generation-matched delete -> an expired record is free). The broker's token path is NOT shared; +// it reads the GCE metadata server, which does not exist on Actions runners. + +export const LEASE_TTL_MS = 35 * 60 * 1_000 +export const RENEW_INTERVAL_MS = 5 * 60 * 1_000 + +const GENERATION = /^[1-9][0-9]{0,30}$/ + +export class LeaseConflict extends Error { + constructor(message, holder) { + super(message) + this.name = 'LeaseConflict' + this.holder = holder ?? null + } +} + +/** A live record we cannot parse is never treated as free; wedging beats double-rollout. */ +export class LeaseUnreadable extends Error { + constructor(message) { + super(message) + this.name = 'LeaseUnreadable' + } +} + +function parseRecord(raw) { + if (!raw || typeof raw !== 'object') return null + if (typeof raw.holder_key !== 'string' || raw.holder_key.length === 0) return null + if (!Number.isSafeInteger(raw.acquired_at) || !Number.isSafeInteger(raw.expires_at)) return null + return { + repository: typeof raw.repository === 'string' ? raw.repository : 'unknown', + workflow: typeof raw.workflow === 'string' ? raw.workflow : 'unknown', + run_id: typeof raw.run_id === 'string' ? raw.run_id : 'unknown', + run_url: typeof raw.run_url === 'string' ? raw.run_url : 'unknown', + run_attempt: typeof raw.run_attempt === 'string' ? raw.run_attempt : 'unknown', + acquired_at: raw.acquired_at, + expires_at: raw.expires_at, + holder_key: raw.holder_key + } +} + +function describe(record) { + return `${record.repository} / ${record.workflow} (run ${record.run_id}, attempt ${record.run_attempt}) ${record.run_url}` +} + +export class CloudSqlRolloutLease { + #bucket + #objectName + #accessToken + #fetcher + #now + #warn + + constructor({ + bucket, + objectName, + accessToken, + fetcher = fetch, + now = Date.now, + warn = (message) => console.log(`::warning::${message}`) + }) { + this.#bucket = bucket + this.#objectName = objectName + this.#accessToken = accessToken + this.#fetcher = fetcher + this.#now = now + this.#warn = warn + } + + get uri() { + return `gs://${this.#bucket}/${this.#objectName}` + } + + async acquire(holder) { + const existing = await this.read() + const now = this.#now() + if (!existing) return this.#claim(holder, '0', now, now, 'created') + if (existing.record.holder_key === holder.holderKey) { + // Same run, another job in the wave chain. Refresh, never fail. + return this.#claim( + holder, + existing.generation, + existing.record.acquired_at, + now, + 'reentrant', + existing.record + ) + } + if (existing.record.expires_at > now) { + throw new LeaseConflict( + `${this.uri} is held by ${describe(existing.record)} until ${new Date(existing.record.expires_at).toISOString()}`, + existing.record + ) + } + this.#warn( + `Taking over an expired Cloud SQL rollout lease on ${this.uri}. Stale holder: ${describe(existing.record)}, expired ${new Date(existing.record.expires_at).toISOString()}.` + ) + return this.#claim(holder, existing.generation, now, now, 'takeover', existing.record) + } + + async renew(holderKey) { + const existing = await this.read() + if (!existing) return { renewed: false, reason: 'absent' } + if (existing.record.holder_key !== holderKey) return { renewed: false, reason: 'foreign' } + const now = this.#now() + const record = { ...existing.record, expires_at: now + LEASE_TTL_MS } + const written = await this.#write(record, existing.generation) + return { renewed: true, generation: written.generation, record } + } + + async release(holderKey) { + const existing = await this.read() + if (!existing) return { released: false, reason: 'absent' } + if (existing.record.holder_key !== holderKey) { + return { released: false, reason: 'foreign', holder: existing.record } + } + const response = await this.#fetcher( + `${this.#metadataUrl()}?ifGenerationMatch=${encodeURIComponent(existing.generation)}`, + { method: 'DELETE', headers: { Authorization: `Bearer ${await this.#token()}` } } + ) + if (response.status === 412) return { released: false, reason: 'conflict' } + if (!response.ok && response.status !== 404) { + throw new Error(`lease release failed: ${response.status}`) + } + return { released: true, generation: existing.generation } + } + + async read() { + const token = await this.#token() + const metadataResponse = await this.#fetcher(this.#metadataUrl(), { + headers: { Authorization: `Bearer ${token}` } + }) + if (metadataResponse.status === 404) return null + if (!metadataResponse.ok) { + throw new Error(`lease inspection failed: ${metadataResponse.status}`) + } + const metadata = await metadataResponse.json() + if (!GENERATION.test(metadata?.generation ?? '')) { + throw new LeaseUnreadable(`${this.uri} has no valid generation`) + } + const bodyResponse = await this.#fetcher(`${this.#metadataUrl()}?alt=media`, { + headers: { Authorization: `Bearer ${token}` } + }) + if (bodyResponse.status === 404) return null + if (!bodyResponse.ok) { + throw new Error(`lease body read failed: ${bodyResponse.status}`) + } + let raw = null + try { + raw = await bodyResponse.json() + } catch { + raw = null + } + const record = parseRecord(raw) + if (!record) { + throw new LeaseUnreadable( + `${this.uri} holds an unreadable lease record; an operator must inspect and delete it before rollouts can resume` + ) + } + return { generation: metadata.generation, record } + } + + async #claim(holder, generation, acquiredAt, now, state, previous) { + const record = { + repository: holder.repository, + workflow: holder.workflow, + run_id: holder.runId, + run_url: holder.runUrl, + run_attempt: holder.runAttempt, + acquired_at: acquiredAt, + expires_at: now + LEASE_TTL_MS, + holder_key: holder.holderKey + } + const written = await this.#write(record, generation) + return { state, generation: written.generation, record, previous: previous ?? null } + } + + async #write(record, generation) { + const response = await this.#fetcher( + `${this.#uploadUrl()}&ifGenerationMatch=${encodeURIComponent(generation)}`, + { + method: 'POST', + headers: { + Authorization: `Bearer ${await this.#token()}`, + 'Content-Type': 'application/json' + }, + body: JSON.stringify(record) + } + ) + if (response.status === 412) { + throw new LeaseConflict(`${this.uri} changed concurrently while we were claiming it`) + } + if (!response.ok) throw new Error(`lease write failed: ${response.status}`) + const metadata = await response.json() + if (!GENERATION.test(metadata?.generation ?? '')) { + throw new LeaseUnreadable(`${this.uri} write returned no valid generation`) + } + return { generation: metadata.generation } + } + + async #token() { + return typeof this.#accessToken === 'function' ? await this.#accessToken() : this.#accessToken + } + + #metadataUrl() { + return `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(this.#bucket)}/o/${encodeURIComponent(this.#objectName)}` + } + + #uploadUrl() { + return `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(this.#bucket)}/o?uploadType=media&name=${encodeURIComponent(this.#objectName)}` + } +} + +export const describeHolder = describe diff --git a/.github/actions/cloud-sql-rollout-lease/storage-lease.test.mjs b/.github/actions/cloud-sql-rollout-lease/storage-lease.test.mjs new file mode 100644 index 00000000000..5d985f6f480 --- /dev/null +++ b/.github/actions/cloud-sql-rollout-lease/storage-lease.test.mjs @@ -0,0 +1,303 @@ +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { createAccessTokenSource } from './gcloud-access-token.mjs' +import { + CloudSqlRolloutLease, + LEASE_TTL_MS, + LeaseConflict, + LeaseUnreadable +} from './storage-lease.mjs' + +const BUCKET = 'onorca-cloud-terraform-state' +const OBJECT = 'terraform/state/cloud-sql-rollout/production.lock' +const NOW = 1_756_000_000_000 + +/** + * Enough of the Cloud Storage JSON API to exercise real compare-and-swap semantics: generations + * increment, ifGenerationMatch is enforced, and a mismatch is a 412. `faults` injects failures. + */ +function fakeStorage({ object = null, faults = [] } = {}) { + const state = { object, requests: [] } + const fetcher = async (rawUrl, init = {}) => { + const url = new URL(rawUrl) + const method = init.method ?? 'GET' + const record = { method, url, path: url.pathname, search: url.searchParams } + state.requests.push(record) + const fault = faults.find((candidate) => candidate.when(record)) + if (fault) return json(fault.status, fault.body ?? {}) + + if (url.pathname.startsWith('/upload/')) { + const want = url.searchParams.get('ifGenerationMatch') + const have = state.object ? state.object.generation : '0' + if (want !== have) return json(412, {}) + const generation = String(Number(have === '0' ? '1000' : have) + 1) + state.object = { generation, body: JSON.parse(init.body) } + return json(200, { generation }) + } + if (method === 'DELETE') { + if (!state.object) return json(404, {}) + if (url.searchParams.get('ifGenerationMatch') !== state.object.generation) return json(412, {}) + state.object = null + return json(204, {}) + } + if (!state.object) return json(404, {}) + if (url.searchParams.get('alt') === 'media') return json(200, state.object.body) + return json(200, { generation: state.object.generation }) + } + return { state, fetcher } +} + +function json(status, body) { + return { + status, + ok: status >= 200 && status < 300, + json: async () => body + } +} + +function storedRecord({ holderKey, expiresAt, repository = 'stablyai/orca', workflow = 'Deploy Relay Production' }) { + return { + repository, + workflow, + run_id: '9001', + run_url: 'https://github.com/stablyai/orca/actions/runs/9001', + run_attempt: '1', + acquired_at: NOW - 60_000, + expires_at: expiresAt, + holder_key: holderKey + } +} + +function leaseFor(storage, { warn = () => {} } = {}) { + return new CloudSqlRolloutLease({ + bucket: BUCKET, + objectName: OBJECT, + accessToken: 'test-token', + fetcher: storage.fetcher, + now: () => NOW, + warn + }) +} + +const HOLDER = { + holderKey: 'stablyai/orca-cloud/42', + repository: 'stablyai/orca-cloud', + workflow: 'Deploy Relay Production Same-Cap', + runId: '42', + runUrl: 'https://github.com/stablyai/orca-cloud/actions/runs/42', + runAttempt: '1' +} + +test('acquires a lease on an empty object with ifGenerationMatch=0', async () => { + const storage = fakeStorage() + const claim = await leaseFor(storage).acquire(HOLDER) + + assert.equal(claim.state, 'created') + const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/')) + assert.equal(upload.search.get('ifGenerationMatch'), '0') + assert.equal(upload.search.get('name'), OBJECT) + assert.deepEqual(storage.state.object.body, { + repository: 'stablyai/orca-cloud', + workflow: 'Deploy Relay Production Same-Cap', + run_id: '42', + run_url: 'https://github.com/stablyai/orca-cloud/actions/runs/42', + run_attempt: '1', + acquired_at: NOW, + expires_at: NOW + LEASE_TTL_MS, + holder_key: 'stablyai/orca-cloud/42' + }) +}) + +test('refuses a live lease held by another run and never writes', async () => { + const storage = fakeStorage({ + object: { + generation: '1500', + body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 }) + } + }) + + const error = await leaseFor(storage) + .acquire(HOLDER) + .catch((thrown) => thrown) + + assert.ok(error instanceof LeaseConflict) + assert.equal(error.holder.repository, 'stablyai/orca') + assert.equal(error.holder.run_url, 'https://github.com/stablyai/orca/actions/runs/9001') + assert.equal( + storage.state.requests.filter((request) => request.method !== 'GET').length, + 0, + 'a foreign live lease must not be written' + ) + assert.equal(storage.state.object.generation, '1500') +}) + +test('re-enters a live lease this run already holds and extends it', async () => { + const storage = fakeStorage({ + object: { + generation: '1500', + body: storedRecord({ holderKey: HOLDER.holderKey, expiresAt: NOW + 60_000 }) + } + }) + const warnings = [] + const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(HOLDER) + + assert.equal(claim.state, 'reentrant') + assert.deepEqual(warnings, [], 're-entering our own lease is not a takeover') + assert.equal(claim.record.acquired_at, NOW - 60_000, 'original acquisition time is preserved') + assert.equal(claim.record.expires_at, NOW + LEASE_TTL_MS) + const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/')) + assert.equal(upload.search.get('ifGenerationMatch'), '1500') + assert.equal(storage.state.object.generation, '1501') +}) + +test('takes over an expired lease and warns naming the stale holder', async () => { + const storage = fakeStorage({ + object: { + generation: '1500', + body: storedRecord({ + holderKey: 'stablyai/orca/9001', + expiresAt: NOW - 1, + repository: 'stablyai/orca', + workflow: 'Deploy Relay Production Capacity' + }) + } + }) + const warnings = [] + const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(HOLDER) + + assert.equal(claim.state, 'takeover') + assert.equal(warnings.length, 1) + assert.match(warnings[0], /stablyai\/orca/) + assert.match(warnings[0], /Deploy Relay Production Capacity/) + assert.match(warnings[0], /actions\/runs\/9001/) + const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/')) + assert.equal(upload.search.get('ifGenerationMatch'), '1500') + assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey) +}) + +test('releases with a generation match and leaves the object gone', async () => { + const storage = fakeStorage() + const lease = leaseFor(storage) + const claim = await lease.acquire(HOLDER) + + const released = await lease.release(HOLDER.holderKey) + + assert.deepEqual(released, { released: true, generation: claim.generation }) + const remove = storage.state.requests.find((request) => request.method === 'DELETE') + assert.equal(remove.search.get('ifGenerationMatch'), claim.generation) + assert.equal(storage.state.object, null) +}) + +test('refuses to release a lease another run now holds', async () => { + const storage = fakeStorage({ + object: { + generation: '1500', + body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 }) + } + }) + + const released = await leaseFor(storage).release(HOLDER.holderKey) + + assert.equal(released.released, false) + assert.equal(released.reason, 'foreign') + assert.equal(storage.state.object.generation, '1500') +}) + +test('fails closed when the bucket answers 5xx', async () => { + const storage = fakeStorage({ + faults: [{ when: (request) => request.method === 'GET', status: 503 }] + }) + + const error = await leaseFor(storage) + .acquire(HOLDER) + .catch((thrown) => thrown) + + assert.match(error.message, /lease inspection failed: 503/) + assert.equal(storage.state.requests.filter((request) => request.method !== 'GET').length, 0) +}) + +test('fails closed when permission is denied', async () => { + const storage = fakeStorage({ + faults: [{ when: (request) => request.method === 'GET', status: 403 }] + }) + + const error = await leaseFor(storage) + .acquire(HOLDER) + .catch((thrown) => thrown) + + assert.match(error.message, /lease inspection failed: 403/) +}) + +test('treats an unreadable record as held, not free', async () => { + const storage = fakeStorage({ + object: { generation: '1500', body: { holder_key: 'stablyai/orca/9001' } } + }) + + const error = await leaseFor(storage) + .acquire(HOLDER) + .catch((thrown) => thrown) + + assert.ok(error instanceof LeaseUnreadable) + assert.equal(storage.state.object.generation, '1500') +}) + +test('reports a 412 during acquisition as a conflict', async () => { + const storage = fakeStorage({ + faults: [{ when: (request) => request.path.startsWith('/upload/'), status: 412 }] + }) + + const error = await leaseFor(storage) + .acquire(HOLDER) + .catch((thrown) => thrown) + + assert.ok(error instanceof LeaseConflict) + assert.match(error.message, /changed concurrently/) +}) + +test('renewal rewrites only expires_at on the observed generation', async () => { + const storage = fakeStorage() + const lease = leaseFor(storage) + await lease.acquire(HOLDER) + storage.state.object.body.expires_at = NOW - 1 + + const renewed = await lease.renew(HOLDER.holderKey) + + assert.equal(renewed.renewed, true) + assert.equal(storage.state.object.body.expires_at, NOW + LEASE_TTL_MS) + assert.equal(storage.state.object.body.acquired_at, NOW) + assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey) +}) + +test('renewal stops once the object belongs to someone else', async () => { + const storage = fakeStorage({ + object: { + generation: '1500', + body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 }) + } + }) + + assert.deepEqual(await leaseFor(storage).renew(HOLDER.holderKey), { + renewed: false, + reason: 'foreign' + }) +}) + +test('the access token source re-mints only after the reuse window', () => { + let clock = 0 + let mints = 0 + const source = createAccessTokenSource({ + run: () => `token-${++mints}`, + now: () => clock + }) + + assert.equal(source(), 'token-1') + clock = 39 * 60 * 1_000 + assert.equal(source(), 'token-1') + clock = 41 * 60 * 1_000 + assert.equal(source(), 'token-2') +}) + +test('the access token source rejects an empty gcloud response', () => { + const source = createAccessTokenSource({ run: () => '' }) + assert.throws(() => source(), /empty token/) +}) diff --git a/.github/workflows/cloud-bootstrap-relay-staging-capacity.yml b/.github/workflows/cloud-bootstrap-relay-staging-capacity.yml new file mode 100644 index 00000000000..29e68510b21 --- /dev/null +++ b/.github/workflows/cloud-bootstrap-relay-staging-capacity.yml @@ -0,0 +1,676 @@ +name: Bootstrap Relay Staging Capacity + +on: + workflow_dispatch: + inputs: + confirmation: + description: Enter BOOTSTRAP_STAGING_CAPACITY + required: true + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + bootstrap: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + GCP_REGION: us-central1 + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + LEGACY_C3_IMAGE_DIGEST: sha256:2d0f6e6db2b0eb9d6aba188698de8330f8c30b4e76badfcf0fac3f3eb9508a87 + steps: + - uses: actions/checkout@v4 + + - id: deploy-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - id: capacity-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: access_token + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Require explicit bootstrap confirmation + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: test "${CONFIRMATION}" = "BOOTSTRAP_STAGING_CAPACITY" + + - name: Read and verify reviewed 600/60 topology + shell: bash + run: | + node dev/scripts/infra.mjs init --env staging + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \ + > "${RUNNER_TEMP}/relay-gce-state.json" + DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars \ + <<< 'local.relay_director_cells_json' | jq -r '.')" + DESIRED_IMAGES_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars \ + <<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.image })' \ + | jq -r '.')" + DESIRED_ZONES_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars \ + <<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.zone })' \ + | jq -r '.')" + DESIRED_TARGET_SIZES_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars \ + <<< 'jsonencode(local.relay_gce_cell_target_sizes)' | jq -r '.')" + jq -e \ + --argjson images "${DESIRED_IMAGES_JSON}" \ + --argjson target_sizes "${DESIRED_TARGET_SIZES_JSON}" \ + '([.[] | select(.id == "staging-gce-c2")] | length == 1) and + ([.[] | select(.id == "staging-gce-c3")] | length == 1) and + (any(.[]; .id == "staging-gce-c2" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and + (any(.[]; .id == "staging-gce-c3" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and + ($images["staging-gce-c2"] == $images["staging-gce-c3"]) and + ($target_sizes["staging-gce-c2"] == 1) and + ($target_sizes["staging-gce-c3"] == 1)' \ + <<< "${DESIRED_CELLS_JSON}" >/dev/null + jq \ + --argjson desired "${DESIRED_CELLS_JSON}" \ + --argjson images "${DESIRED_IMAGES_JSON}" \ + --argjson zones "${DESIRED_ZONES_JSON}" \ + '($desired | map({key: .id, value: .}) | from_entries) as $cells | + to_entries | map(. as $entry | { + key: $entry.key, + value: ($entry.value + { + origin: $cells[$entry.key].url, + image: $images[$entry.key], + zone: $zones[$entry.key], + connection_hard_cap: $cells[$entry.key].connectionHardCap, + connection_unobserved_bound: $cells[$entry.key].connectionUnobservedBound + }) + }) | from_entries' \ + "${RUNNER_TEMP}/relay-gce-state.json" \ + > "${RUNNER_TEMP}/relay-gce-topology.json" + ACTIVE_REVISION="$(gcloud run services describe orca-cloud-relay-staging \ + --project "${GCP_PROJECT_ID}" \ + --region us-central1 \ + --format=json \ + | jq -r ' + [.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')" + test -n "${ACTIVE_REVISION}" + DESIRED_IMAGE="$(jq -r '.["staging-gce-c2"]' <<< "${DESIRED_IMAGES_JSON}")" + gcloud run revisions describe "${ACTIVE_REVISION}" \ + --project "${GCP_PROJECT_ID}" \ + --region us-central1 \ + --format=json \ + | jq -e \ + --arg image "${DESIRED_IMAGE}" \ + --arg capacity_service_account "${CAPACITY_SERVICE_ACCOUNT}" \ + '(.spec.containers[0].image == $image) and + any(.spec.containers[0].env[]?; + .name == "ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT" and + .value == $capacity_service_account)' >/dev/null + CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format=json \ + | jq -cer ' + [.spec.containers[0].env[]? | + select(.name == "ORCA_RELAY_CELLS_JSON") | .value] | + if length == 1 then .[0] | fromjson else error("missing director topology") end')" + jq -e --argjson desired "${DESIRED_CELLS_JSON}" ' + def without_bootstrap_capacity: + map(if .id == "staging-gce-c2" or .id == "staging-gce-c3" + then del(.connectionHardCap, .connectionUnobservedBound) + else . end); + without_bootstrap_capacity == ($desired | without_bootstrap_capacity) + ' <<< "${CURRENT_CELLS_JSON}" >/dev/null + + - name: Bootstrap C2 then C3 + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + + topology="${RUNNER_TEMP}/relay-gce-topology.json" + + fixed_one_instance_name() { + local cell_id="$1" + local mig_name zone + mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")" + zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")" + gcloud compute instance-groups managed list-instances \ + "${mig_name}" \ + --project "${GCP_PROJECT_ID}" \ + --zone "${zone}" \ + --format=json \ + | jq -er ' + if length == 1 and .[0].instanceStatus == "RUNNING" and + .[0].currentAction == "NONE" + then .[0].instance | split("/") | last + else error("legacy cell does not have one stable running instance") end' + } + + fixed_one_instance_id() { + local cell_id="$1" + local instance_name="$2" + local zone + zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")" + gcloud compute instances describe "${instance_name}" \ + --project "${GCP_PROJECT_ID}" \ + --zone "${zone}" \ + --format='value(id)' + } + + write_legacy_metrics() { + local cell_id="$1" + local instance_id="$2" + local after="$3" + local output="$4" + gcloud logging read \ + "resource.type=\"gce_instance\" AND + resource.labels.instance_id=\"${instance_id}\" AND + jsonPayload.event=\"orca_relay_runtime_metrics\" AND + jsonPayload.cellId=\"${cell_id}\" AND + timestamp>=\"${after}\"" \ + --project "${GCP_PROJECT_ID}" \ + --limit 10 \ + --order desc \ + --format json \ + | jq '[.[] | { + timestamp, + cellId: .jsonPayload.cellId, + metricVersion: .jsonPayload.metricVersion, + totalConnections: .jsonPayload.totalConnections, + preAuthConnections: .jsonPayload.preAuthConnections, + controls: .jsonPayload.controls, + splices: .jsonPayload.splices, + pendingSplices: .jsonPayload.pendingSplices, + queuedBytes: .jsonPayload.queuedBytes + }]' > "${output}" + } + + verify_legacy_cell() { + local cell_id="$1" + local admission="$2" + local after="$3" + local expected_instance_id="$4" + local runtime_started_after="${5:-}" + local previous_incarnation_digest="${6:-}" + local hard_cap="${7:-}" + local unobserved_bound="${8:-}" + local capacity_state="${9:-}" + local current_instance current_instance_id metrics origin result + local runtime_start_args=() incarnation_args=() capacity_args=() + current_instance="$(fixed_one_instance_name "${cell_id}")" + current_instance_id="$(fixed_one_instance_id "${cell_id}" "${current_instance}")" + test "${current_instance_id}" = "${expected_instance_id}" + metrics="${RUNNER_TEMP}/${cell_id}-legacy-runtime-metrics.json" + origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")" + if test -n "${runtime_started_after}"; then + runtime_start_args=(--runtime-started-after "${runtime_started_after}") + fi + if test -n "${previous_incarnation_digest}"; then + incarnation_args=(--previous-incarnation-digest "${previous_incarnation_digest}") + fi + if test -n "${hard_cap}"; then + capacity_args=(--hard-cap "${hard_cap}" --unobserved-bound "${unobserved_bound}") + fi + if test -n "${capacity_state}"; then + capacity_args+=(--capacity-state "${capacity_state}") + fi + for _attempt in $(seq 1 18); do + write_legacy_metrics \ + "${cell_id}" "${current_instance_id}" "${after}" "${metrics}" + if result="$(node dev/scripts/verify-relay-legacy-bootstrap.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${origin}" \ + --cell-id "${cell_id}" \ + --admission "${admission}" \ + --expected-image-digest "${LEGACY_C3_IMAGE_DIGEST}" \ + --metrics-after "${after}" \ + --metrics-file "${metrics}" \ + "${runtime_start_args[@]}" \ + "${incarnation_args[@]}" \ + "${capacity_args[@]}")"; then + echo "${result}" + return 0 + fi + sleep 10 + done + return 1 + } + + post_admin() { + local origin="$1" + local path="$2" + local body="$3" + curl --fail --silent --show-error \ + --request POST \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "${body}" \ + "${origin}${path}" + } + + runtime_kind() { + local cell_id="$1" + local origin desired_digest digest + origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")" + desired_digest="$(jq -r --arg cell "${cell_id}" \ + '.[$cell].image | split("@") | last' "${topology}")" + digest="$(post_admin "${origin}" /v1/admin/runtime-status '{"v":1}' \ + | jq -er '.imageDigest')" + if test "${digest}" = "${LEGACY_C3_IMAGE_DIGEST}"; then + echo legacy + elif test "${digest}" = "${desired_digest}"; then + echo modern + else + echo 'bootstrap cell image is neither legacy nor reviewed' >&2 + return 1 + fi + } + + cell_admission() { + local cell_id="$1" + post_admin "${DIRECTOR_ORIGIN}" /v1/admin/cell-status \ + "$(jq -cn --arg cell "${cell_id}" '{v:1, cellId:$cell}')" \ + | jq -er '.status.admissionState | + if . == "general" or . == "migration-only" then . + else error("bootstrap admission is not recoverable") end' + } + + verify_modern_cell() { + local cell_id="$1" + local admission="$2" + local origin + origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")" + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${origin}" \ + --cell-id "${cell_id}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission "${admission}" \ + --draining forbidden \ + --activity allowed + } + + ensure_modern_general() { + local cell_id="$1" + local admission="$2" + verify_modern_cell "${cell_id}" "${admission}" + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${cell_id}" \ + --mode restore \ + --general-cell-ids "${cell_id}" + verify_modern_cell "${cell_id}" general + } + + prepare_legacy_c3_fallback() { + legacy_c3_metrics_boundary="$(node -e \ + 'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')" + legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)" + legacy_c3_instance_id="$(fixed_one_instance_id \ + staging-gce-c3 "${legacy_c3_instance}")" + verify_legacy_cell \ + staging-gce-c3 general \ + "${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" + node dev/scripts/probe-relay-legacy-admission.mjs \ + --cell-origin https://c3.relay-staging.onorca.dev + legacy_c3_restart_started_after= + legacy_c3_old_incarnation= + } + + normalize_legacy_c3() { + legacy_pre_boundary="$(node -e \ + 'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')" + legacy_c2_instance="$(fixed_one_instance_name staging-gce-c2)" + legacy_c2_instance_id="$(fixed_one_instance_id \ + staging-gce-c2 "${legacy_c2_instance}")" + verify_legacy_cell \ + staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}" + node dev/scripts/probe-relay-legacy-admission.mjs \ + --cell-origin https://c2.relay-staging.onorca.dev + + legacy_c3_isolated=false + restore_legacy_c3_fallback() { + if test "${legacy_c3_isolated}" = true; then + verify_legacy_cell \ + staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}" + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id staging-gce-c3 \ + --mode restore-fallback \ + --general-cell-ids staging-gce-c2 + fi + } + + trap restore_legacy_c3_fallback EXIT + legacy_c3_isolated=true + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin https://c3.relay-staging.onorca.dev \ + --cell-id staging-gce-c3 \ + --mode isolate + legacy_c3_drain_boundary="$(node -e \ + 'process.stdout.write(new Date().toISOString())')" + legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)" + legacy_c3_instance_id="$(fixed_one_instance_id \ + staging-gce-c3 "${legacy_c3_instance}")" + legacy_c3_drained="$(verify_legacy_cell \ + staging-gce-c3 migration-only \ + "${legacy_c3_drain_boundary}" "${legacy_c3_instance_id}")" + echo "${legacy_c3_drained}" + legacy_c3_old_incarnation="$(jq -er '.incarnationDigest' \ + <<< "${legacy_c3_drained}")" + legacy_c3_restart_started_after="$(node -e \ + 'process.stdout.write(new Date().toISOString())')" + gcloud compute instance-groups managed recreate-instances \ + orca-cloud-staging-relay-gce-c3 \ + --instances "${legacy_c3_instance}" \ + --project "${GCP_PROJECT_ID}" \ + --zone us-central1-a \ + --quiet + gcloud compute instance-groups managed wait-until \ + orca-cloud-staging-relay-gce-c3 \ + --stable \ + --project "${GCP_PROJECT_ID}" \ + --zone us-central1-a \ + --timeout 900 + legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)" + legacy_c3_instance_id="$(fixed_one_instance_id \ + staging-gce-c3 "${legacy_c3_instance}")" + legacy_c3_metrics_boundary="$(node -e \ + 'process.stdout.write(new Date().toISOString())')" + verify_legacy_cell \ + staging-gce-c3 migration-only \ + "${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \ + "${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}" + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id staging-gce-c3 \ + --mode restore \ + --general-cell-ids staging-gce-c2,staging-gce-c3 + verify_legacy_cell \ + staging-gce-c3 general \ + "${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \ + "${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}" + legacy_c3_isolated=false + trap - EXIT + } + + roll_cell() ( + local cell_id="$1" + local fallback_cell_id="$2" + local target_kind="$3" + local fallback_kind="$4" + local active_revision cell_origin current_cells_json desired_bound desired_cap + local desired_cells_json director_result image mig_name plan + local fallback_origin plan_changes plan_result restored target_drain_boundary + local target_instance target_instance_id zone + cell_origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")" + fallback_origin="$(jq -r --arg cell "${fallback_cell_id}" \ + '.[$cell].origin' "${topology}")" + zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")" + mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")" + image="$(jq -r --arg cell "${cell_id}" '.[$cell].image' "${topology}")" + desired_cap="$(jq -r --arg cell "${cell_id}" \ + '.[$cell].connection_hard_cap' "${topology}")" + desired_bound="$(jq -r --arg cell "${cell_id}" \ + '.[$cell].connection_unobserved_bound' "${topology}")" + plan="${RUNNER_TEMP}/${cell_id}-capacity-bootstrap.tfplan" + restored=false + + restore_fallback() { + if test "${restored}" = false; then + verify_fallback + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${cell_id}" \ + --mode restore-fallback \ + --general-cell-ids "${fallback_cell_id}" + fi + } + + verify_fallback() { + if test "${fallback_kind}" = legacy; then + verify_legacy_cell \ + "${fallback_cell_id}" general \ + "${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \ + "${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}" + return + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${fallback_origin}" \ + --cell-id "${fallback_cell_id}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + } + + verify_fallback + trap restore_fallback EXIT + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${cell_origin}" \ + --cell-id "${cell_id}" \ + --mode isolate + target_drain_boundary="$(node -e \ + 'process.stdout.write(new Date().toISOString())')" + if test "${target_kind}" = legacy; then + target_instance="$(fixed_one_instance_name "${cell_id}")" + target_instance_id="$(fixed_one_instance_id "${cell_id}" "${target_instance}")" + verify_legacy_cell \ + "${cell_id}" migration-only \ + "${target_drain_boundary}" "${target_instance_id}" \ + '' '' "${desired_cap}" "${desired_bound}" absent-or-stale + else + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${cell_origin}" \ + --cell-id "${cell_id}" \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity quiescent + fi + + active_revision="$(gcloud run services describe orca-cloud-relay-staging \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format=json \ + | jq -r ' + [.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')" + test -n "${active_revision}" + current_cells_json="$(gcloud run revisions describe "${active_revision}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format=json \ + | jq -cer ' + [.spec.containers[0].env[]? | + select(.name == "ORCA_RELAY_CELLS_JSON") | .value] | + if length == 1 then .[0] | fromjson else error("missing director topology") end')" + desired_cells_json="$(jq -ce \ + --arg cell "${cell_id}" \ + --argjson cap "${desired_cap}" \ + --argjson bound "${desired_bound}" \ + 'map(if .id == $cell then . + { + connectionHardCap: $cap, + connectionUnobservedBound: $bound + } else . end)' <<< "${current_cells_json}")" + director_result="$(node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service orca-cloud-relay-staging \ + --image "${image}" \ + --role director \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \ + --capacity-cell-id "${cell_id}" \ + --director-cells-json "${desired_cells_json}" \ + --min-instances 0 \ + --prune-revisions true \ + --release-id "bootstrap-${cell_id}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}")" + echo "${director_result}" + if test "${target_kind}" = legacy; then + verify_legacy_cell \ + "${cell_id}" migration-only \ + "${target_drain_boundary}" "${target_instance_id}" \ + '' '' "${desired_cap}" "${desired_bound}" + else + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${cell_origin}" \ + --cell-id "${cell_id}" \ + --hard-cap "${desired_cap}" \ + --unobserved-bound "${desired_bound}" \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity quiescent + fi + + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars \ + "-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \ + "-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \ + -out="${plan}" + plan_result="$(terraform -chdir=infra/terraform show -json "${plan}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode bootstrap-cell \ + --cell-id "${cell_id}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --image "${image}" \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")" + echo "${plan_result}" + plan_changes="$(jq -r '.changes' <<< "${plan_result}")" + [[ "${plan_changes}" =~ ^(0|2)$ ]] + if test "${plan_changes}" = 2; then + terraform -chdir=infra/terraform apply -auto-approve "${plan}" + else + target_instance="$(fixed_one_instance_name "${cell_id}")" + gcloud compute instance-groups managed recreate-instances "${mig_name}" \ + --instances "${target_instance}" \ + --project "${GCP_PROJECT_ID}" \ + --zone "${zone}" \ + --quiet + fi + gcloud compute instance-groups managed wait-until "${mig_name}" \ + --stable \ + --project "${GCP_PROJECT_ID}" \ + --zone "${zone}" \ + --timeout 900 + + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${cell_origin}" \ + --cell-id "${cell_id}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission migration-only \ + --draining forbidden \ + --activity allowed + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${cell_id}" \ + --mode restore \ + --general-cell-ids staging-gce-c2,staging-gce-c3 + restored=true + trap - EXIT + ) + + c2_kind="$(runtime_kind staging-gce-c2)" + c3_kind="$(runtime_kind staging-gce-c3)" + c2_admission="$(cell_admission staging-gce-c2)" + c3_admission="$(cell_admission staging-gce-c3)" + bootstrap_phase="$(node dev/scripts/classify-relay-staging-bootstrap.mjs \ + --c2-kind "${c2_kind}" \ + --c2-admission "${c2_admission}" \ + --c3-kind "${c3_kind}" \ + --c3-admission "${c3_admission}")" + jq -cn --arg phase "${bootstrap_phase}" \ + '{event:"relay_staging_bootstrap_phase", phase:$phase}' + + case "${bootstrap_phase}" in + normalize-and-roll-both) + normalize_legacy_c3 + roll_cell staging-gce-c2 staging-gce-c3 legacy legacy + roll_cell staging-gce-c3 staging-gce-c2 legacy modern + ;; + resume-c2-then-c3) + prepare_legacy_c3_fallback + roll_cell staging-gce-c2 staging-gce-c3 legacy legacy + roll_cell staging-gce-c3 staging-gce-c2 legacy modern + ;; + roll-c2) + ensure_modern_general staging-gce-c3 "${c3_admission}" + roll_cell staging-gce-c2 staging-gce-c3 legacy modern + ;; + roll-c3) + ensure_modern_general staging-gce-c2 "${c2_admission}" + roll_cell staging-gce-c3 staging-gce-c2 legacy modern + ;; + complete) + ensure_modern_general staging-gce-c2 "${c2_admission}" + ensure_modern_general staging-gce-c3 "${c3_admission}" + ;; + *) + echo 'unsupported staging bootstrap phase' >&2 + exit 1 + ;; + esac + + - name: Verify both bootstrapped cells + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + for number in 2 3; do + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "https://c${number}.relay-staging.onorca.dev" \ + --cell-id "staging-gce-c${number}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + done diff --git a/.github/workflows/cloud-deploy-relay-asia-topology.yml b/.github/workflows/cloud-deploy-relay-asia-topology.yml new file mode 100644 index 00000000000..f15fc5ae0e2 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-asia-topology.yml @@ -0,0 +1,245 @@ +name: Deploy Relay Asia Topology + +on: + workflow_dispatch: + inputs: + environment: + description: Target Relay environment + required: true + type: choice + options: [staging, production] + mode: + description: Validate a saved plan or apply that exact plan + required: true + default: plan + type: choice + options: [plan, apply] + cell-ids: + description: Exact reviewed comma-separated Asia cell set + required: true + type: string + image: + description: Full environment Relay image pinned by sha256 digest + required: true + type: string + confirmation: + description: Enter APPLY_RELAY_ASIA_TOPOLOGY for apply mode + required: false + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }} + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + topology: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 30 + environment: ${{ inputs.environment }} + env: + DEPLOY_MODE: ${{ inputs.mode }} + TARGET_ENVIRONMENT: ${{ inputs.environment }} + TARGET_CELL_IDS: ${{ inputs.cell-ids }} + TARGET_IMAGE: ${{ inputs.image }} + TARGET_REGION: asia-east2 + GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }} + CLOUD_SQL_INSTANCE: ${{ inputs.environment == 'production' && 'orca-cloud-auth-db' || 'orca-cloud-staging-auth-db' }} + VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER: db-custom-4-15360 + VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION: POSTGRES_17 + TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }} + TF_VARS: ${{ inputs.environment == 'production' && 'environments/production.tfvars' || 'environments/staging.tfvars' }} + TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }} + TOPOLOGY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT }} + steps: + - uses: actions/checkout@v4 + + - name: Validate the reviewed request before authentication + shell: bash + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + set -euo pipefail + test -n "${TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${TOPOLOGY_SERVICE_ACCOUNT}" + case "${TARGET_ENVIRONMENT}:${TARGET_CELL_IDS}" in + staging:staging-gce-c4) ;; + production:production-gce-c27,production-gce-c28,production-gce-c29) ;; + *) echo "cell-ids do not match the reviewed environment topology" >&2; exit 1 ;; + esac + [[ "${TARGET_IMAGE}" =~ ^us-central1-docker\.pkg\.dev/${GCP_PROJECT_ID}/orca-cloud/relay@sha256:[0-9a-f]{64}$ ]] + if test "${DEPLOY_MODE}" = apply; then + test "${CONFIRMATION}" = APPLY_RELAY_ASIA_TOPOLOGY + else + test "${DEPLOY_MODE}" = plan + test -z "${CONFIRMATION}" + fi + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: 1.15.8 + terraform_wrapper: false + + - uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ env.TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ env.TOPOLOGY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }} + object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }} + + - name: Require the checked Cloud SQL connection budget + shell: bash + run: | + set -euo pipefail + budget="$(node dev/scripts/relay-cloud-sql-connection-budget.mjs)" + checked_max="$(jq -er '.maxConnections' <<< "${budget}")" + jq -e '.withinBudget == true' <<< "${budget}" >/dev/null + if test "${TARGET_ENVIRONMENT}" = production; then + instance="$(gcloud sql instances describe "${CLOUD_SQL_INSTANCE}" \ + --project "${GCP_PROJECT_ID}" --format=json)" + live_flag="$(jq -er '[.settings.databaseFlags[]? | + select(.name == "max_connections") | .value] | + if length <= 1 then (.[0] // "") else error("duplicate max_connections flags") end' \ + <<< "${instance}")" + if test -n "${live_flag}"; then + live_max="${live_flag}" + live_source=explicit-flag + else + # The verified production database uses Cloud SQL's 400-connection + # default for this exact shape; fail closed if its shape changes. + test "$(jq -er '.settings.tier' <<< "${instance}")" = \ + "${VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER}" + test "$(jq -er '.databaseVersion' <<< "${instance}")" = \ + "${VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION}" + live_max=400 + live_source=verified-shape-default + fi + test "${live_max}" = "${checked_max}" + else + live_max="not-read-for-staging" + live_source=not-read-for-staging + fi + { + echo "### Relay Cloud SQL connection budget" + echo "- Checked maximum: ${checked_max}" + echo "- Configured maximum: $(jq -er '.configuredMaximum' <<< "${budget}")" + echo "- Rollout operating maximum: $(jq -er '.operatingMaximum' <<< "${budget}")" + echo "- Explicit reserve: $(jq -er '.explicitReserve' <<< "${budget}")" + echo "- Production live max_connections: ${live_max}" + echo "- Production live maximum source: ${live_source}" + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Initialize the exact environment state + run: terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}" + + - id: targets + name: Build the exact additive target set + shell: bash + run: | + set -euo pipefail + file="${RUNNER_TEMP}/relay-asia-targets" + : > "${file}" + printf '%s\n' \ + '-target=google_compute_subnetwork.relay_gce_additional["asia-east2"]' \ + '-target=google_compute_router.relay_gce_additional["asia-east2"]' \ + '-target=google_compute_router_nat.relay_gce_additional["asia-east2"]' \ + '-target=google_compute_url_map.relay_gce[0]' >> "${file}" + IFS=, read -ra cells <<< "${TARGET_CELL_IDS}" + for cell_id in "${cells[@]}"; do + printf '%s\n' \ + "-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \ + "-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \ + "-target=google_compute_backend_service.relay_gce_cell[\"${cell_id}\"]" >> "${file}" + done + echo "file=${file}" >> "${GITHUB_OUTPUT}" + + - name: Create and validate the saved topology plan + id: plan + shell: bash + run: | + set -euo pipefail + plan="${RUNNER_TEMP}/relay-asia-topology.tfplan" + plan_json="${RUNNER_TEMP}/relay-asia-topology.json" + mapfile -t targets < "${{ steps.targets.outputs.file }}" + terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \ + -var-file="${TF_VARS}" \ + -var manage_artifact_dns=false \ + "${targets[@]}" -out="${plan}" + terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}" + committed="${RUNNER_TEMP}/relay-committed-asia-topology.json" + jq -e '{ + relay_gce_cells: .variables.relay_gce_cells.value, + relay_gce_additional_region_subnetwork_cidrs: + .variables.relay_gce_additional_region_subnetwork_cidrs.value + }' "${plan_json}" > "${committed}" + node dev/scripts/prepare-relay-asia-topology-input.mjs \ + --existing-json "${committed}" \ + --environment "${TARGET_ENVIRONMENT}" \ + --cell-ids "${TARGET_CELL_IDS}" \ + --image "${TARGET_IMAGE}" + result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \ + --plan-json "${plan_json}" \ + --environment "${TARGET_ENVIRONMENT}" \ + --cell-ids "${TARGET_CELL_IDS}" \ + --region "${TARGET_REGION}" \ + --image "${TARGET_IMAGE}")" + changes="$(jq -er '.changes' <<< "${result}")" + digest="$(sha256sum "${plan}" | awk '{print $1}')" + echo "plan=${plan}" >> "${GITHUB_OUTPUT}" + echo "changes=${changes}" >> "${GITHUB_OUTPUT}" + { + echo "### Relay Asia topology saved plan" + echo "- Environment: ${TARGET_ENVIRONMENT}" + echo "- Cells: ${TARGET_CELL_IDS}" + echo "- Region: ${TARGET_REGION}" + echo "- Mutating resources: ${changes}" + echo "- Saved-plan SHA-256: ${digest}" + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Apply only the validated saved plan + if: ${{ inputs.mode == 'apply' }} + run: terraform -chdir=infra/terraform apply -input=false -auto-approve "${{ steps.plan.outputs.plan }}" + + - name: Prove the exact topology targets converged + if: ${{ inputs.mode == 'apply' }} + shell: bash + run: | + set -euo pipefail + mapfile -t targets < "${{ steps.targets.outputs.file }}" + plan="${RUNNER_TEMP}/relay-asia-topology-readback.tfplan" + plan_json="${RUNNER_TEMP}/relay-asia-topology-readback.json" + terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \ + -var-file="${TF_VARS}" \ + -var manage_artifact_dns=false \ + "${targets[@]}" -out="${plan}" + terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}" + result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \ + --plan-json "${plan_json}" \ + --environment "${TARGET_ENVIRONMENT}" \ + --cell-ids "${TARGET_CELL_IDS}" \ + --region "${TARGET_REGION}" \ + --image "${TARGET_IMAGE}")" + test "$(jq -er '.changes' <<< "${result}")" = 0 + + - name: Record the required selector-safe next step + if: ${{ inputs.mode == 'apply' }} + run: | + { + echo "### Required next step" + echo "The VMs are not eligible for ordinary placement yet." + echo "Register the exact new cells atomically as migration-only before any director configuration lists them." + echo "Rollback is migration-only admission; do not destroy the Asia network on rollout day." + } >> "${GITHUB_STEP_SUMMARY}" diff --git a/.github/workflows/cloud-deploy-relay-fence-broker.yml b/.github/workflows/cloud-deploy-relay-fence-broker.yml new file mode 100644 index 00000000000..cda05fd8e5d --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-fence-broker.yml @@ -0,0 +1,93 @@ +name: Deploy Relay Fence Broker + +on: + workflow_dispatch: + inputs: + image-digest: + description: Immutable broker image digest built from this main commit + required: true + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + deploy: + if: >- + ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && + github.ref == 'refs/heads/main' && + vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER != '' && + vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT != '' && + vars.PRODUCTION_GCP_REGION != '' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + SERVICE_NAME: orca-cloud-relay-fence + IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay-fence-broker + IMAGE_DIGEST: ${{ inputs.image-digest }} + steps: + - uses: actions/checkout@v4 + + - uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + + - name: Resolve exact-commit broker image + run: | + [[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]] + IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}" + SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \ + --format='value(image_summary.digest)')" + test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}" + TAGS="$(gcloud artifacts docker tags list "${IMAGE_REPOSITORY}" \ + --filter="version:${IMAGE_DIGEST}" \ + --format=json)" + jq -e --arg tag "/tags/sha-${GITHUB_SHA}" \ + 'any(.[]; .tag | endswith($tag))' <<< "${TAGS}" + echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}" + + - name: Deploy broker image only + run: | + gcloud run services update "${SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --image "${IMAGE}" \ + --quiet + + - name: Verify ready singleton revision + run: | + SERVICE="$(gcloud run services describe "${SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format=json)" + jq -e '.status.conditions[] | select(.type == "Ready" and .status == "True")' \ + <<< "${SERVICE}" + REVISION="$(jq -r \ + '[.status.traffic[] | select((.percent // 0) == 100)] | + if length == 1 then .[0].revisionName // empty else empty end' \ + <<< "${SERVICE}")" + test -n "${REVISION}" + SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format='value(spec.containers[0].image)')" + test "${SERVED_IMAGE}" = "${IMAGE}" diff --git a/.github/workflows/cloud-deploy-relay-production-capacity-job.yml b/.github/workflows/cloud-deploy-relay-production-capacity-job.yml new file mode 100644 index 00000000000..eeea3d11a49 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-capacity-job.yml @@ -0,0 +1,820 @@ +name: Deploy Relay Production Capacity Job + +on: + workflow_call: + inputs: + mode: + required: true + type: string + target-cell-id: + required: true + type: string + confirmation: + required: true + type: string + monitor-run-id: + required: true + type: string + monitor-run-attempt: + required: true + type: string + evidence-mode: + required: true + type: string + wave-cell-ids: + required: true + type: string + wave-index: + required: true + type: string + source-wave-run-id: + required: true + type: string + +permissions: + actions: read + contents: read + id-token: write + +defaults: + run: + working-directory: cloud + +jobs: + capacity: + if: ${{ github.ref == 'refs/heads/main' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 75 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + DIRECTOR_SERVICE_NAME: orca-cloud-relay + DIRECTOR_ORIGIN: https://relay.onorca.dev + TARGET_CELL_ID: ${{ inputs.target-cell-id }} + CAPACITY_CELL_IDS: production-gce-c7,production-gce-c8,production-gce-c9,production-gce-c10,production-gce-c13,production-gce-c14,production-gce-c15,production-gce-c16,production-gce-c19,production-gce-c20,production-gce-c21,production-gce-c22,production-gce-c23,production-gce-c24,production-gce-c25,production-gce-c26 + PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d + COMPATIBLE_DIRECTOR_IMAGE_DIGEST: sha256:01b7fc3e6dce66180034f268a2dc92c05458706c5b3a0dc4450dcdd6161f6e73 + COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f + CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + DEPLOY_MODE: ${{ inputs.mode }} + EVIDENCE_MODE: ${{ inputs.evidence-mode }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }} + WAVE_INDEX: ${{ inputs.wave-index }} + SOURCE_WAVE_RUN_ID: ${{ inputs.source-wave-run-id }} + steps: + - name: Require exact reusable-workflow invocation + run: | + [[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]] + if test "${EVIDENCE_MODE}" = continuation; then + test "${DEPLOY_MODE}" = apply + [[ "${WAVE_INDEX}" =~ ^[0-3]$ ]] + test "${WAVE_CELL_IDS}" != none + test "${SOURCE_WAVE_RUN_ID}" = none + elif test "${EVIDENCE_MODE}" = resume; then + test "${DEPLOY_MODE}" = apply + test "${WAVE_INDEX}" = resume + test "${WAVE_CELL_IDS}" != none + [[ "${SOURCE_WAVE_RUN_ID}" =~ ^[0-9]+$ ]] + else + test "${EVIDENCE_MODE}" = single + test "${WAVE_CELL_IDS}" = none + test "${WAVE_INDEX}" = 0 + test "${SOURCE_WAVE_RUN_ID}" = none + fi + + - name: Require production workflow configuration + env: + DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + CAPACITY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + run: | + test -n "${GCP_REGION}" + test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${DEPLOY_SERVICE_ACCOUNT}" + test -n "${CAPACITY_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${CAPACITY_SERVICE_ACCOUNT}" + + - uses: actions/checkout@v4 + + - id: resume-provenance + if: ${{ inputs.evidence-mode == 'resume' }} + env: + GH_TOKEN: ${{ github.token }} + run: | + test "${MONITOR_RUN_ATTEMPT}" = 1 + case "${MONITOR_RUN_ID}:${SOURCE_WAVE_RUN_ID}:${WAVE_CELL_IDS}:${TARGET_CELL_ID}" in + 31554591366:31555510376:production-gce-c16,production-gce-c15,production-gce-c14,production-gce-c13:production-gce-c13) + EXPECTED_SHA=a917e8e1fc1a2654e8cb81ba39b57733ec56be9c + EXPECTED_SOURCE_ATTEMPT=1 + ;; + 31562760783:31563664692:production-gce-c10,production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c10) + EXPECTED_SHA=6082e9ca89a918ca51f0c87db003f5e8805b64b7 + EXPECTED_SOURCE_ATTEMPT=1 + ;; + 31571019947:31572080665:production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c8) + EXPECTED_SHA=e59958130c9d9b7a6cd805df2678d08997842c7c + EXPECTED_SOURCE_ATTEMPT=2 + ;; + *) exit 1 ;; + esac + MONITOR_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${MONITOR_RUN_ID}" \ + --jq 'select(.name == "Monitor Relay Production" and + .path == ".github/workflows/cloud-monitor-relay-production.yml" and + .head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and + .event == "workflow_dispatch" and .conclusion == "success" and .run_attempt == 1) | + .head_sha')" + SOURCE_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \ + --jq 'select(.name == "Deploy Relay Production Capacity" and + .path == ".github/workflows/cloud-deploy-relay-production-capacity.yml" and + .head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and + .event == "workflow_dispatch" and .conclusion == "failure") | + .head_sha')" + SOURCE_ATTEMPT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \ + --jq '.run_attempt')" + [[ "${MONITOR_SHA}" =~ ^[0-9a-f]{40}$ ]] + test "${MONITOR_SHA}" = "${EXPECTED_SHA}" + test "${SOURCE_SHA}" = "${MONITOR_SHA}" + test "${SOURCE_ATTEMPT}" = "${EXPECTED_SOURCE_ATTEMPT}" + echo "commit-sha=${MONITOR_SHA}" >> "${GITHUB_OUTPUT}" + + - name: Require fresh dry-run evidence reference + if: ${{ inputs.mode == 'apply' }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + + - name: Download private dry-run evidence + if: ${{ inputs.mode == 'apply' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - uses: pnpm/action-setup@v4 + with: + package_json_file: cloud/package.json + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - name: Verify dry-run artifact before cloud authentication + if: ${{ inputs.mode == 'apply' }} + run: | + EVIDENCE_COMMIT_SHA="${GITHUB_SHA}" + if test "${EVIDENCE_MODE:-single}" = resume; then + EVIDENCE_COMMIT_SHA="${{ steps.resume-provenance.outputs.commit-sha }}" + fi + node dev/scripts/relay-monitor-evidence.mjs verify-restore \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${EVIDENCE_COMMIT_SHA}" \ + --mode dry-run + + - name: Reject previously consumed dry-run evidence + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }} + env: + GH_TOKEN: ${{ github.token }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api \ + "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = "0" + + - name: Download this workflow's wave authority + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-wave-authority + github-token: ${{ github.token }} + run-id: ${{ github.run_id }} + + - name: Download the failed wave authority for resume + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-wave-authority + github-token: ${{ github.token }} + run-id: ${{ inputs.source-wave-run-id }} + + - name: Require wave evidence consumed by this workflow + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${GITHUB_RUN_ID}" + + - name: Require wave evidence consumed by the failed source workflow + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${SOURCE_WAVE_RUN_ID}" + + - id: deploy-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + release: 'false' + + - name: Require exact mutation confirmation + if: ${{ inputs.mode != 'verify' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if test "${EVIDENCE_MODE:-single}" = resume; then + test "${CONFIRMATION}" = "RESUME_SELECTED_CELL_TO_1000 ${TARGET_CELL_ID}" + elif test "${DEPLOY_MODE}" = apply; then + test "${CONFIRMATION}" = "RAISE_SELECTED_CELL_TO_1000" + else + test "${CONFIRMATION}" = "ROLL_BACK_SELECTED_CELL_TO_600 ${TARGET_CELL_ID}" + fi + + - name: Initialize the exact production backend + run: node dev/scripts/infra.mjs init --env production + + - name: Build the exact selected-cell configuration + shell: bash + run: | + if test "${DEPLOY_MODE}" = rollback; then + TARGET_HARD_CAP=600 + else + TARGET_HARD_CAP=1000 + fi + TARGET_UNOBSERVED_BOUND=60 + TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}" + [[ "${TARGET_HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]] + CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev" + CELLS_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars \ + -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')" + OVERRIDE_CELLS_JSON="$(jq -ce \ + --arg cell "${TARGET_CELL_ID}" \ + --argjson cap "${TARGET_HARD_CAP}" \ + --argjson bound "${TARGET_UNOBSERVED_BOUND}" \ + '.[$cell].connection_hard_cap = $cap | + .[$cell].connection_unobserved_bound = $bound' \ + <<< "${CELLS_JSON}")" + jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \ + '{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-capacity.tfvars.json" + BASE_CELLS_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars \ + -var manage_artifact_dns=false \ + <<< 'local.relay_director_cells_json' | jq -er '.')" + IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image" + ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone" + DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars \ + -var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \ + -var manage_artifact_dns=false \ + <<< "${IMAGE_EXPRESSION}" | jq -r '.')" + TARGET_ZONE="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars \ + -var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \ + -var manage_artifact_dns=false \ + <<< "${ZONE_EXPRESSION}" | jq -r '.')" + MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \ + | jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')" + jq -e --arg cell "${TARGET_CELL_ID}" \ + 'any(.[]; .id == $cell and .connectionHardCap == 1000 and + .connectionUnobservedBound == 60)' \ + <<< "${BASE_CELLS_JSON}" >/dev/null + [[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]] + DESIRED_IMAGE_DIGEST="${DESIRED_IMAGE##*@}" + [[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]] + test "${MIG_NAME}" = "orca-cloud-relay-gce-${TARGET_HOSTNAME}" + { + echo "CELL_ORIGIN=${CELL_ORIGIN}" + echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}" + echo "TARGET_HARD_CAP=${TARGET_HARD_CAP}" + echo "TARGET_UNOBSERVED_BOUND=${TARGET_UNOBSERVED_BOUND}" + echo "DESIRED_IMAGE=${DESIRED_IMAGE}" + echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}" + echo "TARGET_ZONE=${TARGET_ZONE}" + echo "MIG_NAME=${MIG_NAME}" + echo "BASE_CELLS_JSON=${BASE_CELLS_JSON}" + } >> "${GITHUB_ENV}" + + - name: Require the exact compatible production image and topology + shell: bash + run: | + SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + ACTIVE_REVISION="$(jq -r \ + '[.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \ + <<< "${SERVICE_JSON}")" + test -n "${ACTIVE_REVISION}" + ACTIVE_REVISION_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + ACTIVE_IMAGE="$(jq -er '.spec.containers[0].image' <<< "${ACTIVE_REVISION_JSON}")" + ACTIVE_IMAGE_DIGEST="${ACTIVE_IMAGE##*@}" + if test "${ACTIVE_IMAGE}" != "${DESIRED_IMAGE}"; then + test "${ACTIVE_IMAGE_DIGEST}" = "${COMPATIBLE_DIRECTOR_IMAGE_DIGEST}" + test "${DESIRED_IMAGE_DIGEST}" = "${COMPATIBLE_CELL_IMAGE_DIGEST}" + fi + CURRENT_CELLS_JSON="$(jq -cer '[.spec.containers[0].env[]? | + select(.name == "ORCA_RELAY_CELLS_JSON") | .value] | + if length == 1 then .[0] | fromjson else error("missing director topology") end' \ + <<< "${ACTIVE_REVISION_JSON}")" + CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON="$( + node dev/scripts/read-relay-production-capacity-identity.mjs \ + <<< "${ACTIVE_REVISION_JSON}" + )" + CLASSIFICATION="$(jq -nc \ + --argjson baseCells "${BASE_CELLS_JSON}" \ + --argjson currentCells "${CURRENT_CELLS_JSON}" \ + --arg capacityCellIds "${CAPACITY_CELL_IDS}" \ + --arg targetCellId "${TARGET_CELL_ID}" \ + --argjson targetHardCap "${TARGET_HARD_CAP}" \ + --argjson currentCapacityServiceAccount \ + "${CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON}" \ + '{baseCells:$baseCells, currentCells:$currentCells, + capacityCellIds:($capacityCellIds | split(",")), + targetCellId:$targetCellId, targetHardCap:$targetHardCap, + currentCapacityServiceAccount:$currentCapacityServiceAccount}' \ + | node dev/scripts/classify-relay-production-capacity-director.mjs \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")" + TOPOLOGY_PHASE="$(jq -er '.topologyPhase' <<< "${CLASSIFICATION}")" + DESIRED_CELLS_JSON="$(jq -cer '.desiredCells' <<< "${CLASSIFICATION}")" + DIRECTOR_READY="$(jq -er \ + 'if (.directorReady | type) == "boolean" then + (.directorReady | tostring) + else error("invalid directorReady classification") end' \ + <<< "${CLASSIFICATION}")" + { + echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}" + echo "TOPOLOGY_PHASE=${TOPOLOGY_PHASE}" + echo "DIRECTOR_READY=${DIRECTOR_READY}" + echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" + } >> "${GITHUB_ENV}" + + - name: Require the exact wave predecessor topology + if: ${{ inputs.mode == 'apply' && (inputs.evidence-mode == 'continuation' || inputs.evidence-mode == 'resume') }} + run: test "${TOPOLOGY_PHASE}" = predecessor + + - name: Verify fresh dry-run evidence against the live selector + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-mutation \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run \ + --mutation-mode capacity-transition \ + --source-cell-id "${TARGET_CELL_ID}" \ + --director-origin "${DIRECTOR_ORIGIN}" + + - name: Recheck every live safety signal + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + pnpm incident:relay-preflight -- \ + --state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" + + - name: Recheck exact wave state and every live safety signal + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + node dev/scripts/relay-production-capacity-wave.mjs build-preflight \ + --state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \ + --wave-cell-ids "${WAVE_CELL_IDS}" \ + --wave-index "${WAVE_INDEX}" \ + --target-cell-id "${TARGET_CELL_ID}" \ + --output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" + RETRY_ARGS=() + if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi + pnpm incident:relay-preflight -- \ + --state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \ + "${RETRY_ARGS[@]}" + + - name: Recheck exact isolated resume state and every live safety signal + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + node dev/scripts/relay-production-capacity-wave.mjs build-resume-preflight \ + --state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \ + --wave-cell-ids "${WAVE_CELL_IDS}" \ + --target-cell-id "${TARGET_CELL_ID}" \ + --output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" + pnpm incident:relay-preflight -- \ + --state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \ + --retry-freshness + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission migration-only \ + --draining required \ + --activity allowed \ + --runtime required \ + --expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}" + + - name: Consume the single-use dry-run evidence + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}" + + - name: Publish the consumed-evidence marker + if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }} + uses: actions/upload-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + - name: Verify current selected-cell capacity + if: ${{ inputs.mode == 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + CURRENT_CAP="${TARGET_HARD_CAP}" + CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}" + if test "${TOPOLOGY_PHASE}" = predecessor; then + CURRENT_CAP=600 + CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${CURRENT_CAP}" \ + --unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed \ + --expected-image-digests "${CURRENT_IMAGE_DIGEST}" + + - name: Arm fail-closed mutation cleanup + if: ${{ inputs.mode != 'verify' }} + run: echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + + - name: Reversibly isolate only the selected cell + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + test "${MUTATION_STARTED:-false}" = true || exit 0 + node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode isolate + + - name: Drain the selected cell or prove an offline rollback + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + if node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode drain; then + echo "OFFLINE_ROLLBACK=false" >> "${GITHUB_ENV}" + elif test "${DEPLOY_MODE}" = rollback; then + echo "OFFLINE_ROLLBACK=true" >> "${GITHUB_ENV}" + else + exit 1 + fi + + - id: restart-auth-one + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - id: restart-gate-one + name: Require restart-safe selected-cell activity + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-one.outputs.id_token }} + run: | + if test "${OFFLINE_ROLLBACK:-false}" = true; then + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --heartbeat stale \ + --admission migration-only \ + --draining either \ + --activity restart-safe \ + --runtime unavailable + echo "settled=true" >> "${GITHUB_OUTPUT}" + exit 0 + fi + CURRENT_CAP=600 + if test "${TOPOLOGY_PHASE}" = desired; then + CURRENT_CAP="${TARGET_HARD_CAP}" + elif test "${TARGET_HARD_CAP}" = 600; then + CURRENT_CAP=1000 + fi + GATE_LOG="${RUNNER_TEMP}/relay-capacity-restart-gate-one.log" + set +e + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${CURRENT_CAP}" \ + --unobserved-bound 60 \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity restart-safe \ + --runtime required \ + --timeout-ms 450000 \ + --expected-image-digests \ + "${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" \ + 2> "${GATE_LOG}" + GATE_EXIT=$? + set -e + cat "${GATE_LOG}" >&2 + if test "${GATE_EXIT}" = 0; then + echo "settled=true" >> "${GITHUB_OUTPUT}" + exit 0 + fi + if test "$(wc -l < "${GATE_LOG}" | tr -d ' ')" = 1 && + grep -Eq '^capacity transition verification timed out: \{.*\}$' "${GATE_LOG}"; then + echo "settled=false" >> "${GITHUB_OUTPUT}" + exit 0 + fi + exit "${GATE_EXIT}" + + - id: restart-auth-two + if: ${{ steps.restart-gate-one.outputs.settled == 'false' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Require extended restart-safe selected-cell activity + if: ${{ steps.restart-gate-one.outputs.settled == 'false' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-two.outputs.id_token }} + run: | + CURRENT_CAP=600 + if test "${TOPOLOGY_PHASE}" = desired; then + CURRENT_CAP="${TARGET_HARD_CAP}" + elif test "${TARGET_HARD_CAP}" = 600; then + CURRENT_CAP=1000 + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${CURRENT_CAP}" \ + --unobserved-bound 60 \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity restart-safe \ + --runtime required \ + --timeout-ms 450000 \ + --expected-image-digests \ + "${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" + + - name: Deploy only the reviewed director topology + if: ${{ inputs.mode != 'verify' }} + run: | + if test "${DIRECTOR_READY}" = true; then exit 0; fi + RELEASE_ID="capacity-${TARGET_HOSTNAME}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}" + node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service "${DIRECTOR_SERVICE_NAME}" \ + --image "${ACTIVE_IMAGE}" \ + --role director \ + --max-instances 5 \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \ + --capacity-cell-id "${TARGET_CELL_ID}" \ + --director-cells-json "${DESIRED_CELLS_JSON}" \ + --min-instances 5 \ + --prune-revisions false \ + --release-id "${RELEASE_ID}" + + - id: director-transition-auth + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Require fail-closed director transition + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.director-transition-auth.outputs.id_token }} + run: | + if test "${OFFLINE_ROLLBACK:-false}" = true; then + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --heartbeat stale \ + --admission migration-only \ + --draining either \ + --activity restart-safe \ + --runtime unavailable + exit 0 + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${TARGET_HARD_CAP}" \ + --unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity restart-safe \ + --runtime required \ + --expected-image-digests \ + "${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" + + - id: capacity-auth + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Plan and apply only the empty selected cell + if: ${{ inputs.mode != 'verify' }} + shell: bash + run: | + terraform -chdir=infra/terraform plan \ + -var-file=environments/production.tfvars \ + -var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \ + -var manage_artifact_dns=false \ + "-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + "-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + -out="${RUNNER_TEMP}/relay-capacity-cell.tfplan" + PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \ + "${RUNNER_TEMP}/relay-capacity-cell.tfplan" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode bootstrap-cell \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${TARGET_HARD_CAP}" \ + --unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \ + --image "${DESIRED_IMAGE}" \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")" + echo "${PLAN_RESULT}" + PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")" + [[ "${PLAN_CHANGES}" =~ ^(0|1|2)$ ]] + if test "${PLAN_CHANGES}" != 0; then + terraform -chdir=infra/terraform apply \ + -auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan" + else + INSTANCE="$(gcloud compute instance-groups managed list-instances \ + "${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \ + --format=json | jq -er 'if length == 1 and + .[0].instanceStatus == "RUNNING" and .[0].currentAction == "NONE" + then .[0].instance | split("/") | last + else error("selected cell is not one stable running instance") end')" + gcloud compute instance-groups managed recreate-instances \ + "${MIG_NAME}" --instances "${INSTANCE}" \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet + fi + gcloud compute instance-groups managed wait-until \ + "${MIG_NAME}" --stable --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" --timeout 900 + + - id: capacity-transition-auth + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Verify fresh exact selected-cell heartbeat before admission + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${TARGET_HARD_CAP}" \ + --unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission migration-only \ + --draining forbidden \ + --activity allowed \ + --expected-image-digests "${DESIRED_IMAGE_DIGEST}" + + - name: Restore only the selected cell to general admission + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }} + run: | + node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode activate + + - name: Verify the live general selected cell + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${TARGET_HARD_CAP}" \ + --unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed \ + --expected-image-digests "${DESIRED_IMAGE_DIGEST}" + + - id: cleanup-auth + if: ${{ failure() && inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Keep the selected cell isolated after a failed mutation + if: ${{ failure() && inputs.mode != 'verify' }} + continue-on-error: true + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }} + run: | + test "${MUTATION_STARTED:-false}" = true || exit 0 + CLEANUP_STATUS=0 + node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode isolate || CLEANUP_STATUS=$? + node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode drain || CLEANUP_STATUS=$? + exit "${CLEANUP_STATUS}" diff --git a/.github/workflows/cloud-deploy-relay-production-capacity.yml b/.github/workflows/cloud-deploy-relay-production-capacity.yml new file mode 100644 index 00000000000..830abe8f7cd --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-capacity.yml @@ -0,0 +1,352 @@ +name: Deploy Relay Production Capacity + +on: + workflow_dispatch: + inputs: + mode: + description: Verify, change one cell, or raise a sequential wave + required: true + default: verify + type: choice + options: + - verify + - apply + - rollback + - wave-apply + - wave-resume + target-cell-id: + description: Exact serving cell for verify, apply, or rollback + required: true + default: production-gce-c26 + type: choice + options: + - production-gce-c7 + - production-gce-c8 + - production-gce-c9 + - production-gce-c10 + - production-gce-c13 + - production-gce-c14 + - production-gce-c15 + - production-gce-c16 + - production-gce-c19 + - production-gce-c20 + - production-gce-c21 + - production-gce-c22 + - production-gce-c23 + - production-gce-c24 + - production-gce-c25 + - production-gce-c26 + wave-cell-ids: + description: Ordered comma-separated wave of two to four serving cells + required: false + default: none + type: string + confirmation: + description: Enter the exact single-cell or wave confirmation + required: false + type: string + monitor-run-id: + description: Successful fresh dry-run monitor workflow run ID for apply + required: false + type: string + monitor-run-attempt: + description: Exact dry-run monitor workflow attempt for apply + required: false + type: string + source-wave-run-id: + description: Failed wave run that isolated the resume target + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + single_cell: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode != 'wave-apply' && inputs.mode != 'wave-resume') }} + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: ${{ inputs.mode }} + target-cell-id: ${{ inputs.target-cell-id }} + confirmation: ${{ inputs.confirmation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: single + wave-cell-ids: none + wave-index: '0' + source-wave-run-id: none + secrets: inherit + + resume_cell: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-resume' && github.ref == 'refs/heads/main') }} + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: apply + target-cell-id: ${{ inputs.target-cell-id }} + confirmation: ${{ inputs.confirmation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: resume + wave-cell-ids: ${{ inputs.wave-cell-ids }} + wave-index: resume + source-wave-run-id: ${{ inputs.source-wave-run-id }} + secrets: inherit + + wave_gate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-apply' && github.ref == 'refs/heads/main') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 30 + environment: production + outputs: + cells: ${{ steps.wave.outputs.cells }} + env: + DIRECTOR_ORIGIN: https://relay.onorca.dev + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence + PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d + COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f + WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }} + steps: + - name: Require production workflow configuration + env: + DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + run: | + test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${DEPLOY_SERVICE_ACCOUNT}" + + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + package_json_file: cloud/package.json + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - id: wave + name: Validate the exact wave request + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + CELLS="$(node dev/scripts/relay-production-capacity-wave.mjs validate \ + --wave-cell-ids "${WAVE_CELL_IDS}" \ + --confirmation "${CONFIRMATION}")" + echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}" + + - name: Require fresh dry-run evidence reference + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + + - name: Download private dry-run evidence + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ github.workspace }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - name: Verify dry-run artifact before cloud authentication + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-restore \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run + + - name: Reject previously consumed dry-run evidence + env: + GH_TOKEN: ${{ github.token }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api \ + "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = "0" + + - id: deploy-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + release: 'false' + + - name: Verify wave evidence against the live selector + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + FIRST_CELL="$(jq -er '.[0]' <<< '${{ steps.wave.outputs.cells }}')" + node dev/scripts/relay-monitor-evidence.mjs verify-mutation \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run \ + --mutation-mode capacity-transition \ + --source-cell-id "${FIRST_CELL}" \ + --director-origin "${DIRECTOR_ORIGIN}" + + - name: Recheck every live safety signal + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + pnpm incident:relay-preflight -- \ + --state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" + + - name: Require exact 600/60 predecessor wave cells + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + while read -r CELL_ID; do + HOSTNAME="${CELL_ID#production-gce-}" + [[ "${HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]] + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "https://${HOSTNAME}.relay.onorca.dev" \ + --cell-id "${CELL_ID}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed \ + --expected-image-digests \ + "${PREDECESSOR_IMAGE_DIGEST},${COMPATIBLE_CELL_IMAGE_DIGEST}" + done < <(jq -r '.[]' <<< '${{ steps.wave.outputs.cells }}') + + - name: Consume the single-use dry-run evidence + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}" + + - name: Publish the consumed-evidence marker + uses: actions/upload-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + wave_cell_1: + needs: wave_gate + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: apply + target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[0] }} + confirmation: RAISE_SELECTED_CELL_TO_1000 + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: continuation + wave-cell-ids: ${{ inputs.wave-cell-ids }} + wave-index: '0' + source-wave-run-id: none + secrets: inherit + + wave_cell_2: + needs: [wave_gate, wave_cell_1] + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: apply + target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[1] }} + confirmation: RAISE_SELECTED_CELL_TO_1000 + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: continuation + wave-cell-ids: ${{ inputs.wave-cell-ids }} + wave-index: '1' + source-wave-run-id: none + secrets: inherit + + wave_cell_3: + if: ${{ needs.wave_cell_2.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[2] != null }} + needs: [wave_gate, wave_cell_2] + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: apply + target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[2] }} + confirmation: RAISE_SELECTED_CELL_TO_1000 + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: continuation + wave-cell-ids: ${{ inputs.wave-cell-ids }} + wave-index: '2' + source-wave-run-id: none + secrets: inherit + + wave_cell_4: + if: ${{ needs.wave_cell_3.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[3] != null }} + needs: [wave_gate, wave_cell_3] + uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml + with: + mode: apply + target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[3] }} + confirmation: RAISE_SELECTED_CELL_TO_1000 + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + evidence-mode: continuation + wave-cell-ids: ${{ inputs.wave-cell-ids }} + wave-index: '3' + source-wave-run-id: none + secrets: inherit + + # Every wave job re-enters the run's lease with release: 'false'; only this job frees it. + release_lease: + if: always() + needs: + - single_cell + - resume_cell + - wave_gate + - wave_cell_1 + - wave_cell_2 + - wave_cell_3 + - wave_cell_4 + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 10 + environment: production + steps: + - uses: actions/checkout@v4 + + - uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + release: 'true' diff --git a/.github/workflows/cloud-deploy-relay-production-director.yml b/.github/workflows/cloud-deploy-relay-production-director.yml new file mode 100644 index 00000000000..97abe2d227b --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-director.yml @@ -0,0 +1,267 @@ +name: Deploy Relay Production Director + +on: + workflow_dispatch: + inputs: + image-digest: + description: "Immutable relay image digest (sha256: plus 64 lowercase hex characters)" + required: true + type: string + regional-placement-mode: + description: Preserve the live switch, explicitly enable Asia preference, or force US-first + required: true + default: preserve + type: choice + options: [preserve, enable, disable] + prune-incompatible-revisions: + description: Retain only the newly verified serving and rollback revisions + required: true + default: false + type: boolean + confirmation: + description: Enter the exact confirmation required by a destructive option + required: false + type: string + expected-rehome-generation: + description: Exact durable regional-rehome generation; it must remain disabled + required: true + type: string + bootstrap-runtime-identity: + description: One-time move from the stamped-cell identity to the director identity + required: true + default: false + type: boolean + predecessor-image-digest: + description: Exact immutable serving predecessor digest for the one-time identity bootstrap + required: true + type: string + +permissions: + contents: read + id-token: write + +# Director updates and candidate operations both mutate production relay control state. +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + deploy: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + DIRECTOR_SERVICE_NAME: orca-cloud-relay + IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay + REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled + IMAGE_DIGEST: ${{ inputs.image-digest }} + REGIONAL_PLACEMENT_MODE: ${{ inputs.regional-placement-mode }} + PRUNE_INCOMPATIBLE_REVISIONS: ${{ inputs.prune-incompatible-revisions }} + # Floor the served revision must keep, matching relay_min_instances in + # environments/production.tfvars. This gate only fails a bad deploy; Terraform + # still owns the value, and the candidate inherits it from the serving revision. + DIRECTOR_MIN_INSTANCES: 5 + DIRECTOR_MAX_INSTANCES: 5 + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + PREDECESSOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_RUNTIME_SERVICE_ACCOUNT }} + REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain + EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }} + BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }} + PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }} + steps: + - uses: actions/checkout@v4 + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Resolve immutable production image + shell: bash + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if [[ ! "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]; then + echo "image-digest must be an immutable lowercase sha256 digest" >&2 + exit 1 + fi + IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}" + SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" --format='value(image_summary.digest)')" + test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}" + [[ "${PRUNE_INCOMPATIBLE_REVISIONS}" =~ ^(true|false)$ ]] + [[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + [[ "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]] + [[ "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]] + [[ "${BOOTSTRAP_RUNTIME_IDENTITY}" =~ ^(true|false)$ ]] + if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then + [[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false + test "${REGIONAL_PLACEMENT_MODE}" = preserve + test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY + elif test "${PRUNE_INCOMPATIBLE_REVISIONS}" = true; then + test "${REGIONAL_PLACEMENT_MODE}" = preserve + test "${CONFIRMATION}" = PRUNE_INCOMPATIBLE_RELAY_DIRECTOR_REVISIONS + elif test "${REGIONAL_PLACEMENT_MODE}" = disable; then + test "${CONFIRMATION}" = FORCE_RELAY_US_FIRST + else + test -z "${CONFIRMATION}" + fi + echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}" + + # Why: the deploy INHERITS the serving revision's floor, so when that revision has + # already lost it the candidate inherits zero, the in-script gate compares zero against + # zero and passes, and the post-deploy check below only notices after traffic moved. + # The documented rollback target is created at minimum instances zero, so promoting it + # arms exactly that. Refuse to inherit a degraded floor rather than latch it. + - name: Require a healthy serving floor before deploying + shell: bash + run: | + SERVING="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -r '[.status.traffic[] | select((.percent // 0) > 0)] + | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')" + test -n "${SERVING}" + FLOOR="$(gcloud run revisions describe "${SERVING}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \ + --format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")" + if [[ "${FLOOR:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then + echo "serving revision ${SERVING} holds ${FLOOR:-0} minimum instances," \ + "below ${DIRECTOR_MIN_INSTANCES}; deploying would inherit and latch it." >&2 + echo "Restore the floor first: gcloud run services update ${DIRECTOR_SERVICE_NAME}" \ + "--min-instances=${DIRECTOR_MIN_INSTANCES}" >&2 + exit 1 + fi + CEILING="$(gcloud run revisions describe "${SERVING}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \ + --format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")" + test "${CEILING}" = "${DIRECTOR_MAX_INSTANCES}" + echo "serving revision ${SERVING} holds ${FLOOR} minimum instances" + echo "SERVING_REVISION=${SERVING}" >> "${GITHUB_ENV}" + + # Why: no --min-instances here. The candidate inherits the Terraform-owned + # scaling, and this step ends with 100% traffic on it. Pinning 1 rebuilt the + # per-instance admission shortage that took placement failures to ~70%. + - name: Deploy director blue/green + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + served_version="$(gcloud run revisions describe "${SERVING_REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -r '[.spec.containers[0].env[]? | + select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") | + (.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) | + (.version // .key // empty)] | + if length == 1 then .[0] else empty end')" + if [[ "${served_version}" =~ ^[1-9][0-9]*$ ]]; then + current_version="${served_version}" + else + test "${REGIONAL_PLACEMENT_MODE}" = preserve + current_version="$(gcloud secrets versions describe latest \ + --project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \ + --format='value(name)' | awk -F/ '{print $NF}')" + [[ "${current_version}" =~ ^[1-9][0-9]*$ ]] + fi + current="$(gcloud secrets versions access "${current_version}" \ + --project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")" + [[ "${current}" =~ ^(true|false)$ ]] + case "${REGIONAL_PLACEMENT_MODE}" in + preserve) desired="${current}" ;; + enable) desired=true ;; + disable) desired=false ;; + *) echo "regional-placement-mode is invalid" >&2; exit 1 ;; + esac + if test "${current}" != "${desired}"; then + target_version="$(printf '%s' "${desired}" | gcloud secrets versions add \ + "${REGIONAL_PLACEMENT_SECRET}" --project "${GCP_PROJECT_ID}" --data-file=- \ + --format='value(name)' --quiet | awk -F/ '{print $NF}')" + else + target_version="${current_version}" + fi + [[ "${target_version}" =~ ^[1-9][0-9]*$ ]] + RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}" + node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service "${DIRECTOR_SERVICE_NAME}" \ + --image "${IMAGE}" \ + --role director \ + --runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \ + --predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" \ + --bootstrap-runtime-identity "${BOOTSTRAP_RUNTIME_IDENTITY}" \ + --predecessor-image-digest "${PREDECESSOR_IMAGE_DIGEST}" \ + --rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \ + --rehome-audience "${REHOME_AUDIENCE}" \ + --rehome-control-origin https://relay.onorca.dev \ + --admin-audience https://relay.onorca.dev/v1/admin/drain \ + --expected-rehome-generation "${EXPECTED_REHOME_GENERATION}" \ + --max-instances "${DIRECTOR_MAX_INSTANCES}" \ + --prune-revisions "${PRUNE_INCOMPATIBLE_REVISIONS}" \ + --release-id "${RELEASE_ID}" \ + --regional-placement-secret-version "${target_version}" + echo "REGIONAL_PLACEMENT_ENABLED=${desired}" >> "${GITHUB_ENV}" + echo "REGIONAL_PLACEMENT_VERSION=${target_version}" >> "${GITHUB_ENV}" + + - name: Verify served revision and native health + shell: bash + run: | + SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format=json)" + REVISION="$(jq -r '[.status.traffic[] | select((.percent // 0) > 0)] | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' <<< "${SERVICE_JSON}")" + test -n "${REVISION}" + SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format='value(spec.containers[0].image)')" + test "${SERVED_IMAGE}" = "${IMAGE}" + SERVED_REGIONAL_PLACEMENT_SECRET="$(gcloud run revisions describe "${REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -cer '[.spec.containers[0].env[] | + select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") | + (.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) | + {secret: (.secret // .name), version: (.version // .key)}] | + if length == 1 then .[0] else error("regional placement secret missing") end')" + test "$(jq -r '.secret' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \ + "${REGIONAL_PLACEMENT_SECRET}" + test "$(jq -r '.version' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \ + "${REGIONAL_PLACEMENT_VERSION}" + test "$(gcloud secrets versions access "${REGIONAL_PLACEMENT_VERSION}" --project "${GCP_PROJECT_ID}" \ + --secret "${REGIONAL_PLACEMENT_SECRET}")" = "${REGIONAL_PLACEMENT_ENABLED}" + # Why: a served revision with no warm-instance floor still passes health and digest + # checks while quietly shrinking per-instance admission capacity. + SERVED_MIN_INSTANCES="$(gcloud run revisions describe "${REVISION}" \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")" + if [[ "${SERVED_MIN_INSTANCES:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then + echo "served revision ${REVISION} holds ${SERVED_MIN_INSTANCES:-0} minimum instances, expected at least ${DIRECTOR_MIN_INSTANCES}" >&2 + exit 1 + fi + SERVED_MAX_INSTANCES="$(gcloud run revisions describe "${REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \ + --format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")" + test "${SERVED_MAX_INSTANCES}" = "${DIRECTOR_MAX_INSTANCES}" + SERVICE_URL="$(jq -r '.status.url' <<< "${SERVICE_JSON}")" + node dev/scripts/smoke-relay.mjs "${SERVICE_URL}" diff --git a/.github/workflows/cloud-deploy-relay-production-multi-target.yml b/.github/workflows/cloud-deploy-relay-production-multi-target.yml new file mode 100644 index 00000000000..f6f56d93cf6 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-multi-target.yml @@ -0,0 +1,504 @@ +name: Deploy Relay Production Multi-Target + +on: + workflow_dispatch: + inputs: + source-cell-id: + description: Existing Terraform source cell ID + required: true + type: string + target-cell-ids: + description: Comma-separated distinct Terraform target cell IDs + required: true + type: string + general-cell-ids: + description: Comma-separated proven cells that remain eligible for ordinary placement + required: false + type: string + unobserved-connection-bound: + description: Exact worst-case unobserved connection bound proven by the passing load gate + required: false + type: string + failed-target-cell-id: + description: Registered failed target to fence and supersede + required: false + type: string + replacement-target-cell-id: + description: Healthy replacement for registered failed target + required: false + type: string + mode: + description: Preflight/audit are read-only; other modes mutate production + required: true + default: preflight + type: choice + options: + - audit + - preflight + - cutover-admission + - add-migration-cells + - promote-general-cell + - retire-migration-cell + - execute + - recover-forward + - fence-source + - supersede-target + confirmation: + description: Enter CUTOVER_SELECTOR, ADD_MIGRATION_CELLS, PROMOTE_GENERAL_CELL, RETIRE_MIGRATION_CELL, EVACUATE_MULTI, RECOVER_FORWARD, or FENCE_SOURCE + required: false + type: string + selector-attempt-id: + description: Exact durable selector attempt ID for admission mutations + required: false + type: string + monitor-run-id: + description: Successful fresh dry-run monitor workflow run ID + required: false + type: string + monitor-run-attempt: + description: Exact dry-run monitor workflow attempt + required: false + type: string + broker-operation-id: + description: Stable durable broker operation ID for target supersession + required: false + type: string + completed-fence-attempt-id: + description: Exact older completed fence attempt to recover without replay + required: false + type: string + completed-fence-commit: + description: Exact older fence commit bound to the completed attempt + required: false + type: string + completed-fence-operation: + description: Exact DONE Compute resize operation to adopt + required: false + type: string + completed-fence-state-serial: + description: Exact Terraform serial before the completed fence + required: false + type: string + completed-fence-plan-generation: + description: Exact saved-plan object generation + required: false + type: string + completed-fence-state-generation: + description: Exact current Terraform state object generation + required: false + type: string + completed-fence-state-sha256: + description: Exact current Terraform state object SHA-256 + required: false + type: string + expected-lease-generation: + description: Exact live lease generation authorized for conditional takeover + required: false + type: string + expected-lease-operation-id: + description: Exact live lease operation ID authorized for takeover + required: false + type: string + expected-lease-request-digest: + description: Exact live lease request digest authorized for takeover + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + deploy: + if: >- + ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && + github.ref == 'refs/heads/main' && + vars.PRODUCTION_GCP_REGION != '' && + (inputs.mode == 'supersede-target' || + (inputs.mode != 'supersede-target' && + vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER != '' && + vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT != '')) }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + DIRECTOR_ORIGIN: https://relay.onorca.dev + ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain + SOURCE_CELL_ID: ${{ inputs.source-cell-id }} + TARGET_CELL_IDS: ${{ inputs.target-cell-ids }} + GENERAL_CELL_IDS: ${{ inputs.general-cell-ids }} + UNOBSERVED_CONNECTION_BOUND: ${{ inputs.unobserved-connection-bound }} + FAILED_TARGET_CELL_ID: ${{ inputs.failed-target-cell-id }} + REPLACEMENT_TARGET_CELL_ID: ${{ inputs.replacement-target-cell-id }} + DEPLOY_MODE: ${{ inputs.mode }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }} + BROKER_OPERATION_ID: ${{ inputs.broker-operation-id }} + COMPLETED_FENCE_ATTEMPT_ID: ${{ inputs.completed-fence-attempt-id }} + COMPLETED_FENCE_COMMIT: ${{ inputs.completed-fence-commit }} + COMPLETED_FENCE_OPERATION: ${{ inputs.completed-fence-operation }} + COMPLETED_FENCE_STATE_SERIAL: ${{ inputs.completed-fence-state-serial }} + COMPLETED_FENCE_PLAN_GENERATION: ${{ inputs.completed-fence-plan-generation }} + COMPLETED_FENCE_STATE_GENERATION: ${{ inputs.completed-fence-state-generation }} + COMPLETED_FENCE_STATE_SHA256: ${{ inputs.completed-fence-state-sha256 }} + EXPECTED_LEASE_GENERATION: ${{ inputs.expected-lease-generation }} + EXPECTED_LEASE_OPERATION_ID: ${{ inputs.expected-lease-operation-id }} + EXPECTED_LEASE_REQUEST_DIGEST: ${{ inputs.expected-lease-request-digest }} + steps: + - uses: actions/checkout@v4 + + - name: Require private fence-broker environment + if: >- + ${{ inputs.mode == 'fence-source' || + inputs.mode == 'supersede-target' }} + env: + FENCE_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }} + FENCE_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }} + FENCE_BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }} + run: | + test -n "${FENCE_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${FENCE_SERVICE_ACCOUNT}" + test -n "${FENCE_BROKER_URI}" + + - name: Reject direct-runner Terraform fence aborts + if: ${{ inputs.mode == 'abort-fence-source' }} + run: | + echo "Terraform fence aborts require a reviewed private-broker recovery path." >&2 + exit 1 + + - name: Require fresh dry-run evidence reference + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + + - name: Download private dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - uses: pnpm/action-setup@v4 + with: + package_json_file: cloud/package.json + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - name: Verify dry-run artifact before cloud authentication + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-restore \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run + + - name: Reject previously consumed dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + env: + GH_TOKEN: ${{ github.token }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api \ + "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = "0" + + - id: google-auth + if: ${{ inputs.mode != 'supersede-target' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + + - name: Require explicit mutation confirmation + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if [[ "${DEPLOY_MODE}" = "cutover-admission" ]]; then + test "${CONFIRMATION}" = "CUTOVER_SELECTOR" + elif [[ "${DEPLOY_MODE}" = "add-migration-cells" ]]; then + test "${CONFIRMATION}" = "ADD_MIGRATION_CELLS" + elif [[ "${DEPLOY_MODE}" = "promote-general-cell" ]]; then + test "${CONFIRMATION}" = "PROMOTE_GENERAL_CELL" + elif [[ "${DEPLOY_MODE}" = "retire-migration-cell" ]]; then + test "${CONFIRMATION}" = "RETIRE_MIGRATION_CELL" + elif [[ "${DEPLOY_MODE}" = "execute" ]]; then + test "${CONFIRMATION}" = "EVACUATE_MULTI" + elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then + test "${CONFIRMATION}" = "RECOVER_FORWARD" + elif [[ "${DEPLOY_MODE}" = "fence-source" ]]; then + test "${CONFIRMATION}" = "FENCE_SOURCE" + elif [[ "${DEPLOY_MODE}" = "supersede-target" ]]; then + test "${CONFIRMATION}" = "SUPERSEDE_TARGET" + elif [[ "${DEPLOY_MODE}" = "abort-fence-source" ]]; then + test "${CONFIRMATION}" = "ABORT_FENCE" + else + test "${CONFIRMATION}" = "FENCE_SOURCE" + fi + + - name: Require exact source-fence broker contract + if: ${{ inputs.mode == 'fence-source' }} + run: | + test "${SOURCE_CELL_ID}" = "production-gce-c3" + test "${TARGET_CELL_IDS}" = "production-gce-c7,production-gce-c8,production-gce-c10,production-gce-c13,production-gce-c17,production-gce-c18" + [[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then + [[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]] + test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}" + [[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]] + else + test -z "${EXPECTED_LEASE_OPERATION_ID}" + test -z "${EXPECTED_LEASE_REQUEST_DIGEST}" + fi + + - name: Require exact broker cell contract + if: ${{ inputs.mode == 'supersede-target' }} + run: | + test "${SOURCE_CELL_ID}" = "production-gce-c3" + test "${FAILED_TARGET_CELL_ID}" = "production-gce-c12" + test "${REPLACEMENT_TARGET_CELL_ID}" = "production-gce-c13" + test "${TARGET_CELL_IDS}" = "production-gce-c12,production-gce-c13" + if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then + [[ "${COMPLETED_FENCE_ATTEMPT_ID}" =~ ^[0-9a-f-]{36}$ ]] + [[ "${COMPLETED_FENCE_COMMIT}" =~ ^[0-9a-f]{40}$ ]] + [[ "${COMPLETED_FENCE_OPERATION}" =~ ^[A-Za-z0-9._-]{1,256}$ ]] + [[ "${COMPLETED_FENCE_STATE_SERIAL}" =~ ^[0-9]+$ ]] + [[ "${COMPLETED_FENCE_PLAN_GENERATION}" =~ ^[1-9][0-9]*$ ]] + [[ "${COMPLETED_FENCE_STATE_GENERATION}" =~ ^[1-9][0-9]*$ ]] + [[ "${COMPLETED_FENCE_STATE_SHA256}" =~ ^[0-9a-f]{64}$ ]] + test -n "${EXPECTED_LEASE_GENERATION}" + fi + if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then + [[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]] + test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}" + [[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]] + else + test -z "${EXPECTED_LEASE_OPERATION_ID}" + test -z "${EXPECTED_LEASE_REQUEST_DIGEST}" + fi + + - name: Read reviewed Terraform topology + if: ${{ inputs.mode != 'supersede-target' }} + run: | + node dev/scripts/infra.mjs init --env production + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json" + RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)" + DIRECTOR_MIN_INSTANCES="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars <<< 'var.relay_min_instances')" + [[ "${DIRECTOR_MIN_INSTANCES}" =~ ^[1-9][0-9]*$ ]] + echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}" + echo "DIRECTOR_MIN_INSTANCES=${DIRECTOR_MIN_INSTANCES}" >> "${GITHUB_ENV}" + + - name: Verify fresh dry-run evidence against live selector + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + SCOPED_RECOVERY_ARGS=() + if [[ ("${DEPLOY_MODE}" = "execute" || + "${DEPLOY_MODE}" = "recover-forward") && + "${SOURCE_CELL_ID}" = "production-gce-c12" ]]; then + SCOPED_RECOVERY_ARGS=( + --scoped-recovery-source-cell-id + production-gce-c3 + ) + fi + node dev/scripts/relay-monitor-evidence.mjs verify-mutation \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run \ + --mutation-mode "${DEPLOY_MODE}" \ + --source-cell-id "${SOURCE_CELL_ID}" \ + "${SCOPED_RECOVERY_ARGS[@]}" \ + --director-origin "${DIRECTOR_ORIGIN}" + + - name: Recheck all live safety signals + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + pnpm incident:relay-preflight -- \ + --state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" + + - name: Create single-use dry-run marker + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}" + + - name: Consume dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }} + uses: actions/upload-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + - id: google-fence-broker-auth + if: >- + ${{ inputs.mode == 'fence-source' || + inputs.mode == 'supersede-target' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }} + id_token_include_email: true + + - name: Invoke private target-supersession broker + if: ${{ inputs.mode == 'supersede-target' }} + env: + BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }} + BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }} + run: | + [[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then + REQUEST="$(jq -cn \ + --arg operationId "${BROKER_OPERATION_ID}" \ + --arg fenceCommit "${GITHUB_SHA}" \ + --arg attemptId "${COMPLETED_FENCE_ATTEMPT_ID}" \ + --arg completedCommit "${COMPLETED_FENCE_COMMIT}" \ + --arg gceOperation "${COMPLETED_FENCE_OPERATION}" \ + --arg stateSerial "${COMPLETED_FENCE_STATE_SERIAL}" \ + --arg planGeneration "${COMPLETED_FENCE_PLAN_GENERATION}" \ + --arg stateGeneration "${COMPLETED_FENCE_STATE_GENERATION}" \ + --arg stateSha256 "${COMPLETED_FENCE_STATE_SHA256}" \ + --arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \ + --arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \ + --arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \ + '{v:1,operationId:$operationId,fenceCommit:$fenceCommit, + completedFenceRecovery:{attemptId:$attemptId,fenceCommit:$completedCommit, + gceOperation:$gceOperation,terraformStateSerial:($stateSerial|tonumber), + planObjectGeneration:$planGeneration, + terraformStateObjectGeneration:$stateGeneration, + terraformStateObjectSha256:$stateSha256}, + expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId, + requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')" + elif [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then + REQUEST="$(jq -cn \ + --arg operationId "${BROKER_OPERATION_ID}" \ + --arg fenceCommit "${GITHUB_SHA}" \ + --arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \ + --arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \ + --arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \ + '{v:1,operationId:$operationId,fenceCommit:$fenceCommit, + expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId, + requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')" + else + REQUEST="$(jq -cn \ + --arg operationId "${BROKER_OPERATION_ID}" \ + --arg fenceCommit "${GITHUB_SHA}" \ + '{v:1,operationId:$operationId,fenceCommit:$fenceCommit,confirmation:"SUPERSEDE_TARGET"}')" + fi + curl --fail-with-body --max-time 1790 \ + --request POST "${BROKER_URI}/v1/supersede-target" \ + --header "Authorization: Bearer ${BROKER_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "${REQUEST}" + + - name: Invoke private source-fence broker + if: ${{ inputs.mode == 'fence-source' }} + env: + BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }} + BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }} + run: | + if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then + REQUEST="$(jq -cn \ + --arg operationId "${BROKER_OPERATION_ID}" \ + --arg fenceCommit "${GITHUB_SHA}" \ + --arg targetCellIds "${TARGET_CELL_IDS}" \ + --arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \ + --arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \ + --arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \ + '{v:1,operationId:$operationId,fenceCommit:$fenceCommit, + targetCellIds:($targetCellIds|split(",")), + expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId, + requestDigest:$leaseRequestDigest},confirmation:"FENCE_SOURCE"}')" + else + REQUEST="$(jq -cn \ + --arg operationId "${BROKER_OPERATION_ID}" \ + --arg fenceCommit "${GITHUB_SHA}" \ + --arg targetCellIds "${TARGET_CELL_IDS}" \ + '{v:1,operationId:$operationId,fenceCommit:$fenceCommit, + targetCellIds:($targetCellIds|split(",")),confirmation:"FENCE_SOURCE"}')" + fi + curl --fail-with-body --max-time 1790 \ + --request POST "${BROKER_URI}/v1/fence-source" \ + --header "Authorization: Bearer ${BROKER_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "${REQUEST}" + + - name: Preflight or run multi-target evacuation + if: >- + ${{ inputs.mode != 'fence-source' && + inputs.mode != 'supersede-target' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/deploy-relay-gce-multi-target.mjs \ + --project "${GCP_PROJECT_ID}" \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --admin-audience "${ADMIN_AUDIENCE}" \ + --topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \ + --source-cell-id "${SOURCE_CELL_ID}" \ + --target-cell-ids "${TARGET_CELL_IDS}" \ + --general-cell-ids "${GENERAL_CELL_IDS}" \ + --unobserved-connection-bound "${UNOBSERVED_CONNECTION_BOUND}" \ + --director-region "${{ vars.PRODUCTION_GCP_REGION }}" \ + --director-service "orca-cloud-relay" \ + --director-min-instances "${DIRECTOR_MIN_INSTANCES}" \ + --selector-attempt-id "${SELECTOR_ATTEMPT_ID}" \ + --failed-target-cell-id "${FAILED_TARGET_CELL_ID}" \ + --replacement-target-cell-id "${REPLACEMENT_TARGET_CELL_ID}" \ + --runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \ + --environment production \ + --fence-commit "${GITHUB_SHA}" \ + --terraform-dir infra/terraform \ + --terraform-var-file environments/production.tfvars \ + --mode "${DEPLOY_MODE}" \ + --connection-ceiling 1000 \ + --minimum-lease-remaining-ms 600000 diff --git a/.github/workflows/cloud-deploy-relay-production-same-cap-job.yml b/.github/workflows/cloud-deploy-relay-production-same-cap-job.yml new file mode 100644 index 00000000000..2821bc26660 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-same-cap-job.yml @@ -0,0 +1,644 @@ +name: Deploy Relay Production Same-Cap Job + +on: + workflow_call: + inputs: + mode: { required: true, type: string } + target-cell-id: { required: true, type: string } + target-image-digest: { required: true, type: string } + rollback-image-digest: { required: true, type: string } + target-rehome-protocol: { required: true, type: string } + rollback-rehome-protocol: { required: true, type: string } + expected-selector-generation: { required: true, type: string } + expected-existing-only-cells: { required: true, type: string } + expected-migration-only-cells: { required: true, type: string } + expected-general-cells: { required: true, type: string } + expected-rehome-generation: { required: true, type: string } + monitor-run-id: { required: true, type: string } + monitor-run-attempt: { required: true, type: string } + wave-index: { required: true, type: string } + +permissions: + actions: read + contents: read + id-token: write + +defaults: + run: + working-directory: cloud + +jobs: + rollout: + if: ${{ github.ref == 'refs/heads/main' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 75 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + DIRECTOR_ORIGIN: https://relay.onorca.dev + IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay + TARGET_CELL_ID: ${{ inputs.target-cell-id }} + DEPLOY_MODE: ${{ inputs.mode }} + TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }} + ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }} + TARGET_REHOME_PROTOCOL: ${{ inputs.target-rehome-protocol }} + ROLLBACK_REHOME_PROTOCOL: ${{ inputs.rollback-rehome-protocol }} + EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }} + EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }} + EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }} + EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }} + EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }} + WAVE_INDEX: ${{ inputs.wave-index }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence + steps: + - name: Require exact reusable-workflow configuration + env: + DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + CAPACITY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + run: | + [[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]] + [[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + [[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + test "${TARGET_IMAGE_DIGEST}" != "${ROLLBACK_IMAGE_DIGEST}" + [[ "${TARGET_REHOME_PROTOCOL}" =~ ^[01]$ ]] + [[ "${ROLLBACK_REHOME_PROTOCOL}" =~ ^[01]$ ]] + [[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + [[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + [[ "${WAVE_INDEX}" =~ ^[0-3]$ ]] + if test "${DEPLOY_MODE}" = verify; then + EFFECTIVE_SELECTOR_GENERATION="${EXPECTED_SELECTOR_GENERATION}" + else + EFFECTIVE_SELECTOR_GENERATION="$((EXPECTED_SELECTOR_GENERATION + (2 * WAVE_INDEX)))" + fi + echo "EFFECTIVE_SELECTOR_GENERATION=${EFFECTIVE_SELECTOR_GENERATION}" >> "${GITHUB_ENV}" + if test "${DEPLOY_MODE}" != verify && test "${GITHUB_RUN_ATTEMPT}" != 1; then + echo "mutations are single-dispatch: re-runs replay aged evidence," >&2 + echo "so recover each remaining cell with its own fresh monitor" >&2 + echo "dry-run and canary-apply dispatch instead" >&2 + exit 1 + fi + test -n "${GCP_REGION}" + test -n "${DEPLOY_WIF}" + test -n "${DEPLOY_SERVICE_ACCOUNT}" + test -n "${CAPACITY_WIF}" + test -n "${CAPACITY_SERVICE_ACCOUNT}" + test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" + + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: { package_json_file: cloud/package.json } + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - uses: hashicorp/setup-terraform@v3 + with: { terraform_wrapper: false } + + - name: Require fresh aggregate monitor evidence reference + if: ${{ inputs.mode != 'verify' }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + + - name: Download private aggregate monitor evidence + if: ${{ inputs.mode != 'verify' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ github.workspace }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - name: Verify monitor evidence provenance + if: ${{ inputs.mode != 'verify' }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-authority \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run \ + --required-migration-policy strict \ + --wave-index "${WAVE_INDEX}" + + - name: Download this wave's single-use safety authority + if: ${{ inputs.mode != 'verify' }} + uses: actions/download-artifact@v4 + with: + name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-same-cap-monitor-authority + github-token: ${{ github.token }} + run-id: ${{ github.run_id }} + + - name: Require safety evidence consumed by this workflow + if: ${{ inputs.mode != 'verify' }} + run: | + # Mutations are single-dispatch: a fresh dispatch cannot resume a + # partial batch (the canary authority binds the batch-entry selector + # generation), so each remaining cell is recovered by its own fresh + # monitor dry-run and canary-apply dispatch, never by re-running + # aged evidence. + test "${GITHUB_RUN_ATTEMPT}" = 1 + MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + test "$(< "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}")" = \ + "${GITHUB_RUN_ID}" + + - id: deploy-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + release: 'false' + + - name: Recheck aggregate SQL, pool, reconnect, migration, and selector safety + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + RETRY_ARGS=() + if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi + pnpm incident:relay-preflight -- \ + --state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" \ + --wave-index "${WAVE_INDEX}" "${RETRY_ARGS[@]}" + + - name: Require durable rehome disabled and exact selector + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode inspect \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${EFFECTIVE_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_REHOME_GENERATION}" \ + | jq -e '.control.enabled == false' >/dev/null + + - name: Initialize the exact production backend + run: node dev/scripts/infra.mjs init --env production + + - name: Resolve immutable same-cap cell configuration + shell: bash + run: | + TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}" + case "${TARGET_HOSTNAME}" in + c7|c8|c9|c10|c13|c14|c15|c16|c19|c20|c21|c22|c23|c24|c25|c26) + EXPECTED_HARD_CAP=1000 + EXPECTED_REGION=us-central1 + ;; + c27|c28|c29) + EXPECTED_HARD_CAP=3000 + EXPECTED_REGION=asia-east2 + ;; + *) exit 1 ;; + esac + EXPECTED_UNOBSERVED_BOUND=60 + CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev" + CELLS_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')" + SOURCE_CELLS="$(terraform -chdir=infra/terraform console \ + -var-file=environments/production.tfvars -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_region_rehome_source_cell_ids)' | jq -er '.')" + if test "${EXPECTED_REGION}" = us-central1; then + jq -e --arg cell "${TARGET_CELL_ID}" 'index($cell) != null' \ + <<< "${SOURCE_CELLS}" >/dev/null + fi + CURRENT_SHAPE="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${CELLS_JSON}")" + test "$(jq -r '.connection_hard_cap' <<< "${CURRENT_SHAPE}")" = "${EXPECTED_HARD_CAP}" + test "$(jq -r '.connection_unobserved_bound' <<< "${CURRENT_SHAPE}")" = \ + "${EXPECTED_UNOBSERVED_BOUND}" + TARGET_ZONE="$(jq -r '.zone' <<< "${CURRENT_SHAPE}")" + MIG_NAME="orca-cloud-relay-gce-${TARGET_HOSTNAME}" + if test "${DEPLOY_MODE}" = rollback; then + DESIRED_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}" + CURRENT_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}" + DESIRED_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}" + CURRENT_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}" + else + DESIRED_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}" + CURRENT_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}" + DESIRED_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}" + CURRENT_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}" + fi + DESIRED_IMAGE="${IMAGE_REPOSITORY}@${DESIRED_IMAGE_DIGEST}" + OVERRIDE_CELLS_JSON="$(jq -ce --arg cell "${TARGET_CELL_ID}" \ + --arg image "${DESIRED_IMAGE}" '.[$cell].image = $image' <<< "${CELLS_JSON}")" + jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \ + '{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-same-cap.tfvars.json" + SERVED_DIGEST="$(gcloud artifacts docker images describe "${DESIRED_IMAGE}" \ + --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')" + test "${SERVED_DIGEST}" = "${DESIRED_IMAGE_DIGEST}" + { + echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}" + echo "CELL_ORIGIN=${CELL_ORIGIN}" + echo "TARGET_ZONE=${TARGET_ZONE}" + echo "MIG_NAME=${MIG_NAME}" + echo "EXPECTED_HARD_CAP=${EXPECTED_HARD_CAP}" + echo "EXPECTED_UNOBSERVED_BOUND=${EXPECTED_UNOBSERVED_BOUND}" + echo "EXPECTED_REGION=${EXPECTED_REGION}" + echo "DESIRED_IMAGE=${DESIRED_IMAGE}" + echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}" + echo "CURRENT_IMAGE_DIGEST=${CURRENT_IMAGE_DIGEST}" + echo "DESIRED_REHOME_PROTOCOL=${DESIRED_REHOME_PROTOCOL}" + echo "CURRENT_REHOME_PROTOCOL=${CURRENT_REHOME_PROTOCOL}" + } >> "${GITHUB_ENV}" + + - name: Verify exact current generation, digest, cap, and rollback point + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + CURRENT_RUNTIME="$(curl --fail-with-body --max-time 30 \ + --request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')" + # A rollback that failed between template apply and admission restore + # leaves the cell already on the rollback image; resume from that + # state instead of demanding the pre-rollback predecessor. + LIVE_IMAGE_DIGEST="$(jq -r '.imageDigest' <<< "${CURRENT_RUNTIME}")" + if test "${DEPLOY_MODE}" = rollback \ + && test "${LIVE_IMAGE_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"; then + ROLLBACK_RESUME=true + PREDECESSOR_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}" + PREDECESSOR_REHOME_PROTOCOL="${DESIRED_REHOME_PROTOCOL}" + else + ROLLBACK_RESUME=false + PREDECESSOR_IMAGE_DIGEST="${CURRENT_IMAGE_DIGEST}" + PREDECESSOR_REHOME_PROTOCOL="${CURRENT_REHOME_PROTOCOL}" + fi + RESTORED_MIGRATION_CELLS="$(jq -rn \ + --arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \ + --arg target "${TARGET_CELL_ID}" \ + '$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')" + RESTORED_GENERAL_CELLS="$(jq -rn \ + --arg value "${EXPECTED_GENERAL_CELLS/none/}" \ + --arg target "${TARGET_CELL_ID}" \ + '$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')" + test -n "${RESTORED_MIGRATION_CELLS}" || RESTORED_MIGRATION_CELLS=none + test -n "${RESTORED_GENERAL_CELLS}" || RESTORED_GENERAL_CELLS=none + ISOLATED_MIGRATION_CELLS="$(jq -rn \ + --arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \ + --arg target "${TARGET_CELL_ID}" \ + '$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')" + ISOLATED_GENERAL_CELLS="$(jq -rn \ + --arg value "${EXPECTED_GENERAL_CELLS/none/}" \ + --arg target "${TARGET_CELL_ID}" \ + '$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')" + test -n "${ISOLATED_MIGRATION_CELLS}" || ISOLATED_MIGRATION_CELLS=none + test -n "${ISOLATED_GENERAL_CELLS}" || ISOLATED_GENERAL_CELLS=none + { + echo "ROLLBACK_RESUME=${ROLLBACK_RESUME}" + # The failsafe consumes these; deriving them here keeps them + # defined for a failure in any later step. + echo "ISOLATED_MIGRATION_CELLS=${ISOLATED_MIGRATION_CELLS}" + echo "ISOLATED_GENERAL_CELLS=${ISOLATED_GENERAL_CELLS}" + # No restart happens on resume, so isolate below is skipped and + # cannot advance the selector generation. + echo "SELECTOR_GENERATION_AFTER_ISOLATE=${EFFECTIVE_SELECTOR_GENERATION}" + # A failed-canary rollback enters with the target migration-only, + # so the restore inspect cannot reuse the entry membership inputs. + echo "RESTORED_MIGRATION_CELLS=${RESTORED_MIGRATION_CELLS}" + echo "RESTORED_GENERAL_CELLS=${RESTORED_GENERAL_CELLS}" + } >> "${GITHUB_ENV}" + if ! jq -e --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \ + --arg digest "${PREDECESSOR_IMAGE_DIGEST}" \ + --arg region "${EXPECTED_REGION}" \ + --argjson hardCap "${EXPECTED_HARD_CAP}" \ + --argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \ + --argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \ + --argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \ + && test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \ + '.role == "cell" and .cellId == $cell and .cellUrl == $origin and + (.region == $region or + ($region == "us-central1" and $protocol == 0 and .region == null)) and + .imageDigest == $digest and + .connectionCapacity.hardCap == $hardCap and + .connectionCapacity.unobservedBound == $unobservedBound and + (.draining == false or $drainingOk) and + (.regionalRehomeProtocol // 0) == $protocol' <<< "${CURRENT_RUNTIME}" >/dev/null + then + jq -r --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \ + --arg digest "${PREDECESSOR_IMAGE_DIGEST}" \ + --arg region "${EXPECTED_REGION}" \ + --argjson hardCap "${EXPECTED_HARD_CAP}" \ + --argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \ + --argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \ + --argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \ + && test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \ + '[ + if .role != "cell" then "role" else empty end, + if .cellId != $cell then "cellId" else empty end, + if .cellUrl != $origin then "cellUrl" else empty end, + if (.region != $region and + ($region != "us-central1" or $protocol != 0 or .region != null)) + then "region" else empty end, + if .imageDigest != $digest then "imageDigest" else empty end, + if .connectionCapacity.hardCap != $hardCap then "hardCap" else empty end, + if .connectionCapacity.unobservedBound != $unobservedBound then "unobservedBound" else empty end, + if (.draining != false and ($drainingOk | not)) then "draining" else empty end, + if (.regionalRehomeProtocol // 0) != $protocol then "regionalRehomeProtocol" else empty end + ] | "runtime predecessor mismatch fields=" + join(",")' \ + <<< "${CURRENT_RUNTIME}" >&2 + exit 1 + fi + # The exact legacy digest binds omitted pre-region fields to US and protocol 0. + jq -r '[ + if .region == null then "region" else empty end, + if .regionalRehomeProtocol == null then "regionalRehomeProtocol" else empty end + ] | if length > 0 then "runtime predecessor normalized legacy fields=" + join(",") else empty end' \ + <<< "${CURRENT_RUNTIME}" + CURRENT_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \ + --request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + SOURCE_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \ + <<< "${CURRENT_DIRECTOR_STATUS}")" + if test "${ROLLBACK_RESUME}" = true && ! jq -e \ + '.status.admissionState == "migration-only"' \ + <<< "${CURRENT_DIRECTOR_STATUS}" >/dev/null; then + echo 'resume requires the isolated migration-only cell a failed rollback leaves' >&2 + exit 1 + fi + [[ "${SOURCE_INCARNATION}" =~ ^[0-9a-f-]{36}$ ]] + echo "SOURCE_INCARNATION=${SOURCE_INCARNATION}" >> "${GITHUB_ENV}" + # Rollback is the documented recovery from a failed canary, which + # leaves the cell migration-only (and possibly still marked + # draining); apply and verify still require a pristine general cell. + if test "${DEPLOY_MODE}" = rollback; then + PRECHECK_ADMISSION=general-or-migration-only + PRECHECK_DRAINING=either + else + PRECHECK_ADMISSION=general + PRECHECK_DRAINING=forbidden + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh --admission "${PRECHECK_ADMISSION}" \ + --draining "${PRECHECK_DRAINING}" --activity allowed \ + --expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}" + + - name: Finish read-only verification + if: ${{ inputs.mode == 'verify' }} + run: echo 'Exact same-cap rollback point verified.' + + - name: Reversibly isolate and drain only the selected cell + if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }} + run: | + echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + # A cell isolated by a failed canary is already migration-only, so + # isolate is a no-op there that does not advance the selector; the + # result's generation is authoritative either way. + ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --mode isolate)" + echo "${ISOLATE_RESULT}" + ISOLATE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")" + echo "SELECTOR_GENERATION_AFTER_ISOLATE=${ISOLATE_GENERATION}" >> "${GITHUB_ENV}" + node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --mode drain + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat either --admission migration-only --draining required \ + --activity restart-safe --expected-image-digests "${CURRENT_IMAGE_DIGEST}" \ + --timeout-ms 900000 + + - id: capacity-auth + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + + - name: Require converged Terraform state and a stable MIG on resume + if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME == 'true' }} + shell: bash + env: + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + run: | + # Zero resource changes prove the prior run's apply completed and no + # restart will follow, keeping the incarnation check honest. Root + # outputs may lag a targeted apply, so judge resource_changes only. + terraform -chdir=infra/terraform plan \ + -var-file=environments/production.tfvars \ + -var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \ + -var manage_artifact_dns=false \ + "-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + "-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + -out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan" + if ! terraform -chdir=infra/terraform show -json \ + "${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \ + | jq -e '[.resource_changes[]? + | select(.change.actions | any(. != "no-op" and . != "read"))] + | length == 0' >/dev/null + then + # An apply that failed before its template apply also resumes here + # (the cell still serves the rollback image), and repo drift since + # the cell's last roll (for example newly added rehome trust + # config) then legitimately replaces the template. Nothing is + # applied on resume either way, so accept exactly the drift the + # reviewed validator would let a real apply ship for the image the + # cell already serves: the template leaves and re-enters the + # rollback image, as exactly the template-and-MIG change pair. + terraform -chdir=infra/terraform show -json \ + "${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \ + | jq -r '"resume found unconverged resources: " + + ([.resource_changes[]? + | select(.change.actions | any(. != "no-op" and . != "read")) + | .address] | join(","))' + echo 'requiring reviewed rollback-image drift' + terraform -chdir=infra/terraform show -json \ + "${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --image "${DESIRED_IMAGE}" \ + --rollback-image "${DESIRED_IMAGE}" \ + --rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \ + --rehome-audience https://relay.onorca.dev/v1/admin/host-drain \ + | jq -e '.changes == 2' >/dev/null + fi + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + + - name: Apply only the selected same-cap template and MIG + if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }} + shell: bash + env: + CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + run: | + terraform -chdir=infra/terraform plan \ + -var-file=environments/production.tfvars \ + -var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \ + -var manage_artifact_dns=false \ + "-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + "-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \ + -out="${RUNNER_TEMP}/relay-same-cap.tfplan" + terraform -chdir=infra/terraform show -json "${RUNNER_TEMP}/relay-same-cap.tfplan" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" --image "${DESIRED_IMAGE}" \ + --rollback-image "${IMAGE_REPOSITORY}@${CURRENT_IMAGE_DIGEST}" \ + --rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \ + --rehome-audience https://relay.onorca.dev/v1/admin/host-drain + terraform -chdir=infra/terraform apply -auto-approve \ + "${RUNNER_TEMP}/relay-same-cap.tfplan" + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + + - id: post-auth + if: ${{ inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Verify new incarnation, exact image, protocol, and durable safety + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh --admission migration-only --draining forbidden \ + --activity allowed --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \ + --regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" --timeout-ms 900000 + TARGET_RUNTIME="$(curl --fail-with-body --max-time 30 \ + --request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')" + jq -e --arg digest "${DESIRED_IMAGE_DIGEST}" \ + --argjson protocol "${DESIRED_REHOME_PROTOCOL}" \ + '.imageDigest == $digest and (.regionalRehomeProtocol // 0) == $protocol' \ + <<< "${TARGET_RUNTIME}" >/dev/null + TARGET_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \ + --request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + TARGET_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \ + <<< "${TARGET_DIRECTOR_STATUS}")" + if test "${ROLLBACK_RESUME}" = true; then + echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + # No restart happened; the incarnation legitimately stays put. + test "${TARGET_INCARNATION}" = "${SOURCE_INCARNATION}" + else + test "${TARGET_INCARNATION}" != "${SOURCE_INCARNATION}" + fi + echo "TARGET_INCARNATION=${TARGET_INCARNATION}" >> "${GITHUB_ENV}" + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode inspect --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${SELECTOR_GENERATION_AFTER_ISOLATE}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \ + --expected-general-cells "${ISOLATED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_REHOME_GENERATION}" \ + | jq -e '.control.enabled == false' >/dev/null + + - name: Prove exact per-host trust and idempotent no-neighbor behavior + if: ${{ inputs.mode != 'verify' && ((inputs.mode == 'rollback' && inputs.rollback-rehome-protocol == '1') || (inputs.mode != 'rollback' && inputs.target-rehome-protocol == '1')) }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }} + run: | + node dev/scripts/probe-relay-rehome-trust.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-id "${TARGET_CELL_ID}" \ + --cell-incarnation "${TARGET_INCARNATION}" + + - name: Restore only the verified selected cell to general admission + if: ${{ inputs.mode != 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }} + run: | + echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + ACTIVATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --mode activate)" + echo "${ACTIVATE_RESULT}" + SELECTOR_GENERATION_AFTER_ACTIVATE="$(jq -er '.generation' \ + <<< "${ACTIVATE_RESULT}")" + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh --admission general --draining forbidden --activity allowed \ + --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \ + --regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode inspect --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${SELECTOR_GENERATION_AFTER_ACTIVATE}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${RESTORED_MIGRATION_CELLS}" \ + --expected-general-cells "${RESTORED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_REHOME_GENERATION}" \ + | jq -e '.control.enabled == false' >/dev/null + + - id: cleanup-auth + if: ${{ failure() && inputs.mode != 'verify' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Keep a failed cell isolated and rehome disabled + if: ${{ failure() && inputs.mode != 'verify' }} + continue-on-error: true + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }} + run: | + test "${MUTATION_STARTED:-false}" = true || exit 0 + ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --mode isolate)" + echo "${ISOLATE_RESULT}" + # The isolate result carries the authoritative post-isolate generation; + # fixed offsets are wrong whenever an earlier isolate was a no-op. + FAILSAFE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")" + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode inspect --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${FAILSAFE_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \ + --expected-general-cells "${ISOLATED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_REHOME_GENERATION}" diff --git a/.github/workflows/cloud-deploy-relay-production-same-cap.yml b/.github/workflows/cloud-deploy-relay-production-same-cap.yml new file mode 100644 index 00000000000..1994966d083 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production-same-cap.yml @@ -0,0 +1,310 @@ +name: Deploy Relay Production Same-Cap + +on: + workflow_dispatch: + inputs: + mode: + description: Verify, roll one canary, roll a bounded batch, or roll back + required: true + default: verify + type: choice + options: [verify, canary-apply, batch-apply, rollback] + cell-ids: + description: Ordered comma-separated serving cells; one canary or two to four batch cells + required: true + type: string + target-image-digest: + description: Exact immutable compatibility image digest + required: true + type: string + rollback-image-digest: + description: Exact immutable currently serving rollback digest + required: true + type: string + target-rehome-protocol: + description: Exact target regional-rehome protocol + required: true + default: '1' + type: choice + options: ['0', '1'] + rollback-rehome-protocol: + description: Exact rollback regional-rehome protocol + required: true + default: '0' + type: choice + options: ['0', '1'] + expected-selector-generation: + description: Exact selector generation before the first cell + required: true + type: string + expected-existing-only-cells: + description: Exact existing-only membership, or none + required: true + type: string + expected-migration-only-cells: + description: Exact migration-only membership, or none + required: true + type: string + expected-general-cells: + description: Exact general membership, or none + required: true + type: string + expected-rehome-generation: + description: Exact durable regional-rehome control generation; it must be disabled + required: true + type: string + monitor-run-id: + description: Fresh successful aggregate dry-run monitor workflow run + required: false + type: string + monitor-run-attempt: + description: Exact monitor attempt + required: false + type: string + canary-run-id: + description: Successful same-commit canary run required for batch-apply + required: false + type: string + confirmation: + description: Exact digest-and-cell-bound mutation confirmation + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + gate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 10 + environment: production + outputs: + cells: ${{ steps.wave.outputs.cells }} + job-mode: ${{ steps.wave.outputs.job-mode }} + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: { node-version: 24 } + + - id: wave + env: + MODE: ${{ inputs.mode }} + CELL_IDS: ${{ inputs.cell-ids }} + TARGET_DIGEST: ${{ inputs.target-image-digest }} + ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }} + CONFIRMATION: ${{ inputs.confirmation }} + CANARY_RUN_ID: ${{ inputs.canary-run-id }} + run: | + CELLS="$(node dev/scripts/relay-production-same-cap-wave.mjs validate \ + --mode "${MODE}" --cell-ids "${CELL_IDS}" \ + --target-digest "${TARGET_DIGEST}" --rollback-digest "${ROLLBACK_DIGEST}" \ + --confirmation "${CONFIRMATION}" --canary-run-id "${CANARY_RUN_ID}")" + echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}" + if [[ "${MODE}" =~ ^(canary-apply|batch-apply)$ ]]; then + echo 'job-mode=apply' >> "${GITHUB_OUTPUT}" + else + echo "job-mode=${MODE}" >> "${GITHUB_OUTPUT}" + fi + + - name: Download exact prior canary authority + if: ${{ inputs.mode == 'batch-apply' }} + uses: actions/download-artifact@v4 + with: + name: relay-same-cap-canary-${{ inputs.canary-run-id }} + path: ${{ runner.temp }}/relay-same-cap-canary + github-token: ${{ github.token }} + run-id: ${{ inputs.canary-run-id }} + + - name: Verify canary authority against this batch + if: ${{ inputs.mode == 'batch-apply' }} + env: + CANARY_RUN_ID: ${{ inputs.canary-run-id }} + run: | + node dev/scripts/relay-production-same-cap-wave.mjs verify-canary \ + --file "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" \ + --commit-sha "${GITHUB_SHA}" --run-id "${CANARY_RUN_ID}" \ + --target-digest "${{ inputs.target-image-digest }}" \ + --rollback-digest "${{ inputs.rollback-image-digest }}" \ + --selector-generation "${{ inputs.expected-selector-generation }}" \ + --rehome-generation "${{ inputs.expected-rehome-generation }}" + + - name: Reject previously consumed aggregate safety evidence + if: ${{ inputs.mode != 'verify' }} + env: + GH_TOKEN: ${{ github.token }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = 0 + mkdir -p "${RUNNER_TEMP}/relay-same-cap-monitor-authority" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}" + + - name: Consume aggregate safety evidence for this exact wave + if: ${{ inputs.mode != 'verify' }} + uses: actions/upload-artifact@v4 + with: + name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-same-cap-monitor-authority/relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + cell_1: + needs: gate + uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml + with: + mode: ${{ needs.gate.outputs.job-mode }} + target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[0] }} + target-image-digest: ${{ inputs.target-image-digest }} + rollback-image-digest: ${{ inputs.rollback-image-digest }} + target-rehome-protocol: ${{ inputs.target-rehome-protocol }} + rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + expected-rehome-generation: ${{ inputs.expected-rehome-generation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + wave-index: '0' + secrets: inherit + + cell_2: + if: ${{ needs.cell_1.result == 'success' && fromJSON(needs.gate.outputs.cells)[1] != null }} + needs: [gate, cell_1] + uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml + with: + mode: ${{ needs.gate.outputs.job-mode }} + target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[1] }} + target-image-digest: ${{ inputs.target-image-digest }} + rollback-image-digest: ${{ inputs.rollback-image-digest }} + target-rehome-protocol: ${{ inputs.target-rehome-protocol }} + rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + expected-rehome-generation: ${{ inputs.expected-rehome-generation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + wave-index: '1' + secrets: inherit + + cell_3: + if: ${{ needs.cell_2.result == 'success' && fromJSON(needs.gate.outputs.cells)[2] != null }} + needs: [gate, cell_2] + uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml + with: + mode: ${{ needs.gate.outputs.job-mode }} + target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[2] }} + target-image-digest: ${{ inputs.target-image-digest }} + rollback-image-digest: ${{ inputs.rollback-image-digest }} + target-rehome-protocol: ${{ inputs.target-rehome-protocol }} + rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + expected-rehome-generation: ${{ inputs.expected-rehome-generation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + wave-index: '2' + secrets: inherit + + cell_4: + if: ${{ needs.cell_3.result == 'success' && fromJSON(needs.gate.outputs.cells)[3] != null }} + needs: [gate, cell_3] + uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml + with: + mode: ${{ needs.gate.outputs.job-mode }} + target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[3] }} + target-image-digest: ${{ inputs.target-image-digest }} + rollback-image-digest: ${{ inputs.rollback-image-digest }} + target-rehome-protocol: ${{ inputs.target-rehome-protocol }} + rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + expected-rehome-generation: ${{ inputs.expected-rehome-generation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + wave-index: '3' + secrets: inherit + + seal_canary: + if: ${{ inputs.mode == 'canary-apply' }} + needs: [gate, cell_1] + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: { node-version: 24 } + + - name: Seal exact successful canary authority + run: | + mkdir -p "${RUNNER_TEMP}/relay-same-cap-canary" + node dev/scripts/relay-production-same-cap-wave.mjs create-canary \ + --cell-id "${{ inputs.cell-ids }}" \ + --target-digest "${{ inputs.target-image-digest }}" \ + --rollback-digest "${{ inputs.rollback-image-digest }}" \ + --confirmation "${{ inputs.confirmation }}" \ + --commit-sha "${GITHUB_SHA}" --run-id "${GITHUB_RUN_ID}" \ + --selector-generation "${{ inputs.expected-selector-generation }}" \ + --rehome-generation "${{ inputs.expected-rehome-generation }}" \ + > "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" + + - uses: actions/upload-artifact@v4 + with: + name: relay-same-cap-canary-${{ github.run_id }} + path: ${{ runner.temp }}/relay-same-cap-canary/authority.json + retention-days: 30 + if-no-files-found: error + + # Every cell job re-enters the run's lease with release: 'false'; only this job frees it. + release_lease: + if: always() + needs: + - gate + - cell_1 + - cell_2 + - cell_3 + - cell_4 + - seal_canary + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 10 + environment: production + steps: + - uses: actions/checkout@v4 + + - uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + release: 'true' diff --git a/.github/workflows/cloud-deploy-relay-production.yml b/.github/workflows/cloud-deploy-relay-production.yml new file mode 100644 index 00000000000..6de990e4c90 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-production.yml @@ -0,0 +1,219 @@ +name: Deploy Relay Production Candidate + +on: + workflow_dispatch: + inputs: + source-cell-id: + description: Existing Terraform cell ID to evacuate + required: true + type: string + target-cell-id: + description: Distinct Terraform candidate cell ID + required: true + type: string + mode: + description: Audit/preflight are read-only; recover/continue resume committed work; disable/enable/reset/execute mutate admission + required: true + default: preflight + type: choice + options: + - audit + - preflight + - recover-forward + - continue-evacuation + - disable-cell + - enable-empty-cell + - reset-empty-candidate + - execute + confirmation: + description: Enter RECOVER_FORWARD, CONTINUE_EVACUATION, DISABLE_CELL, ENABLE_CELL, RESET_CANDIDATE, or EVACUATE for the matching mutation + required: false + type: string + monitor-run-id: + description: Successful fresh dry-run monitor workflow run ID + required: false + type: string + monitor-run-attempt: + description: Exact dry-run monitor workflow attempt + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + candidate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + DIRECTOR_ORIGIN: https://relay.onorca.dev + ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain + SOURCE_CELL_ID: ${{ inputs.source-cell-id }} + TARGET_CELL_ID: ${{ inputs.target-cell-id }} + DEPLOY_MODE: ${{ inputs.mode }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + steps: + - uses: actions/checkout@v4 + + - name: Require fresh dry-run evidence reference + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + + - name: Download private dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - uses: pnpm/action-setup@v4 + with: + package_json_file: cloud/package.json + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - name: Verify dry-run artifact before cloud authentication + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-restore \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run + + - name: Reject previously consumed dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + env: + GH_TOKEN: ${{ github.token }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api \ + "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = "0" + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + + - name: Require explicit mutation confirmation + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if [[ "${DEPLOY_MODE}" = "execute" ]]; then + test "${CONFIRMATION}" = "EVACUATE" + elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then + test "${CONFIRMATION}" = "RECOVER_FORWARD" + elif [[ "${DEPLOY_MODE}" = "continue-evacuation" ]]; then + test "${CONFIRMATION}" = "CONTINUE_EVACUATION" + elif [[ "${DEPLOY_MODE}" = "disable-cell" ]]; then + test "${CONFIRMATION}" = "DISABLE_CELL" + elif [[ "${DEPLOY_MODE}" = "enable-empty-cell" ]]; then + test "${CONFIRMATION}" = "ENABLE_CELL" + else + test "${CONFIRMATION}" = "RESET_CANDIDATE" + fi + + - name: Read reviewed Terraform topology + run: | + node dev/scripts/infra.mjs init --env production + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json" + RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)" + echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}" + + - name: Verify fresh dry-run evidence against live selector + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-mutation \ + --directory "${RUNNER_TEMP}/relay-monitor-evidence" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" \ + --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode dry-run \ + --mutation-mode "${DEPLOY_MODE}" \ + --source-cell-id "${SOURCE_CELL_ID}" \ + --director-origin "${DIRECTOR_ORIGIN}" + + - name: Recheck all live safety signals + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + pnpm incident:relay-preflight -- \ + --state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" + + - name: Create single-use dry-run marker + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + run: | + MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}" + + - name: Consume dry-run evidence + if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }} + uses: actions/upload-artifact@v4 + with: + name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + - name: Preflight or evacuate exact GCE candidate + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/deploy-relay-gce-candidate.mjs \ + --project "${GCP_PROJECT_ID}" \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --admin-audience "${ADMIN_AUDIENCE}" \ + --topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \ + --source-cell-id "${SOURCE_CELL_ID}" \ + --target-cell-id "${TARGET_CELL_ID}" \ + --runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \ + --mode "${DEPLOY_MODE}" diff --git a/.github/workflows/cloud-deploy-relay-staging-gce-candidate.yml b/.github/workflows/cloud-deploy-relay-staging-gce-candidate.yml new file mode 100644 index 00000000000..e646980a787 --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-staging-gce-candidate.yml @@ -0,0 +1,109 @@ +name: Deploy Relay Staging GCE Candidate + +on: + workflow_dispatch: + inputs: + source-cell-id: + description: Existing Terraform cell ID to evacuate + required: true + type: string + target-cell-id: + description: Distinct Terraform candidate cell ID + required: true + type: string + mode: + description: Preflight is read-only; reset repairs an empty candidate; execute evacuates + required: true + default: preflight + type: choice + options: + - preflight + - reset-empty-candidate + - execute + confirmation: + description: Enter RESET_CANDIDATE for reset or EVACUATE for execute + required: false + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + candidate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + ADMIN_AUDIENCE: https://relay-staging.onorca.dev/v1/admin/drain + SOURCE_CELL_ID: ${{ inputs.source-cell-id }} + TARGET_CELL_ID: ${{ inputs.target-cell-id }} + DEPLOY_MODE: ${{ inputs.mode }} + steps: + - uses: actions/checkout@v4 + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Require explicit mutation confirmation + if: ${{ inputs.mode != 'preflight' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if [[ "${DEPLOY_MODE}" = "execute" ]]; then + test "${CONFIRMATION}" = "EVACUATE" + else + test "${CONFIRMATION}" = "RESET_CANDIDATE" + fi + + - name: Read reviewed Terraform topology + run: | + node dev/scripts/infra.mjs init --env staging + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json" + RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)" + echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}" + + - name: Preflight or evacuate exact GCE candidate + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/deploy-relay-gce-candidate.mjs \ + --project "${GCP_PROJECT_ID}" \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --admin-audience "${ADMIN_AUDIENCE}" \ + --topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \ + --source-cell-id "${SOURCE_CELL_ID}" \ + --target-cell-id "${TARGET_CELL_ID}" \ + --runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \ + --mode "${DEPLOY_MODE}" diff --git a/.github/workflows/cloud-deploy-relay-staging.yml b/.github/workflows/cloud-deploy-relay-staging.yml new file mode 100644 index 00000000000..cdde494f2ca --- /dev/null +++ b/.github/workflows/cloud-deploy-relay-staging.yml @@ -0,0 +1,112 @@ +name: Deploy Relay Staging + +on: + workflow_dispatch: + inputs: + expected-image-digest: + description: Exact checked-in production Relay sha256 digest to deploy + required: true + type: string + +permissions: + contents: read + id-token: write + +concurrency: + # Staging deploy, candidate, auth, and power operations must never overlap. + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + deploy: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + GCP_REGION: ${{ vars.STAGING_GCP_REGION }} + DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging + REPOSITORY_ID: orca-cloud + IMAGE_NAME: relay + EXPECTED_IMAGE_DIGEST: ${{ inputs.expected-image-digest }} + CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + ASIA_PROOF_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }} + REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled + steps: + - uses: actions/checkout@v4 + + - name: Require the expected immutable image + run: '[[ "${EXPECTED_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]' + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: 1.15.8 + terraform_wrapper: false + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Bind the request to the checked-in staging C4 image + shell: bash + run: | + set -euo pipefail + terraform -chdir=infra/terraform init -reconfigure \ + -backend-config=backend/staging.hcl -input=false + IMAGE="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + <<< 'var.relay_gce_cells["staging-gce-c4"].image' | jq -er '.')" + test "${IMAGE}" = \ + "${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${EXPECTED_IMAGE_DIGEST}" + echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}" + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Require the mirrored immutable image + run: | + DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \ + --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')" + test "${DIGEST}" = "${EXPECTED_IMAGE_DIGEST}" + + - name: Deploy director blue/green + run: | + regional_version="$(gcloud secrets versions describe latest \ + --project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \ + --format='value(name)' | awk -F/ '{print $NF}')" + [[ "${regional_version}" =~ ^[1-9][0-9]*$ ]] + RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}" + node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service "${DIRECTOR_SERVICE_NAME}" \ + --image "${IMAGE}" \ + --role director \ + --max-instances 2 \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \ + --asia-proof-service-account "${ASIA_PROOF_SERVICE_ACCOUNT}" \ + --regional-placement-secret-version "${regional_version}" \ + --min-instances 0 \ + --release-id "${RELEASE_ID}" + + - name: Smoke director health + run: | + URL="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(status.url)')" + node dev/scripts/smoke-relay.mjs "${URL}" diff --git a/.github/workflows/cloud-monitor-relay-clock-skew.yml b/.github/workflows/cloud-monitor-relay-clock-skew.yml new file mode 100644 index 00000000000..86156807bf0 --- /dev/null +++ b/.github/workflows/cloud-monitor-relay-clock-skew.yml @@ -0,0 +1,76 @@ +name: Monitor Relay Cell Clock Skew + +# Why: the 2026-08-01 sustained HOST_OFFLINE incident traced to one cell's +# clock running ~100ms ahead, which deterministically failed every host +# challenge under a zero-tolerance freshness check. /health and /ready cannot +# see clock skew; this monitor alarms before drift reaches the (now 2s) +# client tolerance. + +on: + workflow_dispatch: + schedule: + - cron: '17 * * * *' + +permissions: + contents: read + +defaults: + run: + working-directory: cloud + +jobs: + skew: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: ubuntu-latest + steps: + - name: Measure Date-header skew for every relay cell + run: | + set -u + # Date headers carry whole seconds; compare floored seconds on both + # sides so perfect sync reads 0/±1 and never flaps. An absolute + # floor-skew of >= 2 means real drift of at least ~1s — approaching + # the client's 2s challenge tolerance. Millisecond-precision deltas + # come from the desktop's named-check logging when activations fail. + ALARM_S=2 + failures=0 + reachable=0 + unserved=0 + # Keep this upper bound at or above the highest provisioned cell in + # environments/production.tfvars; unlisted cells are silently unmonitored. + for n in $(seq 1 22); do + cell="c${n}" + url="https://${cell}.relay.onorca.dev/health" + # Why: *.relay.onorca.dev is a wildcard, so the load balancer answers with its own + # accurate Date for a fenced, dead, or never-provisioned cell. Timing that reads as + # perfect sync. Only an HTTP 200 is the relay process itself answering, so only a + # 200 carries a clock worth judging. + response="$(curl -sS -D - -o /dev/null --max-time 8 "${url}" 2>/dev/null || true)" + status="$(printf '%s' "${response}" | awk 'NR==1 {print $2}')" + header="$(printf '%s' "${response}" | tr -d '\r' | grep -i '^date:' || true)" + if [ "${status:-000}" != "200" ] || [ -z "${header}" ]; then + # Expected for the fenced cells; a dead unfenced cell is caught by the heartbeat + # and readiness alerts, not here. Named either way so it is never invisible. + echo "${cell}: not serving (status ${status:-none}); no relay clock to judge" + unserved=$((unserved + 1)) + continue + fi + reachable=$((reachable + 1)) + server_s="$(date -d "${header#*: }" +%s)" + local_s="$(date +%s)" + skew=$((server_s - local_s)) + abs=${skew#-} + if [ "${abs}" -ge "${ALARM_S}" ]; then + echo "::error::${cell}: clock skew ${skew}s reaches ±${ALARM_S}s alarm" + failures=$((failures + 1)) + else + echo "${cell}: skew ${skew}s" + fi + done + echo "cells serving: ${reachable}, not serving: ${unserved}, alarms: ${failures}" + if [ "${reachable}" -eq 0 ]; then + # Now meaningful: previously the load balancer answered for every name, so this + # could never fire and a total fleet outage reported "all healthy". + echo "::error::no relay cell is serving; monitor blind" + exit 1 + fi + exit "$((failures > 0 ? 1 : 0))" diff --git a/.github/workflows/cloud-monitor-relay-production-job.yml b/.github/workflows/cloud-monitor-relay-production-job.yml new file mode 100644 index 00000000000..3ecde1c5f1d --- /dev/null +++ b/.github/workflows/cloud-monitor-relay-production-job.yml @@ -0,0 +1,218 @@ +name: Monitor Relay Production Job + +on: + workflow_call: + inputs: + mode: + required: true + type: string + expected-selector-generation: + required: true + type: string + expected-existing-only-cells: + required: true + type: string + expected-migration-only-cells: + required: true + type: string + expected-general-cells: + required: true + type: string + migration-policy: + required: true + type: string + recovery-source-cell-id: + required: true + type: string + capacity-cell-id: + required: true + type: string + +permissions: + actions: read + contents: read + id-token: write + +defaults: + run: + working-directory: cloud + +jobs: + monitor: + if: >- + ${{ github.ref == 'refs/heads/main' && + vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER != '' && + vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT != '' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 100 + environment: production + env: + EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }} + EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }} + EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }} + EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }} + MIGRATION_POLICY: ${{ inputs.migration-policy }} + RECOVERY_SOURCE_CELL_ID: ${{ inputs.recovery-source-cell-id }} + CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }} + INCIDENT_ID: relay-${{ github.run_id }}-${{ inputs.mode }} + MONITOR_MODE: ${{ inputs.mode }} + OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-incident + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + package_json_file: cloud/package.json + + - uses: actions/setup-node@v4 + with: + node-version: 24 + cache: pnpm + cache-dependency-path: cloud/pnpm-lock.yaml + + - run: pnpm install --frozen-lockfile + + - id: prior-attempt + if: ${{ github.run_attempt > 1 }} + run: echo "value=$((GITHUB_RUN_ATTEMPT - 1))" >> "${GITHUB_OUTPUT}" + + - name: Restore prior private monitor state + if: ${{ github.run_attempt > 1 }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ steps.prior-attempt.outputs.value }} + path: ${{ github.workspace }}/relay-incident + github-token: ${{ github.token }} + run-id: ${{ github.run_id }} + + - name: Verify restored state provenance + if: ${{ github.run_attempt > 1 }} + run: | + node dev/scripts/relay-monitor-evidence.mjs verify-restore \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "${INCIDENT_ID}" \ + --run-id "${GITHUB_RUN_ID}" \ + --run-attempt "${{ steps.prior-attempt.outputs.value }}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode "${MONITOR_MODE}" + echo "RESTART_FLAG=--restart" >> "${GITHUB_ENV}" + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - name: Verify exact-audience admin identity + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)" + + - name: Run read-only relay dry-run + if: ${{ inputs.mode == 'dry-run' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + pnpm --filter @orca-cloud/relay-ops incident:monitor \ + --environment production \ + --incident-id "${INCIDENT_ID}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --migration-policy "${MIGRATION_POLICY}" \ + --recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \ + --capacity-cell-id "${CAPACITY_CELL_ID}" \ + --interval-seconds 60 \ + --output-directory "${OUTPUT_DIRECTORY}" \ + --duration-minutes 15 \ + --pre-drain-dry-run \ + ${RESTART_FLAG:-} + + - name: Run first relay monitor segment + if: ${{ inputs.mode == 'monitor' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + pnpm --filter @orca-cloud/relay-ops incident:monitor \ + --environment production \ + --incident-id "${INCIDENT_ID}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --migration-policy "${MIGRATION_POLICY}" \ + --recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \ + --capacity-cell-id "${CAPACITY_CELL_ID}" \ + --interval-seconds 60 \ + --output-directory "${OUTPUT_DIRECTORY}" \ + --duration-minutes 90 \ + --max-samples-this-run 45 \ + ${RESTART_FLAG:-} + + - id: google-auth-refresh + if: ${{ inputs.mode == 'monitor' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Run remaining relay monitor window + if: ${{ inputs.mode == 'monitor' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-refresh.outputs.id_token }} + run: | + node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)" + pnpm --filter @orca-cloud/relay-ops incident:monitor \ + --environment production \ + --incident-id "${INCIDENT_ID}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --migration-policy "${MIGRATION_POLICY}" \ + --recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \ + --capacity-cell-id "${CAPACITY_CELL_ID}" \ + --interval-seconds 60 \ + --output-directory "${OUTPUT_DIRECTORY}" \ + --duration-minutes 90 \ + --restart + + - name: Seal private evidence provenance + if: ${{ always() }} + run: | + node dev/scripts/relay-monitor-evidence.mjs create \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "${INCIDENT_ID}" \ + --run-id "${GITHUB_RUN_ID}" \ + --run-attempt "${GITHUB_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --mode "${MONITOR_MODE}" + + - name: Publish aggregate job summary + if: ${{ always() }} + run: | + if [[ -f "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" ]]; then + cat "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" >> "${GITHUB_STEP_SUMMARY}" + else + echo "Relay monitor failed before its first aggregate checkpoint." \ + >> "${GITHUB_STEP_SUMMARY}" + fi + + - name: Upload private aggregate evidence + if: ${{ always() }} + uses: actions/upload-artifact@v4 + with: + name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ github.workspace }}/relay-incident + if-no-files-found: error + retention-days: 14 diff --git a/.github/workflows/cloud-monitor-relay-production.yml b/.github/workflows/cloud-monitor-relay-production.yml new file mode 100644 index 00000000000..402e2aa9f69 --- /dev/null +++ b/.github/workflows/cloud-monitor-relay-production.yml @@ -0,0 +1,75 @@ +name: Monitor Relay Production + +on: + workflow_dispatch: + inputs: + mode: + description: Run the required 15-minute pre-drain gate or a 90-minute incident watch + required: true + default: dry-run + type: choice + options: + - dry-run + - monitor + expected-selector-generation: + description: Exact durable admission-selector generation + required: true + type: string + expected-existing-only-cells: + description: Exact comma-separated existing-only cells, or none + required: true + type: string + expected-migration-only-cells: + description: Exact comma-separated migration-only cells, or none + required: true + type: string + expected-general-cells: + description: Exact comma-separated general cells, or none + required: true + type: string + migration-policy: + description: Migration checks matched to the intended mutation + required: true + default: strict + type: choice + options: + - strict + - recover-forward + - capacity-transition + recovery-source-cell-id: + description: Existing-only recovery source cell, or none for strict monitoring + required: true + default: none + type: string + capacity-cell-id: + description: General cell for a capacity-transition monitor, or none + required: true + default: none + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + monitor: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + uses: ./.github/workflows/cloud-monitor-relay-production-job.yml + with: + mode: ${{ inputs.mode }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + migration-policy: ${{ inputs.migration-policy }} + recovery-source-cell-id: ${{ inputs.recovery-source-cell-id }} + capacity-cell-id: ${{ inputs.capacity-cell-id }} diff --git a/.github/workflows/cloud-operate-relay-asia-admission.yml b/.github/workflows/cloud-operate-relay-asia-admission.yml new file mode 100644 index 00000000000..2196c06dc5c --- /dev/null +++ b/.github/workflows/cloud-operate-relay-asia-admission.yml @@ -0,0 +1,512 @@ +name: Operate Relay Asia Admission + +on: + workflow_dispatch: + inputs: + environment: + description: Target Relay environment + required: true + type: choice + options: [staging, production] + mode: + description: Inspect, initialize, verify, atomically register, promote, or roll back admission + required: true + default: verify + type: choice + options: [inspect, initialize, verify, register, configure, promote, rollback] + cell-ids: + description: Exact reviewed comma-separated Asia cell wave + required: true + type: string + selector-generation: + description: Exact live selector generation; leave empty only for inspect + required: false + type: string + selector-membership-sha256: + description: Exact fingerprint printed by inspect; required only for initialize + required: false + type: string + selector-attempt-id: + description: Durable unique attempt ID; empty for inspect, verify, and configure + required: false + type: string + image-digest: + description: Expected compatible Relay sha256 digest + required: true + type: string + director-image-digest: + description: Director sha256 digest; required only for configure + required: false + type: string + evidence-run-id: + description: Successful staging or C27 evidence workflow run ID; required for production promotion + required: false + type: string + evidence-run-attempt: + description: Exact evidence workflow run attempt; required for production promotion + required: false + type: string + confirmation: + description: Exact typed confirmation for a mutation + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }} + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + admission: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 30 + environment: ${{ inputs.environment }} + env: + DIRECTOR_ORIGIN: ${{ inputs.environment == 'production' && 'https://relay.onorca.dev' || 'https://relay-staging.onorca.dev' }} + AUTH_ORIGIN: ${{ inputs.environment == 'production' && 'https://login.onorca.dev' || 'https://auth-staging.onorca.dev' }} + DIRECTOR_SERVICE: ${{ inputs.environment == 'production' && 'orca-cloud-relay' || 'orca-cloud-relay-staging' }} + REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled + GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }} + GCP_REGION: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_REGION || vars.STAGING_GCP_REGION }} + DIRECTOR_MAX_INSTANCES: ${{ inputs.environment == 'production' && '5' || '2' }} + TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }} + TARGET_ENVIRONMENT: ${{ inputs.environment }} + OPERATION_MODE: ${{ inputs.mode }} + TARGET_CELL_IDS: ${{ inputs.cell-ids }} + EXPECTED_SELECTOR_GENERATION: ${{ inputs.selector-generation }} + EXPECTED_SELECTOR_MEMBERSHIP_SHA256: ${{ inputs.selector-membership-sha256 }} + SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }} + IMAGE_DIGEST: ${{ inputs.image-digest }} + DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }} + EVIDENCE_RUN_ID: ${{ inputs.evidence-run-id }} + EVIDENCE_RUN_ATTEMPT: ${{ inputs.evidence-run-attempt }} + OPERATION_CONFIRMATION: ${{ inputs.confirmation }} + DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + DEPLOY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + steps: + - uses: actions/checkout@v4 + + - name: Validate exact operation inputs before authentication + id: inputs + shell: bash + run: | + set -euo pipefail + test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}" + test -n "${DEPLOY_SERVICE_ACCOUNT}" + [[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]] + evidence_kind=none + if test "${OPERATION_MODE}" = inspect; then + test -z "${EXPECTED_SELECTOR_GENERATION}" + test -z "${SELECTOR_ATTEMPT_ID}" + test -z "${OPERATION_CONFIRMATION}" + test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" + test -z "${DIRECTOR_IMAGE_DIGEST}" + else + [[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + fi + if test "${OPERATION_MODE}" = initialize; then + test "${EXPECTED_SELECTOR_GENERATION}" = 0 + [[ "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" =~ ^[a-f0-9]{64}$ ]] + test -z "${DIRECTOR_IMAGE_DIGEST}" + [[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + test "${OPERATION_CONFIRMATION}" = INITIALIZE_ADMISSION_SELECTOR + elif test "${OPERATION_MODE}" = verify; then + test -z "${SELECTOR_ATTEMPT_ID}" + test -z "${OPERATION_CONFIRMATION}" + test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" + test -z "${DIRECTOR_IMAGE_DIGEST}" + elif test "${OPERATION_MODE}" = inspect; then + : + elif test "${OPERATION_MODE}" = configure; then + test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" + test -z "${SELECTOR_ATTEMPT_ID}" + [[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]] + test "${OPERATION_CONFIRMATION}" = CONFIGURE_ASIA_DIRECTOR + else + test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" + test -z "${DIRECTOR_IMAGE_DIGEST}" + [[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + case "${OPERATION_MODE}:${OPERATION_CONFIRMATION}" in + register:REGISTER_ASIA_MIGRATION_ONLY) ;; + promote:PROMOTE_ASIA_GENERAL) ;; + rollback:ROLLBACK_ASIA_MIGRATION_ONLY) ;; + *) echo "typed confirmation does not match the requested mutation" >&2; exit 1 ;; + esac + fi + if test "${TARGET_ENVIRONMENT}:${OPERATION_MODE}" = production:promote; then + [[ "${EVIDENCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + [[ "${EVIDENCE_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + case "${TARGET_CELL_IDS}" in + production-gce-c27) + test "${#SELECTOR_ATTEMPT_ID}" -le 119 + evidence_kind=staging + artifact_name="relay-asia-staging-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}" + ;; + production-gce-c28,production-gce-c29) + evidence_kind=c27 + artifact_name="relay-asia-c27-canary-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}" + ;; + *) echo "production promotion wave is not reviewed" >&2; exit 1 ;; + esac + else + test -z "${EVIDENCE_RUN_ID}" + test -z "${EVIDENCE_RUN_ATTEMPT}" + artifact_name=none + fi + { + echo "evidence_kind=${evidence_kind}" + echo "artifact_name=${artifact_name}" + } >> "${GITHUB_OUTPUT}" + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - uses: pnpm/action-setup@v4 + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + + - name: Install exact C27 canary dependencies + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + run: pnpm install --frozen-lockfile + + - name: Build the C27 canary Relay contract + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + run: pnpm --filter @orca-cloud/relay-contract build + + - name: Download immutable rollout evidence + if: ${{ steps.inputs.outputs.evidence_kind != 'none' }} + uses: actions/download-artifact@v4 + with: + name: ${{ steps.inputs.outputs.artifact_name }} + path: ${{ runner.temp }}/relay-asia-input-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.evidence-run-id }} + + - name: Verify evidence provenance and rollout binding before authentication + if: ${{ steps.inputs.outputs.evidence_kind != 'none' }} + env: + GH_TOKEN: ${{ github.token }} + EVIDENCE_KIND: ${{ steps.inputs.outputs.evidence_kind }} + shell: bash + run: | + set -euo pipefail + run_json="${RUNNER_TEMP}/relay-asia-evidence-run.json" + verified="${RUNNER_TEMP}/relay-asia-evidence-verified" + gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${EVIDENCE_RUN_ID}/attempts/${EVIDENCE_RUN_ATTEMPT}" > "${run_json}" + evidence_commit_sha="$( + jq -er '.head_sha | select(type == "string" and test("^[a-f0-9]{40}$"))' "${run_json}" + )" + command=(node dev/scripts/relay-asia-rollout-evidence.mjs "verify-${EVIDENCE_KIND}" + --evidence "${RUNNER_TEMP}/relay-asia-input-evidence/evidence.json" + --run-json "${run_json}" + --commit-sha "${evidence_commit_sha}" + --image-digest "${IMAGE_DIGEST}" + --now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + --output "${verified}") + if test "${EVIDENCE_KIND}" = c27; then + command+=(--selector-generation "${EXPECTED_SELECTOR_GENERATION}") + fi + "${command[@]}" + test "$(< "${verified}")" = verified + + - id: auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: ${{ env.DIRECTOR_ORIGIN }}/v1/admin/drain + id_token_include_email: true + + - name: Require the exact director image before promotion + if: ${{ inputs.mode == 'promote' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + runtime="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')" + test "$(jq -r '.role' <<< "${runtime}")" = director + test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}" + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }} + object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }} + if: ${{ inputs.mode == 'configure' || (inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27') }} + + - uses: hashicorp/setup-terraform@v3 + if: ${{ inputs.mode == 'configure' }} + with: + terraform_version: 1.15.8 + terraform_wrapper: false + + - name: Run the exact generation-bound admission operation + id: admission-operation + if: ${{ inputs.mode != 'configure' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + run: | + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment "${TARGET_ENVIRONMENT}" \ + --mode "${OPERATION_MODE}" \ + --cell-ids "${TARGET_CELL_IDS}" \ + --expected-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-membership-sha256 "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" \ + --attempt-id "${SELECTOR_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + generation="$(jq -er '.generation' <<< "${result}")" + states="$(jq -cS '.states // {}' <<< "${result}")" + membership="$(jq -cS '.membership // empty' <<< "${result}")" + membership_sha256="$(jq -r '.membershipSha256 // empty' <<< "${result}")" + echo "generation=${generation}" >> "${GITHUB_OUTPUT}" + result_dir="${RUNNER_TEMP}/relay-asia-admission-result" + mkdir -p "${result_dir}" + node dev/scripts/sanitize-relay-asia-admission-result.mjs \ + <<< "${result}" > "${result_dir}/result.json" + { + echo "### Relay Asia admission" + echo "- Mode: ${OPERATION_MODE}" + echo "- Cells: ${TARGET_CELL_IDS}" + echo "- Result generation: ${generation}" + echo "- States: \`${states}\`" + if test -n "${membership}"; then echo "- Membership: \`${membership}\`"; fi + if test -n "${membership_sha256}"; then + echo "- Membership SHA-256: \`${membership_sha256}\`" + fi + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Verify C27 state and start the timed canary + id: c27-start + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment production \ + --mode verify \ + --cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \ + --expected-generation "${{ steps.admission-operation.outputs.generation }}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general + test "$(jq -r '.states["production-gce-c28"]' <<< "${result}")" = migration-only + test "$(jq -r '.states["production-gce-c29"]' <<< "${result}")" = migration-only + echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}" + + - name: Run a real five-minute C27 control and splice canary + id: c27-load + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + shell: bash + run: | + set -euo pipefail + log="${RUNNER_TEMP}/relay-asia-c27-load.jsonl" + report="${RUNNER_TEMP}/relay-asia-c27-load.json" + node dev/scripts/load-relay-controls.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --auth-origin "${AUTH_ORIGIN}" \ + --preferred-region asia-east2 \ + --relay-asia-load-principals 1 \ + --controls 1 \ + --splices 1 \ + --capacity-hard-cap 3000 \ + --ramp-seconds 0 \ + --duration-seconds 300 \ + --splice-hold-seconds 60 \ + --required-lease-horizons 2 > "${log}" + jq -cer 'select(.event == "relay_load_complete")' "${log}" | tail -n 1 > "${report}" + echo "ended_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}" + + - name: Collect regional, Relay SQL, and Cloud SQL canary evidence + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + CANARY_STARTED_AT: ${{ steps.c27-start.outputs.started_at }} + shell: bash + run: | + set -euo pipefail + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment production \ + --mode verify \ + --cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \ + --expected-generation "${{ steps.admission-operation.outputs.generation }}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general + ended_at="${{ steps.c27-load.outputs.ended_at }}" + sleep 60 + output="${RUNNER_TEMP}/relay-asia-output-evidence" + logs="${RUNNER_TEMP}/relay-asia-c27-runtime-metrics.json" + mkdir -p "${output}" + gcloud logging read \ + "timestamp>=\"${CANARY_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \ + --project "${GCP_PROJECT_ID}" \ + --limit 20000 \ + --format json > "${logs}" + node dev/scripts/relay-asia-rollout-evidence.mjs create-c27 \ + --repository "${GITHUB_REPOSITORY}" \ + --run-id "${GITHUB_RUN_ID}" \ + --run-attempt "${GITHUB_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --image-digest "${IMAGE_DIGEST}" \ + --selector-generation "${{ steps.admission-operation.outputs.generation }}" \ + --started-at "${CANARY_STARTED_AT}" \ + --ended-at "${ended_at}" \ + --load-report "${RUNNER_TEMP}/relay-asia-c27-load.json" \ + --logs-json "${logs}" \ + --output "${output}/evidence.json" + jq -r '.metrics | to_entries[] | "- \(.key): \(.value)"' \ + "${output}/evidence.json" >> "${GITHUB_STEP_SUMMARY}" + + - name: Upload immutable C27 canary evidence + id: c27-evidence-upload + if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }} + uses: actions/upload-artifact@v4 + with: + name: relay-asia-c27-canary-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/relay-asia-output-evidence/evidence.json + if-no-files-found: error + retention-days: 7 + + - name: Upload sanitized admission result + if: ${{ inputs.mode != 'configure' && steps.admission-operation.outcome == 'success' }} + uses: actions/upload-artifact@v4 + with: + name: relay-asia-admission-result-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/relay-asia-admission-result/result.json + if-no-files-found: error + retention-days: 7 + + - name: Return an unproven C27 canary to migration-only + if: ${{ always() && inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' && steps.admission-operation.outcome != 'skipped' && steps.c27-evidence-upload.outcome != 'success' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment production \ + --mode recover-promotion \ + --cell-ids production-gce-c27 \ + --expected-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --attempt-id "${SELECTOR_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi + promoted_generation="$(jq -er '.generation' <<< "${promoted}")" + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment production \ + --mode rollback \ + --cell-ids production-gce-c27 \ + --expected-generation "${promoted_generation}" \ + --attempt-id "${SELECTOR_ATTEMPT_ID}-rollback" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = migration-only + + - name: Require registered migration-only cells before director configuration + if: ${{ inputs.mode == 'configure' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + run: | + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment "${TARGET_ENVIRONMENT}" \ + --mode registered \ + --cell-ids "${TARGET_CELL_IDS}" \ + --expected-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true + + - name: Build the additive director cell configuration + if: ${{ inputs.mode == 'configure' }} + id: director-config + shell: bash + run: | + set -euo pipefail + terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}" + topology="${RUNNER_TEMP}/relay-asia-state-topology.json" + current="${RUNNER_TEMP}/relay-current-director-cells.json" + desired="${RUNNER_TEMP}/relay-asia-director-cells.json" + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${topology}" + service="$(gcloud run services describe "${DIRECTOR_SERVICE}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + revision="$(jq -er '[.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end' \ + <<< "${service}")" + gcloud run revisions describe "${revision}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -er '.spec.containers[0].env[] | select(.name == "ORCA_RELAY_CELLS_JSON") | .value | fromjson' \ + > "${current}" + node dev/scripts/prepare-relay-asia-director-cells.mjs \ + --current-json "${current}" \ + --topology-json "${topology}" \ + --output "${desired}" \ + --cell-ids "${TARGET_CELL_IDS}" \ + --image-digest "${IMAGE_DIGEST}" + echo "file=${desired}" >> "${GITHUB_OUTPUT}" + + - name: Deploy the registered additive director topology + if: ${{ inputs.mode == 'configure' }} + env: + DIRECTOR_CELLS_FILE: ${{ steps.director-config.outputs.file }} + run: | + current="$(gcloud secrets versions access latest \ + --project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")" + [[ "${current}" =~ ^(true|false)$ ]] + image="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}" + release_id="asia-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}" + node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service "${DIRECTOR_SERVICE}" \ + --image "${image}" \ + --role director \ + --max-instances "${DIRECTOR_MAX_INSTANCES}" \ + --release-id "${release_id}" \ + --director-cells-json "$(< "${DIRECTOR_CELLS_FILE}")" \ + --prune-revisions false + + - name: Verify selector and heartbeats after director configuration + if: ${{ inputs.mode == 'configure' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + run: | + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment "${TARGET_ENVIRONMENT}" \ + --mode verify \ + --cell-ids "${TARGET_CELL_IDS}" \ + --expected-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true + revision="$(gcloud run services describe "${DIRECTOR_SERVICE}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -er '[.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end')" + revision_json="$(gcloud run revisions describe "${revision}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + jq -e --arg image "us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}" \ + '.spec.containers[0].image == $image' <<< "${revision_json}" > /dev/null + jq -er --arg secret "${REGIONAL_PLACEMENT_SECRET}" \ + '[.spec.containers[0].env[] | + select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") | + (.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) | + {secret: (.secret // .name), version: (.version // .key)} | + select(.secret == $secret and (.version | test("^[1-9][0-9]*$")))] | + if length == 1 then .[0].version else error("regional switch version missing") end' \ + <<< "${revision_json}" > "${RUNNER_TEMP}/relay-regional-placement-version" + regional_version="$(< "${RUNNER_TEMP}/relay-regional-placement-version")" + [[ "$(gcloud secrets versions access "${regional_version}" --project "${GCP_PROJECT_ID}" \ + --secret "${REGIONAL_PLACEMENT_SECRET}")" =~ ^(true|false)$ ]] + echo "Director configuration now lists the registered migration-only Asia cells." >> "${GITHUB_STEP_SUMMARY}" diff --git a/.github/workflows/cloud-operate-relay-production-rehome-job.yml b/.github/workflows/cloud-operate-relay-production-rehome-job.yml new file mode 100644 index 00000000000..058ee2e54cf --- /dev/null +++ b/.github/workflows/cloud-operate-relay-production-rehome-job.yml @@ -0,0 +1,327 @@ +name: Operate Relay Production Rehome Job + +on: + workflow_call: + inputs: + mode: { required: true, type: string } + director-image-digest: { required: true, type: string } + rollback-image-digest: { required: true, type: string } + expected-selector-generation: { required: true, type: string } + expected-existing-only-cells: { required: true, type: string } + expected-migration-only-cells: { required: true, type: string } + expected-general-cells: { required: true, type: string } + expected-control-generation: { required: true, type: string } + not-before: { required: true, type: string } + rate-per-minute: { required: true, type: string } + preference-max-age-ms: { required: true, type: string } + drain-grace-ms: { required: true, type: string } + confirmation: { required: true, type: string } + monitor-run-id: { required: true, type: string } + monitor-run-attempt: { required: true, type: string } + +permissions: + actions: read + contents: read + id-token: write + +defaults: + run: + working-directory: cloud + +jobs: + control: + if: ${{ github.ref == 'refs/heads/main' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 30 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + DIRECTOR_SERVICE: orca-cloud-relay + DIRECTOR_ORIGIN: https://relay.onorca.dev + REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain + MODE: ${{ inputs.mode }} + DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }} + ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }} + EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }} + EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }} + EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }} + EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }} + EXPECTED_CONTROL_GENERATION: ${{ inputs.expected-control-generation }} + NOT_BEFORE: ${{ inputs.not-before }} + RATE_PER_MINUTE: ${{ inputs.rate-per-minute }} + PREFERENCE_MAX_AGE_MS: ${{ inputs.preference-max-age-ms }} + DRAIN_GRACE_MS: ${{ inputs.drain-grace-ms }} + CONFIRMATION: ${{ inputs.confirmation }} + MONITOR_RUN_ID: ${{ inputs.monitor-run-id }} + MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }} + OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence + steps: + - name: Require exact reusable-workflow configuration + env: + DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + run: | + [[ "${MODE}" =~ ^(inspect|enable|pause|disable)$ ]] + [[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + [[ "${EXPECTED_CONTROL_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + test -n "${DEPLOY_WIF}" + test -n "${DEPLOY_SERVICE_ACCOUNT}" + if [[ "${MODE}" =~ ^(inspect|enable)$ ]]; then + [[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + [[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + test -n "${GCP_REGION}" + test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" + fi + case "${MODE}" in + inspect) + test -z "${CONFIRMATION}" + ;; + enable) + test "${CONFIRMATION}" = ENABLE_REGIONAL_REHOMING + test "${RATE_PER_MINUTE}" = 10 + [[ "${NOT_BEFORE}" =~ ^[1-9][0-9]*$ ]] + ;; + pause) + test "${CONFIRMATION}" = PAUSE_REGIONAL_REHOMING + ;; + disable) + test "${CONFIRMATION}" = DISABLE_REGIONAL_REHOMING + ;; + esac + + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Apply emergency durable pause or disable before diagnostics + if: ${{ inputs.mode == 'pause' || inputs.mode == 'disable' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \ + --not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \ + --preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \ + --drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \ + | tee "${RUNNER_TEMP}/relay-rehome-control.json" + + - uses: pnpm/action-setup@v4 + if: ${{ inputs.mode == 'enable' }} + with: { package_json_file: cloud/package.json } + + - run: pnpm install --frozen-lockfile + if: ${{ inputs.mode == 'enable' }} + + - name: Download fresh aggregate safety evidence + if: ${{ inputs.mode == 'enable' }} + uses: actions/download-artifact@v4 + with: + name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ github.workspace }}/relay-monitor-evidence + github-token: ${{ github.token }} + run-id: ${{ inputs.monitor-run-id }} + + - name: Verify enable evidence provenance + if: ${{ inputs.mode == 'enable' }} + run: | + [[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + [[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]] + node dev/scripts/relay-monitor-evidence.mjs verify-authority \ + --directory "${OUTPUT_DIRECTORY}" \ + --incident-id "relay-${MONITOR_RUN_ID}-dry-run" \ + --run-id "${MONITOR_RUN_ID}" --run-attempt "${MONITOR_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" --mode dry-run \ + --required-migration-policy strict + + - name: Reject previously consumed enable safety evidence + if: ${{ inputs.mode == 'enable' }} + env: + GH_TOKEN: ${{ github.token }} + run: | + MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \ + --jq '.total_count')" + test "${COUNT}" = 0 + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-terraform-state + object: terraform/state/cloud-sql-rollout/production.lock + + - name: Verify exact serving and rollback director identities + if: ${{ inputs.mode == 'inspect' || inputs.mode == 'enable' }} + env: + DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }} + run: | + SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + SERVING_REVISION="$(jq -er \ + '[.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName + else error("director does not have one serving revision") end' \ + <<< "${SERVICE_JSON}")" + ROLLBACK_REVISION="$(jq -er \ + '[.status.traffic[] | select(.tag == "selector-rollback")] | + if length == 1 then .[0].revisionName else error("rollback tag missing") end' \ + <<< "${SERVICE_JSON}")" + verify_revision() { + local revision="$1" expected_digest="$2" + local json + json="$(gcloud run revisions describe "${revision}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + test "$(jq -r '.spec.serviceAccountName' <<< "${json}")" = \ + "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" + test "$(jq -r '.spec.containers[0].image | split("@") | last' <<< "${json}")" = \ + "${expected_digest}" + test "$(jq -r '[.spec.containers[0].env[] | select(.name == + "ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT") | .value] | if length == 1 + then .[0] else empty end' <<< "${json}")" = "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" + test "$(jq -r '[.spec.containers[0].env[] | select(.name == + "ORCA_RELAY_REHOME_AUDIENCE") | .value] | if length == 1 then .[0] + else empty end' <<< "${json}")" = "${REHOME_AUDIENCE}" + } + verify_revision "${SERVING_REVISION}" "${DIRECTOR_IMAGE_DIGEST}" + verify_revision "${ROLLBACK_REVISION}" "${ROLLBACK_IMAGE_DIGEST}" + + - name: Seal 24-hour aggregate region observation evidence + if: ${{ inputs.mode == 'enable' }} + run: | + mkdir -p "${RUNNER_TEMP}/relay-region-observation" + # 6 director instances x 120 samples/hour x 25h = 18000; a clipped + # read empties the oldest hourly buckets and fails the seal. + gcloud logging read \ + 'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND jsonPayload.event="orca_relay_runtime_metrics" AND jsonPayload.role="director"' \ + --project "${GCP_PROJECT_ID}" --freshness=25h --limit=30000 --format=json \ + | node dev/scripts/relay-region-observation-evidence.mjs create \ + --commit-sha "${GITHUB_SHA}" \ + --director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \ + --selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --control-generation "${EXPECTED_CONTROL_GENERATION}" \ + > "${RUNNER_TEMP}/relay-region-observation/evidence.json" + + - name: Upload sealed 24-hour aggregate region evidence + if: ${{ inputs.mode == 'enable' }} + uses: actions/upload-artifact@v4 + with: + name: relay-region-observation-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/relay-region-observation/evidence.json + retention-days: 90 + if-no-files-found: error + + - name: Verify sealed 24-hour enable authority + if: ${{ inputs.mode == 'enable' }} + run: | + node dev/scripts/relay-region-observation-evidence.mjs verify \ + --file "${RUNNER_TEMP}/relay-region-observation/evidence.json" \ + --commit-sha "${GITHUB_SHA}" \ + --director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \ + --selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --control-generation "${EXPECTED_CONTROL_GENERATION}" + + - name: Recheck every aggregate safety signal before enable + if: ${{ inputs.mode == 'enable' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + pnpm incident:relay-preflight -- \ + --state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" + + - name: Seal single-use enable safety authority + if: ${{ inputs.mode == 'enable' }} + run: | + MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}" + mkdir -p "${RUNNER_TEMP}/relay-rehome-enable-authority" + printf '%s\n' "${GITHUB_RUN_ID}" \ + > "${RUNNER_TEMP}/relay-rehome-enable-authority/${MARKER_NAME}" + + - name: Consume enable safety evidence before durable mutation + if: ${{ inputs.mode == 'enable' }} + uses: actions/upload-artifact@v4 + with: + name: relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + path: ${{ runner.temp }}/relay-rehome-enable-authority/relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }} + retention-days: 90 + if-no-files-found: error + + - name: Inspect regional rehome control + if: ${{ inputs.mode == 'inspect' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode inspect --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \ + | tee "${RUNNER_TEMP}/relay-rehome-control.json" + + - name: Apply exact durable regional rehome enable + if: ${{ inputs.mode == 'enable' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \ + --expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \ + --expected-general-cells "${EXPECTED_GENERAL_CELLS}" \ + --expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \ + --not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \ + --preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \ + --drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \ + | tee "${RUNNER_TEMP}/relay-rehome-control.json" + + - name: Read fresh aggregate completion and abort evidence + run: | + gcloud logging read \ + 'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND textPayload:"[orca-relay] regional rehome inventory"' \ + --project "${GCP_PROJECT_ID}" --freshness=15m --limit=20 --format=json \ + | node dev/scripts/relay-rehome-aggregate-evidence.mjs --max-age-ms 900000 \ + | tee "${RUNNER_TEMP}/relay-rehome-inventory.json" + + - name: Publish aggregate control evidence + run: | + { + echo '### Regional rehome control' + jq -r '"- mode: `\(.mode)`\n- generation: `\(.control.generation)`\n- enabled: `\(.control.enabled)`"' \ + "${RUNNER_TEMP}/relay-rehome-control.json" + jq -r '"- active: `\(.active)`\n- awaiting receipt: `\(.awaitingReceipt)`\n- target registered: `\(.targetRegistered)`\n- completed (24h): `\(.completedLast24Hours)`\n- aborted (24h): `\(.abortedLast24Hours)`"' \ + "${RUNNER_TEMP}/relay-rehome-inventory.json" + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Fail closed after an unsuccessful enable run + if: ${{ failure() && inputs.mode == 'enable' && steps.google-auth.outcome == 'success' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/operate-relay-regional-rehome.mjs \ + --mode recover-enable --director-origin "${DIRECTOR_ORIGIN}" \ + --expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \ + --confirmation RECOVER_FAILED_REGIONAL_REHOME_ENABLE \ + | tee "${RUNNER_TEMP}/relay-rehome-enable-recovery.json" + jq -e \ + '.mode == "recover-enable" and .control.enabled == false' \ + "${RUNNER_TEMP}/relay-rehome-enable-recovery.json" >/dev/null diff --git a/.github/workflows/cloud-operate-relay-production-rehome.yml b/.github/workflows/cloud-operate-relay-production-rehome.yml new file mode 100644 index 00000000000..40bf5ebbd4f --- /dev/null +++ b/.github/workflows/cloud-operate-relay-production-rehome.yml @@ -0,0 +1,106 @@ +name: Operate Relay Production Rehome + +on: + workflow_dispatch: + inputs: + mode: + description: Inspect or apply the durable regional-rehome switch + required: true + default: inspect + type: choice + options: [inspect, enable, pause, disable] + director-image-digest: + description: Exact immutable serving director digest + required: true + type: string + rollback-image-digest: + description: Exact immutable selector-rollback director digest + required: true + type: string + expected-selector-generation: + description: Exact admission selector generation + required: true + type: string + expected-existing-only-cells: + description: Exact existing-only membership, or none + required: true + type: string + expected-migration-only-cells: + description: Exact migration-only membership, or none + required: true + type: string + expected-general-cells: + description: Exact general membership, or none + required: true + type: string + expected-control-generation: + description: Exact durable rehome generation + required: true + type: string + not-before: + description: Exact epoch milliseconds; ignored only by inspect + required: true + default: '0' + type: string + rate-per-minute: + description: Exact global host rate; initial enable is fixed at 10 + required: true + default: '10' + type: string + preference-max-age-ms: + description: Maximum fresh preference age + required: true + default: '86400000' + type: string + drain-grace-ms: + description: Per-host source drain grace + required: true + default: '3600000' + type: string + monitor-run-id: + description: Fresh successful aggregate dry-run required only by enable + required: false + type: string + monitor-run-attempt: + description: Exact monitor attempt required only by enable + required: false + type: string + confirmation: + description: ENABLE_REGIONAL_REHOMING, PAUSE_REGIONAL_REHOMING, or DISABLE_REGIONAL_REHOMING + required: false + type: string + +permissions: + actions: read + contents: read + id-token: write + +concurrency: + group: production-cloud-sql-rollout + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + operate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + uses: ./.github/workflows/cloud-operate-relay-production-rehome-job.yml + with: + mode: ${{ inputs.mode }} + director-image-digest: ${{ inputs.director-image-digest }} + rollback-image-digest: ${{ inputs.rollback-image-digest }} + expected-selector-generation: ${{ inputs.expected-selector-generation }} + expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }} + expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }} + expected-general-cells: ${{ inputs.expected-general-cells }} + expected-control-generation: ${{ inputs.expected-control-generation }} + not-before: ${{ inputs.not-before }} + rate-per-minute: ${{ inputs.rate-per-minute }} + preference-max-age-ms: ${{ inputs.preference-max-age-ms }} + drain-grace-ms: ${{ inputs.drain-grace-ms }} + confirmation: ${{ inputs.confirmation }} + monitor-run-id: ${{ inputs.monitor-run-id }} + monitor-run-attempt: ${{ inputs.monitor-run-attempt }} + secrets: inherit diff --git a/.github/workflows/cloud-power-relay-staging.yml b/.github/workflows/cloud-power-relay-staging.yml new file mode 100644 index 00000000000..0e8311e4ec2 --- /dev/null +++ b/.github/workflows/cloud-power-relay-staging.yml @@ -0,0 +1,101 @@ +name: Power Relay Staging + +on: + schedule: + # A zero-activity guard makes this a no-op when an internal test is still running. + - cron: '0 9 * * *' + workflow_dispatch: + inputs: + mode: + description: Inspect, wake, or sleep the staging Relay data plane + required: true + default: status + type: choice + options: + - status + - wake + - sleep + wake-cells: + description: Wake configured admission cells, or include disabled candidate cells + required: true + default: configured + type: choice + options: + - configured + - all + confirmation: + description: Enter WAKE_STAGING or SLEEP_STAGING for a manual mutation + required: false + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + power: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: staging + env: + POWER_MODE: ${{ github.event_name == 'schedule' && 'sleep' || inputs.mode }} + WAKE_CELLS: ${{ inputs.wake-cells || 'configured' }} + steps: + - uses: actions/checkout@v4 + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Require explicit manual mutation confirmation + if: ${{ github.event_name == 'workflow_dispatch' && inputs.mode != 'status' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: | + if [[ "${POWER_MODE}" = "wake" ]]; then + test "${CONFIRMATION}" = "WAKE_STAGING" + else + test "${CONFIRMATION}" = "SLEEP_STAGING" + fi + + - name: Read reviewed staging topology + run: | + node dev/scripts/infra.mjs init --env staging + terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json" + + - name: Inspect or change staging power state + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/power-staging-relay.mjs \ + --mode "${POWER_MODE}" \ + --wake-cells "${WAKE_CELLS}" \ + --topology-file "${RUNNER_TEMP}/relay-gce-topology.json" diff --git a/.github/workflows/cloud-prove-relay-asia-staging.yml b/.github/workflows/cloud-prove-relay-asia-staging.yml new file mode 100644 index 00000000000..9275dedc536 --- /dev/null +++ b/.github/workflows/cloud-prove-relay-asia-staging.yml @@ -0,0 +1,329 @@ +name: Prove Relay Asia Staging + +on: + workflow_dispatch: + inputs: + image-digest: + description: Exact immutable Relay digest deployed on staging C4 + required: true + type: string + selector-generation: + description: Exact selector generation with C4 migration-only + required: true + type: string + promote-attempt-id: + description: Durable unique C4 promotion attempt ID + required: true + type: string + rollback-attempt-id: + description: Durable unique C4 rollback attempt ID + required: true + type: string + confirmation: + description: Enter PROVE_ASIA_STAGING + required: true + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + prove: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }} + runs-on: [self-hosted, linux, x64, relay-asia-east2-load] + timeout-minutes: 75 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + AUTH_ORIGIN: https://auth-staging.onorca.dev + IMAGE_DIGEST: ${{ inputs.image-digest }} + INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }} + PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }} + ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }} + steps: + - uses: actions/checkout@v4 + + - name: Validate the exact staging proof request + shell: bash + run: | + set -euo pipefail + test "${{ inputs.confirmation }}" = PROVE_ASIA_STAGING + [[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]] + [[ "${INITIAL_SELECTOR_GENERATION}" =~ ^[1-9][0-9]*$ ]] + [[ "${PROMOTE_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + [[ "${ROLLBACK_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]] + test "${PROMOTE_ATTEMPT_ID}" != "${ROLLBACK_ATTEMPT_ID}" + + - id: auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - uses: pnpm/action-setup@v4 + + - name: Require the exact staging director image before promotion + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + runtime="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')" + test "$(jq -r '.role' <<< "${runtime}")" = director + test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}" + + - name: Install exact load-harness dependencies + run: pnpm install --frozen-lockfile + + - name: Build the Relay load-harness contract + run: pnpm --filter @orca-cloud/relay-contract build + + - name: Promote only staging C4 + id: promote + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging \ + --mode promote \ + --cell-ids staging-gce-c4 \ + --expected-generation "${INITIAL_SELECTOR_GENERATION}" \ + --attempt-id "${PROMOTE_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + generation="$(jq -er '.generation' <<< "${result}")" + test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = general + echo "generation=${generation}" >> "${GITHUB_OUTPUT}" + echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}" + + - name: Run sharded two-horizon and mixed splice proofs + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof" + mkdir -p "${proof_dir}" + fd_limit="$(ulimit -n)" + if test "${fd_limit}" != unlimited; then + [[ "${fd_limit}" =~ ^[0-9]+$ ]] + test "${fd_limit}" -ge 4096 + fi + run_phase() { + phase="$1" + controls="$2" + splices="$3" + pids=() + stop_shards() { + for pid in "${pids[@]}"; do kill "${pid}" 2>/dev/null || true; done + for pid in "${pids[@]}"; do wait "${pid}" 2>/dev/null || true; done + } + trap stop_shards EXIT + for shard in 0 1 2 3; do + slow=0 + wedged=0 + boundary_args=() + request_unit_args=() + if test "${phase}" = launch && test "${shard}" = 0; then + slow=4 + wedged=1 + fi + if test "${phase}" = launch && test "${shard}" = 0; then + boundary_args=( + --region-behavior-probes 1 + --capacity-cell-id staging-gce-c4 + --capacity-cell-origin https://c4.relay-staging.onorca.dev + --capacity-unobserved-bound 60 + --rebind-probes 2 + --skip-rebind-overflow-check + ) + fi + node dev/scripts/load-relay-controls.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --auth-origin "${AUTH_ORIGIN}" \ + --preferred-region asia-east2 \ + --relay-asia-load-principals 32 \ + --controls "${controls}" \ + --splices "${splices}" \ + --slow-reader-splices "${slow}" \ + --wedged-reader-splices "${wedged}" \ + --capacity-hard-cap 3000 \ + "${boundary_args[@]}" \ + "${request_unit_args[@]}" \ + --phase-barrier-dir "${proof_dir}/${phase}-barrier" \ + --aggregate-controls "$((controls * 4))" \ + --aggregate-splices "$((splices * 4))" \ + --aggregate-reader-splices "$([[ "${phase}" = launch ]] && echo 5 || echo 0)" \ + --aggregate-reader-bytes "$([[ "${phase}" = launch ]] && echo 12582912 || echo 0)" \ + --required-lease-horizons 2 \ + --splice-ramp-seconds 120 \ + --max-generator-rss-growth-mib 512 \ + --ramp-seconds 180 \ + --duration-seconds 210 \ + --shard-count 4 \ + --shard-index "${shard}" \ + > "${proof_dir}/${phase}-${shard}.jsonl" & + pids+=("$!") + done + failed=0 + for pid in "${pids[@]}"; do + if ! wait "${pid}"; then failed=1; break; fi + done + if test "${failed}" = 1; then + stop_shards + for shard in 0 1 2 3; do + jq -cer 'select(.event == "relay_load_progress" or .event == "relay_load_complete") | + {event, shardIndex, active, peakActive, connected, connectionFailures, + rampConnectionFailures, connectionFailuresByReason, unexpectedCloses, + protocolErrors, refreshErrors, socketErrors, elapsedSeconds}' \ + "${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1 || true + done + trap - EXIT + return 1 + fi + trap - EXIT + for shard in 0 1 2 3; do + jq -cer 'select(.event == "relay_load_complete")' \ + "${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1 + done | jq -s . > "${proof_dir}/${phase}.json" + } + run_phase launch 5 5 + + - name: Collect and validate aggregate staging proof evidence + env: + PROOF_STARTED_AT: ${{ steps.promote.outputs.started_at }} + shell: bash + run: | + set -euo pipefail + proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof" + ended_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + sleep 60 + gcloud logging read \ + "timestamp>=\"${PROOF_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \ + --project "${GCP_PROJECT_ID}" --limit 20000 --format json \ + > "${proof_dir}/runtime-metrics.json" + node dev/scripts/relay-asia-rollout-evidence.mjs create-staging \ + --repository "${GITHUB_REPOSITORY}" \ + --run-id "${GITHUB_RUN_ID}" \ + --run-attempt "${GITHUB_RUN_ATTEMPT}" \ + --commit-sha "${GITHUB_SHA}" \ + --image-digest "${IMAGE_DIGEST}" \ + --selector-generation "${{ steps.promote.outputs.generation }}" \ + --started-at "${PROOF_STARTED_AT}" \ + --ended-at "${ended_at}" \ + --launch-report "${proof_dir}/launch.json" \ + --logs-json "${proof_dir}/runtime-metrics.json" \ + --output "${proof_dir}/evidence.json" + + - name: Return staging C4 to migration-only + if: ${{ always() && steps.promote.outcome != 'skipped' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging \ + --mode recover-promotion \ + --cell-ids staging-gce-c4 \ + --expected-generation "${INITIAL_SELECTOR_GENERATION}" \ + --attempt-id "${PROMOTE_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi + promoted_generation="$(jq -er '.generation' <<< "${promoted}")" + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging \ + --mode rollback \ + --cell-ids staging-gce-c4 \ + --expected-generation "${promoted_generation}" \ + --attempt-id "${ROLLBACK_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only + + - name: Upload immutable staging readiness evidence + if: ${{ success() }} + uses: actions/upload-artifact@v4 + with: + name: relay-asia-staging-${{ github.run_id }}-${{ github.run_attempt }} + path: ${{ runner.temp }}/relay-asia-staging-proof/evidence.json + if-no-files-found: error + retention-days: 7 + + recover: + if: ${{ always() && github.ref == 'refs/heads/main' }} + needs: prove + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 15 + environment: staging + env: + IMAGE_DIGEST: ${{ inputs.image-digest }} + INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }} + PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }} + ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }} + steps: + - uses: actions/checkout@v4 + + - id: auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Recover staging C4 with a fresh identity + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging \ + --mode recover-promotion \ + --cell-ids staging-gce-c4 \ + --expected-generation "${INITIAL_SELECTOR_GENERATION}" \ + --attempt-id "${PROMOTE_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi + promoted_generation="$(jq -er '.generation' <<< "${promoted}")" + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging \ + --mode rollback \ + --cell-ids staging-gce-c4 \ + --expected-generation "${promoted_generation}" \ + --attempt-id "${ROLLBACK_ATTEMPT_ID}" \ + --image-digest "${IMAGE_DIGEST}")" + test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only diff --git a/.github/workflows/cloud-prove-relay-staging-capacity.yml b/.github/workflows/cloud-prove-relay-staging-capacity.yml new file mode 100644 index 00000000000..7098078b5ba --- /dev/null +++ b/.github/workflows/cloud-prove-relay-staging-capacity.yml @@ -0,0 +1,917 @@ +name: Prove Relay Staging Capacity + +on: + workflow_dispatch: + inputs: + mode: + description: Verify or change C3 capacity, restore admission, or refresh empty Asia C4 + required: true + default: verify + type: choice + options: + - verify + - apply + - restore-admission + - refresh-asia-c4-image + expected-hard-cap: + description: Exact cap declared for staging-gce-c3 in the reviewed staging tfvars + required: true + default: '600' + type: choice + options: + - '1000' + - '600' + expected-unobserved-bound: + description: Exact bound declared for staging-gce-c3 in the reviewed staging tfvars + required: true + default: '60' + type: choice + options: + - '60' + - '0' + confirmation: + description: Exact confirmation required for a mutation + required: false + type: string + expected-selector-generation: + description: Exact staging selector generation for a C4 image refresh + required: false + type: string + predecessor-image-digest: + description: Exact current C4 sha256 digest + required: false + type: string + target-image-digest: + description: Exact desired C4 sha256 digest + required: false + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: relay-staging-mutation + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + capacity: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (vars.STAGING_GCP_REGION != '' && inputs.mode != 'refresh-asia-c4-image') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + GCP_REGION: ${{ vars.STAGING_GCP_REGION }} + DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + CELL_ORIGIN: https://c3.relay-staging.onorca.dev + TARGET_CELL_ID: staging-gce-c3 + FALLBACK_CELL_ORIGIN: https://c2.relay-staging.onorca.dev + FALLBACK_CELL_ID: staging-gce-c2 + CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + EXPECTED_HARD_CAP: ${{ inputs.expected-hard-cap }} + EXPECTED_UNOBSERVED_BOUND: ${{ inputs.expected-unobserved-bound }} + steps: + - uses: actions/checkout@v4 + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + if: ${{ inputs.mode != 'restore-admission' }} + with: + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Initialize the exact staging backend + if: ${{ inputs.mode != 'restore-admission' }} + run: node dev/scripts/infra.mjs init --env staging + + - name: Require reviewed desired capacity and image + if: ${{ inputs.mode != 'restore-admission' }} + shell: bash + run: | + CAP_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_hard_cap" + BOUND_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_unobserved_bound" + IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image" + ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone" + DESIRED_CAP="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars <<< "${CAP_EXPRESSION}")" + DESIRED_BOUND="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars <<< "${BOUND_EXPRESSION}")" + DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars <<< "${IMAGE_EXPRESSION}" | jq -r '.')" + TARGET_ZONE="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars <<< "${ZONE_EXPRESSION}" | jq -r '.')" + MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \ + | jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')" + DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars \ + <<< 'local.relay_director_cells_json' | jq -r '.')" + CELL_ORIGIN="$(jq -r --arg cell "${TARGET_CELL_ID}" \ + '.[] | select(.id == $cell) | .url' <<< "${DESIRED_CELLS_JSON}")" + test "${DESIRED_CAP}" = "${EXPECTED_HARD_CAP}" + test "${DESIRED_BOUND}" = "${EXPECTED_UNOBSERVED_BOUND}" + [[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]] + [[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]] + [[ "${MIG_NAME}" =~ ^[a-z0-9-]+$ ]] + test "${CELL_ORIGIN}" = "https://c3.relay-staging.onorca.dev" + jq -e \ + --arg cell "${TARGET_CELL_ID}" \ + --arg fallback "${FALLBACK_CELL_ID}" \ + --argjson cap "${EXPECTED_HARD_CAP}" \ + --argjson bound "${EXPECTED_UNOBSERVED_BOUND}" \ + '(any(.[]; .id == $cell and .connectionHardCap == $cap and .connectionUnobservedBound == $bound)) and + (any(.[]; .id == $fallback and .connectionHardCap == 600 and .connectionUnobservedBound == 60))' \ + <<< "${DESIRED_CELLS_JSON}" >/dev/null + echo "DESIRED_IMAGE=${DESIRED_IMAGE}" >> "${GITHUB_ENV}" + echo "TARGET_ZONE=${TARGET_ZONE}" >> "${GITHUB_ENV}" + echo "MIG_NAME=${MIG_NAME}" >> "${GITHUB_ENV}" + echo "CELL_ORIGIN=${CELL_ORIGIN}" >> "${GITHUB_ENV}" + echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" >> "${GITHUB_ENV}" + + - name: Verify exact compatible director image + if: ${{ inputs.mode != 'restore-admission' }} + shell: bash + run: | + SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)" + ACTIVE_REVISION="$(jq -r \ + '[.status.traffic[] | select((.percent // 0) > 0)] + | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \ + <<< "${SERVICE_JSON}")" + test -n "${ACTIVE_REVISION}" + ACTIVE_IMAGE="$(gcloud run revisions describe "${ACTIVE_REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \ + --format='value(spec.containers[0].image)')" + test "${ACTIVE_IMAGE}" = "${DESIRED_IMAGE}" + echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}" >> "${GITHUB_ENV}" + + - name: Require explicit transition confirmation + if: ${{ inputs.mode == 'apply' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: test "${CONFIRMATION}" = "FENCE_AND_TRANSITION_STAGING_C3" + + - name: Require explicit admission restore confirmation + if: ${{ inputs.mode == 'restore-admission' }} + env: + CONFIRMATION: ${{ inputs.confirmation }} + run: test "${CONFIRMATION}" = "RESTORE_STAGING_C2_C3_GENERAL" + + - name: Require capacity identity and exact predecessor state + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + case "${EXPECTED_HARD_CAP}/${EXPECTED_UNOBSERVED_BOUND}" in + 1000/0) + PREDECESSOR_C3_CAP=600 + PREDECESSOR_C3_BOUND=60 + ;; + 1000/60) + PREDECESSOR_C3_CAP=1000 + PREDECESSOR_C3_BOUND=0 + ;; + 600/60) + PREDECESSOR_C3_CAP=1000 + PREDECESSOR_C3_BOUND=60 + ;; + *) + echo "Unsupported staging capacity transition" >&2 + exit 1 + ;; + esac + ACTIVE_REVISION="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -r ' + [.status.traffic[] | select((.percent // 0) > 0)] | + if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')" + test -n "${ACTIVE_REVISION}" + CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \ + --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \ + | jq -cer ' + [.spec.containers[0].env[]? | + select(.name == "ORCA_RELAY_CELLS_JSON") | .value] | + if length == 1 then .[0] | fromjson else error("missing director topology") end')" + PREDECESSOR_CELLS_JSON="$(jq -ce \ + --arg cell "${TARGET_CELL_ID}" \ + --argjson cap "${PREDECESSOR_C3_CAP}" \ + --argjson bound "${PREDECESSOR_C3_BOUND}" \ + 'map(if .id == $cell then . + { + connectionHardCap: $cap, + connectionUnobservedBound: $bound + } else . end)' <<< "${DESIRED_CELLS_JSON}")" + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${FALLBACK_CELL_ORIGIN}" \ + --cell-id "${FALLBACK_CELL_ID}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission either \ + --draining forbidden \ + --activity allowed + if jq -ne \ + --argjson current "${CURRENT_CELLS_JSON}" \ + --argjson expected "${DESIRED_CELLS_JSON}" \ + '$current == $expected'; then + if node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed; then + TRANSITION_PHASE=cell-active + elif node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission migration-only \ + --draining forbidden \ + --activity allowed; then + TRANSITION_PHASE=cell-ready + else + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity restart-safe + TRANSITION_PHASE=director-ready + fi + else + jq -ne \ + --argjson current "${CURRENT_CELLS_JSON}" \ + --argjson expected "${PREDECESSOR_CELLS_JSON}" \ + '$current == $expected' + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${PREDECESSOR_C3_CAP}" \ + --unobserved-bound "${PREDECESSOR_C3_BOUND}" \ + --heartbeat fresh \ + --admission either \ + --draining either \ + --activity allowed + TRANSITION_PHASE=predecessor + fi + echo "TRANSITION_PHASE=${TRANSITION_PHASE}" >> "${GITHUB_ENV}" + + - name: Restore C2 as the safe placement fallback + if: ${{ inputs.mode == 'restore-admission' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${FALLBACK_CELL_ORIGIN}" \ + --cell-id "${FALLBACK_CELL_ID}" \ + --heartbeat fresh \ + --admission either \ + --draining forbidden \ + --activity allowed + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode restore-fallback \ + --general-cell-ids "${FALLBACK_CELL_ID}" + + - name: Require a healthy non-draining C3 before restoring it + if: ${{ inputs.mode == 'restore-admission' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission either \ + --draining forbidden \ + --activity allowed + + - name: Require a healthy general C2 before isolating C3 + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + if test "${TRANSITION_PHASE}" = cell-active; then + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode restore-fallback \ + --general-cell-ids "${FALLBACK_CELL_ID}" + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${FALLBACK_CELL_ORIGIN}" \ + --cell-id "${FALLBACK_CELL_ID}" \ + --hard-cap 600 \ + --unobserved-bound 60 \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + + - name: Reversibly isolate and drain C3 + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + if test "${TRANSITION_PHASE}" = cell-ready; then + exit 0 + fi + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode isolate + + - name: Verify restart-safe migration-only target + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + if test "${TRANSITION_PHASE}" = cell-ready; then + exit 0 + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --heartbeat either \ + --admission migration-only \ + --draining required \ + --activity restart-safe + + - name: Verify current capacity + if: ${{ inputs.mode == 'verify' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + + - name: Deploy reviewed director topology and remove pre-protocol revisions + if: ${{ inputs.mode == 'apply' }} + run: | + if test "${TRANSITION_PHASE}" != predecessor; then + echo "DIRECTOR_CONFIG_CHANGED=false" >> "${GITHUB_ENV}" + exit 0 + fi + RELEASE_ID="capacity-compat-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}" + DEPLOY_RESULT="$(node dev/scripts/deploy-relay-blue-green.mjs \ + --project "${GCP_PROJECT_ID}" \ + --region "${GCP_REGION}" \ + --service "${DIRECTOR_SERVICE_NAME}" \ + --image "${ACTIVE_IMAGE}" \ + --role director \ + --capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \ + --capacity-cell-id "${TARGET_CELL_ID}" \ + --director-cells-json "${DESIRED_CELLS_JSON}" \ + --min-instances 0 \ + --prune-revisions true \ + --release-id "${RELEASE_ID}")" + echo "${DEPLOY_RESULT}" + DIRECTOR_CONFIG_CHANGED="$(jq -r '.topologyChanged' <<< "${DEPLOY_RESULT}")" + [[ "${DIRECTOR_CONFIG_CHANGED}" =~ ^(true|false)$ ]] + echo "DIRECTOR_CONFIG_CHANGED=${DIRECTOR_CONFIG_CHANGED}" >> "${GITHUB_ENV}" + + - name: Require fail-closed director transition + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + if test "${TRANSITION_PHASE}" = cell-ready; then + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission migration-only \ + --draining forbidden \ + --activity allowed + exit 0 + fi + HEARTBEAT_EXPECTATION=either + if test "${DIRECTOR_CONFIG_CHANGED}" = true; then + HEARTBEAT_EXPECTATION=stale + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat "${HEARTBEAT_EXPECTATION}" \ + --admission migration-only \ + --draining required \ + --activity restart-safe + + - name: Plan and apply only the exact empty cell + if: ${{ inputs.mode == 'apply' }} + shell: bash + run: | + recreate_fixed_one_instance() { + local instance + instance="$(gcloud compute instance-groups managed list-instances \ + "${MIG_NAME}" \ + --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" \ + --format=json \ + | jq -er ' + if length == 1 and .[0].instanceStatus == "RUNNING" and + .[0].currentAction == "NONE" + then .[0].instance | split("/") | last + else error("capacity cell does not have one stable running instance") end')" + gcloud compute instance-groups managed recreate-instances \ + "${MIG_NAME}" \ + --instances "${instance}" \ + --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" \ + --quiet + } + + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c3"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c3"]' \ + -out="${RUNNER_TEMP}/relay-capacity-cell.tfplan" + PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \ + "${RUNNER_TEMP}/relay-capacity-cell.tfplan" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode cell \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --image "${DESIRED_IMAGE}")" + echo "${PLAN_RESULT}" + CELL_PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")" + [[ "${CELL_PLAN_CHANGES}" =~ ^(0|2)$ ]] + if test "${TRANSITION_PHASE}" = cell-ready; then + test "${CELL_PLAN_CHANGES}" = 0 + elif test "${TRANSITION_PHASE}" = cell-active; then + test "${CELL_PLAN_CHANGES}" = 0 + recreate_fixed_one_instance + elif test "${CELL_PLAN_CHANGES}" = 0; then + recreate_fixed_one_instance + else + terraform -chdir=infra/terraform apply \ + -auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan" + fi + gcloud compute instance-groups managed wait-until \ + "${MIG_NAME}" \ + --stable \ + --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" \ + --timeout 900 + + - name: Verify exact live cap and fresh matching heartbeat + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission migration-only \ + --draining forbidden \ + --activity allowed + + - name: Make C3 the only staging placement cell + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode activate + + - name: Verify the sole general canary after transition + if: ${{ inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + + - name: Restore the reviewed C2 and C3 placement set + if: ${{ inputs.mode == 'restore-admission' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode restore \ + --general-cell-ids staging-gce-c2,staging-gce-c3 + + - name: Verify restored C3 admission and capacity + if: ${{ inputs.mode == 'restore-admission' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --hard-cap "${EXPECTED_HARD_CAP}" \ + --unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \ + --heartbeat fresh \ + --admission general \ + --draining forbidden \ + --activity allowed + + - name: Preserve C2 as the safe fallback after a failed transition + if: ${{ failure() && inputs.mode == 'apply' }} + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }} + run: | + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" \ + --mode restore-fallback \ + --general-cell-ids "${FALLBACK_CELL_ID}" + + refresh-asia-c4-image: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main' && vars.STAGING_GCP_REGION != '' && inputs.mode == 'refresh-asia-c4-image') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 90 + environment: staging + env: + GCP_PROJECT_ID: onorca-cloud-staging + GCP_REGION: ${{ vars.STAGING_GCP_REGION }} + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + CELL_ORIGIN: https://c4.relay-staging.onorca.dev + TARGET_CELL_ID: staging-gce-c4 + EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }} + PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }} + TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }} + APPROVED_PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7 + steps: + - uses: actions/checkout@v4 + + - name: Require the exact bounded C4 refresh + env: + CONFIRMATION: ${{ inputs.confirmation }} + shell: bash + run: | + set -euo pipefail + test "${CONFIRMATION}" = REFRESH_STAGING_ASIA_C4_IMAGE + [[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]] + [[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + [[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + test "${PREDECESSOR_IMAGE_DIGEST}" != "${TARGET_IMAGE_DIGEST}" + test "${PREDECESSOR_IMAGE_DIGEST}" = "${APPROVED_PREDECESSOR_IMAGE_DIGEST}" + + - id: google-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: 1.15.8 + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Initialize the exact staging backend + run: node dev/scripts/infra.mjs init --env staging + + - name: Resolve the reviewed C4 image and shape + shell: bash + run: | + set -euo pipefail + cells="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')" + shape="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${cells}")" + test "$(jq -r '.hostname' <<< "${shape}")" = c4 + test "$(jq -r '.region' <<< "${shape}")" = asia-east2 + test "$(jq -r '.zone' <<< "${shape}")" = asia-east2-a + test "$(jq -r '.machine_type' <<< "${shape}")" = e2-standard-4 + test "$(jq -r '.capacity_requests' <<< "${shape}")" = 6000 + test "$(jq -r '.database_pool_max' <<< "${shape}")" = 10 + test "$(jq -r '.connection_hard_cap' <<< "${shape}")" = 3000 + test "$(jq -r '.connection_unobserved_bound' <<< "${shape}")" = 60 + test "$(jq -r '.initially_enabled' <<< "${shape}")" = false + desired_image="$(jq -r '.image' <<< "${shape}")" + test "${desired_image}" = \ + "us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${TARGET_IMAGE_DIGEST}" + served_digest="$(gcloud artifacts docker images describe "${desired_image}" \ + --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')" + test "${served_digest}" = "${TARGET_IMAGE_DIGEST}" + mig_name="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \ + | jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')" + test "${mig_name}" = orca-cloud-staging-relay-gce-c4 + { + echo "DESIRED_IMAGE=${desired_image}" + echo "ROLLBACK_IMAGE=us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${PREDECESSOR_IMAGE_DIGEST}" + echo "TARGET_ZONE=asia-east2-a" + echo "MIG_NAME=${mig_name}" + } >> "${GITHUB_ENV}" + + - name: Save, validate, and classify the exact C4 plan + id: plan + shell: bash + run: | + set -euo pipefail + plan="${RUNNER_TEMP}/relay-c4-image-refresh.tfplan" + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \ + -out="${plan}" + result="$(terraform -chdir=infra/terraform show -json "${plan}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \ + --unobserved-bound 60 --image "${DESIRED_IMAGE}" \ + --rollback-image "${ROLLBACK_IMAGE}")" + case "$(jq -r '[.changes,.changeKind] | join(":")' <<< "${result}")" in + 2:replacement) refresh_phase=predecessor ;; + 0:none|*:obsolete-template-delete) refresh_phase=applied ;; + *:manager-convergence|*:replacement-with-obsolete-template) refresh_phase=converging ;; + *) exit 1 ;; + esac + { + echo "PLAN_CHANGES=$(jq -r '.changes' <<< "${result}")" + echo "REFRESH_PHASE=${refresh_phase}" + } >> "${GITHUB_ENV}" + + - id: state-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Verify the exact selector and current C4 state + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.state-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \ + --expected-generation '' --expected-membership-sha256 '' --attempt-id '' \ + --image-digest "${TARGET_IMAGE_DIGEST}")" + test "$(jq -r '.generation' <<< "${inspect}")" = "${EXPECTED_SELECTOR_GENERATION}" + test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \ + migration-only + expected_digests="${TARGET_IMAGE_DIGEST}" + if test "${REFRESH_PHASE}" = predecessor; then + expected_digests="${PREDECESSOR_IMAGE_DIGEST}" + elif test "${REFRESH_PHASE}" = converging; then + expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}" + fi + if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \ + "${CELL_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')"; then + current_digest="$(jq -er '.imageDigest' <<< "${runtime}")" + case ",${expected_digests}," in + *,"${current_digest}",*) ;; + *) exit 1 ;; + esac + source_incarnation='' + draining=forbidden + if test "${REFRESH_PHASE}" != applied; then + status="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + source_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")" + [[ "${source_incarnation}" =~ ^[0-9a-f-]{36}$ ]] + if test "$(jq -r '.draining' <<< "${runtime}")" = true; then + draining=required + fi + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \ + --heartbeat fresh --admission migration-only --draining "${draining}" \ + --activity quiescent --expected-image-digests "${expected_digests}" + runtime_available=true + else + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \ + --admission migration-only --draining either --activity restart-safe \ + --timeout-ms 300000 + source_incarnation='' + runtime_available=false + fi + { + echo "MIG_STABLE_AT_MS=0" + echo "MUTATION_STARTED=false" + echo "REPLACEMENT_STARTED_AT_MS=0" + echo "RUNTIME_AVAILABLE=${runtime_available}" + echo "SOURCE_INCARNATION=${source_incarnation}" + } >> "${GITHUB_ENV}" + + - id: fence-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Fence C4 and prove it stayed empty before replacement + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + if test "${REFRESH_PHASE}" = applied; then exit 0; fi + echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + if test "${RUNTIME_AVAILABLE}" = false; then exit 0; fi + node dev/scripts/prepare-relay-capacity-canary.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --mode isolate + fence_digests="${PREDECESSOR_IMAGE_DIGEST}" + if test "${REFRESH_PHASE}" = converging; then + fence_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}" + fi + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \ + --heartbeat fresh --admission migration-only --draining required \ + --activity quiescent --expected-image-digests "${fence_digests}" + + - name: Apply the exact saved C4 plan + shell: bash + run: | + set -euo pipefail + if test "${PLAN_CHANGES}" = 0 && test "${RUNTIME_AVAILABLE}" = true; then exit 0; fi + if test "${REFRESH_PHASE}" = applied && test "${RUNTIME_AVAILABLE}" = false; then + echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}" + fi + if test "${REFRESH_PHASE}" != applied; then + replacement_started_at_ms="$(date -u +%s%3N)" + echo "REPLACEMENT_STARTED_AT_MS=${replacement_started_at_ms}" >> "${GITHUB_ENV}" + fi + if test "${PLAN_CHANGES}" != 0; then + terraform -chdir=infra/terraform apply -auto-approve \ + "${RUNNER_TEMP}/relay-c4-image-refresh.tfplan" + fi + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + if test "${RUNTIME_AVAILABLE}" = false && test "${REFRESH_PHASE}" = applied; then + instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \ + | jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and + .[0].currentAction == "NONE" then .[0].instance | split("/") | last + else error("C4 is not one stable running instance") end')" + gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \ + --instances "${instance}" --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" --quiet + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + fi + echo "MIG_STABLE_AT_MS=$(date -u +%s%3N)" >> "${GITHUB_ENV}" + + - id: post-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Verify new C4 incarnation, image, and unchanged isolation + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \ + --heartbeat fresh --admission migration-only --draining forbidden \ + --activity quiescent --expected-image-digests "${TARGET_IMAGE_DIGEST}" \ + --timeout-ms 240000 + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \ + --expected-generation "${EXPECTED_SELECTOR_GENERATION}" \ + --expected-membership-sha256 '' --attempt-id '' \ + --image-digest "${TARGET_IMAGE_DIGEST}")" + test "$(jq -r '.generation' <<< "${result}")" = "${EXPECTED_SELECTOR_GENERATION}" + test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \ + migration-only + for _ in $(seq 1 36); do + status="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \ + test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \ + -ge "${MIG_STABLE_AT_MS}"; then break; fi + sleep 5 + done + target_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")" + test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true + test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \ + -ge "${MIG_STABLE_AT_MS}" + if test "${REFRESH_PHASE}" != applied; then + if test -n "${SOURCE_INCARNATION}"; then + test "${target_incarnation}" != "${SOURCE_INCARNATION}" + fi + test "$(jq -r '.status.runtime.startedAt' <<< "${status}")" \ + -ge "${REPLACEMENT_STARTED_AT_MS}" + fi + + - name: Require an empty targeted Terraform readback + shell: bash + run: | + set -euo pipefail + plan="${RUNNER_TEMP}/relay-c4-image-readback.tfplan" + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \ + -out="${plan}" + result="$(terraform -chdir=infra/terraform show -json "${plan}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \ + --unobserved-bound 60 --image "${DESIRED_IMAGE}" \ + --rollback-image "${ROLLBACK_IMAGE}")" + test "$(jq -r '.changes' <<< "${result}")" = 0 diff --git a/.github/workflows/cloud-publish-relay-production.yml b/.github/workflows/cloud-publish-relay-production.yml new file mode 100644 index 00000000000..d7bb42b4c55 --- /dev/null +++ b/.github/workflows/cloud-publish-relay-production.yml @@ -0,0 +1,131 @@ +name: Publish Relay Production Image + +on: + workflow_dispatch: + inputs: + mode: + description: Publish a new production image or mirror an existing immutable image to staging + required: true + default: publish + type: choice + options: [publish, mirror-staging] + image-digest: + description: Exact existing production digest for mirror-staging mode + required: false + type: string + confirmation: + description: Enter MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING for mirror-staging mode + required: false + type: string + +permissions: + contents: read + id-token: write + +concurrency: + group: publish-relay-production + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + publish: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + environment: production + env: + GCP_PROJECT_ID: onorca-cloud + GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }} + REPOSITORY_ID: orca-cloud + IMAGE_NAME: relay + PUBLISH_MODE: ${{ inputs.mode }} + MIRROR_DIGEST: ${{ inputs.image-digest }} + MIRROR_CONFIRMATION: ${{ inputs.confirmation }} + steps: + - uses: actions/checkout@v4 + + - name: Validate the exact publish request before authentication + shell: bash + run: | + set -euo pipefail + if test "${PUBLISH_MODE}" = mirror-staging; then + [[ "${MIRROR_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]] + test "${MIRROR_CONFIRMATION}" = MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING + else + test "${PUBLISH_MODE}" = publish + test -z "${MIRROR_DIGEST}" + test -z "${MIRROR_CONFIRMATION}" + fi + + - uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: docker/setup-buildx-action@v3 + + - name: Configure Docker auth + run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet + + - name: Build and publish immutable image + if: ${{ inputs.mode == 'publish' }} + run: | + IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}:sha-${GITHUB_SHA}" + docker build -f apps/relay/Dockerfile -t "${IMAGE_TAG}" . + docker push "${IMAGE_TAG}" + DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')" + test -n "${DIGEST}" + IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${DIGEST}" + { + echo '### Terraform candidate image' + echo + echo "\`${IMAGE}\`" + echo + echo 'Declare this digest on a distinct disabled candidate cell in a reviewed Terraform PR.' + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Build and publish immutable fence broker + if: ${{ inputs.mode == 'publish' }} + run: | + IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker:sha-${GITHUB_SHA}" + docker build \ + --build-arg "ORCA_RELAY_FENCE_IMAGE_COMMIT=${GITHUB_SHA}" \ + -f apps/relay-fence-broker/Dockerfile \ + -t "${IMAGE_TAG}" . + docker push "${IMAGE_TAG}" + DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')" + test -n "${DIGEST}" + IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker@${DIGEST}" + { + echo + echo '### Terraform fence broker image' + echo + echo "\`${IMAGE}\`" + } >> "${GITHUB_STEP_SUMMARY}" + + - name: Mirror the exact production manifest to staging + if: ${{ inputs.mode == 'mirror-staging' }} + shell: bash + run: | + set -euo pipefail + source_image="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${MIRROR_DIGEST}" + target_tag="${GCP_REGION}-docker.pkg.dev/onorca-cloud-staging/${REPOSITORY_ID}/${IMAGE_NAME}:production-${MIRROR_DIGEST#sha256:}" + source_digest="$(gcloud artifacts docker images describe "${source_image}" \ + --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')" + test "${source_digest}" = "${MIRROR_DIGEST}" + docker pull "${source_image}" + docker tag "${source_image}" "${target_tag}" + docker push "${target_tag}" + target_digest="$(gcloud artifacts docker images describe "${target_tag}" \ + --project onorca-cloud-staging --format='value(image_summary.digest)')" + test "${target_digest}" = "${MIRROR_DIGEST}" + { + echo '### Mirrored Relay image' + echo + printf 'Production and staging now resolve the same immutable digest: %s.\n' \ + "${MIRROR_DIGEST}" + } >> "${GITHUB_STEP_SUMMARY}" diff --git a/.github/workflows/cloud-recover-relay-staging-c4-image.yml b/.github/workflows/cloud-recover-relay-staging-c4-image.yml new file mode 100644 index 00000000000..dc6d4eaaebe --- /dev/null +++ b/.github/workflows/cloud-recover-relay-staging-c4-image.yml @@ -0,0 +1,410 @@ +name: Recover Relay Staging C4 Image + +on: + workflow_run: + workflows: [Prove Relay Staging Capacity] + types: [completed] + workflow_dispatch: + inputs: + confirmation: + description: Enter RECOVER_STAGING_ASIA_C4_IMAGE + required: true + type: string + +permissions: + actions: read + contents: read + id-token: write + +defaults: + run: + working-directory: cloud + +jobs: + gate: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event_name == 'workflow_dispatch' || (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion != 'success')) }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 5 + outputs: + recover: ${{ steps.trigger.outputs.recover }} + steps: + - name: Bind recovery to the exact failed C4 job + id: trigger + env: + CONFIRMATION: ${{ inputs.confirmation }} + GH_TOKEN: ${{ github.token }} + SOURCE_RUN_ID: ${{ github.event.workflow_run.id }} + SOURCE_RUN_EVENT: ${{ github.event.workflow_run.event }} + shell: bash + run: | + set -euo pipefail + if test "${GITHUB_EVENT_NAME}" = workflow_dispatch; then + test "${CONFIRMATION}" = RECOVER_STAGING_ASIA_C4_IMAGE + echo "recover=true" >> "${GITHUB_OUTPUT}" + exit 0 + fi + test "${SOURCE_RUN_EVENT}" = workflow_dispatch + [[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + jobs="$(gh api --paginate \ + "repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")" + count="$(jq -s '[.[].jobs[] | select(.name == "refresh-asia-c4-image" and + (.conclusion == "failure" or .conclusion == "cancelled" or + .conclusion == "timed_out"))] | length' <<< "${jobs}")" + if test "${count}" = 0; then + echo "recover=false" >> "${GITHUB_OUTPUT}" + exit 0 + fi + test "${count}" = 1 + echo "recover=true" >> "${GITHUB_OUTPUT}" + + recover: + needs: gate + if: ${{ needs.gate.outputs.recover == 'true' }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 90 + environment: staging + concurrency: + group: relay-staging-mutation + cancel-in-progress: false + env: + GCP_PROJECT_ID: onorca-cloud-staging + DIRECTOR_ORIGIN: https://relay-staging.onorca.dev + CELL_ORIGIN: https://c4.relay-staging.onorca.dev + TARGET_CELL_ID: staging-gce-c4 + TARGET_ZONE: asia-east2-a + MIG_NAME: orca-cloud-staging-relay-gce-c4 + PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7 + TARGET_IMAGE_DIGEST: sha256:5aedbca5c86de24c8b4d4bf7e3b444b76c712f281ede916cb9d90f70cad1e563 + steps: + - uses: actions/checkout@v4 + + - id: auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + + - uses: google-github-actions/setup-gcloud@v2 + + - uses: ./.github/actions/cloud-sql-rollout-lease + with: + bucket: onorca-cloud-staging-terraform-state + object: terraform/state/cloud-sql-rollout/staging.lock + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: 1.15.8 + terraform_wrapper: false + + - uses: actions/setup-node@v4 + with: + node-version: 24 + + - name: Initialize the exact staging backend + run: node dev/scripts/infra.mjs init --env staging + + - id: preflight-auth + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Inspect the exact C4 recovery state + id: preflight + timeout-minutes: 3 + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.preflight-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \ + --expected-generation '' --expected-membership-sha256 '' --attempt-id '' \ + --image-digest "${PREDECESSOR_IMAGE_DIGEST}")" + generation="$(jq -er '.generation' <<< "${inspect}")" + [[ "${generation}" =~ ^(0|[1-9][0-9]*)$ ]] + test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \ + migration-only + echo "selector_generation=${generation}" >> "${GITHUB_OUTPUT}" + if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \ + "${CELL_ORIGIN}/v1/admin/runtime-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1}')"; then + current_digest="$(jq -er '.imageDigest' <<< "${runtime}")" + [[ "${current_digest}" =~ ^sha256:[a-f0-9]{64}$ ]] + status="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + jq -e '.draining | type == "boolean"' <<< "${runtime}" >/dev/null + { + echo "runtime_available=true" + echo "current_digest=${current_digest}" + echo "draining=$(jq -r '.draining' <<< "${runtime}")" + echo "ready=$(jq -r '.status.runtime.ready == true' <<< "${status}")" + } >> "${GITHUB_OUTPUT}" + else + echo "runtime_available=false" >> "${GITHUB_OUTPUT}" + fi + + - name: Classify both exact recovery end states + id: recovery-plan + timeout-minutes: 10 + shell: bash + run: | + set -euo pipefail + image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay" + predecessor_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}" + target_image="${image_repository}@${TARGET_IMAGE_DIGEST}" + served_digest="$(gcloud artifacts docker images describe "${predecessor_image}" \ + --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')" + test "${served_digest}" = "${PREDECESSOR_IMAGE_DIGEST}" + cells="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')" + test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].image' <<< "${cells}")" = \ + "${target_image}" + target_plan="${RUNNER_TEMP}/relay-c4-image-target.tfplan" + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false -lock-timeout=5m \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \ + -out="${target_plan}" + target_result="$(terraform -chdir=infra/terraform show -json "${target_plan}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \ + --unobserved-bound 60 --image "${target_image}" \ + --rollback-image "${predecessor_image}")" + target_state="$(jq -r '[.changes,.changeKind] | join(":")' <<< "${target_result}")" + case "${target_state}" in + 0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;; + *) exit 1 ;; + esac + recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" \ + --arg image "${predecessor_image}" '.[$cell].image = $image' <<< "${cells}")" + jq -n --argjson cells "${recovery_cells}" \ + '{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-recovery.tfvars.json" + plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan" + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars \ + -var-file="${RUNNER_TEMP}/relay-c4-recovery.tfvars.json" \ + -var manage_artifact_dns=false -lock-timeout=5m \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \ + -out="${plan}" + result="$(terraform -chdir=infra/terraform show -json "${plan}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \ + --unobserved-bound 60 --image "${predecessor_image}" \ + --rollback-image "${target_image}")" + changes="$(jq -r '.changes' <<< "${result}")" + change_kind="$(jq -r '.changeKind' <<< "${result}")" + case "${changes}:${change_kind}" in + 0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;; + *) exit 1 ;; + esac + mig="$(gcloud compute instance-groups managed describe "${MIG_NAME}" \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)" + mig_stable="$(jq -r '.status.isStable == true and .status.versionTarget.isReached == true' \ + <<< "${mig}")" + instances="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)" + instance_stable="$(jq -r 'length == 1 and .[0].instanceStatus == "RUNNING" and + .[0].currentAction == "NONE"' <<< "${instances}")" + action=rollback-predecessor + recovery_digest="${PREDECESSOR_IMAGE_DIGEST}" + if test "${{ steps.preflight.outputs.runtime_available }}" = true && \ + test "${{ steps.preflight.outputs.ready }}" = true && \ + test "${{ steps.preflight.outputs.draining }}" = false && \ + test "${mig_stable}" = true && test "${instance_stable}" = true; then + if test "${{ steps.preflight.outputs.current_digest }}" = "${TARGET_IMAGE_DIGEST}" && \ + test "${target_state}" = 0:none; then + action=verify-target + recovery_digest="${TARGET_IMAGE_DIGEST}" + elif test "${{ steps.preflight.outputs.current_digest }}" = \ + "${PREDECESSOR_IMAGE_DIGEST}" && test "${changes}:${change_kind}" = 0:none; then + action=verify-predecessor + fi + fi + { + echo "changes=${changes}" + echo "change_kind=${change_kind}" + echo "action=${action}" + echo "recovery_digest=${recovery_digest}" + } >> "${GITHUB_OUTPUT}" + + - id: fence-auth + if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Fence C4 before predecessor recovery + if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }} + timeout-minutes: 6 + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }} + shell: bash + run: | + set -euo pipefail + expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}" + current_digest="${{ steps.preflight.outputs.current_digest }}" + if test -n "${current_digest}" && [[ ",${expected_digests}," != *",${current_digest},"* ]]; then + expected_digests="${expected_digests},${current_digest}" + fi + if curl --silent --show-error --fail-with-body --max-time 30 --request POST \ + "${CELL_ORIGIN}/v1/admin/drain" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' --data '{"v":1,"graceMs":0}' \ + >/dev/null; then + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \ + --heartbeat fresh --admission migration-only --draining required \ + --activity quiescent \ + --expected-image-digests "${expected_digests}" \ + --timeout-ms 240000 + else + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \ + --admission migration-only --draining either --activity restart-safe \ + --timeout-ms 240000 + fi + + - name: Apply and stabilize the saved predecessor plan + id: apply + if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }} + timeout-minutes: 20 + env: + CHANGES: ${{ steps.recovery-plan.outputs.changes }} + CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }} + shell: bash + run: | + set -euo pipefail + plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan" + if test "${CHANGES}" != 0; then + terraform -chdir=infra/terraform apply -auto-approve "${plan}" + fi + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}" + + - name: Restart only when the plan did not replace C4 + id: restore + if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }} + timeout-minutes: 20 + env: + APPLY_STABLE_AT_MS: ${{ steps.apply.outputs.stable_at_ms }} + CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }} + shell: bash + run: | + set -euo pipefail + if [[ "${CHANGE_KIND}" =~ ^(replacement|replacement-with-obsolete-template|manager-convergence)$ ]]; then + echo "stable_at_ms=${APPLY_STABLE_AT_MS}" >> "${GITHUB_OUTPUT}" + exit 0 + fi + instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \ + | jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and + .[0].currentAction == "NONE" then .[0].instance | split("/") | last + else error("C4 is not one stable running instance") end')" + gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \ + --instances "${instance}" --project "${GCP_PROJECT_ID}" \ + --zone "${TARGET_ZONE}" --quiet + gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \ + --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900 + echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}" + + - name: Require an empty selected-image recovery readback + timeout-minutes: 8 + env: + RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }} + shell: bash + run: | + set -euo pipefail + image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay" + recovery_image="${image_repository}@${RECOVERY_DIGEST}" + other_image="${image_repository}@${TARGET_IMAGE_DIGEST}" + if test "${RECOVERY_DIGEST}" = "${TARGET_IMAGE_DIGEST}"; then + other_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}" + fi + cells="$(terraform -chdir=infra/terraform console \ + -var-file=environments/staging.tfvars -var manage_artifact_dns=false \ + <<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')" + recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" --arg image "${recovery_image}" \ + '.[$cell].image = $image' <<< "${cells}")" + jq -n --argjson cells "${recovery_cells}" \ + '{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-readback.tfvars.json" + readback="${RUNNER_TEMP}/relay-c4-image-recovery-readback.tfplan" + terraform -chdir=infra/terraform plan \ + -var-file=environments/staging.tfvars \ + -var-file="${RUNNER_TEMP}/relay-c4-readback.tfvars.json" \ + -var manage_artifact_dns=false -lock-timeout=5m \ + '-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \ + '-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \ + -out="${readback}" + readback_result="$(terraform -chdir=infra/terraform show -json "${readback}" \ + | node dev/scripts/validate-relay-capacity-plan.mjs \ + --mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \ + --unobserved-bound 60 --image "${recovery_image}" \ + --rollback-image "${other_image}")" + test "$(jq -r '.changes' <<< "${readback_result}")" = 0 + + - id: verify-auth + if: ${{ always() }} + uses: google-github-actions/auth@v2 + with: + workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }} + service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }} + token_format: id_token + id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain + id_token_include_email: true + + - name: Verify the recovered image and unchanged isolation + if: ${{ always() && steps.verify-auth.outcome == 'success' }} + timeout-minutes: 8 + env: + ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.verify-auth.outputs.id_token }} + SELECTOR_GENERATION: ${{ steps.preflight.outputs.selector_generation }} + STABLE_AT_MS: ${{ steps.restore.outputs.stable_at_ms }} + RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }} + shell: bash + run: | + set -euo pipefail + node dev/scripts/verify-relay-capacity-transition.mjs \ + --director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \ + --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \ + --heartbeat fresh --admission migration-only --draining forbidden \ + --activity quiescent --expected-image-digests "${RECOVERY_DIGEST}" \ + --timeout-ms 240000 + stable_at_ms="${STABLE_AT_MS:-0}" + for _ in $(seq 1 36); do + status="$(curl --fail-with-body --max-time 30 --request POST \ + "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \ + --header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \ + --header 'Content-Type: application/json' \ + --data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")" + if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \ + test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \ + -ge "${stable_at_ms}"; then break; fi + sleep 5 + done + test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true + test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \ + -ge "${stable_at_ms}" + result="$(node dev/scripts/operate-relay-asia-admission.mjs \ + --environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \ + --expected-generation "${SELECTOR_GENERATION}" \ + --expected-membership-sha256 '' --attempt-id '' \ + --image-digest "${RECOVERY_DIGEST}")" + test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \ + migration-only diff --git a/.github/workflows/cloud-requeue-relay-staging-c4-recovery.yml b/.github/workflows/cloud-requeue-relay-staging-c4-recovery.yml new file mode 100644 index 00000000000..d137c0b1fdc --- /dev/null +++ b/.github/workflows/cloud-requeue-relay-staging-c4-recovery.yml @@ -0,0 +1,59 @@ +name: Requeue Relay Staging C4 Recovery + +on: + workflow_run: + workflows: [Recover Relay Staging C4 Image] + types: [completed] + +permissions: + actions: write + contents: read + +concurrency: + group: relay-staging-c4-recovery-requeue + cancel-in-progress: false + +defaults: + run: + working-directory: cloud + +jobs: + requeue: + if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion == 'cancelled') }} + runs-on: blacksmith-2vcpu-ubuntu-2204 + timeout-minutes: 5 + steps: + - name: Requeue only a cancelled protected recovery job + env: + GH_TOKEN: ${{ github.token }} + SOURCE_RUN_ID: ${{ github.event.workflow_run.id }} + shell: bash + run: | + set -euo pipefail + [[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]] + jobs="$(gh api --paginate \ + "repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")" + count="$(jq -s '[.[].jobs[] | select(.name == "recover" and + .conclusion == "cancelled" and .started_at == null)] | length' <<< "${jobs}")" + if test "${count}" = 0; then exit 0; fi + test "${count}" = 1 + runs="$(gh api \ + "repos/${GITHUB_REPOSITORY}/actions/workflows/cloud-recover-relay-staging-c4-image.yml/runs?branch=main&per_page=100")" + active=0 + while IFS= read -r run_id; do + active_jobs="$(gh api --paginate \ + "repos/${GITHUB_REPOSITORY}/actions/runs/${run_id}/jobs?filter=latest")" + if jq -se ' + ([.[].jobs[] | select(.name == "gate" and .conclusion == "success")] | length) == 1 and + ([.[].jobs[] | select(.name == "recover" and .status != "completed")] | length) == 1 + ' <<< "${active_jobs}" >/dev/null; then + active=1 + break + fi + done < <(jq -r --arg source "${SOURCE_RUN_ID}" \ + '.workflow_runs[] | select((.id | tostring) != $source and + .status != "completed") | .id' <<< "${runs}") + if test "${active}" != 0; then exit 0; fi + gh workflow run cloud-recover-relay-staging-c4-image.yml \ + --repo "${GITHUB_REPOSITORY}" --ref main \ + -f confirmation=RECOVER_STAGING_ASIA_C4_IMAGE diff --git a/cloud/.gitleaks.toml b/cloud/.gitleaks.toml index 430d17622b8..fc5c725c37d 100644 --- a/cloud/.gitleaks.toml +++ b/cloud/.gitleaks.toml @@ -5,3 +5,11 @@ useDefault = true description = "Explicit Relay test signing key" regexTarget = "secret" regexes = ['''^test-assignment-key-with-at-least-32-bytes$'''] + +# The Cloud SQL rollout lease records `owner/repo/run_id` as the holder of a lease. The action's +# unit tests build fixture holders from that shape, which the generic key rule reads as a secret. +[[allowlists]] +description = "Cloud SQL rollout lease holder keys in the action's unit tests" +regexTarget = "secret" +paths = ['''\.github/actions/cloud-sql-rollout-lease/[a-z-]+\.test\.mjs$'''] +regexes = ['''^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+/[0-9]+$'''] diff --git a/cloud/dev/scripts/relay-public-workflow-contract.test.mjs b/cloud/dev/scripts/relay-public-workflow-contract.test.mjs new file mode 100644 index 00000000000..56393d07bd1 --- /dev/null +++ b/cloud/dev/scripts/relay-public-workflow-contract.test.mjs @@ -0,0 +1,82 @@ +import assert from 'node:assert/strict' +import test from 'node:test' +import { + isEntrypoint, + jobIf, + jobNeeds, + jobs, + readWorkflow, + workflowFiles +} from './cloud-sql-rollout-lock-census.mjs' +import { relayWorkflowFile } from './relay-repository.mjs' + +// Why: this repository publishes the relay's operate surface next to the desktop app. Three +// invariants make that safe, and each of them is one careless edit away from being lost. +const OPERATIONS_GATE = "vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true'" + +// Cloud Verify is the only cloud workflow that must run on every pull request. +const UNGATED = relayWorkflowFile('verify.yml') + +const relayWorkflows = () => workflowFiles().filter((file) => file !== UNGATED) + +test('the copy carries every relay workflow', () => { + assert.equal(relayWorkflows().length, 24) +}) + +// Why: workflow_run chains match by display name, not filename. Renaming a file is safe; renaming +// one of these silently breaks the recovery chain with no failing run to notice. +test('the recovery chain keeps the display names it is matched by', () => { + const names = Object.fromEntries( + ['prove-relay-staging-capacity.yml', 'recover-relay-staging-c4-image.yml', 'requeue-relay-staging-c4-recovery.yml'].map( + (name) => [name, /^name: (.+)$/m.exec(readWorkflow(relayWorkflowFile(name)))?.[1]] + ) + ) + assert.deepEqual(names, { + 'prove-relay-staging-capacity.yml': 'Prove Relay Staging Capacity', + 'recover-relay-staging-c4-image.yml': 'Recover Relay Staging C4 Image', + 'requeue-relay-staging-c4-recovery.yml': 'Requeue Relay Staging C4 Recovery' + }) + const recover = readWorkflow(relayWorkflowFile('recover-relay-staging-c4-image.yml')) + const requeue = readWorkflow(relayWorkflowFile('requeue-relay-staging-c4-recovery.yml')) + assert.ok(recover.includes(`workflows: [${names['prove-relay-staging-capacity.yml']}]`)) + assert.ok(requeue.includes(`workflows: [${names['recover-relay-staging-c4-image.yml']}]`)) +}) + +// Why: this repository holds none of the GCP credentials these workflows would need. Every one +// authenticates through Workload Identity read from a variable, so any repository secret other +// than the automatic token would be a credential the owner has to store here. +test('no cloud workflow reads a repository secret', () => { + for (const file of workflowFiles()) { + for (const [, name] of readWorkflow(file).matchAll(/secrets\.([A-Za-z_][A-Za-z0-9_]*)/g)) { + assert.equal(name, 'GITHUB_TOKEN', `${file} reads secrets.${name}`) + } + } +}) + +// Why: the operations gate is what makes the whole surface inert until the owner enables it. A +// job that can start without a gated dependency would run the moment someone dispatches it. +test('every job that can start on its own is gated on the operations variable', () => { + const reachable = [] + for (const file of relayWorkflows()) { + const text = readWorkflow(file) + if (!isEntrypoint(text)) continue + for (const job of jobs(text)) { + if (jobNeeds(job.text).length > 0) continue + reachable.push(`${file}:${job.id}`) + assert.ok(jobIf(job.text).includes(OPERATIONS_GATE), `${file}:${job.id} is not gated`) + } + } + assert.ok(reachable.length >= 20, `only ${reachable.length} root jobs were checked`) +}) + +// Why: reusable jobs inherit the caller's gate. Gating them again would be dead configuration +// that reads as protection, and every caller is already checked above. +test('reusable workflows carry no gate of their own', () => { + for (const file of relayWorkflows()) { + const text = readWorkflow(file) + if (isEntrypoint(text)) continue + for (const job of jobs(text)) { + assert.ok(!jobIf(job.text).includes(OPERATIONS_GATE), `${file}:${job.id} regates a reusable job`) + } + } +})