Merge origin/main and port the Windows structured gate onto the new router

#18248 deleted canUseStructuredNativeChat and replaced it with
resolveAgentLaunchRoute, which still carried a blanket `platform !== 'win32'`
refusal -- the exact refusal this lane removes. Taking main's deletion would
have gone green on tc and lint while silently dropping the whole Windows
enablement, so the gate is ported rather than merged.

- resolveAgentLaunchRoute now takes a REQUIRED tri-state
  windowsProcessStartTime ('available' | 'unavailable' | 'unknown') and a
  required worktreeUsesWslPath. Required, not optional, so the compiler
  enumerates every call site -- it immediately found a 7th behind a
  forwarding wrapper that a grep of the call sites had missed.
- 'unknown' is refused: a host that never answered the probe must not have a
  PID adopted against it.
- Derivation lives once in agent-launch-routing-windows-gate.ts.
  hasCachedWindowsTerminalCapabilities distinguishes unprobed from probed-false.
- resolveStructuredAgentSessionLocation: both WSL branches now key on
  executionHostId. main's resolveRuntimeFileTarget no longer returns
  connectionId at all, so the folder branch's `!target.connectionId` was
  permanently true; the file is @ts-nocheck so nothing caught it.
- create-intent stubs updated to main's target contract.
- readWorktreeUsesWslPath defaults both store collections: an unhydrated
  store must read as "not WSL", not throw on the launch path.

Gates: pnpm tc 0. Gate tests ablation-proven in BOTH directions -- reverting
to the blanket refusal reddens the 2 enablement tests; forcing allow reddens
the 4 refusal tests. ssh-multi-factor-authentication failures reproduce on
pure origin/main and are not attributable to this merge.
This commit is contained in:
Merge Sim
2026-09-03 13:58:51 -07:00
1317 changed files with 147791 additions and 7144 deletions
+3
View File
@@ -18,3 +18,6 @@
# the reviewable change, and pin LF because they are compared byte-for-byte.
# Not -diff: the shell diff is the review surface when a wrapper does change.
/src/main/__fixtures__/shell-wrapper-snapshots/*.txt linguist-generated=true text eol=lf
# Generated runtime English subset: compared byte-for-byte by
# verify:localization-runtime-catalog, so a CRLF checkout would fail the gate.
/src/renderer/src/i18n/en-runtime-required.json linguist-generated=true text eol=lf
+5
View File
@@ -3,3 +3,8 @@
/config/scripts/*localization*.mjs @brennanb2025
/config/scripts/*locale*.mjs @brennanb2025
/config/i18next.config.ts @brennanb2025
# The relay's deploy and operate surface: workspace, workflows, and the rollout lease action.
/cloud/ @Jinwoo-H
/.github/workflows/cloud-*.yml @Jinwoo-H
/.github/actions/cloud-sql-rollout-lease/ @Jinwoo-H
@@ -0,0 +1,152 @@
# Cloud SQL rollout lease
A compare-and-swap lease on one Cloud Storage object, used to serialize Cloud SQL
**connection-budget** rollouts across two repositories.
`concurrency.group: production-cloud-sql-rollout` only serializes runs inside a single repository.
Once the relay workflows live in `stablyai/orca` and the app workflows stay in
`stablyai/orca-cloud`, there are two independent queues pointed at one shared Cloud SQL instance.
`relay-cloud-sql-connection-budget.mjs` computes `rolloutOverlap` as a `Math.max` over the relay
director, api, auth and relay-cell candidates, which is only sound when exactly one rollout is in
flight. This lease is what keeps that assumption true. Keep the per-repo concurrency groups **and**
the lease; they solve different halves of the problem.
## What it protects
No workflow runs a Cloud SQL schema migration. Every locked workflow either deploys a Cloud Run
revision or applies a GCE instance template against the shared instance, so the lease must cover
**all rollouts**, not just migrations.
## Usage
The lease step must run **after** `google-github-actions/setup-gcloud`, and after
`actions/checkout` — `uses: ./.github/actions/...` resolves against the checked-out workspace.
It belongs in the first job of the workflow that holds a GCP credential, which is not always the
gate job: `deploy-relay-production-same-cap`'s gate runs no `gcloud`, so its first acquire happens
in the first cell job.
```yaml
- uses: google-github-actions/auth@v2
with: { workload_identity_provider: ..., service_account: ... }
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
```
Buckets and objects in use:
| Environment | Bucket | Object |
| ----------- | -------------------------------------- | --------------------------------------------------- |
| production | `onorca-cloud-terraform-state` | `terraform/state/cloud-sql-rollout/production.lock` |
| staging | `onorca-cloud-staging-terraform-state` | `terraform/state/cloud-sql-rollout/staging.lock` |
Workflows that serve both environments (`deploy-relay-asia-topology`,
`operate-relay-asia-admission`) select the pair with an `inputs.environment == 'production'`
ternary on both `bucket` and `object`. `deploy-staging` keeps its own `deploy-artifacts-staging`
concurrency group but takes the staging lease, because it rolls the staging API revision.
The object sits beside `terraform/state/relay-fence-broker/<env>.lock`. The IAM grant names both the
relay and app service accounts, so it is a **foundation-root** resource: `roles/storage.objectAdmin`
conditioned on the `terraform/state/cloud-sql-rollout/` prefix, **plus** an unconditioned
`roles/storage.legacyBucketReader`. Without the second role the generation-matched write fails in a
way that looks like a permissions flake.
## One lease per run, not per job
`deploy-relay-production-capacity` calls its reusable job six times and
`deploy-relay-production-same-cap` four times. Each call is a separate job on a separate runner, so
a naive per-job acquire/release would leave the object free between waves — for runs that have taken
up to 85 minutes.
The lease is therefore keyed to the **run**, not the job. `holder-key` defaults to
`${{ github.repository }}/${{ github.run_id }}`, and a job that finds its own holder key on a live
lease **re-enters** it: the record is refreshed, not rejected. Every job in the chain acquires; only
the last one releases.
```yaml
jobs:
gate:
steps:
- uses: ./.github/actions/cloud-sql-rollout-lease
with: { bucket: ..., object: ..., release: 'false' } # intermediate
wave-1: # ... release: 'false' on every wave job
release_lease:
needs: [gate, wave-1, wave-2, wave-3, wave-4]
if: always()
steps:
- uses: google-github-actions/auth@v2
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with: { bucket: ..., object: ..., release: 'true' } # final
```
`release: 'false'` still acquires and still runs its `post` step; `post` only skips the delete. A
single-job workflow leaves `release` at its `true` default and needs no extra job. So does a
workflow whose several jobs can never hold the lease at once: `prove-relay-staging-capacity`'s two
lease-holding jobs are guarded by complementary `inputs.mode` conditions, and the contract test
checks that exclusivity rather than assuming it.
If the final job never runs (runner killed, run cancelled hard), the lease expires on its TTL.
## Timing
- **TTL 35 minutes**, matching `apps/relay-fence-broker/src/mutation-lease.ts`.
- **Renewal every 5 minutes.** `main` spawns a detached background Node process that rewrites
`expires_at` on the same generation-matched path; `post` kills it by pid read back from
`$GITHUB_STATE`. The renewer stops on its own the moment the object stops being ours, and has a
six-hour backstop in case `post` never runs. Its log is written to
`$RUNNER_TEMP/cloud-sql-rollout-lease-renewer.log` and echoed by `post`.
- Renewal is mandatory, not optional: capacity runs have taken 85 minutes, well past any sane TTL.
## Failure behaviour
| Situation | Behaviour |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Object absent | Acquire with `ifGenerationMatch: 0`. |
| Live lease, our own holder key | Re-enter. Refresh `expires_at`, keep `acquired_at`. Never fails. |
| Live lease, another holder | **Fail the job immediately**, printing the holder's repository, workflow and run URL. Never queues, never steals. |
| Expired lease | Take over with the observed generation and emit `::warning::` naming the stale holder. |
| `412` on write | Someone raced us. Fail as a conflict. |
| Bucket unreachable, `403`, `5xx` | **Fail closed.** |
| Record present but unparseable | **Fail closed.** A record we cannot read is never treated as free; an operator must inspect and delete it. |
| Release finds a foreign holder | Warn and leave it alone. Our lease had already expired. |
| Release fails | Warn only. `post` never fails a job over a release; the TTL bounds the damage. |
## Why `monitor-relay-production` must not use this
`monitor-relay-production` is in the `production-cloud-sql-rollout` concurrency group but is
**read-only**: its identity holds only monitoring, logging, Cloud SQL and compute _viewer_ roles,
and it runs `gcloud sql instances describe`, never a mutation. It consumes no connection budget.
Putting it on the durable lease would let a monitoring run block a real rollout, and a rollout block
monitoring exactly when an operator most needs it. Keep its same-repo concurrency group; keep it off
the lease. The lock census contract test records it in the not-a-candidate map with this reason.
## Token acquisition
`gcloud auth print-access-token`, not a hand-rolled exchange of the `external_account` credentials
file. Every consuming workflow already runs `setup-gcloud`, gcloud already handles every ADC flavour
including the service-account impersonation leg, and this action must stay zero-dependency because
it is duplicated by hand into the public repo. The GCE metadata server that
`apps/relay-fence-broker/src/google-metadata.ts` uses does **not** exist on GitHub or Blacksmith
runners; only the compare-and-swap algorithm is shared with the fence broker.
## Duplication
This directory is copied verbatim into `stablyai/orca`. It has no `package.json`, no
`node_modules`, and imports nothing outside itself — `action-contract.test.mjs` enforces all three.
Cross-repo consumption via `uses: stablyai/orca/.github/actions/...@<sha>` was rejected: it would
put public-repo code inside private app deploys that hold a production credential, and neither
repository protects `main` today.
## Tests
```
node --test .github/actions/cloud-sql-rollout-lease/
```
`storage-lease.test.mjs` drives the real compare-and-swap path against an in-memory Cloud Storage
fake that enforces generations. No network.
@@ -0,0 +1,52 @@
import assert from 'node:assert/strict'
import { readFileSync, readdirSync } from 'node:fs'
import { test } from 'node:test'
const here = new URL('./', import.meta.url)
const action = readFileSync(new URL('action.yml', here), 'utf8')
const modules = readdirSync(here).filter((name) => name.endsWith('.mjs'))
const shipped = modules.filter((name) => !name.endsWith('.test.mjs'))
test('is a node24 JavaScript action with an always-run post step', () => {
// A composite action has no `post:`, so the lease could never be released on cancel or failure.
assert.match(action, /^ {2}using: node24$/m)
assert.doesNotMatch(action, /using: composite/)
assert.match(action, /^ {2}main: main\.mjs$/m)
assert.match(action, /^ {2}post: post\.mjs$/m)
assert.match(action, /^ {2}post-if: always\(\)$/m)
})
test('declares the inputs the wave-chain callers depend on', () => {
for (const input of ['bucket:', 'object:', 'holder-key:', 'release:']) {
assert.match(action, new RegExp(`^ {2}${input}$`, 'm'), input)
}
assert.match(action, /default: \$\{\{ github\.repository \}\}\/\$\{\{ github\.run_id \}\}/)
assert.match(action, /default: 'true'/)
})
test('stays self-contained so it can be duplicated into the public repo', () => {
assert.deepEqual(
readdirSync(here).filter((name) => name === 'package.json' || name === 'node_modules'),
[],
'the action must run with zero installed dependencies'
)
for (const name of modules) {
const source = readFileSync(new URL(name, here), 'utf8')
for (const match of source.matchAll(/^import\b[\s\S]*?from '([^']+)'/gm)) {
const specifier = match[1]
const local = specifier.startsWith('.')
assert.ok(
specifier.startsWith('node:') || (local && !specifier.includes('..')),
`${name} imports ${specifier}; only node: builtins and same-directory modules are allowed`
)
}
}
})
test('never reaches for the GCE metadata server', () => {
// Runners have no metadata.google.internal; the fence broker's token path must not be copied.
for (const name of shipped) {
const source = readFileSync(new URL(name, here), 'utf8')
assert.doesNotMatch(source, /metadata\.google\.internal/, name)
}
})
@@ -0,0 +1,41 @@
name: Cloud SQL rollout lease
description: >-
Serialize Cloud SQL connection-budget rollouts across repositories with a compare-and-swap lease
on a Cloud Storage object. Fails immediately when another run holds the lease; never queues,
never steals.
inputs:
bucket:
description: Terraform state bucket that holds the lease object.
required: true
object:
description: Lease object name, e.g. terraform/state/cloud-sql-rollout/production.lock
required: true
holder-key:
description: >-
Identity that owns the lease. Every job in one run must pass the same value; a job that finds
its own holder key on a live lease re-enters it instead of failing.
required: false
default: ${{ github.repository }}/${{ github.run_id }}
release:
description: >-
Release the lease in the post step. Set to "false" on every job of a multi-job wave except the
final always() job, which sets "true".
required: false
default: 'true'
outputs:
holder-key:
description: The holder key written to the lease object.
generation:
description: Cloud Storage generation of the lease object after acquisition.
expires-at:
description: ISO-8601 instant at which the lease expires without renewal.
reentrant:
description: '"true" when this job re-entered a lease its own run already held.'
runs:
using: node24
main: main.mjs
post: post.mjs
post-if: always()
@@ -0,0 +1,46 @@
import { execFileSync } from 'node:child_process'
// DESIGN CHOICE: shell out to `gcloud auth print-access-token` instead of exchanging the
// external_account credentials file that google-github-actions/auth writes.
//
// Every workflow on the Cloud SQL rollout lease already runs google-github-actions/setup-gcloud
// right after auth (verified across all 11 mutating members), so gcloud is on PATH and already
// bound to the federated identity. Doing the exchange ourselves would mean reimplementing the STS
// token swap plus the service-account impersonation leg, in an action that must stay
// zero-dependency and is duplicated by hand into a second repo. gcloud already handles every ADC
// flavour and refreshes on its own. The metadata server is not an option: it does not exist on
// GitHub or Blacksmith runners.
//
// Consequence, documented in the README: the lease step MUST come after setup-gcloud.
const TOKEN_REUSE_MS = 40 * 60 * 1_000 // GCP access tokens live ~60 min; re-mint well before that.
export function createAccessTokenSource({ run = runGcloud, now = Date.now } = {}) {
let cached = null
return () => {
if (cached && cached.mintedAt + TOKEN_REUSE_MS > now()) {
return cached.token
}
const token = run()
if (!token) {
throw new Error('gcloud auth print-access-token returned an empty token')
}
cached = { token, mintedAt: now() }
return token
}
}
function runGcloud() {
const binary = process.platform === 'win32' ? 'gcloud.cmd' : 'gcloud'
try {
return execFileSync(binary, ['auth', 'print-access-token'], {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
timeout: 60_000
}).trim()
} catch (error) {
// Never surface stdout; it is the token on success and noise on failure.
const detail = String(error?.stderr ?? '').trim() || error?.message || 'unknown failure'
throw new Error(`could not mint a GCP access token via gcloud: ${detail}`)
}
}
@@ -0,0 +1,20 @@
// Who we claim to be on the lease object. Shared by main and the detached renewer so both agree
// on the holder key without re-deriving it from a different set of environment variables.
export function holderIdentity(explicitHolderKey) {
const repository = process.env.GITHUB_REPOSITORY ?? 'unknown'
const runId = process.env.GITHUB_RUN_ID ?? 'unknown'
const server = process.env.GITHUB_SERVER_URL ?? 'https://github.com'
const holderKey = explicitHolderKey || `${repository}/${runId}`
if (/[\r\n]/.test(holderKey)) {
throw new Error('holder-key must be single-line')
}
return {
holderKey,
repository,
workflow: process.env.GITHUB_WORKFLOW ?? 'unknown',
runId,
runUrl: `${server}/${repository}/actions/runs/${runId}`,
runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? 'unknown'
}
}
@@ -0,0 +1,81 @@
import { spawn } from 'node:child_process'
import { fileURLToPath } from 'node:url'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { holderIdentity } from './holder-identity.mjs'
import { fail, input, notice, renewerLogPath, saveState, setOutput, warn } from './runner-state.mjs'
import { CloudSqlRolloutLease, LeaseConflict, describeHolder } from './storage-lease.mjs'
const bucket = input('bucket')
const objectName = input('object')
const release = input('release') !== 'false'
if (!bucket || !objectName) {
fail('cloud-sql-rollout-lease requires both `bucket` and `object`')
process.exit(1)
}
const holder = holderIdentity(input('holder-key'))
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource()
})
let claim
try {
claim = await lease.acquire(holder)
} catch (error) {
if (error instanceof LeaseConflict) {
fail(
`${error.message}. Cloud SQL rollouts are serialized across repositories; this run will not queue or steal the lease. Wait for the holder to finish, then re-run.`
)
if (error.holder) {
console.log(`Lease holder repository: ${error.holder.repository}`)
console.log(`Lease holder workflow: ${error.holder.workflow}`)
console.log(`Lease holder run: ${error.holder.run_url}`)
}
} else {
// Bucket unreachable, permission denied, unreadable record: fail closed.
fail(`could not acquire ${lease.uri}: ${error.message}`)
}
process.exit(1)
}
// Persist before anything else can throw, so `post` always releases what we hold.
saveState('acquired', 'true')
saveState('bucket', bucket)
saveState('object', objectName)
saveState('holder_key', holder.holderKey)
saveState('release', release ? 'true' : 'false')
setOutput('holder-key', holder.holderKey)
setOutput('generation', claim.generation)
setOutput('expires-at', new Date(claim.record.expires_at).toISOString())
setOutput('reentrant', claim.state === 'reentrant' ? 'true' : 'false')
if (claim.state === 'reentrant') {
notice(
`Re-entered the Cloud SQL rollout lease on ${lease.uri} already held by this run; refreshed to ${new Date(claim.record.expires_at).toISOString()}.`
)
} else if (claim.state === 'takeover') {
notice(`Took over ${lease.uri} from ${describeHolder(claim.previous)}.`)
} else {
notice(
`Acquired ${lease.uri} until ${new Date(claim.record.expires_at).toISOString()} (holder ${holder.holderKey}).`
)
}
try {
const renewer = spawn(
process.execPath,
[fileURLToPath(new URL('./renew.mjs', import.meta.url)), bucket, objectName, holder.holderKey],
{ detached: true, stdio: 'ignore', env: process.env }
)
renewer.unref()
saveState('renewer_pid', String(renewer.pid))
const log = renewerLogPath()
notice(`Lease renewer running as pid ${renewer.pid}${log ? `, logging to ${log}` : ''}.`)
} catch (error) {
// A missing renewer is survivable for short jobs; the TTL still covers 35 minutes.
warn(`could not start the lease renewer: ${error.message}. The lease will expire on its TTL.`)
}
@@ -0,0 +1,81 @@
import { readFileSync } from 'node:fs'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { notice, renewerLogPath, savedState, warn } from './runner-state.mjs'
import { CloudSqlRolloutLease, describeHolder } from './storage-lease.mjs'
stopRenewer()
printRenewerLog()
if (savedState('acquired') !== 'true') {
notice('No Cloud SQL rollout lease was acquired by this step; nothing to release.')
process.exit(0)
}
const bucket = savedState('bucket')
const objectName = savedState('object')
const holderKey = savedState('holder_key')
if (savedState('release') !== 'true') {
notice(
`Holding gs://${bucket}/${objectName} for the rest of run ${holderKey}; a later job with release=true must free it.`
)
process.exit(0)
}
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource()
})
try {
const result = await lease.release(holderKey)
if (result.released) {
notice(`Released ${lease.uri} at generation ${result.generation}.`)
} else if (result.reason === 'absent') {
notice(`${lease.uri} was already gone; nothing to release.`)
} else if (result.reason === 'foreign') {
warn(
`${lease.uri} is now held by ${describeHolder(result.holder)}; leaving it alone. Our lease had already expired.`
)
} else {
warn(`${lease.uri} changed while releasing it; leaving it to expire on its TTL.`)
}
} catch (error) {
// Never fail a job in post over a release; the TTL bounds the damage to 35 minutes.
warn(`could not release ${lease.uri}: ${error.message}. It will expire on its TTL.`)
}
function stopRenewer() {
const pid = Number(savedState('renewer_pid'))
if (!Number.isInteger(pid) || pid <= 0) {
return
}
try {
process.kill(pid, 'SIGTERM')
notice(`Stopped the lease renewer (pid ${pid}).`)
} catch (error) {
if (error?.code !== 'ESRCH') {
warn(`could not stop the lease renewer ${pid}: ${error.message}`)
}
}
}
function printRenewerLog() {
const path = renewerLogPath()
if (!path) {
return
}
let text = ''
try {
text = readFileSync(path, 'utf8')
} catch {
return
}
if (!text.trim()) {
return
}
console.log('::group::Cloud SQL rollout lease renewer log')
console.log(text.trimEnd())
console.log('::endgroup::')
}
@@ -0,0 +1,73 @@
import { appendFileSync } from 'node:fs'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { renewerLogPath } from './runner-state.mjs'
import { CloudSqlRolloutLease, RENEW_INTERVAL_MS } from './storage-lease.mjs'
// Detached renewer. `main` spawns it, `post` kills it. It rewrites expires_at on the same
// generation-matched path as acquisition, and stops the moment the object stops being ours.
const MAX_LIFETIME_MS = 6 * 60 * 60 * 1_000 // Backstop if post never runs (runner killed).
const [bucket, objectName, holderKey] = process.argv.slice(2)
const logPath = renewerLogPath()
const startedAt = Date.now()
function log(message) {
if (!logPath) {
return
}
try {
appendFileSync(logPath, `${new Date().toISOString()} ${message}\n`)
} catch {
// A renewer that cannot log must still renew.
}
}
if (!bucket || !objectName || !holderKey) {
log('renewer started without bucket/object/holder-key; exiting')
process.exit(1)
}
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource(),
warn: (message) => log(`warning ${message}`)
})
let stopping = false
for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) {
process.on(signal, () => {
stopping = true
log(`received ${signal}; stopping`)
process.exit(0)
})
}
log(`renewer started for ${lease.uri} holder=${holderKey} interval=${RENEW_INTERVAL_MS}ms`)
while (!stopping) {
await new Promise((resolve) => {
setTimeout(resolve, RENEW_INTERVAL_MS)
})
if (stopping) {
break
}
if (Date.now() - startedAt > MAX_LIFETIME_MS) {
log('renewer hit its maximum lifetime; stopping so the lease can expire')
break
}
try {
const result = await lease.renew(holderKey)
if (!result.renewed) {
log(`lease is no longer ours (${result.reason}); stopping`)
break
}
log(
`renewed until ${new Date(result.record.expires_at).toISOString()} at generation ${result.generation}`
)
} catch (error) {
// Transient GCS or token failures are retried on the next tick; the TTL covers 7 misses.
log(`renewal attempt failed: ${error.message}`)
}
}
@@ -0,0 +1,55 @@
import { appendFileSync } from 'node:fs'
// Action-state and output plumbing via the runner's file protocol, so the action needs no
// @actions/core dependency. Values are single-line by construction; anything else is rejected.
/** The detached renewer's stdio is ignored, so it appends here instead and `post` echoes it. */
export function renewerLogPath() {
const dir = process.env.RUNNER_TEMP
return dir ? `${dir}/cloud-sql-rollout-lease-renewer.log` : null
}
export function input(name) {
return (process.env[`INPUT_${name.replace(/ /g, '_').toUpperCase()}`] ?? '').trim()
}
export function savedState(name) {
return (process.env[`STATE_${name}`] ?? '').trim()
}
export function saveState(name, value) {
appendToEnvFile('GITHUB_STATE', name, value)
}
export function setOutput(name, value) {
appendToEnvFile('GITHUB_OUTPUT', name, value)
}
export function notice(message) {
console.log(`::notice::${oneLine(message)}`)
}
export function warn(message) {
console.log(`::warning::${oneLine(message)}`)
}
export function fail(message) {
console.log(`::error::${oneLine(message)}`)
process.exitCode = 1
}
function appendToEnvFile(variable, name, value) {
const text = String(value)
if (/[\r\n]/.test(text)) {
throw new Error(`${name} must be single-line`)
}
const path = process.env[variable]
if (!path) {
return
} // Running outside a runner (local smoke run); nothing to persist.
appendFileSync(path, `${name}=${text}\n`)
}
function oneLine(message) {
return String(message).replace(/\r?\n/g, ' ')
}
@@ -0,0 +1,240 @@
// Compare-and-swap lease over a single Cloud Storage object.
//
// Ported from apps/relay-fence-broker/src/mutation-lease.ts rather than imported: this action is
// duplicated verbatim into stablyai/orca, so it must carry no repo-local imports. Only the
// algorithm is shared (read metadata -> write with ifGenerationMatch -> 412 is a conflict ->
// generation-matched delete -> an expired record is free). The broker's token path is NOT shared;
// it reads the GCE metadata server, which does not exist on Actions runners.
export const LEASE_TTL_MS = 35 * 60 * 1_000
export const RENEW_INTERVAL_MS = 5 * 60 * 1_000
const GENERATION = /^[1-9][0-9]{0,30}$/
export class LeaseConflict extends Error {
constructor(message, holder) {
super(message)
this.name = 'LeaseConflict'
this.holder = holder ?? null
}
}
/** A live record we cannot parse is never treated as free; wedging beats double-rollout. */
export class LeaseUnreadable extends Error {
constructor(message) {
super(message)
this.name = 'LeaseUnreadable'
}
}
function parseRecord(raw) {
if (!raw || typeof raw !== 'object') {
return null
}
if (typeof raw.holder_key !== 'string' || raw.holder_key.length === 0) {
return null
}
if (!Number.isSafeInteger(raw.acquired_at) || !Number.isSafeInteger(raw.expires_at)) {
return null
}
return {
repository: typeof raw.repository === 'string' ? raw.repository : 'unknown',
workflow: typeof raw.workflow === 'string' ? raw.workflow : 'unknown',
run_id: typeof raw.run_id === 'string' ? raw.run_id : 'unknown',
run_url: typeof raw.run_url === 'string' ? raw.run_url : 'unknown',
run_attempt: typeof raw.run_attempt === 'string' ? raw.run_attempt : 'unknown',
acquired_at: raw.acquired_at,
expires_at: raw.expires_at,
holder_key: raw.holder_key
}
}
function describe(record) {
return `${record.repository} / ${record.workflow} (run ${record.run_id}, attempt ${record.run_attempt}) ${record.run_url}`
}
export class CloudSqlRolloutLease {
#bucket
#objectName
#accessToken
#fetcher
#now
#warn
constructor({
bucket,
objectName,
accessToken,
fetcher = fetch,
now = Date.now,
warn = (message) => console.log(`::warning::${message}`)
}) {
this.#bucket = bucket
this.#objectName = objectName
this.#accessToken = accessToken
this.#fetcher = fetcher
this.#now = now
this.#warn = warn
}
get uri() {
return `gs://${this.#bucket}/${this.#objectName}`
}
async acquire(holder) {
const existing = await this.read()
const now = this.#now()
if (!existing) {
return this.#claim(holder, '0', now, now, 'created')
}
if (existing.record.holder_key === holder.holderKey) {
// Same run, another job in the wave chain. Refresh, never fail.
return this.#claim(
holder,
existing.generation,
existing.record.acquired_at,
now,
'reentrant',
existing.record
)
}
if (existing.record.expires_at > now) {
throw new LeaseConflict(
`${this.uri} is held by ${describe(existing.record)} until ${new Date(existing.record.expires_at).toISOString()}`,
existing.record
)
}
this.#warn(
`Taking over an expired Cloud SQL rollout lease on ${this.uri}. Stale holder: ${describe(existing.record)}, expired ${new Date(existing.record.expires_at).toISOString()}.`
)
return this.#claim(holder, existing.generation, now, now, 'takeover', existing.record)
}
async renew(holderKey) {
const existing = await this.read()
if (!existing) {
return { renewed: false, reason: 'absent' }
}
if (existing.record.holder_key !== holderKey) {
return { renewed: false, reason: 'foreign' }
}
const now = this.#now()
const record = { ...existing.record, expires_at: now + LEASE_TTL_MS }
const written = await this.#write(record, existing.generation)
return { renewed: true, generation: written.generation, record }
}
async release(holderKey) {
const existing = await this.read()
if (!existing) {
return { released: false, reason: 'absent' }
}
if (existing.record.holder_key !== holderKey) {
return { released: false, reason: 'foreign', holder: existing.record }
}
const response = await this.#fetcher(
`${this.#metadataUrl()}?ifGenerationMatch=${encodeURIComponent(existing.generation)}`,
{ method: 'DELETE', headers: { Authorization: `Bearer ${await this.#token()}` } }
)
if (response.status === 412) {
return { released: false, reason: 'conflict' }
}
if (!response.ok && response.status !== 404) {
throw new Error(`lease release failed: ${response.status}`)
}
return { released: true, generation: existing.generation }
}
async read() {
const token = await this.#token()
const metadataResponse = await this.#fetcher(this.#metadataUrl(), {
headers: { Authorization: `Bearer ${token}` }
})
if (metadataResponse.status === 404) {
return null
}
if (!metadataResponse.ok) {
throw new Error(`lease inspection failed: ${metadataResponse.status}`)
}
const metadata = await metadataResponse.json()
if (!GENERATION.test(metadata?.generation ?? '')) {
throw new LeaseUnreadable(`${this.uri} has no valid generation`)
}
const bodyResponse = await this.#fetcher(`${this.#metadataUrl()}?alt=media`, {
headers: { Authorization: `Bearer ${token}` }
})
if (bodyResponse.status === 404) {
return null
}
if (!bodyResponse.ok) {
throw new Error(`lease body read failed: ${bodyResponse.status}`)
}
let raw = null
try {
raw = await bodyResponse.json()
} catch {
raw = null
}
const record = parseRecord(raw)
if (!record) {
throw new LeaseUnreadable(
`${this.uri} holds an unreadable lease record; an operator must inspect and delete it before rollouts can resume`
)
}
return { generation: metadata.generation, record }
}
async #claim(holder, generation, acquiredAt, now, state, previous) {
const record = {
repository: holder.repository,
workflow: holder.workflow,
run_id: holder.runId,
run_url: holder.runUrl,
run_attempt: holder.runAttempt,
acquired_at: acquiredAt,
expires_at: now + LEASE_TTL_MS,
holder_key: holder.holderKey
}
const written = await this.#write(record, generation)
return { state, generation: written.generation, record, previous: previous ?? null }
}
async #write(record, generation) {
const response = await this.#fetcher(
`${this.#uploadUrl()}&ifGenerationMatch=${encodeURIComponent(generation)}`,
{
method: 'POST',
headers: {
Authorization: `Bearer ${await this.#token()}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(record)
}
)
if (response.status === 412) {
throw new LeaseConflict(`${this.uri} changed concurrently while we were claiming it`)
}
if (!response.ok) {
throw new Error(`lease write failed: ${response.status}`)
}
const metadata = await response.json()
if (!GENERATION.test(metadata?.generation ?? '')) {
throw new LeaseUnreadable(`${this.uri} write returned no valid generation`)
}
return { generation: metadata.generation }
}
async #token() {
return typeof this.#accessToken === 'function' ? await this.#accessToken() : this.#accessToken
}
#metadataUrl() {
return `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(this.#bucket)}/o/${encodeURIComponent(this.#objectName)}`
}
#uploadUrl() {
return `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(this.#bucket)}/o?uploadType=media&name=${encodeURIComponent(this.#objectName)}`
}
}
export const describeHolder = describe
@@ -0,0 +1,324 @@
import assert from 'node:assert/strict'
import { test } from 'node:test'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import {
CloudSqlRolloutLease,
LEASE_TTL_MS,
LeaseConflict,
LeaseUnreadable
} from './storage-lease.mjs'
const BUCKET = 'onorca-cloud-terraform-state'
const OBJECT = 'terraform/state/cloud-sql-rollout/production.lock'
const NOW = 1_756_000_000_000
/**
* Enough of the Cloud Storage JSON API to exercise real compare-and-swap semantics: generations
* increment, ifGenerationMatch is enforced, and a mismatch is a 412. `faults` injects failures.
*/
function fakeStorage({ object = null, faults = [] } = {}) {
const state = { object, requests: [] }
const fetcher = async (rawUrl, init = {}) => {
const url = new URL(rawUrl)
const method = init.method ?? 'GET'
const record = { method, url, path: url.pathname, search: url.searchParams }
state.requests.push(record)
const fault = faults.find((candidate) => candidate.when(record))
if (fault) {
return json(fault.status, fault.body ?? {})
}
if (url.pathname.startsWith('/upload/')) {
const want = url.searchParams.get('ifGenerationMatch')
const have = state.object ? state.object.generation : '0'
if (want !== have) {
return json(412, {})
}
const generation = String(Number(have === '0' ? '1000' : have) + 1)
state.object = { generation, body: JSON.parse(init.body) }
return json(200, { generation })
}
if (method === 'DELETE') {
if (!state.object) {
return json(404, {})
}
if (url.searchParams.get('ifGenerationMatch') !== state.object.generation) {
return json(412, {})
}
state.object = null
return json(204, {})
}
if (!state.object) {
return json(404, {})
}
if (url.searchParams.get('alt') === 'media') {
return json(200, state.object.body)
}
return json(200, { generation: state.object.generation })
}
return { state, fetcher }
}
function json(status, body) {
return {
status,
ok: status >= 200 && status < 300,
json: async () => body
}
}
function storedRecord({
holderKey,
expiresAt,
repository = 'stablyai/orca',
workflow = 'Deploy Relay Production'
}) {
return {
repository,
workflow,
run_id: '9001',
run_url: 'https://github.com/stablyai/orca/actions/runs/9001',
run_attempt: '1',
acquired_at: NOW - 60_000,
expires_at: expiresAt,
holder_key: holderKey
}
}
function leaseFor(storage, { warn = () => {} } = {}) {
return new CloudSqlRolloutLease({
bucket: BUCKET,
objectName: OBJECT,
accessToken: 'test-token',
fetcher: storage.fetcher,
now: () => NOW,
warn
})
}
const HOLDER = {
holderKey: 'stablyai/orca-cloud/42',
repository: 'stablyai/orca-cloud',
workflow: 'Deploy Relay Production Same-Cap',
runId: '42',
runUrl: 'https://github.com/stablyai/orca-cloud/actions/runs/42',
runAttempt: '1'
}
test('acquires a lease on an empty object with ifGenerationMatch=0', async () => {
const storage = fakeStorage()
const claim = await leaseFor(storage).acquire(HOLDER)
assert.equal(claim.state, 'created')
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '0')
assert.equal(upload.search.get('name'), OBJECT)
assert.deepEqual(storage.state.object.body, {
repository: 'stablyai/orca-cloud',
workflow: 'Deploy Relay Production Same-Cap',
run_id: '42',
run_url: 'https://github.com/stablyai/orca-cloud/actions/runs/42',
run_attempt: '1',
acquired_at: NOW,
expires_at: NOW + LEASE_TTL_MS,
holder_key: 'stablyai/orca-cloud/42'
})
})
test('refuses a live lease held by another run and never writes', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseConflict)
assert.equal(error.holder.repository, 'stablyai/orca')
assert.equal(error.holder.run_url, 'https://github.com/stablyai/orca/actions/runs/9001')
assert.equal(
storage.state.requests.filter((request) => request.method !== 'GET').length,
0,
'a foreign live lease must not be written'
)
assert.equal(storage.state.object.generation, '1500')
})
test('re-enters a live lease this run already holds and extends it', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: HOLDER.holderKey, expiresAt: NOW + 60_000 })
}
})
const warnings = []
const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(
HOLDER
)
assert.equal(claim.state, 'reentrant')
assert.deepEqual(warnings, [], 're-entering our own lease is not a takeover')
assert.equal(claim.record.acquired_at, NOW - 60_000, 'original acquisition time is preserved')
assert.equal(claim.record.expires_at, NOW + LEASE_TTL_MS)
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '1500')
assert.equal(storage.state.object.generation, '1501')
})
test('takes over an expired lease and warns naming the stale holder', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({
holderKey: 'stablyai/orca/9001',
expiresAt: NOW - 1,
repository: 'stablyai/orca',
workflow: 'Deploy Relay Production Capacity'
})
}
})
const warnings = []
const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(
HOLDER
)
assert.equal(claim.state, 'takeover')
assert.equal(warnings.length, 1)
assert.match(warnings[0], /stablyai\/orca/)
assert.match(warnings[0], /Deploy Relay Production Capacity/)
assert.match(warnings[0], /actions\/runs\/9001/)
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '1500')
assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey)
})
test('releases with a generation match and leaves the object gone', async () => {
const storage = fakeStorage()
const lease = leaseFor(storage)
const claim = await lease.acquire(HOLDER)
const released = await lease.release(HOLDER.holderKey)
assert.deepEqual(released, { released: true, generation: claim.generation })
const remove = storage.state.requests.find((request) => request.method === 'DELETE')
assert.equal(remove.search.get('ifGenerationMatch'), claim.generation)
assert.equal(storage.state.object, null)
})
test('refuses to release a lease another run now holds', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
const released = await leaseFor(storage).release(HOLDER.holderKey)
assert.equal(released.released, false)
assert.equal(released.reason, 'foreign')
assert.equal(storage.state.object.generation, '1500')
})
test('fails closed when the bucket answers 5xx', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.method === 'GET', status: 503 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.match(error.message, /lease inspection failed: 503/)
assert.equal(storage.state.requests.filter((request) => request.method !== 'GET').length, 0)
})
test('fails closed when permission is denied', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.method === 'GET', status: 403 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.match(error.message, /lease inspection failed: 403/)
})
test('treats an unreadable record as held, not free', async () => {
const storage = fakeStorage({
object: { generation: '1500', body: { holder_key: 'stablyai/orca/9001' } }
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseUnreadable)
assert.equal(storage.state.object.generation, '1500')
})
test('reports a 412 during acquisition as a conflict', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.path.startsWith('/upload/'), status: 412 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseConflict)
assert.match(error.message, /changed concurrently/)
})
test('renewal rewrites only expires_at on the observed generation', async () => {
const storage = fakeStorage()
const lease = leaseFor(storage)
await lease.acquire(HOLDER)
storage.state.object.body.expires_at = NOW - 1
const renewed = await lease.renew(HOLDER.holderKey)
assert.equal(renewed.renewed, true)
assert.equal(storage.state.object.body.expires_at, NOW + LEASE_TTL_MS)
assert.equal(storage.state.object.body.acquired_at, NOW)
assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey)
})
test('renewal stops once the object belongs to someone else', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
assert.deepEqual(await leaseFor(storage).renew(HOLDER.holderKey), {
renewed: false,
reason: 'foreign'
})
})
test('the access token source re-mints only after the reuse window', () => {
let clock = 0
let mints = 0
const source = createAccessTokenSource({
run: () => `token-${++mints}`,
now: () => clock
})
assert.equal(source(), 'token-1')
clock = 39 * 60 * 1_000
assert.equal(source(), 'token-1')
clock = 41 * 60 * 1_000
assert.equal(source(), 'token-2')
})
test('the access token source rejects an empty gcloud response', () => {
const source = createAccessTokenSource({ run: () => '' })
assert.throws(() => source(), /empty token/)
})
@@ -13,6 +13,10 @@ function readRootEntries(sha) {
return stdout.split('\0').filter(Boolean)
}
// Why: the Cloud workspace import is the one reviewed root addition; it stays
// listed until it lands on main, after which the base tree carries it.
const REVIEWED_ROOT_ENTRIES = new Set(['cloud'])
function checkRootDirectoryEntries(argv) {
if (argv.length !== 2) {
console.error(`Usage: ${process.argv[1]} <base-sha> <head-sha>`)
@@ -21,7 +25,9 @@ function checkRootDirectoryEntries(argv) {
const [baseSha, headSha] = argv
const baseEntries = new Set(readRootEntries(baseSha))
const blockedEntries = readRootEntries(headSha).filter((entry) => !baseEntries.has(entry))
const blockedEntries = readRootEntries(headSha).filter(
(entry) => !baseEntries.has(entry) && !REVIEWED_ROOT_ENTRIES.has(entry)
)
if (blockedEntries.length === 0) {
console.log('Root directory guard passed: no new root-level files or folders.')
@@ -0,0 +1,676 @@
name: Bootstrap Relay Staging Capacity
on:
workflow_dispatch:
inputs:
confirmation:
description: Enter BOOTSTRAP_STAGING_CAPACITY
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
bootstrap:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: us-central1
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
LEGACY_C3_IMAGE_DIGEST: sha256:2d0f6e6db2b0eb9d6aba188698de8330f8c30b4e76badfcf0fac3f3eb9508a87
steps:
- uses: actions/checkout@v4
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: capacity-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: access_token
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit bootstrap confirmation
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "BOOTSTRAP_STAGING_CAPACITY"
- name: Read and verify reviewed 600/60 topology
shell: bash
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
> "${RUNNER_TEMP}/relay-gce-state.json"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
DESIRED_IMAGES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.image })' \
| jq -r '.')"
DESIRED_ZONES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.zone })' \
| jq -r '.')"
DESIRED_TARGET_SIZES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(local.relay_gce_cell_target_sizes)' | jq -r '.')"
jq -e \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson target_sizes "${DESIRED_TARGET_SIZES_JSON}" \
'([.[] | select(.id == "staging-gce-c2")] | length == 1) and
([.[] | select(.id == "staging-gce-c3")] | length == 1) and
(any(.[]; .id == "staging-gce-c2" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
(any(.[]; .id == "staging-gce-c3" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
($images["staging-gce-c2"] == $images["staging-gce-c3"]) and
($target_sizes["staging-gce-c2"] == 1) and
($target_sizes["staging-gce-c3"] == 1)' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
jq \
--argjson desired "${DESIRED_CELLS_JSON}" \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson zones "${DESIRED_ZONES_JSON}" \
'($desired | map({key: .id, value: .}) | from_entries) as $cells |
to_entries | map(. as $entry | {
key: $entry.key,
value: ($entry.value + {
origin: $cells[$entry.key].url,
image: $images[$entry.key],
zone: $zones[$entry.key],
connection_hard_cap: $cells[$entry.key].connectionHardCap,
connection_unobserved_bound: $cells[$entry.key].connectionUnobservedBound
})
}) | from_entries' \
"${RUNNER_TEMP}/relay-gce-state.json" \
> "${RUNNER_TEMP}/relay-gce-topology.json"
ACTIVE_REVISION="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
DESIRED_IMAGE="$(jq -r '.["staging-gce-c2"]' <<< "${DESIRED_IMAGES_JSON}")"
gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -e \
--arg image "${DESIRED_IMAGE}" \
--arg capacity_service_account "${CAPACITY_SERVICE_ACCOUNT}" \
'(.spec.containers[0].image == $image) and
any(.spec.containers[0].env[]?;
.name == "ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT" and
.value == $capacity_service_account)' >/dev/null
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
jq -e --argjson desired "${DESIRED_CELLS_JSON}" '
def without_bootstrap_capacity:
map(if .id == "staging-gce-c2" or .id == "staging-gce-c3"
then del(.connectionHardCap, .connectionUnobservedBound)
else . end);
without_bootstrap_capacity == ($desired | without_bootstrap_capacity)
' <<< "${CURRENT_CELLS_JSON}" >/dev/null
- name: Bootstrap C2 then C3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
topology="${RUNNER_TEMP}/relay-gce-topology.json"
fixed_one_instance_name() {
local cell_id="$1"
local mig_name zone
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instance-groups managed list-instances \
"${mig_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("legacy cell does not have one stable running instance") end'
}
fixed_one_instance_id() {
local cell_id="$1"
local instance_name="$2"
local zone
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instances describe "${instance_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format='value(id)'
}
write_legacy_metrics() {
local cell_id="$1"
local instance_id="$2"
local after="$3"
local output="$4"
gcloud logging read \
"resource.type=\"gce_instance\" AND
resource.labels.instance_id=\"${instance_id}\" AND
jsonPayload.event=\"orca_relay_runtime_metrics\" AND
jsonPayload.cellId=\"${cell_id}\" AND
timestamp>=\"${after}\"" \
--project "${GCP_PROJECT_ID}" \
--limit 10 \
--order desc \
--format json \
| jq '[.[] | {
timestamp,
cellId: .jsonPayload.cellId,
metricVersion: .jsonPayload.metricVersion,
totalConnections: .jsonPayload.totalConnections,
preAuthConnections: .jsonPayload.preAuthConnections,
controls: .jsonPayload.controls,
splices: .jsonPayload.splices,
pendingSplices: .jsonPayload.pendingSplices,
queuedBytes: .jsonPayload.queuedBytes
}]' > "${output}"
}
verify_legacy_cell() {
local cell_id="$1"
local admission="$2"
local after="$3"
local expected_instance_id="$4"
local runtime_started_after="${5:-}"
local previous_incarnation_digest="${6:-}"
local hard_cap="${7:-}"
local unobserved_bound="${8:-}"
local capacity_state="${9:-}"
local current_instance current_instance_id metrics origin result
local runtime_start_args=() incarnation_args=() capacity_args=()
current_instance="$(fixed_one_instance_name "${cell_id}")"
current_instance_id="$(fixed_one_instance_id "${cell_id}" "${current_instance}")"
test "${current_instance_id}" = "${expected_instance_id}"
metrics="${RUNNER_TEMP}/${cell_id}-legacy-runtime-metrics.json"
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
if test -n "${runtime_started_after}"; then
runtime_start_args=(--runtime-started-after "${runtime_started_after}")
fi
if test -n "${previous_incarnation_digest}"; then
incarnation_args=(--previous-incarnation-digest "${previous_incarnation_digest}")
fi
if test -n "${hard_cap}"; then
capacity_args=(--hard-cap "${hard_cap}" --unobserved-bound "${unobserved_bound}")
fi
if test -n "${capacity_state}"; then
capacity_args+=(--capacity-state "${capacity_state}")
fi
for _attempt in $(seq 1 18); do
write_legacy_metrics \
"${cell_id}" "${current_instance_id}" "${after}" "${metrics}"
if result="$(node dev/scripts/verify-relay-legacy-bootstrap.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--admission "${admission}" \
--expected-image-digest "${LEGACY_C3_IMAGE_DIGEST}" \
--metrics-after "${after}" \
--metrics-file "${metrics}" \
"${runtime_start_args[@]}" \
"${incarnation_args[@]}" \
"${capacity_args[@]}")"; then
echo "${result}"
return 0
fi
sleep 10
done
return 1
}
post_admin() {
local origin="$1"
local path="$2"
local body="$3"
curl --fail --silent --show-error \
--request POST \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${body}" \
"${origin}${path}"
}
runtime_kind() {
local cell_id="$1"
local origin desired_digest digest
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
desired_digest="$(jq -r --arg cell "${cell_id}" \
'.[$cell].image | split("@") | last' "${topology}")"
digest="$(post_admin "${origin}" /v1/admin/runtime-status '{"v":1}' \
| jq -er '.imageDigest')"
if test "${digest}" = "${LEGACY_C3_IMAGE_DIGEST}"; then
echo legacy
elif test "${digest}" = "${desired_digest}"; then
echo modern
else
echo 'bootstrap cell image is neither legacy nor reviewed' >&2
return 1
fi
}
cell_admission() {
local cell_id="$1"
post_admin "${DIRECTOR_ORIGIN}" /v1/admin/cell-status \
"$(jq -cn --arg cell "${cell_id}" '{v:1, cellId:$cell}')" \
| jq -er '.status.admissionState |
if . == "general" or . == "migration-only" then .
else error("bootstrap admission is not recoverable") end'
}
verify_modern_cell() {
local cell_id="$1"
local admission="$2"
local origin
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission "${admission}" \
--draining forbidden \
--activity allowed
}
ensure_modern_general() {
local cell_id="$1"
local admission="$2"
verify_modern_cell "${cell_id}" "${admission}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids "${cell_id}"
verify_modern_cell "${cell_id}" general
}
prepare_legacy_c3_fallback() {
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c3.relay-staging.onorca.dev
legacy_c3_restart_started_after=
legacy_c3_old_incarnation=
}
normalize_legacy_c3() {
legacy_pre_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c2_instance="$(fixed_one_instance_name staging-gce-c2)"
legacy_c2_instance_id="$(fixed_one_instance_id \
staging-gce-c2 "${legacy_c2_instance}")"
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c2.relay-staging.onorca.dev
legacy_c3_isolated=false
restore_legacy_c3_fallback() {
if test "${legacy_c3_isolated}" = true; then
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore-fallback \
--general-cell-ids staging-gce-c2
fi
}
trap restore_legacy_c3_fallback EXIT
legacy_c3_isolated=true
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin https://c3.relay-staging.onorca.dev \
--cell-id staging-gce-c3 \
--mode isolate
legacy_c3_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_drained="$(verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_drain_boundary}" "${legacy_c3_instance_id}")"
echo "${legacy_c3_drained}"
legacy_c3_old_incarnation="$(jq -er '.incarnationDigest' \
<<< "${legacy_c3_drained}")"
legacy_c3_restart_started_after="$(node -e \
'process.stdout.write(new Date().toISOString())')"
gcloud compute instance-groups managed recreate-instances \
orca-cloud-staging-relay-gce-c3 \
--instances "${legacy_c3_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--quiet
gcloud compute instance-groups managed wait-until \
orca-cloud-staging-relay-gce-c3 \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--timeout 900
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
legacy_c3_isolated=false
trap - EXIT
}
roll_cell() (
local cell_id="$1"
local fallback_cell_id="$2"
local target_kind="$3"
local fallback_kind="$4"
local active_revision cell_origin current_cells_json desired_bound desired_cap
local desired_cells_json director_result image mig_name plan
local fallback_origin plan_changes plan_result restored target_drain_boundary
local target_instance target_instance_id zone
cell_origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
fallback_origin="$(jq -r --arg cell "${fallback_cell_id}" \
'.[$cell].origin' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
image="$(jq -r --arg cell "${cell_id}" '.[$cell].image' "${topology}")"
desired_cap="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_hard_cap' "${topology}")"
desired_bound="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_unobserved_bound' "${topology}")"
plan="${RUNNER_TEMP}/${cell_id}-capacity-bootstrap.tfplan"
restored=false
restore_fallback() {
if test "${restored}" = false; then
verify_fallback
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore-fallback \
--general-cell-ids "${fallback_cell_id}"
fi
}
verify_fallback() {
if test "${fallback_kind}" = legacy; then
verify_legacy_cell \
"${fallback_cell_id}" general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
return
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${fallback_origin}" \
--cell-id "${fallback_cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
}
verify_fallback
trap restore_fallback EXIT
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--mode isolate
target_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
if test "${target_kind}" = legacy; then
target_instance="$(fixed_one_instance_name "${cell_id}")"
target_instance_id="$(fixed_one_instance_id "${cell_id}" "${target_instance}")"
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}" absent-or-stale
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
active_revision="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${active_revision}"
current_cells_json="$(gcloud run revisions describe "${active_revision}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
desired_cells_json="$(jq -ce \
--arg cell "${cell_id}" \
--argjson cap "${desired_cap}" \
--argjson bound "${desired_bound}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${current_cells_json}")"
director_result="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service orca-cloud-relay-staging \
--image "${image}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${cell_id}" \
--director-cells-json "${desired_cells_json}" \
--min-instances 0 \
--prune-revisions true \
--release-id "bootstrap-${cell_id}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}")"
echo "${director_result}"
if test "${target_kind}" = legacy; then
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}"
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap "${desired_cap}" \
--unobserved-bound "${desired_bound}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
-out="${plan}"
plan_result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--image "${image}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${plan_result}"
plan_changes="$(jq -r '.changes' <<< "${plan_result}")"
[[ "${plan_changes}" =~ ^(0|2)$ ]]
if test "${plan_changes}" = 2; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
else
target_instance="$(fixed_one_instance_name "${cell_id}")"
gcloud compute instance-groups managed recreate-instances "${mig_name}" \
--instances "${target_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--quiet
fi
gcloud compute instance-groups managed wait-until "${mig_name}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--timeout 900
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
restored=true
trap - EXIT
)
c2_kind="$(runtime_kind staging-gce-c2)"
c3_kind="$(runtime_kind staging-gce-c3)"
c2_admission="$(cell_admission staging-gce-c2)"
c3_admission="$(cell_admission staging-gce-c3)"
bootstrap_phase="$(node dev/scripts/classify-relay-staging-bootstrap.mjs \
--c2-kind "${c2_kind}" \
--c2-admission "${c2_admission}" \
--c3-kind "${c3_kind}" \
--c3-admission "${c3_admission}")"
jq -cn --arg phase "${bootstrap_phase}" \
'{event:"relay_staging_bootstrap_phase", phase:$phase}'
case "${bootstrap_phase}" in
normalize-and-roll-both)
normalize_legacy_c3
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
resume-c2-then-c3)
prepare_legacy_c3_fallback
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
roll-c2)
ensure_modern_general staging-gce-c3 "${c3_admission}"
roll_cell staging-gce-c2 staging-gce-c3 legacy modern
;;
roll-c3)
ensure_modern_general staging-gce-c2 "${c2_admission}"
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
complete)
ensure_modern_general staging-gce-c2 "${c2_admission}"
ensure_modern_general staging-gce-c3 "${c3_admission}"
;;
*)
echo 'unsupported staging bootstrap phase' >&2
exit 1
;;
esac
- name: Verify both bootstrapped cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
for number in 2 3; do
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://c${number}.relay-staging.onorca.dev" \
--cell-id "staging-gce-c${number}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
done
@@ -0,0 +1,243 @@
name: Deploy Relay Asia Topology
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Validate a saved plan or apply that exact plan
required: true
default: plan
type: choice
options: [plan, apply]
cell-ids:
description: Exact reviewed comma-separated Asia cell set
required: true
type: string
image:
description: Full environment Relay image pinned by sha256 digest
required: true
type: string
confirmation:
description: Enter APPLY_RELAY_ASIA_TOPOLOGY for apply mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
topology:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
TARGET_IMAGE: ${{ inputs.image }}
TARGET_REGION: asia-east2
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
CLOUD_SQL_INSTANCE: ${{ inputs.environment == 'production' && 'orca-cloud-auth-db' || 'orca-cloud-staging-auth-db' }}
VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER: db-custom-4-15360
VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION: POSTGRES_17
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TF_VARS: ${{ inputs.environment == 'production' && 'environments/production.tfvars' || 'environments/staging.tfvars' }}
TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
TOPOLOGY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate the reviewed request before authentication
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
test -n "${TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${TOPOLOGY_SERVICE_ACCOUNT}"
case "${TARGET_ENVIRONMENT}:${TARGET_CELL_IDS}" in
staging:staging-gce-c4) ;;
production:production-gce-c27,production-gce-c28,production-gce-c29) ;;
*) echo "cell-ids do not match the reviewed environment topology" >&2; exit 1 ;;
esac
[[ "${TARGET_IMAGE}" =~ ^us-central1-docker\.pkg\.dev/${GCP_PROJECT_ID}/orca-cloud/relay@sha256:[0-9a-f]{64}$ ]]
if test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = APPLY_RELAY_ASIA_TOPOLOGY
else
test "${DEPLOY_MODE}" = plan
test -z "${CONFIRMATION}"
fi
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.TOPOLOGY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
- name: Require the checked Cloud SQL connection budget
shell: bash
run: |
set -euo pipefail
budget="$(node dev/scripts/relay-cloud-sql-connection-budget.mjs)"
checked_max="$(jq -er '.maxConnections' <<< "${budget}")"
jq -e '.withinBudget == true' <<< "${budget}" >/dev/null
if test "${TARGET_ENVIRONMENT}" = production; then
instance="$(gcloud sql instances describe "${CLOUD_SQL_INSTANCE}" \
--project "${GCP_PROJECT_ID}" --format=json)"
live_flag="$(jq -er '[.settings.databaseFlags[]? |
select(.name == "max_connections") | .value] |
if length <= 1 then (.[0] // "") else error("duplicate max_connections flags") end' \
<<< "${instance}")"
if test -n "${live_flag}"; then
live_max="${live_flag}"
live_source=explicit-flag
else
# The verified production database uses Cloud SQL's 400-connection
# default for this exact shape; fail closed if its shape changes.
test "$(jq -er '.settings.tier' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER}"
test "$(jq -er '.databaseVersion' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION}"
live_max=400
live_source=verified-shape-default
fi
test "${live_max}" = "${checked_max}"
else
live_max="not-read-for-staging"
live_source=not-read-for-staging
fi
{
echo "### Relay Cloud SQL connection budget"
echo "- Checked maximum: ${checked_max}"
echo "- Configured maximum: $(jq -er '.configuredMaximum' <<< "${budget}")"
echo "- Rollout operating maximum: $(jq -er '.operatingMaximum' <<< "${budget}")"
echo "- Explicit reserve: $(jq -er '.explicitReserve' <<< "${budget}")"
echo "- Production live max_connections: ${live_max}"
echo "- Production live maximum source: ${live_source}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Initialize the exact environment state
run: terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
- id: targets
name: Build the exact additive target set
shell: bash
run: |
set -euo pipefail
file="${RUNNER_TEMP}/relay-asia-targets"
: > "${file}"
printf '%s\n' \
'-target=google_compute_subnetwork.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router_nat.relay_gce_additional["asia-east2"]' \
'-target=google_compute_url_map.relay_gce[0]' >> "${file}"
IFS=, read -ra cells <<< "${TARGET_CELL_IDS}"
for cell_id in "${cells[@]}"; do
printf '%s\n' \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_backend_service.relay_gce_cell[\"${cell_id}\"]" >> "${file}"
done
echo "file=${file}" >> "${GITHUB_OUTPUT}"
- name: Create and validate the saved topology plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-asia-topology.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology.json"
mapfile -t targets < "${{ steps.targets.outputs.file }}"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
committed="${RUNNER_TEMP}/relay-committed-asia-topology.json"
jq -e '{
relay_gce_cells: .variables.relay_gce_cells.value,
relay_gce_additional_region_subnetwork_cidrs:
.variables.relay_gce_additional_region_subnetwork_cidrs.value
}' "${plan_json}" > "${committed}"
node dev/scripts/prepare-relay-asia-topology-input.mjs \
--existing-json "${committed}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image "${TARGET_IMAGE}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
changes="$(jq -er '.changes' <<< "${result}")"
digest="$(sha256sum "${plan}" | awk '{print $1}')"
echo "plan=${plan}" >> "${GITHUB_OUTPUT}"
echo "changes=${changes}" >> "${GITHUB_OUTPUT}"
{
echo "### Relay Asia topology saved plan"
echo "- Environment: ${TARGET_ENVIRONMENT}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Region: ${TARGET_REGION}"
echo "- Mutating resources: ${changes}"
echo "- Saved-plan SHA-256: ${digest}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Apply only the validated saved plan
if: ${{ inputs.mode == 'apply' }}
run: terraform -chdir=infra/terraform apply -input=false -auto-approve "${{ steps.plan.outputs.plan }}"
- name: Prove the exact topology targets converged
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
set -euo pipefail
mapfile -t targets < "${{ steps.targets.outputs.file }}"
plan="${RUNNER_TEMP}/relay-asia-topology-readback.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology-readback.json"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
test "$(jq -er '.changes' <<< "${result}")" = 0
- name: Record the required selector-safe next step
if: ${{ inputs.mode == 'apply' }}
run: |
{
echo "### Required next step"
echo "The VMs are not eligible for ordinary placement yet."
echo "Register the exact new cells atomically as migration-only before any director configuration lists them."
echo "Rollback is migration-only admission; do not destroy the Asia network on rollout day."
} >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,90 @@
name: Deploy Relay Fence Broker
on:
workflow_dispatch:
inputs:
image-digest:
description: Immutable broker image digest built from this main commit
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
SERVICE_NAME: orca-cloud-relay-fence
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay-fence-broker
IMAGE_DIGEST: ${{ inputs.image-digest }}
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Resolve exact-commit broker image
run: |
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
TAGS="$(gcloud artifacts docker tags list "${IMAGE_REPOSITORY}" \
--filter="version:${IMAGE_DIGEST}" \
--format=json)"
jq -e --arg tag "/tags/sha-${GITHUB_SHA}" \
'any(.[]; .tag | endswith($tag))' <<< "${TAGS}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
- name: Deploy broker image only
run: |
gcloud run services update "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--image "${IMAGE}" \
--quiet
- name: Verify ready singleton revision
run: |
SERVICE="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
jq -e '.status.conditions[] | select(.type == "Ready" and .status == "True")' \
<<< "${SERVICE}"
REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) == 100)] |
if length == 1 then .[0].revisionName // empty else empty end' \
<<< "${SERVICE}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
@@ -0,0 +1,815 @@
name: Deploy Relay Production Capacity Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
target-cell-id:
required: true
type: string
confirmation:
required: true
type: string
monitor-run-id:
required: true
type: string
monitor-run-attempt:
required: true
type: string
evidence-mode:
required: true
type: string
wave-cell-ids:
required: true
type: string
wave-index:
required: true
type: string
source-wave-run-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 75
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
CAPACITY_CELL_IDS: production-gce-c7,production-gce-c8,production-gce-c9,production-gce-c10,production-gce-c13,production-gce-c14,production-gce-c15,production-gce-c16,production-gce-c19,production-gce-c20,production-gce-c21,production-gce-c22,production-gce-c23,production-gce-c24,production-gce-c25,production-gce-c26
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_DIRECTOR_IMAGE_DIGEST: sha256:01b7fc3e6dce66180034f268a2dc92c05458706c5b3a0dc4450dcdd6161f6e73
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DEPLOY_MODE: ${{ inputs.mode }}
EVIDENCE_MODE: ${{ inputs.evidence-mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
WAVE_INDEX: ${{ inputs.wave-index }}
SOURCE_WAVE_RUN_ID: ${{ inputs.source-wave-run-id }}
steps:
- name: Require exact reusable-workflow invocation
working-directory: .
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
if test "${EVIDENCE_MODE}" = continuation; then
test "${DEPLOY_MODE}" = apply
[[ "${WAVE_INDEX}" =~ ^[0-3]$ ]]
test "${WAVE_CELL_IDS}" != none
test "${SOURCE_WAVE_RUN_ID}" = none
elif test "${EVIDENCE_MODE}" = resume; then
test "${DEPLOY_MODE}" = apply
test "${WAVE_INDEX}" = resume
test "${WAVE_CELL_IDS}" != none
[[ "${SOURCE_WAVE_RUN_ID}" =~ ^[0-9]+$ ]]
else
test "${EVIDENCE_MODE}" = single
test "${WAVE_CELL_IDS}" = none
test "${WAVE_INDEX}" = 0
test "${SOURCE_WAVE_RUN_ID}" = none
fi
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
run: |
test -n "${GCP_REGION}"
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- id: resume-provenance
if: ${{ inputs.evidence-mode == 'resume' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
test "${MONITOR_RUN_ATTEMPT}" = 1
case "${MONITOR_RUN_ID}:${SOURCE_WAVE_RUN_ID}:${WAVE_CELL_IDS}:${TARGET_CELL_ID}" in
31554591366:31555510376:production-gce-c16,production-gce-c15,production-gce-c14,production-gce-c13:production-gce-c13)
EXPECTED_SHA=a917e8e1fc1a2654e8cb81ba39b57733ec56be9c
EXPECTED_SOURCE_ATTEMPT=1
;;
31562760783:31563664692:production-gce-c10,production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c10)
EXPECTED_SHA=6082e9ca89a918ca51f0c87db003f5e8805b64b7
EXPECTED_SOURCE_ATTEMPT=1
;;
31571019947:31572080665:production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c8)
EXPECTED_SHA=e59958130c9d9b7a6cd805df2678d08997842c7c
EXPECTED_SOURCE_ATTEMPT=2
;;
*) exit 1 ;;
esac
MONITOR_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${MONITOR_RUN_ID}" \
--jq 'select(.name == "Monitor Relay Production" and
.path == ".github/workflows/cloud-monitor-relay-production.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "success" and .run_attempt == 1) |
.head_sha')"
SOURCE_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq 'select(.name == "Deploy Relay Production Capacity" and
.path == ".github/workflows/cloud-deploy-relay-production-capacity.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "failure") |
.head_sha')"
SOURCE_ATTEMPT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq '.run_attempt')"
[[ "${MONITOR_SHA}" =~ ^[0-9a-f]{40}$ ]]
test "${MONITOR_SHA}" = "${EXPECTED_SHA}"
test "${SOURCE_SHA}" = "${MONITOR_SHA}"
test "${SOURCE_ATTEMPT}" = "${EXPECTED_SOURCE_ATTEMPT}"
echo "commit-sha=${MONITOR_SHA}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode == 'apply' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode == 'apply' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode == 'apply' }}
run: |
EVIDENCE_COMMIT_SHA="${GITHUB_SHA}"
if test "${EVIDENCE_MODE:-single}" = resume; then
EVIDENCE_COMMIT_SHA="${{ steps.resume-provenance.outputs.commit-sha }}"
fi
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${EVIDENCE_COMMIT_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- name: Download this workflow's wave authority
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Download the failed wave authority for resume
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ inputs.source-wave-run-id }}
- name: Require wave evidence consumed by this workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${GITHUB_RUN_ID}"
- name: Require wave evidence consumed by the failed source workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${SOURCE_WAVE_RUN_ID}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Require exact mutation confirmation
if: ${{ inputs.mode != 'verify' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if test "${EVIDENCE_MODE:-single}" = resume; then
test "${CONFIRMATION}" = "RESUME_SELECTED_CELL_TO_1000 ${TARGET_CELL_ID}"
elif test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = "RAISE_SELECTED_CELL_TO_1000"
else
test "${CONFIRMATION}" = "ROLL_BACK_SELECTED_CELL_TO_600 ${TARGET_CELL_ID}"
fi
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Build the exact selected-cell configuration
shell: bash
run: |
if test "${DEPLOY_MODE}" = rollback; then
TARGET_HARD_CAP=600
else
TARGET_HARD_CAP=1000
fi
TARGET_UNOBSERVED_BOUND=60
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
[[ "${TARGET_HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
OVERRIDE_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${TARGET_HARD_CAP}" \
--argjson bound "${TARGET_UNOBSERVED_BOUND}" \
'.[$cell].connection_hard_cap = $cap |
.[$cell].connection_unobserved_bound = $bound' \
<<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-capacity.tfvars.json"
BASE_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'local.relay_director_cells_json' | jq -er '.')"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
<<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
<<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
jq -e --arg cell "${TARGET_CELL_ID}" \
'any(.[]; .id == $cell and .connectionHardCap == 1000 and
.connectionUnobservedBound == 60)' \
<<< "${BASE_CELLS_JSON}" >/dev/null
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
DESIRED_IMAGE_DIGEST="${DESIRED_IMAGE##*@}"
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
test "${MIG_NAME}" = "orca-cloud-relay-gce-${TARGET_HOSTNAME}"
{
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "TARGET_HARD_CAP=${TARGET_HARD_CAP}"
echo "TARGET_UNOBSERVED_BOUND=${TARGET_UNOBSERVED_BOUND}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "BASE_CELLS_JSON=${BASE_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact compatible production image and topology
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_REVISION_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_IMAGE="$(jq -er '.spec.containers[0].image' <<< "${ACTIVE_REVISION_JSON}")"
ACTIVE_IMAGE_DIGEST="${ACTIVE_IMAGE##*@}"
if test "${ACTIVE_IMAGE}" != "${DESIRED_IMAGE}"; then
test "${ACTIVE_IMAGE_DIGEST}" = "${COMPATIBLE_DIRECTOR_IMAGE_DIGEST}"
test "${DESIRED_IMAGE_DIGEST}" = "${COMPATIBLE_CELL_IMAGE_DIGEST}"
fi
CURRENT_CELLS_JSON="$(jq -cer '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end' \
<<< "${ACTIVE_REVISION_JSON}")"
CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON="$(
node dev/scripts/read-relay-production-capacity-identity.mjs \
<<< "${ACTIVE_REVISION_JSON}"
)"
CLASSIFICATION="$(jq -nc \
--argjson baseCells "${BASE_CELLS_JSON}" \
--argjson currentCells "${CURRENT_CELLS_JSON}" \
--arg capacityCellIds "${CAPACITY_CELL_IDS}" \
--arg targetCellId "${TARGET_CELL_ID}" \
--argjson targetHardCap "${TARGET_HARD_CAP}" \
--argjson currentCapacityServiceAccount \
"${CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON}" \
'{baseCells:$baseCells, currentCells:$currentCells,
capacityCellIds:($capacityCellIds | split(",")),
targetCellId:$targetCellId, targetHardCap:$targetHardCap,
currentCapacityServiceAccount:$currentCapacityServiceAccount}' \
| node dev/scripts/classify-relay-production-capacity-director.mjs \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
TOPOLOGY_PHASE="$(jq -er '.topologyPhase' <<< "${CLASSIFICATION}")"
DESIRED_CELLS_JSON="$(jq -cer '.desiredCells' <<< "${CLASSIFICATION}")"
DIRECTOR_READY="$(jq -er \
'if (.directorReady | type) == "boolean" then
(.directorReady | tostring)
else error("invalid directorReady classification") end' \
<<< "${CLASSIFICATION}")"
{
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}"
echo "TOPOLOGY_PHASE=${TOPOLOGY_PHASE}"
echo "DIRECTOR_READY=${DIRECTOR_READY}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact wave predecessor topology
if: ${{ inputs.mode == 'apply' && (inputs.evidence-mode == 'continuation' || inputs.evidence-mode == 'resume') }}
run: test "${TOPOLOGY_PHASE}" = predecessor
- name: Verify fresh dry-run evidence against the live selector
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${TARGET_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Recheck exact wave state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--wave-index "${WAVE_INDEX}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
RETRY_ARGS=()
if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
"${RETRY_ARGS[@]}"
- name: Recheck exact isolated resume state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-resume-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
--retry-freshness
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining required \
--activity allowed \
--runtime required \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
- name: Consume the single-use dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Verify current selected-cell capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
CURRENT_CAP="${TARGET_HARD_CAP}"
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
if test "${TOPOLOGY_PHASE}" = predecessor; then
CURRENT_CAP=600
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${CURRENT_IMAGE_DIGEST}"
- name: Arm fail-closed mutation cleanup
if: ${{ inputs.mode != 'verify' }}
run: echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
- name: Reversibly isolate only the selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Drain the selected cell or prove an offline rollback
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
if node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain; then
echo "OFFLINE_ROLLBACK=false" >> "${GITHUB_ENV}"
elif test "${DEPLOY_MODE}" = rollback; then
echo "OFFLINE_ROLLBACK=true" >> "${GITHUB_ENV}"
else
exit 1
fi
- id: restart-auth-one
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: restart-gate-one
name: Require restart-safe selected-cell activity
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-one.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
GATE_LOG="${RUNNER_TEMP}/relay-capacity-restart-gate-one.log"
set +e
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" \
2> "${GATE_LOG}"
GATE_EXIT=$?
set -e
cat "${GATE_LOG}" >&2
if test "${GATE_EXIT}" = 0; then
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
if test "$(wc -l < "${GATE_LOG}" | tr -d ' ')" = 1 &&
grep -Eq '^capacity transition verification timed out: \{.*\}$' "${GATE_LOG}"; then
echo "settled=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
exit "${GATE_EXIT}"
- id: restart-auth-two
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require extended restart-safe selected-cell activity
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-two.outputs.id_token }}
run: |
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- name: Deploy only the reviewed director topology
if: ${{ inputs.mode != 'verify' }}
run: |
if test "${DIRECTOR_READY}" = true; then exit 0; fi
RELEASE_ID="capacity-${TARGET_HOSTNAME}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--max-instances 5 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--min-instances 5 \
--prune-revisions false \
--release-id "${RELEASE_ID}"
- id: director-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require fail-closed director transition
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.director-transition-auth.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Plan and apply only the empty selected cell
if: ${{ inputs.mode != 'verify' }}
shell: bash
run: |
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${PLAN_RESULT}"
PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${PLAN_CHANGES}" =~ ^(0|1|2)$ ]]
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
else
INSTANCE="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--format=json | jq -er 'if length == 1 and
.[0].instanceStatus == "RUNNING" and .[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("selected cell is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" --instances "${INSTANCE}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" --stable --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --timeout 900
- id: capacity-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify fresh exact selected-cell heartbeat before admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- name: Restore only the selected cell to general admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the live general selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep the selected cell isolated after a failed mutation
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
CLEANUP_STATUS=0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate || CLEANUP_STATUS=$?
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain || CLEANUP_STATUS=$?
exit "${CLEANUP_STATUS}"
@@ -0,0 +1,351 @@
name: Deploy Relay Production Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify, change one cell, or raise a sequential wave
required: true
default: verify
type: choice
options:
- verify
- apply
- rollback
- wave-apply
- wave-resume
target-cell-id:
description: Exact serving cell for verify, apply, or rollback
required: true
default: production-gce-c26
type: choice
options:
- production-gce-c7
- production-gce-c8
- production-gce-c9
- production-gce-c10
- production-gce-c13
- production-gce-c14
- production-gce-c15
- production-gce-c16
- production-gce-c19
- production-gce-c20
- production-gce-c21
- production-gce-c22
- production-gce-c23
- production-gce-c24
- production-gce-c25
- production-gce-c26
wave-cell-ids:
description: Ordered comma-separated wave of two to four serving cells
required: false
default: none
type: string
confirmation:
description: Enter the exact single-cell or wave confirmation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID for apply
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt for apply
required: false
type: string
source-wave-run-id:
description: Failed wave run that isolated the resume target
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
single_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode != 'wave-apply' && inputs.mode != 'wave-resume') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: ${{ inputs.mode }}
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: single
wave-cell-ids: none
wave-index: '0'
source-wave-run-id: none
secrets: inherit
resume_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-resume' && github.ref == 'refs/heads/main') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: resume
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: resume
source-wave-run-id: ${{ inputs.source-wave-run-id }}
secrets: inherit
wave_gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-apply' && github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
env:
DIRECTOR_ORIGIN: https://relay.onorca.dev
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
steps:
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
run: |
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- id: wave
name: Validate the exact wave request
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
CELLS="$(node dev/scripts/relay-production-capacity-wave.mjs validate \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--confirmation "${CONFIRMATION}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify dry-run artifact before cloud authentication
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Verify wave evidence against the live selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
FIRST_CELL="$(jq -er '.[0]' <<< '${{ steps.wave.outputs.cells }}')"
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${FIRST_CELL}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Require exact 600/60 predecessor wave cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
while read -r CELL_ID; do
HOSTNAME="${CELL_ID#production-gce-}"
[[ "${HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://${HOSTNAME}.relay.onorca.dev" \
--cell-id "${CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests \
"${PREDECESSOR_IMAGE_DIGEST},${COMPATIBLE_CELL_IMAGE_DIGEST}"
done < <(jq -r '.[]' <<< '${{ steps.wave.outputs.cells }}')
- name: Consume the single-use dry-run evidence
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
wave_cell_1:
needs: wave_gate
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[0] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '0'
source-wave-run-id: none
secrets: inherit
wave_cell_2:
needs: [wave_gate, wave_cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[1] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '1'
source-wave-run-id: none
secrets: inherit
wave_cell_3:
if: ${{ needs.wave_cell_2.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[2] != null }}
needs: [wave_gate, wave_cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[2] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '2'
source-wave-run-id: none
secrets: inherit
wave_cell_4:
if: ${{ needs.wave_cell_3.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[3] != null }}
needs: [wave_gate, wave_cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[3] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '3'
source-wave-run-id: none
secrets: inherit
# Every wave job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- single_cell
- resume_cell
- wave_gate
- wave_cell_1
- wave_cell_2
- wave_cell_3
- wave_cell_4
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -0,0 +1,267 @@
name: Deploy Relay Production Director
on:
workflow_dispatch:
inputs:
image-digest:
description: "Immutable relay image digest (sha256: plus 64 lowercase hex characters)"
required: true
type: string
regional-placement-mode:
description: Preserve the live switch, explicitly enable Asia preference, or force US-first
required: true
default: preserve
type: choice
options: [preserve, enable, disable]
prune-incompatible-revisions:
description: Retain only the newly verified serving and rollback revisions
required: true
default: false
type: boolean
confirmation:
description: Enter the exact confirmation required by a destructive option
required: false
type: string
expected-rehome-generation:
description: Exact durable regional-rehome generation; it must remain disabled
required: true
type: string
bootstrap-runtime-identity:
description: One-time move from the stamped-cell identity to the director identity
required: true
default: false
type: boolean
predecessor-image-digest:
description: Exact immutable serving predecessor digest for the one-time identity bootstrap
required: true
type: string
permissions:
contents: read
id-token: write
# Director updates and candidate operations both mutate production relay control state.
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
IMAGE_DIGEST: ${{ inputs.image-digest }}
REGIONAL_PLACEMENT_MODE: ${{ inputs.regional-placement-mode }}
PRUNE_INCOMPATIBLE_REVISIONS: ${{ inputs.prune-incompatible-revisions }}
# Floor the served revision must keep, matching relay_min_instances in
# environments/production.tfvars. This gate only fails a bad deploy; Terraform
# still owns the value, and the candidate inherits it from the serving revision.
DIRECTOR_MIN_INSTANCES: 5
DIRECTOR_MAX_INSTANCES: 5
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
PREDECESSOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_RUNTIME_SERVICE_ACCOUNT }}
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Resolve immutable production image
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ ! "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "image-digest must be an immutable lowercase sha256 digest" >&2
exit 1
fi
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
[[ "${PRUNE_INCOMPATIBLE_REVISIONS}" =~ ^(true|false)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${BOOTSTRAP_RUNTIME_IDENTITY}" =~ ^(true|false)$ ]]
if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY
elif test "${PRUNE_INCOMPATIBLE_REVISIONS}" = true; then
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = PRUNE_INCOMPATIBLE_RELAY_DIRECTOR_REVISIONS
elif test "${REGIONAL_PLACEMENT_MODE}" = disable; then
test "${CONFIRMATION}" = FORCE_RELAY_US_FIRST
else
test -z "${CONFIRMATION}"
fi
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
# Why: the deploy INHERITS the serving revision's floor, so when that revision has
# already lost it the candidate inherits zero, the in-script gate compares zero against
# zero and passes, and the post-deploy check below only notices after traffic moved.
# The documented rollback target is created at minimum instances zero, so promoting it
# arms exactly that. Refuse to inherit a degraded floor rather than latch it.
- name: Require a healthy serving floor before deploying
shell: bash
run: |
SERVING="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${SERVING}"
FLOOR="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${FLOOR:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "serving revision ${SERVING} holds ${FLOOR:-0} minimum instances," \
"below ${DIRECTOR_MIN_INSTANCES}; deploying would inherit and latch it." >&2
echo "Restore the floor first: gcloud run services update ${DIRECTOR_SERVICE_NAME}" \
"--min-instances=${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
CEILING="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${CEILING}" = "${DIRECTOR_MAX_INSTANCES}"
echo "serving revision ${SERVING} holds ${FLOOR} minimum instances"
echo "SERVING_REVISION=${SERVING}" >> "${GITHUB_ENV}"
# Why: no --min-instances here. The candidate inherits the Terraform-owned
# scaling, and this step ends with 100% traffic on it. Pinning 1 rebuilt the
# per-instance admission shortage that took placement failures to ~70%.
- name: Deploy director blue/green
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
served_version="$(gcloud run revisions describe "${SERVING_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
(.version // .key // empty)] |
if length == 1 then .[0] else empty end')"
if [[ "${served_version}" =~ ^[1-9][0-9]*$ ]]; then
current_version="${served_version}"
else
test "${REGIONAL_PLACEMENT_MODE}" = preserve
current_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${current_version}" =~ ^[1-9][0-9]*$ ]]
fi
current="$(gcloud secrets versions access "${current_version}" \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
case "${REGIONAL_PLACEMENT_MODE}" in
preserve) desired="${current}" ;;
enable) desired=true ;;
disable) desired=false ;;
*) echo "regional-placement-mode is invalid" >&2; exit 1 ;;
esac
if test "${current}" != "${desired}"; then
target_version="$(printf '%s' "${desired}" | gcloud secrets versions add \
"${REGIONAL_PLACEMENT_SECRET}" --project "${GCP_PROJECT_ID}" --data-file=- \
--format='value(name)' --quiet | awk -F/ '{print $NF}')"
else
target_version="${current_version}"
fi
[[ "${target_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" \
--bootstrap-runtime-identity "${BOOTSTRAP_RUNTIME_IDENTITY}" \
--predecessor-image-digest "${PREDECESSOR_IMAGE_DIGEST}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience "${REHOME_AUDIENCE}" \
--rehome-control-origin https://relay.onorca.dev \
--admin-audience https://relay.onorca.dev/v1/admin/drain \
--expected-rehome-generation "${EXPECTED_REHOME_GENERATION}" \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--prune-revisions "${PRUNE_INCOMPATIBLE_REVISIONS}" \
--release-id "${RELEASE_ID}" \
--regional-placement-secret-version "${target_version}"
echo "REGIONAL_PLACEMENT_ENABLED=${desired}" >> "${GITHUB_ENV}"
echo "REGIONAL_PLACEMENT_VERSION=${target_version}" >> "${GITHUB_ENV}"
- name: Verify served revision and native health
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
REVISION="$(jq -r '[.status.traffic[] | select((.percent // 0) > 0)] | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' <<< "${SERVICE_JSON}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
SERVED_REGIONAL_PLACEMENT_SECRET="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)}] |
if length == 1 then .[0] else error("regional placement secret missing") end')"
test "$(jq -r '.secret' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_SECRET}"
test "$(jq -r '.version' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_VERSION}"
test "$(gcloud secrets versions access "${REGIONAL_PLACEMENT_VERSION}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" = "${REGIONAL_PLACEMENT_ENABLED}"
# Why: a served revision with no warm-instance floor still passes health and digest
# checks while quietly shrinking per-instance admission capacity.
SERVED_MIN_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${SERVED_MIN_INSTANCES:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "served revision ${REVISION} holds ${SERVED_MIN_INSTANCES:-0} minimum instances, expected at least ${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
SERVED_MAX_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${SERVED_MAX_INSTANCES}" = "${DIRECTOR_MAX_INSTANCES}"
SERVICE_URL="$(jq -r '.status.url' <<< "${SERVICE_JSON}")"
node dev/scripts/smoke-relay.mjs "${SERVICE_URL}"
@@ -0,0 +1,497 @@
name: Deploy Relay Production Multi-Target
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform source cell ID
required: true
type: string
target-cell-ids:
description: Comma-separated distinct Terraform target cell IDs
required: true
type: string
general-cell-ids:
description: Comma-separated proven cells that remain eligible for ordinary placement
required: false
type: string
unobserved-connection-bound:
description: Exact worst-case unobserved connection bound proven by the passing load gate
required: false
type: string
failed-target-cell-id:
description: Registered failed target to fence and supersede
required: false
type: string
replacement-target-cell-id:
description: Healthy replacement for registered failed target
required: false
type: string
mode:
description: Preflight/audit are read-only; other modes mutate production
required: true
default: preflight
type: choice
options:
- audit
- preflight
- cutover-admission
- add-migration-cells
- promote-general-cell
- retire-migration-cell
- execute
- recover-forward
- fence-source
- supersede-target
confirmation:
description: Enter CUTOVER_SELECTOR, ADD_MIGRATION_CELLS, PROMOTE_GENERAL_CELL, RETIRE_MIGRATION_CELL, EVACUATE_MULTI, RECOVER_FORWARD, or FENCE_SOURCE
required: false
type: string
selector-attempt-id:
description: Exact durable selector attempt ID for admission mutations
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
broker-operation-id:
description: Stable durable broker operation ID for target supersession
required: false
type: string
completed-fence-attempt-id:
description: Exact older completed fence attempt to recover without replay
required: false
type: string
completed-fence-commit:
description: Exact older fence commit bound to the completed attempt
required: false
type: string
completed-fence-operation:
description: Exact DONE Compute resize operation to adopt
required: false
type: string
completed-fence-state-serial:
description: Exact Terraform serial before the completed fence
required: false
type: string
completed-fence-plan-generation:
description: Exact saved-plan object generation
required: false
type: string
completed-fence-state-generation:
description: Exact current Terraform state object generation
required: false
type: string
completed-fence-state-sha256:
description: Exact current Terraform state object SHA-256
required: false
type: string
expected-lease-generation:
description: Exact live lease generation authorized for conditional takeover
required: false
type: string
expected-lease-operation-id:
description: Exact live lease operation ID authorized for takeover
required: false
type: string
expected-lease-request-digest:
description: Exact live lease request digest authorized for takeover
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_IDS: ${{ inputs.target-cell-ids }}
GENERAL_CELL_IDS: ${{ inputs.general-cell-ids }}
UNOBSERVED_CONNECTION_BOUND: ${{ inputs.unobserved-connection-bound }}
FAILED_TARGET_CELL_ID: ${{ inputs.failed-target-cell-id }}
REPLACEMENT_TARGET_CELL_ID: ${{ inputs.replacement-target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
BROKER_OPERATION_ID: ${{ inputs.broker-operation-id }}
COMPLETED_FENCE_ATTEMPT_ID: ${{ inputs.completed-fence-attempt-id }}
COMPLETED_FENCE_COMMIT: ${{ inputs.completed-fence-commit }}
COMPLETED_FENCE_OPERATION: ${{ inputs.completed-fence-operation }}
COMPLETED_FENCE_STATE_SERIAL: ${{ inputs.completed-fence-state-serial }}
COMPLETED_FENCE_PLAN_GENERATION: ${{ inputs.completed-fence-plan-generation }}
COMPLETED_FENCE_STATE_GENERATION: ${{ inputs.completed-fence-state-generation }}
COMPLETED_FENCE_STATE_SHA256: ${{ inputs.completed-fence-state-sha256 }}
EXPECTED_LEASE_GENERATION: ${{ inputs.expected-lease-generation }}
EXPECTED_LEASE_OPERATION_ID: ${{ inputs.expected-lease-operation-id }}
EXPECTED_LEASE_REQUEST_DIGEST: ${{ inputs.expected-lease-request-digest }}
steps:
- uses: actions/checkout@v4
- name: Require private fence-broker environment
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
env:
FENCE_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
FENCE_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
FENCE_BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
test -n "${FENCE_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${FENCE_SERVICE_ACCOUNT}"
test -n "${FENCE_BROKER_URI}"
- name: Reject direct-runner Terraform fence aborts
if: ${{ inputs.mode == 'abort-fence-source' }}
run: |
echo "Terraform fence aborts require a reviewed private-broker recovery path." >&2
exit 1
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
if: ${{ inputs.mode != 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "cutover-admission" ]]; then
test "${CONFIRMATION}" = "CUTOVER_SELECTOR"
elif [[ "${DEPLOY_MODE}" = "add-migration-cells" ]]; then
test "${CONFIRMATION}" = "ADD_MIGRATION_CELLS"
elif [[ "${DEPLOY_MODE}" = "promote-general-cell" ]]; then
test "${CONFIRMATION}" = "PROMOTE_GENERAL_CELL"
elif [[ "${DEPLOY_MODE}" = "retire-migration-cell" ]]; then
test "${CONFIRMATION}" = "RETIRE_MIGRATION_CELL"
elif [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE_MULTI"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "fence-source" ]]; then
test "${CONFIRMATION}" = "FENCE_SOURCE"
elif [[ "${DEPLOY_MODE}" = "supersede-target" ]]; then
test "${CONFIRMATION}" = "SUPERSEDE_TARGET"
elif [[ "${DEPLOY_MODE}" = "abort-fence-source" ]]; then
test "${CONFIRMATION}" = "ABORT_FENCE"
else
test "${CONFIRMATION}" = "FENCE_SOURCE"
fi
- name: Require exact source-fence broker contract
if: ${{ inputs.mode == 'fence-source' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${TARGET_CELL_IDS}" = "production-gce-c7,production-gce-c8,production-gce-c10,production-gce-c13,production-gce-c17,production-gce-c18"
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Require exact broker cell contract
if: ${{ inputs.mode == 'supersede-target' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${FAILED_TARGET_CELL_ID}" = "production-gce-c12"
test "${REPLACEMENT_TARGET_CELL_ID}" = "production-gce-c13"
test "${TARGET_CELL_IDS}" = "production-gce-c12,production-gce-c13"
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
[[ "${COMPLETED_FENCE_ATTEMPT_ID}" =~ ^[0-9a-f-]{36}$ ]]
[[ "${COMPLETED_FENCE_COMMIT}" =~ ^[0-9a-f]{40}$ ]]
[[ "${COMPLETED_FENCE_OPERATION}" =~ ^[A-Za-z0-9._-]{1,256}$ ]]
[[ "${COMPLETED_FENCE_STATE_SERIAL}" =~ ^[0-9]+$ ]]
[[ "${COMPLETED_FENCE_PLAN_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_SHA256}" =~ ^[0-9a-f]{64}$ ]]
test -n "${EXPECTED_LEASE_GENERATION}"
fi
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Read reviewed Terraform topology
if: ${{ inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
DIRECTOR_MIN_INSTANCES="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars <<< 'var.relay_min_instances')"
[[ "${DIRECTOR_MIN_INSTANCES}" =~ ^[1-9][0-9]*$ ]]
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
echo "DIRECTOR_MIN_INSTANCES=${DIRECTOR_MIN_INSTANCES}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
SCOPED_RECOVERY_ARGS=()
if [[ ("${DEPLOY_MODE}" = "execute" ||
"${DEPLOY_MODE}" = "recover-forward") &&
"${SOURCE_CELL_ID}" = "production-gce-c12" ]]; then
SCOPED_RECOVERY_ARGS=(
--scoped-recovery-source-cell-id
production-gce-c3
)
fi
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
"${SCOPED_RECOVERY_ARGS[@]}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- id: google-fence-broker-auth
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
id_token_include_email: true
- name: Invoke private target-supersession broker
if: ${{ inputs.mode == 'supersede-target' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg attemptId "${COMPLETED_FENCE_ATTEMPT_ID}" \
--arg completedCommit "${COMPLETED_FENCE_COMMIT}" \
--arg gceOperation "${COMPLETED_FENCE_OPERATION}" \
--arg stateSerial "${COMPLETED_FENCE_STATE_SERIAL}" \
--arg planGeneration "${COMPLETED_FENCE_PLAN_GENERATION}" \
--arg stateGeneration "${COMPLETED_FENCE_STATE_GENERATION}" \
--arg stateSha256 "${COMPLETED_FENCE_STATE_SHA256}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
completedFenceRecovery:{attemptId:$attemptId,fenceCommit:$completedCommit,
gceOperation:$gceOperation,terraformStateSerial:($stateSerial|tonumber),
planObjectGeneration:$planGeneration,
terraformStateObjectGeneration:$stateGeneration,
terraformStateObjectSha256:$stateSha256},
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
elif [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,confirmation:"SUPERSEDE_TARGET"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/supersede-target" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Invoke private source-fence broker
if: ${{ inputs.mode == 'fence-source' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"FENCE_SOURCE"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),confirmation:"FENCE_SOURCE"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/fence-source" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Preflight or run multi-target evacuation
if: >-
${{ inputs.mode != 'fence-source' &&
inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-multi-target.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-ids "${TARGET_CELL_IDS}" \
--general-cell-ids "${GENERAL_CELL_IDS}" \
--unobserved-connection-bound "${UNOBSERVED_CONNECTION_BOUND}" \
--director-region "${{ vars.PRODUCTION_GCP_REGION }}" \
--director-service "orca-cloud-relay" \
--director-min-instances "${DIRECTOR_MIN_INSTANCES}" \
--selector-attempt-id "${SELECTOR_ATTEMPT_ID}" \
--failed-target-cell-id "${FAILED_TARGET_CELL_ID}" \
--replacement-target-cell-id "${REPLACEMENT_TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--environment production \
--fence-commit "${GITHUB_SHA}" \
--terraform-dir infra/terraform \
--terraform-var-file environments/production.tfvars \
--mode "${DEPLOY_MODE}" \
--connection-ceiling 1000 \
--minimum-lease-remaining-ms 600000
@@ -0,0 +1,641 @@
name: Deploy Relay Production Same-Cap Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
target-cell-id: { required: true, type: string }
target-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
target-rehome-protocol: { required: true, type: string }
rollback-rehome-protocol: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-rehome-generation: { required: true, type: string }
monitor-run-id: { required: true, type: string }
monitor-run-attempt: { required: true, type: string }
wave-index: { required: true, type: string }
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
rollout:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 75
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay.onorca.dev
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
TARGET_REHOME_PROTOCOL: ${{ inputs.target-rehome-protocol }}
ROLLBACK_REHOME_PROTOCOL: ${{ inputs.rollback-rehome-protocol }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
WAVE_INDEX: ${{ inputs.wave-index }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${TARGET_IMAGE_DIGEST}" != "${ROLLBACK_IMAGE_DIGEST}"
[[ "${TARGET_REHOME_PROTOCOL}" =~ ^[01]$ ]]
[[ "${ROLLBACK_REHOME_PROTOCOL}" =~ ^[01]$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${WAVE_INDEX}" =~ ^[0-3]$ ]]
if test "${DEPLOY_MODE}" = verify; then
EFFECTIVE_SELECTOR_GENERATION="${EXPECTED_SELECTOR_GENERATION}"
else
EFFECTIVE_SELECTOR_GENERATION="$((EXPECTED_SELECTOR_GENERATION + (2 * WAVE_INDEX)))"
fi
echo "EFFECTIVE_SELECTOR_GENERATION=${EFFECTIVE_SELECTOR_GENERATION}" >> "${GITHUB_ENV}"
if test "${DEPLOY_MODE}" != verify && test "${GITHUB_RUN_ATTEMPT}" != 1; then
echo "mutations are single-dispatch: re-runs replay aged evidence," >&2
echo "so recover each remaining cell with its own fresh monitor" >&2
echo "dry-run and canary-apply dispatch instead" >&2
exit 1
fi
test -n "${GCP_REGION}"
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WIF}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with: { package_json_file: cloud/package.json }
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with: { terraform_wrapper: false }
- name: Require fresh aggregate monitor evidence reference
if: ${{ inputs.mode != 'verify' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private aggregate monitor evidence
if: ${{ inputs.mode != 'verify' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify monitor evidence provenance
if: ${{ inputs.mode != 'verify' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--required-migration-policy strict \
--wave-index "${WAVE_INDEX}"
- name: Download this wave's single-use safety authority
if: ${{ inputs.mode != 'verify' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-same-cap-monitor-authority
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Require safety evidence consumed by this workflow
if: ${{ inputs.mode != 'verify' }}
run: |
# Mutations are single-dispatch: a fresh dispatch cannot resume a
# partial batch (the canary authority binds the batch-entry selector
# generation), so each remaining cell is recovered by its own fresh
# monitor dry-run and canary-apply dispatch, never by re-running
# aged evidence.
test "${GITHUB_RUN_ATTEMPT}" = 1
MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}")" = \
"${GITHUB_RUN_ID}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Recheck aggregate SQL, pool, reconnect, migration, and selector safety
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
RETRY_ARGS=()
if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-index "${WAVE_INDEX}" "${RETRY_ARGS[@]}"
- name: Require durable rehome disabled and exact selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect \
--director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EFFECTIVE_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Resolve immutable same-cap cell configuration
shell: bash
run: |
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
case "${TARGET_HOSTNAME}" in
c7|c8|c9|c10|c13|c14|c15|c16|c19|c20|c21|c22|c23|c24|c25|c26)
EXPECTED_HARD_CAP=1000
EXPECTED_REGION=us-central1
;;
c27|c28|c29)
EXPECTED_HARD_CAP=3000
EXPECTED_REGION=asia-east2
;;
*) exit 1 ;;
esac
EXPECTED_UNOBSERVED_BOUND=60
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
SOURCE_CELLS="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_region_rehome_source_cell_ids)' | jq -er '.')"
if test "${EXPECTED_REGION}" = us-central1; then
jq -e --arg cell "${TARGET_CELL_ID}" 'index($cell) != null' \
<<< "${SOURCE_CELLS}" >/dev/null
fi
CURRENT_SHAPE="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${CELLS_JSON}")"
test "$(jq -r '.connection_hard_cap' <<< "${CURRENT_SHAPE}")" = "${EXPECTED_HARD_CAP}"
test "$(jq -r '.connection_unobserved_bound' <<< "${CURRENT_SHAPE}")" = \
"${EXPECTED_UNOBSERVED_BOUND}"
TARGET_ZONE="$(jq -r '.zone' <<< "${CURRENT_SHAPE}")"
MIG_NAME="orca-cloud-relay-gce-${TARGET_HOSTNAME}"
if test "${DEPLOY_MODE}" = rollback; then
DESIRED_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
else
DESIRED_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
fi
DESIRED_IMAGE="${IMAGE_REPOSITORY}@${DESIRED_IMAGE_DIGEST}"
OVERRIDE_CELLS_JSON="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${DESIRED_IMAGE}" '.[$cell].image = $image' <<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-same-cap.tfvars.json"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${DESIRED_IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"
{
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "EXPECTED_HARD_CAP=${EXPECTED_HARD_CAP}"
echo "EXPECTED_UNOBSERVED_BOUND=${EXPECTED_UNOBSERVED_BOUND}"
echo "EXPECTED_REGION=${EXPECTED_REGION}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "CURRENT_IMAGE_DIGEST=${CURRENT_IMAGE_DIGEST}"
echo "DESIRED_REHOME_PROTOCOL=${DESIRED_REHOME_PROTOCOL}"
echo "CURRENT_REHOME_PROTOCOL=${CURRENT_REHOME_PROTOCOL}"
} >> "${GITHUB_ENV}"
- name: Verify exact current generation, digest, cap, and rollback point
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
CURRENT_RUNTIME="$(curl --fail-with-body --max-time 30 \
--request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
# A rollback that failed between template apply and admission restore
# leaves the cell already on the rollback image; resume from that
# state instead of demanding the pre-rollback predecessor.
LIVE_IMAGE_DIGEST="$(jq -r '.imageDigest' <<< "${CURRENT_RUNTIME}")"
if test "${DEPLOY_MODE}" = rollback \
&& test "${LIVE_IMAGE_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"; then
ROLLBACK_RESUME=true
PREDECESSOR_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${DESIRED_REHOME_PROTOCOL}"
else
ROLLBACK_RESUME=false
PREDECESSOR_IMAGE_DIGEST="${CURRENT_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${CURRENT_REHOME_PROTOCOL}"
fi
RESTORED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
RESTORED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
test -n "${RESTORED_MIGRATION_CELLS}" || RESTORED_MIGRATION_CELLS=none
test -n "${RESTORED_GENERAL_CELLS}" || RESTORED_GENERAL_CELLS=none
ISOLATED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
ISOLATED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
test -n "${ISOLATED_MIGRATION_CELLS}" || ISOLATED_MIGRATION_CELLS=none
test -n "${ISOLATED_GENERAL_CELLS}" || ISOLATED_GENERAL_CELLS=none
{
echo "ROLLBACK_RESUME=${ROLLBACK_RESUME}"
# The failsafe consumes these; deriving them here keeps them
# defined for a failure in any later step.
echo "ISOLATED_MIGRATION_CELLS=${ISOLATED_MIGRATION_CELLS}"
echo "ISOLATED_GENERAL_CELLS=${ISOLATED_GENERAL_CELLS}"
# No restart happens on resume, so isolate below is skipped and
# cannot advance the selector generation.
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${EFFECTIVE_SELECTOR_GENERATION}"
# A failed-canary rollback enters with the target migration-only,
# so the restore inspect cannot reuse the entry membership inputs.
echo "RESTORED_MIGRATION_CELLS=${RESTORED_MIGRATION_CELLS}"
echo "RESTORED_GENERAL_CELLS=${RESTORED_GENERAL_CELLS}"
} >> "${GITHUB_ENV}"
if ! jq -e --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \
&& test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \
'.role == "cell" and .cellId == $cell and .cellUrl == $origin and
(.region == $region or
($region == "us-central1" and $protocol == 0 and .region == null)) and
.imageDigest == $digest and
.connectionCapacity.hardCap == $hardCap and
.connectionCapacity.unobservedBound == $unobservedBound and
(.draining == false or $drainingOk) and
(.regionalRehomeProtocol // 0) == $protocol' <<< "${CURRENT_RUNTIME}" >/dev/null
then
jq -r --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \
&& test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \
'[
if .role != "cell" then "role" else empty end,
if .cellId != $cell then "cellId" else empty end,
if .cellUrl != $origin then "cellUrl" else empty end,
if (.region != $region and
($region != "us-central1" or $protocol != 0 or .region != null))
then "region" else empty end,
if .imageDigest != $digest then "imageDigest" else empty end,
if .connectionCapacity.hardCap != $hardCap then "hardCap" else empty end,
if .connectionCapacity.unobservedBound != $unobservedBound then "unobservedBound" else empty end,
if (.draining != false and ($drainingOk | not)) then "draining" else empty end,
if (.regionalRehomeProtocol // 0) != $protocol then "regionalRehomeProtocol" else empty end
] | "runtime predecessor mismatch fields=" + join(",")' \
<<< "${CURRENT_RUNTIME}" >&2
exit 1
fi
# The exact legacy digest binds omitted pre-region fields to US and protocol 0.
jq -r '[
if .region == null then "region" else empty end,
if .regionalRehomeProtocol == null then "regionalRehomeProtocol" else empty end
] | if length > 0 then "runtime predecessor normalized legacy fields=" + join(",") else empty end' \
<<< "${CURRENT_RUNTIME}"
CURRENT_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \
--request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
SOURCE_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${CURRENT_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true && ! jq -e \
'.status.admissionState == "migration-only"' \
<<< "${CURRENT_DIRECTOR_STATUS}" >/dev/null; then
echo 'resume requires the isolated migration-only cell a failed rollback leaves' >&2
exit 1
fi
[[ "${SOURCE_INCARNATION}" =~ ^[0-9a-f-]{36}$ ]]
echo "SOURCE_INCARNATION=${SOURCE_INCARNATION}" >> "${GITHUB_ENV}"
# Rollback is the documented recovery from a failed canary, which
# leaves the cell migration-only (and possibly still marked
# draining); apply and verify still require a pristine general cell.
if test "${DEPLOY_MODE}" = rollback; then
PRECHECK_ADMISSION=general-or-migration-only
PRECHECK_DRAINING=either
else
PRECHECK_ADMISSION=general
PRECHECK_DRAINING=forbidden
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission "${PRECHECK_ADMISSION}" \
--draining "${PRECHECK_DRAINING}" --activity allowed \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
- name: Finish read-only verification
if: ${{ inputs.mode == 'verify' }}
run: echo 'Exact same-cap rollback point verified.'
- name: Reversibly isolate and drain only the selected cell
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# A cell isolated by a failed canary is already migration-only, so
# isolate is a no-op there that does not advance the selector; the
# result's generation is authoritative either way.
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate)"
echo "${ISOLATE_RESULT}"
ISOLATE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${ISOLATE_GENERATION}" >> "${GITHUB_ENV}"
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode drain
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat either --admission migration-only --draining required \
--activity restart-safe --expected-image-digests "${CURRENT_IMAGE_DIGEST}" \
--timeout-ms 900000
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- name: Require converged Terraform state and a stable MIG on resume
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME == 'true' }}
shell: bash
env:
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
# Zero resource changes prove the prior run's apply completed and no
# restart will follow, keeping the incarnation check honest. Root
# outputs may lag a targeted apply, so judge resource_changes only.
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
if ! terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -e '[.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))]
| length == 0' >/dev/null
then
# An apply that failed before its template apply also resumes here
# (the cell still serves the rollback image), and repo drift since
# the cell's last roll (for example newly added rehome trust
# config) then legitimately replaces the template. Nothing is
# applied on resume either way, so accept exactly the drift the
# reviewed validator would let a real apply ship for the image the
# cell already serves: the template leaves and re-enters the
# rollback image, as exactly the template-and-MIG change pair.
terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -r '"resume found unconverged resources: " +
([.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))
| .address] | join(","))'
echo 'requiring reviewed rollback-image drift'
terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--rollback-image "${DESIRED_IMAGE}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
| jq -e '.changes == 2' >/dev/null
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
- name: Apply only the selected same-cap template and MIG
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
shell: bash
env:
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
terraform -chdir=infra/terraform show -json "${RUNNER_TEMP}/relay-same-cap.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" --image "${DESIRED_IMAGE}" \
--rollback-image "${IMAGE_REPOSITORY}@${CURRENT_IMAGE_DIGEST}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-same-cap.tfplan"
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
- id: post-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new incarnation, exact image, protocol, and durable safety
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity allowed --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" --timeout-ms 900000
TARGET_RUNTIME="$(curl --fail-with-body --max-time 30 \
--request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
jq -e --arg digest "${DESIRED_IMAGE_DIGEST}" \
--argjson protocol "${DESIRED_REHOME_PROTOCOL}" \
'.imageDigest == $digest and (.regionalRehomeProtocol // 0) == $protocol' \
<<< "${TARGET_RUNTIME}" >/dev/null
TARGET_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \
--request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
TARGET_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${TARGET_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# No restart happened; the incarnation legitimately stays put.
test "${TARGET_INCARNATION}" = "${SOURCE_INCARNATION}"
else
test "${TARGET_INCARNATION}" != "${SOURCE_INCARNATION}"
fi
echo "TARGET_INCARNATION=${TARGET_INCARNATION}" >> "${GITHUB_ENV}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ISOLATE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- name: Prove exact per-host trust and idempotent no-neighbor behavior
if: ${{ inputs.mode != 'verify' && ((inputs.mode == 'rollback' && inputs.rollback-rehome-protocol == '1') || (inputs.mode != 'rollback' && inputs.target-rehome-protocol == '1')) }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
node dev/scripts/probe-relay-rehome-trust.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-id "${TARGET_CELL_ID}" \
--cell-incarnation "${TARGET_INCARNATION}"
- name: Restore only the verified selected cell to general admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
ACTIVATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode activate)"
echo "${ACTIVATE_RESULT}"
SELECTOR_GENERATION_AFTER_ACTIVATE="$(jq -er '.generation' \
<<< "${ACTIVATE_RESULT}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission general --draining forbidden --activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ACTIVATE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${RESTORED_MIGRATION_CELLS}" \
--expected-general-cells "${RESTORED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep a failed cell isolated and rehome disabled
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate)"
echo "${ISOLATE_RESULT}"
# The isolate result carries the authoritative post-isolate generation;
# fixed offsets are wrong whenever an earlier isolate was a no-op.
FAILSAFE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${FAILSAFE_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}"
@@ -0,0 +1,310 @@
name: Deploy Relay Production Same-Cap
on:
workflow_dispatch:
inputs:
mode:
description: Verify, roll one canary, roll a bounded batch, or roll back
required: true
default: verify
type: choice
options: [verify, canary-apply, batch-apply, rollback]
cell-ids:
description: Ordered comma-separated serving cells; one canary or two to four batch cells
required: true
type: string
target-image-digest:
description: Exact immutable compatibility image digest
required: true
type: string
rollback-image-digest:
description: Exact immutable currently serving rollback digest
required: true
type: string
target-rehome-protocol:
description: Exact target regional-rehome protocol
required: true
default: '1'
type: choice
options: ['0', '1']
rollback-rehome-protocol:
description: Exact rollback regional-rehome protocol
required: true
default: '0'
type: choice
options: ['0', '1']
expected-selector-generation:
description: Exact selector generation before the first cell
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-rehome-generation:
description: Exact durable regional-rehome control generation; it must be disabled
required: true
type: string
monitor-run-id:
description: Fresh successful aggregate dry-run monitor workflow run
required: false
type: string
monitor-run-attempt:
description: Exact monitor attempt
required: false
type: string
canary-run-id:
description: Successful same-commit canary run required for batch-apply
required: false
type: string
confirmation:
description: Exact digest-and-cell-bound mutation confirmation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
job-mode: ${{ steps.wave.outputs.job-mode }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
- id: wave
env:
MODE: ${{ inputs.mode }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
CONFIRMATION: ${{ inputs.confirmation }}
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
run: |
CELLS="$(node dev/scripts/relay-production-same-cap-wave.mjs validate \
--mode "${MODE}" --cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" --rollback-digest "${ROLLBACK_DIGEST}" \
--confirmation "${CONFIRMATION}" --canary-run-id "${CANARY_RUN_ID}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
if [[ "${MODE}" =~ ^(canary-apply|batch-apply)$ ]]; then
echo 'job-mode=apply' >> "${GITHUB_OUTPUT}"
else
echo "job-mode=${MODE}" >> "${GITHUB_OUTPUT}"
fi
- name: Download exact prior canary authority
if: ${{ inputs.mode == 'batch-apply' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-canary-${{ inputs.canary-run-id }}
path: ${{ runner.temp }}/relay-same-cap-canary
github-token: ${{ github.token }}
run-id: ${{ inputs.canary-run-id }}
- name: Verify canary authority against this batch
if: ${{ inputs.mode == 'batch-apply' }}
env:
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
run: |
node dev/scripts/relay-production-same-cap-wave.mjs verify-canary \
--file "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${CANARY_RUN_ID}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}"
- name: Reject previously consumed aggregate safety evidence
if: ${{ inputs.mode != 'verify' }}
env:
GH_TOKEN: ${{ github.token }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = 0
mkdir -p "${RUNNER_TEMP}/relay-same-cap-monitor-authority"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}"
- name: Consume aggregate safety evidence for this exact wave
if: ${{ inputs.mode != 'verify' }}
uses: actions/upload-artifact@v4
with:
name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-same-cap-monitor-authority/relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
cell_1:
needs: gate
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[0] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '0'
secrets: inherit
cell_2:
if: ${{ needs.cell_1.result == 'success' && fromJSON(needs.gate.outputs.cells)[1] != null }}
needs: [gate, cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[1] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '1'
secrets: inherit
cell_3:
if: ${{ needs.cell_2.result == 'success' && fromJSON(needs.gate.outputs.cells)[2] != null }}
needs: [gate, cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[2] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '2'
secrets: inherit
cell_4:
if: ${{ needs.cell_3.result == 'success' && fromJSON(needs.gate.outputs.cells)[3] != null }}
needs: [gate, cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[3] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '3'
secrets: inherit
seal_canary:
if: ${{ inputs.mode == 'canary-apply' }}
needs: [gate, cell_1]
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
- name: Seal exact successful canary authority
run: |
mkdir -p "${RUNNER_TEMP}/relay-same-cap-canary"
node dev/scripts/relay-production-same-cap-wave.mjs create-canary \
--cell-id "${{ inputs.cell-ids }}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--confirmation "${{ inputs.confirmation }}" \
--commit-sha "${GITHUB_SHA}" --run-id "${GITHUB_RUN_ID}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}" \
> "${RUNNER_TEMP}/relay-same-cap-canary/authority.json"
- uses: actions/upload-artifact@v4
with:
name: relay-same-cap-canary-${{ github.run_id }}
path: ${{ runner.temp }}/relay-same-cap-canary/authority.json
retention-days: 30
if-no-files-found: error
# Every cell job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- gate
- cell_1
- cell_2
- cell_3
- cell_4
- seal_canary
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -0,0 +1,217 @@
name: Deploy Relay Production Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Audit/preflight are read-only; recover/continue resume committed work; disable/enable/reset/execute mutate admission
required: true
default: preflight
type: choice
options:
- audit
- preflight
- recover-forward
- continue-evacuation
- disable-cell
- enable-empty-cell
- reset-empty-candidate
- execute
confirmation:
description: Enter RECOVER_FORWARD, CONTINUE_EVACUATION, DISABLE_CELL, ENABLE_CELL, RESET_CANDIDATE, or EVACUATE for the matching mutation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
steps:
- uses: actions/checkout@v4
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "continue-evacuation" ]]; then
test "${CONFIRMATION}" = "CONTINUE_EVACUATION"
elif [[ "${DEPLOY_MODE}" = "disable-cell" ]]; then
test "${CONFIRMATION}" = "DISABLE_CELL"
elif [[ "${DEPLOY_MODE}" = "enable-empty-cell" ]]; then
test "${CONFIRMATION}" = "ENABLE_CELL"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -0,0 +1,109 @@
name: Deploy Relay Staging GCE Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Preflight is read-only; reset repairs an empty candidate; execute evacuates
required: true
default: preflight
type: choice
options:
- preflight
- reset-empty-candidate
- execute
confirmation:
description: Enter RESET_CANDIDATE for reset or EVACUATE for execute
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
ADMIN_AUDIENCE: https://relay-staging.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -0,0 +1,112 @@
name: Deploy Relay Staging
on:
workflow_dispatch:
inputs:
expected-image-digest:
description: Exact checked-in production Relay sha256 digest to deploy
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
# Staging deploy, candidate, auth, and power operations must never overlap.
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
EXPECTED_IMAGE_DIGEST: ${{ inputs.expected-image-digest }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
ASIA_PROOF_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
steps:
- uses: actions/checkout@v4
- name: Require the expected immutable image
run: '[[ "${EXPECTED_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]'
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Bind the request to the checked-in staging C4 image
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure \
-backend-config=backend/staging.hcl -input=false
IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'var.relay_gce_cells["staging-gce-c4"].image' | jq -er '.')"
test "${IMAGE}" = \
"${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${EXPECTED_IMAGE_DIGEST}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require the mirrored immutable image
run: |
DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${DIGEST}" = "${EXPECTED_IMAGE_DIGEST}"
- name: Deploy director blue/green
run: |
regional_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${regional_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
--max-instances 2 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--asia-proof-service-account "${ASIA_PROOF_SERVICE_ACCOUNT}" \
--regional-placement-secret-version "${regional_version}" \
--min-instances 0 \
--release-id "${RELEASE_ID}"
- name: Smoke director health
run: |
URL="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(status.url)')"
node dev/scripts/smoke-relay.mjs "${URL}"
@@ -0,0 +1,77 @@
name: Monitor Relay Cell Clock Skew
# Why: the 2026-08-01 sustained HOST_OFFLINE incident traced to one cell's
# clock running ~100ms ahead, which deterministically failed every host
# challenge under a zero-tolerance freshness check. /health and /ready cannot
# see clock skew; this monitor alarms before drift reaches the (now 2s)
# client tolerance.
on:
workflow_dispatch:
schedule:
- cron: '17 * * * *'
permissions:
contents: read
defaults:
run:
working-directory: cloud
jobs:
skew:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Measure Date-header skew for every relay cell
working-directory: .
run: |
set -u
# Date headers carry whole seconds; compare floored seconds on both
# sides so perfect sync reads 0/±1 and never flaps. An absolute
# floor-skew of >= 2 means real drift of at least ~1s — approaching
# the client's 2s challenge tolerance. Millisecond-precision deltas
# come from the desktop's named-check logging when activations fail.
ALARM_S=2
failures=0
reachable=0
unserved=0
# Keep this upper bound at or above the highest provisioned cell in
# environments/production.tfvars; unlisted cells are silently unmonitored.
for n in $(seq 1 22); do
cell="c${n}"
url="https://${cell}.relay.onorca.dev/health"
# Why: *.relay.onorca.dev is a wildcard, so the load balancer answers with its own
# accurate Date for a fenced, dead, or never-provisioned cell. Timing that reads as
# perfect sync. Only an HTTP 200 is the relay process itself answering, so only a
# 200 carries a clock worth judging.
response="$(curl -sS -D - -o /dev/null --max-time 8 "${url}" 2>/dev/null || true)"
status="$(printf '%s' "${response}" | awk 'NR==1 {print $2}')"
header="$(printf '%s' "${response}" | tr -d '\r' | grep -i '^date:' || true)"
if [ "${status:-000}" != "200" ] || [ -z "${header}" ]; then
# Expected for the fenced cells; a dead unfenced cell is caught by the heartbeat
# and readiness alerts, not here. Named either way so it is never invisible.
echo "${cell}: not serving (status ${status:-none}); no relay clock to judge"
unserved=$((unserved + 1))
continue
fi
reachable=$((reachable + 1))
server_s="$(date -d "${header#*: }" +%s)"
local_s="$(date +%s)"
skew=$((server_s - local_s))
abs=${skew#-}
if [ "${abs}" -ge "${ALARM_S}" ]; then
echo "::error::${cell}: clock skew ${skew}s reaches ±${ALARM_S}s alarm"
failures=$((failures + 1))
else
echo "${cell}: skew ${skew}s"
fi
done
echo "cells serving: ${reachable}, not serving: ${unserved}, alarms: ${failures}"
if [ "${reachable}" -eq 0 ]; then
# Now meaningful: previously the load balancer answered for every name, so this
# could never fire and a total fleet outage reported "all healthy".
echo "::error::no relay cell is serving; monitor blind"
exit 1
fi
exit "$((failures > 0 ? 1 : 0))"
@@ -0,0 +1,213 @@
name: Monitor Relay Production Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
expected-selector-generation:
required: true
type: string
expected-existing-only-cells:
required: true
type: string
expected-migration-only-cells:
required: true
type: string
expected-general-cells:
required: true
type: string
migration-policy:
required: true
type: string
recovery-source-cell-id:
required: true
type: string
capacity-cell-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 100
environment: production
env:
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
MIGRATION_POLICY: ${{ inputs.migration-policy }}
RECOVERY_SOURCE_CELL_ID: ${{ inputs.recovery-source-cell-id }}
CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }}
INCIDENT_ID: relay-${{ github.run_id }}-${{ inputs.mode }}
MONITOR_MODE: ${{ inputs.mode }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-incident
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- id: prior-attempt
if: ${{ github.run_attempt > 1 }}
run: echo "value=$((GITHUB_RUN_ATTEMPT - 1))" >> "${GITHUB_OUTPUT}"
- name: Restore prior private monitor state
if: ${{ github.run_attempt > 1 }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ steps.prior-attempt.outputs.value }}
path: ${{ github.workspace }}/relay-incident
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Verify restored state provenance
if: ${{ github.run_attempt > 1 }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${{ steps.prior-attempt.outputs.value }}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
echo "RESTART_FLAG=--restart" >> "${GITHUB_ENV}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- name: Verify exact-audience admin identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
- name: Run read-only relay dry-run
if: ${{ inputs.mode == 'dry-run' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 15 \
--pre-drain-dry-run \
${RESTART_FLAG:-}
- name: Run first relay monitor segment
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--max-samples-this-run 45 \
${RESTART_FLAG:-}
- id: google-auth-refresh
if: ${{ inputs.mode == 'monitor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Run remaining relay monitor window
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-refresh.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--restart
- name: Seal private evidence provenance
if: ${{ always() }}
run: |
node dev/scripts/relay-monitor-evidence.mjs create \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
- name: Publish aggregate job summary
if: ${{ always() }}
run: |
if [[ -f "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" ]]; then
cat "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" >> "${GITHUB_STEP_SUMMARY}"
else
echo "Relay monitor failed before its first aggregate checkpoint." \
>> "${GITHUB_STEP_SUMMARY}"
fi
- name: Upload private aggregate evidence
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ github.workspace }}/relay-incident
if-no-files-found: error
retention-days: 14
@@ -0,0 +1,75 @@
name: Monitor Relay Production
on:
workflow_dispatch:
inputs:
mode:
description: Run the required 15-minute pre-drain gate or a 90-minute incident watch
required: true
default: dry-run
type: choice
options:
- dry-run
- monitor
expected-selector-generation:
description: Exact durable admission-selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact comma-separated existing-only cells, or none
required: true
type: string
expected-migration-only-cells:
description: Exact comma-separated migration-only cells, or none
required: true
type: string
expected-general-cells:
description: Exact comma-separated general cells, or none
required: true
type: string
migration-policy:
description: Migration checks matched to the intended mutation
required: true
default: strict
type: choice
options:
- strict
- recover-forward
- capacity-transition
recovery-source-cell-id:
description: Existing-only recovery source cell, or none for strict monitoring
required: true
default: none
type: string
capacity-cell-id:
description: General cell for a capacity-transition monitor, or none
required: true
default: none
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-monitor-relay-production-job.yml
with:
mode: ${{ inputs.mode }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
migration-policy: ${{ inputs.migration-policy }}
recovery-source-cell-id: ${{ inputs.recovery-source-cell-id }}
capacity-cell-id: ${{ inputs.capacity-cell-id }}
@@ -0,0 +1,514 @@
name: Operate Relay Asia Admission
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Inspect, initialize, verify, atomically register, promote, or roll back admission
required: true
default: verify
type: choice
options: [inspect, initialize, verify, register, configure, promote, rollback]
cell-ids:
description: Exact reviewed comma-separated Asia cell wave
required: true
type: string
selector-generation:
description: Exact live selector generation; leave empty only for inspect
required: false
type: string
selector-membership-sha256:
description: Exact fingerprint printed by inspect; required only for initialize
required: false
type: string
selector-attempt-id:
description: Durable unique attempt ID; empty for inspect, verify, and configure
required: false
type: string
image-digest:
description: Expected compatible Relay sha256 digest
required: true
type: string
director-image-digest:
description: Director sha256 digest; required only for configure
required: false
type: string
evidence-run-id:
description: Successful staging or C27 evidence workflow run ID; required for production promotion
required: false
type: string
evidence-run-attempt:
description: Exact evidence workflow run attempt; required for production promotion
required: false
type: string
confirmation:
description: Exact typed confirmation for a mutation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
admission:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DIRECTOR_ORIGIN: ${{ inputs.environment == 'production' && 'https://relay.onorca.dev' || 'https://relay-staging.onorca.dev' }}
AUTH_ORIGIN: ${{ inputs.environment == 'production' && 'https://login.onorca.dev' || 'https://auth-staging.onorca.dev' }}
DIRECTOR_SERVICE: ${{ inputs.environment == 'production' && 'orca-cloud-relay' || 'orca-cloud-relay-staging' }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
GCP_REGION: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_REGION || vars.STAGING_GCP_REGION }}
DIRECTOR_MAX_INSTANCES: ${{ inputs.environment == 'production' && '5' || '2' }}
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
OPERATION_MODE: ${{ inputs.mode }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
EXPECTED_SELECTOR_MEMBERSHIP_SHA256: ${{ inputs.selector-membership-sha256 }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
IMAGE_DIGEST: ${{ inputs.image-digest }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
EVIDENCE_RUN_ID: ${{ inputs.evidence-run-id }}
EVIDENCE_RUN_ATTEMPT: ${{ inputs.evidence-run-attempt }}
OPERATION_CONFIRMATION: ${{ inputs.confirmation }}
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate exact operation inputs before authentication
id: inputs
shell: bash
run: |
set -euo pipefail
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
evidence_kind=none
if test "${OPERATION_MODE}" = inspect; then
test -z "${EXPECTED_SELECTOR_GENERATION}"
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
else
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
fi
if test "${OPERATION_MODE}" = initialize; then
test "${EXPECTED_SELECTOR_GENERATION}" = 0
[[ "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" =~ ^[a-f0-9]{64}$ ]]
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${OPERATION_CONFIRMATION}" = INITIALIZE_ADMISSION_SELECTOR
elif test "${OPERATION_MODE}" = verify; then
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
elif test "${OPERATION_MODE}" = inspect; then
:
elif test "${OPERATION_MODE}" = configure; then
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${SELECTOR_ATTEMPT_ID}"
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
test "${OPERATION_CONFIRMATION}" = CONFIGURE_ASIA_DIRECTOR
else
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
case "${OPERATION_MODE}:${OPERATION_CONFIRMATION}" in
register:REGISTER_ASIA_MIGRATION_ONLY) ;;
promote:PROMOTE_ASIA_GENERAL) ;;
rollback:ROLLBACK_ASIA_MIGRATION_ONLY) ;;
*) echo "typed confirmation does not match the requested mutation" >&2; exit 1 ;;
esac
fi
if test "${TARGET_ENVIRONMENT}:${OPERATION_MODE}" = production:promote; then
[[ "${EVIDENCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${EVIDENCE_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
case "${TARGET_CELL_IDS}" in
production-gce-c27)
test "${#SELECTOR_ATTEMPT_ID}" -le 119
evidence_kind=staging
artifact_name="relay-asia-staging-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
;;
production-gce-c28,production-gce-c29)
evidence_kind=c27
artifact_name="relay-asia-c27-canary-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
;;
*) echo "production promotion wave is not reviewed" >&2; exit 1 ;;
esac
else
test -z "${EVIDENCE_RUN_ID}"
test -z "${EVIDENCE_RUN_ATTEMPT}"
artifact_name=none
fi
{
echo "evidence_kind=${evidence_kind}"
echo "artifact_name=${artifact_name}"
} >> "${GITHUB_OUTPUT}"
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
- name: Install exact C27 canary dependencies
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
run: pnpm install --frozen-lockfile
- name: Build the C27 canary Relay contract
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
run: pnpm --filter @orca-cloud/relay-contract build
- name: Download immutable rollout evidence
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
uses: actions/download-artifact@v4
with:
name: ${{ steps.inputs.outputs.artifact_name }}
path: ${{ runner.temp }}/relay-asia-input-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.evidence-run-id }}
- name: Verify evidence provenance and rollout binding before authentication
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
env:
GH_TOKEN: ${{ github.token }}
EVIDENCE_KIND: ${{ steps.inputs.outputs.evidence_kind }}
shell: bash
run: |
set -euo pipefail
run_json="${RUNNER_TEMP}/relay-asia-evidence-run.json"
verified="${RUNNER_TEMP}/relay-asia-evidence-verified"
gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${EVIDENCE_RUN_ID}/attempts/${EVIDENCE_RUN_ATTEMPT}" > "${run_json}"
evidence_commit_sha="$(
jq -er '.head_sha | select(type == "string" and test("^[a-f0-9]{40}$"))' "${run_json}"
)"
command=(node dev/scripts/relay-asia-rollout-evidence.mjs "verify-${EVIDENCE_KIND}"
--evidence "${RUNNER_TEMP}/relay-asia-input-evidence/evidence.json"
--run-json "${run_json}"
--commit-sha "${evidence_commit_sha}"
--image-digest "${IMAGE_DIGEST}"
--now "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
--output "${verified}")
if test "${EVIDENCE_KIND}" = c27; then
command+=(--selector-generation "${EXPECTED_SELECTOR_GENERATION}")
fi
"${command[@]}"
test "$(< "${verified}")" = verified
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ env.DIRECTOR_ORIGIN }}/v1/admin/drain
id_token_include_email: true
- name: Require the exact director image before promotion
if: ${{ inputs.mode == 'promote' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
if: ${{ inputs.mode == 'configure' || (inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27') }}
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode == 'configure' }}
with:
terraform_version: 1.15.8
terraform_wrapper: false
- name: Run the exact generation-bound admission operation
id: admission-operation
if: ${{ inputs.mode != 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode "${OPERATION_MODE}" \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
states="$(jq -cS '.states // {}' <<< "${result}")"
membership="$(jq -cS '.membership // empty' <<< "${result}")"
membership_sha256="$(jq -r '.membershipSha256 // empty' <<< "${result}")"
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
result_dir="${RUNNER_TEMP}/relay-asia-admission-result"
mkdir -p "${result_dir}"
node dev/scripts/sanitize-relay-asia-admission-result.mjs \
<<< "${result}" > "${result_dir}/result.json"
{
echo "### Relay Asia admission"
echo "- Mode: ${OPERATION_MODE}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Result generation: ${generation}"
echo "- States: \`${states}\`"
if test -n "${membership}"; then echo "- Membership: \`${membership}\`"; fi
if test -n "${membership_sha256}"; then
echo "- Membership SHA-256: \`${membership_sha256}\`"
fi
} >> "${GITHUB_STEP_SUMMARY}"
- name: Verify C27 state and start the timed canary
id: c27-start
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general
test "$(jq -r '.states["production-gce-c28"]' <<< "${result}")" = migration-only
test "$(jq -r '.states["production-gce-c29"]' <<< "${result}")" = migration-only
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run a real five-minute C27 control and splice canary
id: c27-load
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
shell: bash
run: |
set -euo pipefail
log="${RUNNER_TEMP}/relay-asia-c27-load.jsonl"
report="${RUNNER_TEMP}/relay-asia-c27-load.json"
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region asia-east2 \
--relay-asia-load-principals 1 \
--controls 1 \
--splices 1 \
--capacity-hard-cap 3000 \
--ramp-seconds 0 \
--duration-seconds 300 \
--splice-hold-seconds 60 \
--required-lease-horizons 2 > "${log}"
jq -cer 'select(.event == "relay_load_complete")' "${log}" | tail -n 1 > "${report}"
echo "ended_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Collect regional, Relay SQL, and Cloud SQL canary evidence
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
CANARY_STARTED_AT: ${{ steps.c27-start.outputs.started_at }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general
ended_at="${{ steps.c27-load.outputs.ended_at }}"
sleep 60
output="${RUNNER_TEMP}/relay-asia-output-evidence"
logs="${RUNNER_TEMP}/relay-asia-c27-runtime-metrics.json"
mkdir -p "${output}"
gcloud logging read \
"timestamp>=\"${CANARY_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" \
--limit 20000 \
--format json > "${logs}"
node dev/scripts/relay-asia-rollout-evidence.mjs create-c27 \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.admission-operation.outputs.generation }}" \
--started-at "${CANARY_STARTED_AT}" \
--ended-at "${ended_at}" \
--load-report "${RUNNER_TEMP}/relay-asia-c27-load.json" \
--logs-json "${logs}" \
--output "${output}/evidence.json"
jq -r '.metrics | to_entries[] | "- \(.key): \(.value)"' \
"${output}/evidence.json" >> "${GITHUB_STEP_SUMMARY}"
- name: Upload immutable C27 canary evidence
id: c27-evidence-upload
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-c27-canary-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-output-evidence/evidence.json
if-no-files-found: error
retention-days: 7
- name: Upload sanitized admission result
if: ${{ inputs.mode != 'configure' && steps.admission-operation.outcome == 'success' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-admission-result-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-admission-result/result.json
if-no-files-found: error
retention-days: 7
- name: Return an unproven C27 canary to migration-only
if: ${{ always() && inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' && steps.admission-operation.outcome != 'skipped' && steps.c27-evidence-upload.outcome != 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode recover-promotion \
--cell-ids production-gce-c27 \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode rollback \
--cell-ids production-gce-c27 \
--expected-generation "${promoted_generation}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}-rollback" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = migration-only
- name: Require registered migration-only cells before director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode registered \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
- name: Build the additive director cell configuration
if: ${{ inputs.mode == 'configure' }}
id: director-config
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
topology="${RUNNER_TEMP}/relay-asia-state-topology.json"
current="${RUNNER_TEMP}/relay-current-director-cells.json"
desired="${RUNNER_TEMP}/relay-asia-director-cells.json"
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${topology}"
service="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
revision="$(jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end' \
<<< "${service}")"
gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '.spec.containers[0].env[] | select(.name == "ORCA_RELAY_CELLS_JSON") | .value | fromjson' \
> "${current}"
node dev/scripts/prepare-relay-asia-director-cells.mjs \
--current-json "${current}" \
--topology-json "${topology}" \
--output "${desired}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image-digest "${IMAGE_DIGEST}"
echo "file=${desired}" >> "${GITHUB_OUTPUT}"
- name: Deploy the registered additive director topology
if: ${{ inputs.mode == 'configure' }}
env:
DIRECTOR_CELLS_FILE: ${{ steps.director-config.outputs.file }}
run: |
current="$(gcloud secrets versions access latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
image="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}"
release_id="asia-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE}" \
--image "${image}" \
--role director \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--release-id "${release_id}" \
--director-cells-json "$(< "${DIRECTOR_CELLS_FILE}")" \
--prune-revisions false
- name: Verify selector and heartbeats after director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode verify \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
revision="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end')"
revision_json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
jq -e --arg image "us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}" \
'.spec.containers[0].image == $image' <<< "${revision_json}" > /dev/null
jq -er --arg secret "${REGIONAL_PLACEMENT_SECRET}" \
'[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)} |
select(.secret == $secret and (.version | test("^[1-9][0-9]*$")))] |
if length == 1 then .[0].version else error("regional switch version missing") end' \
<<< "${revision_json}" > "${RUNNER_TEMP}/relay-regional-placement-version"
regional_version="$(< "${RUNNER_TEMP}/relay-regional-placement-version")"
[[ "$(gcloud secrets versions access "${regional_version}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" =~ ^(true|false)$ ]]
echo "Director configuration now lists the registered migration-only Asia cells." >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,328 @@
name: Operate Relay Production Rehome Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
director-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-control-generation: { required: true, type: string }
not-before: { required: true, type: string }
rate-per-minute: { required: true, type: string }
preference-max-age-ms: { required: true, type: string }
drain-grace-ms: { required: true, type: string }
confirmation: { required: true, type: string }
monitor-run-id: { required: true, type: string }
monitor-run-attempt: { required: true, type: string }
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
control:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
MODE: ${{ inputs.mode }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_CONTROL_GENERATION: ${{ inputs.expected-control-generation }}
NOT_BEFORE: ${{ inputs.not-before }}
RATE_PER_MINUTE: ${{ inputs.rate-per-minute }}
PREFERENCE_MAX_AGE_MS: ${{ inputs.preference-max-age-ms }}
DRAIN_GRACE_MS: ${{ inputs.drain-grace-ms }}
CONFIRMATION: ${{ inputs.confirmation }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${MODE}" =~ ^(inspect|enable|pause|disable)$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_CONTROL_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
if [[ "${MODE}" =~ ^(inspect|enable)$ ]]; then
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test -n "${GCP_REGION}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
fi
case "${MODE}" in
inspect)
test -z "${CONFIRMATION}"
;;
enable)
test "${CONFIRMATION}" = ENABLE_REGIONAL_REHOMING
test "${RATE_PER_MINUTE}" = 10
[[ "${NOT_BEFORE}" =~ ^[1-9][0-9]*$ ]]
;;
pause)
test "${CONFIRMATION}" = PAUSE_REGIONAL_REHOMING
;;
disable)
test "${CONFIRMATION}" = DISABLE_REGIONAL_REHOMING
;;
esac
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Apply emergency durable pause or disable before diagnostics
if: ${{ inputs.mode == 'pause' || inputs.mode == 'disable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- uses: pnpm/action-setup@v4
if: ${{ inputs.mode == 'enable' }}
with: { package_json_file: cloud/package.json }
- run: pnpm install --frozen-lockfile
if: ${{ inputs.mode == 'enable' }}
- name: Download fresh aggregate safety evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify enable evidence provenance
if: ${{ inputs.mode == 'enable' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" --run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" --mode dry-run \
--required-migration-policy strict
- name: Reject previously consumed enable safety evidence
if: ${{ inputs.mode == 'enable' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = 0
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Verify exact serving and rollback director identities
if: ${{ inputs.mode == 'inspect' || inputs.mode == 'enable' }}
env:
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
SERVING_REVISION="$(jq -er \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName
else error("director does not have one serving revision") end' \
<<< "${SERVICE_JSON}")"
ROLLBACK_REVISION="$(jq -er \
'[.status.traffic[] | select(.tag == "selector-rollback")] |
if length == 1 then .[0].revisionName else error("rollback tag missing") end' \
<<< "${SERVICE_JSON}")"
verify_revision() {
local revision="$1" expected_digest="$2"
local json
json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
test "$(jq -r '.spec.serviceAccountName' <<< "${json}")" = \
"${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '.spec.containers[0].image | split("@") | last' <<< "${json}")" = \
"${expected_digest}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT") | .value] | if length == 1
then .[0] else empty end' <<< "${json}")" = "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_AUDIENCE") | .value] | if length == 1 then .[0]
else empty end' <<< "${json}")" = "${REHOME_AUDIENCE}"
}
verify_revision "${SERVING_REVISION}" "${DIRECTOR_IMAGE_DIGEST}"
verify_revision "${ROLLBACK_REVISION}" "${ROLLBACK_IMAGE_DIGEST}"
- name: Seal 24-hour aggregate region observation evidence
if: ${{ inputs.mode == 'enable' }}
run: |
mkdir -p "${RUNNER_TEMP}/relay-region-observation"
# 6 director instances x 120 samples/hour x 25h = 18000; a clipped
# read empties the oldest hourly buckets and fails the seal.
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND jsonPayload.event="orca_relay_runtime_metrics" AND jsonPayload.role="director"' \
--project "${GCP_PROJECT_ID}" --freshness=25h --limit=30000 --format=json \
| node dev/scripts/relay-region-observation-evidence.mjs create \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}" \
> "${RUNNER_TEMP}/relay-region-observation/evidence.json"
- name: Upload sealed 24-hour aggregate region evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-region-observation-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-region-observation/evidence.json
retention-days: 90
if-no-files-found: error
- name: Verify sealed 24-hour enable authority
if: ${{ inputs.mode == 'enable' }}
run: |
node dev/scripts/relay-region-observation-evidence.mjs verify \
--file "${RUNNER_TEMP}/relay-region-observation/evidence.json" \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}"
- name: Recheck every aggregate safety signal before enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Seal single-use enable safety authority
if: ${{ inputs.mode == 'enable' }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-rehome-enable-authority"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-rehome-enable-authority/${MARKER_NAME}"
- name: Consume enable safety evidence before durable mutation
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-rehome-enable-authority/relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Inspect regional rehome control
if: ${{ inputs.mode == 'inspect' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Apply exact durable regional rehome enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Read fresh aggregate completion and abort evidence
run: |
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND textPayload:"[orca-relay] regional rehome inventory"' \
--project "${GCP_PROJECT_ID}" --freshness=15m --limit=20 --format=json \
| node dev/scripts/relay-rehome-aggregate-evidence.mjs --max-age-ms 900000 \
| tee "${RUNNER_TEMP}/relay-rehome-inventory.json"
- name: Publish aggregate control evidence
run: |
{
echo '### Regional rehome control'
jq -r '"- mode: `\(.mode)`\n- generation: `\(.control.generation)`\n- enabled: `\(.control.enabled)`"' \
"${RUNNER_TEMP}/relay-rehome-control.json"
jq -r '"- active: `\(.active)`\n- awaiting receipt: `\(.awaitingReceipt)`\n- target registered: `\(.targetRegistered)`\n- completed (24h): `\(.completedLast24Hours)`\n- aborted (24h): `\(.abortedLast24Hours)`"' \
"${RUNNER_TEMP}/relay-rehome-inventory.json"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Fail closed after an unsuccessful enable run
if: ${{ failure() && inputs.mode == 'enable' && steps.google-auth.outcome == 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode recover-enable --director-origin "${DIRECTOR_ORIGIN}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--confirmation RECOVER_FAILED_REGIONAL_REHOME_ENABLE \
| tee "${RUNNER_TEMP}/relay-rehome-enable-recovery.json"
jq -e \
'.mode == "recover-enable" and .control.enabled == false' \
"${RUNNER_TEMP}/relay-rehome-enable-recovery.json" >/dev/null
@@ -0,0 +1,106 @@
name: Operate Relay Production Rehome
on:
workflow_dispatch:
inputs:
mode:
description: Inspect or apply the durable regional-rehome switch
required: true
default: inspect
type: choice
options: [inspect, enable, pause, disable]
director-image-digest:
description: Exact immutable serving director digest
required: true
type: string
rollback-image-digest:
description: Exact immutable selector-rollback director digest
required: true
type: string
expected-selector-generation:
description: Exact admission selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-control-generation:
description: Exact durable rehome generation
required: true
type: string
not-before:
description: Exact epoch milliseconds; ignored only by inspect
required: true
default: '0'
type: string
rate-per-minute:
description: Exact global host rate; initial enable is fixed at 10
required: true
default: '10'
type: string
preference-max-age-ms:
description: Maximum fresh preference age
required: true
default: '86400000'
type: string
drain-grace-ms:
description: Per-host source drain grace
required: true
default: '3600000'
type: string
monitor-run-id:
description: Fresh successful aggregate dry-run required only by enable
required: false
type: string
monitor-run-attempt:
description: Exact monitor attempt required only by enable
required: false
type: string
confirmation:
description: ENABLE_REGIONAL_REHOMING, PAUSE_REGIONAL_REHOMING, or DISABLE_REGIONAL_REHOMING
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
operate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-operate-relay-production-rehome-job.yml
with:
mode: ${{ inputs.mode }}
director-image-digest: ${{ inputs.director-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-control-generation: ${{ inputs.expected-control-generation }}
not-before: ${{ inputs.not-before }}
rate-per-minute: ${{ inputs.rate-per-minute }}
preference-max-age-ms: ${{ inputs.preference-max-age-ms }}
drain-grace-ms: ${{ inputs.drain-grace-ms }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
secrets: inherit
@@ -0,0 +1,101 @@
name: Power Relay Staging
on:
schedule:
# A zero-activity guard makes this a no-op when an internal test is still running.
- cron: '0 9 * * *'
workflow_dispatch:
inputs:
mode:
description: Inspect, wake, or sleep the staging Relay data plane
required: true
default: status
type: choice
options:
- status
- wake
- sleep
wake-cells:
description: Wake configured admission cells, or include disabled candidate cells
required: true
default: configured
type: choice
options:
- configured
- all
confirmation:
description: Enter WAKE_STAGING or SLEEP_STAGING for a manual mutation
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
power:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
POWER_MODE: ${{ github.event_name == 'schedule' && 'sleep' || inputs.mode }}
WAKE_CELLS: ${{ inputs.wake-cells || 'configured' }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit manual mutation confirmation
if: ${{ github.event_name == 'workflow_dispatch' && inputs.mode != 'status' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${POWER_MODE}" = "wake" ]]; then
test "${CONFIRMATION}" = "WAKE_STAGING"
else
test "${CONFIRMATION}" = "SLEEP_STAGING"
fi
- name: Read reviewed staging topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
- name: Inspect or change staging power state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/power-staging-relay.mjs \
--mode "${POWER_MODE}" \
--wake-cells "${WAKE_CELLS}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json"
@@ -0,0 +1,331 @@
name: Prove Relay Asia Staging
on:
workflow_dispatch:
inputs:
image-digest:
description: Exact immutable Relay digest deployed on staging C4
required: true
type: string
selector-generation:
description: Exact selector generation with C4 migration-only
required: true
type: string
promote-attempt-id:
description: Durable unique C4 promotion attempt ID
required: true
type: string
rollback-attempt-id:
description: Durable unique C4 rollback attempt ID
required: true
type: string
confirmation:
description: Enter PROVE_ASIA_STAGING
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
prove:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: [self-hosted, linux, x64, relay-asia-east2-load]
timeout-minutes: 75
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
AUTH_ORIGIN: https://auth-staging.onorca.dev
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact staging proof request
shell: bash
run: |
set -euo pipefail
test "${{ inputs.confirmation }}" = PROVE_ASIA_STAGING
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "${INITIAL_SELECTOR_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${PROMOTE_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
[[ "${ROLLBACK_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${PROMOTE_ATTEMPT_ID}" != "${ROLLBACK_ATTEMPT_ID}"
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- name: Require the exact staging director image before promotion
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- name: Install exact load-harness dependencies
run: pnpm install --frozen-lockfile
- name: Build the Relay load-harness contract
run: pnpm --filter @orca-cloud/relay-contract build
- name: Promote only staging C4
id: promote
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode promote \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = general
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run sharded two-horizon and mixed splice proofs
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
mkdir -p "${proof_dir}"
fd_limit="$(ulimit -n)"
if test "${fd_limit}" != unlimited; then
[[ "${fd_limit}" =~ ^[0-9]+$ ]]
test "${fd_limit}" -ge 4096
fi
run_phase() {
phase="$1"
controls="$2"
splices="$3"
pids=()
stop_shards() {
for pid in "${pids[@]}"; do kill "${pid}" 2>/dev/null || true; done
for pid in "${pids[@]}"; do wait "${pid}" 2>/dev/null || true; done
}
trap stop_shards EXIT
for shard in 0 1 2 3; do
slow=0
wedged=0
boundary_args=()
request_unit_args=()
if test "${phase}" = launch && test "${shard}" = 0; then
slow=4
wedged=1
fi
if test "${phase}" = launch && test "${shard}" = 0; then
boundary_args=(
--region-behavior-probes 1
--capacity-cell-id staging-gce-c4
--capacity-cell-origin https://c4.relay-staging.onorca.dev
--capacity-unobserved-bound 60
--rebind-probes 2
--skip-rebind-overflow-check
)
fi
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region asia-east2 \
--relay-asia-load-principals 32 \
--controls "${controls}" \
--splices "${splices}" \
--slow-reader-splices "${slow}" \
--wedged-reader-splices "${wedged}" \
--capacity-hard-cap 3000 \
"${boundary_args[@]}" \
"${request_unit_args[@]}" \
--phase-barrier-dir "${proof_dir}/${phase}-barrier" \
--aggregate-controls "$((controls * 4))" \
--aggregate-splices "$((splices * 4))" \
--aggregate-reader-splices "$([[ "${phase}" = launch ]] && echo 5 || echo 0)" \
--aggregate-reader-bytes "$([[ "${phase}" = launch ]] && echo 12582912 || echo 0)" \
--required-lease-horizons 2 \
--splice-ramp-seconds 120 \
--max-generator-rss-growth-mib 512 \
--ramp-seconds 180 \
--duration-seconds 210 \
--shard-count 4 \
--shard-index "${shard}" \
> "${proof_dir}/${phase}-${shard}.jsonl" &
pids+=("$!")
done
failed=0
for pid in "${pids[@]}"; do
if ! wait "${pid}"; then failed=1; break; fi
done
if test "${failed}" = 1; then
stop_shards
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_progress" or .event == "relay_load_complete") |
{event, shardIndex, active, peakActive, connected, connectionFailures,
rampConnectionFailures, connectionFailuresByReason, unexpectedCloses,
protocolErrors, refreshErrors, socketErrors, elapsedSeconds}' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1 || true
done
trap - EXIT
return 1
fi
trap - EXIT
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_complete")' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1
done | jq -s . > "${proof_dir}/${phase}.json"
}
run_phase launch 5 5
- name: Collect and validate aggregate staging proof evidence
env:
PROOF_STARTED_AT: ${{ steps.promote.outputs.started_at }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
ended_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
sleep 60
gcloud logging read \
"timestamp>=\"${PROOF_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" --limit 20000 --format json \
> "${proof_dir}/runtime-metrics.json"
node dev/scripts/relay-asia-rollout-evidence.mjs create-staging \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.promote.outputs.generation }}" \
--started-at "${PROOF_STARTED_AT}" \
--ended-at "${ended_at}" \
--launch-report "${proof_dir}/launch.json" \
--logs-json "${proof_dir}/runtime-metrics.json" \
--output "${proof_dir}/evidence.json"
- name: Return staging C4 to migration-only
if: ${{ always() && steps.promote.outcome != 'skipped' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
- name: Upload immutable staging readiness evidence
if: ${{ success() }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-staging-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-staging-proof/evidence.json
if-no-files-found: error
retention-days: 7
recover:
if: ${{ always() && github.ref == 'refs/heads/main' }}
needs: prove
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 15
environment: staging
env:
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Recover staging C4 with a fresh identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
@@ -0,0 +1,917 @@
name: Prove Relay Staging Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify or change C3 capacity, restore admission, or refresh empty Asia C4
required: true
default: verify
type: choice
options:
- verify
- apply
- restore-admission
- refresh-asia-c4-image
expected-hard-cap:
description: Exact cap declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '600'
type: choice
options:
- '1000'
- '600'
expected-unobserved-bound:
description: Exact bound declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '60'
type: choice
options:
- '60'
- '0'
confirmation:
description: Exact confirmation required for a mutation
required: false
type: string
expected-selector-generation:
description: Exact staging selector generation for a C4 image refresh
required: false
type: string
predecessor-image-digest:
description: Exact current C4 sha256 digest
required: false
type: string
target-image-digest:
description: Exact desired C4 sha256 digest
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && inputs.mode != 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c3.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c3
FALLBACK_CELL_ORIGIN: https://c2.relay-staging.onorca.dev
FALLBACK_CELL_ID: staging-gce-c2
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
EXPECTED_HARD_CAP: ${{ inputs.expected-hard-cap }}
EXPECTED_UNOBSERVED_BOUND: ${{ inputs.expected-unobserved-bound }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode != 'restore-admission' }}
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
if: ${{ inputs.mode != 'restore-admission' }}
run: node dev/scripts/infra.mjs init --env staging
- name: Require reviewed desired capacity and image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
CAP_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_hard_cap"
BOUND_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_unobserved_bound"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_CAP="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${CAP_EXPRESSION}")"
DESIRED_BOUND="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${BOUND_EXPRESSION}")"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
CELL_ORIGIN="$(jq -r --arg cell "${TARGET_CELL_ID}" \
'.[] | select(.id == $cell) | .url' <<< "${DESIRED_CELLS_JSON}")"
test "${DESIRED_CAP}" = "${EXPECTED_HARD_CAP}"
test "${DESIRED_BOUND}" = "${EXPECTED_UNOBSERVED_BOUND}"
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
[[ "${MIG_NAME}" =~ ^[a-z0-9-]+$ ]]
test "${CELL_ORIGIN}" = "https://c3.relay-staging.onorca.dev"
jq -e \
--arg cell "${TARGET_CELL_ID}" \
--arg fallback "${FALLBACK_CELL_ID}" \
--argjson cap "${EXPECTED_HARD_CAP}" \
--argjson bound "${EXPECTED_UNOBSERVED_BOUND}" \
'(any(.[]; .id == $cell and .connectionHardCap == $cap and .connectionUnobservedBound == $bound)) and
(any(.[]; .id == $fallback and .connectionHardCap == 600 and .connectionUnobservedBound == 60))' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
echo "DESIRED_IMAGE=${DESIRED_IMAGE}" >> "${GITHUB_ENV}"
echo "TARGET_ZONE=${TARGET_ZONE}" >> "${GITHUB_ENV}"
echo "MIG_NAME=${MIG_NAME}" >> "${GITHUB_ENV}"
echo "CELL_ORIGIN=${CELL_ORIGIN}" >> "${GITHUB_ENV}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" >> "${GITHUB_ENV}"
- name: Verify exact compatible director image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_IMAGE="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${ACTIVE_IMAGE}" = "${DESIRED_IMAGE}"
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}" >> "${GITHUB_ENV}"
- name: Require explicit transition confirmation
if: ${{ inputs.mode == 'apply' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "FENCE_AND_TRANSITION_STAGING_C3"
- name: Require explicit admission restore confirmation
if: ${{ inputs.mode == 'restore-admission' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "RESTORE_STAGING_C2_C3_GENERAL"
- name: Require capacity identity and exact predecessor state
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
case "${EXPECTED_HARD_CAP}/${EXPECTED_UNOBSERVED_BOUND}" in
1000/0)
PREDECESSOR_C3_CAP=600
PREDECESSOR_C3_BOUND=60
;;
1000/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=0
;;
600/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=60
;;
*)
echo "Unsupported staging capacity transition" >&2
exit 1
;;
esac
ACTIVE_REVISION="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
PREDECESSOR_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${PREDECESSOR_C3_CAP}" \
--argjson bound "${PREDECESSOR_C3_BOUND}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${DESIRED_CELLS_JSON}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
if jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${DESIRED_CELLS_JSON}" \
'$current == $expected'; then
if node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-active
elif node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-ready
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe
TRANSITION_PHASE=director-ready
fi
else
jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${PREDECESSOR_CELLS_JSON}" \
'$current == $expected'
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${PREDECESSOR_C3_CAP}" \
--unobserved-bound "${PREDECESSOR_C3_BOUND}" \
--heartbeat fresh \
--admission either \
--draining either \
--activity allowed
TRANSITION_PHASE=predecessor
fi
echo "TRANSITION_PHASE=${TRANSITION_PHASE}" >> "${GITHUB_ENV}"
- name: Restore C2 as the safe placement fallback
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
- name: Require a healthy non-draining C3 before restoring it
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
- name: Require a healthy general C2 before isolating C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-active; then
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Reversibly isolate and drain C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Verify restart-safe migration-only target
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe
- name: Verify current capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Deploy reviewed director topology and remove pre-protocol revisions
if: ${{ inputs.mode == 'apply' }}
run: |
if test "${TRANSITION_PHASE}" != predecessor; then
echo "DIRECTOR_CONFIG_CHANGED=false" >> "${GITHUB_ENV}"
exit 0
fi
RELEASE_ID="capacity-compat-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
DEPLOY_RESULT="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--min-instances 0 \
--prune-revisions true \
--release-id "${RELEASE_ID}")"
echo "${DEPLOY_RESULT}"
DIRECTOR_CONFIG_CHANGED="$(jq -r '.topologyChanged' <<< "${DEPLOY_RESULT}")"
[[ "${DIRECTOR_CONFIG_CHANGED}" =~ ^(true|false)$ ]]
echo "DIRECTOR_CONFIG_CHANGED=${DIRECTOR_CONFIG_CHANGED}" >> "${GITHUB_ENV}"
- name: Require fail-closed director transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
exit 0
fi
HEARTBEAT_EXPECTATION=either
if test "${DIRECTOR_CONFIG_CHANGED}" = true; then
HEARTBEAT_EXPECTATION=stale
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat "${HEARTBEAT_EXPECTATION}" \
--admission migration-only \
--draining required \
--activity restart-safe
- name: Plan and apply only the exact empty cell
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
recreate_fixed_one_instance() {
local instance
instance="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("capacity cell does not have one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" \
--instances "${instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--quiet
}
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c3"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c3"]' \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}")"
echo "${PLAN_RESULT}"
CELL_PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${CELL_PLAN_CHANGES}" =~ ^(0|2)$ ]]
if test "${TRANSITION_PHASE}" = cell-ready; then
test "${CELL_PLAN_CHANGES}" = 0
elif test "${TRANSITION_PHASE}" = cell-active; then
test "${CELL_PLAN_CHANGES}" = 0
recreate_fixed_one_instance
elif test "${CELL_PLAN_CHANGES}" = 0; then
recreate_fixed_one_instance
else
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--timeout 900
- name: Verify exact live cap and fresh matching heartbeat
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
- name: Make C3 the only staging placement cell
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the sole general canary after transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Restore the reviewed C2 and C3 placement set
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
- name: Verify restored C3 admission and capacity
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Preserve C2 as the safe fallback after a failed transition
if: ${{ failure() && inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
refresh-asia-c4-image:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && github.ref == 'refs/heads/main' && inputs.mode == 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
APPROVED_PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
steps:
- uses: actions/checkout@v4
- name: Require the exact bounded C4 refresh
env:
CONFIRMATION: ${{ inputs.confirmation }}
shell: bash
run: |
set -euo pipefail
test "${CONFIRMATION}" = REFRESH_STAGING_ASIA_C4_IMAGE
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PREDECESSOR_IMAGE_DIGEST}" != "${TARGET_IMAGE_DIGEST}"
test "${PREDECESSOR_IMAGE_DIGEST}" = "${APPROVED_PREDECESSOR_IMAGE_DIGEST}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- name: Resolve the reviewed C4 image and shape
shell: bash
run: |
set -euo pipefail
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
shape="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${cells}")"
test "$(jq -r '.hostname' <<< "${shape}")" = c4
test "$(jq -r '.region' <<< "${shape}")" = asia-east2
test "$(jq -r '.zone' <<< "${shape}")" = asia-east2-a
test "$(jq -r '.machine_type' <<< "${shape}")" = e2-standard-4
test "$(jq -r '.capacity_requests' <<< "${shape}")" = 6000
test "$(jq -r '.database_pool_max' <<< "${shape}")" = 10
test "$(jq -r '.connection_hard_cap' <<< "${shape}")" = 3000
test "$(jq -r '.connection_unobserved_bound' <<< "${shape}")" = 60
test "$(jq -r '.initially_enabled' <<< "${shape}")" = false
desired_image="$(jq -r '.image' <<< "${shape}")"
test "${desired_image}" = \
"us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${desired_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${TARGET_IMAGE_DIGEST}"
mig_name="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
test "${mig_name}" = orca-cloud-staging-relay-gce-c4
{
echo "DESIRED_IMAGE=${desired_image}"
echo "ROLLBACK_IMAGE=us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${PREDECESSOR_IMAGE_DIGEST}"
echo "TARGET_ZONE=asia-east2-a"
echo "MIG_NAME=${mig_name}"
} >> "${GITHUB_ENV}"
- name: Save, validate, and classify the exact C4 plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
case "$(jq -r '[.changes,.changeKind] | join(":")' <<< "${result}")" in
2:replacement) refresh_phase=predecessor ;;
0:none|*:obsolete-template-delete) refresh_phase=applied ;;
*:manager-convergence|*:replacement-with-obsolete-template) refresh_phase=converging ;;
*) exit 1 ;;
esac
{
echo "PLAN_CHANGES=$(jq -r '.changes' <<< "${result}")"
echo "REFRESH_PHASE=${refresh_phase}"
} >> "${GITHUB_ENV}"
- id: state-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the exact selector and current C4 state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.state-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${inspect}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
expected_digests="${TARGET_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = predecessor; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST}"
elif test "${REFRESH_PHASE}" = converging; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
case ",${expected_digests}," in
*,"${current_digest}",*) ;;
*) exit 1 ;;
esac
source_incarnation=''
draining=forbidden
if test "${REFRESH_PHASE}" != applied; then
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
source_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
[[ "${source_incarnation}" =~ ^[0-9a-f-]{36}$ ]]
if test "$(jq -r '.draining' <<< "${runtime}")" = true; then
draining=required
fi
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining "${draining}" \
--activity quiescent --expected-image-digests "${expected_digests}"
runtime_available=true
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 300000
source_incarnation=''
runtime_available=false
fi
{
echo "MIG_STABLE_AT_MS=0"
echo "MUTATION_STARTED=false"
echo "REPLACEMENT_STARTED_AT_MS=0"
echo "RUNTIME_AVAILABLE=${runtime_available}"
echo "SOURCE_INCARNATION=${source_incarnation}"
} >> "${GITHUB_ENV}"
- id: fence-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 and prove it stayed empty before replacement
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
if test "${REFRESH_PHASE}" = applied; then exit 0; fi
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
if test "${RUNTIME_AVAILABLE}" = false; then exit 0; fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate
fence_digests="${PREDECESSOR_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = converging; then
fence_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent --expected-image-digests "${fence_digests}"
- name: Apply the exact saved C4 plan
shell: bash
run: |
set -euo pipefail
if test "${PLAN_CHANGES}" = 0 && test "${RUNTIME_AVAILABLE}" = true; then exit 0; fi
if test "${REFRESH_PHASE}" = applied && test "${RUNTIME_AVAILABLE}" = false; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
fi
if test "${REFRESH_PHASE}" != applied; then
replacement_started_at_ms="$(date -u +%s%3N)"
echo "REPLACEMENT_STARTED_AT_MS=${replacement_started_at_ms}" >> "${GITHUB_ENV}"
fi
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
if test "${RUNTIME_AVAILABLE}" = false && test "${REFRESH_PHASE}" = applied; then
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
fi
echo "MIG_STABLE_AT_MS=$(date -u +%s%3N)" >> "${GITHUB_ENV}"
- id: post-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new C4 incarnation, image, and unchanged isolation
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${TARGET_IMAGE_DIGEST}" \
--timeout-ms 240000
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${result}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"; then break; fi
sleep 5
done
target_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"
if test "${REFRESH_PHASE}" != applied; then
if test -n "${SOURCE_INCARNATION}"; then
test "${target_incarnation}" != "${SOURCE_INCARNATION}"
fi
test "$(jq -r '.status.runtime.startedAt' <<< "${status}")" \
-ge "${REPLACEMENT_STARTED_AT_MS}"
fi
- name: Require an empty targeted Terraform readback
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
test "$(jq -r '.changes' <<< "${result}")" = 0
@@ -0,0 +1,131 @@
name: Publish Relay Production Image
on:
workflow_dispatch:
inputs:
mode:
description: Publish a new production image or mirror an existing immutable image to staging
required: true
default: publish
type: choice
options: [publish, mirror-staging]
image-digest:
description: Exact existing production digest for mirror-staging mode
required: false
type: string
confirmation:
description: Enter MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING for mirror-staging mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: publish-relay-production
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
publish:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
PUBLISH_MODE: ${{ inputs.mode }}
MIRROR_DIGEST: ${{ inputs.image-digest }}
MIRROR_CONFIRMATION: ${{ inputs.confirmation }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact publish request before authentication
shell: bash
run: |
set -euo pipefail
if test "${PUBLISH_MODE}" = mirror-staging; then
[[ "${MIRROR_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${MIRROR_CONFIRMATION}" = MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING
else
test "${PUBLISH_MODE}" = publish
test -z "${MIRROR_DIGEST}"
test -z "${MIRROR_CONFIRMATION}"
fi
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: docker/setup-buildx-action@v3
- name: Configure Docker auth
run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet
- name: Build and publish immutable image
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}:sha-${GITHUB_SHA}"
docker build -f apps/relay/Dockerfile -t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${DIGEST}"
{
echo '### Terraform candidate image'
echo
echo "\`${IMAGE}\`"
echo
echo 'Declare this digest on a distinct disabled candidate cell in a reviewed Terraform PR.'
} >> "${GITHUB_STEP_SUMMARY}"
- name: Build and publish immutable fence broker
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker:sha-${GITHUB_SHA}"
docker build \
--build-arg "ORCA_RELAY_FENCE_IMAGE_COMMIT=${GITHUB_SHA}" \
-f apps/relay-fence-broker/Dockerfile \
-t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker@${DIGEST}"
{
echo
echo '### Terraform fence broker image'
echo
echo "\`${IMAGE}\`"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Mirror the exact production manifest to staging
if: ${{ inputs.mode == 'mirror-staging' }}
shell: bash
run: |
set -euo pipefail
source_image="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${MIRROR_DIGEST}"
target_tag="${GCP_REGION}-docker.pkg.dev/onorca-cloud-staging/${REPOSITORY_ID}/${IMAGE_NAME}:production-${MIRROR_DIGEST#sha256:}"
source_digest="$(gcloud artifacts docker images describe "${source_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${source_digest}" = "${MIRROR_DIGEST}"
docker pull "${source_image}"
docker tag "${source_image}" "${target_tag}"
docker push "${target_tag}"
target_digest="$(gcloud artifacts docker images describe "${target_tag}" \
--project onorca-cloud-staging --format='value(image_summary.digest)')"
test "${target_digest}" = "${MIRROR_DIGEST}"
{
echo '### Mirrored Relay image'
echo
printf 'Production and staging now resolve the same immutable digest: %s.\n' \
"${MIRROR_DIGEST}"
} >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,411 @@
name: Recover Relay Staging C4 Image
on:
workflow_run:
workflows: [Prove Relay Staging Capacity]
types: [completed]
workflow_dispatch:
inputs:
confirmation:
description: Enter RECOVER_STAGING_ASIA_C4_IMAGE
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event_name == 'workflow_dispatch' || (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion != 'success')) }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
outputs:
recover: ${{ steps.trigger.outputs.recover }}
steps:
- name: Bind recovery to the exact failed C4 job
working-directory: .
id: trigger
env:
CONFIRMATION: ${{ inputs.confirmation }}
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
SOURCE_RUN_EVENT: ${{ github.event.workflow_run.event }}
shell: bash
run: |
set -euo pipefail
if test "${GITHUB_EVENT_NAME}" = workflow_dispatch; then
test "${CONFIRMATION}" = RECOVER_STAGING_ASIA_C4_IMAGE
echo "recover=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${SOURCE_RUN_EVENT}" = workflow_dispatch
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "refresh-asia-c4-image" and
(.conclusion == "failure" or .conclusion == "cancelled" or
.conclusion == "timed_out"))] | length' <<< "${jobs}")"
if test "${count}" = 0; then
echo "recover=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${count}" = 1
echo "recover=true" >> "${GITHUB_OUTPUT}"
recover:
needs: gate
if: ${{ needs.gate.outputs.recover == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
TARGET_ZONE: asia-east2-a
MIG_NAME: orca-cloud-staging-relay-gce-c4
PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
TARGET_IMAGE_DIGEST: sha256:5aedbca5c86de24c8b4d4bf7e3b444b76c712f281ede916cb9d90f70cad1e563
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- id: preflight-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Inspect the exact C4 recovery state
id: preflight
timeout-minutes: 3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.preflight-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${PREDECESSOR_IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${inspect}")"
[[ "${generation}" =~ ^(0|[1-9][0-9]*)$ ]]
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
echo "selector_generation=${generation}" >> "${GITHUB_OUTPUT}"
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
[[ "${current_digest}" =~ ^sha256:[a-f0-9]{64}$ ]]
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
jq -e '.draining | type == "boolean"' <<< "${runtime}" >/dev/null
{
echo "runtime_available=true"
echo "current_digest=${current_digest}"
echo "draining=$(jq -r '.draining' <<< "${runtime}")"
echo "ready=$(jq -r '.status.runtime.ready == true' <<< "${status}")"
} >> "${GITHUB_OUTPUT}"
else
echo "runtime_available=false" >> "${GITHUB_OUTPUT}"
fi
- name: Classify both exact recovery end states
id: recovery-plan
timeout-minutes: 10
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
predecessor_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
target_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${predecessor_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${PREDECESSOR_IMAGE_DIGEST}"
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].image' <<< "${cells}")" = \
"${target_image}"
target_plan="${RUNNER_TEMP}/relay-c4-image-target.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars -lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${target_plan}"
target_result="$(terraform -chdir=infra/terraform show -json "${target_plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${target_image}" \
--rollback-image "${predecessor_image}")"
target_state="$(jq -r '[.changes,.changeKind] | join(":")' <<< "${target_result}")"
case "${target_state}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${predecessor_image}" '.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-recovery.tfvars.json"
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-recovery.tfvars.json" \
-lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${predecessor_image}" \
--rollback-image "${target_image}")"
changes="$(jq -r '.changes' <<< "${result}")"
change_kind="$(jq -r '.changeKind' <<< "${result}")"
case "${changes}:${change_kind}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
mig="$(gcloud compute instance-groups managed describe "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
mig_stable="$(jq -r '.status.isStable == true and .status.versionTarget.isReached == true' \
<<< "${mig}")"
instances="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
instance_stable="$(jq -r 'length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"' <<< "${instances}")"
action=rollback-predecessor
recovery_digest="${PREDECESSOR_IMAGE_DIGEST}"
if test "${{ steps.preflight.outputs.runtime_available }}" = true && \
test "${{ steps.preflight.outputs.ready }}" = true && \
test "${{ steps.preflight.outputs.draining }}" = false && \
test "${mig_stable}" = true && test "${instance_stable}" = true; then
if test "${{ steps.preflight.outputs.current_digest }}" = "${TARGET_IMAGE_DIGEST}" && \
test "${target_state}" = 0:none; then
action=verify-target
recovery_digest="${TARGET_IMAGE_DIGEST}"
elif test "${{ steps.preflight.outputs.current_digest }}" = \
"${PREDECESSOR_IMAGE_DIGEST}" && test "${changes}:${change_kind}" = 0:none; then
action=verify-predecessor
fi
fi
{
echo "changes=${changes}"
echo "change_kind=${change_kind}"
echo "action=${action}"
echo "recovery_digest=${recovery_digest}"
} >> "${GITHUB_OUTPUT}"
- id: fence-auth
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 before predecessor recovery
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 6
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
current_digest="${{ steps.preflight.outputs.current_digest }}"
if test -n "${current_digest}" && [[ ",${expected_digests}," != *",${current_digest},"* ]]; then
expected_digests="${expected_digests},${current_digest}"
fi
if curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/drain" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1,"graceMs":0}' \
>/dev/null; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent \
--expected-image-digests "${expected_digests}" \
--timeout-ms 240000
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 240000
fi
- name: Apply and stabilize the saved predecessor plan
id: apply
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
CHANGES: ${{ steps.recovery-plan.outputs.changes }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
if test "${CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Restart only when the plan did not replace C4
id: restore
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
APPLY_STABLE_AT_MS: ${{ steps.apply.outputs.stable_at_ms }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
if [[ "${CHANGE_KIND}" =~ ^(replacement|replacement-with-obsolete-template|manager-convergence)$ ]]; then
echo "stable_at_ms=${APPLY_STABLE_AT_MS}" >> "${GITHUB_OUTPUT}"
exit 0
fi
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Require an empty selected-image recovery readback
timeout-minutes: 8
env:
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
recovery_image="${image_repository}@${RECOVERY_DIGEST}"
other_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
if test "${RECOVERY_DIGEST}" = "${TARGET_IMAGE_DIGEST}"; then
other_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
fi
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" --arg image "${recovery_image}" \
'.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-readback.tfvars.json"
readback="${RUNNER_TEMP}/relay-c4-image-recovery-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-readback.tfvars.json" \
-lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${readback}"
readback_result="$(terraform -chdir=infra/terraform show -json "${readback}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${recovery_image}" \
--rollback-image "${other_image}")"
test "$(jq -r '.changes' <<< "${readback_result}")" = 0
- id: verify-auth
if: ${{ always() }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the recovered image and unchanged isolation
if: ${{ always() && steps.verify-auth.outcome == 'success' }}
timeout-minutes: 8
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.verify-auth.outputs.id_token }}
SELECTOR_GENERATION: ${{ steps.preflight.outputs.selector_generation }}
STABLE_AT_MS: ${{ steps.restore.outputs.stable_at_ms }}
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${RECOVERY_DIGEST}" \
--timeout-ms 240000
stable_at_ms="${STABLE_AT_MS:-0}"
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"; then break; fi
sleep 5
done
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${RECOVERY_DIGEST}")"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
@@ -0,0 +1,60 @@
name: Requeue Relay Staging C4 Recovery
on:
workflow_run:
workflows: [Recover Relay Staging C4 Image]
types: [completed]
permissions:
actions: write
contents: read
concurrency:
group: relay-staging-c4-recovery-requeue
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
requeue:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion == 'cancelled') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- name: Requeue only a cancelled protected recovery job
working-directory: .
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
shell: bash
run: |
set -euo pipefail
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "recover" and
.conclusion == "cancelled" and .started_at == null)] | length' <<< "${jobs}")"
if test "${count}" = 0; then exit 0; fi
test "${count}" = 1
runs="$(gh api \
"repos/${GITHUB_REPOSITORY}/actions/workflows/cloud-recover-relay-staging-c4-image.yml/runs?branch=main&per_page=100")"
active=0
while IFS= read -r run_id; do
active_jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${run_id}/jobs?filter=latest")"
if jq -se '
([.[].jobs[] | select(.name == "gate" and .conclusion == "success")] | length) == 1 and
([.[].jobs[] | select(.name == "recover" and .status != "completed")] | length) == 1
' <<< "${active_jobs}" >/dev/null; then
active=1
break
fi
done < <(jq -r --arg source "${SOURCE_RUN_ID}" \
'.workflow_runs[] | select((.id | tostring) != $source and
.status != "completed") | .id' <<< "${runs}")
if test "${active}" != 0; then exit 0; fi
gh workflow run cloud-recover-relay-staging-c4-image.yml \
--repo "${GITHUB_REPOSITORY}" --ref main \
-f confirmation=RECOVER_STAGING_ASIA_C4_IMAGE
+120
View File
@@ -0,0 +1,120 @@
name: Cloud Verify
on:
pull_request:
paths:
- cloud/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
push:
branches: [main]
paths:
- cloud/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
permissions:
contents: read
concurrency:
group: cloud-verify-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
working-directory: cloud
jobs:
security:
name: Secret scan
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Scan Cloud history with Gitleaks
run: >-
docker run --rm
--volume "${GITHUB_WORKSPACE}:/repo:ro"
zricethezav/gitleaks@sha256:cdbb7c955abce02001a9f6c9f602fb195b7fadc1e812065883f695d1eeaba854
git /repo --config /repo/cloud/.gitleaks.toml
--log-opts="--all -- cloud :(glob).github/workflows/cloud-*.yml .github/actions/cloud-sql-rollout-lease"
- name: Scan the single-commit Cloud snapshot with TruffleHog
run: >-
docker run --rm
--volume "${GITHUB_WORKSPACE}:/repo:ro"
trufflesecurity/trufflehog@sha256:5dc064868ba7933601b5cbaea6954954d524ddd5dc6222a9667acea70068bf7d
filesystem /repo --no-verification --fail
--include-paths=/repo/cloud/.trufflehog-include-paths.txt
--exclude-paths=/repo/cloud/.trufflehog-exclude-paths.txt
# Compiles the workspace. No Postgres service: nothing here reaches a
# database, and the service container costs ~13s of startup.
build:
runs-on: blacksmith-4vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- run: pnpm build
- run: pnpm typecheck
# Runs in parallel with build. `pnpm test` compiles the one workspace
# package it needs through the relay pretest hook, so it does not depend on
# `pnpm build` having run.
test:
runs-on: blacksmith-4vcpu-ubuntu-2204
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_DB: orca_relay_test
POSTGRES_PASSWORD: relay_test
POSTGRES_USER: relay_test
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U relay_test -d orca_relay_test"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
ORCA_RELAY_TEST_POSTGRES_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- run: pnpm test
# Fork pull requests reach this job, so it never configures a backend, never plans, and never
# holds a credential. Only the relay root ships here; foundation and apps stay private.
terraform:
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
- run: terraform -chdir=infra/terraform fmt -check -recursive
- run: terraform -chdir=infra/terraform init -backend=false -input=false
- run: terraform -chdir=infra/terraform validate
+6
View File
@@ -152,6 +152,12 @@ jobs:
- name: Verify localization catalog
run: pnpm run verify:localization-catalog
# Why: the renderer ships only the English entries i18next cannot rebuild
# from each call site's inline default, so the generated subset has to
# track en.json and those defaults.
- name: Verify runtime-required localization catalog
run: pnpm run verify:localization-runtime-catalog
# Why: extraction writes sorted evidence to an isolated temporary path,
# so feature PRs need one normalized AST pass rather than a three-OS matrix.
- name: Verify localization extraction
+2 -1
View File
@@ -3,5 +3,6 @@
"singleQuote": true,
"semi": false,
"printWidth": 100,
"trailingComma": "none"
"trailingComma": "none",
"ignorePatterns": ["cloud/**", ".github/actions/cloud-sql-rollout-lease/**"]
}
+8 -1
View File
@@ -174,5 +174,12 @@
}
}
],
"ignorePatterns": ["**/node_modules", "**/dist", "**/out", "tests/e2e/.cross-version-checkouts"]
"ignorePatterns": [
"**/node_modules",
"**/dist",
"**/out",
"cloud/**",
".github/actions/cloud-sql-rollout-lease/**",
"tests/e2e/.cross-version-checkouts"
]
}
+3
View File
@@ -252,6 +252,9 @@ Pair with your desktop app to monitor and steer your agents from your phone.
Want to contribute or run locally? See our [CONTRIBUTING.md](.github/CONTRIBUTING.md) guide.
The relay that pairs the mobile app with a desktop host is also in this repository under
[`cloud/`](cloud/README.md), with a separate pnpm workspace and setup guide.
<a href="https://github.com/stablyai/orca/graphs/contributors">
<img src="https://contrib.rocks/image?repo=stablyai/orca" alt="Orca contributors" />
</a>
+15
View File
@@ -0,0 +1,15 @@
.git/
.github/
.local/
.terraform/
node_modules/
dist/
coverage/
.env
.env.local
.env.local.generated
*.log
*.tfplan
*.tfstate
*.tfstate.*
+10
View File
@@ -0,0 +1,10 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
+24
View File
@@ -0,0 +1,24 @@
node_modules/
dist/
coverage/
.turbo/
.local/
.env
.env.local
.env.local.generated
*.log
# Terraform/OpenTofu local state and plans must stay out of git.
**/.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
override.tf
override.tf.json
*_override.tf
*_override.tf.json
# Local dev signing key + SQLite live under data/ — never commit (contains a
# private key and dev PII). Prod supplies the key via ORCA_CLOUD_SIGNING_KEY_PEM.
data/
+15
View File
@@ -0,0 +1,15 @@
[extend]
useDefault = true
[[allowlists]]
description = "Explicit Relay test signing key"
regexTarget = "secret"
regexes = ['''^test-assignment-key-with-at-least-32-bytes$''']
# The Cloud SQL rollout lease records `owner/repo/run_id` as the holder of a lease. The action's
# unit tests build fixture holders from that shape, which the generic key rule reads as a secret.
[[allowlists]]
description = "Cloud SQL rollout lease holder keys in the action's unit tests"
regexTarget = "secret"
paths = ['''\.github/actions/cloud-sql-rollout-lease/[a-z-]+\.test\.mjs$''']
regexes = ['''^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+/[0-9]+$''']
+2
View File
@@ -0,0 +1,2 @@
24
+3
View File
@@ -0,0 +1,3 @@
engine-strict=false
package-manager-strict=true
+1
View File
@@ -0,0 +1 @@
(^|/)cloud/apps/relay/src/postgres-idle-client-error\.test\.ts$
+2
View File
@@ -0,0 +1,2 @@
(^|/)cloud/
(^|/)\.github/workflows/cloud-[^/]+\.yml$
+92
View File
@@ -0,0 +1,92 @@
# Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and
desktops never talk to each other directly: each opens an outbound WebSocket
to a relay cell, the relay pairs the two sessions, and it splices frames
between them. A director assigns hosts to cells and coordinates migrations;
cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from `cloud/`, not the repository root. The source is covered by
the repository's root [MIT license](../LICENSE).
## Packages
- `packages/relay-contract`: the wire contract shared by the relay, the
desktop app, and the mobile app (frame shapes, close codes, admission budgets,
splice state machine).
- `apps/relay`: the relay server. The same image runs as a director or a cell
depending on `ORCA_RELAY_ROLE`.
- `apps/relay-fence-broker`: a private, IAM-only service that owns the durable
mutation lease, the Terraform checkout, and the narrow Compute mutation used
when a registered target is superseded. The workflow that calls it holds read
and invoke rights only, never those mutation permissions.
- `apps/relay-ops`: the relay operations console and the incident monitor
behind `pnpm ops:relay`, `pnpm incident:relay`, and
`pnpm incident:relay-preflight`.
## Infrastructure and operations
- `infra/terraform`: the relay Terraform root. It owns the cells, the director,
the shared Cloud SQL instance, DNS, observability, and every GitHub Workload
Identity provider the relay workflows authenticate through. `backend/` holds
the per-environment backend configuration and `environments/` the tfvars.
Drive it through `pnpm infra:init`, `pnpm infra:plan`, and `pnpm infra:apply`.
- `dev/scripts`: the deploy, capacity, admission, rehome, monitoring, and load
scripts the workflows call, plus the contract tests that pin each workflow
and Terraform surface. Run them with `pnpm test`.
- `dev/contracts` and `dev/fixtures`: the checked-in data those contract tests
read, including the Terraform root partition.
- `docs/`: the relay runbooks, capacity-testing guide, incident-monitor
reference, and the workflow variable reference in `docs/relay-workflows.md`.
## Workflows
The 24 `.github/workflows/cloud-*.yml` workflows are the relay's deploy and
operate surface: publish and deploy the director, roll GCE cell capacity,
operate Asia admission and regional rehoming, prove staging capacity, monitor
production, and power staging up and down. `.github/actions/cloud-sql-rollout-lease`
is the compare-and-swap lease that serializes every rollout against the shared
Cloud SQL instance.
Every one of them is inert. Each top-level job is gated on
`vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true'`, a repository variable that is
unset here, so the two scheduled triggers and every manual dispatch skip
without running a step. Only the repository owner, holding the GCP identities
these workflows authenticate as, can turn them on.
`Cloud Verify` is not gated. It builds, typechecks, lints, tests, secret-scans,
and validates the relay Terraform on every change under `cloud/`, and it runs
on fork pull requests, so it configures no backend and holds no credential.
## What is not here
The `terraform-foundation` and `terraform-apps` roots and the API and auth
services live in the private `stablyai/orca-cloud` repository. Scripts and
tests that spanned both trees were narrowed to the relay side rather than
carrying a dangling reference.
## Local development
```sh
cd cloud
pnpm install
pnpm build
pnpm test
```
`pnpm test` runs the SQLite-backed suites. Tests that need PostgreSQL run only
when `ORCA_RELAY_TEST_POSTGRES_URL` points at a disposable PostgreSQL 16 or 17
database, for example:
```sh
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
```
Configuration is read from environment variables validated in
`apps/relay/src/config.ts`. `ORCA_RELAY_ASSIGNMENT_SIGNING_KEY` (at least 32
bytes) is the only required value; everything else has a local default.
+33
View File
@@ -0,0 +1,33 @@
FROM node:24-bookworm-slim AS build
WORKDIR /workspace
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json ./
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
RUN pnpm install --frozen-lockfile
COPY apps/relay-fence-broker apps/relay-fence-broker
RUN pnpm --filter @orca-cloud/relay-fence-broker build
FROM hashicorp/terraform:1.15.8 AS terraform
FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
ARG ORCA_RELAY_FENCE_IMAGE_COMMIT
RUN test "$(printf '%s' "${ORCA_RELAY_FENCE_IMAGE_COMMIT}" | grep -E '^[a-f0-9]{40}$')"
ENV NODE_ENV=production
ENV PORT=8080
ENV ORCA_RELAY_FENCE_IMAGE_COMMIT=${ORCA_RELAY_FENCE_IMAGE_COMMIT}
ENV IAC_TOOL=terraform
WORKDIR /workspace
COPY --from=build /usr/local /usr/local
COPY --from=terraform /bin/terraform /usr/local/bin/terraform
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
COPY --from=build /workspace/apps/relay-fence-broker/dist apps/relay-fence-broker/dist
COPY dev/scripts dev/scripts
COPY infra/terraform infra/terraform
RUN corepack enable \
&& pnpm install --prod --frozen-lockfile --filter @orca-cloud/relay-fence-broker... \
&& useradd --create-home --uid 10001 broker \
&& chown -R broker:broker /workspace
USER broker
EXPOSE 8080
CMD ["node", "apps/relay-fence-broker/dist/index.js"]
@@ -0,0 +1,27 @@
{
"name": "@orca-cloud/relay-fence-broker",
"private": true,
"version": "0.0.0",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "pnpm clean && tsc -p tsconfig.build.json",
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
"dev": "tsx watch src/index.ts",
"lint": "tsc -p tsconfig.json --noEmit",
"start": "node dist/index.js",
"test": "vitest run",
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"@hono/node-server": "^1.19.14",
"hono": "^4.12.27",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^24.10.0",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vitest": "^4.0.8"
}
}
@@ -0,0 +1,316 @@
import { describe, expect, it, vi } from 'vitest'
import { createApp } from './app.js'
import type { RelayFenceBrokerConfig } from './config.js'
import type {
GoogleStorageMutationLease,
MutationLease
} from './mutation-lease.js'
const commit = 'a'.repeat(40)
const config: RelayFenceBrokerConfig = {
port: 8080,
project: 'onorca-cloud',
stateBucket: 'onorca-cloud-terraform-state',
leaseObject: 'terraform/state/relay-fence-broker/production.lock',
directorOrigin: 'https://relay.onorca.dev',
adminAudience: 'https://relay.onorca.dev/v1/admin/drain',
requesterServiceAccount: 'requester@example.com',
runtimeServiceAccount: 'runtime@example.com',
sourceCellId: 'production-gce-c3',
failedTargetCellId: 'production-gce-c11',
replacementTargetCellId: 'production-gce-c12',
imageCommit: commit,
terraformDir: 'infra/terraform',
unobservedConnectionBound: 10,
connectionCeiling: 600
}
function request(fenceCommit = commit): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit,
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function recoveryRequest(): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: commit,
completedFenceRecovery: {
attemptId: '11111111-1111-4111-8111-111111111111',
fenceCommit: 'b'.repeat(40),
gceOperation: 'operation-1',
terraformStateSerial: 61,
planObjectGeneration: '123',
terraformStateObjectGeneration: '456',
terraformStateObjectSha256: 'c'.repeat(64)
},
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
},
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function leaseTakeoverRequest(): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: commit,
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
},
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function sourceFenceRequest(
overrides: Record<string, unknown> = {}
): Request {
return new Request('http://broker/v1/fence-source', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c3-final-fence',
fenceCommit: commit,
targetCellIds: ['production-gce-c7', 'production-gce-c12'],
confirmation: 'FENCE_SOURCE',
...overrides
})
})
}
describe('relay fence broker', () => {
it('runs only after acquiring the durable lease', async () => {
const events: string[] = []
const lease = {
acquire: vi.fn(async () => {
events.push('acquire')
return {
generation: '7',
record: {}
} as MutationLease
}),
release: vi.fn(async () => {
events.push('release')
})
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
supersede: async () => {
events.push('supersede')
}
})
const response = await app.request(request())
expect(response.status).toBe(200)
expect(events).toEqual(['acquire', 'supersede', 'release'])
})
it('rejects a commit not bound to the immutable image', async () => {
const lease = {
acquire: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const response = await app.request(request('b'.repeat(40)))
expect(response.status).toBe(409)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('retains the lease when supersession fails', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '7', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
supersede: async () => {
throw new Error('stopped safely')
}
})
const response = await app.request(request())
expect(response.status).toBe(500)
expect(lease.release).not.toHaveBeenCalled()
})
it('passes exact completed-attempt and live-lease recovery pins', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '8', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const supersede = vi.fn(async () => {})
const app = createApp(config, { lease, supersede })
const response = await app.request(recoveryRequest())
expect(response.status).toBe(200)
expect(lease.acquire).toHaveBeenCalledWith(
'c11-to-c12-forward',
expect.objectContaining({
completedFenceRecovery: expect.objectContaining({
terraformStateSerial: 61
})
}),
{
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
)
expect(supersede).toHaveBeenCalledWith(
config,
expect.objectContaining({
attemptId: '11111111-1111-4111-8111-111111111111'
})
)
})
it('conditionally resumes the exact live supersession lease', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '8', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const supersede = vi.fn(async () => {})
const app = createApp(config, { lease, supersede })
const response = await app.request(leaseTakeoverRequest())
expect(response.status).toBe(200)
expect(lease.acquire).toHaveBeenCalledWith(
'c11-to-c12-forward',
expect.objectContaining({
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
}),
{
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
)
expect(supersede).toHaveBeenCalledWith(config, undefined)
})
it('rejects a live supersession lease for another operation', async () => {
const lease = {
acquire: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const request = leaseTakeoverRequest()
const body = (await request.json()) as {
expectedLease: { operationId: string }
}
body.expectedLease.operationId = 'another-operation'
const response = await app.request(
new Request(request.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body)
})
)
expect(response.status).toBe(400)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('fences the configured source only after acquiring the durable lease', async () => {
const events: string[] = []
const expectedLease = {
generation: '6',
operationId: 'c3-final-fence',
requestDigest: 'd'.repeat(64)
}
const lease = {
acquire: vi.fn(async () => {
events.push('acquire')
return { generation: '7', record: {} } as MutationLease
}),
release: vi.fn(async () => {
events.push('release')
})
} as unknown as GoogleStorageMutationLease
const fenceSource = vi.fn(async () => {
events.push('fence')
})
const app = createApp(config, { lease, fenceSource })
const response = await app.request(sourceFenceRequest({ expectedLease }))
expect(response.status).toBe(200)
expect(events).toEqual(['acquire', 'fence', 'release'])
expect(lease.acquire).toHaveBeenCalledWith(
'c3-final-fence',
expect.objectContaining({ targetCellIds: expect.any(Array) }),
expectedLease
)
expect(fenceSource).toHaveBeenCalledWith(config, [
'production-gce-c7',
'production-gce-c12'
])
})
it('rejects duplicate targets and the configured source', async () => {
const lease = { acquire: vi.fn() } as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const duplicate = await app.request(
sourceFenceRequest({
targetCellIds: ['production-gce-c7', 'production-gce-c7']
})
)
const source = await app.request(
sourceFenceRequest({ targetCellIds: [config.sourceCellId] })
)
expect(duplicate.status).toBe(400)
expect(source.status).toBe(400)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('retains the source-fence lease after a controlled failure', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '7', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
fenceSource: async () => {
throw new Error('stopped safely')
}
})
const response = await app.request(sourceFenceRequest())
expect(response.status).toBe(500)
expect(lease.release).not.toHaveBeenCalled()
})
})
+162
View File
@@ -0,0 +1,162 @@
import { Hono } from 'hono'
import { z } from 'zod'
import type { RelayFenceBrokerConfig } from './config.js'
import {
GoogleStorageMutationLease,
MutationLeaseConflict
} from './mutation-lease.js'
import {
runSourceFence,
runTargetSupersession
} from './fence-operation-runner.js'
const expectedLeaseSchema = z
.object({
generation: z.string().regex(/^[1-9][0-9]{0,30}$/),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
requestDigest: z.string().regex(/^[a-f0-9]{64}$/)
})
.strict()
const supersessionRequestSchema = z.object({
v: z.literal(1),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
completedFenceRecovery: z
.object({
attemptId: z.string().uuid(),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
gceOperation: z.string().min(1).max(256),
terraformStateSerial: z.number().int().nonnegative().safe(),
planObjectGeneration: z.string().regex(/^[1-9][0-9]{0,30}$/),
terraformStateObjectGeneration: z.string().regex(/^[1-9][0-9]{0,30}$/),
terraformStateObjectSha256: z.string().regex(/^[a-f0-9]{64}$/)
})
.strict()
.optional(),
expectedLease: expectedLeaseSchema.optional(),
confirmation: z.literal('SUPERSEDE_TARGET')
})
.strict()
.refine(
(value) =>
(!value.completedFenceRecovery || Boolean(value.expectedLease)) &&
(!value.expectedLease || value.expectedLease.operationId === value.operationId)
)
const cellIdSchema = z.string().regex(/^[a-z][a-z0-9-]{0,127}$/)
const sourceFenceRequestSchema = z
.object({
v: z.literal(1),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
targetCellIds: z.array(cellIdSchema).min(1).max(16),
expectedLease: expectedLeaseSchema.optional(),
confirmation: z.literal('FENCE_SOURCE')
})
.strict()
.refine(
(value) =>
new Set(value.targetCellIds).size === value.targetCellIds.length &&
(!value.expectedLease || value.expectedLease.operationId === value.operationId)
)
type AppDependencies = {
lease?: GoogleStorageMutationLease
supersede?: (
config: RelayFenceBrokerConfig,
recovery?: z.infer<typeof supersessionRequestSchema>['completedFenceRecovery']
) => Promise<void>
fenceSource?: (
config: RelayFenceBrokerConfig,
targetCellIds: string[]
) => Promise<void>
}
export function createApp(
config: RelayFenceBrokerConfig,
dependencies: AppDependencies = {}
): Hono {
const app = new Hono()
const lease =
dependencies.lease ??
new GoogleStorageMutationLease(
config.stateBucket,
config.leaseObject,
config.imageCommit
)
const supersede = dependencies.supersede ?? runTargetSupersession
const fenceSource = dependencies.fenceSource ?? runSourceFence
app.get('/healthz', (context) =>
context.json({ ok: true, fenceCommit: config.imageCommit })
)
app.post('/v1/supersede-target', async (context) => {
const parsed = supersessionRequestSchema.safeParse(
await context.req.json().catch(() => null)
)
if (!parsed.success) return context.json({ error: 'invalid_request' }, 400)
if (parsed.data.fenceCommit !== config.imageCommit) {
return context.json({ error: 'fence_commit_mismatch' }, 409)
}
let acquired
try {
acquired = await lease.acquire(
parsed.data.operationId,
parsed.data,
parsed.data.expectedLease
)
} catch (error) {
if (error instanceof MutationLeaseConflict) {
return context.json({ error: 'mutation_lease_conflict' }, 409)
}
throw error
}
await supersede(config, parsed.data.completedFenceRecovery)
await lease.release(acquired)
return context.json({
ok: true,
operationId: parsed.data.operationId,
fenceCommit: config.imageCommit
})
})
app.post('/v1/fence-source', async (context) => {
const parsed = sourceFenceRequestSchema.safeParse(
await context.req.json().catch(() => null)
)
if (
!parsed.success ||
parsed.data.targetCellIds.includes(config.sourceCellId)
) {
return context.json({ error: 'invalid_request' }, 400)
}
if (parsed.data.fenceCommit !== config.imageCommit) {
return context.json({ error: 'fence_commit_mismatch' }, 409)
}
let acquired
try {
acquired = await lease.acquire(
parsed.data.operationId,
parsed.data,
parsed.data.expectedLease
)
} catch (error) {
if (error instanceof MutationLeaseConflict) {
return context.json({ error: 'mutation_lease_conflict' }, 409)
}
throw error
}
await fenceSource(config, parsed.data.targetCellIds)
await lease.release(acquired)
return context.json({
ok: true,
operationId: parsed.data.operationId,
fenceCommit: config.imageCommit
})
})
app.onError((error, context) => {
console.error(error)
return context.json({ error: 'broker_operation_failed' }, 500)
})
return app
}
@@ -0,0 +1,92 @@
import { z } from 'zod'
const environmentSchema = z.object({
PORT: z.coerce.number().int().positive().max(65_535).default(8080),
ORCA_RELAY_FENCE_PROJECT: z.string().regex(/^[a-z][a-z0-9-]{4,29}$/),
ORCA_RELAY_FENCE_STATE_BUCKET: z.string().min(3).max(222),
ORCA_RELAY_FENCE_LEASE_OBJECT: z.string().min(1).max(512),
ORCA_RELAY_FENCE_DIRECTOR_ORIGIN: z.string().url(),
ORCA_RELAY_FENCE_ADMIN_AUDIENCE: z.string().url(),
ORCA_RELAY_FENCE_REQUESTER_SERVICE_ACCOUNT: z.string().email(),
ORCA_RELAY_FENCE_RUNTIME_SERVICE_ACCOUNT: z.string().email(),
ORCA_RELAY_FENCE_SOURCE_CELL_ID: z.string().regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID: z
.string()
.regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID: z
.string()
.regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_IMAGE_COMMIT: z.string().regex(/^[a-f0-9]{40}$/),
// Resolved against the broker's working directory, which the image sets to the copied tree root.
ORCA_RELAY_FENCE_TERRAFORM_DIR: z.string().min(1).default('infra/terraform'),
ORCA_RELAY_FENCE_UNOBSERVED_CONNECTION_BOUND: z.coerce
.number()
.int()
.nonnegative()
.max(499),
ORCA_RELAY_FENCE_CONNECTION_CEILING: z.coerce
.number()
.int()
.positive()
.max(600)
.default(600)
})
export type RelayFenceBrokerConfig = {
port: number
project: string
stateBucket: string
leaseObject: string
directorOrigin: string
adminAudience: string
requesterServiceAccount: string
runtimeServiceAccount: string
sourceCellId: string
failedTargetCellId: string
replacementTargetCellId: string
imageCommit: string
terraformDir: string
unobservedConnectionBound: number
connectionCeiling: number
}
export function loadConfig(
environment: NodeJS.ProcessEnv = process.env
): RelayFenceBrokerConfig {
const parsed = environmentSchema.parse(environment)
const director = new URL(parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN)
const audience = new URL(parsed.ORCA_RELAY_FENCE_ADMIN_AUDIENCE)
if (
director.origin !== parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN ||
audience.origin !== director.origin ||
audience.pathname !== '/v1/admin/drain' ||
audience.search ||
audience.hash
) {
throw new Error('broker director and admin audience must use the canonical drain origin')
}
const cells = new Set([
parsed.ORCA_RELAY_FENCE_SOURCE_CELL_ID,
parsed.ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID,
parsed.ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID
])
if (cells.size !== 3) throw new Error('broker cells must be distinct')
return {
port: parsed.PORT,
project: parsed.ORCA_RELAY_FENCE_PROJECT,
stateBucket: parsed.ORCA_RELAY_FENCE_STATE_BUCKET,
leaseObject: parsed.ORCA_RELAY_FENCE_LEASE_OBJECT,
directorOrigin: parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN,
adminAudience: parsed.ORCA_RELAY_FENCE_ADMIN_AUDIENCE,
requesterServiceAccount: parsed.ORCA_RELAY_FENCE_REQUESTER_SERVICE_ACCOUNT,
runtimeServiceAccount: parsed.ORCA_RELAY_FENCE_RUNTIME_SERVICE_ACCOUNT,
sourceCellId: parsed.ORCA_RELAY_FENCE_SOURCE_CELL_ID,
failedTargetCellId: parsed.ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID,
replacementTargetCellId: parsed.ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID,
imageCommit: parsed.ORCA_RELAY_FENCE_IMAGE_COMMIT,
terraformDir: parsed.ORCA_RELAY_FENCE_TERRAFORM_DIR,
unobservedConnectionBound:
parsed.ORCA_RELAY_FENCE_UNOBSERVED_CONNECTION_BOUND,
connectionCeiling: parsed.ORCA_RELAY_FENCE_CONNECTION_CEILING
}
}
@@ -0,0 +1,57 @@
import { describe, expect, it } from 'vitest'
import type { RelayFenceBrokerConfig } from './config.js'
import {
fenceChildEnvironment,
sourceFenceArguments
} from './fence-operation-runner.js'
const config = {
project: 'onorca-cloud',
directorOrigin: 'https://relay.onorca.dev',
adminAudience: 'https://relay.onorca.dev/v1/admin/drain',
sourceCellId: 'production-gce-c3',
runtimeServiceAccount: 'runtime@example.com',
imageCommit: 'a'.repeat(40),
terraformDir: 'infra/terraform',
unobservedConnectionBound: 10,
connectionCeiling: 600
} as RelayFenceBrokerConfig
describe('fence operation runner', () => {
it('passes distinct read and mutation identity tokens', () => {
expect(
fenceChildEnvironment(
config,
'read.token.value',
'mutation.token.value',
{ PRESERVED: 'yes' }
)
).toMatchObject({
PRESERVED: 'yes',
IAC_TOOL: 'terraform',
ORCA_RELAY_ADMIN_ID_TOKEN: 'read.token.value',
ORCA_RELAY_FENCE_MUTATION_ID_TOKEN: 'mutation.token.value',
ORCA_RELAY_FENCE_IMAGE_COMMIT: 'a'.repeat(40)
})
})
it('binds a source fence to deterministic targets and the production var file', () => {
const args = sourceFenceArguments(config, '/tmp/topology.json', [
'production-gce-c12',
'production-gce-c7'
])
expect(args).toEqual(
expect.arrayContaining([
'--source-cell-id',
'production-gce-c3',
'--target-cell-ids',
'production-gce-c12,production-gce-c7',
'--terraform-var-file',
'environments/production.tfvars',
'--mode',
'fence-source'
])
)
})
})
@@ -0,0 +1,185 @@
import { execFile } from 'node:child_process'
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { promisify } from 'node:util'
import type { RelayFenceBrokerConfig } from './config.js'
import {
metadataIdentityToken,
metadataServiceAccountEmail
} from './google-metadata.js'
const exec = promisify(execFile)
const MAX_OUTPUT_BYTES = 10 * 1024 * 1024
type CompletedFenceRecovery = {
attemptId: string
fenceCommit: string
gceOperation: string
terraformStateSerial: number
planObjectGeneration: string
terraformStateObjectGeneration: string
terraformStateObjectSha256: string
}
async function run(file: string, args: string[], environment?: NodeJS.ProcessEnv) {
return await exec(file, args, {
env: environment,
maxBuffer: MAX_OUTPUT_BYTES
})
}
export function fenceChildEnvironment(
config: RelayFenceBrokerConfig,
readToken: string,
mutationToken: string,
environment: NodeJS.ProcessEnv = process.env
): NodeJS.ProcessEnv {
return {
...environment,
IAC_TOOL: 'terraform',
ORCA_RELAY_ADMIN_ID_TOKEN: readToken,
ORCA_RELAY_FENCE_MUTATION_ID_TOKEN: mutationToken,
ORCA_RELAY_FENCE_IMAGE_COMMIT: config.imageCommit
}
}
function relayFenceArguments(
config: RelayFenceBrokerConfig,
topologyFile: string,
targetCellIds: string[]
): string[] {
return [
'dev/scripts/deploy-relay-gce-multi-target.mjs',
'--project',
config.project,
'--director-origin',
config.directorOrigin,
'--admin-audience',
config.adminAudience,
'--topology-file',
topologyFile,
'--source-cell-id',
config.sourceCellId,
'--target-cell-ids',
[...targetCellIds].sort().join(','),
'--unobserved-connection-bound',
String(config.unobservedConnectionBound),
'--runtime-service-account',
config.runtimeServiceAccount,
'--environment',
'production',
'--fence-commit',
config.imageCommit,
'--terraform-dir',
config.terraformDir,
'--terraform-var-file',
'environments/production.tfvars',
'--connection-ceiling',
String(config.connectionCeiling),
'--minimum-lease-remaining-ms',
'600000'
]
}
async function runRelayFenceCommand(
config: RelayFenceBrokerConfig,
argumentsForTopology: (
topologyFile: string,
brokerServiceAccount: string
) => string[]
): Promise<void> {
const directory = await mkdtemp(join(tmpdir(), 'orca-relay-fence-operation-'))
const topologyFile = join(directory, 'topology.json')
try {
await run('terraform', [
`-chdir=${config.terraformDir}`,
'init',
'-input=false',
'-backend-config=backend/production.hcl'
])
const topology = await run('terraform', [
`-chdir=${config.terraformDir}`,
'output',
'-json',
'relay_gce_cell_deployments'
])
await writeFile(topologyFile, topology.stdout, { mode: 0o600 })
const brokerToken = await metadataIdentityToken(config.adminAudience)
const brokerServiceAccount = await metadataServiceAccountEmail()
const environment = fenceChildEnvironment(
config,
brokerToken,
brokerToken
)
await run(
'node',
argumentsForTopology(topologyFile, brokerServiceAccount),
environment
)
} finally {
await rm(directory, { recursive: true, force: true })
}
}
export function sourceFenceArguments(
config: RelayFenceBrokerConfig,
topologyFile: string,
targetCellIds: string[]
): string[] {
return [
...relayFenceArguments(config, topologyFile, targetCellIds),
'--mode',
'fence-source'
]
}
export async function runSourceFence(
config: RelayFenceBrokerConfig,
targetCellIds: string[]
): Promise<void> {
await runRelayFenceCommand(config, (topologyFile) =>
sourceFenceArguments(config, topologyFile, targetCellIds)
)
}
export async function runTargetSupersession(
config: RelayFenceBrokerConfig,
recovery?: CompletedFenceRecovery
): Promise<void> {
await runRelayFenceCommand(config, (topologyFile, brokerServiceAccount) => {
const targets = [
config.failedTargetCellId,
config.replacementTargetCellId
]
const args = [
...relayFenceArguments(config, topologyFile, targets),
'--failed-target-cell-id',
config.failedTargetCellId,
'--replacement-target-cell-id',
config.replacementTargetCellId,
'--mode',
'supersede-target'
]
if (recovery) {
args.push(
'--completed-fence-attempt-id',
recovery.attemptId,
'--completed-fence-commit',
recovery.fenceCommit,
'--completed-fence-operation',
recovery.gceOperation,
'--completed-fence-state-serial',
String(recovery.terraformStateSerial),
'--completed-fence-plan-generation',
recovery.planObjectGeneration,
'--completed-fence-state-generation',
recovery.terraformStateObjectGeneration,
'--completed-fence-state-sha256',
recovery.terraformStateObjectSha256,
'--fence-broker-service-account',
brokerServiceAccount
)
}
return args
})
}
@@ -0,0 +1,44 @@
const METADATA_ROOT =
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default'
async function metadataText(path: string, fetcher: typeof fetch): Promise<string> {
const response = await fetcher(`${METADATA_ROOT}/${path}`, {
headers: { 'Metadata-Flavor': 'Google' }
})
if (!response.ok) throw new Error(`metadata request failed: ${response.status}`)
return await response.text()
}
export async function metadataAccessToken(fetcher: typeof fetch = fetch): Promise<string> {
const body = JSON.parse(await metadataText('token', fetcher)) as {
access_token?: unknown
}
if (typeof body.access_token !== 'string' || body.access_token.length < 20) {
throw new Error('metadata access token is missing')
}
return body.access_token
}
export async function metadataServiceAccountEmail(
fetcher: typeof fetch = fetch
): Promise<string> {
const email = (await metadataText('email', fetcher)).trim()
if (!/^[^@\s]+@[^@\s]+\.gserviceaccount\.com$/.test(email)) {
throw new Error('metadata service account email is invalid')
}
return email
}
export async function metadataIdentityToken(
audience: string,
fetcher: typeof fetch = fetch
): Promise<string> {
const token = await metadataText(
`identity?audience=${encodeURIComponent(audience)}&format=full`,
fetcher
)
if (!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)) {
throw new Error('metadata identity token is invalid')
}
return token
}
@@ -0,0 +1,10 @@
import { serve } from '@hono/node-server'
import { createApp } from './app.js'
import { loadConfig } from './config.js'
const config = loadConfig()
const app = createApp(config)
serve({ fetch: app.fetch, port: config.port }, (info) => {
console.log(`[relay-fence-broker] listening on port ${info.port}`)
})
@@ -0,0 +1,164 @@
import { describe, expect, it, vi } from 'vitest'
import {
GoogleStorageMutationLease,
MutationLeaseConflict
} from './mutation-lease.js'
function json(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { 'Content-Type': 'application/json' }
})
}
const accessToken = () => json({ access_token: 'a'.repeat(40) })
const request = {
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: 'a'.repeat(40),
confirmation: 'SUPERSEDE_TARGET'
}
describe('GoogleStorageMutationLease', () => {
it('creates and conditionally releases a new lease', async () => {
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(new Response(null, { status: 404 }))
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(new Response(null, { status: 204 }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request)
await leaseStore.release(lease)
expect(lease.generation).toBe('7')
expect(fetcher.mock.calls[2]?.[0]).toContain('ifGenerationMatch=0')
expect(fetcher.mock.calls[4]?.[0]).toContain('ifGenerationMatch=7')
})
it('rejects a different request while the durable lease is live', async () => {
const existing = {
operationId: 'other-operation',
requestDigest: 'b'.repeat(64),
imageCommit: 'a'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
await expect(
leaseStore.acquire(request.operationId, request)
).rejects.toBeInstanceOf(MutationLeaseConflict)
expect(fetcher).toHaveBeenCalledTimes(3)
})
it('takes over an expired lease with an exact generation precondition', async () => {
const existing = {
operationId: 'other-operation',
requestDigest: 'b'.repeat(64),
imageCommit: 'a'.repeat(40),
acquiredAt: 500,
expiresAt: 999
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
.mockResolvedValueOnce(json({ generation: '8' }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request)
expect(lease.generation).toBe('8')
expect(fetcher.mock.calls[3]?.[0]).toContain('ifGenerationMatch=7')
})
it('conditionally replaces the exact authorized live lease', async () => {
const existing = {
operationId: request.operationId,
requestDigest: 'b'.repeat(64),
imageCommit: 'b'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
.mockResolvedValueOnce(json({ generation: '8' }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request, {
generation: '7',
operationId: existing.operationId,
requestDigest: existing.requestDigest
})
expect(lease.generation).toBe('8')
expect(fetcher.mock.calls[3]?.[0]).toContain('ifGenerationMatch=7')
})
it('rejects a live-lease takeover when any expected field differs', async () => {
const existing = {
operationId: request.operationId,
requestDigest: 'b'.repeat(64),
imageCommit: 'b'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
await expect(
leaseStore.acquire(request.operationId, request, {
generation: '8',
operationId: existing.operationId,
requestDigest: existing.requestDigest
})
).rejects.toBeInstanceOf(MutationLeaseConflict)
expect(fetcher).toHaveBeenCalledTimes(3)
})
})
@@ -0,0 +1,152 @@
import { createHash } from 'node:crypto'
import { metadataAccessToken } from './google-metadata.js'
const LEASE_TTL_MS = 35 * 60 * 1_000
type LeaseRecord = {
operationId: string
requestDigest: string
imageCommit: string
acquiredAt: number
expiresAt: number
}
type ObjectMetadata = {
generation: string
}
export class MutationLeaseConflict extends Error {}
export type MutationLease = {
generation: string
record: LeaseRecord
}
export type ExpectedMutationLease = {
generation: string
operationId: string
requestDigest: string
}
export class GoogleStorageMutationLease {
constructor(
private readonly bucket: string,
private readonly objectName: string,
private readonly imageCommit: string,
private readonly fetcher: typeof fetch = fetch,
private readonly now: () => number = Date.now
) {}
async acquire(
operationId: string,
request: unknown,
expectedExisting?: ExpectedMutationLease
): Promise<MutationLease> {
const token = await metadataAccessToken(this.fetcher)
const existing = await this.read(token)
const requestDigest = createHash('sha256')
.update(JSON.stringify(request))
.digest('hex')
const exactTakeover =
existing &&
expectedExisting?.generation === existing.metadata.generation &&
expectedExisting.operationId === existing.record.operationId &&
expectedExisting.requestDigest === existing.record.requestDigest
if (
(!existing && expectedExisting) ||
(existing &&
existing.record.requestDigest !== requestDigest &&
existing.record.expiresAt > this.now() &&
!exactTakeover)
) {
throw new MutationLeaseConflict('another relay mutation owns the durable lease')
}
const acquiredAt = this.now()
const record: LeaseRecord = {
operationId,
requestDigest,
imageCommit: this.imageCommit,
acquiredAt,
expiresAt: acquiredAt + LEASE_TTL_MS
}
const generation = existing?.metadata.generation ?? '0'
const response = await this.fetcher(
`${this.uploadUrl()}&ifGenerationMatch=${encodeURIComponent(generation)}`,
{
method: 'POST',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(record)
}
)
if (response.status === 412) {
throw new MutationLeaseConflict('relay mutation lease changed concurrently')
}
if (!response.ok) throw new Error(`mutation lease acquisition failed: ${response.status}`)
const metadata = (await response.json()) as Partial<ObjectMetadata>
if (!/^[1-9][0-9]{0,30}$/.test(metadata.generation ?? '')) {
throw new Error('mutation lease has no valid generation')
}
return { generation: metadata.generation!, record }
}
async release(lease: MutationLease): Promise<void> {
const token = await metadataAccessToken(this.fetcher)
const response = await this.fetcher(
`${this.metadataUrl()}?ifGenerationMatch=${encodeURIComponent(lease.generation)}`,
{
method: 'DELETE',
headers: { Authorization: `Bearer ${token}` }
}
)
if (!response.ok && response.status !== 404) {
throw new Error(`mutation lease release failed: ${response.status}`)
}
}
private async read(
token: string
): Promise<{ metadata: ObjectMetadata; record: LeaseRecord } | null> {
const metadataResponse = await this.fetcher(this.metadataUrl(), {
headers: { Authorization: `Bearer ${token}` }
})
if (metadataResponse.status === 404) return null
if (!metadataResponse.ok) {
throw new Error(`mutation lease inspection failed: ${metadataResponse.status}`)
}
const metadata = (await metadataResponse.json()) as Partial<ObjectMetadata>
if (!/^[1-9][0-9]{0,30}$/.test(metadata.generation ?? '')) {
throw new Error('existing mutation lease has no valid generation')
}
const bodyResponse = await this.fetcher(`${this.metadataUrl()}?alt=media`, {
headers: { Authorization: `Bearer ${token}` }
})
if (!bodyResponse.ok) {
throw new Error(`mutation lease body read failed: ${bodyResponse.status}`)
}
const record = (await bodyResponse.json()) as Partial<LeaseRecord>
if (
typeof record.operationId !== 'string' ||
!/^[a-f0-9]{64}$/.test(record.requestDigest ?? '') ||
!/^[a-f0-9]{40}$/.test(record.imageCommit ?? '') ||
!Number.isSafeInteger(record.acquiredAt) ||
!Number.isSafeInteger(record.expiresAt)
) {
throw new Error('existing mutation lease is invalid')
}
return {
metadata: { generation: metadata.generation! },
record: record as LeaseRecord
}
}
private metadataUrl(): string {
return `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(this.bucket)}/o/${encodeURIComponent(this.objectName)}`
}
private uploadUrl(): string {
return `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(this.bucket)}/o?uploadType=media&name=${encodeURIComponent(this.objectName)}`
}
}
@@ -0,0 +1,10 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"declaration": true,
"noEmit": false,
"outDir": "dist",
"rootDir": "src"
},
"exclude": ["src/**/*.test.ts"]
}
@@ -0,0 +1,5 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": { "noEmit": true },
"include": ["src/**/*.ts"]
}
+66
View File
@@ -0,0 +1,66 @@
# Orca Relay Operations
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local `gcloud` and `gh` credentials on the server; credentials and per-user Relay state never enter the browser. One cached `gcloud auth print-access-token` refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
## Run locally
Prerequisites:
- Node 24 and pnpm 10
- `gcloud` authenticated for `onorca-cloud` and `onorca-cloud-staging`
- `gh` authenticated with read access to `stablyai/orca-cloud`
From the repository root:
```sh
pnpm install
pnpm ops:relay
```
Open <http://127.0.0.1:2455>. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
## Share through Tailscale
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
```sh
tailscale serve --bg http://127.0.0.1:2455
tailscale serve status
```
Share the HTTPS URL printed by `tailscale serve status` with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
```sh
tailscale serve reset
```
For a persistent host, run `pnpm --filter @orca-cloud/relay-ops build` and supervise `pnpm --filter @orca-cloud/relay-ops start` with the host's normal process manager. The process needs the same non-interactive `gcloud` and `gh` identities.
## Optional staging controls
Controls are intentionally local-only and disabled unless explicitly enabled:
```sh
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
```
Even in this mode the service never changes GCP directly. It dispatches `.github/workflows/power-relay-staging.yml`, preserves the workflow's typed `WAKE_STAGING` / `SLEEP_STAGING` confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
## Data and security boundaries
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
- Missing cell inventory renders as `Unknown`, never `Sleeping`. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale.
- Responses use `no-store`, a restrictive CSP, frame denial, and no-referrer headers.
## Verification
```sh
pnpm --filter @orca-cloud/relay-ops test
pnpm --filter @orca-cloud/relay-ops typecheck
pnpm --filter @orca-cloud/relay-ops build
```
+29
View File
@@ -0,0 +1,29 @@
{
"name": "@orca-cloud/relay-ops",
"private": true,
"version": "0.0.0",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "pnpm clean && tsc -p tsconfig.build.json && node -e \"require('fs').cpSync('public','dist/public',{recursive:true})\"",
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
"dev": "tsx watch src/index.ts",
"incident:monitor": "tsx src/incident-monitor-cli.ts",
"incident:preflight": "tsx src/incident-live-preflight-cli.ts",
"lint": "tsc -p tsconfig.json --noEmit",
"start": "node dist/index.js",
"test": "vitest run",
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"@hono/node-server": "^1.19.14",
"hono": "^4.12.27",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^24.10.0",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vitest": "^4.0.8"
}
}
+273
View File
@@ -0,0 +1,273 @@
const state = { environment: 'production', window: 360, snapshot: null, config: null, loading: false }
const $ = (selector) => document.querySelector(selector)
const all = (selector) => [...document.querySelectorAll(selector)]
const escapeHtml = (value) => String(value).replace(/[&<>'"]/g, (character) => ({
'&': '&amp;', '<': '&lt;', '>': '&gt;', "'": '&#039;', '"': '&quot;'
})[character])
function formatNumber(value, maximumFractionDigits = 0) {
if (value === null || value === undefined) return '—'
return new Intl.NumberFormat('en-US', { notation: value >= 10_000 ? 'compact' : 'standard', maximumFractionDigits }).format(value)
}
function formatBytes(value) {
if (value === null || value === undefined) return '—'
const units = ['B', 'KB', 'MB', 'GB', 'TB']
let amount = value
let unit = 0
while (Math.abs(amount) >= 1000 && unit < units.length - 1) { amount /= 1000; unit += 1 }
return `${formatNumber(amount, amount < 10 ? 1 : 0)} ${units[unit]}`
}
function formatMetric(metric, value) {
if (value === null || value === undefined) return '—'
if (metric.unit === 'bytes') return formatBytes(value)
if (metric.unit === 'milliseconds') return `${formatNumber(value, 1)} ms`
return formatNumber(value, value < 10 ? 1 : 0)
}
function timeAgo(value) {
const seconds = Math.max(0, Math.round((Date.now() - Date.parse(value)) / 1000))
if (seconds < 60) return `${seconds}s ago`
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ago`
if (seconds < 86400) return `${Math.floor(seconds / 3600)}h ago`
return `${Math.floor(seconds / 86400)}d ago`
}
function shortImage(value) {
const digest = value?.match(/sha256:([a-f0-9]{64})/)?.[1]
if (digest) return digest.slice(0, 10)
const tag = value?.split(':').at(-1)
return tag?.slice(0, 16) ?? '—'
}
function badge(label, healthy) {
return `<span class="badge ${healthy === true ? 'healthy' : healthy === false ? 'unhealthy' : ''}">${escapeHtml(label)}</span>`
}
function renderSummary(snapshot) {
const { summary } = snapshot
const utilization = summary.poweredCapacity === null
? null
: summary.poweredCapacity > 0
? summary.observedConnections / summary.poweredCapacity * 100
: 0
const items = [
['Observed connections', formatNumber(summary.observedConnections, 1), 'Latest 1-minute aggregate mean'],
['Active relay sessions', formatNumber(summary.observedSplices, 1), `${formatNumber(summary.observedControls, 1)} desktop-control mean`],
['Healthy cells', `${summary.activeCells ?? '—'} / ${summary.totalCells}`, summary.poweredCapacity === null ? 'Cell inventory unavailable' : `${formatNumber(summary.poweredCapacity)} powered request units`],
['Capacity signal', utilization === null ? '—' : `${formatNumber(utilization, 2)}%`, `${formatNumber(summary.configuredCapacity)} configured admission units`]
]
$('#summary').innerHTML = items.map(([label, value, caption]) => `
<article class="metric-card">
<p class="eyebrow">${escapeHtml(label)}</p>
<div class="value">${escapeHtml(value)}</div>
<div class="caption">${escapeHtml(caption)}</div>
</article>`).join('')
}
function cellState(cell) {
if (cell.targetSize === null) return ['Unknown', null]
if (cell.targetSize === 0) return ['Sleeping', null]
if (cell.backendHealth === 'healthy' && cell.endpoint.ready) return ['Healthy', true]
if (cell.stable && cell.backendHealth === 'empty') return ['Starting', null]
return ['Attention', false]
}
function renderTopology(snapshot) {
const director = snapshot.resources.director
const sleeping = snapshot.resources.cells.every((cell) => cell.targetSize === 0)
&& snapshot.resources.sql?.activationPolicy === 'NEVER'
const directorHealthy = director?.ready && snapshot.resources.directorEndpoint.health
$('#topology-meta').textContent = `${snapshot.environment.project} · ${snapshot.environment.region}`
const cellNodes = snapshot.resources.cells.map((cell) => {
const [label, healthy] = cellState(cell)
return `<div class="topology-node">
<strong>${escapeHtml(cell.hostname.toUpperCase())} ${badge(label, healthy)}</strong>
<span>${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)} · ${formatNumber(cell.capacityRequests)} units</span>
</div>`
}).join('')
$('#topology').innerHTML = `
<div class="topology-node"><strong>Desktop + phone</strong><span>Encrypted Relay traffic</span></div>
<div class="connector" aria-hidden="true"></div>
<div class="topology-node"><strong>Director ${badge(sleeping ? 'Sleeping' : directorHealthy ? 'Ready' : 'Attention', sleeping ? null : Boolean(directorHealthy))}</strong><span>${escapeHtml(snapshot.environment.directorOrigin)}</span></div>
<div class="connector" aria-hidden="true"></div>
<div class="topology-cells">${cellNodes}</div>`
}
function chartPaths(points, width = 400, height = 112) {
if (points.length === 0) return null
const values = points.map((point) => point.value)
const minimum = Math.min(0, ...values)
const maximum = Math.max(...values)
const spread = maximum - minimum || 1
const coordinates = points.map((point, index) => {
const x = points.length === 1 ? width : index / (points.length - 1) * width
const y = height - ((point.value - minimum) / spread * (height - 10) + 5)
return [x, y]
})
const line = coordinates.map(([x, y], index) => `${index === 0 ? 'M' : 'L'}${x.toFixed(1)},${y.toFixed(1)}`).join(' ')
const area = `${line} L${width},${height} L0,${height} Z`
return { line, area }
}
function renderCharts(snapshot) {
const names = ['controls', 'splices', 'assignment_5xx', 'postgres_retries', 'auth_failures', 'event_loop_ms_p99']
$('#charts').innerHTML = names.map((name) => {
const metric = snapshot.monitoring.metrics[name]
const paths = chartPaths(metric.points)
const graph = paths ? `<svg class="chart" viewBox="0 0 400 112" preserveAspectRatio="none" role="img" aria-label="${escapeHtml(metric.label)} over time">
<line x1="0" y1="111" x2="400" y2="111"></line>
<path class="area" d="${paths.area}"></path><path class="line" d="${paths.line}"></path>
</svg>` : '<div class="chart-empty">No samples in this window</div>'
return `<article class="panel chart-card"><div class="chart-head"><div><p class="eyebrow">${escapeHtml(metric.label)}</p><div class="chart-value">${escapeHtml(formatMetric(metric, metric.latest))}</div></div><span class="chart-unit">${escapeHtml(metric.unit)}</span></div>${graph}</article>`
}).join('')
}
function renderCells(snapshot) {
const controlsByCell = snapshot.monitoring.metrics.controls.latestByCell
const splicesByCell = snapshot.monitoring.metrics.splices.latestByCell
$('#cells').innerHTML = snapshot.resources.cells.map((cell) => {
const [label, healthy] = cellState(cell)
const observed = (controlsByCell[cell.cellId] ?? 0) + (splicesByCell[cell.cellId] ?? 0)
return `<tr>
<td><strong>${escapeHtml(cell.hostname.toUpperCase())}</strong><div class="row-caption">${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)}</div></td>
<td>${badge(label, healthy)}<div class="row-caption">MIG ${cell.runningInstances ?? '—'}/${cell.targetSize ?? '—'}</div></td>
<td>${formatNumber(observed, 1)}<div class="row-caption">1-minute mean</div></td>
<td>${formatNumber(cell.capacityRequests)}<div class="row-caption">DB pool ${formatNumber(cell.databasePoolMax)} · ${cell.configuredAdmission ? 'configured' : 'candidate only'}</div></td>
<td class="mono">${escapeHtml(shortImage(cell.imageDigest))}</td>
</tr>`
}).join('')
}
function serviceRow(name, healthy, detail, suffix = '') {
return `<div class="service-row"><div><div class="row-title">${escapeHtml(name)}</div><div class="row-caption">${escapeHtml(detail)}</div></div>${badge(suffix || (healthy ? 'Ready' : 'Attention'), healthy)}</div>`
}
function renderServices(snapshot) {
const { resources } = snapshot
const sleeping = resources.cells.every((cell) => cell.targetSize === 0)
&& resources.sql?.activationPolicy === 'NEVER'
const certDays = resources.certificate?.expireTime
? Math.floor((Date.parse(resources.certificate.expireTime) - Date.now()) / 86400000)
: null
$('#services').innerHTML = [
serviceRow('Director', sleeping ? null : Boolean(resources.director?.ready && resources.directorEndpoint.health), `${resources.director?.revision ?? 'Revision unavailable'} · ${shortImage(resources.director?.image)}`, sleeping ? 'Sleeping' : ''),
serviceRow('Authentication', sleeping ? null : Boolean(resources.auth?.ready && resources.authEndpoint.health), `${resources.auth?.revision ?? 'Revision unavailable'} · ${shortImage(resources.auth?.image)}`, sleeping ? 'Sleeping' : ''),
serviceRow('Cloud SQL', resources.sql?.state === 'RUNNABLE' || resources.sql?.activationPolicy === 'NEVER', `${resources.sql?.tier ?? 'unknown'} · ${resources.sql?.activationPolicy ?? 'unknown'}`, resources.sql?.state ?? 'Unknown'),
serviceRow('Wildcard TLS', resources.certificate?.state === 'ACTIVE', resources.certificate?.domains.join(', ') ?? 'Certificate unavailable', certDays === null ? 'Unknown' : `${certDays}d`)
].join('')
}
function renderAlerts(snapshot) {
const policies = snapshot.monitoring.alertPolicies
$('#alerts').innerHTML = policies.length ? policies.map((policy) => `<div class="list-row"><div><div class="row-title">${escapeHtml(policy.displayName.replace('Orca Relay: ', ''))}</div><div class="row-caption">Cloud Monitoring policy</div></div>${badge(policy.enabled ? 'Enabled' : 'Disabled', policy.enabled)}</div>`).join('') : '<p class="muted">No Relay alert policies returned.</p>'
}
function renderWorkflows(snapshot) {
$('#workflows').innerHTML = snapshot.workflows.length ? snapshot.workflows.slice(0, 6).map((run) => {
const healthy = run.conclusion === 'success'
const stateLabel = run.status === 'completed' ? (run.conclusion ?? 'completed') : run.status
return `<a class="list-row" href="${escapeHtml(run.url)}" target="_blank" rel="noreferrer"><div><div class="row-title">${escapeHtml(run.name)}</div><div class="row-caption">${escapeHtml(run.headSha)} · ${timeAgo(run.updatedAt)}</div></div>${badge(stateLabel, healthy)}</a>`
}).join('') : '<p class="muted">Workflow history unavailable.</p>'
}
function renderCost(snapshot) {
const cost = snapshot.cost
$('#cost').innerHTML = `<div class="cost-total"><strong>$${formatNumber(cost.monthlyUsd)}</strong><span class="muted">/ month</span></div>
<p class="row-caption">Modeled range $${formatNumber(cost.rangeUsd[0])}–$${formatNumber(cost.rangeUsd[1])}. Not billed cost.</p>
<div class="cost-lines">${cost.lines.map((line) => `<div class="cost-line"><span>${escapeHtml(line.label)}</span><strong>$${formatNumber(line.monthlyUsd, 2)}</strong></div>`).join('')}</div>
<p class="cost-note"><strong>Exact billing:</strong> ${escapeHtml(cost.actualBilling.reason)}<br>${escapeHtml(cost.caveats[1])}</p>`
}
function renderWarnings(snapshot) {
const warning = $('#warnings')
warning.classList.toggle('hidden', snapshot.warnings.length === 0)
warning.textContent = snapshot.warnings.length ? `Partial data: ${snapshot.warnings.join(' ')}` : ''
}
function render(snapshot) {
state.snapshot = snapshot
$('#environment-label').textContent = `${snapshot.environment.label} · ${snapshot.environment.region}`
$('#freshness').textContent = snapshot.stale
? `Last good update ${timeAgo(snapshot.generatedAt)} · refresh degraded`
: `Updated ${timeAgo(snapshot.generatedAt)}`
$('#freshness-dot').className = snapshot.stale ? 'status-dot neutral' : 'status-dot healthy'
renderWarnings(snapshot)
renderSummary(snapshot)
renderTopology(snapshot)
renderCharts(snapshot)
renderCells(snapshot)
renderServices(snapshot)
renderAlerts(snapshot)
renderWorkflows(snapshot)
renderCost(snapshot)
$('#power-panel').classList.toggle('hidden', !(state.environment === 'staging' && state.config?.stagingControlsEnabled))
$('#compute-link').href = snapshot.environment.consoleLinks.compute
$('#alert-link').href = snapshot.environment.consoleLinks.alerts
}
async function loadSnapshot() {
if (state.loading) return
state.loading = true
$('#refresh').disabled = true
$('#error').classList.add('hidden')
$('#freshness-dot').className = 'status-dot neutral'
$('#freshness').textContent = 'Refreshing current state…'
try {
const response = await fetch(`/api/snapshot?environment=${state.environment}&window=${state.window}`)
const body = await response.json()
if (!response.ok) throw new Error(body.error ?? 'Snapshot failed')
render(body)
} catch (error) {
$('#freshness-dot').className = 'status-dot unhealthy'
$('#freshness').textContent = 'Refresh failed'
$('#error').textContent = error instanceof Error ? error.message : 'Relay operations data is unavailable.'
$('#error').classList.remove('hidden')
} finally {
state.loading = false
$('#refresh').disabled = false
}
}
async function dispatchPower(mode) {
const confirmation = mode === 'wake' ? 'WAKE_STAGING' : mode === 'sleep' ? 'SLEEP_STAGING' : ''
if (confirmation) {
const entered = window.prompt(`Type ${confirmation} to dispatch the guarded workflow.`) ?? ''
if (entered !== confirmation) return
}
all('[data-power]').forEach((button) => { button.disabled = true })
$('#power-result').textContent = 'Dispatching…'
try {
const response = await fetch('/api/staging/power', {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-csrf-token': state.config.csrfToken },
body: JSON.stringify({ mode, confirmation })
})
const body = await response.json()
if (!response.ok) throw new Error(body.error)
$('#power-result').textContent = 'Workflow accepted. Refresh after it completes.'
} catch (error) {
$('#power-result').textContent = error instanceof Error ? error.message : 'Dispatch failed.'
} finally {
all('[data-power]').forEach((button) => { button.disabled = false })
}
}
all('[data-environment]').forEach((button) => button.addEventListener('click', () => {
state.environment = button.dataset.environment
all('[data-environment]').forEach((candidate) => candidate.setAttribute('aria-pressed', String(candidate === button)))
loadSnapshot()
}))
$('#window').addEventListener('change', (event) => { state.window = Number(event.target.value); loadSnapshot() })
$('#refresh').addEventListener('click', loadSnapshot)
all('[data-power]').forEach((button) => button.addEventListener('click', () => dispatchPower(button.dataset.power)))
async function start() {
try { state.config = await fetch('/api/config').then((response) => response.json()) } catch { state.config = {} }
await loadSnapshot()
window.setInterval(loadSnapshot, 60_000)
}
start()
+115
View File
@@ -0,0 +1,115 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="color-scheme" content="light dark" />
<title>Orca Relay Operations</title>
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='9' fill='%23171717'/%3E%3Ctext x='16' y='22' text-anchor='middle' font-family='sans-serif' font-weight='700' font-size='17' fill='white'%3EO%3C/text%3E%3C/svg%3E" />
<link rel="stylesheet" href="/styles.css" />
<script type="module" src="/app.js"></script>
</head>
<body>
<header class="topbar">
<div class="brand">
<span class="brand-mark" aria-hidden="true">O</span>
<div>
<strong>Orca Relay</strong>
<span>Operations</span>
</div>
</div>
<div class="toolbar">
<div class="segmented" aria-label="Environment">
<button type="button" data-environment="production" aria-pressed="true">Production</button>
<button type="button" data-environment="staging" aria-pressed="false">Staging</button>
</div>
<select id="window" aria-label="Metrics window">
<option value="60">Last hour</option>
<option value="360" selected>Last 6 hours</option>
<option value="1440">Last 24 hours</option>
</select>
<button type="button" id="refresh" class="button secondary">Refresh</button>
</div>
</header>
<main>
<section class="page-heading">
<div>
<p class="eyebrow" id="environment-label">Production · us-central1</p>
<h1>Relay data plane</h1>
<p class="lede">Private, aggregate operations view. No account, host, device, or pairing identifiers.</p>
</div>
<div class="freshness">
<span class="status-dot neutral" id="freshness-dot"></span>
<span id="freshness">Loading current state…</span>
</div>
</section>
<div id="error" class="banner error hidden" role="alert"></div>
<div id="warnings" class="banner warning hidden"></div>
<section class="summary-grid" id="summary" aria-label="Relay summary">
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
</section>
<section class="panel topology-panel">
<div class="panel-heading">
<div><p class="eyebrow">Topology</p><h2>Request path</h2></div>
<span class="meta" id="topology-meta"></span>
</div>
<div class="topology" id="topology"></div>
</section>
<section class="section-heading"><div><p class="eyebrow">Traffic</p><h2>Signals in selected window</h2></div></section>
<section class="chart-grid" id="charts"></section>
<section class="two-column">
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Compute</p><h2>Relay cells</h2></div><a class="meta panel-link" id="compute-link" target="_blank" rel="noreferrer">Open in GCP ↗</a></div>
<p class="panel-note">Configured admission is shown below. Live director admission and heartbeat state stays unavailable because this dashboard has no Relay-admin identity.</p>
<div class="table-wrap"><table>
<thead><tr><th>Cell</th><th>State</th><th>Observed</th><th>Capacity</th><th>Image</th></tr></thead>
<tbody id="cells"></tbody>
</table></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Control plane</p><h2>Services</h2></div></div>
<div class="service-list" id="services"></div>
</article>
</section>
<section class="three-column">
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Guardrails</p><h2>Alert policies</h2></div><a class="meta panel-link" id="alert-link" target="_blank" rel="noreferrer">Open incidents ↗</a></div>
<div class="list" id="alerts"></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Delivery</p><h2>Recent workflows</h2></div></div>
<div class="list" id="workflows"></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Planning</p><h2>Monthly run-rate</h2></div></div>
<div id="cost"></div>
</article>
</section>
<section class="panel hidden" id="power-panel">
<div class="panel-heading">
<div><p class="eyebrow">Staging only</p><h2>Power workflow</h2></div>
<span class="badge">GitHub guarded</span>
</div>
<p class="muted">Dispatches the reviewed GitHub workflow. It never changes GCP resources directly from this process.</p>
<div class="power-actions">
<button class="button secondary" type="button" data-power="status">Check status</button>
<button class="button secondary" type="button" data-power="wake">Wake configured cells</button>
<button class="button secondary" type="button" data-power="sleep">Sleep staging</button>
<span id="power-result" class="meta"></span>
</div>
</section>
</main>
<footer>Orca Relay Operations · Server-side credentials · Read-only by default</footer>
</body>
</html>
+165
View File
@@ -0,0 +1,165 @@
:root {
--radius: 10px;
--background: #fff;
--foreground: #0a0a0a;
--card: #fff;
--primary: #171717;
--primary-foreground: #fafafa;
--secondary: #f5f5f5;
--muted: #f5f5f5;
--muted-foreground: #737373;
--accent: #f5f5f5;
--border: #e5e5e5;
--ring: #a1a1a1;
--destructive: #e40014;
--success: #15803d;
--warning: #895503;
--font-mono: 'SF Mono', SFMono-Regular, ui-monospace, Menlo, Consolas, monospace;
}
@media (prefers-color-scheme: dark) {
:root {
--background: #0a0a0a;
--foreground: #fafafa;
--card: #171717;
--primary: #e5e5e5;
--primary-foreground: #171717;
--secondary: #262626;
--muted: #262626;
--muted-foreground: #a1a1a1;
--accent: #262626;
--border: rgb(255 255 255 / 0.07);
--ring: #737373;
--destructive: #ff6568;
--success: #4ade80;
--warning: #fbbf24;
}
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--background);
color: var(--foreground);
font-family: 'Geist', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
font-size: 14px;
letter-spacing: .01em;
}
button, select { font: inherit; }
button:focus-visible, select:focus-visible { outline: 2px solid var(--ring); outline-offset: 2px; }
.topbar {
height: 64px;
border-bottom: 1px solid var(--border);
padding: 0 max(24px, calc((100vw - 1440px) / 2));
display: flex;
align-items: center;
justify-content: space-between;
position: sticky;
top: 0;
background: color-mix(in srgb, var(--background) 92%, transparent);
backdrop-filter: blur(16px);
z-index: 10;
}
.brand { display: flex; align-items: center; gap: 10px; }
.brand-mark {
width: 30px; height: 30px; border-radius: 9px; background: var(--primary);
color: var(--primary-foreground); display: grid; place-items: center; font-weight: 700;
}
.brand div { display: flex; align-items: baseline; gap: 7px; }
.brand span:last-child { color: var(--muted-foreground); font-size: 12px; }
.toolbar { display: flex; align-items: center; gap: 8px; }
.segmented { display: flex; padding: 3px; gap: 2px; border-radius: 8px; background: var(--secondary); }
.segmented button { border: 0; background: transparent; color: var(--muted-foreground); padding: 5px 10px; border-radius: 6px; cursor: pointer; }
.segmented button[aria-pressed="true"] { background: var(--card); color: var(--foreground); box-shadow: 0 1px 2px rgb(0 0 0 / .08); }
select, .button { height: 32px; border: 1px solid var(--border); border-radius: 8px; background: var(--card); color: var(--foreground); padding: 0 10px; }
.button { cursor: pointer; font-weight: 500; }
.button:hover { background: var(--accent); }
.button:disabled { opacity: .5; cursor: wait; }
main { max-width: 1440px; margin: 0 auto; padding: 42px 24px 72px; }
.page-heading { display: flex; justify-content: space-between; align-items: end; gap: 24px; margin-bottom: 28px; }
h1, h2, p { margin: 0; }
h1 { font-size: 28px; line-height: 1.2; letter-spacing: -.025em; margin-top: 5px; }
h2 { font-size: 16px; line-height: 1.25; letter-spacing: -.01em; margin-top: 3px; }
.lede { color: var(--muted-foreground); margin-top: 8px; max-width: 680px; }
.eyebrow { color: var(--muted-foreground); font-size: 11px; line-height: 1; font-weight: 600; text-transform: uppercase; letter-spacing: .05em; }
.freshness { color: var(--muted-foreground); display: flex; align-items: center; gap: 8px; font-size: 12px; white-space: nowrap; }
.status-dot { width: 7px; height: 7px; border-radius: 999px; background: var(--ring); display: inline-block; }
.status-dot.healthy { background: var(--success); }
.status-dot.unhealthy { background: var(--destructive); }
.summary-grid { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
.metric-card, .panel { border: 1px solid var(--border); border-radius: 14px; background: var(--card); }
.metric-card { padding: 18px; min-height: 116px; }
.metric-card .value { font-size: 29px; font-weight: 600; letter-spacing: -.035em; margin-top: 16px; }
.metric-card .caption { color: var(--muted-foreground); font-size: 12px; margin-top: 3px; }
.skeleton { background: linear-gradient(90deg, var(--card), var(--muted), var(--card)); background-size: 200% 100%; animation: shimmer 1.6s infinite; }
@keyframes shimmer { to { background-position: -200% 0; } }
.panel { padding: 18px; }
.topology-panel { margin-bottom: 36px; }
.panel-heading, .section-heading { display: flex; align-items: center; justify-content: space-between; gap: 16px; margin-bottom: 18px; }
.section-heading { margin-top: 34px; }
.topology { display: grid; grid-template-columns: 1fr 48px 1fr 48px 2fr; align-items: stretch; gap: 8px; }
.topology-node { border: 1px solid var(--border); background: var(--background); border-radius: 10px; padding: 14px; min-width: 0; }
.topology-node strong { display: block; font-size: 13px; }
.topology-node span { display: block; color: var(--muted-foreground); font-size: 12px; margin-top: 4px; overflow: hidden; text-overflow: ellipsis; }
.topology-cells { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 8px; }
.connector { display: grid; place-items: center; color: var(--muted-foreground); }
.connector::before { content: ''; width: 100%; height: 1px; background: var(--border); }
.chart-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 36px; }
.chart-card { min-height: 220px; }
.chart-head { display: flex; justify-content: space-between; align-items: start; }
.chart-value { font-size: 22px; font-weight: 600; letter-spacing: -.03em; }
.chart-unit { color: var(--muted-foreground); font-size: 11px; }
.chart { width: 100%; height: 122px; margin-top: 18px; overflow: visible; }
.chart path.area { fill: color-mix(in srgb, var(--foreground) 5%, transparent); }
.chart path.line { fill: none; stroke: var(--foreground); stroke-width: 1.5; vector-effect: non-scaling-stroke; }
.chart line { stroke: var(--border); stroke-width: 1; }
.chart-empty { color: var(--muted-foreground); height: 120px; display: grid; place-items: center; font-size: 12px; }
.two-column { display: grid; grid-template-columns: 1.7fr 1fr; gap: 12px; margin-bottom: 12px; }
.three-column { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
.two-column > *, .three-column > *, .chart-grid > * { min-width: 0; }
.table-wrap { overflow-x: auto; }
table { width: 100%; border-collapse: collapse; font-size: 12px; }
th { color: var(--muted-foreground); font-weight: 500; text-align: left; padding: 0 10px 10px; }
td { padding: 12px 10px; border-top: 1px solid var(--border); vertical-align: middle; }
td:first-child, th:first-child { padding-left: 0; }
td:last-child, th:last-child { padding-right: 0; }
.mono { font-family: var(--font-mono); font-size: 11px; }
.badge { display: inline-flex; align-items: center; border: 1px solid var(--border); border-radius: 999px; padding: 2px 7px; font-size: 11px; color: var(--muted-foreground); }
.badge.healthy { color: var(--success); border-color: color-mix(in srgb, var(--success) 25%, var(--border)); }
.badge.unhealthy { color: var(--destructive); border-color: color-mix(in srgb, var(--destructive) 25%, var(--border)); }
.service-list, .list { display: grid; }
.service-row, .list-row { padding: 11px 0; border-top: 1px solid var(--border); display: flex; align-items: center; justify-content: space-between; gap: 12px; min-width: 0; }
.service-row:first-child, .list-row:first-child { border-top: 0; padding-top: 0; }
.service-row:last-child, .list-row:last-child { padding-bottom: 0; }
.row-title { font-size: 12px; font-weight: 500; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row-caption, .meta, .muted { color: var(--muted-foreground); font-size: 11px; }
.panel-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.45; margin: -8px 0 14px; }
.panel-link:hover { color: var(--foreground); text-decoration: underline; }
.row-caption { margin-top: 3px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
a { color: inherit; text-decoration: none; }
a:hover .row-title { text-decoration: underline; }
.cost-total { display: flex; align-items: baseline; gap: 7px; margin-bottom: 12px; }
.cost-total strong { font-size: 28px; letter-spacing: -.035em; }
.cost-line { display: flex; justify-content: space-between; gap: 10px; padding: 7px 0; border-top: 1px solid var(--border); font-size: 12px; }
.cost-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.5; margin-top: 12px; }
.banner { border: 1px solid var(--border); border-radius: 10px; padding: 11px 13px; margin-bottom: 12px; font-size: 12px; }
.banner.error { color: var(--destructive); }
.banner.warning { color: var(--warning); }
.hidden { display: none !important; }
.power-actions { display: flex; align-items: center; gap: 8px; margin-top: 14px; flex-wrap: wrap; }
footer { border-top: 1px solid var(--border); padding: 24px; color: var(--muted-foreground); font-size: 11px; text-align: center; }
@media (max-width: 1050px) {
.summary-grid, .chart-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
.three-column { grid-template-columns: 1fr; }
.topology { grid-template-columns: 1fr; }
.connector { height: 20px; }
.connector::before { height: 100%; width: 1px; }
}
@media (max-width: 760px) {
.topbar { height: auto; min-height: 64px; padding: 12px 16px; align-items: stretch; flex-direction: column; gap: 12px; }
.brand div { display: grid; gap: 0; }
.toolbar { flex-wrap: wrap; justify-content: flex-start; }
main { padding: 28px 16px 56px; }
.page-heading { align-items: flex-start; flex-direction: column; }
.summary-grid, .chart-grid, .two-column { grid-template-columns: 1fr; }
.topology-cells { grid-template-columns: 1fr; }
table { min-width: 600px; }
}
@@ -0,0 +1,85 @@
import { describe, expect, it } from 'vitest'
import { buildCostModel } from './cost-model.js'
import {
RELAY_OPS_ENVIRONMENTS,
relayOpsCellsFromTerraform,
type RelayOpsCellConfig
} from './environment-config.js'
import type { ResourceInventory } from './resource-inventory.js'
const regionalCellsSource = `
relay_gce_cells = {
"staging-gce-c3" = {
hostname = "c3"
zone = "us-central1-a"
machine_type = "e2-standard-2"
capacity_requests = 4000
}
"staging-gce-c4" = {
hostname = "c4"
region = "asia-east2"
zone = "asia-east2-a"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
}
`
function inventory(
targetSize: number,
activationPolicy: string,
cells: RelayOpsCellConfig[] = RELAY_OPS_ENVIRONMENTS.staging.cells
): ResourceInventory {
return {
director: null,
auth: null,
sql: { state: targetSize ? 'RUNNABLE' : 'STOPPED', activationPolicy, tier: 'db-custom-1-3840', availabilityType: 'ZONAL', databaseVersion: 'POSTGRES_17' },
certificate: null,
directorEndpoint: { health: null, ready: null, latencyMs: null },
authEndpoint: { health: null, ready: null, latencyMs: null },
cells: cells.map((cell) => ({
...cell, migName: `mig-${cell.hostname}`, targetSize, runningInstances: targetSize,
stable: true, template: 'template', imageDigest: null,
backendHealth: targetSize ? 'healthy' : 'empty',
endpoint: { health: null, ready: null, latencyMs: null }
})),
warnings: []
}
}
describe('buildCostModel', () => {
it('shows the sleeping staging floor without VM or SQL compute', () => {
const result = buildCostModel(RELAY_OPS_ENVIRONMENTS.staging, inventory(0, 'NEVER'))
expect(result.kind).toBe('planning-estimate')
expect(result.monthlyUsd).toBe(34)
expect(result.lines.find((line) => line.label === 'Relay cell VMs')?.monthlyUsd).toBe(0)
expect(result.actualBilling.available).toBe(false)
expect(result.caveats[0]).toContain('not the Cloud Billing invoice')
})
it('prices durable machine inventory and network floors by region', () => {
const cells = relayOpsCellsFromTerraform({
environment: 'staging',
domain: 'relay-staging.onorca.dev',
source: regionalCellsSource
})
const environment = { ...RELAY_OPS_ENVIRONMENTS.staging, cells }
const result = buildCostModel(environment, inventory(1, 'NEVER', cells))
const machines = result.lines.find((line) => line.label === 'Relay cell VMs')
const network = result.lines.find((line) => line.label === 'Load balancer and network floor')
expect(machines).toEqual({
label: 'Relay cell VMs',
monthlyUsd: 185.79,
basis: '2 configured VM cells at regional machine rates × 730 hours'
})
expect(network).toEqual({
label: 'Load balancer and network floor',
monthlyUsd: 29,
basis: 'shared HTTPS foundation plus NAT floor in 2 configured regions'
})
})
})
+115
View File
@@ -0,0 +1,115 @@
import type {
RelayOpsEnvironment,
RelayOpsMachineType,
RelayOpsRegion
} from './environment-config.js'
import type { ResourceInventory } from './resource-inventory.js'
export type CostLine = {
label: string
monthlyUsd: number
basis: string
}
export type CostModel = {
kind: 'planning-estimate'
monthlyUsd: number
rangeUsd: [number, number]
actualBilling: { available: false; reason: string }
lines: CostLine[]
caveats: string[]
}
const HOURS_PER_MONTH = 730
const MACHINE_HOURLY_USD: Record<
RelayOpsRegion,
Record<RelayOpsMachineType, number>
> = {
'us-central1': {
'e2-standard-2': 0.06701142,
'e2-standard-4': 0.13402284
},
'asia-east2': {
'e2-standard-2': 0.0938,
'e2-standard-4': 0.1875
}
}
const SHARED_NETWORK_FLOOR_USD = 19
const REGIONAL_NAT_FLOOR_USD = 5
function round(value: number): number {
return Math.round(value * 100) / 100
}
export function buildCostModel(
environment: RelayOpsEnvironment,
resources: ResourceInventory
): CostModel {
const runningCells = resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0)
const runningCellCount = runningCells.reduce((sum, cell) => sum + (cell.targetSize ?? 0), 0)
const compute = runningCells.reduce(
(sum, cell) =>
sum + (cell.targetSize ?? 0) * MACHINE_HOURLY_USD[cell.region][cell.machineType],
0
) * HOURS_PER_MONTH
const disks = runningCellCount * 30 * 0.1
const sqlRunning = resources.sql?.activationPolicy === 'ALWAYS'
const sql = sqlRunning ? (environment.id === 'production' ? 105 : 52) : 0
const cloudRunMinimums =
(resources.director?.minInstances ?? 0) + (resources.auth?.minInstances ?? 0)
const cloudRun = cloudRunMinimums * 10
const configuredRegions = new Set(
(resources.cells.length > 0 ? resources.cells : environment.cells).map((cell) => cell.region)
)
const networkFoundation =
SHARED_NETWORK_FLOOR_USD + configuredRegions.size * REGIONAL_NAT_FLOOR_USD
const observability = environment.id === 'production' ? 12 : 5
const lines: CostLine[] = [
{
label: 'Relay cell VMs',
monthlyUsd: round(compute),
basis: `${runningCellCount} configured VM cells at regional machine rates × 730 hours`
},
{
label: 'Cell boot disks',
monthlyUsd: round(disks),
basis: `${runningCellCount} × 30 GB balanced persistent disk`
},
{
label: 'Cloud SQL',
monthlyUsd: sql,
basis: sqlRunning ? `${resources.sql?.tier ?? 'configured tier'} active` : 'stopped'
},
{
label: 'Cloud Run minimums',
monthlyUsd: cloudRun,
basis: `${cloudRunMinimums} configured minimum instances`
},
{
label: 'Load balancer and network floor',
monthlyUsd: networkFoundation,
basis: `shared HTTPS foundation plus NAT floor in ${configuredRegions.size} configured region${configuredRegions.size === 1 ? '' : 's'}`
},
{
label: 'Logs and monitoring allowance',
monthlyUsd: observability,
basis: 'planning allowance; varies with traffic and retention'
}
]
const monthlyUsd = round(lines.reduce((sum, line) => sum + line.monthlyUsd, 0))
return {
kind: 'planning-estimate',
monthlyUsd,
rangeUsd: [round(monthlyUsd * 0.85), round(monthlyUsd * 1.3)],
actualBilling: {
available: false,
reason: 'Cloud Billing export is not configured in either Relay project.'
},
lines,
caveats: [
'This is a modeled run-rate, not the Cloud Billing invoice.',
'Network egress, actual request traffic, credits, discounts, taxes, and free tiers are excluded.',
'Use the GCP Billing report for authoritative spend and forecasts.'
]
}
}
@@ -0,0 +1,62 @@
import { describe, expect, it } from 'vitest'
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
import type { DashboardSnapshot } from './dashboard-snapshot.js'
import type { GcloudClient } from './gcloud-client.js'
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
function snapshot(kind: 'good' | 'unavailable'): DashboardSnapshot {
const good = kind === 'good'
return {
generatedAt: '2026-07-15T12:00:00.000Z',
resources: {
director: good ? {} : null,
auth: good ? {} : null,
sql: good ? {} : null,
cells: [{ targetSize: good ? 1 : null }]
},
monitoring: {
warnings: good
? []
: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
},
summary: { observedConnections: good ? 7 : 0 },
warnings: good ? [] : ['Google Cloud credentials are unavailable. Run gcloud auth login.'],
stale: false,
staleReason: null
} as unknown as DashboardSnapshot
}
describe('DashboardSnapshotCache', () => {
it('keeps the last good view when a later credential refresh fails', async () => {
let calls = 0
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
calls += 1
return snapshot(calls === 1 ? 'good' : 'unavailable')
})
const first = await cache.read('production', 30)
const second = await cache.read('production', 31)
expect(first.stale).toBe(false)
expect(second.stale).toBe(true)
expect(second.summary.observedConnections).toBe(7)
expect(second.staleReason).toContain('credentials')
})
it('keeps the last good view when a later collector throws', async () => {
let calls = 0
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
calls += 1
if (calls > 1) throw new Error('sensitive collector context')
return snapshot('good')
})
await cache.read('production', 30)
const second = await cache.read('production', 31)
expect(second.stale).toBe(true)
expect(second.summary.observedConnections).toBe(7)
expect(JSON.stringify(second)).not.toContain('sensitive collector context')
})
})
@@ -0,0 +1,167 @@
import type { RelayOpsEnvironmentId } from './environment-config.js'
import { relayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { readRelayWorkflowRuns } from './github-runs.js'
import { readMonitoringSnapshot } from './monitoring-snapshot.js'
import { readResourceInventory } from './resource-inventory.js'
import { buildCostModel } from './cost-model.js'
export type DashboardSnapshot = Awaited<ReturnType<typeof buildDashboardSnapshot>>
export async function buildDashboardSnapshot(
environmentId: RelayOpsEnvironmentId,
gcloud: GcloudClient,
options: { windowMinutes?: number; fetchImpl?: typeof fetch; now?: Date } = {}
) {
const generatedAt = (options.now ?? new Date()).toISOString()
const environment = relayOpsEnvironment(environmentId)
const [monitoringResult, resourceResult, workflowResult] = await Promise.allSettled([
readMonitoringSnapshot(environment, gcloud, {
...(options.now === undefined ? {} : { now: options.now }),
...(options.windowMinutes === undefined ? {} : { windowMinutes: options.windowMinutes }),
...(options.fetchImpl === undefined ? {} : { fetchImpl: options.fetchImpl })
}),
readResourceInventory(environment, gcloud, options.fetchImpl),
readRelayWorkflowRuns()
])
if (monitoringResult.status === 'rejected' || resourceResult.status === 'rejected') {
const failed = [
monitoringResult.status === 'rejected' ? 'Monitoring snapshot' : null,
resourceResult.status === 'rejected' ? 'Resource inventory' : null
].filter(Boolean)
throw new Error(`${failed.join(' and ')} unavailable`)
}
const resources = resourceResult.value
const monitoring = monitoringResult.value
const warnings = [...resources.warnings, ...monitoring.warnings]
const poweredDigests = new Set(
resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0).map((cell) => cell.imageDigest)
)
if (poweredDigests.has(null)) warnings.push('A powered cell image digest is unavailable.')
if (poweredDigests.size > 1) warnings.push('Powered cells are not serving one immutable digest.')
const expectedCertificateDomain = `*.${new URL(environment.cells[0]!.origin).hostname
.split('.').slice(1).join('.')}`
if (resources.certificate && !resources.certificate.domains.includes(expectedCertificateDomain)) {
warnings.push('The Relay certificate domain does not match the configured cell domain.')
}
if (workflowResult.status === 'rejected') warnings.push('GitHub workflow history is unavailable.')
const observedConnections = monitoring.metrics.total_connections.latest ?? 0
const observedControls = monitoring.metrics.controls.latest ?? 0
const observedSplices = monitoring.metrics.splices.latest ?? 0
const configuredCapacity = environment.cells
.filter((cell) => cell.configuredAdmission)
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
const cellInventoryAvailable = resources.cells.every((cell) => cell.targetSize !== null)
const poweredCapacity = cellInventoryAvailable
? resources.cells
.filter((cell) => (cell.targetSize ?? 0) > 0)
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
: null
return {
schemaVersion: 1,
generatedAt,
environment: {
id: environment.id,
label: environment.label,
project: environment.project,
region: environment.region,
directorOrigin: environment.directorOrigin,
authOrigin: environment.authOrigin,
consoleLinks: {
project: `https://console.cloud.google.com/home/dashboard?project=${environment.project}`,
alerts: `https://console.cloud.google.com/monitoring/alerting?project=${environment.project}`,
compute: `https://console.cloud.google.com/compute/instanceGroups/list?project=${environment.project}`
}
},
summary: {
observedConnections,
observedControls,
observedSplices,
configuredCapacity,
poweredCapacity,
activeCells: cellInventoryAvailable
? resources.cells.filter(
(cell) => (cell.targetSize ?? 0) > 0 && cell.backendHealth === 'healthy'
).length
: null,
totalCells: resources.cells.length
},
resources,
monitoring,
workflows: workflowResult.status === 'fulfilled' ? workflowResult.value : [],
cost: buildCostModel(environment, resources),
warnings,
stale: false,
staleReason: null as string | null
}
}
type SnapshotCacheEntry = { snapshot: DashboardSnapshot; expiresAt: number }
type SnapshotBuilder = (
environment: RelayOpsEnvironmentId,
gcloud: GcloudClient,
options: { windowMinutes?: number }
) => Promise<DashboardSnapshot>
export class DashboardSnapshotCache {
private readonly entries = new Map<string, SnapshotCacheEntry>()
private readonly pending = new Map<string, Promise<DashboardSnapshot>>()
private readonly lastGood = new Map<RelayOpsEnvironmentId, DashboardSnapshot>()
constructor(
private readonly gcloud: GcloudClient,
private readonly ttlMs = 30_000,
private readonly builder: SnapshotBuilder = buildDashboardSnapshot
) {}
async read(environment: RelayOpsEnvironmentId, windowMinutes: number): Promise<DashboardSnapshot> {
const key = `${environment}:${windowMinutes}`
const cached = this.entries.get(key)
if (cached && cached.expiresAt > Date.now()) return cached.snapshot
const existing = this.pending.get(key)
if (existing) return await existing
const request = this.builder(environment, this.gcloud, { windowMinutes })
.then((snapshot) => {
const coreInventoryUnavailable =
snapshot.resources.director === null &&
snapshot.resources.auth === null &&
snapshot.resources.sql === null &&
snapshot.resources.cells.every((cell) => cell.targetSize === null)
const monitoringCredentialsUnavailable = snapshot.monitoring.warnings.some(
(warning) => warning.includes('credentials are unavailable')
)
const lastGood = this.lastGood.get(environment)
const result = (coreInventoryUnavailable || monitoringCredentialsUnavailable) && lastGood
? {
...lastGood,
stale: true,
staleReason: 'Local Google Cloud credentials are temporarily unavailable.',
warnings: [...new Set([...lastGood.warnings, ...snapshot.warnings])]
}
: snapshot
if (!coreInventoryUnavailable && !monitoringCredentialsUnavailable) {
this.lastGood.set(environment, snapshot)
}
this.entries.set(key, { snapshot: result, expiresAt: Date.now() + this.ttlMs })
return result
})
.catch((error: unknown) => {
const lastGood = this.lastGood.get(environment)
if (!lastGood) throw error
const stale = {
...lastGood,
stale: true,
staleReason: 'The latest operations refresh failed; showing the last good snapshot.',
warnings: [...new Set([
...lastGood.warnings,
'The latest operations refresh failed before a complete snapshot was available.'
])]
}
this.entries.set(key, { snapshot: stale, expiresAt: Date.now() + this.ttlMs })
return stale
})
.finally(() => this.pending.delete(key))
this.pending.set(key, request)
return await request
}
}
@@ -0,0 +1,148 @@
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import {
RELAY_OPS_ENVIRONMENTS,
relayOpsCellsFromTerraform
} from './environment-config.js'
const durableUsCell = `
"production-gce-c26" = {
hostname = "c26"
zone = "us-central1-a"
machine_type = "e2-standard-4"
capacity_requests = 4000
initially_enabled = false
}
`
const durableAsiaCells = `
"production-gce-c27" = {
hostname = "c27"
region = "asia-east2"
zone = "asia-east2-a"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
"production-gce-c28" = {
hostname = "c28"
region = "asia-east2"
zone = "asia-east2-b"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
"production-gce-c29" = {
hostname = "c29"
region = "asia-east2"
zone = "asia-east2-c"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
`
const durableAsiaSource = `
relay_gce_cells = {${durableUsCell}${durableAsiaCells}}
`
const durableUsOnlySource = `
relay_gce_cells = {${durableUsCell}}
`
// Why: relay-ops sat at 18 cells for four days after C19-C22 shipped, which threw
// `selector membership must contain every configured cell exactly once` and blocked
// every production mutation. Reading Terraform here makes that drift fail the build.
function terraformCells(environment: 'production' | 'staging'): Array<{
cellId: string
region: string
zone: string
machineType: string
capacityRequests: number
databasePoolMax: number
}> {
const tfvars = readFileSync(
fileURLToPath(new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url)),
'utf8'
)
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(tfvars)?.[1] ?? ''
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
.map((match) => ({
cellId: match[1] ?? '',
region: /region\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? 'us-central1',
zone: /zone\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
machineType: /machine_type\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
capacityRequests: Number(/capacity_requests\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1]),
databasePoolMax: Number(/database_pool_max\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1] ?? 10)
}))
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
}
const cellOrdinal = (cellId: string): number => Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
describe('relay operations environment config', () => {
it.each(['production', 'staging'] as const)(
'matches the %s cells Terraform actually provisions',
(environment) => {
const expected = terraformCells(environment)
expect(expected.length).toBeGreaterThan(0)
expect(
RELAY_OPS_ENVIRONMENTS[environment].cells.map((cell) => ({
cellId: cell.cellId,
region: cell.region,
zone: cell.zone,
machineType: cell.machineType,
capacityRequests: cell.capacityRequests,
databasePoolMax: cell.databasePoolMax
}))
).toEqual(expected)
}
)
it('derives hostname and origin from the cell ordinal', () => {
const cells = RELAY_OPS_ENVIRONMENTS.production.cells
expect(cells.at(-1)).toMatchObject({
hostname: `c${cells.length}`,
origin: `https://c${cells.length}.relay.onorca.dev`
})
})
it('inventories Asia cells only when they exist in durable Terraform', () => {
const cells = relayOpsCellsFromTerraform({
environment: 'production',
domain: 'relay.onorca.dev',
source: durableAsiaSource
})
expect(cells.slice(1)).toEqual([
{
cellId: 'production-gce-c27', hostname: 'c27', origin: 'https://c27.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-a', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
},
{
cellId: 'production-gce-c28', hostname: 'c28', origin: 'https://c28.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-b', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
},
{
cellId: 'production-gce-c29', hostname: 'c29', origin: 'https://c29.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-c', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
}
])
const usOnlyCells = relayOpsCellsFromTerraform({
environment: 'production',
domain: 'relay.onorca.dev',
source: durableUsOnlySource
})
expect(usOnlyCells).toHaveLength(1)
expect(usOnlyCells.some((cell) => cell.region === 'asia-east2')).toBe(false)
expect(usOnlyCells.some((cell) => cell.cellId === 'production-gce-c27')).toBe(false)
})
})
@@ -0,0 +1,125 @@
import { readFileSync } from 'node:fs'
import { z } from 'zod'
export type RelayOpsEnvironmentId = 'production' | 'staging'
export type RelayOpsRegion = 'us-central1' | 'asia-east2'
export type RelayOpsMachineType = 'e2-standard-2' | 'e2-standard-4'
export type RelayOpsCellConfig = {
cellId: string
hostname: string
origin: string
region: RelayOpsRegion
zone: string
machineType: RelayOpsMachineType
capacityRequests: number
databasePoolMax: number
configuredAdmission: boolean
}
export type RelayOpsEnvironment = {
id: RelayOpsEnvironmentId
label: string
project: string
region: string
directorOrigin: string
authOrigin: string
directorService: string
authService: string
sqlInstance: string
migPrefix: string
certificateName: string
cells: RelayOpsCellConfig[]
}
const RegionSchema = z.enum(['us-central1', 'asia-east2'])
const MachineTypeSchema = z.enum(['e2-standard-2', 'e2-standard-4'])
const EnvironmentSchema = z.enum(['production', 'staging'])
function cellOrdinal(cellId: string): number {
return Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
}
function required(body: string, pattern: RegExp, label: string): string {
const value = pattern.exec(body)?.[1]
if (!value) throw new Error(`Relay Ops could not read ${label} from durable Terraform config`)
return value
}
export function relayOpsCellsFromTerraform(input: {
environment: RelayOpsEnvironmentId
domain: string
source: string
}): RelayOpsCellConfig[] {
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(input.source)?.[1]
if (!block) throw new Error('Relay Ops could not read durable Relay cells')
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
.map((match) => {
const cellId = match[1]!
const body = match[2]!
const hostname = required(body, /\bhostname\s*=\s*"([^"]+)"/, `${cellId} hostname`)
const configuredAdmission = /\binitially_enabled\s*=\s*(true|false)/.exec(body)?.[1]
return {
cellId,
hostname,
origin: `https://${hostname}.${input.domain}`,
region: RegionSchema.parse(
/\bregion\s*=\s*"([^"]+)"/.exec(body)?.[1] ?? 'us-central1'
),
zone: required(body, /\bzone\s*=\s*"([^"]+)"/, `${cellId} zone`),
machineType: MachineTypeSchema.parse(
required(body, /\bmachine_type\s*=\s*"([^"]+)"/, `${cellId} machine type`)
),
capacityRequests: Number(
required(body, /\bcapacity_requests\s*=\s*(\d+)/, `${cellId} capacity`)
),
databasePoolMax: Number(/\bdatabase_pool_max\s*=\s*(\d+)/.exec(body)?.[1] ?? 10),
configuredAdmission: configuredAdmission === undefined || configuredAdmission === 'true'
}
})
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
}
function durableCells(environment: RelayOpsEnvironmentId, domain: string): RelayOpsCellConfig[] {
const source = readFileSync(
// Repository root; infra/terraform moves with this tree, so the relative depth holds.
new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url),
'utf8'
)
return relayOpsCellsFromTerraform({ environment, domain, source })
}
export const RELAY_OPS_ENVIRONMENTS: Record<RelayOpsEnvironmentId, RelayOpsEnvironment> = {
production: {
id: 'production',
label: 'Production',
project: 'onorca-cloud',
region: 'us-central1',
directorOrigin: 'https://relay.onorca.dev',
authOrigin: 'https://login.onorca.dev',
directorService: 'orca-cloud-relay',
authService: 'orca-cloud-auth',
sqlInstance: 'orca-cloud-auth-db',
migPrefix: 'orca-cloud-relay-gce-',
certificateName: 'orca-cloud-relay-gce',
cells: durableCells('production', 'relay.onorca.dev')
},
staging: {
id: 'staging',
label: 'Staging',
project: 'onorca-cloud-staging',
region: 'us-central1',
directorOrigin: 'https://relay-staging.onorca.dev',
authOrigin: 'https://auth-staging.onorca.dev',
directorService: 'orca-cloud-relay-staging',
authService: 'orca-cloud-auth-staging',
sqlInstance: 'orca-cloud-staging-auth-db',
migPrefix: 'orca-cloud-staging-relay-gce-',
certificateName: 'orca-cloud-staging-relay-gce',
cells: durableCells('staging', 'relay-staging.onorca.dev')
}
}
export function relayOpsEnvironment(value: unknown): RelayOpsEnvironment {
return RELAY_OPS_ENVIRONMENTS[EnvironmentSchema.parse(value)]
}
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest'
import { createGcloudClient } from './gcloud-client.js'
describe('createGcloudClient', () => {
it('shares and caches one credential refresh across concurrent readers', async () => {
let calls = 0
const token = 'a'.repeat(40)
const client = createGcloudClient(async () => {
calls += 1
await Promise.resolve()
return token
})
const values = await Promise.all([
client.accessToken(),
client.accessToken(),
client.accessToken()
])
expect(values).toEqual([token, token, token])
expect(await client.accessToken()).toBe(token)
expect(calls).toBe(1)
})
it('caches bounded identity tokens by audience', async () => {
const commands: string[][] = []
const token = 'aaa.bbb.ccc'
const client = createGcloudClient(async (args) => {
commands.push(args)
return token
})
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
expect(commands).toEqual([
[
'auth',
'print-identity-token',
'--audiences=https://relay.example/admin',
'--include-email'
]
])
})
})
+77
View File
@@ -0,0 +1,77 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
const execFileAsync = promisify(execFile)
const TOKEN_PATTERN = /^[A-Za-z0-9._~+\/-]{32,8192}$/
export type GcloudClient = {
accessToken(): Promise<string>
identityToken?(audience: string): Promise<string>
}
export class GcloudCommandError extends Error {
constructor(readonly operation: string) {
super(`${operation} is unavailable`)
}
}
async function runGcloud(args: string[]): Promise<string> {
try {
const result = await execFileAsync('gcloud', args, {
encoding: 'utf8',
timeout: 90_000,
maxBuffer: 8 * 1024 * 1024,
env: {
...process.env,
CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK: '1',
CLOUDSDK_CORE_DISABLE_PROMPTS: '1',
CLOUDSDK_CORE_DISABLE_USAGE_REPORTING: '1'
}
})
return result.stdout.trim()
} catch {
// Gcloud stderr can echo command context; keep dashboard errors intentionally non-sensitive.
throw new GcloudCommandError(`gcloud ${args.slice(0, 3).join(' ')}`)
}
}
export function createGcloudClient(
tokenCommand: (args: string[]) => Promise<string> = runGcloud
): GcloudClient {
let cachedToken: { value: string; expiresAt: number } | null = null
const identityTokens = new Map<string, { value: string; expiresAt: number }>()
let pendingToken: Promise<string> | null = null
return {
async accessToken(): Promise<string> {
if (cachedToken && cachedToken.expiresAt > Date.now()) return cachedToken.value
if (pendingToken) return await pendingToken
// One refresh avoids concurrent gcloud processes contending on the local credential store.
pendingToken = tokenCommand(['auth', 'print-access-token'])
.then((token) => {
if (!TOKEN_PATTERN.test(token)) throw new GcloudCommandError('gcloud access token')
cachedToken = { value: token, expiresAt: Date.now() + 5 * 60_000 }
return token
})
.finally(() => { pendingToken = null })
return await pendingToken
},
async identityToken(audience: string): Promise<string> {
const cached = identityTokens.get(audience)
if (cached && cached.expiresAt > Date.now()) return cached.value
const token = await tokenCommand([
'auth',
'print-identity-token',
`--audiences=${audience}`,
'--include-email'
])
if (
token.length > 8_192 ||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)
) {
throw new GcloudCommandError('gcloud identity token')
}
identityTokens.set(audience, { value: token, expiresAt: Date.now() + 5 * 60_000 })
return token
}
}
}
+77
View File
@@ -0,0 +1,77 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { z } from 'zod'
import { relayRepositoryApiPath } from './relay-repository.js'
const execFileAsync = promisify(execFile)
const WorkflowRunSchema = z.object({
id: z.number().int().positive(),
name: z.string(),
event: z.string(),
status: z.string(),
conclusion: z.string().nullable(),
head_sha: z.string(),
html_url: z.string().url(),
created_at: z.string(),
updated_at: z.string()
})
const WorkflowRunsSchema = z.object({
workflow_runs: z.array(WorkflowRunSchema)
})
export type RelayWorkflowRun = {
id: number
name: string
status: string
conclusion: string | null
headSha: string
url: string
createdAt: string
updatedAt: string
}
const RELAY_WORKFLOW_PATTERN = /(Relay|Auth|Power)/i
export async function readRelayWorkflowRuns(): Promise<RelayWorkflowRun[]> {
let stdout: string
try {
const result = await execFileAsync(
'gh',
[
'api',
'--method',
'GET',
relayRepositoryApiPath('actions/runs'),
'-f',
'per_page=100'
],
{ encoding: 'utf8', timeout: 30_000, maxBuffer: 4 * 1024 * 1024 }
)
stdout = result.stdout
} catch {
throw new Error('GitHub workflow history is unavailable')
}
const runs = WorkflowRunsSchema.parse(JSON.parse(stdout) as unknown).workflow_runs
const counts = new Map<string, number>()
return runs
.filter((run) => {
if (!RELAY_WORKFLOW_PATTERN.test(run.name)) return false
const count = counts.get(run.name) ?? 0
if (count >= 2) return false
counts.set(run.name, count + 1)
return true
})
.slice(0, 12)
.map((run) => ({
id: run.id,
name: run.name,
status: run.status,
conclusion: run.conclusion,
headSha: run.head_sha.slice(0, 8),
url: run.html_url,
createdAt: run.created_at,
updatedAt: run.updated_at
}))
}
@@ -0,0 +1,373 @@
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import {
livePreflightGcloud,
runIncidentLivePreflight
} from './incident-live-preflight-cli.js'
import type { IncidentSample } from './incident-monitor.js'
import type { AdmissionSelector } from './incident-selector.js'
const directories: string[] = []
const now = Date.parse('2026-07-28T12:00:00.000Z')
const selector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
function stateFile(
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition' = 'strict',
overrides: Record<string, unknown> = {}
): string {
const directory = mkdtempSync(join(tmpdir(), 'relay-live-preflight-'))
directories.push(directory)
const path = join(directory, 'state.json')
const expectedSelector = migrationPolicy === 'capacity-transition'
? {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
: selector
writeFileSync(path, JSON.stringify({
schemaVersion: 4,
environment: 'production',
expectedSelector,
migrationPolicy,
recoverySourceCellId:
migrationPolicy === 'recover-forward' ? 'production-gce-c1' : null,
capacityCellId:
migrationPolicy === 'capacity-transition' ? 'production-gce-c1' : null,
preDrainDryRun: true,
startedAt: new Date(now - 17 * 60_000).toISOString(),
windowStartedAt: new Date(now - 16 * 60_000).toISOString(),
durationMinutes: 15,
intervalMs: 60_000,
sampleCount: 16,
lastSampleAt: new Date(now - 60_007).toISOString(),
frozenAt: null,
completedAt: new Date(now - 60_000).toISOString(),
...overrides
}))
return path
}
function sample(): IncidentSample {
const observedAt = new Date(now).toISOString()
const signal = (value: number) => ({ value, observedAt })
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId: 'production-gce-c1',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'existing-only'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1),
'director.ready': signal(1),
'director.latency_ms': signal(1),
'auth.health': signal(1),
'auth.ready': signal(1),
'auth.latency_ms': signal(1),
'cell.production-gce-c1.health': signal(1),
'cell.production-gce-c1.ready': signal(1),
'cell.production-gce-c1.latency_ms': signal(1)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.1),
'cloud_sql.memory': signal(0.1),
'cloud_sql.backends': signal(1),
'cloud_sql.lock_waits': signal(0),
'cloud_sql.deadlocks': signal(0),
'director.instances': signal(5),
'director.cpu': signal(0.1),
'director.memory': signal(0.1),
'director.concurrency': signal(1),
'director.errors': signal(0),
'auth.errors': signal(0)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0),
'relay.pool_wait_ms': signal(0),
'relay.postgres_retries': signal(0),
'relay.postgres_retry_exhausted': signal(0),
'cell.production-gce-c1.connections': signal(1),
'cell.production-gce-c1.queued_bytes': signal(0)
}
},
'director-admin': {
observedAt,
signals: {
'cell.production-gce-c1.admission_state': signal(0),
'cell.production-gce-c1.heartbeat_fresh': signal(1),
'cell.production-gce-c1.heartbeat_age_ms': signal(1),
'cell.production-gce-c1.migration_blocked': signal(0),
'cell.production-gce-c1.migration_target_inactive': signal(0)
}
}
}
}
}
afterEach(() => {
for (const directory of directories.splice(0)) {
rmSync(directory, { recursive: true, force: true })
}
})
describe('relay incident live preflight', () => {
it('accepts the package-manager argument separator', async () => {
await expect(runIncidentLivePreflight(
['--', '--state-file', stateFile()],
{ now: () => now, collect: async () => sample() }
)).resolves.toBeUndefined()
})
it('accepts one complete fresh green sample', async () => {
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => sample() }
)).resolves.toBeUndefined()
})
it('rejects monitor evidence beyond the 25-minute lineage bound', async () => {
const path = stateFile('strict', {
startedAt: new Date(now - 26 * 60_000 - 1).toISOString()
})
await expect(runIncidentLivePreflight(
['--state-file', path],
{ now: () => now, collect: async () => sample() }
)).rejects.toThrow('monitor evidence is incomplete or stale')
})
it('scales the evidence age bound by same-cap wave index', async () => {
const agedState = (ageMs: number) => stateFile('strict', {
startedAt: new Date(now - ageMs - 17 * 60_000).toISOString(),
windowStartedAt: new Date(now - ageMs - 16 * 60_000).toISOString(),
lastSampleAt: new Date(now - ageMs - 7).toISOString(),
completedAt: new Date(now - ageMs).toISOString()
})
const deps = { now: () => now, collect: async () => sample() }
// One predecessor cell roll (~16 min) exceeds wave 0 but fits wave 1.
const oneRollOld = agedState(17 * 60_000)
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld, '--wave-index', '0'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld, '--wave-index', '1'], deps
)).resolves.toBeUndefined()
// Both edges of one predecessor job timeout: 5min + 75min exactly.
await expect(runIncidentLivePreflight(
['--state-file', agedState(80 * 60_000), '--wave-index', '1'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(80 * 60_000 + 1), '--wave-index', '1'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', agedState(155 * 60_000), '--wave-index', '2'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(155 * 60_000 + 1), '--wave-index', '2'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', agedState(230 * 60_000), '--wave-index', '3'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(230 * 60_000 + 1), '--wave-index', '3'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
// The wave index is a strict single-use 0-3 argument.
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', '4'], deps
)).rejects.toThrow('usage:')
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', ''], deps
)).rejects.toThrow('usage:')
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', '1', '--wave-index', '1'],
deps
)).rejects.toThrow('usage:')
})
it('expects the wave-adjusted live selector generation', async () => {
const agedPath = stateFile('strict', {
startedAt: new Date(now - 34 * 60_000).toISOString(),
windowStartedAt: new Date(now - 33 * 60_000).toISOString(),
lastSampleAt: new Date(now - 17 * 60_000 - 7).toISOString(),
completedAt: new Date(now - 17 * 60_000).toISOString()
})
const liveAt = (generation: number) =>
async (expectedSelector: AdmissionSelector) => ({
...sample(),
selector: { ...selector, generation },
expectedSelector
})
// One predecessor roll advanced the live selector by exactly 2.
await expect(runIncidentLivePreflight(
['--state-file', agedPath, '--wave-index', '1'],
{ now: () => now, collect: liveAt(selector.generation + 2) }
)).resolves.toBeUndefined()
// The sealed pre-roll generation must no longer satisfy wave 1.
await expect(runIncidentLivePreflight(
['--state-file', agedPath, '--wave-index', '1'],
{ now: () => now, collect: liveAt(selector.generation) }
)).rejects.toThrow('director-admin/selector_mismatch')
// Wave 0 still expects the sealed generation itself.
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: liveAt(selector.generation) }
)).resolves.toBeUndefined()
})
it('fails closed on a live threshold breach', async () => {
const unhealthy = sample()
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => unhealthy }
)).rejects.toThrow('cloud-monitoring/threshold_max')
})
it('enforces the signed migration policy', async () => {
const inactiveTarget = sample()
inactiveTarget.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
]!.value = 30
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => inactiveTarget }
)).rejects.toThrow('director-admin/threshold_max')
await expect(runIncidentLivePreflight(
['--state-file', stateFile('recover-forward')],
{ now: () => now, collect: async () => inactiveTarget }
)).resolves.toBeUndefined()
inactiveTarget.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
]!.value = 1
await expect(runIncidentLivePreflight(
['--state-file', stateFile('recover-forward')],
{ now: () => now, collect: async () => inactiveTarget }
)).rejects.toThrow('director-admin/threshold_max')
})
it('binds capacity-transition evidence to its general cell', async () => {
const capacitySample = sample()
const capacitySelector = {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
capacitySample.selector = capacitySelector
capacitySample.expectedSelector = capacitySelector
capacitySample.cells[0]!.expectedAdmissionState = 'general'
capacitySample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
]!.value = 2
await expect(runIncidentLivePreflight(
['--state-file', stateFile('capacity-transition')],
{ now: () => now, collect: async () => capacitySample }
)).resolves.toBeUndefined()
capacitySample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
]!.value = 1
await expect(runIncidentLivePreflight(
['--state-file', stateFile('capacity-transition')],
{ now: () => now, collect: async () => capacitySample }
)).rejects.toThrow('director-admin/threshold_max')
})
it('rejects stale live evidence', async () => {
const stale = sample()
stale.sources['active-probe']!.observedAt = new Date(now - 60_001).toISOString()
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => stale }
)).rejects.toThrow('active-probe/source_stale')
})
it('retries freshness-only failures when explicitly requested', async () => {
const stale = sample()
stale.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
new Date(now - 180_001).toISOString()
const missing = sample()
delete missing.sources['relay-logs']
const collect = vi.fn()
.mockResolvedValueOnce(stale)
.mockResolvedValueOnce(missing)
.mockResolvedValueOnce(sample())
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).resolves.toBeUndefined()
expect(collect).toHaveBeenCalledTimes(3)
expect(wait).toHaveBeenCalledTimes(2)
expect(wait).toHaveBeenNthCalledWith(1, 15_000)
expect(wait).toHaveBeenNthCalledWith(2, 15_000)
})
it('does not retry a threshold failure', async () => {
const unhealthy = sample()
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
new Date(now - 180_001).toISOString()
const collect = vi.fn(async () => unhealthy)
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).rejects.toThrow('cloud-monitoring/threshold_max')
expect(collect).toHaveBeenCalledOnce()
expect(wait).not.toHaveBeenCalled()
})
it('fails closed after the bounded freshness retry window', async () => {
const stale = sample()
stale.sources['cloud-monitoring']!.observedAt = new Date(now - 180_001).toISOString()
const collect = vi.fn(async () => stale)
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).rejects.toThrow('cloud-monitoring/source_stale')
expect(collect).toHaveBeenCalledTimes(5)
expect(wait).toHaveBeenCalledTimes(4)
})
it('uses the supplied admin token without minting through gcloud', async () => {
const identityToken = vi.fn(async () => 'minted.token.value')
const gcloud = livePreflightGcloud(
{ accessToken: async () => 'access-token', identityToken },
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'supplied.token.value' }
)
await expect(gcloud.identityToken!('audience')).resolves.toBe(
'supplied.token.value'
)
expect(identityToken).not.toHaveBeenCalled()
})
})
@@ -0,0 +1,195 @@
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
import { z } from 'zod'
import { createGcloudClient } from './gcloud-client.js'
import { suppliedIdentityToken } from './incident-monitor-cli.js'
import { AdmissionSelectorSchema, type AdmissionSelector } from './incident-selector.js'
import {
evaluateIncidentSample,
preDrainDryRunPassed,
type IncidentSample
} from './incident-monitor.js'
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
const FRESHNESS_RETRY_ATTEMPTS = 5
const FRESHNESS_RETRY_INTERVAL_MS = 15_000
const MONITOR_EVIDENCE_MAX_AGE_MS = 5 * 60_000
// Matches the same-cap cell job timeout-minutes; bounds each predecessor wave.
const WAVE_PREDECESSOR_TIMEOUT_MS = 75 * 60_000
const WAVE_INDEX_PATTERN = /^[0-3]$/
const FRESHNESS_FAILURE_CODES = new Set([
'signal_missing',
'signal_stale',
'source_missing',
'source_stale'
])
export function livePreflightGcloud(
gcloud: ReturnType<typeof createGcloudClient>,
environment: NodeJS.ProcessEnv = process.env
): ReturnType<typeof createGcloudClient> {
const token = suppliedIdentityToken(environment.ORCA_RELAY_ADMIN_ID_TOKEN)
return token ? { ...gcloud, identityToken: async () => token } : gcloud
}
const PreflightStateSchema = z.object({
schemaVersion: z.literal(4),
environment: z.literal('production'),
expectedSelector: AdmissionSelectorSchema,
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
recoverySourceCellId: z.string().nullable(),
capacityCellId: z.string().nullable(),
preDrainDryRun: z.literal(true),
startedAt: z.string(),
windowStartedAt: z.string(),
durationMinutes: z.literal(15),
intervalMs: z.literal(60_000),
sampleCount: z.number().int().min(16),
lastSampleAt: z.string(),
frozenAt: z.null(),
completedAt: z.string()
}).superRefine((state, context) => {
const validRecovery =
state.migrationPolicy === 'recover-forward' &&
state.capacityCellId === null &&
state.recoverySourceCellId !== null &&
state.expectedSelector.membership.existingOnly.includes(
state.recoverySourceCellId
)
const validStrict =
state.migrationPolicy === 'strict' &&
state.recoverySourceCellId === null &&
state.capacityCellId === null
const validCapacity =
state.migrationPolicy === 'capacity-transition' &&
state.recoverySourceCellId === null &&
state.capacityCellId !== null &&
state.expectedSelector.membership.general.includes(state.capacityCellId)
if (!validRecovery && !validStrict && !validCapacity) {
context.addIssue({
code: 'custom',
message: 'relay live preflight migration policy is invalid'
})
}
})
export async function runIncidentLivePreflight(
argv: string[],
dependencies: {
now?: () => number
wait?: (ms: number) => Promise<void>
collect?: (expectedSelector: AdmissionSelector) => Promise<IncidentSample>
gcloud?: ReturnType<typeof createGcloudClient>
environment?: NodeJS.ProcessEnv
} = {}
): Promise<void> {
const args = argv[0] === '--' ? argv.slice(1) : argv
const freshnessRetryCount = args.filter((arg) => arg === '--retry-freshness').length
const rest = args.filter((arg) => arg !== '--retry-freshness')
const stateArgs: string[] = []
let waveIndex = '0'
let waveIndexCount = 0
for (let index = 0; index < rest.length; index += 1) {
if (rest[index] === '--wave-index') {
waveIndexCount += 1
waveIndex = rest[index + 1] ?? ''
index += 1
} else {
stateArgs.push(rest[index] as string)
}
}
if (
freshnessRetryCount > 1 ||
waveIndexCount > 1 ||
!WAVE_INDEX_PATTERN.test(waveIndex) ||
stateArgs.length !== 2 ||
stateArgs[0] !== '--state-file' ||
!stateArgs[1]
) {
throw new Error(
'usage: --state-file <verified-monitor-state> [--wave-index <0-3>] [--retry-freshness]'
)
}
const state = PreflightStateSchema.parse(
JSON.parse(await readFile(resolve(stateArgs[1]), 'utf8'))
)
const now = dependencies.now ?? Date.now
const completedAt = Date.parse(state.completedAt)
const windowStartedAt = Date.parse(state.windowStartedAt)
const lastSampleAt = Date.parse(state.lastSampleAt)
const evidenceAgeMs = now() - completedAt
// Later same-cap waves start after sequential predecessor cell rolls, so the
// freshness bound grows by one cell-job timeout per predecessor; the live
// samples collected below still hold every wave to current health.
const maxEvidenceAgeMs =
MONITOR_EVIDENCE_MAX_AGE_MS + Number(waveIndex) * WAVE_PREDECESSOR_TIMEOUT_MS
if (
!preDrainDryRunPassed(state) ||
!Number.isFinite(windowStartedAt) ||
completedAt - windowStartedAt < 15 * 60_000 ||
!Number.isFinite(lastSampleAt) ||
lastSampleAt > completedAt ||
completedAt - lastSampleAt > state.intervalMs ||
!Number.isFinite(completedAt) ||
evidenceAgeMs < 0 ||
evidenceAgeMs > maxEvidenceAgeMs
) {
throw new Error('relay live preflight monitor evidence is incomplete or stale')
}
const gcloud = livePreflightGcloud(
dependencies.gcloud ?? createGcloudClient(),
dependencies.environment
)
// Each predecessor same-cap apply wave reversibly isolates and restores its
// cell, advancing the selector generation by exactly 2 with membership
// unchanged (rollback is single-cell, so it never reaches a later wave), so
// the live selector comparison must expect the wave-adjusted generation.
const collectOptions = {
environment: state.environment,
expectedSelector: {
...state.expectedSelector,
generation: state.expectedSelector.generation + 2 * Number(waveIndex)
},
...(dependencies.now ? { now: dependencies.now } : {})
}
const injected = dependencies.collect
const collect = injected
? () => injected(collectOptions.expectedSelector)
: createIncidentSampleCollector(gcloud, collectOptions)
const wait = dependencies.wait ?? ((ms: number) => new Promise<void>((resolveWait) => {
setTimeout(resolveWait, ms)
}))
const attempts = freshnessRetryCount === 1 ? FRESHNESS_RETRY_ATTEMPTS : 1
for (let attempt = 1; attempt <= attempts; attempt++) {
const evaluation = evaluateIncidentSample(
await collect(),
now(),
state.migrationPolicy,
state.recoverySourceCellId,
state.capacityCellId
)
if (evaluation.status === 'green') return
const freshnessOnly = evaluation.failures.every((failure) =>
FRESHNESS_FAILURE_CODES.has(failure.code)
)
if (!freshnessOnly || attempt === attempts) {
throw new Error(
`relay live preflight failed: ${evaluation.failures
.map((failure) => `${failure.source}/${failure.code}`)
.join(',')}`
)
}
console.warn(
`relay live preflight awaiting fresh evidence (${attempt}/${attempts - 1})`
)
await wait(FRESHNESS_RETRY_INTERVAL_MS)
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
runIncidentLivePreflight(process.argv.slice(2)).catch((error: unknown) => {
console.error(error instanceof Error ? error.message : 'relay live preflight failed')
process.exitCode = 1
})
}
@@ -0,0 +1,454 @@
import { mkdtempSync, readFileSync, rmSync, statSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import {
parseIncidentMonitorArguments,
runIncidentMonitorCli
} from './incident-monitor-cli.js'
import type { IncidentSample } from './incident-monitor.js'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
const directories: string[] = []
const startedAt = Date.parse('2026-07-28T00:00:00.000Z')
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map((cell) => cell.cellId)
const selector = {
generation: 1,
membership: {
existingOnly: productionCells.slice(1),
migrationOnly: [],
general: [productionCells[0]!]
}
}
const selectorArguments = [
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
productionCells[0]!
]
const signal = (value: number, at: number) => ({
value,
observedAt: new Date(at).toISOString()
})
function sample(at: number): IncidentSample {
const observedAt = new Date(at).toISOString()
const cellId = 'production-gce-c1'
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId,
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1, at),
'director.ready': signal(1, at),
'director.latency_ms': signal(1, at),
'auth.health': signal(1, at),
'auth.ready': signal(1, at),
'auth.latency_ms': signal(1, at),
[`cell.${cellId}.health`]: signal(1, at),
[`cell.${cellId}.ready`]: signal(1, at),
[`cell.${cellId}.latency_ms`]: signal(1, at)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.1, at),
'cloud_sql.memory': signal(0.1, at),
'cloud_sql.backends': signal(1, at),
'cloud_sql.lock_waits': signal(0, at),
'cloud_sql.deadlocks': signal(0, at),
'director.instances': signal(5, at),
'director.cpu': signal(0.1, at),
'director.memory': signal(0.1, at),
'director.concurrency': signal(1, at),
'director.errors': signal(0, at),
'auth.errors': signal(0, at)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0, at),
'relay.pool_wait_ms': signal(0, at),
'relay.postgres_retries': signal(0, at),
'relay.postgres_retry_exhausted': signal(0, at),
[`cell.${cellId}.connections`]: signal(1, at),
[`cell.${cellId}.queued_bytes`]: signal(0, at)
}
},
'director-admin': {
observedAt,
signals: {
[`cell.${cellId}.admission_state`]: signal(2, at),
[`cell.${cellId}.heartbeat_fresh`]: signal(1, at),
[`cell.${cellId}.heartbeat_age_ms`]: signal(1, at),
[`cell.${cellId}.migration_blocked`]: signal(0, at),
[`cell.${cellId}.migration_target_inactive`]: signal(0, at)
}
}
}
}
}
afterEach(() => {
for (const directory of directories.splice(0)) {
rmSync(directory, { recursive: true, force: true })
}
})
describe('incident monitor CLI', () => {
it('requires an exact selector and rejects invalid membership', () => {
expect(() => parseIncidentMonitorArguments([])).toThrow(
'--expected-selector-generation'
)
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
'production-gce-c99'
])
).toThrow('every configured cell exactly once')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--interval-seconds',
'61'
])
).toThrow('between 1 and 60')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'14'
])
).toThrow('between 15 and 90')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
'--expected-selector-generation',
'0',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
productionCells[0]!,
'--expected-general-cells',
'none'
])
).toThrow('generation 0 cannot represent migration-only')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'recover-forward',
'--pre-drain-dry-run'
])
).toThrow('requires an existing-only --recovery-source-cell-id')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'capacity-transition',
'--pre-drain-dry-run'
])
).toThrow('requires a general --capacity-cell-id')
expect(
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'capacity-transition',
'--capacity-cell-id',
productionCells[0]!,
'--pre-drain-dry-run'
])
).toMatchObject({
migrationPolicy: 'capacity-transition',
capacityCellId: productionCells[0]!,
recoverySourceCellId: null
})
})
it('writes private durable checkpoints for a green pre-drain dry run', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const output: string[] = []
const code = await runIncidentMonitorCli(
[
'--incident-id',
'incident-1',
...selectorArguments,
'--pre-drain-dry-run',
'--output-directory',
directory
],
{
cwd: directory,
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => sample(now),
writeOutput: (value) => output.push(value)
}
)
expect(code).toBe(0)
const statePath = join(directory, 'incident-1.state.json')
const summaryPath = join(directory, 'incident-1.summaries.jsonl')
const markdownPath = join(directory, 'incident-1.summary.md')
expect(statSync(statePath).mode & 0o077).toBe(0)
expect(statSync(summaryPath).mode & 0o077).toBe(0)
expect(statSync(markdownPath).mode & 0o077).toBe(0)
const summaries = readFileSync(summaryPath, 'utf8')
.trim()
.split('\n')
.map((line) => JSON.parse(line))
expect(summaries.map((entry) => entry.checkpointMinute)).toEqual([0, 5, 15])
expect(output.join('')).not.toContain('token')
expect(readFileSync(markdownPath, 'utf8')).toContain(
'| 0 | 15 | green | 16 | none |'
)
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
frozenAt: null,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
sampleCount: 16
})
})
it('runs a recovery dry run without masking blocked migrations', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const recoverySelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: productionCells.slice(1)
}
}
const recoverySample = (): IncidentSample => {
const current = sample(now)
current.selector = recoverySelector
current.expectedSelector = recoverySelector
current.cells[0]!.expectedAdmissionState = 'existing-only'
current.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0, now)
current.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(30, now)
return current
}
const args = [
'--incident-id',
'incident-1',
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
'production-gce-c1',
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
productionCells.slice(1).join(','),
'--pre-drain-dry-run',
'--migration-policy',
'recover-forward',
'--recovery-source-cell-id',
'production-gce-c1',
'--output-directory',
directory
]
const dependencies = {
cwd: directory,
now: () => now,
wait: async (ms: number) => {
now += ms
},
collect: async () => recoverySample(),
writeOutput: () => {}
}
await expect(runIncidentMonitorCli(args, dependencies)).resolves.toBe(0)
expect(
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
).toMatchObject({
migrationPolicy: 'recover-forward',
recoverySourceCellId: 'production-gce-c1',
capacityCellId: null,
frozenAt: null
})
})
it('fails a frozen pre-drain gate after its first sample', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
let collections = 0
let waits = 0
const code = await runIncidentMonitorCli(
[
'--incident-id',
'incident-1',
...selectorArguments,
'--pre-drain-dry-run',
'--output-directory',
directory
],
{
cwd: directory,
now: () => now,
wait: async (ms) => {
waits++
now += ms
},
collect: async () => {
collections++
const unhealthy = sample(now)
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] =
signal(801, now)
return unhealthy
},
writeOutput: () => {}
}
)
expect(code).toBe(2)
expect(collections).toBe(1)
expect(waits).toBe(0)
expect(
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
).toMatchObject({
completedAt: new Date(startedAt).toISOString(),
frozenAt: new Date(startedAt).toISOString(),
sampleCount: 1
})
})
it('requires --restart and preserves a latched freeze', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const args = [
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'15',
'--output-directory',
directory
]
await runIncidentMonitorCli(args, {
cwd: directory,
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => {
const unhealthy = sample(now)
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(801, now)
return unhealthy
},
writeOutput: () => {}
})
await expect(
runIncidentMonitorCli(args, {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).rejects.toThrow('pass --restart')
const changedAdmission = [...args]
changedAdmission[changedAdmission.indexOf('--expected-selector-generation') + 1] = '2'
await expect(
runIncidentMonitorCli([...changedAdmission, '--restart'], {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).rejects.toThrow('do not match')
await expect(
runIncidentMonitorCli([...args, '--restart'], {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).resolves.toBe(2)
})
it('resumes a gracefully segmented monitor without resetting continuity', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const args = [
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'15',
'--output-directory',
directory
]
const dependencies = {
cwd: directory,
now: () => now,
wait: async (ms: number) => {
now += ms
},
collect: async () => sample(now),
writeOutput: () => {}
}
await expect(
runIncidentMonitorCli([...args, '--max-samples-this-run', '2'], dependencies)
).resolves.toBe(0)
const statePath = join(directory, 'incident-1.state.json')
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: null,
sampleCount: 2,
lastSampleAt: new Date(startedAt + 60_000).toISOString()
})
await expect(
runIncidentMonitorCli([...args, '--restart'], dependencies)
).resolves.toBe(0)
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
windowSequence: 0,
sampleCount: 17
})
})
})
@@ -0,0 +1,493 @@
import { randomUUID } from 'node:crypto'
import {
appendFile,
chmod,
mkdir,
open,
readFile,
rename,
stat,
writeFile
} from 'node:fs/promises'
import { dirname, resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
import { z } from 'zod'
import { relayOpsEnvironment } from './environment-config.js'
import { createGcloudClient } from './gcloud-client.js'
import {
AdmissionSelectorSchema,
normalizeSelectorMembership,
type AdmissionSelector
} from './incident-selector.js'
import {
initialIncidentMonitorState,
preDrainDryRunPassed,
runIncidentMonitor,
type IncidentCheckpoint,
type IncidentSample,
type IncidentMonitorState
} from './incident-monitor.js'
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
const StateSchema = z.object({
schemaVersion: z.literal(4),
incidentId: z.string(),
environment: z.enum(['production', 'staging']),
expectedSelector: AdmissionSelectorSchema,
preDrainDryRun: z.boolean(),
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
recoverySourceCellId: z.string().nullable(),
capacityCellId: z.string().nullable(),
startedAt: z.string(),
windowStartedAt: z.string().nullable(),
windowSequence: z.number().int().nonnegative(),
durationMinutes: z.number().int(),
intervalMs: z.number().int(),
nextCheckpointIndex: z.number().int().nonnegative(),
sampleCount: z.number().int().nonnegative(),
totalSampleCount: z.number().int().nonnegative(),
lastSampleAt: z.string().nullable(),
continuityEvents: z.array(z.object({
recordedAt: z.string(),
windowSequence: z.number().int().nonnegative(),
failures: z.array(z.object({
code: z.string(),
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
signal: z.string().optional(),
observed: z.number().optional(),
threshold: z.number().optional()
}))
})),
frozenAt: z.string().nullable(),
failures: z.array(z.object({
code: z.string(),
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
signal: z.string().optional(),
observed: z.number().optional(),
threshold: z.number().optional()
})),
completedAt: z.string().nullable()
})
type CliOptions = {
environment: 'production' | 'staging'
incidentId: string
durationMinutes: number
intervalMs: number
expectedSelector: AdmissionSelector
stateFile: string
summaryFile: string
markdownFile: string
restart: boolean
preDrainDryRun: boolean
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition'
recoverySourceCellId: string | null
capacityCellId: string | null
maxSamplesThisRun: number | null
}
function parsePositiveInteger(value: string | undefined, name: string): number {
const parsed = Number(value)
if (!Number.isSafeInteger(parsed) || parsed <= 0) {
throw new Error(`${name} must be a positive integer`)
}
return parsed
}
export function parseIncidentMonitorArguments(argv: string[], cwd = process.cwd()): CliOptions {
const flags = new Set(['restart', 'pre-drain-dry-run'])
const valueArguments = new Set([
'duration-minutes',
'environment',
'expected-selector-generation',
'expected-existing-only-cells',
'expected-migration-only-cells',
'expected-general-cells',
'incident-id',
'interval-seconds',
'max-samples-this-run',
'migration-policy',
'output-directory',
'recovery-source-cell-id',
'capacity-cell-id'
])
const values: Record<string, string> = {}
const enabledFlags = new Set<string>()
for (let index = 0; index < argv.length; index++) {
const argument = argv[index]
if (!argument?.startsWith('--')) throw new Error(`invalid argument ${argument ?? ''}`)
const name = argument.slice(2)
if (flags.has(name)) {
enabledFlags.add(name)
continue
}
if (!valueArguments.has(name)) throw new Error(`unknown argument --${name}`)
const value = argv[++index]
if (!value || value.startsWith('--')) throw new Error(`missing --${name} value`)
values[name] = value
}
const environment = z.enum(['production', 'staging']).parse(
values.environment ?? 'production'
)
const incidentId = values['incident-id'] ?? randomUUID()
if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]{7,127}$/.test(incidentId)) {
throw new Error('--incident-id must be 8-128 safe characters')
}
const selectorGeneration = Number(values['expected-selector-generation'])
if (!Number.isSafeInteger(selectorGeneration) || selectorGeneration < 0) {
throw new Error('--expected-selector-generation must be a nonnegative integer')
}
const configuredCells = new Set(
relayOpsEnvironment(environment).cells.map((cell) => cell.cellId)
)
const cellList = (name: string): string[] => {
const value = values[name]
if (value === undefined) throw new Error(`--${name} is required; use none for an empty set`)
return value === 'none' ? [] : value.split(',').map((cellId) => cellId.trim())
}
const expectedSelector = {
generation: selectorGeneration,
membership: normalizeSelectorMembership(
{
existingOnly: cellList('expected-existing-only-cells'),
migrationOnly: cellList('expected-migration-only-cells'),
general: cellList('expected-general-cells')
},
configuredCells
)
}
if (selectorGeneration === 0 && expectedSelector.membership.migrationOnly.length > 0) {
throw new Error('generation 0 cannot represent migration-only admission')
}
const preDrainDryRun = enabledFlags.has('pre-drain-dry-run')
const migrationPolicy = z.enum([
'strict',
'recover-forward',
'capacity-transition'
]).parse(
values['migration-policy'] ?? 'strict'
)
const recoverySourceCellId =
values['recovery-source-cell-id'] === undefined ||
values['recovery-source-cell-id'] === 'none'
? null
: values['recovery-source-cell-id']
const capacityCellId =
values['capacity-cell-id'] === undefined || values['capacity-cell-id'] === 'none'
? null
: values['capacity-cell-id']
const durationMinutes = parsePositiveInteger(
values['duration-minutes'] ?? (preDrainDryRun ? '15' : '90'),
'--duration-minutes'
)
const intervalMs =
parsePositiveInteger(values['interval-seconds'] ?? '60', '--interval-seconds') * 1_000
if (durationMinutes < 15 || durationMinutes > 90) {
throw new Error('--duration-minutes must be between 15 and 90')
}
if (intervalMs > 60_000) {
throw new Error('--interval-seconds must be between 1 and 60')
}
if (preDrainDryRun && durationMinutes !== 15) {
throw new Error('--pre-drain-dry-run requires --duration-minutes 15')
}
if (migrationPolicy !== 'strict' && !preDrainDryRun) {
throw new Error(`--migration-policy ${migrationPolicy} requires --pre-drain-dry-run`)
}
if (
migrationPolicy === 'recover-forward' &&
(
recoverySourceCellId === null ||
!configuredCells.has(recoverySourceCellId) ||
!expectedSelector.membership.existingOnly.includes(recoverySourceCellId)
)
) {
throw new Error(
'--migration-policy recover-forward requires an existing-only --recovery-source-cell-id'
)
}
if (migrationPolicy !== 'recover-forward' && recoverySourceCellId !== null) {
throw new Error('--recovery-source-cell-id requires --migration-policy recover-forward')
}
if (
migrationPolicy === 'capacity-transition' &&
(
capacityCellId === null ||
!configuredCells.has(capacityCellId) ||
!expectedSelector.membership.general.includes(capacityCellId)
)
) {
throw new Error(
'--migration-policy capacity-transition requires a general --capacity-cell-id'
)
}
if (migrationPolicy !== 'capacity-transition' && capacityCellId !== null) {
throw new Error('--capacity-cell-id requires --migration-policy capacity-transition')
}
const directory = resolve(cwd, values['output-directory'] ?? '.relay-incidents')
const maxSamplesThisRun = values['max-samples-this-run']
? parsePositiveInteger(values['max-samples-this-run'], '--max-samples-this-run')
: null
return {
environment,
incidentId,
durationMinutes,
intervalMs,
expectedSelector,
stateFile: resolve(directory, `${incidentId}.state.json`),
summaryFile: resolve(directory, `${incidentId}.summaries.jsonl`),
markdownFile: resolve(directory, `${incidentId}.summary.md`),
restart: enabledFlags.has('restart'),
preDrainDryRun,
migrationPolicy,
recoverySourceCellId,
capacityCellId,
maxSamplesThisRun
}
}
async function fileExists(path: string): Promise<boolean> {
try {
await stat(path)
return true
} catch {
return false
}
}
async function syncFile(path: string): Promise<void> {
const handle = await open(path, 'r')
try {
await handle.sync()
} finally {
await handle.close()
}
}
async function persistState(path: string, state: IncidentMonitorState): Promise<void> {
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
await chmod(dirname(path), 0o700)
const temporaryPath = `${path}.tmp`
await writeFile(temporaryPath, `${JSON.stringify(state)}\n`, { mode: 0o600 })
await chmod(temporaryPath, 0o600)
await syncFile(temporaryPath)
await rename(temporaryPath, path)
await syncFile(path)
}
async function appendCheckpoint(path: string, checkpoint: IncidentCheckpoint): Promise<void> {
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
await chmod(dirname(path), 0o700)
if (await fileExists(path)) {
const existing = (await readFile(path, 'utf8'))
.trim()
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line) as IncidentCheckpoint)
if (
existing.some((entry) =>
entry.windowSequence === checkpoint.windowSequence &&
entry.checkpointMinute === checkpoint.checkpointMinute
)
) return
}
await appendFile(path, `${JSON.stringify(checkpoint)}\n`, { mode: 0o600 })
await chmod(path, 0o600)
await syncFile(path)
}
function markdownFailure(checkpoint: IncidentCheckpoint): string {
if (checkpoint.failures.length === 0) return 'none'
return checkpoint.failures
.map((failure) => {
const signal = failure.signal ? `/${failure.signal}` : ''
return `${failure.source}/${failure.code}${signal}`
})
.join(', ')
}
async function writeMarkdownSummary(
path: string,
summaryPath: string,
incidentId: string,
environment: string
): Promise<void> {
const checkpoints = (await readFile(summaryPath, 'utf8'))
.trim()
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line) as IncidentCheckpoint)
const rows = checkpoints.map((checkpoint) => [
`| ${checkpoint.windowSequence}`,
checkpoint.checkpointMinute,
checkpoint.status,
checkpoint.sampleCount,
`${markdownFailure(checkpoint)} |`
].join(' | '))
const markdown = [
'# Relay incident monitor',
'',
`Incident: \`${incidentId}\``,
'',
`Environment: \`${environment}\``,
'',
`Expected selector: \`${JSON.stringify(checkpoints[0]?.expectedSelector ?? null)}\``,
'',
`Migration policy: \`${checkpoints[0]?.migrationPolicy ?? 'unknown'}\``,
'',
`Recovery source: \`${checkpoints[0]?.recoverySourceCellId ?? 'none'}\``,
'',
`Capacity cell: \`${checkpoints[0]?.capacityCellId ?? 'none'}\``,
'',
'| Window | Minute | Status | Samples | Failures |',
'| ---: | ---: | --- | ---: | --- |',
...rows,
''
].join('\n')
const temporaryPath = `${path}.tmp`
await writeFile(temporaryPath, markdown, { mode: 0o600 })
await chmod(temporaryPath, 0o600)
await syncFile(temporaryPath)
await rename(temporaryPath, path)
await syncFile(path)
}
export function suppliedIdentityToken(value: string | undefined): string | null {
if (value === undefined) return null
if (
value.length > 8_192 ||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(value)
) {
throw new Error('ORCA_RELAY_ADMIN_ID_TOKEN is invalid')
}
return value
}
async function readInitialState(
options: CliOptions,
now: () => number = Date.now
): Promise<IncidentMonitorState> {
const exists = await fileExists(options.stateFile)
if (exists && !options.restart) {
throw new Error('incident state already exists; pass --restart to resume it')
}
if (!exists && options.restart) throw new Error('no incident state exists to restart')
if (!exists) {
return initialIncidentMonitorState({
incidentId: options.incidentId,
environment: options.environment,
expectedSelector: options.expectedSelector,
preDrainDryRun: options.preDrainDryRun,
migrationPolicy: options.migrationPolicy,
recoverySourceCellId: options.recoverySourceCellId,
capacityCellId: options.capacityCellId,
startedAt: new Date(now()).toISOString(),
durationMinutes: options.durationMinutes,
intervalMs: options.intervalMs
})
}
const state = StateSchema.parse(
JSON.parse(await readFile(options.stateFile, 'utf8'))
) as IncidentMonitorState
if (
state.incidentId !== options.incidentId ||
state.environment !== options.environment ||
JSON.stringify(state.expectedSelector) !== JSON.stringify(options.expectedSelector) ||
state.preDrainDryRun !== options.preDrainDryRun ||
state.migrationPolicy !== options.migrationPolicy ||
state.recoverySourceCellId !== options.recoverySourceCellId ||
state.capacityCellId !== options.capacityCellId ||
state.durationMinutes !== options.durationMinutes ||
state.intervalMs !== options.intervalMs
) {
throw new Error('restart arguments do not match durable incident state')
}
return state
}
export async function runIncidentMonitorCli(
argv: string[],
dependencies: {
cwd?: string
now?: () => number
wait?: (ms: number) => Promise<void>
gcloud?: ReturnType<typeof createGcloudClient>
collect?: () => Promise<IncidentSample>
writeOutput?: (value: string) => void
environment?: NodeJS.ProcessEnv
} = {}
): Promise<number> {
const options = parseIncidentMonitorArguments(argv, dependencies.cwd)
const state = await readInitialState(options, dependencies.now)
const baseGcloud = dependencies.gcloud ?? createGcloudClient()
const token = suppliedIdentityToken(
(dependencies.environment ?? process.env).ORCA_RELAY_ADMIN_ID_TOKEN
)
await persistState(options.stateFile, state)
const gcloud = token
? { ...baseGcloud, identityToken: async () => token }
: baseGcloud
const collect =
dependencies.collect ??
createIncidentSampleCollector(gcloud, {
environment: options.environment,
expectedSelector: options.expectedSelector,
...(dependencies.now ? { now: dependencies.now } : {})
})
let samplesThisRun = 0
const segmentedCollect = async (): Promise<IncidentSample> => {
try {
return await collect()
} finally {
samplesThisRun++
}
}
const wait =
dependencies.wait ??
((ms: number) => new Promise((resolvePromise) => setTimeout(resolvePromise, ms)))
const segmentComplete = Symbol('segment-complete')
const output = dependencies.writeOutput ?? ((value) => process.stdout.write(`${value}\n`))
let result: IncidentMonitorState
try {
result = await runIncidentMonitor(state, {
now: dependencies.now ?? Date.now,
wait: async (ms) => {
if (
options.maxSamplesThisRun !== null &&
samplesThisRun >= options.maxSamplesThisRun
) {
throw segmentComplete
}
await wait(ms)
},
collect: segmentedCollect,
persist: async (nextState) => await persistState(options.stateFile, nextState),
checkpoint: async (checkpoint) => {
await appendCheckpoint(options.summaryFile, checkpoint)
await writeMarkdownSummary(
options.markdownFile,
options.summaryFile,
options.incidentId,
options.environment
)
output(JSON.stringify(checkpoint))
}
})
} catch (error) {
if (error !== segmentComplete) throw error
return 0
}
if (options.preDrainDryRun && !preDrainDryRunPassed(result)) return 2
return result.frozenAt ? 2 : 0
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
runIncidentMonitorCli(process.argv.slice(2))
.then((code) => {
process.exitCode = code
})
.catch((error: unknown) => {
console.error(error instanceof Error ? error.message : 'incident monitor failed')
process.exitCode = 1
})
}
@@ -0,0 +1,418 @@
import { describe, expect, it } from 'vitest'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { effectiveAdmissionState } from './incident-selector.js'
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
import {
directorSignals,
GOOGLE_METRICS,
readGoogleMetric,
readGoogleMetricWithEmptyRetry,
relayFiveMinuteDeltaSignal
} from './incident-monitor-sources.js'
const now = Date.parse('2026-07-28T10:00:00.000Z')
const startAt = new Date(now - 5 * 60_000).toISOString()
const endAt = new Date(now).toISOString()
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map(
({ cellId }) => cellId
)
describe('incident monitor sources', () => {
it('uses legacy booleans only at selector generation zero', () => {
const membership = {
existingOnly: ['c1'],
migrationOnly: [],
general: ['c2']
}
expect(
effectiveAdmissionState({ generation: 0, membership }, true, 'c1')
).toBe('general')
expect(
effectiveAdmissionState({ generation: 1, membership }, true, 'c1')
).toBe('existing-only')
})
it('sums DELTA metrics across the window and scopes every target exactly', async () => {
const filters: string[] = []
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
filters.push(url.searchParams.get('filter') ?? '')
return Response.json({
timeSeries: [
{
points: [
{
interval: { endTime: new Date(now - 120_000).toISOString() },
value: { int64Value: '2' }
},
{
interval: { endTime: new Date(now - 60_000).toISOString() },
value: { int64Value: '3' }
}
]
}
]
})
}
const environment = RELAY_OPS_ENVIRONMENTS.production
const directorErrors = GOOGLE_METRICS.find(
(definition) => definition.signal === 'director.errors'
)!
const deadlocks = GOOGLE_METRICS.find(
(definition) => definition.signal === 'cloud_sql.deadlocks'
)!
await expect(
readGoogleMetric(
environment,
directorErrors,
'secret-access-token',
startAt,
endAt,
fetchImpl
)
).resolves.toEqual({
value: 5,
observedAt: new Date(now - 60_000).toISOString()
})
await readGoogleMetric(
environment,
deadlocks,
'secret-access-token',
startAt,
endAt,
fetchImpl
)
expect(filters[0]).toContain(
'resource.label."service_name"="orca-cloud-relay"'
)
expect(filters[0]).toContain('metric.label."response_code"!="503"')
expect(filters[1]).toContain(
'resource.label."database_id"="onorca-cloud:orca-cloud-auth-db"'
)
})
it('zero-fills an expired sparse lock-wait point', async () => {
let pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
const fetchImpl: typeof fetch = async () => Response.json({
timeSeries: [{
points: [{
interval: { endTime: new Date(pointAt).toISOString() },
value: { int64Value: '1' }
}]
}]
})
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl
)).resolves.toEqual({
value: 1,
observedAt: new Date(pointAt).toISOString()
})
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => now + 3_000
)).resolves.toEqual({
value: 1,
observedAt: new Date(pointAt).toISOString()
})
pointAt--
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl
)).resolves.toEqual({ value: 0, observedAt: endAt })
})
it('freshens a sparse zero without masking a recent nonzero lock wait', async () => {
let value = 0
const pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
const readAt = now + 11_879
const fetchImpl: typeof fetch = async () => Response.json({
timeSeries: [{
points: [{
interval: { endTime: new Date(pointAt).toISOString() },
value: { int64Value: String(value) }
}]
}]
})
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
const sparseZero = await readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => readAt
)
expect(sparseZero).toEqual({
value: 0,
observedAt: new Date(readAt).toISOString()
})
expect(readAt - pointAt).toBe(191_879)
expect(readAt - Date.parse(sparseZero!.observedAt)).toBe(0)
value = 20
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => readAt
)).resolves.toEqual({
value: 20,
observedAt: new Date(pointAt).toISOString()
})
})
it('preserves a recent nonzero lock wait across staggered series', async () => {
const nonzeroAt = now - 179_000
const zeroAt = now - 178_000
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
const metric = await readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json({
timeSeries: [
{
points: [{
interval: { endTime: new Date(nonzeroAt).toISOString() },
value: { int64Value: '7' }
}]
},
{
points: [{
interval: { endTime: new Date(zeroAt).toISOString() },
value: { int64Value: '0' }
}]
}
]
})
)
expect(metric).toEqual({
value: 7,
observedAt: new Date(nonzeroAt).toISOString()
})
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json({
timeSeries: [{
points: [
{
interval: { endTime: new Date(nonzeroAt).toISOString() },
value: { int64Value: '7' }
},
{
interval: { endTime: new Date(zeroAt).toISOString() },
value: { int64Value: '0' }
}
]
}]
})
)).resolves.toEqual({ value: 0, observedAt: endAt })
})
it('retries an empty required metric without weakening its value', async () => {
let calls = 0
const waits: number[] = []
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.cpu'
)!
await expect(readGoogleMetricWithEmptyRetry(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json(calls++ === 0
? { timeSeries: [] }
: {
timeSeries: [{
points: [{
interval: { endTime: new Date(now - 60_000).toISOString() },
value: { doubleValue: 0.81 }
}]
}]
}),
() => now,
async (ms) => { waits.push(ms) }
)).resolves.toEqual({
value: 0.81,
observedAt: new Date(now - 60_000).toISOString()
})
expect(calls).toBe(2)
expect(waits).toEqual([2_000])
})
it('still reports a required metric missing after bounded retries', async () => {
let calls = 0
const waits: number[] = []
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'director.concurrency'
)!
await expect(readGoogleMetricWithEmptyRetry(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => {
calls++
return Response.json({ timeSeries: [] })
},
() => now,
async (ms) => { waits.push(ms) }
)).resolves.toBeNull()
expect(calls).toBe(3)
expect(waits).toEqual([2_000, 2_000])
})
it('timestamps sparse retry aggregates at query completion', () => {
expect(relayFiveMinuteDeltaSignal({
available: true,
points: [
{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 },
{ at: new Date(now - 4 * 60_000).toISOString(), value: 2 }
]
}, endAt)).toEqual({ value: 2, observedAt: endAt })
expect(relayFiveMinuteDeltaSignal({
available: true,
points: [{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 }]
}, endAt)).toEqual({ value: 0, observedAt: endAt })
expect(relayFiveMinuteDeltaSignal({
available: false,
points: []
}, endAt)).toBeNull()
})
it('aggregates admin state without returning tokens or response identities', async () => {
const identityToken = 'secret.header.signature'
const sensitiveIdentity = 'user@example.test'
const gcloud: GcloudClient = {
accessToken: async () => 'unused',
identityToken: async () => identityToken
}
let activeRequests = 0
let maximumActiveRequests = 0
let requestCount = 0
const fetchImpl: typeof fetch = async (_input, init) => {
requestCount++
activeRequests++
maximumActiveRequests = Math.max(maximumActiveRequests, activeRequests)
expect(new Headers(init?.headers).get('authorization')).toBe(
`Bearer ${identityToken}`
)
const body = JSON.parse(String(init?.body)) as {
cellId?: string
sourceCellId?: string
targetCellId?: string
}
await Promise.resolve()
activeRequests--
if (!body.cellId && !body.sourceCellId) {
return Response.json({
selector: {
generation: 1,
membership: {
existingOnly: productionCells.slice(2),
migrationOnly: ['production-gce-c2'],
general: ['production-gce-c1']
}
}
})
}
if (body.cellId) {
return Response.json({
status: {
// Selector-era monitoring must ignore this legacy compatibility bit.
enabled: body.cellId !== 'production-gce-c1',
connectionCapacity: {
hardCap: body.cellId === 'production-gce-c1' ? 1_000 : 600
},
runtime: {
lastHeartbeatAt: now - 1_000,
heartbeatFresh: true
},
userId: sensitiveIdentity
}
})
}
return Response.json({
blocked: 0,
blockedExpiredUnregistered:
body.sourceCellId === 'production-gce-c1' &&
body.targetCellId === 'production-gce-c2'
? 1
: 0,
registeredTargetInactive: 0,
userId: sensitiveIdentity
})
}
const result = await directorSignals(
'production',
{
generation: 1,
membership: {
existingOnly: productionCells.slice(2),
migrationOnly: ['production-gce-c2'],
general: ['production-gce-c1']
}
},
gcloud,
now,
fetchImpl
)
expect(requestCount).toBe(productionCells.length * 2)
expect(maximumActiveRequests).toBe(1)
expect(
result.source.signals['cell.production-gce-c1.migration_blocked']
).toMatchObject({ value: 1 })
expect(result.cells.find((cell) => cell.cellId === 'production-gce-c1')).toMatchObject({
expectedAdmissionState: 'general'
})
expect(
result.source.signals['cell.production-gce-c1.admission_state']
).toMatchObject({ value: 2 })
expect(
result.source.signals['cell.production-gce-c1.connection_hard_cap']
).toMatchObject({ value: 1_000 })
expect(
result.source.signals['cell.production-gce-c2.admission_state']
).toMatchObject({ value: 1 })
const serialized = JSON.stringify(result)
expect(serialized).not.toContain(identityToken)
expect(serialized).not.toContain(sensitiveIdentity)
})
})
@@ -0,0 +1,646 @@
import { z } from 'zod'
import { buildDashboardSnapshot } from './dashboard-snapshot.js'
import type {
RelayOpsEnvironment,
RelayOpsEnvironmentId
} from './environment-config.js'
import { relayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import type { RelayMetricSnapshot } from './monitoring-snapshot.js'
import {
AdmissionSelectorSchema,
effectiveAdmissionState,
normalizeSelectorMembership,
selectorCellState,
type AdmissionSelector,
} from './incident-selector.js'
import {
INCIDENT_MONITOR_THRESHOLDS,
type IncidentSample,
type IncidentSignal,
type IncidentSource
} from './incident-monitor.js'
const NumericSchema = z.union([z.number(), z.string()])
.transform(Number)
.pipe(z.number().finite())
const MonitoringPointSchema = z.object({
interval: z.object({ endTime: z.string() }),
value: z.object({
doubleValue: NumericSchema.optional(),
int64Value: NumericSchema.optional(),
distributionValue: z.object({
mean: NumericSchema.optional(),
range: z.object({ max: NumericSchema.optional() }).optional()
}).optional()
})
})
const MonitoringResponseSchema = z.object({
timeSeries: z.array(z.object({ points: z.array(MonitoringPointSchema) })).default([]),
nextPageToken: z.string().optional()
})
const CellStatusSchema = z.object({
status: z.object({
enabled: z.boolean(),
connectionCapacity: z
.object({ hardCap: z.number().int().positive() })
.nullable(),
runtime: z.object({
lastHeartbeatAt: z.number(),
heartbeatFresh: z.boolean()
}).nullable()
})
})
const SelectorStatusSchema = z.object({
selector: AdmissionSelectorSchema
})
const MigrationStatusSchema = z.object({
blocked: z.number().int().nonnegative(),
registeredTargetInactive: z.number().int().nonnegative(),
blockedExpiredUnregistered: z.number().int().nonnegative()
})
export type GoogleMetricDefinition = {
signal: string
type: string
resourceFilter: string
aggregation: 'latest-max' | 'latest-sum' | 'window-sum'
emptyIsZero?: boolean
zeroAfterMs?: number
}
export const GOOGLE_METRICS: GoogleMetricDefinition[] = [
{
signal: 'cloud_sql.cpu',
type: 'cloudsql.googleapis.com/database/cpu/utilization',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-max'
},
{
signal: 'cloud_sql.memory',
type: 'cloudsql.googleapis.com/database/memory/utilization',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-max'
},
{
signal: 'cloud_sql.backends',
type: 'cloudsql.googleapis.com/database/postgresql/num_backends',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-sum'
},
{
signal: 'cloud_sql.lock_waits',
type: 'cloudsql.googleapis.com/database/postgresql/backends_in_wait',
resourceFilter:
'resource.type="cloudsql_database" AND metric.label."wait_event_type"="Lock"',
aggregation: 'latest-max',
emptyIsZero: true,
zeroAfterMs: INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
},
{
signal: 'cloud_sql.deadlocks',
type: 'cloudsql.googleapis.com/database/postgresql/deadlock_count',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'window-sum',
emptyIsZero: true
},
{
signal: 'director.instances',
type: 'run.googleapis.com/container/instance_count',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-sum'
},
{
signal: 'director.cpu',
type: 'run.googleapis.com/container/cpu/utilizations',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-max'
},
{
signal: 'director.memory',
type: 'run.googleapis.com/container/memory/utilizations',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-max'
},
{
signal: 'director.concurrency',
type: 'run.googleapis.com/container/max_request_concurrencies',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."state"="active"',
aggregation: 'latest-max'
},
{
signal: 'director.errors',
type: 'run.googleapis.com/request_count',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx" AND metric.label."response_code"!="503"',
aggregation: 'window-sum',
emptyIsZero: true
},
{
signal: 'auth.errors',
type: 'run.googleapis.com/request_count',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx"',
aggregation: 'window-sum',
emptyIsZero: true
}
]
function pointValue(point: z.infer<typeof MonitoringPointSchema>): number {
return (
point.value.doubleValue ??
point.value.int64Value ??
point.value.distributionValue?.range?.max ??
point.value.distributionValue?.mean ??
0
)
}
function serviceFilter(signal: string, directorService: string, authService: string): string {
if (signal.startsWith('director.')) {
return `resource.label."service_name"="${directorService}"`
}
if (signal.startsWith('auth.')) {
return `resource.label."service_name"="${authService}"`
}
return ''
}
function targetFilter(
definition: GoogleMetricDefinition,
environment: RelayOpsEnvironment
): string {
if (definition.signal.startsWith('cloud_sql.')) {
return `resource.label."database_id"="${environment.project}:${environment.sqlInstance}"`
}
return serviceFilter(
definition.signal,
environment.directorService,
environment.authService
)
}
async function googleJson(
fetchImpl: typeof fetch,
token: string,
url: URL | string,
init: RequestInit = {}
): Promise<unknown> {
const response = await fetchImpl(url, {
...init,
headers: {
authorization: `Bearer ${token}`,
...(init.body ? { 'content-type': 'application/json' } : {})
},
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Google telemetry returned ${response.status}`)
return await response.json()
}
export async function readGoogleMetric(
environment: RelayOpsEnvironment,
definition: GoogleMetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch,
now: () => number = () => Date.parse(endAt)
): Promise<IncidentSignal | null> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
)
const filters = [
`metric.type="${definition.type}"`,
definition.resourceFilter,
targetFilter(definition, environment)
].filter(Boolean)
url.searchParams.set('filter', filters.join(' AND '))
url.searchParams.set('interval.startTime', startAt)
url.searchParams.set('interval.endTime', endAt)
url.searchParams.set('view', 'FULL')
url.searchParams.set('pageSize', '1000')
const parsed = MonitoringResponseSchema.parse(
await googleJson(fetchImpl, token, url)
)
if (parsed.nextPageToken) throw new Error('Google metric pagination is incomplete')
const queryEndMs = Date.parse(endAt)
const readAtMs = Math.max(queryEndMs, now())
const readAt = new Date(readAtMs).toISOString()
const points = parsed.timeSeries.flatMap((series) => series.points)
if (points.length === 0) {
return definition.emptyIsZero ? { value: 0, observedAt: readAt } : null
}
const zeroAfterMs = definition.zeroAfterMs
if (definition.emptyIsZero && zeroAfterMs !== undefined) {
const latestSeriesPoints = parsed.timeSeries.flatMap((series) => {
const seriesNewestAt = Math.max(
...series.points.map((point) => Date.parse(point.interval.endTime))
)
return series.points.filter(
(point) => Date.parse(point.interval.endTime) === seriesNewestAt
)
})
const futurePoints = latestSeriesPoints.filter(
(point) => Date.parse(point.interval.endTime) > queryEndMs
)
if (futurePoints.length > 0) {
return {
value: Math.max(...futurePoints.map(pointValue)),
observedAt: new Date(Math.max(
...futurePoints.map((point) => Date.parse(point.interval.endTime))
)).toISOString()
}
}
const recentNonzero = latestSeriesPoints.filter((point) => {
const pointAt = Date.parse(point.interval.endTime)
return pointValue(point) > 0 && queryEndMs - pointAt <= zeroAfterMs
})
if (recentNonzero.length === 0) return { value: 0, observedAt: readAt }
return {
value: Math.max(...recentNonzero.map(pointValue)),
observedAt: new Date(Math.min(
...recentNonzero.map((point) => Date.parse(point.interval.endTime))
)).toISOString()
}
}
const newestAt = Math.max(...points.map((point) => Date.parse(point.interval.endTime)))
const selected = definition.aggregation === 'window-sum'
? points
: points.filter((point) => Date.parse(point.interval.endTime) === newestAt)
const values = selected.map(pointValue)
const value =
definition.aggregation !== 'latest-max'
? values.reduce((total, entry) => total + entry, 0)
: Math.max(...values)
return {
value,
observedAt: new Date(newestAt).toISOString()
}
}
export async function readGoogleMetricWithEmptyRetry(
environment: RelayOpsEnvironment,
definition: GoogleMetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch,
now: () => number = () => Date.parse(endAt),
wait: (ms: number) => Promise<void> = async (ms) =>
await new Promise((resolve) => setTimeout(resolve, ms))
): Promise<IncidentSignal | null> {
for (let attempt = 0; attempt < 3; attempt++) {
const signal = await readGoogleMetric(
environment,
definition,
token,
startAt,
endAt,
fetchImpl,
now
)
if (signal !== null || definition.emptyIsZero) return signal
if (attempt < 2) await wait(2_000)
}
return null
}
function addSignal(
signals: Record<string, IncidentSignal>,
name: string,
value: number | null,
observedAt: string | null
): void {
if (value === null || observedAt === null) return
signals[name] = { value, observedAt }
}
function endpointSignals(
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>,
nowAt: string
): IncidentSource {
const signals: Record<string, IncidentSignal> = {}
const addEndpoint = (
prefix: string,
endpoint: { health: boolean | null; ready: boolean | null; latencyMs: number | null },
unavailableIsZero = false
) => {
addSignal(
signals,
`${prefix}.health`,
endpoint.health === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.health),
nowAt
)
addSignal(
signals,
`${prefix}.ready`,
endpoint.ready === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.ready),
nowAt
)
addSignal(signals, `${prefix}.latency_ms`, endpoint.latencyMs, nowAt)
}
addEndpoint('director', snapshot.resources.directorEndpoint)
addEndpoint('auth', snapshot.resources.authEndpoint)
for (const cell of snapshot.resources.cells) {
addEndpoint(`cell.${cell.cellId}`, cell.endpoint, true)
}
return { observedAt: nowAt, signals }
}
export function relayFiveMinuteDeltaSignal(
metric: Pick<RelayMetricSnapshot, 'available' | 'points'>,
endAt: string
): IncidentSignal | null {
if (!metric.available) return null
const endMs = Date.parse(endAt)
if (!Number.isFinite(endMs)) throw new Error('Relay telemetry end time is invalid')
return {
value: metric.points
.filter((point) => {
const pointMs = Date.parse(point.at)
return pointMs >= endMs - 300_000 && pointMs <= endMs
})
.reduce((total, point) => total + point.value, 0),
observedAt: endAt
}
}
function relaySignals(
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>
): IncidentSource {
const metrics = snapshot.monitoring.metrics
const signals: Record<string, IncidentSignal> = {}
addSignal(
signals,
'relay.pool_waiting',
metrics.db_waiters_max.latest,
metrics.db_waiters_max.latestAt
)
addSignal(
signals,
'relay.pool_wait_ms',
metrics.db_wait_ms_max.latest,
metrics.db_wait_ms_max.latestAt
)
const retries = relayFiveMinuteDeltaSignal(
metrics.postgres_retries,
snapshot.monitoring.endAt
)
if (retries) signals['relay.postgres_retries'] = retries
const retryExhausted = relayFiveMinuteDeltaSignal(
metrics.postgres_retry_exhausted,
snapshot.monitoring.endAt
)
if (retryExhausted) signals['relay.postgres_retry_exhausted'] = retryExhausted
for (const cell of snapshot.resources.cells) {
addSignal(
signals,
`cell.${cell.cellId}.connections`,
metrics.total_connections.latestByCell[cell.cellId] ?? null,
metrics.total_connections.latestAt
)
addSignal(
signals,
`cell.${cell.cellId}.queued_bytes`,
metrics.queued_bytes.latestByCell[cell.cellId] ?? null,
metrics.queued_bytes.latestAt
)
}
const observedTimes = Object.values(signals).map((entry) => Date.parse(entry.observedAt))
if (observedTimes.length === 0) throw new Error('Relay telemetry is unavailable')
const observedAt = new Date(Math.max(...observedTimes)).toISOString()
return { observedAt, signals }
}
async function adminPost(
fetchImpl: typeof fetch,
origin: string,
token: string,
path: string,
body: unknown
): Promise<unknown> {
const response = await fetchImpl(`${origin}${path}`, {
method: 'POST',
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
body: JSON.stringify(body),
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Relay admin telemetry returned ${response.status}`)
return await response.json()
}
export async function directorSignals(
environmentId: RelayOpsEnvironmentId,
expectedSelector: AdmissionSelector,
gcloud: GcloudClient,
nowMs: number,
fetchImpl: typeof fetch
): Promise<{
source: IncidentSource
selector: AdmissionSelector
cells: IncidentSample['cells']
}> {
const environment = relayOpsEnvironment(environmentId)
if (!gcloud.identityToken) throw new Error('gcloud identity-token support is unavailable')
const token = await gcloud.identityToken(`${environment.directorOrigin}/v1/admin/drain`)
const configuredCellIds = new Set(environment.cells.map((cell) => cell.cellId))
const rawSelector = SelectorStatusSchema.parse(
await adminPost(
fetchImpl,
environment.directorOrigin,
token,
'/v1/admin/admission-selector/status',
{ v: 1 }
)
).selector
const selector = {
generation: rawSelector.generation,
membership: normalizeSelectorMembership(rawSelector.membership, configuredCellIds)
}
const statuses: Array<{
cell: RelayOpsEnvironment['cells'][number]
status: z.infer<typeof CellStatusSchema>['status']
}> = []
for (const cell of environment.cells) {
statuses.push({
cell,
status: CellStatusSchema.parse(
await adminPost(fetchImpl, environment.directorOrigin, token, '/v1/admin/cell-status', {
v: 1,
cellId: cell.cellId
})
).status
})
}
const migrationEntries: Array<{
sourceCellId: string
migration: z.infer<typeof MigrationStatusSchema>
}> = []
const migrationTargets = new Set(expectedSelector.membership.migrationOnly)
for (const source of environment.cells) {
for (const target of environment.cells) {
if (source.cellId === target.cellId || !migrationTargets.has(target.cellId)) continue
migrationEntries.push({
sourceCellId: source.cellId,
migration: MigrationStatusSchema.parse(
await adminPost(
fetchImpl,
environment.directorOrigin,
token,
'/v1/admin/evacuation-status',
{
v: 1,
sourceCellId: source.cellId,
targetCellId: target.cellId,
completeReady: false
}
)
)
})
}
}
const migrationBySource = new Map<string, { blocked: number; targetInactive: number }>()
for (const { sourceCellId, migration } of migrationEntries) {
const aggregate = migrationBySource.get(sourceCellId) ?? { blocked: 0, targetInactive: 0 }
aggregate.blocked += migration.blocked + migration.blockedExpiredUnregistered
aggregate.targetInactive += migration.registeredTargetInactive
migrationBySource.set(sourceCellId, aggregate)
}
const nowAt = new Date(nowMs).toISOString()
const signals: Record<string, IncidentSignal> = {}
for (const { cell, status } of statuses) {
const prefix = `cell.${cell.cellId}`
const admissionState = effectiveAdmissionState(
selector,
status.enabled,
cell.cellId
)
addSignal(
signals,
`${prefix}.admission_state`,
['existing-only', 'migration-only', 'general'].indexOf(admissionState),
nowAt
)
addSignal(
signals,
`${prefix}.connection_hard_cap`,
status.connectionCapacity?.hardCap ??
INCIDENT_MONITOR_THRESHOLDS.cellConnections,
nowAt
)
if (status.runtime) {
addSignal(
signals,
`${prefix}.heartbeat_fresh`,
Number(status.runtime.heartbeatFresh),
nowAt
)
addSignal(
signals,
`${prefix}.heartbeat_age_ms`,
Math.max(0, nowMs - status.runtime.lastHeartbeatAt),
nowAt
)
}
const migration = migrationBySource.get(cell.cellId) ?? { blocked: 0, targetInactive: 0 }
addSignal(signals, `${prefix}.migration_blocked`, migration.blocked, nowAt)
addSignal(
signals,
`${prefix}.migration_target_inactive`,
migration.targetInactive,
nowAt
)
}
return {
source: { observedAt: nowAt, signals },
selector,
cells: statuses.map(({ cell }) => ({
cellId: cell.cellId,
runtimeKnown: true,
powered: true,
expectedAdmissionState: selectorCellState(expectedSelector, cell.cellId)
}))
}
}
export type IncidentSampleCollectorOptions = {
environment: RelayOpsEnvironmentId
expectedSelector: AdmissionSelector
fetchImpl?: typeof fetch
now?: () => number
}
export function createIncidentSampleCollector(
gcloud: GcloudClient,
options: IncidentSampleCollectorOptions
): () => Promise<IncidentSample> {
const fetchImpl = options.fetchImpl ?? fetch
const now = options.now ?? Date.now
return async () => {
const nowMs = now()
const nowAt = new Date(nowMs).toISOString()
const startAt = new Date(nowMs - 5 * 60_000).toISOString()
const environment = relayOpsEnvironment(options.environment)
const accessToken = gcloud.accessToken()
const cloudMetricEntries = accessToken.then(async (token) => await Promise.all(
GOOGLE_METRICS.map(async (definition) => [
definition.signal,
await readGoogleMetricWithEmptyRetry(
environment,
definition,
token,
startAt,
nowAt,
fetchImpl,
now
)
] as const)
))
const [snapshot, director, metricEntries] = await Promise.all([
buildDashboardSnapshot(options.environment, gcloud, {
windowMinutes: 5,
now: new Date(nowMs),
fetchImpl
}),
directorSignals(
options.environment,
options.expectedSelector,
gcloud,
nowMs,
fetchImpl
),
cloudMetricEntries
])
const cloudSignals = Object.fromEntries(
metricEntries.filter((entry): entry is [string, IncidentSignal] => entry[1] !== null)
)
const relay = relaySignals(snapshot)
const poweredByCell = new Map(
snapshot.resources.cells.map((cell) => [
cell.cellId,
{
runtimeKnown: cell.targetSize !== null,
powered: (cell.targetSize ?? 0) > 0
}
])
)
return {
collectedAt: nowAt,
selector: director.selector,
expectedSelector: options.expectedSelector,
sources: {
'active-probe': endpointSignals(snapshot, nowAt),
'cloud-monitoring': { observedAt: nowAt, signals: cloudSignals },
'relay-logs': relay,
'director-admin': director.source
},
cells: director.cells.map((cell) => ({
...cell,
runtimeKnown: poweredByCell.get(cell.cellId)?.runtimeKnown ?? false,
powered: poweredByCell.get(cell.cellId)?.powered ?? false
}))
}
}
}
@@ -0,0 +1,786 @@
import { describe, expect, it } from 'vitest'
import {
evaluateIncidentSample,
INCIDENT_CHECKPOINT_MINUTES,
INCIDENT_MONITOR_THRESHOLDS,
INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS,
initialIncidentMonitorState,
preDrainDryRunPassed,
runIncidentMonitor,
type IncidentSample
} from './incident-monitor.js'
const startedAt = Date.parse('2026-07-28T00:00:00.000Z')
const selector = {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
const signal = (value: number, at = startedAt) => ({
value,
observedAt: new Date(at).toISOString()
})
function healthySample(at = startedAt): IncidentSample {
const observedAt = new Date(at).toISOString()
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId: 'production-gce-c1',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1, at),
'director.ready': signal(1, at),
'director.latency_ms': signal(100, at),
'auth.health': signal(1, at),
'auth.ready': signal(1, at),
'auth.latency_ms': signal(100, at),
'cell.production-gce-c1.health': signal(1, at),
'cell.production-gce-c1.ready': signal(1, at),
'cell.production-gce-c1.latency_ms': signal(100, at)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.2, at),
'cloud_sql.memory': signal(0.3, at),
'cloud_sql.backends': signal(12, at),
'cloud_sql.lock_waits': signal(0, at),
'cloud_sql.deadlocks': signal(0, at),
'director.instances': signal(5, at),
'director.cpu': signal(0.2, at),
'director.memory': signal(0.3, at),
'director.concurrency': signal(5, at),
'director.errors': signal(0, at),
'auth.errors': signal(0, at)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0, at),
'relay.pool_wait_ms': signal(1, at),
'relay.postgres_retries': signal(0, at),
'relay.postgres_retry_exhausted': signal(0, at),
'cell.production-gce-c1.connections': signal(100, at),
'cell.production-gce-c1.queued_bytes': signal(0, at)
}
},
'director-admin': {
observedAt,
signals: {
'cell.production-gce-c1.admission_state': signal(2, at),
'cell.production-gce-c1.heartbeat_fresh': signal(1, at),
'cell.production-gce-c1.heartbeat_age_ms': signal(1_000, at),
'cell.production-gce-c1.migration_blocked': signal(0, at),
'cell.production-gce-c1.migration_target_inactive': signal(0, at)
}
}
}
}
}
describe('incident monitor evaluator', () => {
it('accepts a complete fresh sample at every exact boundary', () => {
const sample = healthySample()
sample.sources['active-probe']!.signals['director.latency_ms'] =
signal(INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs)
sample.sources['cloud-monitoring']!.signals['cloud_sql.cpu'] =
signal(INCIDENT_MONITOR_THRESHOLDS.cloudSqlCpuUtilization)
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPoolWaitMs)
sample.sources['relay-logs']!.signals['relay.postgres_retries'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries)
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] =
signal(INCIDENT_MONITOR_THRESHOLDS.cloudSqlBackends)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('freezes when postgres retries exceed the recalibrated ceiling', () => {
const sample = healthySample()
sample.sources['relay-logs']!.signals['relay.postgres_retries'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries + 1)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'freeze',
failures: [
expect.objectContaining({ signal: 'relay.postgres_retries', threshold: 300 })
]
})
})
it('allows missing auth readiness and legacy existing-only connections', () => {
const sample = healthySample()
const legacySelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
sample.selector = legacySelector
sample.expectedSelector = legacySelector
sample.cells[0]!.expectedAdmissionState = 'existing-only'
delete sample.sources['active-probe']!.signals['auth.ready']
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(900)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('fails loudly on every missing or stale source', () => {
const missing = healthySample()
delete missing.sources['relay-logs']
expect(evaluateIncidentSample(missing, startedAt).failures).toContainEqual({
code: 'source_missing',
source: 'relay-logs'
})
const stale = healthySample(startedAt - 180_001)
const failures = evaluateIncidentSample(stale, startedAt).failures
expect(failures.some((failure) => failure.source === 'cloud-monitoring')).toBe(true)
expect(failures.some((failure) => failure.source === 'active-probe')).toBe(true)
})
it('freezes on SQL, director, relay pool, heartbeat, and migration breaches', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.cpu'] = signal(0.81)
sample.sources['cloud-monitoring']!.signals['director.instances'] = signal(7)
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(801)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_age_ms'
] = signal(45_001)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
] = signal(1)
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(501)
const evaluation = evaluateIncidentSample(sample, startedAt)
expect(evaluation.status).toBe('freeze')
expect(evaluation.failures.map((failure) => failure.signal)).toEqual(
expect.arrayContaining([
'cloud_sql.cpu',
'director.instances',
'relay.pool_waiting',
'cell.production-gce-c1.connections',
'cell.production-gce-c1.heartbeat_age_ms',
'cell.production-gce-c1.migration_blocked'
])
)
})
it('uses each cell reported physical connection cap', () => {
const sample = healthySample()
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.connection_hard_cap'
] = signal(1_000)
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(999)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(1_000)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('freeze')
})
it('allows five active directors plus the warm rollback', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['director.instances'] = signal(6)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
})
it('allows bounded relay pool waiting below the latency ceiling', () => {
const sample = healthySample()
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(800)
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] = signal(2_500)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] = signal(2_501)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'relay-logs',
signal: 'relay.pool_wait_ms',
observed: 2_501,
threshold: 2_500
})
})
it('bounds Cloud SQL backends above measured healthy peaks', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] = signal(250)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] = signal(251)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.backends',
observed: 251,
threshold: 250
})
})
it('bounds SQL lock waiters and keeps deadlocks zero-tolerance', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(20)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(21)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.lock_waits',
observed: 21,
threshold: 20
})
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(0)
sample.sources['cloud-monitoring']!.signals['cloud_sql.deadlocks'] = signal(1)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.deadlocks',
observed: 1,
threshold: 0
})
})
it('allows only registered target inactivity during forward recovery', () => {
const sample = healthySample()
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(40)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'director-admin',
signal: 'cell.production-gce-c1.migration_target_inactive',
observed: 40,
threshold: 0
})
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c1'
)
).toMatchObject({
status: 'green',
failures: []
})
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c2'
).failures
).toContainEqual(expect.objectContaining({
signal: 'cell.production-gce-c1.migration_target_inactive'
}))
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
] = signal(1)
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c1'
).failures
).toContainEqual({
code: 'threshold_max',
source: 'director-admin',
signal: 'cell.production-gce-c1.migration_blocked',
observed: 1,
threshold: 0
})
})
it('scopes registered target inactivity to the capacity cell', () => {
const sample = healthySample()
const scopedSelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: ['production-gce-c2', 'production-gce-c3']
}
}
sample.selector = scopedSelector
sample.expectedSelector = scopedSelector
sample.cells[0]!.expectedAdmissionState = 'existing-only'
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
sample.cells.push({
cellId: 'production-gce-c2',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
})
sample.cells.push({
cellId: 'production-gce-c3',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
})
for (const sourceName of ['active-probe', 'relay-logs', 'director-admin'] as const) {
const signals = sample.sources[sourceName]!.signals
for (const [name, value] of Object.entries(signals)) {
if (name.includes('production-gce-c1')) {
for (const cellId of ['production-gce-c2', 'production-gce-c3']) {
signals[name.replace('production-gce-c1', cellId)] = value
}
}
}
}
for (const cellId of ['production-gce-c2', 'production-gce-c3']) {
sample.sources['director-admin']!.signals[
`cell.${cellId}.admission_state`
] = signal(2)
}
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(40)
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c2'
)
).toMatchObject({ status: 'green', failures: [] })
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c3'
)
).toMatchObject({ status: 'green', failures: [] })
sample.sources['director-admin']!.signals[
'cell.production-gce-c2.migration_target_inactive'
] = signal(1)
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c2'
).failures
).toContainEqual(expect.objectContaining({
signal: 'cell.production-gce-c2.migration_target_inactive'
}))
})
it('freezes when expected admission has no powered runtime', () => {
const sample = healthySample()
sample.cells[0]!.powered = false
const evaluation = evaluateIncidentSample(sample, startedAt)
expect(evaluation.failures).toContainEqual({
code: 'expected_admission_without_runtime',
source: 'director-admin',
signal: 'cell.production-gce-c1.powered',
observed: 0,
threshold: 1
})
})
it('ignores stale runtime signals for an expected offline existing-only cell', () => {
const sample = healthySample()
sample.cells[0] = {
...sample.cells[0]!,
powered: false,
expectedAdmissionState: 'existing-only'
}
sample.expectedSelector = {
generation: sample.expectedSelector.generation,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
sample.selector = sample.expectedSelector
sample.sources['active-probe']!.signals['cell.production-gce-c1.health'] = signal(0)
sample.sources['active-probe']!.signals['cell.production-gce-c1.ready'] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_fresh'
] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_age_ms'
] = signal(9_000_000)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('freezes when cell power inventory is unavailable', () => {
const sample = healthySample()
sample.cells[0]!.runtimeKnown = false
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'runtime_power_unknown',
source: 'cloud-monitoring',
signal: 'cell.production-gce-c1.powered'
})
})
it('freezes on selector generation or tri-state membership drift', () => {
const generation = healthySample()
generation.selector = { ...generation.selector, generation: 2 }
expect(evaluateIncidentSample(generation, startedAt).failures).toContainEqual(
expect.objectContaining({ code: 'selector_mismatch' })
)
const membership = healthySample()
membership.selector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
expect(evaluateIncidentSample(membership, startedAt).failures).toContainEqual(
expect.objectContaining({ code: 'selector_mismatch' })
)
})
})
describe('incident monitor lifecycle', () => {
it('persists the exact 90-minute checkpoints while polling every minute', async () => {
let now = startedAt
const checkpoints: number[] = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: false,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 90,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async (summary) => {
checkpoints.push(summary.checkpointMinute)
}
})
expect(checkpoints).toEqual([...INCIDENT_CHECKPOINT_MINUTES])
expect(result.sampleCount).toBe(91)
expect(result.completedAt).not.toBeNull()
expect(result.frozenAt).toBeNull()
})
it('latches monitor freeze across a restart without rewriting its time', async () => {
let now = startedAt
let unhealthy = true
let persisted = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: false,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const stop = new Error('stop after first persistence')
await expect(
runIncidentMonitor(persisted, {
now: () => now,
wait: async () => {
throw stop
},
collect: async () => {
const sample = healthySample(now)
if (unhealthy) {
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(31, now)
}
return sample
},
persist: async (state) => {
persisted = structuredClone(state)
},
checkpoint: async () => {}
})
).rejects.toThrow('stop after first persistence')
const frozenAt = persisted.frozenAt
unhealthy = false
now += 60_000
const result = await runIncidentMonitor(persisted, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.frozenAt).toBe(frozenAt)
expect(preDrainDryRunPassed(result)).toBe(false)
})
it.each([15, 90])(
'restarts a %i-minute continuous window after stale telemetry',
async (durationMinutes) => {
let now = startedAt
let staleInjected = false
const checkpoints: Array<[number, number]> = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: durationMinutes === 15,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => {
if (!staleInjected && now === startedAt + 5 * 60_000) {
staleInjected = true
return healthySample(now - 180_001)
}
return healthySample(now)
},
persist: async () => {},
checkpoint: async (summary) => {
checkpoints.push([summary.windowSequence, summary.checkpointMinute])
}
})
expect(result.windowSequence).toBe(1)
expect(result.windowStartedAt).toBe(
new Date(startedAt + 6 * 60_000).toISOString()
)
expect(result.completedAt).toBe(
new Date(startedAt + (durationMinutes + 6) * 60_000).toISOString()
)
expect(result.sampleCount).toBe(durationMinutes + 1)
expect(result.continuityEvents).toHaveLength(1)
expect(result.continuityEvents[0]!.failures).toEqual(
expect.arrayContaining([
expect.objectContaining({ code: 'source_stale' })
])
)
expect(checkpoints).toContainEqual([1, durationMinutes])
expect(result.frozenAt).toBeNull()
}
)
it('resets at the next fresh sample after a runner gap', async () => {
let now = startedAt + 10 * 60_000
const state = {
...initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
}),
lastSampleAt: new Date(startedAt).toISOString(),
sampleCount: 1,
totalSampleCount: 1
}
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.windowSequence).toBe(1)
expect(result.windowStartedAt).toBe(new Date(startedAt + 10 * 60_000).toISOString())
expect(result.continuityEvents[0]!.failures[0]!.code).toBe('monitor_gap')
expect(result.sampleCount).toBe(16)
})
it('fails a dry run after 25 total minutes of continuity resets', async () => {
let now = startedAt
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () =>
healthySample(now === startedAt + 10 * 60_000 ? now - 180_001 : now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.completedAt).toBe(
new Date(startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS).toISOString()
)
expect(result.frozenAt).not.toBeNull()
expect(result.windowSequence).toBe(1)
expect(result.sampleCount).toBe(15)
expect(result.failures).toContainEqual({
code: 'continuity_deadline_exceeded',
source: 'active-probe',
observed: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS,
threshold: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
})
expect(preDrainDryRunPassed(result)).toBe(false)
})
it('fails an overdue resumed dry run before collecting again', async () => {
const now = startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS + 1
let collections = 0
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async () => {},
collect: async () => {
collections++
return healthySample(now)
},
persist: async () => {},
checkpoint: async () => {}
})
expect(collections).toBe(0)
expect(result.failures).toContainEqual(expect.objectContaining({
code: 'continuity_deadline_exceeded'
}))
})
it('requires a completed green 15-minute dry run', () => {
const state = {
...initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
}),
sampleCount: 16,
completedAt: new Date(startedAt + 15 * 60_000).toISOString()
}
expect(preDrainDryRunPassed(state)).toBe(true)
expect(preDrainDryRunPassed({ ...state, frozenAt: state.startedAt })).toBe(false)
expect(preDrainDryRunPassed({
...state,
completedAt: new Date(startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS + 1).toISOString()
})).toBe(false)
})
it('keeps poll starts on the configured cadence after collection time', async () => {
let now = startedAt
const starts: number[] = []
const waits: number[] = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
waits.push(ms)
now += ms
},
collect: async () => {
starts.push(now)
now += 15_000
return healthySample(now)
},
persist: async () => {},
checkpoint: async () => {}
})
expect(starts.slice(0, 3)).toEqual([
startedAt,
startedAt + 60_000,
startedAt + 120_000
])
expect(waits[0]).toBe(45_000)
})
})
@@ -0,0 +1,800 @@
import {
exactAdmissionSelector,
type AdmissionSelector,
type AdmissionState
} from './incident-selector.js'
export const INCIDENT_MONITOR_THRESHOLDS = {
activeProbeMaxAgeMs: 60_000,
cloudDataMaxAgeMs: 180_000,
relayLogMaxAgeMs: 180_000,
heartbeatMaxAgeMs: 45_000,
endpointLatencyMs: 2_000,
cloudSqlCpuUtilization: 0.8,
cloudSqlMemoryUtilization: 0.9,
// Why: healthy latest-sum backends idle near 100 but spike to 216 in 1-minute
// bursts (~10 min/day exceeded the old bar of 160 on 2026-08-26, freezing a
// pre-drain gate on baseline noise). 250 clears measured healthy peaks while
// firing well before the verified 400-connection ceiling; pool-wait and
// exhausted-retry signals keep their strict thresholds.
cloudSqlBackends: 250,
// Bound the observed recovery load; deadlocks remain zero-tolerance.
cloudSqlLockWaits: 20,
cloudSqlDeadlocks: 0,
// Why: pool amplitude cannot discriminate the 2026-08-23 incident. Healthy
// fleet-wide bursts reach 43 waiters / 2.03s waits several times an hour,
// and a cell roll's reconnect surge peaks at 676 waiters, while the real
// incident peaked at 356 waiters and never crossed 2.5s (waits cap ~2s
// structurally). The old bars of 30/1000 froze pre-drain gates on baseline
// noise (~17% per 15-minute window). Incident-class contention is caught by
// the retry signals below at ~10x separation; these bars now fence only
// genuinely unbounded queueing, which grows past both.
relayPoolWaiting: 800,
relayPoolWaitMs: 2_500,
// Why: successful lock retries are the contention machinery working, not harm.
// Healthy 2026-08-26 baseline bursts to 234/5min (26% of windows crossed the old
// bar of 20, set unmeasured at the monitor's 2026-07-28 birth); the 2026-08-23
// incident ran ~2,200-3,000/5min. 300 clears healthy bursts with ~10x incident
// margin; relayPostgresRetryExhausted below stays at zero tolerance, so any
// transaction that terminally fails still freezes the gate.
relayPostgresRetries: 300,
relayPostgresRetryExhausted: 0,
// Why: public admission is a per-instance semaphore, so fleet assignment capacity is
// concurrency x instances. A floor of 1 let the 2026-08-04 collapse from five instances
// to two pass unnoticed, which is the exact failure this monitor exists to catch. Keep in
// step with relay_min_instances in infra/terraform/environments/production.tfvars.
directorInstancesMin: 5,
// Five serving instances plus one warm scale-to-zero rollback during recovery.
directorInstancesMax: 6,
directorCpuUtilization: 0.8,
directorMemoryUtilization: 0.8,
directorConcurrency: 64,
directorErrors: 0,
authErrors: 0,
// Why: 800 exceeded the 600 hard cap, so this could never trigger on a capped cell. 500 is
// the ordinary admission limit a cell actually stops at (600 cap - 100 control-rebind reserve).
cellConnections: 500,
cellQueuedBytes: 48 * 1024 * 1024,
migrationBlocked: 0
} as const
export const INCIDENT_CHECKPOINT_MINUTES = [0, 5, 15, 30, 45, 60, 75, 90] as const
export const INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS = 25 * 60_000
export type IncidentSourceName =
| 'active-probe'
| 'cloud-monitoring'
| 'relay-logs'
| 'director-admin'
export type IncidentMigrationPolicy =
| 'strict'
| 'recover-forward'
| 'capacity-transition'
export type IncidentSignal = {
value: number
observedAt: string
}
export type IncidentSource = {
observedAt: string
signals: Record<string, IncidentSignal>
}
export type IncidentCellExpectation = {
cellId: string
runtimeKnown: boolean
powered: boolean
expectedAdmissionState: AdmissionState
}
export type IncidentSample = {
collectedAt: string
selector: AdmissionSelector
expectedSelector: AdmissionSelector
sources: Partial<Record<IncidentSourceName, IncidentSource>>
cells: IncidentCellExpectation[]
}
export type IncidentFailure = {
code: string
source: IncidentSourceName
signal?: string
observed?: number
threshold?: number
}
export type IncidentEvaluation = {
status: 'green' | 'freeze'
evaluatedAt: string
failures: IncidentFailure[]
}
export type IncidentCheckpoint = {
schemaVersion: 4
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
windowSequence: number
windowStartedAt: string
checkpointMinute: number
scheduledAt: string
recordedAt: string
status: 'green' | 'freeze'
frozenAt: string | null
sampleCount: number
failures: IncidentFailure[]
thresholds: typeof INCIDENT_MONITOR_THRESHOLDS
}
export type IncidentMonitorState = {
schemaVersion: 4
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
startedAt: string
windowStartedAt: string | null
windowSequence: number
durationMinutes: number
intervalMs: number
nextCheckpointIndex: number
sampleCount: number
totalSampleCount: number
lastSampleAt: string | null
continuityEvents: {
recordedAt: string
windowSequence: number
failures: IncidentFailure[]
}[]
frozenAt: string | null
failures: IncidentFailure[]
completedAt: string | null
}
type NumericRule = {
source: IncidentSourceName
signal: string
comparison: 'max' | 'min' | 'equal'
threshold: number
}
const NUMERIC_RULES: NumericRule[] = [
{ source: 'active-probe', signal: 'director.health', comparison: 'equal', threshold: 1 },
{ source: 'active-probe', signal: 'director.ready', comparison: 'equal', threshold: 1 },
{
source: 'active-probe',
signal: 'director.latency_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
},
{ source: 'active-probe', signal: 'auth.health', comparison: 'equal', threshold: 1 },
{
source: 'active-probe',
signal: 'auth.latency_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.cpu',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlCpuUtilization
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.memory',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlMemoryUtilization
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.backends',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlBackends
},
{
source: 'cloud-monitoring',
signal: 'director.instances',
comparison: 'min',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorInstancesMin
},
{
source: 'cloud-monitoring',
signal: 'director.instances',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorInstancesMax
},
{
source: 'cloud-monitoring',
signal: 'director.cpu',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorCpuUtilization
},
{
source: 'cloud-monitoring',
signal: 'director.memory',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorMemoryUtilization
},
{
source: 'cloud-monitoring',
signal: 'director.concurrency',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorConcurrency
},
{
source: 'cloud-monitoring',
signal: 'director.errors',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorErrors
},
{
source: 'cloud-monitoring',
signal: 'auth.errors',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.authErrors
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.lock_waits',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlLockWaits
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.deadlocks',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlDeadlocks
},
{
source: 'relay-logs',
signal: 'relay.pool_waiting',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPoolWaiting
},
{
source: 'relay-logs',
signal: 'relay.pool_wait_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPoolWaitMs
},
{
source: 'relay-logs',
signal: 'relay.postgres_retries',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries
},
{
source: 'relay-logs',
signal: 'relay.postgres_retry_exhausted',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetryExhausted
}
]
const SOURCE_MAX_AGE: Record<IncidentSourceName, number> = {
'active-probe': INCIDENT_MONITOR_THRESHOLDS.activeProbeMaxAgeMs,
'cloud-monitoring': INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs,
'relay-logs': INCIDENT_MONITOR_THRESHOLDS.relayLogMaxAgeMs,
'director-admin': INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
}
function ageMs(timestamp: string, nowMs: number): number {
const parsed = Date.parse(timestamp)
return Number.isFinite(parsed) ? nowMs - parsed : Number.POSITIVE_INFINITY
}
function addMissingSignal(
failures: IncidentFailure[],
source: IncidentSourceName,
signal: string
): void {
failures.push({ code: 'signal_missing', source, signal })
}
function checkRule(
failures: IncidentFailure[],
source: IncidentSourceName,
signals: Record<string, IncidentSignal>,
rule: NumericRule
): void {
const signal = signals[rule.signal]
if (!signal) return addMissingSignal(failures, source, rule.signal)
const failed =
(rule.comparison === 'max' && signal.value > rule.threshold) ||
(rule.comparison === 'min' && signal.value < rule.threshold) ||
(rule.comparison === 'equal' && signal.value !== rule.threshold)
if (failed) {
failures.push({
code: `threshold_${rule.comparison}`,
source,
signal: rule.signal,
observed: signal.value,
threshold: rule.threshold
})
}
}
function checkCell(
failures: IncidentFailure[],
sample: IncidentSample,
cell: IncidentCellExpectation,
migrationPolicy: IncidentMigrationPolicy,
recoverySourceCellId: string | null,
capacityCellId: string | null
): void {
const probe = sample.sources['active-probe']?.signals
const relay = sample.sources['relay-logs']?.signals
const admin = sample.sources['director-admin']?.signals
if (!cell.runtimeKnown) {
failures.push({
code: 'runtime_power_unknown',
source: 'cloud-monitoring',
signal: `cell.${cell.cellId}.powered`
})
}
if (
cell.runtimeKnown &&
cell.expectedAdmissionState !== 'existing-only' &&
!cell.powered
) {
failures.push({
code: 'expected_admission_without_runtime',
source: 'director-admin',
signal: `cell.${cell.cellId}.powered`,
observed: 0,
threshold: 1
})
}
const checks = [
['active-probe', probe, `cell.${cell.cellId}.health`, cell.powered ? 1 : 0, 'equal'],
['active-probe', probe, `cell.${cell.cellId}.ready`, cell.powered ? 1 : 0, 'equal'],
[
'active-probe',
probe,
`cell.${cell.cellId}.latency_ms`,
INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.admission_state`,
['existing-only', 'migration-only', 'general'].indexOf(
cell.expectedAdmissionState
),
'equal'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.heartbeat_fresh`,
1,
'equal'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.heartbeat_age_ms`,
INCIDENT_MONITOR_THRESHOLDS.heartbeatMaxAgeMs,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.migration_blocked`,
INCIDENT_MONITOR_THRESHOLDS.migrationBlocked,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.migration_target_inactive`,
INCIDENT_MONITOR_THRESHOLDS.migrationBlocked,
'max'
],
[
'relay-logs',
relay,
`cell.${cell.cellId}.connections`,
(admin?.[`cell.${cell.cellId}.connection_hard_cap`]?.value ??
INCIDENT_MONITOR_THRESHOLDS.cellConnections + 1) - 1,
'max'
],
[
'relay-logs',
relay,
`cell.${cell.cellId}.queued_bytes`,
INCIDENT_MONITOR_THRESHOLDS.cellQueuedBytes,
'max'
]
] as const
for (const [source, signals, signalName, threshold, comparison] of checks) {
if (
migrationPolicy === 'recover-forward' &&
cell.cellId === recoverySourceCellId &&
signalName.endsWith('.migration_target_inactive')
) {
if (!signals?.[signalName]) addMissingSignal(failures, source, signalName)
continue
}
if (
migrationPolicy === 'capacity-transition' &&
capacityCellId !== null &&
cell.cellId !== capacityCellId &&
cell.expectedAdmissionState === 'existing-only' &&
signalName.endsWith('.migration_target_inactive')
) {
if (!signals?.[signalName]) addMissingSignal(failures, source, signalName)
continue
}
if (
cell.expectedAdmissionState === 'existing-only' &&
signalName.endsWith('.connections')
) {
continue
}
if (
!cell.powered &&
[
'latency_ms',
'heartbeat_fresh',
'heartbeat_age_ms',
'connections',
'queued_bytes'
].some((suffix) => signalName.endsWith(suffix))
) {
continue
}
if (!signals?.[signalName]) {
addMissingSignal(failures, source, signalName)
continue
}
const value = signals[signalName].value
const failed = comparison === 'equal' ? value !== threshold : value > threshold
if (failed) {
failures.push({
code: `threshold_${comparison}`,
source,
signal: signalName,
observed: value,
threshold
})
}
}
}
export function evaluateIncidentSample(
sample: IncidentSample,
nowMs = Date.now(),
migrationPolicy: IncidentMigrationPolicy = 'strict',
recoverySourceCellId: string | null = null,
capacityCellId: string | null = null
): IncidentEvaluation {
const failures: IncidentFailure[] = []
if (!exactAdmissionSelector(sample.selector, sample.expectedSelector)) {
failures.push({
code: 'selector_mismatch',
source: 'director-admin',
signal: 'selector.generation',
observed: sample.selector.generation,
threshold: sample.expectedSelector.generation
})
}
for (const [sourceName, maxAge] of Object.entries(SOURCE_MAX_AGE) as [
IncidentSourceName,
number
][]) {
const source = sample.sources[sourceName]
if (!source) {
failures.push({ code: 'source_missing', source: sourceName })
continue
}
if (ageMs(source.observedAt, nowMs) < 0 || ageMs(source.observedAt, nowMs) > maxAge) {
failures.push({
code: 'source_stale',
source: sourceName,
observed: ageMs(source.observedAt, nowMs),
threshold: maxAge
})
}
for (const [signalName, signal] of Object.entries(source.signals)) {
if (ageMs(signal.observedAt, nowMs) < 0 || ageMs(signal.observedAt, nowMs) > maxAge) {
failures.push({
code: 'signal_stale',
source: sourceName,
signal: signalName,
observed: ageMs(signal.observedAt, nowMs),
threshold: maxAge
})
}
}
}
for (const rule of NUMERIC_RULES) {
const source = sample.sources[rule.source]
if (source) checkRule(failures, rule.source, source.signals, rule)
}
for (const cell of sample.cells) {
checkCell(
failures,
sample,
cell,
migrationPolicy,
recoverySourceCellId,
capacityCellId
)
}
return {
status: failures.length === 0 ? 'green' : 'freeze',
evaluatedAt: new Date(nowMs).toISOString(),
failures
}
}
export function initialIncidentMonitorState(input: {
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
startedAt: string
durationMinutes: number
intervalMs: number
}): IncidentMonitorState {
if (input.intervalMs < 1_000 || input.intervalMs > 60_000) {
throw new Error('incident monitor interval must be between 1 and 60 seconds')
}
if (input.durationMinutes < 15 || input.durationMinutes > 90) {
throw new Error('incident monitor duration must be between 15 and 90 minutes')
}
return {
schemaVersion: 4,
...input,
windowStartedAt: input.startedAt,
windowSequence: 0,
nextCheckpointIndex: 0,
sampleCount: 0,
totalSampleCount: 0,
lastSampleAt: null,
continuityEvents: [],
frozenAt: null,
failures: [],
completedAt: null
}
}
export type IncidentMonitorDependencies = {
now(): number
wait(ms: number): Promise<void>
collect(): Promise<IncidentSample>
persist(state: IncidentMonitorState): Promise<void>
checkpoint(summary: IncidentCheckpoint): Promise<void>
}
function checkpointMinutes(durationMinutes: number): number[] {
return INCIDENT_CHECKPOINT_MINUTES.filter((minute) => minute <= durationMinutes)
}
const CONTINUITY_FAILURE_CODES = new Set([
'collector_failed',
'monitor_gap',
'signal_stale',
'source_missing',
'source_stale'
])
function resetContinuousWindow(
state: IncidentMonitorState,
recordedAt: string,
failures: IncidentFailure[]
): void {
if (state.windowStartedAt !== null) {
state.windowSequence++
state.windowStartedAt = null
state.nextCheckpointIndex = 0
state.sampleCount = 0
state.completedAt = null
}
state.continuityEvents.push({
recordedAt,
windowSequence: state.windowSequence,
failures
})
}
function completeContinuityDeadline(
state: IncidentMonitorState,
nowMs: number,
lineageStartMs: number
): void {
const recordedAt = new Date(nowMs).toISOString()
state.frozenAt ??= recordedAt
state.failures.push({
code: 'continuity_deadline_exceeded',
source: 'active-probe',
observed: nowMs - lineageStartMs,
threshold: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
})
state.completedAt = recordedAt
}
export async function runIncidentMonitor(
initialState: IncidentMonitorState,
dependencies: IncidentMonitorDependencies
): Promise<IncidentMonitorState> {
const state = structuredClone(initialState)
const lineageStartMs = Date.parse(state.startedAt)
if (!Number.isFinite(lineageStartMs)) {
throw new Error('incident monitor start time is invalid')
}
const checkpoints = checkpointMinutes(state.durationMinutes)
const lineageDeadlineMs = state.preDrainDryRun
? lineageStartMs + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
: Number.POSITIVE_INFINITY
const resumedAt = dependencies.now()
const priorSampleMs = state.lastSampleAt
? Date.parse(state.lastSampleAt)
: lineageStartMs
const gapThreshold = state.intervalMs
if (resumedAt - priorSampleMs > gapThreshold) {
resetContinuousWindow(state, new Date(resumedAt).toISOString(), [{
code: 'monitor_gap',
source: 'active-probe',
observed: resumedAt - priorSampleMs,
threshold: gapThreshold
}])
}
if (state.completedAt !== null) {
await dependencies.persist(state)
return state
}
while (state.completedAt === null) {
if (dependencies.now() > lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
const sampleStartedAt = dependencies.now()
let evaluation: IncidentEvaluation
try {
evaluation = evaluateIncidentSample(
await dependencies.collect(),
dependencies.now(),
state.migrationPolicy,
state.recoverySourceCellId,
state.capacityCellId
)
} catch {
evaluation = {
status: 'freeze',
evaluatedAt: new Date(dependencies.now()).toISOString(),
failures: [{
code: 'collector_failed',
source: 'cloud-monitoring'
}]
}
}
state.totalSampleCount++
state.lastSampleAt = evaluation.evaluatedAt
const continuityFailures = evaluation.failures.filter((failure) =>
CONTINUITY_FAILURE_CODES.has(failure.code)
)
const thresholdFailures = evaluation.failures.filter((failure) =>
!CONTINUITY_FAILURE_CODES.has(failure.code)
)
if (continuityFailures.length > 0) {
resetContinuousWindow(state, evaluation.evaluatedAt, continuityFailures)
} else {
if (state.windowStartedAt === null) {
state.windowStartedAt = evaluation.evaluatedAt
}
state.sampleCount++
}
if (thresholdFailures.length > 0) {
state.frozenAt ??= evaluation.evaluatedAt
state.failures = [...state.failures, ...thresholdFailures]
}
if (state.windowStartedAt === null) {
if (dependencies.now() >= lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
await dependencies.persist(state)
await dependencies.wait(
Math.max(0, Math.min(state.intervalMs, lineageDeadlineMs - dependencies.now()))
)
continue
}
const startMs = Date.parse(state.windowStartedAt)
const endMs = startMs + state.durationMinutes * 60_000
const elapsedMinutes = (dependencies.now() - startMs) / 60_000
while (
state.nextCheckpointIndex < checkpoints.length &&
elapsedMinutes >= checkpoints[state.nextCheckpointIndex]!
) {
const minute = checkpoints[state.nextCheckpointIndex]!
await dependencies.checkpoint({
schemaVersion: 4,
incidentId: state.incidentId,
environment: state.environment,
expectedSelector: state.expectedSelector,
preDrainDryRun: state.preDrainDryRun,
migrationPolicy: state.migrationPolicy,
recoverySourceCellId: state.recoverySourceCellId,
capacityCellId: state.capacityCellId,
windowSequence: state.windowSequence,
windowStartedAt: state.windowStartedAt,
checkpointMinute: minute,
scheduledAt: new Date(startMs + minute * 60_000).toISOString(),
recordedAt: new Date(dependencies.now()).toISOString(),
status: state.frozenAt ? 'freeze' : 'green',
frozenAt: state.frozenAt,
sampleCount: state.sampleCount,
failures: state.failures,
thresholds: INCIDENT_MONITOR_THRESHOLDS
})
state.nextCheckpointIndex++
}
if (state.preDrainDryRun && state.frozenAt !== null) {
state.completedAt = new Date(dependencies.now()).toISOString()
await dependencies.persist(state)
break
}
if (dependencies.now() >= endMs) {
state.completedAt = new Date(dependencies.now()).toISOString()
await dependencies.persist(state)
break
}
if (dependencies.now() >= lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
await dependencies.persist(state)
await dependencies.wait(
Math.max(
0,
Math.min(sampleStartedAt + state.intervalMs, endMs, lineageDeadlineMs) - dependencies.now()
)
)
}
return state
}
export function preDrainDryRunPassed(state: Pick<
IncidentMonitorState,
| 'completedAt'
| 'durationMinutes'
| 'frozenAt'
| 'intervalMs'
| 'preDrainDryRun'
| 'sampleCount'
| 'startedAt'
>): boolean {
const minimumSamples =
Math.ceil((state.durationMinutes * 60_000) / state.intervalMs) + 1
const lineageElapsedMs = state.completedAt === null
? Number.POSITIVE_INFINITY
: Date.parse(state.completedAt) - Date.parse(state.startedAt)
return (
state.preDrainDryRun &&
state.durationMinutes === 15 &&
state.completedAt !== null &&
lineageElapsedMs >= 0 &&
lineageElapsedMs <= INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS &&
state.frozenAt === null &&
state.sampleCount >= minimumSamples
)
}
@@ -0,0 +1,77 @@
import { z } from 'zod'
export const AdmissionStateSchema = z.enum([
'existing-only',
'migration-only',
'general'
])
export type AdmissionState = z.infer<typeof AdmissionStateSchema>
export const SelectorMembershipSchema = z.object({
existingOnly: z.array(z.string()),
migrationOnly: z.array(z.string()),
general: z.array(z.string())
})
export type SelectorMembership = z.infer<typeof SelectorMembershipSchema>
export const AdmissionSelectorSchema = z.object({
generation: z.number().int().nonnegative(),
membership: SelectorMembershipSchema
})
export type AdmissionSelector = z.infer<typeof AdmissionSelectorSchema>
export function normalizeSelectorMembership(
membership: SelectorMembership,
configuredCellIds: ReadonlySet<string>
): SelectorMembership {
const normalized = {
existingOnly: [...membership.existingOnly].sort(),
migrationOnly: [...membership.migrationOnly].sort(),
general: [...membership.general].sort()
}
const all = [
...normalized.existingOnly,
...normalized.migrationOnly,
...normalized.general
]
if (
all.length !== configuredCellIds.size ||
new Set(all).size !== all.length ||
all.some((cellId) => !configuredCellIds.has(cellId))
) {
throw new Error('selector membership must contain every configured cell exactly once')
}
return normalized
}
export function selectorCellState(
selector: AdmissionSelector,
cellId: string
): AdmissionState {
if (selector.membership.existingOnly.includes(cellId)) return 'existing-only'
if (selector.membership.migrationOnly.includes(cellId)) return 'migration-only'
if (selector.membership.general.includes(cellId)) return 'general'
throw new Error(`selector does not contain ${cellId}`)
}
export function effectiveAdmissionState(
selector: AdmissionSelector,
legacyEnabled: boolean,
cellId: string
): AdmissionState {
if (selector.generation === 0) return legacyEnabled ? 'general' : 'existing-only'
return selectorCellState(selector, cellId)
}
export function exactAdmissionSelector(
actual: AdmissionSelector,
expected: AdmissionSelector
): boolean {
return (
actual.generation === expected.generation &&
JSON.stringify(actual.membership) === JSON.stringify(expected.membership)
)
}
+108
View File
@@ -0,0 +1,108 @@
import { randomBytes, timingSafeEqual } from 'node:crypto'
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'
import { serve } from '@hono/node-server'
import { Hono } from 'hono'
import { z } from 'zod'
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
import { createGcloudClient } from './gcloud-client.js'
import {
dispatchStagingPowerWorkflow,
parseStagingPowerRequest
} from './staging-workflow.js'
const QuerySchema = z.object({
environment: z.enum(['production', 'staging']).default('production'),
window: z.coerce.number().int().min(30).max(1440).default(360)
})
const port = z.coerce.number().int().min(1024).max(65_535).parse(process.env.PORT ?? 2455)
const controlsEnabled = process.env.RELAY_OPS_ENABLE_STAGING_CONTROLS === '1'
const csrfToken = randomBytes(32).toString('base64url')
const publicDirectory = resolve(import.meta.dirname, '../public')
const cache = new DashboardSnapshotCache(createGcloudClient())
const app = new Hono()
function safeEqual(left: string, right: string): boolean {
const leftBuffer = Buffer.from(left)
const rightBuffer = Buffer.from(right)
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer)
}
app.use('*', async (context, next) => {
await next()
context.header('Cache-Control', 'no-store')
context.header('Content-Security-Policy', [
"default-src 'self'",
"script-src 'self'",
"style-src 'self'",
"font-src 'self'",
"connect-src 'self'",
"img-src 'self' data:",
"object-src 'none'",
"base-uri 'none'",
"frame-ancestors 'none'",
"form-action 'self'"
].join('; '))
context.header('Referrer-Policy', 'no-referrer')
context.header('X-Content-Type-Options', 'nosniff')
context.header('X-Frame-Options', 'DENY')
})
app.get('/health', (context) => context.json({ status: 'ok' }))
app.get('/api/config', (context) => context.json({
stagingControlsEnabled: controlsEnabled,
csrfToken: controlsEnabled ? csrfToken : null
}))
app.get('/api/snapshot', async (context) => {
const query = QuerySchema.safeParse(context.req.query())
if (!query.success) return context.json({ error: 'Invalid dashboard query' }, 400)
try {
return context.json(await cache.read(query.data.environment, query.data.window))
} catch {
return context.json({
error: 'Relay operations data is unavailable. Check local gcloud and gh authentication.'
}, 503)
}
})
app.post('/api/staging/power', async (context) => {
if (!controlsEnabled) return context.json({ error: 'Staging controls are disabled' }, 403)
const origin = context.req.header('origin')
const expectedOrigin = `http://127.0.0.1:${port}`
if (origin !== expectedOrigin) return context.json({ error: 'Origin rejected' }, 403)
if (!safeEqual(context.req.header('x-csrf-token') ?? '', csrfToken)) {
return context.json({ error: 'Request token rejected' }, 403)
}
try {
const request = parseStagingPowerRequest(await context.req.json())
await dispatchStagingPowerWorkflow(request)
return context.json({ accepted: true })
} catch {
return context.json({ error: 'Invalid or failed staging workflow dispatch' }, 400)
}
})
const staticTypes: Record<string, string> = {
'/app.js': 'text/javascript; charset=utf-8',
'/styles.css': 'text/css; charset=utf-8'
}
for (const [route, contentType] of Object.entries(staticTypes)) {
app.get(route, async (context) => {
const file = route.slice(1)
try {
const content = await readFile(resolve(publicDirectory, file))
return context.body(content, 200, { 'Content-Type': contentType })
} catch {
return context.notFound()
}
})
}
app.get('/', async (context) => {
const html = await readFile(resolve(publicDirectory, 'index.html'), 'utf8')
return context.html(html)
})
serve({ fetch: app.fetch, hostname: '127.0.0.1', port }, () => {
// Loopback is intentional; operators may add authenticated Tailscale Serve separately.
console.log(`Orca Relay Operations: http://127.0.0.1:${port}`)
console.log(`Staging controls: ${controlsEnabled ? 'enabled through GitHub workflow' : 'read-only'}`)
})
@@ -0,0 +1,85 @@
import { describe, expect, it } from 'vitest'
import { RELAY_METRICS, readMonitoringSnapshot } from './monitoring-snapshot.js'
import type { GcloudClient } from './gcloud-client.js'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
const gcloud: GcloudClient = {
accessToken: async () => 'a'.repeat(40)
}
function distribution(at: string, mean: number | undefined, count = 1) {
return {
interval: { endTime: at },
value: { distributionValue: { count, ...(mean === undefined ? {} : { mean }) } }
}
}
describe('readMonitoringSnapshot', () => {
it('aggregates gauge series per minute and distribution deltas by count', async () => {
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
if (url.pathname.endsWith('/alertPolicies')) {
return Response.json({ alertPolicies: [] })
}
const filter = url.searchParams.get('filter') ?? ''
if (filter.includes('orca_relay_controls')) {
return Response.json({ timeSeries: [
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
points: [
distribution('2026-07-15T12:00:10Z', 1),
distribution('2026-07-15T12:00:50Z', 2)
]
},
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'two' } },
points: [distribution('2026-07-15T12:00:20Z', 3)]
},
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'stale' } },
points: [distribution('2026-07-15T11:59:20Z', 100)]
}
] })
}
if (filter.includes('orca_relay_forwarded_bytes')) {
return Response.json({ timeSeries: [{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
points: [distribution('2026-07-15T12:00:20Z', 10, 4)]
}] })
}
return Response.json({ timeSeries: [] })
}
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, gcloud, {
now: new Date('2026-07-15T12:01:00Z'),
windowMinutes: 30,
fetchImpl
})
expect(result.warnings).toEqual([])
expect(result.metrics.controls.points).toEqual([
{ at: '2026-07-15T11:59:00.000Z', value: 100 },
{ at: '2026-07-15T12:00:00.000Z', value: 5 }
])
expect(result.metrics.controls.latestByCell).toEqual({ 'production-gce-c1': 5 })
expect(result.metrics.forwarded_bytes.latest).toBe(40)
expect(result.metrics.postgres_retries.available).toBe(true)
expect(Object.keys(result.metrics)).toHaveLength(RELAY_METRICS.length)
})
it('degrades safely when credentials are unavailable', async () => {
const unavailable: GcloudClient = {
accessToken: async () => { throw new Error('sensitive context') }
}
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, unavailable)
expect(result.warnings).toEqual([
'Cloud Monitoring credentials are unavailable. Run gcloud auth login.'
])
expect(result.metrics.postgres_retries.available).toBe(false)
expect(JSON.stringify(result)).not.toContain('sensitive context')
})
})
@@ -0,0 +1,379 @@
import { z } from 'zod'
import type { RelayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
type MetricMode = 'gauge-sum' | 'delta-sum' | 'maximum'
export type RelayMetricName =
| 'total_connections'
| 'controls'
| 'splices'
| 'pending_splices'
| 'queued_bytes'
| 'http_latency_ms'
| 'sql_latency_ms'
| 'heap_used_bytes'
| 'event_loop_ms_p99'
| 'forwarded_bytes'
| 'auth_successes'
| 'auth_failures'
| 'reconnects'
| 'sql_queries'
| 'sql_failures'
| 'assignment_5xx'
| 'postgres_retries'
| 'postgres_retry_exhausted'
| 'db_pool_total'
| 'db_pool_idle'
| 'db_pool_waiting'
| 'db_waiters_max'
| 'db_oldest_wait_ms'
| 'db_wait_ms_max'
type MetricDefinition = {
name: RelayMetricName
label: string
unit: 'count' | 'bytes' | 'milliseconds'
mode: MetricMode
}
export const RELAY_METRICS: MetricDefinition[] = [
{ name: 'total_connections', label: 'Connections', unit: 'count', mode: 'gauge-sum' },
{ name: 'controls', label: 'Desktop controls', unit: 'count', mode: 'gauge-sum' },
{ name: 'splices', label: 'Phone splices', unit: 'count', mode: 'gauge-sum' },
{ name: 'pending_splices', label: 'Pending splices', unit: 'count', mode: 'gauge-sum' },
{ name: 'queued_bytes', label: 'Queued bytes', unit: 'bytes', mode: 'maximum' },
{ name: 'http_latency_ms', label: 'HTTP latency', unit: 'milliseconds', mode: 'maximum' },
{ name: 'sql_latency_ms', label: 'SQL latency', unit: 'milliseconds', mode: 'maximum' },
{ name: 'heap_used_bytes', label: 'Heap used', unit: 'bytes', mode: 'maximum' },
{
name: 'event_loop_ms_p99',
label: 'Event-loop p99',
unit: 'milliseconds',
mode: 'maximum'
},
{ name: 'forwarded_bytes', label: 'Forwarded bytes', unit: 'bytes', mode: 'delta-sum' },
{ name: 'auth_successes', label: 'Auth successes', unit: 'count', mode: 'delta-sum' },
{ name: 'auth_failures', label: 'Auth failures', unit: 'count', mode: 'delta-sum' },
{ name: 'reconnects', label: 'Reconnects', unit: 'count', mode: 'delta-sum' },
{ name: 'sql_queries', label: 'SQL queries', unit: 'count', mode: 'delta-sum' },
{ name: 'sql_failures', label: 'SQL failures', unit: 'count', mode: 'delta-sum' },
{ name: 'assignment_5xx', label: 'Assignment 5xx', unit: 'count', mode: 'delta-sum' },
{
name: 'postgres_retries',
label: 'PostgreSQL retries',
unit: 'count',
mode: 'delta-sum'
},
{
name: 'postgres_retry_exhausted',
label: 'PostgreSQL retry exhausted',
unit: 'count',
mode: 'delta-sum'
},
{ name: 'db_pool_total', label: 'Database pool total', unit: 'count', mode: 'gauge-sum' },
{ name: 'db_pool_idle', label: 'Database pool idle', unit: 'count', mode: 'gauge-sum' },
{
name: 'db_pool_waiting',
label: 'Database pool waiting',
unit: 'count',
mode: 'gauge-sum'
},
{
name: 'db_waiters_max',
label: 'Database waiters max',
unit: 'count',
mode: 'maximum'
},
{
name: 'db_oldest_wait_ms',
label: 'Database oldest wait',
unit: 'milliseconds',
mode: 'maximum'
},
{
name: 'db_wait_ms_max',
label: 'Database wait max',
unit: 'milliseconds',
mode: 'maximum'
}
]
const NumericSchema = z.union([z.number(), z.string()]).transform((value) => Number(value))
const DistributionSchema = z.object({
count: NumericSchema.default(0),
mean: NumericSchema.optional()
})
const PointSchema = z.object({
interval: z.object({ endTime: z.string() }),
value: z.object({
doubleValue: NumericSchema.optional(),
int64Value: NumericSchema.optional(),
distributionValue: DistributionSchema.optional()
})
})
const TimeSeriesSchema = z.object({
metric: z.object({ labels: z.record(z.string()).default({}) }),
resource: z.object({ type: z.string(), labels: z.record(z.string()).default({}) }),
points: z.array(PointSchema).default([])
})
const TimeSeriesResponseSchema = z.object({
timeSeries: z.array(TimeSeriesSchema).default([])
})
export type MetricPoint = { at: string; value: number }
export type RelayMetricSnapshot = MetricDefinition & {
available: boolean
points: MetricPoint[]
latest: number | null
latestAt: string | null
latestByCell: Record<string, number>
}
export type AlertPolicySnapshot = {
id: string
displayName: string
enabled: boolean
documentation: string | null
}
export type MonitoringSnapshot = {
startAt: string
endAt: string
resolutionSeconds: number
metrics: Record<RelayMetricName, RelayMetricSnapshot>
alertPolicies: AlertPolicySnapshot[]
warnings: string[]
}
type ParsedPoint = {
atMs: number
bucketMs: number
value: number
sampleTotal: number
seriesKey: string
cellId: string
}
function pointValue(point: z.infer<typeof PointSchema>): { value: number; sampleTotal: number } {
if (point.value.distributionValue) {
const count = point.value.distributionValue.count
const value = point.value.distributionValue.mean ?? 0
return { value, sampleTotal: value * count }
}
const value = point.value.doubleValue ?? point.value.int64Value ?? 0
return { value, sampleTotal: value }
}
function parsePoints(series: z.infer<typeof TimeSeriesSchema>[]): ParsedPoint[] {
return series.flatMap((entry) => {
const cellId = entry.metric.labels.cell_id ?? 'unknown'
const resourceId =
entry.resource.labels.instance_id ?? entry.resource.labels.revision_name ?? entry.resource.type
const seriesKey = `${cellId}:${resourceId}`
return entry.points.flatMap((point) => {
const atMs = Date.parse(point.interval.endTime)
if (!Number.isFinite(atMs)) return []
const values = pointValue(point)
return [{
atMs,
bucketMs: Math.floor(atMs / 60_000) * 60_000,
value: values.value,
sampleTotal: values.sampleTotal,
seriesKey,
cellId
}]
})
})
}
function aggregatePoints(points: ParsedPoint[], mode: MetricMode): MetricPoint[] {
if (mode === 'gauge-sum') {
const buckets = new Map<number, Map<string, ParsedPoint>>()
for (const point of points) {
const bySeries = buckets.get(point.bucketMs) ?? new Map<string, ParsedPoint>()
const previous = bySeries.get(point.seriesKey)
if (!previous || previous.atMs < point.atMs) bySeries.set(point.seriesKey, point)
buckets.set(point.bucketMs, bySeries)
}
return [...buckets.entries()]
.sort(([left], [right]) => left - right)
.map(([at, bySeries]) => ({
at: new Date(at).toISOString(),
value: [...bySeries.values()].reduce((total, point) => total + point.value, 0)
}))
}
const buckets = new Map<number, number>()
for (const point of points) {
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
const previous = buckets.get(point.bucketMs)
buckets.set(
point.bucketMs,
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
)
}
return [...buckets.entries()]
.sort(([left], [right]) => left - right)
.map(([at, value]) => ({ at: new Date(at).toISOString(), value }))
}
function latestByCell(points: ParsedPoint[], mode: MetricMode): Record<string, number> {
const newestBucketByCell = new Map<string, number>()
for (const point of points) {
newestBucketByCell.set(
point.cellId,
Math.max(newestBucketByCell.get(point.cellId) ?? 0, point.bucketMs)
)
}
const newestBySeries = new Map<string, ParsedPoint>()
for (const point of points) {
if (point.bucketMs !== newestBucketByCell.get(point.cellId)) continue
const previous = newestBySeries.get(point.seriesKey)
if (!previous || previous.atMs < point.atMs) newestBySeries.set(point.seriesKey, point)
}
const totals = new Map<string, number>()
for (const point of newestBySeries.values()) {
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
const previous = totals.get(point.cellId)
totals.set(
point.cellId,
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
)
}
return Object.fromEntries(totals)
}
async function monitoringRequest(
fetchImpl: typeof fetch,
token: string,
url: URL
): Promise<unknown> {
const response = await fetchImpl(url, {
headers: { authorization: `Bearer ${token}` },
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Cloud Monitoring returned ${response.status}`)
return await response.json()
}
async function readMetric(
environment: RelayOpsEnvironment,
definition: MetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch
): Promise<RelayMetricSnapshot> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
)
url.searchParams.set(
'filter',
`metric.type="logging.googleapis.com/user/orca_relay_${definition.name}"`
)
url.searchParams.set('interval.startTime', startAt)
url.searchParams.set('interval.endTime', endAt)
url.searchParams.set('view', 'FULL')
url.searchParams.set('pageSize', '1000')
const body = TimeSeriesResponseSchema.parse(
await monitoringRequest(fetchImpl, token, url)
)
const parsed = parsePoints(body.timeSeries)
const points = aggregatePoints(parsed, definition.mode)
const latest = points.at(-1) ?? null
return {
...definition,
available: true,
points,
latest: latest?.value ?? null,
latestAt: latest?.at ?? null,
latestByCell: latestByCell(parsed, definition.mode)
}
}
async function readAlertPolicies(
environment: RelayOpsEnvironment,
token: string,
fetchImpl: typeof fetch
): Promise<AlertPolicySnapshot[]> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/alertPolicies`
)
url.searchParams.set('pageSize', '100')
const body = (await monitoringRequest(fetchImpl, token, url)) as {
alertPolicies?: Array<{
name?: string
displayName?: string
enabled?: boolean
documentation?: { content?: string }
}>
}
return (body.alertPolicies ?? [])
.filter((policy) => policy.displayName?.startsWith('Orca Relay:'))
.map((policy) => ({
id: policy.name ?? '',
displayName: policy.displayName ?? 'Orca Relay alert',
enabled: policy.enabled === true,
documentation: policy.documentation?.content ?? null
}))
.sort((left, right) => left.displayName.localeCompare(right.displayName))
}
function emptyMetric(definition: MetricDefinition): RelayMetricSnapshot {
return {
...definition,
available: false,
points: [],
latest: null,
latestAt: null,
latestByCell: {}
}
}
export async function readMonitoringSnapshot(
environment: RelayOpsEnvironment,
gcloud: GcloudClient,
options: { now?: Date; windowMinutes?: number; fetchImpl?: typeof fetch } = {}
): Promise<MonitoringSnapshot> {
const now = options.now ?? new Date()
const windowMinutes = Math.min(24 * 60, Math.max(30, options.windowMinutes ?? 360))
const endAt = now.toISOString()
const startAt = new Date(now.getTime() - windowMinutes * 60_000).toISOString()
const fetchImpl = options.fetchImpl ?? fetch
const warnings: string[] = []
let token: string
try {
token = await gcloud.accessToken()
} catch {
return {
startAt,
endAt,
resolutionSeconds: 60,
metrics: Object.fromEntries(
RELAY_METRICS.map((definition) => [definition.name, emptyMetric(definition)])
) as Record<RelayMetricName, RelayMetricSnapshot>,
alertPolicies: [],
warnings: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
}
}
const settled = await Promise.allSettled(
RELAY_METRICS.map((definition) =>
readMetric(environment, definition, token, startAt, endAt, fetchImpl)
)
)
const metrics = {} as Record<RelayMetricName, RelayMetricSnapshot>
settled.forEach((result, index) => {
const definition = RELAY_METRICS[index]!
if (result.status === 'fulfilled') metrics[definition.name] = result.value
else {
metrics[definition.name] = emptyMetric(definition)
warnings.push(`${definition.label} metric is unavailable.`)
}
})
const alertPolicies = await readAlertPolicies(environment, token, fetchImpl).catch(() => {
warnings.push('Cloud Monitoring alert policies are unavailable.')
return []
})
return { startAt, endAt, resolutionSeconds: 60, metrics, alertPolicies, warnings }
}
@@ -0,0 +1,42 @@
import { readdirSync, readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import {
RELAY_GITHUB_REPOSITORY,
RELAY_WORKFLOW_FILE_PREFIX,
relayRepositoryApiPath,
relayWorkflowFile
} from './relay-repository.js'
const sourceDir = fileURLToPath(new URL('.', import.meta.url))
const sources = readdirSync(sourceDir)
.filter((name) => name.endsWith('.ts') && name !== 'relay-repository.ts')
.map((name) => ({ name, text: readFileSync(`${sourceDir}${name}`, 'utf8') }))
describe('relay repository identity', () => {
it('builds API paths and workflow filenames from the one repository name', () => {
expect(relayRepositoryApiPath('actions/runs')).toBe(`repos/${RELAY_GITHUB_REPOSITORY}/actions/runs`)
expect(relayWorkflowFile('power-relay-staging.yml')).toBe(
`${RELAY_WORKFLOW_FILE_PREFIX}power-relay-staging.yml`
)
})
// Why: the public-repo copy renames the repository and prefixes every workflow file. Both have to
// be one edit, so no other module may restate either.
it('is the only module naming a GitHub repository', () => {
for (const { name, text } of sources) {
expect(text, `${name} restates a GitHub repository`).not.toMatch(/stablyai\//)
}
})
it('is the only module naming a workflow file', () => {
for (const { name, text } of sources) {
for (const match of text.matchAll(/'([^']*\.yml)'/g)) {
const file = match[1] ?? ''
expect(text, `${name} names ${file} outside relayWorkflowFile`).toMatch(
new RegExp(`relayWorkflowFile\\('${file.replaceAll('.', '\\.')}'\\)`)
)
}
}
})
})
@@ -0,0 +1,14 @@
// Single place naming the GitHub repository that holds the Relay workflows. When the Relay tree is
// copied to its public repository, only this file changes: the repository moves and every workflow
// file gains a prefix, while the workflow display names stay as they are.
export const RELAY_GITHUB_REPOSITORY = 'stablyai/orca-cloud'
export const RELAY_WORKFLOW_FILE_PREFIX = ''
export function relayWorkflowFile(name: string): string {
return `${RELAY_WORKFLOW_FILE_PREFIX}${name}`
}
export function relayRepositoryApiPath(resource: string): string {
return `repos/${RELAY_GITHUB_REPOSITORY}/${resource}`
}
@@ -0,0 +1,150 @@
import { describe, expect, it } from 'vitest'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { probeEndpointHealth, readResourceInventory } from './resource-inventory.js'
const digest = `sha256:${'a'.repeat(64)}`
const runService = {
template: {
scaling: { minInstanceCount: 0, maxInstanceCount: 2 },
containers: [{ image: 'registry/image:tag' }]
},
conditions: [{ state: 'CONDITION_SUCCEEDED' }],
latestReadyRevision: 'projects/project/revisions/revision-one'
}
describe('readResourceInventory', () => {
it('does not delay a healthy endpoint sample', async () => {
let calls = 0
let waits = 0
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async () => {
calls += 1
return new Response(null, { status: 200 })
},
async () => {
waits += 1
}
)
expect(result.health).toBe(true)
expect(result.ready).toBe(true)
expect(calls).toBe(2)
expect(waits).toBe(0)
})
it('retries one transient endpoint failure within the same sample', async () => {
const calls = new Map<string, number>()
const waits: number[] = []
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async (input) => {
const path = new URL(String(input)).pathname
const call = (calls.get(path) ?? 0) + 1
calls.set(path, call)
return new Response(null, { status: path === '/ready' && call === 1 ? 503 : 200 })
},
async (ms) => {
waits.push(ms)
}
)
expect(result.health).toBe(true)
expect(result.ready).toBe(true)
expect(calls).toEqual(new Map([['/health', 2], ['/ready', 2]]))
expect(waits).toEqual([11_000])
})
it('fails closed when the endpoint retry is also unhealthy', async () => {
let calls = 0
const waits: number[] = []
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async () => {
calls += 1
return new Response(null, { status: 503 })
},
async (ms) => {
waits.push(ms)
}
)
expect(result.health).toBe(false)
expect(result.ready).toBe(false)
expect(calls).toBe(4)
expect(waits).toEqual([11_000])
})
it('uses aggregate REST inventory without probing sleeping staging endpoints', async () => {
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
let publicProbeCalls = 0
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
if (url.hostname.endsWith('onorca.dev')) {
publicProbeCalls += 1
return Response.json({ status: 'ok' })
}
if (url.hostname === 'run.googleapis.com') return Response.json(runService)
if (url.hostname === 'sqladmin.googleapis.com') return Response.json({
state: 'STOPPED',
databaseVersion: 'POSTGRES_17',
settings: {
activationPolicy: 'NEVER',
availabilityType: 'ZONAL',
tier: 'db-custom-1-3840'
}
})
if (url.hostname === 'certificatemanager.googleapis.com') return Response.json({
managed: { domains: ['*.relay-staging.onorca.dev'], state: 'ACTIVE' }
})
if (url.pathname.includes('/instanceGroupManagers/')) {
const name = url.pathname.split('/').at(-1)!
return Response.json({
name,
targetSize: 0,
size: '0',
instanceGroup: `projects/project/zones/zone/instanceGroups/${name}`,
instanceTemplate: `projects/project/global/instanceTemplates/template-${name}`,
status: { isStable: true }
})
}
if (url.pathname.includes('/instanceTemplates/')) return Response.json({
properties: { metadata: { items: [{
key: 'startup-script',
value: `SECRET_TEXT\nORCA_RELAY_IMAGE_DIGEST=%s\\n' '${digest}'`
}] } }
})
if (url.pathname.endsWith('/getHealth')) return Response.json([])
throw new Error(`Unexpected request to ${url.hostname}${url.pathname}`)
}
const result = await readResourceInventory(
RELAY_OPS_ENVIRONMENTS.staging,
gcloud,
fetchImpl
)
expect(publicProbeCalls).toBe(0)
expect(result.cells.every((cell) => cell.targetSize === 0)).toBe(true)
expect(result.cells.every((cell) => cell.endpoint.health === null)).toBe(true)
expect(result.cells.every((cell) => cell.imageDigest === digest)).toBe(true)
expect(JSON.stringify(result)).not.toContain('SECRET_TEXT')
})
it('represents missing credentials as unknown inventory, never sleeping', async () => {
const gcloud: GcloudClient = {
accessToken: async () => { throw new Error('sensitive context') }
}
let fetchCalls = 0
const result = await readResourceInventory(
RELAY_OPS_ENVIRONMENTS.production,
gcloud,
async () => { fetchCalls += 1; return Response.json({}) }
)
expect(fetchCalls).toBe(0)
expect(result.cells.every((cell) => cell.targetSize === null)).toBe(true)
expect(result.cells.every((cell) => cell.backendHealth === 'unknown')).toBe(true)
expect(JSON.stringify(result)).not.toContain('sensitive context')
})
})
@@ -0,0 +1,366 @@
import { z } from 'zod'
import type { RelayOpsEnvironment, RelayOpsCellConfig } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
const RunServiceSchema = z.object({
template: z.object({
scaling: z.object({
minInstanceCount: z.number().optional(),
maxInstanceCount: z.number().optional()
}).optional(),
containers: z.array(z.object({ image: z.string() })).min(1)
}),
conditions: z.array(z.object({ state: z.string() })).default([]),
latestReadyRevision: z.string().optional()
})
const SqlInstanceSchema = z.object({
state: z.string(),
databaseVersion: z.string(),
settings: z.object({
activationPolicy: z.string(),
availabilityType: z.string().optional(),
tier: z.string()
})
})
const MigSchema = z.object({
name: z.string(),
targetSize: z.number(),
size: z.union([z.string(), z.number()]).transform(Number).optional(),
instanceGroup: z.string(),
instanceTemplate: z.string(),
status: z.object({ isStable: z.boolean().default(false) }).default({ isStable: false })
})
const TemplateSchema = z.object({
properties: z.object({
metadata: z.object({
items: z.array(z.object({ key: z.string(), value: z.string().optional() })).default([])
}).optional()
})
})
const BackendHealthGroupSchema = z.object({
healthStatus: z.array(z.object({ healthState: z.string() })).default([])
})
const BackendHealthSchema = z.union([
BackendHealthGroupSchema,
z.array(z.object({ status: BackendHealthGroupSchema }))
])
const CertificateSchema = z.object({
expireTime: z.string().optional(),
managed: z.object({
domains: z.array(z.string()).default([]),
state: z.string()
})
})
export type EndpointHealth = {
health: boolean | null
ready: boolean | null
latencyMs: number | null
}
export type ServiceInventory = {
ready: boolean
revision: string | null
image: string
minInstances: number
maxInstances: number
}
export type CellInventory = RelayOpsCellConfig & {
migName: string
targetSize: number | null
runningInstances: number | null
stable: boolean | null
template: string | null
imageDigest: string | null
backendHealth: 'healthy' | 'unhealthy' | 'empty' | 'unknown'
endpoint: EndpointHealth
}
export type ResourceInventory = {
director: ServiceInventory | null
auth: ServiceInventory | null
sql: {
state: string
activationPolicy: string
tier: string
availabilityType: string
databaseVersion: string
} | null
certificate: { state: string; domains: string[]; expireTime: string | null } | null
directorEndpoint: EndpointHealth
authEndpoint: EndpointHealth
cells: CellInventory[]
warnings: string[]
}
const unavailableEndpoint = (): EndpointHealth => ({ health: null, ready: null, latencyMs: null })
const independentEndpointRetryDelayMs = 11_000
function finalSegment(value: string): string {
return value.split('/').at(-1) ?? value
}
function parseService(value: unknown): ServiceInventory {
const service = RunServiceSchema.parse(value)
return {
ready: service.conditions.length > 0 && service.conditions.every(
(condition) => condition.state === 'CONDITION_SUCCEEDED'
),
revision: service.latestReadyRevision ? finalSegment(service.latestReadyRevision) : null,
image: service.template.containers[0]!.image,
minInstances: service.template.scaling?.minInstanceCount ?? 0,
maxInstances: service.template.scaling?.maxInstanceCount ?? 0
}
}
async function googleRequest(
fetchImpl: typeof fetch,
token: string,
url: string,
init: RequestInit = {}
): Promise<unknown> {
const response = await fetchImpl(url, {
...init,
headers: {
authorization: `Bearer ${token}`,
...(init.body ? { 'content-type': 'application/json' } : {})
},
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Google API returned ${response.status}`)
return await response.json()
}
async function endpointProbe(origin: string, fetchImpl: typeof fetch): Promise<EndpointHealth> {
const startedAt = performance.now()
const check = async (path: '/health' | '/ready'): Promise<boolean> => {
try {
const response = await fetchImpl(`${origin}${path}`, {
redirect: 'error',
signal: AbortSignal.timeout(8_000)
})
return response.ok
} catch {
return false
}
}
const [health, ready] = await Promise.all([check('/health'), check('/ready')])
return { health, ready, latencyMs: Math.round(performance.now() - startedAt) }
}
export async function probeEndpointHealth(
origin: string,
fetchImpl: typeof fetch,
wait: (ms: number) => Promise<void> = async (ms) =>
await new Promise((resolvePromise) => setTimeout(resolvePromise, ms))
): Promise<EndpointHealth> {
const first = await endpointProbe(origin, fetchImpl)
if (
first.health &&
first.ready &&
first.latencyMs !== null &&
first.latencyMs <= INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
) {
return first
}
// Outwait Relay's ten-second readiness cache before treating the retry as independent.
await wait(independentEndpointRetryDelayMs)
return await endpointProbe(origin, fetchImpl)
}
function imageDigest(template: z.infer<typeof TemplateSchema>): string | null {
const startupScript = template.properties.metadata?.items.find(
(item) => item.key === 'startup-script'
)?.value
// Return only the immutable digest; startup metadata contains secret names and operational detail.
return startupScript?.match(/ORCA_RELAY_IMAGE_DIGEST=%s\\n' '(sha256:[a-f0-9]{64})'/)?.[1] ?? null
}
function backendState(value: unknown): CellInventory['backendHealth'] {
const parsedHealth = BackendHealthSchema.parse(value)
const groups = Array.isArray(parsedHealth)
? parsedHealth.map((group) => group.status)
: [parsedHealth]
const states = groups.flatMap((group) => group.healthStatus.map((status) => status.healthState))
if (states.length === 0) return 'empty'
if (states.every((state) => state === 'HEALTHY')) return 'healthy'
return 'unhealthy'
}
function unavailableCell(cell: RelayOpsCellConfig, migName: string): CellInventory {
return {
...cell,
migName,
targetSize: null,
runningInstances: null,
stable: null,
template: null,
imageDigest: null,
backendHealth: 'unknown',
endpoint: unavailableEndpoint()
}
}
async function readCell(
environment: RelayOpsEnvironment,
cell: RelayOpsCellConfig,
mig: z.infer<typeof MigSchema> | null,
token: string,
fetchImpl: typeof fetch
): Promise<CellInventory> {
const migName = `${environment.migPrefix}${cell.hostname}`
if (!mig) return unavailableCell(cell, migName)
// An empty fixed-one MIG cannot serve and must never be woken by observation.
const endpoint = mig.targetSize > 0
? await probeEndpointHealth(cell.origin, fetchImpl)
: unavailableEndpoint()
const templateName = finalSegment(mig.instanceTemplate)
const [templateResult, healthResult] = await Promise.allSettled([
googleRequest(
fetchImpl,
token,
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/instanceTemplates/${templateName}`
),
googleRequest(
fetchImpl,
token,
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/backendServices/${migName}/getHealth`,
{ method: 'POST', body: JSON.stringify({ group: mig.instanceGroup }) }
)
])
return {
...cell,
migName,
targetSize: mig.targetSize,
runningInstances: mig.size ?? (mig.status.isStable ? mig.targetSize : null),
stable: mig.status.isStable,
template: templateName,
imageDigest: templateResult.status === 'fulfilled'
? imageDigest(TemplateSchema.parse(templateResult.value))
: null,
backendHealth: healthResult.status === 'fulfilled'
? backendState(healthResult.value)
: 'unknown',
endpoint
}
}
function parsed<S extends z.ZodTypeAny>(
result: PromiseSettledResult<unknown>,
schema: S,
warning: string,
warnings: string[]
): z.infer<S> | null {
if (result.status === 'rejected') {
warnings.push(warning)
return null
}
const parsedValue = schema.safeParse(result.value)
if (!parsedValue.success) {
warnings.push(warning)
return null
}
return parsedValue.data
}
function unavailableInventory(environment: RelayOpsEnvironment, warning: string): ResourceInventory {
return {
director: null,
auth: null,
sql: null,
certificate: null,
directorEndpoint: unavailableEndpoint(),
authEndpoint: unavailableEndpoint(),
cells: environment.cells.map((cell) =>
unavailableCell(cell, `${environment.migPrefix}${cell.hostname}`)
),
warnings: [warning]
}
}
export async function readResourceInventory(
environment: RelayOpsEnvironment,
gcloud: GcloudClient,
fetchImpl: typeof fetch = fetch
): Promise<ResourceInventory> {
let token: string
try {
token = await gcloud.accessToken()
} catch {
return unavailableInventory(
environment,
'Google Cloud credentials are unavailable. Run gcloud auth login.'
)
}
const runUrl = (service: string) =>
`https://run.googleapis.com/v2/projects/${environment.project}/locations/${environment.region}/services/${service}`
const migUrl = (cell: RelayOpsCellConfig) =>
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/zones/${cell.zone}/instanceGroupManagers/${environment.migPrefix}${cell.hostname}`
const settled = await Promise.allSettled([
googleRequest(fetchImpl, token, runUrl(environment.directorService)),
googleRequest(fetchImpl, token, runUrl(environment.authService)),
googleRequest(
fetchImpl,
token,
`https://sqladmin.googleapis.com/sql/v1beta4/projects/${environment.project}/instances/${environment.sqlInstance}`
),
googleRequest(
fetchImpl,
token,
`https://certificatemanager.googleapis.com/v1/projects/${environment.project}/locations/global/certificates/${environment.certificateName}`
),
...environment.cells.map((cell) => googleRequest(fetchImpl, token, migUrl(cell)))
])
const warnings: string[] = []
const directorValue = parsed(settled[0]!, RunServiceSchema, 'Director service inventory is unavailable.', warnings)
const authValue = parsed(settled[1]!, RunServiceSchema, 'Auth service inventory is unavailable.', warnings)
const sqlValue = parsed(settled[2]!, SqlInstanceSchema, 'Cloud SQL inventory is unavailable.', warnings)
const certificateValue = parsed(
settled[3]!, CertificateSchema, 'TLS certificate inventory is unavailable.', warnings
)
const migValues = environment.cells.map((cell, index) => parsed(
settled[index + 4]!,
MigSchema,
`${cell.hostname.toUpperCase()} MIG inventory is unavailable.`,
warnings
))
const controlPlaneSleeping =
environment.id === 'staging' && sqlValue?.settings.activationPolicy === 'NEVER'
// Health probes would cold-start scale-to-zero Cloud Run services, so sleeping staging is inventory-only.
const [directorEndpoint, authEndpoint] = controlPlaneSleeping
? [unavailableEndpoint(), unavailableEndpoint()]
: await Promise.all([
probeEndpointHealth(environment.directorOrigin, fetchImpl),
probeEndpointHealth(environment.authOrigin, fetchImpl)
])
const cells = await Promise.all(environment.cells.map((cell, index) =>
readCell(environment, cell, migValues[index] ?? null, token, fetchImpl)
))
return {
director: directorValue ? parseService(directorValue) : null,
auth: authValue ? parseService(authValue) : null,
sql: sqlValue ? {
state: sqlValue.state,
activationPolicy: sqlValue.settings.activationPolicy,
tier: sqlValue.settings.tier,
availabilityType: sqlValue.settings.availabilityType ?? 'unknown',
databaseVersion: sqlValue.databaseVersion
} : null,
certificate: certificateValue ? {
state: certificateValue.managed.state,
domains: certificateValue.managed.domains,
expireTime: certificateValue.expireTime ?? null
} : null,
directorEndpoint,
authEndpoint,
cells,
warnings
}
}
@@ -0,0 +1,19 @@
import { describe, expect, it } from 'vitest'
import { parseStagingPowerRequest } from './staging-workflow.js'
describe('parseStagingPowerRequest', () => {
it('accepts the exact reviewed confirmations', () => {
expect(parseStagingPowerRequest({ mode: 'status', confirmation: '' })).toEqual({
mode: 'status', confirmation: ''
})
expect(parseStagingPowerRequest({ mode: 'wake', confirmation: 'WAKE_STAGING' }).mode).toBe('wake')
expect(parseStagingPowerRequest({ mode: 'sleep', confirmation: 'SLEEP_STAGING' }).mode).toBe('sleep')
})
it('rejects missing, swapped, or additional fields', () => {
expect(() => parseStagingPowerRequest({ mode: 'wake', confirmation: '' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'sleep', confirmation: 'WAKE_STAGING' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'production', confirmation: '' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'status', confirmation: '', project: 'other' })).toThrow()
})
})
@@ -0,0 +1,36 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { z } from 'zod'
import { RELAY_GITHUB_REPOSITORY, relayWorkflowFile } from './relay-repository.js'
const execFileAsync = promisify(execFile)
const DispatchSchema = z.discriminatedUnion('mode', [
z.object({ mode: z.literal('status'), confirmation: z.literal('') }).strict(),
z.object({ mode: z.literal('wake'), confirmation: z.literal('WAKE_STAGING') }).strict(),
z.object({ mode: z.literal('sleep'), confirmation: z.literal('SLEEP_STAGING') }).strict()
])
export type StagingPowerRequest = z.infer<typeof DispatchSchema>
export function parseStagingPowerRequest(value: unknown): StagingPowerRequest {
return DispatchSchema.parse(value)
}
export async function dispatchStagingPowerWorkflow(request: StagingPowerRequest): Promise<void> {
const args = [
'workflow', 'run', relayWorkflowFile('power-relay-staging.yml'),
'--repo', RELAY_GITHUB_REPOSITORY,
'-f', `mode=${request.mode}`,
'-f', 'wake-cells=configured'
]
if (request.confirmation) args.push('-f', `confirmation=${request.confirmation}`)
try {
await execFileAsync('gh', args, {
encoding: 'utf8',
timeout: 30_000,
maxBuffer: 1024 * 1024
})
} catch {
throw new Error('Staging power workflow dispatch failed')
}
}
+4
View File
@@ -0,0 +1,4 @@
{
"extends": "./tsconfig.json",
"exclude": ["src/**/*.test.ts"]
}
+11
View File
@@ -0,0 +1,11 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "dist",
"types": ["node", "vitest"],
"exactOptionalPropertyTypes": true,
"verbatimModuleSyntax": true
},
"include": ["src/**/*.ts"]
}
+25
View File
@@ -0,0 +1,25 @@
FROM node:24-alpine AS build
WORKDIR /app
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json ./
COPY packages/relay-contract/package.json packages/relay-contract/package.json
COPY apps/relay/package.json apps/relay/package.json
RUN pnpm install --frozen-lockfile
COPY packages/relay-contract packages/relay-contract
COPY apps/relay apps/relay
RUN pnpm --filter @orca-cloud/relay-contract build && pnpm --filter @orca-cloud/relay build
FROM node:24-alpine AS runtime
ENV NODE_ENV=production
ENV PORT=8080
WORKDIR /app
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY packages/relay-contract/package.json packages/relay-contract/package.json
COPY apps/relay/package.json apps/relay/package.json
COPY --from=build /app/packages/relay-contract/dist packages/relay-contract/dist
COPY --from=build /app/apps/relay/dist apps/relay/dist
RUN pnpm install --prod --frozen-lockfile --filter @orca-cloud/relay...
USER node
EXPOSE 8080
CMD ["node", "apps/relay/dist/index.js"]

Some files were not shown because too many files have changed in this diff Show More