chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)

Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
This commit is contained in:
Jinwoo Hong
2026-09-03 06:55:14 -04:00
committed by GitHub
parent 4cc0b8de61
commit 3eec77c11a
379 changed files with 102741 additions and 8 deletions
+5
View File
@@ -3,3 +3,8 @@
/config/scripts/*localization*.mjs @brennanb2025
/config/scripts/*locale*.mjs @brennanb2025
/config/i18next.config.ts @brennanb2025
# The relay's deploy and operate surface: workspace, workflows, and the rollout lease action.
/cloud/ @Jinwoo-H
/.github/workflows/cloud-*.yml @Jinwoo-H
/.github/actions/cloud-sql-rollout-lease/ @Jinwoo-H
@@ -0,0 +1,152 @@
# Cloud SQL rollout lease
A compare-and-swap lease on one Cloud Storage object, used to serialize Cloud SQL
**connection-budget** rollouts across two repositories.
`concurrency.group: production-cloud-sql-rollout` only serializes runs inside a single repository.
Once the relay workflows live in `stablyai/orca` and the app workflows stay in
`stablyai/orca-cloud`, there are two independent queues pointed at one shared Cloud SQL instance.
`relay-cloud-sql-connection-budget.mjs` computes `rolloutOverlap` as a `Math.max` over the relay
director, api, auth and relay-cell candidates, which is only sound when exactly one rollout is in
flight. This lease is what keeps that assumption true. Keep the per-repo concurrency groups **and**
the lease; they solve different halves of the problem.
## What it protects
No workflow runs a Cloud SQL schema migration. Every locked workflow either deploys a Cloud Run
revision or applies a GCE instance template against the shared instance, so the lease must cover
**all rollouts**, not just migrations.
## Usage
The lease step must run **after** `google-github-actions/setup-gcloud`, and after
`actions/checkout``uses: ./.github/actions/...` resolves against the checked-out workspace.
It belongs in the first job of the workflow that holds a GCP credential, which is not always the
gate job: `deploy-relay-production-same-cap`'s gate runs no `gcloud`, so its first acquire happens
in the first cell job.
```yaml
- uses: google-github-actions/auth@v2
with: { workload_identity_provider: ..., service_account: ... }
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
```
Buckets and objects in use:
| Environment | Bucket | Object |
| ----------- | -------------------------------------- | --------------------------------------------------- |
| production | `onorca-cloud-terraform-state` | `terraform/state/cloud-sql-rollout/production.lock` |
| staging | `onorca-cloud-staging-terraform-state` | `terraform/state/cloud-sql-rollout/staging.lock` |
Workflows that serve both environments (`deploy-relay-asia-topology`,
`operate-relay-asia-admission`) select the pair with an `inputs.environment == 'production'`
ternary on both `bucket` and `object`. `deploy-staging` keeps its own `deploy-artifacts-staging`
concurrency group but takes the staging lease, because it rolls the staging API revision.
The object sits beside `terraform/state/relay-fence-broker/<env>.lock`. The IAM grant names both the
relay and app service accounts, so it is a **foundation-root** resource: `roles/storage.objectAdmin`
conditioned on the `terraform/state/cloud-sql-rollout/` prefix, **plus** an unconditioned
`roles/storage.legacyBucketReader`. Without the second role the generation-matched write fails in a
way that looks like a permissions flake.
## One lease per run, not per job
`deploy-relay-production-capacity` calls its reusable job six times and
`deploy-relay-production-same-cap` four times. Each call is a separate job on a separate runner, so
a naive per-job acquire/release would leave the object free between waves — for runs that have taken
up to 85 minutes.
The lease is therefore keyed to the **run**, not the job. `holder-key` defaults to
`${{ github.repository }}/${{ github.run_id }}`, and a job that finds its own holder key on a live
lease **re-enters** it: the record is refreshed, not rejected. Every job in the chain acquires; only
the last one releases.
```yaml
jobs:
gate:
steps:
- uses: ./.github/actions/cloud-sql-rollout-lease
with: { bucket: ..., object: ..., release: 'false' } # intermediate
wave-1: # ... release: 'false' on every wave job
release_lease:
needs: [gate, wave-1, wave-2, wave-3, wave-4]
if: always()
steps:
- uses: google-github-actions/auth@v2
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with: { bucket: ..., object: ..., release: 'true' } # final
```
`release: 'false'` still acquires and still runs its `post` step; `post` only skips the delete. A
single-job workflow leaves `release` at its `true` default and needs no extra job. So does a
workflow whose several jobs can never hold the lease at once: `prove-relay-staging-capacity`'s two
lease-holding jobs are guarded by complementary `inputs.mode` conditions, and the contract test
checks that exclusivity rather than assuming it.
If the final job never runs (runner killed, run cancelled hard), the lease expires on its TTL.
## Timing
- **TTL 35 minutes**, matching `apps/relay-fence-broker/src/mutation-lease.ts`.
- **Renewal every 5 minutes.** `main` spawns a detached background Node process that rewrites
`expires_at` on the same generation-matched path; `post` kills it by pid read back from
`$GITHUB_STATE`. The renewer stops on its own the moment the object stops being ours, and has a
six-hour backstop in case `post` never runs. Its log is written to
`$RUNNER_TEMP/cloud-sql-rollout-lease-renewer.log` and echoed by `post`.
- Renewal is mandatory, not optional: capacity runs have taken 85 minutes, well past any sane TTL.
## Failure behaviour
| Situation | Behaviour |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Object absent | Acquire with `ifGenerationMatch: 0`. |
| Live lease, our own holder key | Re-enter. Refresh `expires_at`, keep `acquired_at`. Never fails. |
| Live lease, another holder | **Fail the job immediately**, printing the holder's repository, workflow and run URL. Never queues, never steals. |
| Expired lease | Take over with the observed generation and emit `::warning::` naming the stale holder. |
| `412` on write | Someone raced us. Fail as a conflict. |
| Bucket unreachable, `403`, `5xx` | **Fail closed.** |
| Record present but unparseable | **Fail closed.** A record we cannot read is never treated as free; an operator must inspect and delete it. |
| Release finds a foreign holder | Warn and leave it alone. Our lease had already expired. |
| Release fails | Warn only. `post` never fails a job over a release; the TTL bounds the damage. |
## Why `monitor-relay-production` must not use this
`monitor-relay-production` is in the `production-cloud-sql-rollout` concurrency group but is
**read-only**: its identity holds only monitoring, logging, Cloud SQL and compute _viewer_ roles,
and it runs `gcloud sql instances describe`, never a mutation. It consumes no connection budget.
Putting it on the durable lease would let a monitoring run block a real rollout, and a rollout block
monitoring exactly when an operator most needs it. Keep its same-repo concurrency group; keep it off
the lease. The lock census contract test records it in the not-a-candidate map with this reason.
## Token acquisition
`gcloud auth print-access-token`, not a hand-rolled exchange of the `external_account` credentials
file. Every consuming workflow already runs `setup-gcloud`, gcloud already handles every ADC flavour
including the service-account impersonation leg, and this action must stay zero-dependency because
it is duplicated by hand into the public repo. The GCE metadata server that
`apps/relay-fence-broker/src/google-metadata.ts` uses does **not** exist on GitHub or Blacksmith
runners; only the compare-and-swap algorithm is shared with the fence broker.
## Duplication
This directory is copied verbatim into `stablyai/orca`. It has no `package.json`, no
`node_modules`, and imports nothing outside itself — `action-contract.test.mjs` enforces all three.
Cross-repo consumption via `uses: stablyai/orca/.github/actions/...@<sha>` was rejected: it would
put public-repo code inside private app deploys that hold a production credential, and neither
repository protects `main` today.
## Tests
```
node --test .github/actions/cloud-sql-rollout-lease/
```
`storage-lease.test.mjs` drives the real compare-and-swap path against an in-memory Cloud Storage
fake that enforces generations. No network.
@@ -0,0 +1,52 @@
import assert from 'node:assert/strict'
import { readFileSync, readdirSync } from 'node:fs'
import { test } from 'node:test'
const here = new URL('./', import.meta.url)
const action = readFileSync(new URL('action.yml', here), 'utf8')
const modules = readdirSync(here).filter((name) => name.endsWith('.mjs'))
const shipped = modules.filter((name) => !name.endsWith('.test.mjs'))
test('is a node24 JavaScript action with an always-run post step', () => {
// A composite action has no `post:`, so the lease could never be released on cancel or failure.
assert.match(action, /^ {2}using: node24$/m)
assert.doesNotMatch(action, /using: composite/)
assert.match(action, /^ {2}main: main\.mjs$/m)
assert.match(action, /^ {2}post: post\.mjs$/m)
assert.match(action, /^ {2}post-if: always\(\)$/m)
})
test('declares the inputs the wave-chain callers depend on', () => {
for (const input of ['bucket:', 'object:', 'holder-key:', 'release:']) {
assert.match(action, new RegExp(`^ {2}${input}$`, 'm'), input)
}
assert.match(action, /default: \$\{\{ github\.repository \}\}\/\$\{\{ github\.run_id \}\}/)
assert.match(action, /default: 'true'/)
})
test('stays self-contained so it can be duplicated into the public repo', () => {
assert.deepEqual(
readdirSync(here).filter((name) => name === 'package.json' || name === 'node_modules'),
[],
'the action must run with zero installed dependencies'
)
for (const name of modules) {
const source = readFileSync(new URL(name, here), 'utf8')
for (const match of source.matchAll(/^import\b[\s\S]*?from '([^']+)'/gm)) {
const specifier = match[1]
const local = specifier.startsWith('.')
assert.ok(
specifier.startsWith('node:') || (local && !specifier.includes('..')),
`${name} imports ${specifier}; only node: builtins and same-directory modules are allowed`
)
}
}
})
test('never reaches for the GCE metadata server', () => {
// Runners have no metadata.google.internal; the fence broker's token path must not be copied.
for (const name of shipped) {
const source = readFileSync(new URL(name, here), 'utf8')
assert.doesNotMatch(source, /metadata\.google\.internal/, name)
}
})
@@ -0,0 +1,41 @@
name: Cloud SQL rollout lease
description: >-
Serialize Cloud SQL connection-budget rollouts across repositories with a compare-and-swap lease
on a Cloud Storage object. Fails immediately when another run holds the lease; never queues,
never steals.
inputs:
bucket:
description: Terraform state bucket that holds the lease object.
required: true
object:
description: Lease object name, e.g. terraform/state/cloud-sql-rollout/production.lock
required: true
holder-key:
description: >-
Identity that owns the lease. Every job in one run must pass the same value; a job that finds
its own holder key on a live lease re-enters it instead of failing.
required: false
default: ${{ github.repository }}/${{ github.run_id }}
release:
description: >-
Release the lease in the post step. Set to "false" on every job of a multi-job wave except the
final always() job, which sets "true".
required: false
default: 'true'
outputs:
holder-key:
description: The holder key written to the lease object.
generation:
description: Cloud Storage generation of the lease object after acquisition.
expires-at:
description: ISO-8601 instant at which the lease expires without renewal.
reentrant:
description: '"true" when this job re-entered a lease its own run already held.'
runs:
using: node24
main: main.mjs
post: post.mjs
post-if: always()
@@ -0,0 +1,46 @@
import { execFileSync } from 'node:child_process'
// DESIGN CHOICE: shell out to `gcloud auth print-access-token` instead of exchanging the
// external_account credentials file that google-github-actions/auth writes.
//
// Every workflow on the Cloud SQL rollout lease already runs google-github-actions/setup-gcloud
// right after auth (verified across all 11 mutating members), so gcloud is on PATH and already
// bound to the federated identity. Doing the exchange ourselves would mean reimplementing the STS
// token swap plus the service-account impersonation leg, in an action that must stay
// zero-dependency and is duplicated by hand into a second repo. gcloud already handles every ADC
// flavour and refreshes on its own. The metadata server is not an option: it does not exist on
// GitHub or Blacksmith runners.
//
// Consequence, documented in the README: the lease step MUST come after setup-gcloud.
const TOKEN_REUSE_MS = 40 * 60 * 1_000 // GCP access tokens live ~60 min; re-mint well before that.
export function createAccessTokenSource({ run = runGcloud, now = Date.now } = {}) {
let cached = null
return () => {
if (cached && cached.mintedAt + TOKEN_REUSE_MS > now()) {
return cached.token
}
const token = run()
if (!token) {
throw new Error('gcloud auth print-access-token returned an empty token')
}
cached = { token, mintedAt: now() }
return token
}
}
function runGcloud() {
const binary = process.platform === 'win32' ? 'gcloud.cmd' : 'gcloud'
try {
return execFileSync(binary, ['auth', 'print-access-token'], {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
timeout: 60_000
}).trim()
} catch (error) {
// Never surface stdout; it is the token on success and noise on failure.
const detail = String(error?.stderr ?? '').trim() || error?.message || 'unknown failure'
throw new Error(`could not mint a GCP access token via gcloud: ${detail}`)
}
}
@@ -0,0 +1,20 @@
// Who we claim to be on the lease object. Shared by main and the detached renewer so both agree
// on the holder key without re-deriving it from a different set of environment variables.
export function holderIdentity(explicitHolderKey) {
const repository = process.env.GITHUB_REPOSITORY ?? 'unknown'
const runId = process.env.GITHUB_RUN_ID ?? 'unknown'
const server = process.env.GITHUB_SERVER_URL ?? 'https://github.com'
const holderKey = explicitHolderKey || `${repository}/${runId}`
if (/[\r\n]/.test(holderKey)) {
throw new Error('holder-key must be single-line')
}
return {
holderKey,
repository,
workflow: process.env.GITHUB_WORKFLOW ?? 'unknown',
runId,
runUrl: `${server}/${repository}/actions/runs/${runId}`,
runAttempt: process.env.GITHUB_RUN_ATTEMPT ?? 'unknown'
}
}
@@ -0,0 +1,81 @@
import { spawn } from 'node:child_process'
import { fileURLToPath } from 'node:url'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { holderIdentity } from './holder-identity.mjs'
import { fail, input, notice, renewerLogPath, saveState, setOutput, warn } from './runner-state.mjs'
import { CloudSqlRolloutLease, LeaseConflict, describeHolder } from './storage-lease.mjs'
const bucket = input('bucket')
const objectName = input('object')
const release = input('release') !== 'false'
if (!bucket || !objectName) {
fail('cloud-sql-rollout-lease requires both `bucket` and `object`')
process.exit(1)
}
const holder = holderIdentity(input('holder-key'))
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource()
})
let claim
try {
claim = await lease.acquire(holder)
} catch (error) {
if (error instanceof LeaseConflict) {
fail(
`${error.message}. Cloud SQL rollouts are serialized across repositories; this run will not queue or steal the lease. Wait for the holder to finish, then re-run.`
)
if (error.holder) {
console.log(`Lease holder repository: ${error.holder.repository}`)
console.log(`Lease holder workflow: ${error.holder.workflow}`)
console.log(`Lease holder run: ${error.holder.run_url}`)
}
} else {
// Bucket unreachable, permission denied, unreadable record: fail closed.
fail(`could not acquire ${lease.uri}: ${error.message}`)
}
process.exit(1)
}
// Persist before anything else can throw, so `post` always releases what we hold.
saveState('acquired', 'true')
saveState('bucket', bucket)
saveState('object', objectName)
saveState('holder_key', holder.holderKey)
saveState('release', release ? 'true' : 'false')
setOutput('holder-key', holder.holderKey)
setOutput('generation', claim.generation)
setOutput('expires-at', new Date(claim.record.expires_at).toISOString())
setOutput('reentrant', claim.state === 'reentrant' ? 'true' : 'false')
if (claim.state === 'reentrant') {
notice(
`Re-entered the Cloud SQL rollout lease on ${lease.uri} already held by this run; refreshed to ${new Date(claim.record.expires_at).toISOString()}.`
)
} else if (claim.state === 'takeover') {
notice(`Took over ${lease.uri} from ${describeHolder(claim.previous)}.`)
} else {
notice(
`Acquired ${lease.uri} until ${new Date(claim.record.expires_at).toISOString()} (holder ${holder.holderKey}).`
)
}
try {
const renewer = spawn(
process.execPath,
[fileURLToPath(new URL('./renew.mjs', import.meta.url)), bucket, objectName, holder.holderKey],
{ detached: true, stdio: 'ignore', env: process.env }
)
renewer.unref()
saveState('renewer_pid', String(renewer.pid))
const log = renewerLogPath()
notice(`Lease renewer running as pid ${renewer.pid}${log ? `, logging to ${log}` : ''}.`)
} catch (error) {
// A missing renewer is survivable for short jobs; the TTL still covers 35 minutes.
warn(`could not start the lease renewer: ${error.message}. The lease will expire on its TTL.`)
}
@@ -0,0 +1,81 @@
import { readFileSync } from 'node:fs'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { notice, renewerLogPath, savedState, warn } from './runner-state.mjs'
import { CloudSqlRolloutLease, describeHolder } from './storage-lease.mjs'
stopRenewer()
printRenewerLog()
if (savedState('acquired') !== 'true') {
notice('No Cloud SQL rollout lease was acquired by this step; nothing to release.')
process.exit(0)
}
const bucket = savedState('bucket')
const objectName = savedState('object')
const holderKey = savedState('holder_key')
if (savedState('release') !== 'true') {
notice(
`Holding gs://${bucket}/${objectName} for the rest of run ${holderKey}; a later job with release=true must free it.`
)
process.exit(0)
}
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource()
})
try {
const result = await lease.release(holderKey)
if (result.released) {
notice(`Released ${lease.uri} at generation ${result.generation}.`)
} else if (result.reason === 'absent') {
notice(`${lease.uri} was already gone; nothing to release.`)
} else if (result.reason === 'foreign') {
warn(
`${lease.uri} is now held by ${describeHolder(result.holder)}; leaving it alone. Our lease had already expired.`
)
} else {
warn(`${lease.uri} changed while releasing it; leaving it to expire on its TTL.`)
}
} catch (error) {
// Never fail a job in post over a release; the TTL bounds the damage to 35 minutes.
warn(`could not release ${lease.uri}: ${error.message}. It will expire on its TTL.`)
}
function stopRenewer() {
const pid = Number(savedState('renewer_pid'))
if (!Number.isInteger(pid) || pid <= 0) {
return
}
try {
process.kill(pid, 'SIGTERM')
notice(`Stopped the lease renewer (pid ${pid}).`)
} catch (error) {
if (error?.code !== 'ESRCH') {
warn(`could not stop the lease renewer ${pid}: ${error.message}`)
}
}
}
function printRenewerLog() {
const path = renewerLogPath()
if (!path) {
return
}
let text = ''
try {
text = readFileSync(path, 'utf8')
} catch {
return
}
if (!text.trim()) {
return
}
console.log('::group::Cloud SQL rollout lease renewer log')
console.log(text.trimEnd())
console.log('::endgroup::')
}
@@ -0,0 +1,73 @@
import { appendFileSync } from 'node:fs'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { renewerLogPath } from './runner-state.mjs'
import { CloudSqlRolloutLease, RENEW_INTERVAL_MS } from './storage-lease.mjs'
// Detached renewer. `main` spawns it, `post` kills it. It rewrites expires_at on the same
// generation-matched path as acquisition, and stops the moment the object stops being ours.
const MAX_LIFETIME_MS = 6 * 60 * 60 * 1_000 // Backstop if post never runs (runner killed).
const [bucket, objectName, holderKey] = process.argv.slice(2)
const logPath = renewerLogPath()
const startedAt = Date.now()
function log(message) {
if (!logPath) {
return
}
try {
appendFileSync(logPath, `${new Date().toISOString()} ${message}\n`)
} catch {
// A renewer that cannot log must still renew.
}
}
if (!bucket || !objectName || !holderKey) {
log('renewer started without bucket/object/holder-key; exiting')
process.exit(1)
}
const lease = new CloudSqlRolloutLease({
bucket,
objectName,
accessToken: createAccessTokenSource(),
warn: (message) => log(`warning ${message}`)
})
let stopping = false
for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) {
process.on(signal, () => {
stopping = true
log(`received ${signal}; stopping`)
process.exit(0)
})
}
log(`renewer started for ${lease.uri} holder=${holderKey} interval=${RENEW_INTERVAL_MS}ms`)
while (!stopping) {
await new Promise((resolve) => {
setTimeout(resolve, RENEW_INTERVAL_MS)
})
if (stopping) {
break
}
if (Date.now() - startedAt > MAX_LIFETIME_MS) {
log('renewer hit its maximum lifetime; stopping so the lease can expire')
break
}
try {
const result = await lease.renew(holderKey)
if (!result.renewed) {
log(`lease is no longer ours (${result.reason}); stopping`)
break
}
log(
`renewed until ${new Date(result.record.expires_at).toISOString()} at generation ${result.generation}`
)
} catch (error) {
// Transient GCS or token failures are retried on the next tick; the TTL covers 7 misses.
log(`renewal attempt failed: ${error.message}`)
}
}
@@ -0,0 +1,55 @@
import { appendFileSync } from 'node:fs'
// Action-state and output plumbing via the runner's file protocol, so the action needs no
// @actions/core dependency. Values are single-line by construction; anything else is rejected.
/** The detached renewer's stdio is ignored, so it appends here instead and `post` echoes it. */
export function renewerLogPath() {
const dir = process.env.RUNNER_TEMP
return dir ? `${dir}/cloud-sql-rollout-lease-renewer.log` : null
}
export function input(name) {
return (process.env[`INPUT_${name.replace(/ /g, '_').toUpperCase()}`] ?? '').trim()
}
export function savedState(name) {
return (process.env[`STATE_${name}`] ?? '').trim()
}
export function saveState(name, value) {
appendToEnvFile('GITHUB_STATE', name, value)
}
export function setOutput(name, value) {
appendToEnvFile('GITHUB_OUTPUT', name, value)
}
export function notice(message) {
console.log(`::notice::${oneLine(message)}`)
}
export function warn(message) {
console.log(`::warning::${oneLine(message)}`)
}
export function fail(message) {
console.log(`::error::${oneLine(message)}`)
process.exitCode = 1
}
function appendToEnvFile(variable, name, value) {
const text = String(value)
if (/[\r\n]/.test(text)) {
throw new Error(`${name} must be single-line`)
}
const path = process.env[variable]
if (!path) {
return
} // Running outside a runner (local smoke run); nothing to persist.
appendFileSync(path, `${name}=${text}\n`)
}
function oneLine(message) {
return String(message).replace(/\r?\n/g, ' ')
}
@@ -0,0 +1,240 @@
// Compare-and-swap lease over a single Cloud Storage object.
//
// Ported from apps/relay-fence-broker/src/mutation-lease.ts rather than imported: this action is
// duplicated verbatim into stablyai/orca, so it must carry no repo-local imports. Only the
// algorithm is shared (read metadata -> write with ifGenerationMatch -> 412 is a conflict ->
// generation-matched delete -> an expired record is free). The broker's token path is NOT shared;
// it reads the GCE metadata server, which does not exist on Actions runners.
export const LEASE_TTL_MS = 35 * 60 * 1_000
export const RENEW_INTERVAL_MS = 5 * 60 * 1_000
const GENERATION = /^[1-9][0-9]{0,30}$/
export class LeaseConflict extends Error {
constructor(message, holder) {
super(message)
this.name = 'LeaseConflict'
this.holder = holder ?? null
}
}
/** A live record we cannot parse is never treated as free; wedging beats double-rollout. */
export class LeaseUnreadable extends Error {
constructor(message) {
super(message)
this.name = 'LeaseUnreadable'
}
}
function parseRecord(raw) {
if (!raw || typeof raw !== 'object') {
return null
}
if (typeof raw.holder_key !== 'string' || raw.holder_key.length === 0) {
return null
}
if (!Number.isSafeInteger(raw.acquired_at) || !Number.isSafeInteger(raw.expires_at)) {
return null
}
return {
repository: typeof raw.repository === 'string' ? raw.repository : 'unknown',
workflow: typeof raw.workflow === 'string' ? raw.workflow : 'unknown',
run_id: typeof raw.run_id === 'string' ? raw.run_id : 'unknown',
run_url: typeof raw.run_url === 'string' ? raw.run_url : 'unknown',
run_attempt: typeof raw.run_attempt === 'string' ? raw.run_attempt : 'unknown',
acquired_at: raw.acquired_at,
expires_at: raw.expires_at,
holder_key: raw.holder_key
}
}
function describe(record) {
return `${record.repository} / ${record.workflow} (run ${record.run_id}, attempt ${record.run_attempt}) ${record.run_url}`
}
export class CloudSqlRolloutLease {
#bucket
#objectName
#accessToken
#fetcher
#now
#warn
constructor({
bucket,
objectName,
accessToken,
fetcher = fetch,
now = Date.now,
warn = (message) => console.log(`::warning::${message}`)
}) {
this.#bucket = bucket
this.#objectName = objectName
this.#accessToken = accessToken
this.#fetcher = fetcher
this.#now = now
this.#warn = warn
}
get uri() {
return `gs://${this.#bucket}/${this.#objectName}`
}
async acquire(holder) {
const existing = await this.read()
const now = this.#now()
if (!existing) {
return this.#claim(holder, '0', now, now, 'created')
}
if (existing.record.holder_key === holder.holderKey) {
// Same run, another job in the wave chain. Refresh, never fail.
return this.#claim(
holder,
existing.generation,
existing.record.acquired_at,
now,
'reentrant',
existing.record
)
}
if (existing.record.expires_at > now) {
throw new LeaseConflict(
`${this.uri} is held by ${describe(existing.record)} until ${new Date(existing.record.expires_at).toISOString()}`,
existing.record
)
}
this.#warn(
`Taking over an expired Cloud SQL rollout lease on ${this.uri}. Stale holder: ${describe(existing.record)}, expired ${new Date(existing.record.expires_at).toISOString()}.`
)
return this.#claim(holder, existing.generation, now, now, 'takeover', existing.record)
}
async renew(holderKey) {
const existing = await this.read()
if (!existing) {
return { renewed: false, reason: 'absent' }
}
if (existing.record.holder_key !== holderKey) {
return { renewed: false, reason: 'foreign' }
}
const now = this.#now()
const record = { ...existing.record, expires_at: now + LEASE_TTL_MS }
const written = await this.#write(record, existing.generation)
return { renewed: true, generation: written.generation, record }
}
async release(holderKey) {
const existing = await this.read()
if (!existing) {
return { released: false, reason: 'absent' }
}
if (existing.record.holder_key !== holderKey) {
return { released: false, reason: 'foreign', holder: existing.record }
}
const response = await this.#fetcher(
`${this.#metadataUrl()}?ifGenerationMatch=${encodeURIComponent(existing.generation)}`,
{ method: 'DELETE', headers: { Authorization: `Bearer ${await this.#token()}` } }
)
if (response.status === 412) {
return { released: false, reason: 'conflict' }
}
if (!response.ok && response.status !== 404) {
throw new Error(`lease release failed: ${response.status}`)
}
return { released: true, generation: existing.generation }
}
async read() {
const token = await this.#token()
const metadataResponse = await this.#fetcher(this.#metadataUrl(), {
headers: { Authorization: `Bearer ${token}` }
})
if (metadataResponse.status === 404) {
return null
}
if (!metadataResponse.ok) {
throw new Error(`lease inspection failed: ${metadataResponse.status}`)
}
const metadata = await metadataResponse.json()
if (!GENERATION.test(metadata?.generation ?? '')) {
throw new LeaseUnreadable(`${this.uri} has no valid generation`)
}
const bodyResponse = await this.#fetcher(`${this.#metadataUrl()}?alt=media`, {
headers: { Authorization: `Bearer ${token}` }
})
if (bodyResponse.status === 404) {
return null
}
if (!bodyResponse.ok) {
throw new Error(`lease body read failed: ${bodyResponse.status}`)
}
let raw = null
try {
raw = await bodyResponse.json()
} catch {
raw = null
}
const record = parseRecord(raw)
if (!record) {
throw new LeaseUnreadable(
`${this.uri} holds an unreadable lease record; an operator must inspect and delete it before rollouts can resume`
)
}
return { generation: metadata.generation, record }
}
async #claim(holder, generation, acquiredAt, now, state, previous) {
const record = {
repository: holder.repository,
workflow: holder.workflow,
run_id: holder.runId,
run_url: holder.runUrl,
run_attempt: holder.runAttempt,
acquired_at: acquiredAt,
expires_at: now + LEASE_TTL_MS,
holder_key: holder.holderKey
}
const written = await this.#write(record, generation)
return { state, generation: written.generation, record, previous: previous ?? null }
}
async #write(record, generation) {
const response = await this.#fetcher(
`${this.#uploadUrl()}&ifGenerationMatch=${encodeURIComponent(generation)}`,
{
method: 'POST',
headers: {
Authorization: `Bearer ${await this.#token()}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(record)
}
)
if (response.status === 412) {
throw new LeaseConflict(`${this.uri} changed concurrently while we were claiming it`)
}
if (!response.ok) {
throw new Error(`lease write failed: ${response.status}`)
}
const metadata = await response.json()
if (!GENERATION.test(metadata?.generation ?? '')) {
throw new LeaseUnreadable(`${this.uri} write returned no valid generation`)
}
return { generation: metadata.generation }
}
async #token() {
return typeof this.#accessToken === 'function' ? await this.#accessToken() : this.#accessToken
}
#metadataUrl() {
return `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(this.#bucket)}/o/${encodeURIComponent(this.#objectName)}`
}
#uploadUrl() {
return `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(this.#bucket)}/o?uploadType=media&name=${encodeURIComponent(this.#objectName)}`
}
}
export const describeHolder = describe
@@ -0,0 +1,324 @@
import assert from 'node:assert/strict'
import { test } from 'node:test'
import { createAccessTokenSource } from './gcloud-access-token.mjs'
import {
CloudSqlRolloutLease,
LEASE_TTL_MS,
LeaseConflict,
LeaseUnreadable
} from './storage-lease.mjs'
const BUCKET = 'onorca-cloud-terraform-state'
const OBJECT = 'terraform/state/cloud-sql-rollout/production.lock'
const NOW = 1_756_000_000_000
/**
* Enough of the Cloud Storage JSON API to exercise real compare-and-swap semantics: generations
* increment, ifGenerationMatch is enforced, and a mismatch is a 412. `faults` injects failures.
*/
function fakeStorage({ object = null, faults = [] } = {}) {
const state = { object, requests: [] }
const fetcher = async (rawUrl, init = {}) => {
const url = new URL(rawUrl)
const method = init.method ?? 'GET'
const record = { method, url, path: url.pathname, search: url.searchParams }
state.requests.push(record)
const fault = faults.find((candidate) => candidate.when(record))
if (fault) {
return json(fault.status, fault.body ?? {})
}
if (url.pathname.startsWith('/upload/')) {
const want = url.searchParams.get('ifGenerationMatch')
const have = state.object ? state.object.generation : '0'
if (want !== have) {
return json(412, {})
}
const generation = String(Number(have === '0' ? '1000' : have) + 1)
state.object = { generation, body: JSON.parse(init.body) }
return json(200, { generation })
}
if (method === 'DELETE') {
if (!state.object) {
return json(404, {})
}
if (url.searchParams.get('ifGenerationMatch') !== state.object.generation) {
return json(412, {})
}
state.object = null
return json(204, {})
}
if (!state.object) {
return json(404, {})
}
if (url.searchParams.get('alt') === 'media') {
return json(200, state.object.body)
}
return json(200, { generation: state.object.generation })
}
return { state, fetcher }
}
function json(status, body) {
return {
status,
ok: status >= 200 && status < 300,
json: async () => body
}
}
function storedRecord({
holderKey,
expiresAt,
repository = 'stablyai/orca',
workflow = 'Deploy Relay Production'
}) {
return {
repository,
workflow,
run_id: '9001',
run_url: 'https://github.com/stablyai/orca/actions/runs/9001',
run_attempt: '1',
acquired_at: NOW - 60_000,
expires_at: expiresAt,
holder_key: holderKey
}
}
function leaseFor(storage, { warn = () => {} } = {}) {
return new CloudSqlRolloutLease({
bucket: BUCKET,
objectName: OBJECT,
accessToken: 'test-token',
fetcher: storage.fetcher,
now: () => NOW,
warn
})
}
const HOLDER = {
holderKey: 'stablyai/orca-cloud/42',
repository: 'stablyai/orca-cloud',
workflow: 'Deploy Relay Production Same-Cap',
runId: '42',
runUrl: 'https://github.com/stablyai/orca-cloud/actions/runs/42',
runAttempt: '1'
}
test('acquires a lease on an empty object with ifGenerationMatch=0', async () => {
const storage = fakeStorage()
const claim = await leaseFor(storage).acquire(HOLDER)
assert.equal(claim.state, 'created')
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '0')
assert.equal(upload.search.get('name'), OBJECT)
assert.deepEqual(storage.state.object.body, {
repository: 'stablyai/orca-cloud',
workflow: 'Deploy Relay Production Same-Cap',
run_id: '42',
run_url: 'https://github.com/stablyai/orca-cloud/actions/runs/42',
run_attempt: '1',
acquired_at: NOW,
expires_at: NOW + LEASE_TTL_MS,
holder_key: 'stablyai/orca-cloud/42'
})
})
test('refuses a live lease held by another run and never writes', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseConflict)
assert.equal(error.holder.repository, 'stablyai/orca')
assert.equal(error.holder.run_url, 'https://github.com/stablyai/orca/actions/runs/9001')
assert.equal(
storage.state.requests.filter((request) => request.method !== 'GET').length,
0,
'a foreign live lease must not be written'
)
assert.equal(storage.state.object.generation, '1500')
})
test('re-enters a live lease this run already holds and extends it', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: HOLDER.holderKey, expiresAt: NOW + 60_000 })
}
})
const warnings = []
const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(
HOLDER
)
assert.equal(claim.state, 'reentrant')
assert.deepEqual(warnings, [], 're-entering our own lease is not a takeover')
assert.equal(claim.record.acquired_at, NOW - 60_000, 'original acquisition time is preserved')
assert.equal(claim.record.expires_at, NOW + LEASE_TTL_MS)
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '1500')
assert.equal(storage.state.object.generation, '1501')
})
test('takes over an expired lease and warns naming the stale holder', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({
holderKey: 'stablyai/orca/9001',
expiresAt: NOW - 1,
repository: 'stablyai/orca',
workflow: 'Deploy Relay Production Capacity'
})
}
})
const warnings = []
const claim = await leaseFor(storage, { warn: (message) => warnings.push(message) }).acquire(
HOLDER
)
assert.equal(claim.state, 'takeover')
assert.equal(warnings.length, 1)
assert.match(warnings[0], /stablyai\/orca/)
assert.match(warnings[0], /Deploy Relay Production Capacity/)
assert.match(warnings[0], /actions\/runs\/9001/)
const upload = storage.state.requests.find((request) => request.path.startsWith('/upload/'))
assert.equal(upload.search.get('ifGenerationMatch'), '1500')
assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey)
})
test('releases with a generation match and leaves the object gone', async () => {
const storage = fakeStorage()
const lease = leaseFor(storage)
const claim = await lease.acquire(HOLDER)
const released = await lease.release(HOLDER.holderKey)
assert.deepEqual(released, { released: true, generation: claim.generation })
const remove = storage.state.requests.find((request) => request.method === 'DELETE')
assert.equal(remove.search.get('ifGenerationMatch'), claim.generation)
assert.equal(storage.state.object, null)
})
test('refuses to release a lease another run now holds', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
const released = await leaseFor(storage).release(HOLDER.holderKey)
assert.equal(released.released, false)
assert.equal(released.reason, 'foreign')
assert.equal(storage.state.object.generation, '1500')
})
test('fails closed when the bucket answers 5xx', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.method === 'GET', status: 503 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.match(error.message, /lease inspection failed: 503/)
assert.equal(storage.state.requests.filter((request) => request.method !== 'GET').length, 0)
})
test('fails closed when permission is denied', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.method === 'GET', status: 403 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.match(error.message, /lease inspection failed: 403/)
})
test('treats an unreadable record as held, not free', async () => {
const storage = fakeStorage({
object: { generation: '1500', body: { holder_key: 'stablyai/orca/9001' } }
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseUnreadable)
assert.equal(storage.state.object.generation, '1500')
})
test('reports a 412 during acquisition as a conflict', async () => {
const storage = fakeStorage({
faults: [{ when: (request) => request.path.startsWith('/upload/'), status: 412 }]
})
const error = await leaseFor(storage)
.acquire(HOLDER)
.catch((thrown) => thrown)
assert.ok(error instanceof LeaseConflict)
assert.match(error.message, /changed concurrently/)
})
test('renewal rewrites only expires_at on the observed generation', async () => {
const storage = fakeStorage()
const lease = leaseFor(storage)
await lease.acquire(HOLDER)
storage.state.object.body.expires_at = NOW - 1
const renewed = await lease.renew(HOLDER.holderKey)
assert.equal(renewed.renewed, true)
assert.equal(storage.state.object.body.expires_at, NOW + LEASE_TTL_MS)
assert.equal(storage.state.object.body.acquired_at, NOW)
assert.equal(storage.state.object.body.holder_key, HOLDER.holderKey)
})
test('renewal stops once the object belongs to someone else', async () => {
const storage = fakeStorage({
object: {
generation: '1500',
body: storedRecord({ holderKey: 'stablyai/orca/9001', expiresAt: NOW + 60_000 })
}
})
assert.deepEqual(await leaseFor(storage).renew(HOLDER.holderKey), {
renewed: false,
reason: 'foreign'
})
})
test('the access token source re-mints only after the reuse window', () => {
let clock = 0
let mints = 0
const source = createAccessTokenSource({
run: () => `token-${++mints}`,
now: () => clock
})
assert.equal(source(), 'token-1')
clock = 39 * 60 * 1_000
assert.equal(source(), 'token-1')
clock = 41 * 60 * 1_000
assert.equal(source(), 'token-2')
})
test('the access token source rejects an empty gcloud response', () => {
const source = createAccessTokenSource({ run: () => '' })
assert.throws(() => source(), /empty token/)
})
@@ -13,6 +13,10 @@ function readRootEntries(sha) {
return stdout.split('\0').filter(Boolean)
}
// Why: the Cloud workspace import is the one reviewed root addition; it stays
// listed until it lands on main, after which the base tree carries it.
const REVIEWED_ROOT_ENTRIES = new Set(['cloud'])
function checkRootDirectoryEntries(argv) {
if (argv.length !== 2) {
console.error(`Usage: ${process.argv[1]} <base-sha> <head-sha>`)
@@ -21,7 +25,9 @@ function checkRootDirectoryEntries(argv) {
const [baseSha, headSha] = argv
const baseEntries = new Set(readRootEntries(baseSha))
const blockedEntries = readRootEntries(headSha).filter((entry) => !baseEntries.has(entry))
const blockedEntries = readRootEntries(headSha).filter(
(entry) => !baseEntries.has(entry) && !REVIEWED_ROOT_ENTRIES.has(entry)
)
if (blockedEntries.length === 0) {
console.log('Root directory guard passed: no new root-level files or folders.')
@@ -0,0 +1,676 @@
name: Bootstrap Relay Staging Capacity
on:
workflow_dispatch:
inputs:
confirmation:
description: Enter BOOTSTRAP_STAGING_CAPACITY
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
bootstrap:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: us-central1
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
LEGACY_C3_IMAGE_DIGEST: sha256:2d0f6e6db2b0eb9d6aba188698de8330f8c30b4e76badfcf0fac3f3eb9508a87
steps:
- uses: actions/checkout@v4
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: capacity-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: access_token
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit bootstrap confirmation
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "BOOTSTRAP_STAGING_CAPACITY"
- name: Read and verify reviewed 600/60 topology
shell: bash
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
> "${RUNNER_TEMP}/relay-gce-state.json"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
DESIRED_IMAGES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.image })' \
| jq -r '.')"
DESIRED_ZONES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.zone })' \
| jq -r '.')"
DESIRED_TARGET_SIZES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(local.relay_gce_cell_target_sizes)' | jq -r '.')"
jq -e \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson target_sizes "${DESIRED_TARGET_SIZES_JSON}" \
'([.[] | select(.id == "staging-gce-c2")] | length == 1) and
([.[] | select(.id == "staging-gce-c3")] | length == 1) and
(any(.[]; .id == "staging-gce-c2" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
(any(.[]; .id == "staging-gce-c3" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
($images["staging-gce-c2"] == $images["staging-gce-c3"]) and
($target_sizes["staging-gce-c2"] == 1) and
($target_sizes["staging-gce-c3"] == 1)' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
jq \
--argjson desired "${DESIRED_CELLS_JSON}" \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson zones "${DESIRED_ZONES_JSON}" \
'($desired | map({key: .id, value: .}) | from_entries) as $cells |
to_entries | map(. as $entry | {
key: $entry.key,
value: ($entry.value + {
origin: $cells[$entry.key].url,
image: $images[$entry.key],
zone: $zones[$entry.key],
connection_hard_cap: $cells[$entry.key].connectionHardCap,
connection_unobserved_bound: $cells[$entry.key].connectionUnobservedBound
})
}) | from_entries' \
"${RUNNER_TEMP}/relay-gce-state.json" \
> "${RUNNER_TEMP}/relay-gce-topology.json"
ACTIVE_REVISION="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
DESIRED_IMAGE="$(jq -r '.["staging-gce-c2"]' <<< "${DESIRED_IMAGES_JSON}")"
gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -e \
--arg image "${DESIRED_IMAGE}" \
--arg capacity_service_account "${CAPACITY_SERVICE_ACCOUNT}" \
'(.spec.containers[0].image == $image) and
any(.spec.containers[0].env[]?;
.name == "ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT" and
.value == $capacity_service_account)' >/dev/null
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
jq -e --argjson desired "${DESIRED_CELLS_JSON}" '
def without_bootstrap_capacity:
map(if .id == "staging-gce-c2" or .id == "staging-gce-c3"
then del(.connectionHardCap, .connectionUnobservedBound)
else . end);
without_bootstrap_capacity == ($desired | without_bootstrap_capacity)
' <<< "${CURRENT_CELLS_JSON}" >/dev/null
- name: Bootstrap C2 then C3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
topology="${RUNNER_TEMP}/relay-gce-topology.json"
fixed_one_instance_name() {
local cell_id="$1"
local mig_name zone
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instance-groups managed list-instances \
"${mig_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("legacy cell does not have one stable running instance") end'
}
fixed_one_instance_id() {
local cell_id="$1"
local instance_name="$2"
local zone
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instances describe "${instance_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format='value(id)'
}
write_legacy_metrics() {
local cell_id="$1"
local instance_id="$2"
local after="$3"
local output="$4"
gcloud logging read \
"resource.type=\"gce_instance\" AND
resource.labels.instance_id=\"${instance_id}\" AND
jsonPayload.event=\"orca_relay_runtime_metrics\" AND
jsonPayload.cellId=\"${cell_id}\" AND
timestamp>=\"${after}\"" \
--project "${GCP_PROJECT_ID}" \
--limit 10 \
--order desc \
--format json \
| jq '[.[] | {
timestamp,
cellId: .jsonPayload.cellId,
metricVersion: .jsonPayload.metricVersion,
totalConnections: .jsonPayload.totalConnections,
preAuthConnections: .jsonPayload.preAuthConnections,
controls: .jsonPayload.controls,
splices: .jsonPayload.splices,
pendingSplices: .jsonPayload.pendingSplices,
queuedBytes: .jsonPayload.queuedBytes
}]' > "${output}"
}
verify_legacy_cell() {
local cell_id="$1"
local admission="$2"
local after="$3"
local expected_instance_id="$4"
local runtime_started_after="${5:-}"
local previous_incarnation_digest="${6:-}"
local hard_cap="${7:-}"
local unobserved_bound="${8:-}"
local capacity_state="${9:-}"
local current_instance current_instance_id metrics origin result
local runtime_start_args=() incarnation_args=() capacity_args=()
current_instance="$(fixed_one_instance_name "${cell_id}")"
current_instance_id="$(fixed_one_instance_id "${cell_id}" "${current_instance}")"
test "${current_instance_id}" = "${expected_instance_id}"
metrics="${RUNNER_TEMP}/${cell_id}-legacy-runtime-metrics.json"
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
if test -n "${runtime_started_after}"; then
runtime_start_args=(--runtime-started-after "${runtime_started_after}")
fi
if test -n "${previous_incarnation_digest}"; then
incarnation_args=(--previous-incarnation-digest "${previous_incarnation_digest}")
fi
if test -n "${hard_cap}"; then
capacity_args=(--hard-cap "${hard_cap}" --unobserved-bound "${unobserved_bound}")
fi
if test -n "${capacity_state}"; then
capacity_args+=(--capacity-state "${capacity_state}")
fi
for _attempt in $(seq 1 18); do
write_legacy_metrics \
"${cell_id}" "${current_instance_id}" "${after}" "${metrics}"
if result="$(node dev/scripts/verify-relay-legacy-bootstrap.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--admission "${admission}" \
--expected-image-digest "${LEGACY_C3_IMAGE_DIGEST}" \
--metrics-after "${after}" \
--metrics-file "${metrics}" \
"${runtime_start_args[@]}" \
"${incarnation_args[@]}" \
"${capacity_args[@]}")"; then
echo "${result}"
return 0
fi
sleep 10
done
return 1
}
post_admin() {
local origin="$1"
local path="$2"
local body="$3"
curl --fail --silent --show-error \
--request POST \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${body}" \
"${origin}${path}"
}
runtime_kind() {
local cell_id="$1"
local origin desired_digest digest
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
desired_digest="$(jq -r --arg cell "${cell_id}" \
'.[$cell].image | split("@") | last' "${topology}")"
digest="$(post_admin "${origin}" /v1/admin/runtime-status '{"v":1}' \
| jq -er '.imageDigest')"
if test "${digest}" = "${LEGACY_C3_IMAGE_DIGEST}"; then
echo legacy
elif test "${digest}" = "${desired_digest}"; then
echo modern
else
echo 'bootstrap cell image is neither legacy nor reviewed' >&2
return 1
fi
}
cell_admission() {
local cell_id="$1"
post_admin "${DIRECTOR_ORIGIN}" /v1/admin/cell-status \
"$(jq -cn --arg cell "${cell_id}" '{v:1, cellId:$cell}')" \
| jq -er '.status.admissionState |
if . == "general" or . == "migration-only" then .
else error("bootstrap admission is not recoverable") end'
}
verify_modern_cell() {
local cell_id="$1"
local admission="$2"
local origin
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission "${admission}" \
--draining forbidden \
--activity allowed
}
ensure_modern_general() {
local cell_id="$1"
local admission="$2"
verify_modern_cell "${cell_id}" "${admission}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids "${cell_id}"
verify_modern_cell "${cell_id}" general
}
prepare_legacy_c3_fallback() {
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c3.relay-staging.onorca.dev
legacy_c3_restart_started_after=
legacy_c3_old_incarnation=
}
normalize_legacy_c3() {
legacy_pre_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c2_instance="$(fixed_one_instance_name staging-gce-c2)"
legacy_c2_instance_id="$(fixed_one_instance_id \
staging-gce-c2 "${legacy_c2_instance}")"
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c2.relay-staging.onorca.dev
legacy_c3_isolated=false
restore_legacy_c3_fallback() {
if test "${legacy_c3_isolated}" = true; then
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore-fallback \
--general-cell-ids staging-gce-c2
fi
}
trap restore_legacy_c3_fallback EXIT
legacy_c3_isolated=true
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin https://c3.relay-staging.onorca.dev \
--cell-id staging-gce-c3 \
--mode isolate
legacy_c3_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_drained="$(verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_drain_boundary}" "${legacy_c3_instance_id}")"
echo "${legacy_c3_drained}"
legacy_c3_old_incarnation="$(jq -er '.incarnationDigest' \
<<< "${legacy_c3_drained}")"
legacy_c3_restart_started_after="$(node -e \
'process.stdout.write(new Date().toISOString())')"
gcloud compute instance-groups managed recreate-instances \
orca-cloud-staging-relay-gce-c3 \
--instances "${legacy_c3_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--quiet
gcloud compute instance-groups managed wait-until \
orca-cloud-staging-relay-gce-c3 \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--timeout 900
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
legacy_c3_isolated=false
trap - EXIT
}
roll_cell() (
local cell_id="$1"
local fallback_cell_id="$2"
local target_kind="$3"
local fallback_kind="$4"
local active_revision cell_origin current_cells_json desired_bound desired_cap
local desired_cells_json director_result image mig_name plan
local fallback_origin plan_changes plan_result restored target_drain_boundary
local target_instance target_instance_id zone
cell_origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
fallback_origin="$(jq -r --arg cell "${fallback_cell_id}" \
'.[$cell].origin' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
image="$(jq -r --arg cell "${cell_id}" '.[$cell].image' "${topology}")"
desired_cap="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_hard_cap' "${topology}")"
desired_bound="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_unobserved_bound' "${topology}")"
plan="${RUNNER_TEMP}/${cell_id}-capacity-bootstrap.tfplan"
restored=false
restore_fallback() {
if test "${restored}" = false; then
verify_fallback
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore-fallback \
--general-cell-ids "${fallback_cell_id}"
fi
}
verify_fallback() {
if test "${fallback_kind}" = legacy; then
verify_legacy_cell \
"${fallback_cell_id}" general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
return
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${fallback_origin}" \
--cell-id "${fallback_cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
}
verify_fallback
trap restore_fallback EXIT
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--mode isolate
target_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
if test "${target_kind}" = legacy; then
target_instance="$(fixed_one_instance_name "${cell_id}")"
target_instance_id="$(fixed_one_instance_id "${cell_id}" "${target_instance}")"
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}" absent-or-stale
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
active_revision="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${active_revision}"
current_cells_json="$(gcloud run revisions describe "${active_revision}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
desired_cells_json="$(jq -ce \
--arg cell "${cell_id}" \
--argjson cap "${desired_cap}" \
--argjson bound "${desired_bound}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${current_cells_json}")"
director_result="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service orca-cloud-relay-staging \
--image "${image}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${cell_id}" \
--director-cells-json "${desired_cells_json}" \
--min-instances 0 \
--prune-revisions true \
--release-id "bootstrap-${cell_id}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}")"
echo "${director_result}"
if test "${target_kind}" = legacy; then
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}"
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap "${desired_cap}" \
--unobserved-bound "${desired_bound}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
-out="${plan}"
plan_result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--image "${image}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${plan_result}"
plan_changes="$(jq -r '.changes' <<< "${plan_result}")"
[[ "${plan_changes}" =~ ^(0|2)$ ]]
if test "${plan_changes}" = 2; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
else
target_instance="$(fixed_one_instance_name "${cell_id}")"
gcloud compute instance-groups managed recreate-instances "${mig_name}" \
--instances "${target_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--quiet
fi
gcloud compute instance-groups managed wait-until "${mig_name}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--timeout 900
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
restored=true
trap - EXIT
)
c2_kind="$(runtime_kind staging-gce-c2)"
c3_kind="$(runtime_kind staging-gce-c3)"
c2_admission="$(cell_admission staging-gce-c2)"
c3_admission="$(cell_admission staging-gce-c3)"
bootstrap_phase="$(node dev/scripts/classify-relay-staging-bootstrap.mjs \
--c2-kind "${c2_kind}" \
--c2-admission "${c2_admission}" \
--c3-kind "${c3_kind}" \
--c3-admission "${c3_admission}")"
jq -cn --arg phase "${bootstrap_phase}" \
'{event:"relay_staging_bootstrap_phase", phase:$phase}'
case "${bootstrap_phase}" in
normalize-and-roll-both)
normalize_legacy_c3
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
resume-c2-then-c3)
prepare_legacy_c3_fallback
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
roll-c2)
ensure_modern_general staging-gce-c3 "${c3_admission}"
roll_cell staging-gce-c2 staging-gce-c3 legacy modern
;;
roll-c3)
ensure_modern_general staging-gce-c2 "${c2_admission}"
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
complete)
ensure_modern_general staging-gce-c2 "${c2_admission}"
ensure_modern_general staging-gce-c3 "${c3_admission}"
;;
*)
echo 'unsupported staging bootstrap phase' >&2
exit 1
;;
esac
- name: Verify both bootstrapped cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
for number in 2 3; do
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://c${number}.relay-staging.onorca.dev" \
--cell-id "staging-gce-c${number}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
done
@@ -0,0 +1,245 @@
name: Deploy Relay Asia Topology
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Validate a saved plan or apply that exact plan
required: true
default: plan
type: choice
options: [plan, apply]
cell-ids:
description: Exact reviewed comma-separated Asia cell set
required: true
type: string
image:
description: Full environment Relay image pinned by sha256 digest
required: true
type: string
confirmation:
description: Enter APPLY_RELAY_ASIA_TOPOLOGY for apply mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
topology:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
TARGET_IMAGE: ${{ inputs.image }}
TARGET_REGION: asia-east2
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
CLOUD_SQL_INSTANCE: ${{ inputs.environment == 'production' && 'orca-cloud-auth-db' || 'orca-cloud-staging-auth-db' }}
VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER: db-custom-4-15360
VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION: POSTGRES_17
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TF_VARS: ${{ inputs.environment == 'production' && 'environments/production.tfvars' || 'environments/staging.tfvars' }}
TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
TOPOLOGY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate the reviewed request before authentication
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
test -n "${TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${TOPOLOGY_SERVICE_ACCOUNT}"
case "${TARGET_ENVIRONMENT}:${TARGET_CELL_IDS}" in
staging:staging-gce-c4) ;;
production:production-gce-c27,production-gce-c28,production-gce-c29) ;;
*) echo "cell-ids do not match the reviewed environment topology" >&2; exit 1 ;;
esac
[[ "${TARGET_IMAGE}" =~ ^us-central1-docker\.pkg\.dev/${GCP_PROJECT_ID}/orca-cloud/relay@sha256:[0-9a-f]{64}$ ]]
if test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = APPLY_RELAY_ASIA_TOPOLOGY
else
test "${DEPLOY_MODE}" = plan
test -z "${CONFIRMATION}"
fi
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.TOPOLOGY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
- name: Require the checked Cloud SQL connection budget
shell: bash
run: |
set -euo pipefail
budget="$(node dev/scripts/relay-cloud-sql-connection-budget.mjs)"
checked_max="$(jq -er '.maxConnections' <<< "${budget}")"
jq -e '.withinBudget == true' <<< "${budget}" >/dev/null
if test "${TARGET_ENVIRONMENT}" = production; then
instance="$(gcloud sql instances describe "${CLOUD_SQL_INSTANCE}" \
--project "${GCP_PROJECT_ID}" --format=json)"
live_flag="$(jq -er '[.settings.databaseFlags[]? |
select(.name == "max_connections") | .value] |
if length <= 1 then (.[0] // "") else error("duplicate max_connections flags") end' \
<<< "${instance}")"
if test -n "${live_flag}"; then
live_max="${live_flag}"
live_source=explicit-flag
else
# The verified production database uses Cloud SQL's 400-connection
# default for this exact shape; fail closed if its shape changes.
test "$(jq -er '.settings.tier' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER}"
test "$(jq -er '.databaseVersion' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION}"
live_max=400
live_source=verified-shape-default
fi
test "${live_max}" = "${checked_max}"
else
live_max="not-read-for-staging"
live_source=not-read-for-staging
fi
{
echo "### Relay Cloud SQL connection budget"
echo "- Checked maximum: ${checked_max}"
echo "- Configured maximum: $(jq -er '.configuredMaximum' <<< "${budget}")"
echo "- Rollout operating maximum: $(jq -er '.operatingMaximum' <<< "${budget}")"
echo "- Explicit reserve: $(jq -er '.explicitReserve' <<< "${budget}")"
echo "- Production live max_connections: ${live_max}"
echo "- Production live maximum source: ${live_source}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Initialize the exact environment state
run: terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
- id: targets
name: Build the exact additive target set
shell: bash
run: |
set -euo pipefail
file="${RUNNER_TEMP}/relay-asia-targets"
: > "${file}"
printf '%s\n' \
'-target=google_compute_subnetwork.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router_nat.relay_gce_additional["asia-east2"]' \
'-target=google_compute_url_map.relay_gce[0]' >> "${file}"
IFS=, read -ra cells <<< "${TARGET_CELL_IDS}"
for cell_id in "${cells[@]}"; do
printf '%s\n' \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_backend_service.relay_gce_cell[\"${cell_id}\"]" >> "${file}"
done
echo "file=${file}" >> "${GITHUB_OUTPUT}"
- name: Create and validate the saved topology plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-asia-topology.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology.json"
mapfile -t targets < "${{ steps.targets.outputs.file }}"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" \
-var manage_artifact_dns=false \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
committed="${RUNNER_TEMP}/relay-committed-asia-topology.json"
jq -e '{
relay_gce_cells: .variables.relay_gce_cells.value,
relay_gce_additional_region_subnetwork_cidrs:
.variables.relay_gce_additional_region_subnetwork_cidrs.value
}' "${plan_json}" > "${committed}"
node dev/scripts/prepare-relay-asia-topology-input.mjs \
--existing-json "${committed}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image "${TARGET_IMAGE}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
changes="$(jq -er '.changes' <<< "${result}")"
digest="$(sha256sum "${plan}" | awk '{print $1}')"
echo "plan=${plan}" >> "${GITHUB_OUTPUT}"
echo "changes=${changes}" >> "${GITHUB_OUTPUT}"
{
echo "### Relay Asia topology saved plan"
echo "- Environment: ${TARGET_ENVIRONMENT}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Region: ${TARGET_REGION}"
echo "- Mutating resources: ${changes}"
echo "- Saved-plan SHA-256: ${digest}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Apply only the validated saved plan
if: ${{ inputs.mode == 'apply' }}
run: terraform -chdir=infra/terraform apply -input=false -auto-approve "${{ steps.plan.outputs.plan }}"
- name: Prove the exact topology targets converged
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
set -euo pipefail
mapfile -t targets < "${{ steps.targets.outputs.file }}"
plan="${RUNNER_TEMP}/relay-asia-topology-readback.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology-readback.json"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" \
-var manage_artifact_dns=false \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
test "$(jq -er '.changes' <<< "${result}")" = 0
- name: Record the required selector-safe next step
if: ${{ inputs.mode == 'apply' }}
run: |
{
echo "### Required next step"
echo "The VMs are not eligible for ordinary placement yet."
echo "Register the exact new cells atomically as migration-only before any director configuration lists them."
echo "Rollback is migration-only admission; do not destroy the Asia network on rollout day."
} >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,90 @@
name: Deploy Relay Fence Broker
on:
workflow_dispatch:
inputs:
image-digest:
description: Immutable broker image digest built from this main commit
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
SERVICE_NAME: orca-cloud-relay-fence
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay-fence-broker
IMAGE_DIGEST: ${{ inputs.image-digest }}
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Resolve exact-commit broker image
run: |
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
TAGS="$(gcloud artifacts docker tags list "${IMAGE_REPOSITORY}" \
--filter="version:${IMAGE_DIGEST}" \
--format=json)"
jq -e --arg tag "/tags/sha-${GITHUB_SHA}" \
'any(.[]; .tag | endswith($tag))' <<< "${TAGS}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
- name: Deploy broker image only
run: |
gcloud run services update "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--image "${IMAGE}" \
--quiet
- name: Verify ready singleton revision
run: |
SERVICE="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
jq -e '.status.conditions[] | select(.type == "Ready" and .status == "True")' \
<<< "${SERVICE}"
REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) == 100)] |
if length == 1 then .[0].revisionName // empty else empty end' \
<<< "${SERVICE}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
@@ -0,0 +1,822 @@
name: Deploy Relay Production Capacity Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
target-cell-id:
required: true
type: string
confirmation:
required: true
type: string
monitor-run-id:
required: true
type: string
monitor-run-attempt:
required: true
type: string
evidence-mode:
required: true
type: string
wave-cell-ids:
required: true
type: string
wave-index:
required: true
type: string
source-wave-run-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 75
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
CAPACITY_CELL_IDS: production-gce-c7,production-gce-c8,production-gce-c9,production-gce-c10,production-gce-c13,production-gce-c14,production-gce-c15,production-gce-c16,production-gce-c19,production-gce-c20,production-gce-c21,production-gce-c22,production-gce-c23,production-gce-c24,production-gce-c25,production-gce-c26
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_DIRECTOR_IMAGE_DIGEST: sha256:01b7fc3e6dce66180034f268a2dc92c05458706c5b3a0dc4450dcdd6161f6e73
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DEPLOY_MODE: ${{ inputs.mode }}
EVIDENCE_MODE: ${{ inputs.evidence-mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
WAVE_INDEX: ${{ inputs.wave-index }}
SOURCE_WAVE_RUN_ID: ${{ inputs.source-wave-run-id }}
steps:
- name: Require exact reusable-workflow invocation
working-directory: .
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
if test "${EVIDENCE_MODE}" = continuation; then
test "${DEPLOY_MODE}" = apply
[[ "${WAVE_INDEX}" =~ ^[0-3]$ ]]
test "${WAVE_CELL_IDS}" != none
test "${SOURCE_WAVE_RUN_ID}" = none
elif test "${EVIDENCE_MODE}" = resume; then
test "${DEPLOY_MODE}" = apply
test "${WAVE_INDEX}" = resume
test "${WAVE_CELL_IDS}" != none
[[ "${SOURCE_WAVE_RUN_ID}" =~ ^[0-9]+$ ]]
else
test "${EVIDENCE_MODE}" = single
test "${WAVE_CELL_IDS}" = none
test "${WAVE_INDEX}" = 0
test "${SOURCE_WAVE_RUN_ID}" = none
fi
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
run: |
test -n "${GCP_REGION}"
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- id: resume-provenance
if: ${{ inputs.evidence-mode == 'resume' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
test "${MONITOR_RUN_ATTEMPT}" = 1
case "${MONITOR_RUN_ID}:${SOURCE_WAVE_RUN_ID}:${WAVE_CELL_IDS}:${TARGET_CELL_ID}" in
31554591366:31555510376:production-gce-c16,production-gce-c15,production-gce-c14,production-gce-c13:production-gce-c13)
EXPECTED_SHA=a917e8e1fc1a2654e8cb81ba39b57733ec56be9c
EXPECTED_SOURCE_ATTEMPT=1
;;
31562760783:31563664692:production-gce-c10,production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c10)
EXPECTED_SHA=6082e9ca89a918ca51f0c87db003f5e8805b64b7
EXPECTED_SOURCE_ATTEMPT=1
;;
31571019947:31572080665:production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c8)
EXPECTED_SHA=e59958130c9d9b7a6cd805df2678d08997842c7c
EXPECTED_SOURCE_ATTEMPT=2
;;
*) exit 1 ;;
esac
MONITOR_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${MONITOR_RUN_ID}" \
--jq 'select(.name == "Monitor Relay Production" and
.path == ".github/workflows/cloud-monitor-relay-production.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "success" and .run_attempt == 1) |
.head_sha')"
SOURCE_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq 'select(.name == "Deploy Relay Production Capacity" and
.path == ".github/workflows/cloud-deploy-relay-production-capacity.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "failure") |
.head_sha')"
SOURCE_ATTEMPT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq '.run_attempt')"
[[ "${MONITOR_SHA}" =~ ^[0-9a-f]{40}$ ]]
test "${MONITOR_SHA}" = "${EXPECTED_SHA}"
test "${SOURCE_SHA}" = "${MONITOR_SHA}"
test "${SOURCE_ATTEMPT}" = "${EXPECTED_SOURCE_ATTEMPT}"
echo "commit-sha=${MONITOR_SHA}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode == 'apply' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode == 'apply' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode == 'apply' }}
run: |
EVIDENCE_COMMIT_SHA="${GITHUB_SHA}"
if test "${EVIDENCE_MODE:-single}" = resume; then
EVIDENCE_COMMIT_SHA="${{ steps.resume-provenance.outputs.commit-sha }}"
fi
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${EVIDENCE_COMMIT_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- name: Download this workflow's wave authority
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Download the failed wave authority for resume
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ inputs.source-wave-run-id }}
- name: Require wave evidence consumed by this workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${GITHUB_RUN_ID}"
- name: Require wave evidence consumed by the failed source workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${SOURCE_WAVE_RUN_ID}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Require exact mutation confirmation
if: ${{ inputs.mode != 'verify' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if test "${EVIDENCE_MODE:-single}" = resume; then
test "${CONFIRMATION}" = "RESUME_SELECTED_CELL_TO_1000 ${TARGET_CELL_ID}"
elif test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = "RAISE_SELECTED_CELL_TO_1000"
else
test "${CONFIRMATION}" = "ROLL_BACK_SELECTED_CELL_TO_600 ${TARGET_CELL_ID}"
fi
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Build the exact selected-cell configuration
shell: bash
run: |
if test "${DEPLOY_MODE}" = rollback; then
TARGET_HARD_CAP=600
else
TARGET_HARD_CAP=1000
fi
TARGET_UNOBSERVED_BOUND=60
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
[[ "${TARGET_HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
OVERRIDE_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${TARGET_HARD_CAP}" \
--argjson bound "${TARGET_UNOBSERVED_BOUND}" \
'.[$cell].connection_hard_cap = $cap |
.[$cell].connection_unobserved_bound = $bound' \
<<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-capacity.tfvars.json"
BASE_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var manage_artifact_dns=false \
<<< 'local.relay_director_cells_json' | jq -er '.')"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
-var manage_artifact_dns=false \
<<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
-var manage_artifact_dns=false \
<<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
jq -e --arg cell "${TARGET_CELL_ID}" \
'any(.[]; .id == $cell and .connectionHardCap == 1000 and
.connectionUnobservedBound == 60)' \
<<< "${BASE_CELLS_JSON}" >/dev/null
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
DESIRED_IMAGE_DIGEST="${DESIRED_IMAGE##*@}"
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
test "${MIG_NAME}" = "orca-cloud-relay-gce-${TARGET_HOSTNAME}"
{
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "TARGET_HARD_CAP=${TARGET_HARD_CAP}"
echo "TARGET_UNOBSERVED_BOUND=${TARGET_UNOBSERVED_BOUND}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "BASE_CELLS_JSON=${BASE_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact compatible production image and topology
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_REVISION_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_IMAGE="$(jq -er '.spec.containers[0].image' <<< "${ACTIVE_REVISION_JSON}")"
ACTIVE_IMAGE_DIGEST="${ACTIVE_IMAGE##*@}"
if test "${ACTIVE_IMAGE}" != "${DESIRED_IMAGE}"; then
test "${ACTIVE_IMAGE_DIGEST}" = "${COMPATIBLE_DIRECTOR_IMAGE_DIGEST}"
test "${DESIRED_IMAGE_DIGEST}" = "${COMPATIBLE_CELL_IMAGE_DIGEST}"
fi
CURRENT_CELLS_JSON="$(jq -cer '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end' \
<<< "${ACTIVE_REVISION_JSON}")"
CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON="$(
node dev/scripts/read-relay-production-capacity-identity.mjs \
<<< "${ACTIVE_REVISION_JSON}"
)"
CLASSIFICATION="$(jq -nc \
--argjson baseCells "${BASE_CELLS_JSON}" \
--argjson currentCells "${CURRENT_CELLS_JSON}" \
--arg capacityCellIds "${CAPACITY_CELL_IDS}" \
--arg targetCellId "${TARGET_CELL_ID}" \
--argjson targetHardCap "${TARGET_HARD_CAP}" \
--argjson currentCapacityServiceAccount \
"${CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON}" \
'{baseCells:$baseCells, currentCells:$currentCells,
capacityCellIds:($capacityCellIds | split(",")),
targetCellId:$targetCellId, targetHardCap:$targetHardCap,
currentCapacityServiceAccount:$currentCapacityServiceAccount}' \
| node dev/scripts/classify-relay-production-capacity-director.mjs \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
TOPOLOGY_PHASE="$(jq -er '.topologyPhase' <<< "${CLASSIFICATION}")"
DESIRED_CELLS_JSON="$(jq -cer '.desiredCells' <<< "${CLASSIFICATION}")"
DIRECTOR_READY="$(jq -er \
'if (.directorReady | type) == "boolean" then
(.directorReady | tostring)
else error("invalid directorReady classification") end' \
<<< "${CLASSIFICATION}")"
{
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}"
echo "TOPOLOGY_PHASE=${TOPOLOGY_PHASE}"
echo "DIRECTOR_READY=${DIRECTOR_READY}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact wave predecessor topology
if: ${{ inputs.mode == 'apply' && (inputs.evidence-mode == 'continuation' || inputs.evidence-mode == 'resume') }}
run: test "${TOPOLOGY_PHASE}" = predecessor
- name: Verify fresh dry-run evidence against the live selector
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${TARGET_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Recheck exact wave state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--wave-index "${WAVE_INDEX}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
RETRY_ARGS=()
if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
"${RETRY_ARGS[@]}"
- name: Recheck exact isolated resume state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-resume-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
--retry-freshness
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining required \
--activity allowed \
--runtime required \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
- name: Consume the single-use dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Verify current selected-cell capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
CURRENT_CAP="${TARGET_HARD_CAP}"
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
if test "${TOPOLOGY_PHASE}" = predecessor; then
CURRENT_CAP=600
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${CURRENT_IMAGE_DIGEST}"
- name: Arm fail-closed mutation cleanup
if: ${{ inputs.mode != 'verify' }}
run: echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
- name: Reversibly isolate only the selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Drain the selected cell or prove an offline rollback
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
if node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain; then
echo "OFFLINE_ROLLBACK=false" >> "${GITHUB_ENV}"
elif test "${DEPLOY_MODE}" = rollback; then
echo "OFFLINE_ROLLBACK=true" >> "${GITHUB_ENV}"
else
exit 1
fi
- id: restart-auth-one
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: restart-gate-one
name: Require restart-safe selected-cell activity
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-one.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
GATE_LOG="${RUNNER_TEMP}/relay-capacity-restart-gate-one.log"
set +e
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" \
2> "${GATE_LOG}"
GATE_EXIT=$?
set -e
cat "${GATE_LOG}" >&2
if test "${GATE_EXIT}" = 0; then
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
if test "$(wc -l < "${GATE_LOG}" | tr -d ' ')" = 1 &&
grep -Eq '^capacity transition verification timed out: \{.*\}$' "${GATE_LOG}"; then
echo "settled=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
exit "${GATE_EXIT}"
- id: restart-auth-two
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require extended restart-safe selected-cell activity
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-two.outputs.id_token }}
run: |
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- name: Deploy only the reviewed director topology
if: ${{ inputs.mode != 'verify' }}
run: |
if test "${DIRECTOR_READY}" = true; then exit 0; fi
RELEASE_ID="capacity-${TARGET_HOSTNAME}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--max-instances 5 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--min-instances 5 \
--prune-revisions false \
--release-id "${RELEASE_ID}"
- id: director-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require fail-closed director transition
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.director-transition-auth.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Plan and apply only the empty selected cell
if: ${{ inputs.mode != 'verify' }}
shell: bash
run: |
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
-var manage_artifact_dns=false \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${PLAN_RESULT}"
PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${PLAN_CHANGES}" =~ ^(0|1|2)$ ]]
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
else
INSTANCE="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--format=json | jq -er 'if length == 1 and
.[0].instanceStatus == "RUNNING" and .[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("selected cell is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" --instances "${INSTANCE}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" --stable --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --timeout 900
- id: capacity-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify fresh exact selected-cell heartbeat before admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- name: Restore only the selected cell to general admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the live general selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep the selected cell isolated after a failed mutation
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
CLEANUP_STATUS=0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate || CLEANUP_STATUS=$?
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain || CLEANUP_STATUS=$?
exit "${CLEANUP_STATUS}"
@@ -0,0 +1,353 @@
name: Deploy Relay Production Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify, change one cell, or raise a sequential wave
required: true
default: verify
type: choice
options:
- verify
- apply
- rollback
- wave-apply
- wave-resume
target-cell-id:
description: Exact serving cell for verify, apply, or rollback
required: true
default: production-gce-c26
type: choice
options:
- production-gce-c7
- production-gce-c8
- production-gce-c9
- production-gce-c10
- production-gce-c13
- production-gce-c14
- production-gce-c15
- production-gce-c16
- production-gce-c19
- production-gce-c20
- production-gce-c21
- production-gce-c22
- production-gce-c23
- production-gce-c24
- production-gce-c25
- production-gce-c26
wave-cell-ids:
description: Ordered comma-separated wave of two to four serving cells
required: false
default: none
type: string
confirmation:
description: Enter the exact single-cell or wave confirmation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID for apply
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt for apply
required: false
type: string
source-wave-run-id:
description: Failed wave run that isolated the resume target
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
single_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode != 'wave-apply' && inputs.mode != 'wave-resume') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: ${{ inputs.mode }}
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: single
wave-cell-ids: none
wave-index: '0'
source-wave-run-id: none
secrets: inherit
resume_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-resume' && github.ref == 'refs/heads/main') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: resume
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: resume
source-wave-run-id: ${{ inputs.source-wave-run-id }}
secrets: inherit
wave_gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-apply' && github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
env:
DIRECTOR_ORIGIN: https://relay.onorca.dev
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
steps:
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
run: |
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- id: wave
name: Validate the exact wave request
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
CELLS="$(node dev/scripts/relay-production-capacity-wave.mjs validate \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--confirmation "${CONFIRMATION}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify dry-run artifact before cloud authentication
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Verify wave evidence against the live selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
FIRST_CELL="$(jq -er '.[0]' <<< '${{ steps.wave.outputs.cells }}')"
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${FIRST_CELL}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Require exact 600/60 predecessor wave cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
while read -r CELL_ID; do
HOSTNAME="${CELL_ID#production-gce-}"
[[ "${HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://${HOSTNAME}.relay.onorca.dev" \
--cell-id "${CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests \
"${PREDECESSOR_IMAGE_DIGEST},${COMPATIBLE_CELL_IMAGE_DIGEST}"
done < <(jq -r '.[]' <<< '${{ steps.wave.outputs.cells }}')
- name: Consume the single-use dry-run evidence
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
wave_cell_1:
needs: wave_gate
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[0] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '0'
source-wave-run-id: none
secrets: inherit
wave_cell_2:
needs: [wave_gate, wave_cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[1] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '1'
source-wave-run-id: none
secrets: inherit
wave_cell_3:
if: ${{ needs.wave_cell_2.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[2] != null }}
needs: [wave_gate, wave_cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[2] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '2'
source-wave-run-id: none
secrets: inherit
wave_cell_4:
if: ${{ needs.wave_cell_3.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[3] != null }}
needs: [wave_gate, wave_cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[3] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '3'
source-wave-run-id: none
secrets: inherit
# Every wave job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- single_cell
- resume_cell
- wave_gate
- wave_cell_1
- wave_cell_2
- wave_cell_3
- wave_cell_4
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -0,0 +1,267 @@
name: Deploy Relay Production Director
on:
workflow_dispatch:
inputs:
image-digest:
description: "Immutable relay image digest (sha256: plus 64 lowercase hex characters)"
required: true
type: string
regional-placement-mode:
description: Preserve the live switch, explicitly enable Asia preference, or force US-first
required: true
default: preserve
type: choice
options: [preserve, enable, disable]
prune-incompatible-revisions:
description: Retain only the newly verified serving and rollback revisions
required: true
default: false
type: boolean
confirmation:
description: Enter the exact confirmation required by a destructive option
required: false
type: string
expected-rehome-generation:
description: Exact durable regional-rehome generation; it must remain disabled
required: true
type: string
bootstrap-runtime-identity:
description: One-time move from the stamped-cell identity to the director identity
required: true
default: false
type: boolean
predecessor-image-digest:
description: Exact immutable serving predecessor digest for the one-time identity bootstrap
required: true
type: string
permissions:
contents: read
id-token: write
# Director updates and candidate operations both mutate production relay control state.
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
IMAGE_DIGEST: ${{ inputs.image-digest }}
REGIONAL_PLACEMENT_MODE: ${{ inputs.regional-placement-mode }}
PRUNE_INCOMPATIBLE_REVISIONS: ${{ inputs.prune-incompatible-revisions }}
# Floor the served revision must keep, matching relay_min_instances in
# environments/production.tfvars. This gate only fails a bad deploy; Terraform
# still owns the value, and the candidate inherits it from the serving revision.
DIRECTOR_MIN_INSTANCES: 5
DIRECTOR_MAX_INSTANCES: 5
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
PREDECESSOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_RUNTIME_SERVICE_ACCOUNT }}
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Resolve immutable production image
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ ! "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "image-digest must be an immutable lowercase sha256 digest" >&2
exit 1
fi
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
[[ "${PRUNE_INCOMPATIBLE_REVISIONS}" =~ ^(true|false)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${BOOTSTRAP_RUNTIME_IDENTITY}" =~ ^(true|false)$ ]]
if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY
elif test "${PRUNE_INCOMPATIBLE_REVISIONS}" = true; then
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = PRUNE_INCOMPATIBLE_RELAY_DIRECTOR_REVISIONS
elif test "${REGIONAL_PLACEMENT_MODE}" = disable; then
test "${CONFIRMATION}" = FORCE_RELAY_US_FIRST
else
test -z "${CONFIRMATION}"
fi
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
# Why: the deploy INHERITS the serving revision's floor, so when that revision has
# already lost it the candidate inherits zero, the in-script gate compares zero against
# zero and passes, and the post-deploy check below only notices after traffic moved.
# The documented rollback target is created at minimum instances zero, so promoting it
# arms exactly that. Refuse to inherit a degraded floor rather than latch it.
- name: Require a healthy serving floor before deploying
shell: bash
run: |
SERVING="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${SERVING}"
FLOOR="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${FLOOR:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "serving revision ${SERVING} holds ${FLOOR:-0} minimum instances," \
"below ${DIRECTOR_MIN_INSTANCES}; deploying would inherit and latch it." >&2
echo "Restore the floor first: gcloud run services update ${DIRECTOR_SERVICE_NAME}" \
"--min-instances=${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
CEILING="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${CEILING}" = "${DIRECTOR_MAX_INSTANCES}"
echo "serving revision ${SERVING} holds ${FLOOR} minimum instances"
echo "SERVING_REVISION=${SERVING}" >> "${GITHUB_ENV}"
# Why: no --min-instances here. The candidate inherits the Terraform-owned
# scaling, and this step ends with 100% traffic on it. Pinning 1 rebuilt the
# per-instance admission shortage that took placement failures to ~70%.
- name: Deploy director blue/green
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
served_version="$(gcloud run revisions describe "${SERVING_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
(.version // .key // empty)] |
if length == 1 then .[0] else empty end')"
if [[ "${served_version}" =~ ^[1-9][0-9]*$ ]]; then
current_version="${served_version}"
else
test "${REGIONAL_PLACEMENT_MODE}" = preserve
current_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${current_version}" =~ ^[1-9][0-9]*$ ]]
fi
current="$(gcloud secrets versions access "${current_version}" \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
case "${REGIONAL_PLACEMENT_MODE}" in
preserve) desired="${current}" ;;
enable) desired=true ;;
disable) desired=false ;;
*) echo "regional-placement-mode is invalid" >&2; exit 1 ;;
esac
if test "${current}" != "${desired}"; then
target_version="$(printf '%s' "${desired}" | gcloud secrets versions add \
"${REGIONAL_PLACEMENT_SECRET}" --project "${GCP_PROJECT_ID}" --data-file=- \
--format='value(name)' --quiet | awk -F/ '{print $NF}')"
else
target_version="${current_version}"
fi
[[ "${target_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" \
--bootstrap-runtime-identity "${BOOTSTRAP_RUNTIME_IDENTITY}" \
--predecessor-image-digest "${PREDECESSOR_IMAGE_DIGEST}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience "${REHOME_AUDIENCE}" \
--rehome-control-origin https://relay.onorca.dev \
--admin-audience https://relay.onorca.dev/v1/admin/drain \
--expected-rehome-generation "${EXPECTED_REHOME_GENERATION}" \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--prune-revisions "${PRUNE_INCOMPATIBLE_REVISIONS}" \
--release-id "${RELEASE_ID}" \
--regional-placement-secret-version "${target_version}"
echo "REGIONAL_PLACEMENT_ENABLED=${desired}" >> "${GITHUB_ENV}"
echo "REGIONAL_PLACEMENT_VERSION=${target_version}" >> "${GITHUB_ENV}"
- name: Verify served revision and native health
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
REVISION="$(jq -r '[.status.traffic[] | select((.percent // 0) > 0)] | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' <<< "${SERVICE_JSON}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
SERVED_REGIONAL_PLACEMENT_SECRET="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)}] |
if length == 1 then .[0] else error("regional placement secret missing") end')"
test "$(jq -r '.secret' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_SECRET}"
test "$(jq -r '.version' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_VERSION}"
test "$(gcloud secrets versions access "${REGIONAL_PLACEMENT_VERSION}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" = "${REGIONAL_PLACEMENT_ENABLED}"
# Why: a served revision with no warm-instance floor still passes health and digest
# checks while quietly shrinking per-instance admission capacity.
SERVED_MIN_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${SERVED_MIN_INSTANCES:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "served revision ${REVISION} holds ${SERVED_MIN_INSTANCES:-0} minimum instances, expected at least ${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
SERVED_MAX_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${SERVED_MAX_INSTANCES}" = "${DIRECTOR_MAX_INSTANCES}"
SERVICE_URL="$(jq -r '.status.url' <<< "${SERVICE_JSON}")"
node dev/scripts/smoke-relay.mjs "${SERVICE_URL}"
@@ -0,0 +1,499 @@
name: Deploy Relay Production Multi-Target
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform source cell ID
required: true
type: string
target-cell-ids:
description: Comma-separated distinct Terraform target cell IDs
required: true
type: string
general-cell-ids:
description: Comma-separated proven cells that remain eligible for ordinary placement
required: false
type: string
unobserved-connection-bound:
description: Exact worst-case unobserved connection bound proven by the passing load gate
required: false
type: string
failed-target-cell-id:
description: Registered failed target to fence and supersede
required: false
type: string
replacement-target-cell-id:
description: Healthy replacement for registered failed target
required: false
type: string
mode:
description: Preflight/audit are read-only; other modes mutate production
required: true
default: preflight
type: choice
options:
- audit
- preflight
- cutover-admission
- add-migration-cells
- promote-general-cell
- retire-migration-cell
- execute
- recover-forward
- fence-source
- supersede-target
confirmation:
description: Enter CUTOVER_SELECTOR, ADD_MIGRATION_CELLS, PROMOTE_GENERAL_CELL, RETIRE_MIGRATION_CELL, EVACUATE_MULTI, RECOVER_FORWARD, or FENCE_SOURCE
required: false
type: string
selector-attempt-id:
description: Exact durable selector attempt ID for admission mutations
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
broker-operation-id:
description: Stable durable broker operation ID for target supersession
required: false
type: string
completed-fence-attempt-id:
description: Exact older completed fence attempt to recover without replay
required: false
type: string
completed-fence-commit:
description: Exact older fence commit bound to the completed attempt
required: false
type: string
completed-fence-operation:
description: Exact DONE Compute resize operation to adopt
required: false
type: string
completed-fence-state-serial:
description: Exact Terraform serial before the completed fence
required: false
type: string
completed-fence-plan-generation:
description: Exact saved-plan object generation
required: false
type: string
completed-fence-state-generation:
description: Exact current Terraform state object generation
required: false
type: string
completed-fence-state-sha256:
description: Exact current Terraform state object SHA-256
required: false
type: string
expected-lease-generation:
description: Exact live lease generation authorized for conditional takeover
required: false
type: string
expected-lease-operation-id:
description: Exact live lease operation ID authorized for takeover
required: false
type: string
expected-lease-request-digest:
description: Exact live lease request digest authorized for takeover
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_IDS: ${{ inputs.target-cell-ids }}
GENERAL_CELL_IDS: ${{ inputs.general-cell-ids }}
UNOBSERVED_CONNECTION_BOUND: ${{ inputs.unobserved-connection-bound }}
FAILED_TARGET_CELL_ID: ${{ inputs.failed-target-cell-id }}
REPLACEMENT_TARGET_CELL_ID: ${{ inputs.replacement-target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
BROKER_OPERATION_ID: ${{ inputs.broker-operation-id }}
COMPLETED_FENCE_ATTEMPT_ID: ${{ inputs.completed-fence-attempt-id }}
COMPLETED_FENCE_COMMIT: ${{ inputs.completed-fence-commit }}
COMPLETED_FENCE_OPERATION: ${{ inputs.completed-fence-operation }}
COMPLETED_FENCE_STATE_SERIAL: ${{ inputs.completed-fence-state-serial }}
COMPLETED_FENCE_PLAN_GENERATION: ${{ inputs.completed-fence-plan-generation }}
COMPLETED_FENCE_STATE_GENERATION: ${{ inputs.completed-fence-state-generation }}
COMPLETED_FENCE_STATE_SHA256: ${{ inputs.completed-fence-state-sha256 }}
EXPECTED_LEASE_GENERATION: ${{ inputs.expected-lease-generation }}
EXPECTED_LEASE_OPERATION_ID: ${{ inputs.expected-lease-operation-id }}
EXPECTED_LEASE_REQUEST_DIGEST: ${{ inputs.expected-lease-request-digest }}
steps:
- uses: actions/checkout@v4
- name: Require private fence-broker environment
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
env:
FENCE_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
FENCE_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
FENCE_BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
test -n "${FENCE_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${FENCE_SERVICE_ACCOUNT}"
test -n "${FENCE_BROKER_URI}"
- name: Reject direct-runner Terraform fence aborts
if: ${{ inputs.mode == 'abort-fence-source' }}
run: |
echo "Terraform fence aborts require a reviewed private-broker recovery path." >&2
exit 1
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
if: ${{ inputs.mode != 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "cutover-admission" ]]; then
test "${CONFIRMATION}" = "CUTOVER_SELECTOR"
elif [[ "${DEPLOY_MODE}" = "add-migration-cells" ]]; then
test "${CONFIRMATION}" = "ADD_MIGRATION_CELLS"
elif [[ "${DEPLOY_MODE}" = "promote-general-cell" ]]; then
test "${CONFIRMATION}" = "PROMOTE_GENERAL_CELL"
elif [[ "${DEPLOY_MODE}" = "retire-migration-cell" ]]; then
test "${CONFIRMATION}" = "RETIRE_MIGRATION_CELL"
elif [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE_MULTI"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "fence-source" ]]; then
test "${CONFIRMATION}" = "FENCE_SOURCE"
elif [[ "${DEPLOY_MODE}" = "supersede-target" ]]; then
test "${CONFIRMATION}" = "SUPERSEDE_TARGET"
elif [[ "${DEPLOY_MODE}" = "abort-fence-source" ]]; then
test "${CONFIRMATION}" = "ABORT_FENCE"
else
test "${CONFIRMATION}" = "FENCE_SOURCE"
fi
- name: Require exact source-fence broker contract
if: ${{ inputs.mode == 'fence-source' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${TARGET_CELL_IDS}" = "production-gce-c7,production-gce-c8,production-gce-c10,production-gce-c13,production-gce-c17,production-gce-c18"
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Require exact broker cell contract
if: ${{ inputs.mode == 'supersede-target' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${FAILED_TARGET_CELL_ID}" = "production-gce-c12"
test "${REPLACEMENT_TARGET_CELL_ID}" = "production-gce-c13"
test "${TARGET_CELL_IDS}" = "production-gce-c12,production-gce-c13"
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
[[ "${COMPLETED_FENCE_ATTEMPT_ID}" =~ ^[0-9a-f-]{36}$ ]]
[[ "${COMPLETED_FENCE_COMMIT}" =~ ^[0-9a-f]{40}$ ]]
[[ "${COMPLETED_FENCE_OPERATION}" =~ ^[A-Za-z0-9._-]{1,256}$ ]]
[[ "${COMPLETED_FENCE_STATE_SERIAL}" =~ ^[0-9]+$ ]]
[[ "${COMPLETED_FENCE_PLAN_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_SHA256}" =~ ^[0-9a-f]{64}$ ]]
test -n "${EXPECTED_LEASE_GENERATION}"
fi
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Read reviewed Terraform topology
if: ${{ inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
DIRECTOR_MIN_INSTANCES="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars <<< 'var.relay_min_instances')"
[[ "${DIRECTOR_MIN_INSTANCES}" =~ ^[1-9][0-9]*$ ]]
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
echo "DIRECTOR_MIN_INSTANCES=${DIRECTOR_MIN_INSTANCES}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
SCOPED_RECOVERY_ARGS=()
if [[ ("${DEPLOY_MODE}" = "execute" ||
"${DEPLOY_MODE}" = "recover-forward") &&
"${SOURCE_CELL_ID}" = "production-gce-c12" ]]; then
SCOPED_RECOVERY_ARGS=(
--scoped-recovery-source-cell-id
production-gce-c3
)
fi
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
"${SCOPED_RECOVERY_ARGS[@]}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- id: google-fence-broker-auth
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
id_token_include_email: true
- name: Invoke private target-supersession broker
if: ${{ inputs.mode == 'supersede-target' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg attemptId "${COMPLETED_FENCE_ATTEMPT_ID}" \
--arg completedCommit "${COMPLETED_FENCE_COMMIT}" \
--arg gceOperation "${COMPLETED_FENCE_OPERATION}" \
--arg stateSerial "${COMPLETED_FENCE_STATE_SERIAL}" \
--arg planGeneration "${COMPLETED_FENCE_PLAN_GENERATION}" \
--arg stateGeneration "${COMPLETED_FENCE_STATE_GENERATION}" \
--arg stateSha256 "${COMPLETED_FENCE_STATE_SHA256}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
completedFenceRecovery:{attemptId:$attemptId,fenceCommit:$completedCommit,
gceOperation:$gceOperation,terraformStateSerial:($stateSerial|tonumber),
planObjectGeneration:$planGeneration,
terraformStateObjectGeneration:$stateGeneration,
terraformStateObjectSha256:$stateSha256},
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
elif [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,confirmation:"SUPERSEDE_TARGET"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/supersede-target" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Invoke private source-fence broker
if: ${{ inputs.mode == 'fence-source' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"FENCE_SOURCE"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),confirmation:"FENCE_SOURCE"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/fence-source" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Preflight or run multi-target evacuation
if: >-
${{ inputs.mode != 'fence-source' &&
inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-multi-target.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-ids "${TARGET_CELL_IDS}" \
--general-cell-ids "${GENERAL_CELL_IDS}" \
--unobserved-connection-bound "${UNOBSERVED_CONNECTION_BOUND}" \
--director-region "${{ vars.PRODUCTION_GCP_REGION }}" \
--director-service "orca-cloud-relay" \
--director-min-instances "${DIRECTOR_MIN_INSTANCES}" \
--selector-attempt-id "${SELECTOR_ATTEMPT_ID}" \
--failed-target-cell-id "${FAILED_TARGET_CELL_ID}" \
--replacement-target-cell-id "${REPLACEMENT_TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--environment production \
--fence-commit "${GITHUB_SHA}" \
--terraform-dir infra/terraform \
--terraform-var-file environments/production.tfvars \
--mode "${DEPLOY_MODE}" \
--connection-ceiling 1000 \
--minimum-lease-remaining-ms 600000
@@ -0,0 +1,645 @@
name: Deploy Relay Production Same-Cap Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
target-cell-id: { required: true, type: string }
target-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
target-rehome-protocol: { required: true, type: string }
rollback-rehome-protocol: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-rehome-generation: { required: true, type: string }
monitor-run-id: { required: true, type: string }
monitor-run-attempt: { required: true, type: string }
wave-index: { required: true, type: string }
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
rollout:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 75
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay.onorca.dev
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
TARGET_REHOME_PROTOCOL: ${{ inputs.target-rehome-protocol }}
ROLLBACK_REHOME_PROTOCOL: ${{ inputs.rollback-rehome-protocol }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
WAVE_INDEX: ${{ inputs.wave-index }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${TARGET_IMAGE_DIGEST}" != "${ROLLBACK_IMAGE_DIGEST}"
[[ "${TARGET_REHOME_PROTOCOL}" =~ ^[01]$ ]]
[[ "${ROLLBACK_REHOME_PROTOCOL}" =~ ^[01]$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${WAVE_INDEX}" =~ ^[0-3]$ ]]
if test "${DEPLOY_MODE}" = verify; then
EFFECTIVE_SELECTOR_GENERATION="${EXPECTED_SELECTOR_GENERATION}"
else
EFFECTIVE_SELECTOR_GENERATION="$((EXPECTED_SELECTOR_GENERATION + (2 * WAVE_INDEX)))"
fi
echo "EFFECTIVE_SELECTOR_GENERATION=${EFFECTIVE_SELECTOR_GENERATION}" >> "${GITHUB_ENV}"
if test "${DEPLOY_MODE}" != verify && test "${GITHUB_RUN_ATTEMPT}" != 1; then
echo "mutations are single-dispatch: re-runs replay aged evidence," >&2
echo "so recover each remaining cell with its own fresh monitor" >&2
echo "dry-run and canary-apply dispatch instead" >&2
exit 1
fi
test -n "${GCP_REGION}"
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WIF}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with: { package_json_file: cloud/package.json }
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with: { terraform_wrapper: false }
- name: Require fresh aggregate monitor evidence reference
if: ${{ inputs.mode != 'verify' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private aggregate monitor evidence
if: ${{ inputs.mode != 'verify' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify monitor evidence provenance
if: ${{ inputs.mode != 'verify' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--required-migration-policy strict \
--wave-index "${WAVE_INDEX}"
- name: Download this wave's single-use safety authority
if: ${{ inputs.mode != 'verify' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-same-cap-monitor-authority
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Require safety evidence consumed by this workflow
if: ${{ inputs.mode != 'verify' }}
run: |
# Mutations are single-dispatch: a fresh dispatch cannot resume a
# partial batch (the canary authority binds the batch-entry selector
# generation), so each remaining cell is recovered by its own fresh
# monitor dry-run and canary-apply dispatch, never by re-running
# aged evidence.
test "${GITHUB_RUN_ATTEMPT}" = 1
MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}")" = \
"${GITHUB_RUN_ID}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Recheck aggregate SQL, pool, reconnect, migration, and selector safety
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
RETRY_ARGS=()
if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-index "${WAVE_INDEX}" "${RETRY_ARGS[@]}"
- name: Require durable rehome disabled and exact selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect \
--director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EFFECTIVE_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Resolve immutable same-cap cell configuration
shell: bash
run: |
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
case "${TARGET_HOSTNAME}" in
c7|c8|c9|c10|c13|c14|c15|c16|c19|c20|c21|c22|c23|c24|c25|c26)
EXPECTED_HARD_CAP=1000
EXPECTED_REGION=us-central1
;;
c27|c28|c29)
EXPECTED_HARD_CAP=3000
EXPECTED_REGION=asia-east2
;;
*) exit 1 ;;
esac
EXPECTED_UNOBSERVED_BOUND=60
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars -var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
SOURCE_CELLS="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars -var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_region_rehome_source_cell_ids)' | jq -er '.')"
if test "${EXPECTED_REGION}" = us-central1; then
jq -e --arg cell "${TARGET_CELL_ID}" 'index($cell) != null' \
<<< "${SOURCE_CELLS}" >/dev/null
fi
CURRENT_SHAPE="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${CELLS_JSON}")"
test "$(jq -r '.connection_hard_cap' <<< "${CURRENT_SHAPE}")" = "${EXPECTED_HARD_CAP}"
test "$(jq -r '.connection_unobserved_bound' <<< "${CURRENT_SHAPE}")" = \
"${EXPECTED_UNOBSERVED_BOUND}"
TARGET_ZONE="$(jq -r '.zone' <<< "${CURRENT_SHAPE}")"
MIG_NAME="orca-cloud-relay-gce-${TARGET_HOSTNAME}"
if test "${DEPLOY_MODE}" = rollback; then
DESIRED_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
else
DESIRED_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
fi
DESIRED_IMAGE="${IMAGE_REPOSITORY}@${DESIRED_IMAGE_DIGEST}"
OVERRIDE_CELLS_JSON="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${DESIRED_IMAGE}" '.[$cell].image = $image' <<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-same-cap.tfvars.json"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${DESIRED_IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"
{
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "EXPECTED_HARD_CAP=${EXPECTED_HARD_CAP}"
echo "EXPECTED_UNOBSERVED_BOUND=${EXPECTED_UNOBSERVED_BOUND}"
echo "EXPECTED_REGION=${EXPECTED_REGION}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "CURRENT_IMAGE_DIGEST=${CURRENT_IMAGE_DIGEST}"
echo "DESIRED_REHOME_PROTOCOL=${DESIRED_REHOME_PROTOCOL}"
echo "CURRENT_REHOME_PROTOCOL=${CURRENT_REHOME_PROTOCOL}"
} >> "${GITHUB_ENV}"
- name: Verify exact current generation, digest, cap, and rollback point
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
CURRENT_RUNTIME="$(curl --fail-with-body --max-time 30 \
--request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
# A rollback that failed between template apply and admission restore
# leaves the cell already on the rollback image; resume from that
# state instead of demanding the pre-rollback predecessor.
LIVE_IMAGE_DIGEST="$(jq -r '.imageDigest' <<< "${CURRENT_RUNTIME}")"
if test "${DEPLOY_MODE}" = rollback \
&& test "${LIVE_IMAGE_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"; then
ROLLBACK_RESUME=true
PREDECESSOR_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${DESIRED_REHOME_PROTOCOL}"
else
ROLLBACK_RESUME=false
PREDECESSOR_IMAGE_DIGEST="${CURRENT_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${CURRENT_REHOME_PROTOCOL}"
fi
RESTORED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
RESTORED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
test -n "${RESTORED_MIGRATION_CELLS}" || RESTORED_MIGRATION_CELLS=none
test -n "${RESTORED_GENERAL_CELLS}" || RESTORED_GENERAL_CELLS=none
ISOLATED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
ISOLATED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
test -n "${ISOLATED_MIGRATION_CELLS}" || ISOLATED_MIGRATION_CELLS=none
test -n "${ISOLATED_GENERAL_CELLS}" || ISOLATED_GENERAL_CELLS=none
{
echo "ROLLBACK_RESUME=${ROLLBACK_RESUME}"
# The failsafe consumes these; deriving them here keeps them
# defined for a failure in any later step.
echo "ISOLATED_MIGRATION_CELLS=${ISOLATED_MIGRATION_CELLS}"
echo "ISOLATED_GENERAL_CELLS=${ISOLATED_GENERAL_CELLS}"
# No restart happens on resume, so isolate below is skipped and
# cannot advance the selector generation.
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${EFFECTIVE_SELECTOR_GENERATION}"
# A failed-canary rollback enters with the target migration-only,
# so the restore inspect cannot reuse the entry membership inputs.
echo "RESTORED_MIGRATION_CELLS=${RESTORED_MIGRATION_CELLS}"
echo "RESTORED_GENERAL_CELLS=${RESTORED_GENERAL_CELLS}"
} >> "${GITHUB_ENV}"
if ! jq -e --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \
&& test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \
'.role == "cell" and .cellId == $cell and .cellUrl == $origin and
(.region == $region or
($region == "us-central1" and $protocol == 0 and .region == null)) and
.imageDigest == $digest and
.connectionCapacity.hardCap == $hardCap and
.connectionCapacity.unobservedBound == $unobservedBound and
(.draining == false or $drainingOk) and
(.regionalRehomeProtocol // 0) == $protocol' <<< "${CURRENT_RUNTIME}" >/dev/null
then
jq -r --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "$(test "${DEPLOY_MODE}" = rollback \
&& test "${ROLLBACK_RESUME}" != true && echo true || echo false)" \
'[
if .role != "cell" then "role" else empty end,
if .cellId != $cell then "cellId" else empty end,
if .cellUrl != $origin then "cellUrl" else empty end,
if (.region != $region and
($region != "us-central1" or $protocol != 0 or .region != null))
then "region" else empty end,
if .imageDigest != $digest then "imageDigest" else empty end,
if .connectionCapacity.hardCap != $hardCap then "hardCap" else empty end,
if .connectionCapacity.unobservedBound != $unobservedBound then "unobservedBound" else empty end,
if (.draining != false and ($drainingOk | not)) then "draining" else empty end,
if (.regionalRehomeProtocol // 0) != $protocol then "regionalRehomeProtocol" else empty end
] | "runtime predecessor mismatch fields=" + join(",")' \
<<< "${CURRENT_RUNTIME}" >&2
exit 1
fi
# The exact legacy digest binds omitted pre-region fields to US and protocol 0.
jq -r '[
if .region == null then "region" else empty end,
if .regionalRehomeProtocol == null then "regionalRehomeProtocol" else empty end
] | if length > 0 then "runtime predecessor normalized legacy fields=" + join(",") else empty end' \
<<< "${CURRENT_RUNTIME}"
CURRENT_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \
--request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
SOURCE_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${CURRENT_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true && ! jq -e \
'.status.admissionState == "migration-only"' \
<<< "${CURRENT_DIRECTOR_STATUS}" >/dev/null; then
echo 'resume requires the isolated migration-only cell a failed rollback leaves' >&2
exit 1
fi
[[ "${SOURCE_INCARNATION}" =~ ^[0-9a-f-]{36}$ ]]
echo "SOURCE_INCARNATION=${SOURCE_INCARNATION}" >> "${GITHUB_ENV}"
# Rollback is the documented recovery from a failed canary, which
# leaves the cell migration-only (and possibly still marked
# draining); apply and verify still require a pristine general cell.
if test "${DEPLOY_MODE}" = rollback; then
PRECHECK_ADMISSION=general-or-migration-only
PRECHECK_DRAINING=either
else
PRECHECK_ADMISSION=general
PRECHECK_DRAINING=forbidden
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission "${PRECHECK_ADMISSION}" \
--draining "${PRECHECK_DRAINING}" --activity allowed \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
- name: Finish read-only verification
if: ${{ inputs.mode == 'verify' }}
run: echo 'Exact same-cap rollback point verified.'
- name: Reversibly isolate and drain only the selected cell
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# A cell isolated by a failed canary is already migration-only, so
# isolate is a no-op there that does not advance the selector; the
# result's generation is authoritative either way.
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate)"
echo "${ISOLATE_RESULT}"
ISOLATE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${ISOLATE_GENERATION}" >> "${GITHUB_ENV}"
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode drain
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat either --admission migration-only --draining required \
--activity restart-safe --expected-image-digests "${CURRENT_IMAGE_DIGEST}" \
--timeout-ms 900000
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- name: Require converged Terraform state and a stable MIG on resume
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME == 'true' }}
shell: bash
env:
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
# Zero resource changes prove the prior run's apply completed and no
# restart will follow, keeping the incarnation check honest. Root
# outputs may lag a targeted apply, so judge resource_changes only.
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
-var manage_artifact_dns=false \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
if ! terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -e '[.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))]
| length == 0' >/dev/null
then
# An apply that failed before its template apply also resumes here
# (the cell still serves the rollback image), and repo drift since
# the cell's last roll (for example newly added rehome trust
# config) then legitimately replaces the template. Nothing is
# applied on resume either way, so accept exactly the drift the
# reviewed validator would let a real apply ship for the image the
# cell already serves: the template leaves and re-enters the
# rollback image, as exactly the template-and-MIG change pair.
terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -r '"resume found unconverged resources: " +
([.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))
| .address] | join(","))'
echo 'requiring reviewed rollback-image drift'
terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--rollback-image "${DESIRED_IMAGE}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
| jq -e '.changes == 2' >/dev/null
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
- name: Apply only the selected same-cap template and MIG
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
shell: bash
env:
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
-var manage_artifact_dns=false \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
terraform -chdir=infra/terraform show -json "${RUNNER_TEMP}/relay-same-cap.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" --image "${DESIRED_IMAGE}" \
--rollback-image "${IMAGE_REPOSITORY}@${CURRENT_IMAGE_DIGEST}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-same-cap.tfplan"
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
- id: post-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new incarnation, exact image, protocol, and durable safety
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity allowed --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" --timeout-ms 900000
TARGET_RUNTIME="$(curl --fail-with-body --max-time 30 \
--request POST "${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
jq -e --arg digest "${DESIRED_IMAGE_DIGEST}" \
--argjson protocol "${DESIRED_REHOME_PROTOCOL}" \
'.imageDigest == $digest and (.regionalRehomeProtocol // 0) == $protocol' \
<<< "${TARGET_RUNTIME}" >/dev/null
TARGET_DIRECTOR_STATUS="$(curl --fail-with-body --max-time 30 \
--request POST "${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
TARGET_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${TARGET_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# No restart happened; the incarnation legitimately stays put.
test "${TARGET_INCARNATION}" = "${SOURCE_INCARNATION}"
else
test "${TARGET_INCARNATION}" != "${SOURCE_INCARNATION}"
fi
echo "TARGET_INCARNATION=${TARGET_INCARNATION}" >> "${GITHUB_ENV}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ISOLATE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- name: Prove exact per-host trust and idempotent no-neighbor behavior
if: ${{ inputs.mode != 'verify' && ((inputs.mode == 'rollback' && inputs.rollback-rehome-protocol == '1') || (inputs.mode != 'rollback' && inputs.target-rehome-protocol == '1')) }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
node dev/scripts/probe-relay-rehome-trust.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-id "${TARGET_CELL_ID}" \
--cell-incarnation "${TARGET_INCARNATION}"
- name: Restore only the verified selected cell to general admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
ACTIVATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode activate)"
echo "${ACTIVATE_RESULT}"
SELECTOR_GENERATION_AFTER_ACTIVATE="$(jq -er '.generation' \
<<< "${ACTIVATE_RESULT}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission general --draining forbidden --activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ACTIVATE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${RESTORED_MIGRATION_CELLS}" \
--expected-general-cells "${RESTORED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep a failed cell isolated and rehome disabled
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate)"
echo "${ISOLATE_RESULT}"
# The isolate result carries the authoritative post-isolate generation;
# fixed offsets are wrong whenever an earlier isolate was a no-op.
FAILSAFE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${FAILSAFE_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}"
@@ -0,0 +1,310 @@
name: Deploy Relay Production Same-Cap
on:
workflow_dispatch:
inputs:
mode:
description: Verify, roll one canary, roll a bounded batch, or roll back
required: true
default: verify
type: choice
options: [verify, canary-apply, batch-apply, rollback]
cell-ids:
description: Ordered comma-separated serving cells; one canary or two to four batch cells
required: true
type: string
target-image-digest:
description: Exact immutable compatibility image digest
required: true
type: string
rollback-image-digest:
description: Exact immutable currently serving rollback digest
required: true
type: string
target-rehome-protocol:
description: Exact target regional-rehome protocol
required: true
default: '1'
type: choice
options: ['0', '1']
rollback-rehome-protocol:
description: Exact rollback regional-rehome protocol
required: true
default: '0'
type: choice
options: ['0', '1']
expected-selector-generation:
description: Exact selector generation before the first cell
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-rehome-generation:
description: Exact durable regional-rehome control generation; it must be disabled
required: true
type: string
monitor-run-id:
description: Fresh successful aggregate dry-run monitor workflow run
required: false
type: string
monitor-run-attempt:
description: Exact monitor attempt
required: false
type: string
canary-run-id:
description: Successful same-commit canary run required for batch-apply
required: false
type: string
confirmation:
description: Exact digest-and-cell-bound mutation confirmation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
job-mode: ${{ steps.wave.outputs.job-mode }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
- id: wave
env:
MODE: ${{ inputs.mode }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
CONFIRMATION: ${{ inputs.confirmation }}
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
run: |
CELLS="$(node dev/scripts/relay-production-same-cap-wave.mjs validate \
--mode "${MODE}" --cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" --rollback-digest "${ROLLBACK_DIGEST}" \
--confirmation "${CONFIRMATION}" --canary-run-id "${CANARY_RUN_ID}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
if [[ "${MODE}" =~ ^(canary-apply|batch-apply)$ ]]; then
echo 'job-mode=apply' >> "${GITHUB_OUTPUT}"
else
echo "job-mode=${MODE}" >> "${GITHUB_OUTPUT}"
fi
- name: Download exact prior canary authority
if: ${{ inputs.mode == 'batch-apply' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-canary-${{ inputs.canary-run-id }}
path: ${{ runner.temp }}/relay-same-cap-canary
github-token: ${{ github.token }}
run-id: ${{ inputs.canary-run-id }}
- name: Verify canary authority against this batch
if: ${{ inputs.mode == 'batch-apply' }}
env:
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
run: |
node dev/scripts/relay-production-same-cap-wave.mjs verify-canary \
--file "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${CANARY_RUN_ID}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}"
- name: Reject previously consumed aggregate safety evidence
if: ${{ inputs.mode != 'verify' }}
env:
GH_TOKEN: ${{ github.token }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
MARKER_NAME="relay-same-cap-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = 0
mkdir -p "${RUNNER_TEMP}/relay-same-cap-monitor-authority"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-same-cap-monitor-authority/${MARKER_NAME}"
- name: Consume aggregate safety evidence for this exact wave
if: ${{ inputs.mode != 'verify' }}
uses: actions/upload-artifact@v4
with:
name: relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-same-cap-monitor-authority/relay-same-cap-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
cell_1:
needs: gate
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[0] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '0'
secrets: inherit
cell_2:
if: ${{ needs.cell_1.result == 'success' && fromJSON(needs.gate.outputs.cells)[1] != null }}
needs: [gate, cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[1] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '1'
secrets: inherit
cell_3:
if: ${{ needs.cell_2.result == 'success' && fromJSON(needs.gate.outputs.cells)[2] != null }}
needs: [gate, cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[2] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '2'
secrets: inherit
cell_4:
if: ${{ needs.cell_3.result == 'success' && fromJSON(needs.gate.outputs.cells)[3] != null }}
needs: [gate, cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[3] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
wave-index: '3'
secrets: inherit
seal_canary:
if: ${{ inputs.mode == 'canary-apply' }}
needs: [gate, cell_1]
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
- name: Seal exact successful canary authority
run: |
mkdir -p "${RUNNER_TEMP}/relay-same-cap-canary"
node dev/scripts/relay-production-same-cap-wave.mjs create-canary \
--cell-id "${{ inputs.cell-ids }}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--confirmation "${{ inputs.confirmation }}" \
--commit-sha "${GITHUB_SHA}" --run-id "${GITHUB_RUN_ID}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}" \
> "${RUNNER_TEMP}/relay-same-cap-canary/authority.json"
- uses: actions/upload-artifact@v4
with:
name: relay-same-cap-canary-${{ github.run_id }}
path: ${{ runner.temp }}/relay-same-cap-canary/authority.json
retention-days: 30
if-no-files-found: error
# Every cell job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- gate
- cell_1
- cell_2
- cell_3
- cell_4
- seal_canary
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -0,0 +1,219 @@
name: Deploy Relay Production Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Audit/preflight are read-only; recover/continue resume committed work; disable/enable/reset/execute mutate admission
required: true
default: preflight
type: choice
options:
- audit
- preflight
- recover-forward
- continue-evacuation
- disable-cell
- enable-empty-cell
- reset-empty-candidate
- execute
confirmation:
description: Enter RECOVER_FORWARD, CONTINUE_EVACUATION, DISABLE_CELL, ENABLE_CELL, RESET_CANDIDATE, or EVACUATE for the matching mutation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
steps:
- uses: actions/checkout@v4
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "continue-evacuation" ]]; then
test "${CONFIRMATION}" = "CONTINUE_EVACUATION"
elif [[ "${DEPLOY_MODE}" = "disable-cell" ]]; then
test "${CONFIRMATION}" = "DISABLE_CELL"
elif [[ "${DEPLOY_MODE}" = "enable-empty-cell" ]]; then
test "${CONFIRMATION}" = "ENABLE_CELL"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -0,0 +1,109 @@
name: Deploy Relay Staging GCE Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Preflight is read-only; reset repairs an empty candidate; execute evacuates
required: true
default: preflight
type: choice
options:
- preflight
- reset-empty-candidate
- execute
confirmation:
description: Enter RESET_CANDIDATE for reset or EVACUATE for execute
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
ADMIN_AUDIENCE: https://relay-staging.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -0,0 +1,112 @@
name: Deploy Relay Staging
on:
workflow_dispatch:
inputs:
expected-image-digest:
description: Exact checked-in production Relay sha256 digest to deploy
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
# Staging deploy, candidate, auth, and power operations must never overlap.
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
EXPECTED_IMAGE_DIGEST: ${{ inputs.expected-image-digest }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
ASIA_PROOF_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
steps:
- uses: actions/checkout@v4
- name: Require the expected immutable image
run: '[[ "${EXPECTED_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]'
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Bind the request to the checked-in staging C4 image
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure \
-backend-config=backend/staging.hcl -input=false
IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
<<< 'var.relay_gce_cells["staging-gce-c4"].image' | jq -er '.')"
test "${IMAGE}" = \
"${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${EXPECTED_IMAGE_DIGEST}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require the mirrored immutable image
run: |
DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${DIGEST}" = "${EXPECTED_IMAGE_DIGEST}"
- name: Deploy director blue/green
run: |
regional_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${regional_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
--max-instances 2 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--asia-proof-service-account "${ASIA_PROOF_SERVICE_ACCOUNT}" \
--regional-placement-secret-version "${regional_version}" \
--min-instances 0 \
--release-id "${RELEASE_ID}"
- name: Smoke director health
run: |
URL="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(status.url)')"
node dev/scripts/smoke-relay.mjs "${URL}"
@@ -0,0 +1,77 @@
name: Monitor Relay Cell Clock Skew
# Why: the 2026-08-01 sustained HOST_OFFLINE incident traced to one cell's
# clock running ~100ms ahead, which deterministically failed every host
# challenge under a zero-tolerance freshness check. /health and /ready cannot
# see clock skew; this monitor alarms before drift reaches the (now 2s)
# client tolerance.
on:
workflow_dispatch:
schedule:
- cron: '17 * * * *'
permissions:
contents: read
defaults:
run:
working-directory: cloud
jobs:
skew:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
steps:
- name: Measure Date-header skew for every relay cell
working-directory: .
run: |
set -u
# Date headers carry whole seconds; compare floored seconds on both
# sides so perfect sync reads 0/±1 and never flaps. An absolute
# floor-skew of >= 2 means real drift of at least ~1s — approaching
# the client's 2s challenge tolerance. Millisecond-precision deltas
# come from the desktop's named-check logging when activations fail.
ALARM_S=2
failures=0
reachable=0
unserved=0
# Keep this upper bound at or above the highest provisioned cell in
# environments/production.tfvars; unlisted cells are silently unmonitored.
for n in $(seq 1 22); do
cell="c${n}"
url="https://${cell}.relay.onorca.dev/health"
# Why: *.relay.onorca.dev is a wildcard, so the load balancer answers with its own
# accurate Date for a fenced, dead, or never-provisioned cell. Timing that reads as
# perfect sync. Only an HTTP 200 is the relay process itself answering, so only a
# 200 carries a clock worth judging.
response="$(curl -sS -D - -o /dev/null --max-time 8 "${url}" 2>/dev/null || true)"
status="$(printf '%s' "${response}" | awk 'NR==1 {print $2}')"
header="$(printf '%s' "${response}" | tr -d '\r' | grep -i '^date:' || true)"
if [ "${status:-000}" != "200" ] || [ -z "${header}" ]; then
# Expected for the fenced cells; a dead unfenced cell is caught by the heartbeat
# and readiness alerts, not here. Named either way so it is never invisible.
echo "${cell}: not serving (status ${status:-none}); no relay clock to judge"
unserved=$((unserved + 1))
continue
fi
reachable=$((reachable + 1))
server_s="$(date -d "${header#*: }" +%s)"
local_s="$(date +%s)"
skew=$((server_s - local_s))
abs=${skew#-}
if [ "${abs}" -ge "${ALARM_S}" ]; then
echo "::error::${cell}: clock skew ${skew}s reaches ±${ALARM_S}s alarm"
failures=$((failures + 1))
else
echo "${cell}: skew ${skew}s"
fi
done
echo "cells serving: ${reachable}, not serving: ${unserved}, alarms: ${failures}"
if [ "${reachable}" -eq 0 ]; then
# Now meaningful: previously the load balancer answered for every name, so this
# could never fire and a total fleet outage reported "all healthy".
echo "::error::no relay cell is serving; monitor blind"
exit 1
fi
exit "$((failures > 0 ? 1 : 0))"
@@ -0,0 +1,215 @@
name: Monitor Relay Production Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
expected-selector-generation:
required: true
type: string
expected-existing-only-cells:
required: true
type: string
expected-migration-only-cells:
required: true
type: string
expected-general-cells:
required: true
type: string
migration-policy:
required: true
type: string
recovery-source-cell-id:
required: true
type: string
capacity-cell-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 100
environment: production
env:
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
MIGRATION_POLICY: ${{ inputs.migration-policy }}
RECOVERY_SOURCE_CELL_ID: ${{ inputs.recovery-source-cell-id }}
CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }}
INCIDENT_ID: relay-${{ github.run_id }}-${{ inputs.mode }}
MONITOR_MODE: ${{ inputs.mode }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-incident
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
cache: pnpm
cache-dependency-path: cloud/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- id: prior-attempt
if: ${{ github.run_attempt > 1 }}
run: echo "value=$((GITHUB_RUN_ATTEMPT - 1))" >> "${GITHUB_OUTPUT}"
- name: Restore prior private monitor state
if: ${{ github.run_attempt > 1 }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ steps.prior-attempt.outputs.value }}
path: ${{ github.workspace }}/relay-incident
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Verify restored state provenance
if: ${{ github.run_attempt > 1 }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${{ steps.prior-attempt.outputs.value }}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
echo "RESTART_FLAG=--restart" >> "${GITHUB_ENV}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- name: Verify exact-audience admin identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
- name: Run read-only relay dry-run
if: ${{ inputs.mode == 'dry-run' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 15 \
--pre-drain-dry-run \
${RESTART_FLAG:-}
- name: Run first relay monitor segment
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--max-samples-this-run 45 \
${RESTART_FLAG:-}
- id: google-auth-refresh
if: ${{ inputs.mode == 'monitor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Run remaining relay monitor window
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-refresh.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--restart
- name: Seal private evidence provenance
if: ${{ always() }}
run: |
node dev/scripts/relay-monitor-evidence.mjs create \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
- name: Publish aggregate job summary
if: ${{ always() }}
run: |
if [[ -f "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" ]]; then
cat "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" >> "${GITHUB_STEP_SUMMARY}"
else
echo "Relay monitor failed before its first aggregate checkpoint." \
>> "${GITHUB_STEP_SUMMARY}"
fi
- name: Upload private aggregate evidence
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ github.workspace }}/relay-incident
if-no-files-found: error
retention-days: 14
@@ -0,0 +1,75 @@
name: Monitor Relay Production
on:
workflow_dispatch:
inputs:
mode:
description: Run the required 15-minute pre-drain gate or a 90-minute incident watch
required: true
default: dry-run
type: choice
options:
- dry-run
- monitor
expected-selector-generation:
description: Exact durable admission-selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact comma-separated existing-only cells, or none
required: true
type: string
expected-migration-only-cells:
description: Exact comma-separated migration-only cells, or none
required: true
type: string
expected-general-cells:
description: Exact comma-separated general cells, or none
required: true
type: string
migration-policy:
description: Migration checks matched to the intended mutation
required: true
default: strict
type: choice
options:
- strict
- recover-forward
- capacity-transition
recovery-source-cell-id:
description: Existing-only recovery source cell, or none for strict monitoring
required: true
default: none
type: string
capacity-cell-id:
description: General cell for a capacity-transition monitor, or none
required: true
default: none
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-monitor-relay-production-job.yml
with:
mode: ${{ inputs.mode }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
migration-policy: ${{ inputs.migration-policy }}
recovery-source-cell-id: ${{ inputs.recovery-source-cell-id }}
capacity-cell-id: ${{ inputs.capacity-cell-id }}
@@ -0,0 +1,514 @@
name: Operate Relay Asia Admission
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Inspect, initialize, verify, atomically register, promote, or roll back admission
required: true
default: verify
type: choice
options: [inspect, initialize, verify, register, configure, promote, rollback]
cell-ids:
description: Exact reviewed comma-separated Asia cell wave
required: true
type: string
selector-generation:
description: Exact live selector generation; leave empty only for inspect
required: false
type: string
selector-membership-sha256:
description: Exact fingerprint printed by inspect; required only for initialize
required: false
type: string
selector-attempt-id:
description: Durable unique attempt ID; empty for inspect, verify, and configure
required: false
type: string
image-digest:
description: Expected compatible Relay sha256 digest
required: true
type: string
director-image-digest:
description: Director sha256 digest; required only for configure
required: false
type: string
evidence-run-id:
description: Successful staging or C27 evidence workflow run ID; required for production promotion
required: false
type: string
evidence-run-attempt:
description: Exact evidence workflow run attempt; required for production promotion
required: false
type: string
confirmation:
description: Exact typed confirmation for a mutation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
admission:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DIRECTOR_ORIGIN: ${{ inputs.environment == 'production' && 'https://relay.onorca.dev' || 'https://relay-staging.onorca.dev' }}
AUTH_ORIGIN: ${{ inputs.environment == 'production' && 'https://login.onorca.dev' || 'https://auth-staging.onorca.dev' }}
DIRECTOR_SERVICE: ${{ inputs.environment == 'production' && 'orca-cloud-relay' || 'orca-cloud-relay-staging' }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
GCP_REGION: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_REGION || vars.STAGING_GCP_REGION }}
DIRECTOR_MAX_INSTANCES: ${{ inputs.environment == 'production' && '5' || '2' }}
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
OPERATION_MODE: ${{ inputs.mode }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
EXPECTED_SELECTOR_MEMBERSHIP_SHA256: ${{ inputs.selector-membership-sha256 }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
IMAGE_DIGEST: ${{ inputs.image-digest }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
EVIDENCE_RUN_ID: ${{ inputs.evidence-run-id }}
EVIDENCE_RUN_ATTEMPT: ${{ inputs.evidence-run-attempt }}
OPERATION_CONFIRMATION: ${{ inputs.confirmation }}
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate exact operation inputs before authentication
id: inputs
shell: bash
run: |
set -euo pipefail
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
evidence_kind=none
if test "${OPERATION_MODE}" = inspect; then
test -z "${EXPECTED_SELECTOR_GENERATION}"
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
else
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
fi
if test "${OPERATION_MODE}" = initialize; then
test "${EXPECTED_SELECTOR_GENERATION}" = 0
[[ "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" =~ ^[a-f0-9]{64}$ ]]
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${OPERATION_CONFIRMATION}" = INITIALIZE_ADMISSION_SELECTOR
elif test "${OPERATION_MODE}" = verify; then
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
elif test "${OPERATION_MODE}" = inspect; then
:
elif test "${OPERATION_MODE}" = configure; then
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${SELECTOR_ATTEMPT_ID}"
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
test "${OPERATION_CONFIRMATION}" = CONFIGURE_ASIA_DIRECTOR
else
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
case "${OPERATION_MODE}:${OPERATION_CONFIRMATION}" in
register:REGISTER_ASIA_MIGRATION_ONLY) ;;
promote:PROMOTE_ASIA_GENERAL) ;;
rollback:ROLLBACK_ASIA_MIGRATION_ONLY) ;;
*) echo "typed confirmation does not match the requested mutation" >&2; exit 1 ;;
esac
fi
if test "${TARGET_ENVIRONMENT}:${OPERATION_MODE}" = production:promote; then
[[ "${EVIDENCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${EVIDENCE_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
case "${TARGET_CELL_IDS}" in
production-gce-c27)
test "${#SELECTOR_ATTEMPT_ID}" -le 119
evidence_kind=staging
artifact_name="relay-asia-staging-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
;;
production-gce-c28,production-gce-c29)
evidence_kind=c27
artifact_name="relay-asia-c27-canary-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
;;
*) echo "production promotion wave is not reviewed" >&2; exit 1 ;;
esac
else
test -z "${EVIDENCE_RUN_ID}"
test -z "${EVIDENCE_RUN_ATTEMPT}"
artifact_name=none
fi
{
echo "evidence_kind=${evidence_kind}"
echo "artifact_name=${artifact_name}"
} >> "${GITHUB_OUTPUT}"
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
- name: Install exact C27 canary dependencies
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
run: pnpm install --frozen-lockfile
- name: Build the C27 canary Relay contract
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
run: pnpm --filter @orca-cloud/relay-contract build
- name: Download immutable rollout evidence
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
uses: actions/download-artifact@v4
with:
name: ${{ steps.inputs.outputs.artifact_name }}
path: ${{ runner.temp }}/relay-asia-input-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.evidence-run-id }}
- name: Verify evidence provenance and rollout binding before authentication
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
env:
GH_TOKEN: ${{ github.token }}
EVIDENCE_KIND: ${{ steps.inputs.outputs.evidence_kind }}
shell: bash
run: |
set -euo pipefail
run_json="${RUNNER_TEMP}/relay-asia-evidence-run.json"
verified="${RUNNER_TEMP}/relay-asia-evidence-verified"
gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${EVIDENCE_RUN_ID}/attempts/${EVIDENCE_RUN_ATTEMPT}" > "${run_json}"
evidence_commit_sha="$(
jq -er '.head_sha | select(type == "string" and test("^[a-f0-9]{40}$"))' "${run_json}"
)"
command=(node dev/scripts/relay-asia-rollout-evidence.mjs "verify-${EVIDENCE_KIND}"
--evidence "${RUNNER_TEMP}/relay-asia-input-evidence/evidence.json"
--run-json "${run_json}"
--commit-sha "${evidence_commit_sha}"
--image-digest "${IMAGE_DIGEST}"
--now "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
--output "${verified}")
if test "${EVIDENCE_KIND}" = c27; then
command+=(--selector-generation "${EXPECTED_SELECTOR_GENERATION}")
fi
"${command[@]}"
test "$(< "${verified}")" = verified
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ env.DIRECTOR_ORIGIN }}/v1/admin/drain
id_token_include_email: true
- name: Require the exact director image before promotion
if: ${{ inputs.mode == 'promote' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
if: ${{ inputs.mode == 'configure' || (inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27') }}
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode == 'configure' }}
with:
terraform_version: 1.15.8
terraform_wrapper: false
- name: Run the exact generation-bound admission operation
id: admission-operation
if: ${{ inputs.mode != 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode "${OPERATION_MODE}" \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
states="$(jq -cS '.states // {}' <<< "${result}")"
membership="$(jq -cS '.membership // empty' <<< "${result}")"
membership_sha256="$(jq -r '.membershipSha256 // empty' <<< "${result}")"
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
result_dir="${RUNNER_TEMP}/relay-asia-admission-result"
mkdir -p "${result_dir}"
node dev/scripts/sanitize-relay-asia-admission-result.mjs \
<<< "${result}" > "${result_dir}/result.json"
{
echo "### Relay Asia admission"
echo "- Mode: ${OPERATION_MODE}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Result generation: ${generation}"
echo "- States: \`${states}\`"
if test -n "${membership}"; then echo "- Membership: \`${membership}\`"; fi
if test -n "${membership_sha256}"; then
echo "- Membership SHA-256: \`${membership_sha256}\`"
fi
} >> "${GITHUB_STEP_SUMMARY}"
- name: Verify C27 state and start the timed canary
id: c27-start
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general
test "$(jq -r '.states["production-gce-c28"]' <<< "${result}")" = migration-only
test "$(jq -r '.states["production-gce-c29"]' <<< "${result}")" = migration-only
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run a real five-minute C27 control and splice canary
id: c27-load
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
shell: bash
run: |
set -euo pipefail
log="${RUNNER_TEMP}/relay-asia-c27-load.jsonl"
report="${RUNNER_TEMP}/relay-asia-c27-load.json"
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region asia-east2 \
--relay-asia-load-principals 1 \
--controls 1 \
--splices 1 \
--capacity-hard-cap 3000 \
--ramp-seconds 0 \
--duration-seconds 300 \
--splice-hold-seconds 60 \
--required-lease-horizons 2 > "${log}"
jq -cer 'select(.event == "relay_load_complete")' "${log}" | tail -n 1 > "${report}"
echo "ended_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Collect regional, Relay SQL, and Cloud SQL canary evidence
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
CANARY_STARTED_AT: ${{ steps.c27-start.outputs.started_at }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids production-gce-c27,production-gce-c28,production-gce-c29 \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = general
ended_at="${{ steps.c27-load.outputs.ended_at }}"
sleep 60
output="${RUNNER_TEMP}/relay-asia-output-evidence"
logs="${RUNNER_TEMP}/relay-asia-c27-runtime-metrics.json"
mkdir -p "${output}"
gcloud logging read \
"timestamp>=\"${CANARY_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" \
--limit 20000 \
--format json > "${logs}"
node dev/scripts/relay-asia-rollout-evidence.mjs create-c27 \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.admission-operation.outputs.generation }}" \
--started-at "${CANARY_STARTED_AT}" \
--ended-at "${ended_at}" \
--load-report "${RUNNER_TEMP}/relay-asia-c27-load.json" \
--logs-json "${logs}" \
--output "${output}/evidence.json"
jq -r '.metrics | to_entries[] | "- \(.key): \(.value)"' \
"${output}/evidence.json" >> "${GITHUB_STEP_SUMMARY}"
- name: Upload immutable C27 canary evidence
id: c27-evidence-upload
if: ${{ inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-c27-canary-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-output-evidence/evidence.json
if-no-files-found: error
retention-days: 7
- name: Upload sanitized admission result
if: ${{ inputs.mode != 'configure' && steps.admission-operation.outcome == 'success' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-admission-result-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-admission-result/result.json
if-no-files-found: error
retention-days: 7
- name: Return an unproven C27 canary to migration-only
if: ${{ always() && inputs.environment == 'production' && inputs.mode == 'promote' && inputs.cell-ids == 'production-gce-c27' && steps.admission-operation.outcome != 'skipped' && steps.c27-evidence-upload.outcome != 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode recover-promotion \
--cell-ids production-gce-c27 \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode rollback \
--cell-ids production-gce-c27 \
--expected-generation "${promoted_generation}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}-rollback" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["production-gce-c27"]' <<< "${result}")" = migration-only
- name: Require registered migration-only cells before director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode registered \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
- name: Build the additive director cell configuration
if: ${{ inputs.mode == 'configure' }}
id: director-config
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
topology="${RUNNER_TEMP}/relay-asia-state-topology.json"
current="${RUNNER_TEMP}/relay-current-director-cells.json"
desired="${RUNNER_TEMP}/relay-asia-director-cells.json"
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${topology}"
service="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
revision="$(jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end' \
<<< "${service}")"
gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '.spec.containers[0].env[] | select(.name == "ORCA_RELAY_CELLS_JSON") | .value | fromjson' \
> "${current}"
node dev/scripts/prepare-relay-asia-director-cells.mjs \
--current-json "${current}" \
--topology-json "${topology}" \
--output "${desired}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image-digest "${IMAGE_DIGEST}"
echo "file=${desired}" >> "${GITHUB_OUTPUT}"
- name: Deploy the registered additive director topology
if: ${{ inputs.mode == 'configure' }}
env:
DIRECTOR_CELLS_FILE: ${{ steps.director-config.outputs.file }}
run: |
current="$(gcloud secrets versions access latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
image="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}"
release_id="asia-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE}" \
--image "${image}" \
--role director \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--release-id "${release_id}" \
--director-cells-json "$(< "${DIRECTOR_CELLS_FILE}")" \
--prune-revisions false
- name: Verify selector and heartbeats after director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode verify \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
revision="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end')"
revision_json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
jq -e --arg image "us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}" \
'.spec.containers[0].image == $image' <<< "${revision_json}" > /dev/null
jq -er --arg secret "${REGIONAL_PLACEMENT_SECRET}" \
'[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)} |
select(.secret == $secret and (.version | test("^[1-9][0-9]*$")))] |
if length == 1 then .[0].version else error("regional switch version missing") end' \
<<< "${revision_json}" > "${RUNNER_TEMP}/relay-regional-placement-version"
regional_version="$(< "${RUNNER_TEMP}/relay-regional-placement-version")"
[[ "$(gcloud secrets versions access "${regional_version}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" =~ ^(true|false)$ ]]
echo "Director configuration now lists the registered migration-only Asia cells." >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,328 @@
name: Operate Relay Production Rehome Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
director-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-control-generation: { required: true, type: string }
not-before: { required: true, type: string }
rate-per-minute: { required: true, type: string }
preference-max-age-ms: { required: true, type: string }
drain-grace-ms: { required: true, type: string }
confirmation: { required: true, type: string }
monitor-run-id: { required: true, type: string }
monitor-run-attempt: { required: true, type: string }
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
control:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
MODE: ${{ inputs.mode }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_CONTROL_GENERATION: ${{ inputs.expected-control-generation }}
NOT_BEFORE: ${{ inputs.not-before }}
RATE_PER_MINUTE: ${{ inputs.rate-per-minute }}
PREFERENCE_MAX_AGE_MS: ${{ inputs.preference-max-age-ms }}
DRAIN_GRACE_MS: ${{ inputs.drain-grace-ms }}
CONFIRMATION: ${{ inputs.confirmation }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${MODE}" =~ ^(inspect|enable|pause|disable)$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_CONTROL_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
if [[ "${MODE}" =~ ^(inspect|enable)$ ]]; then
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test -n "${GCP_REGION}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
fi
case "${MODE}" in
inspect)
test -z "${CONFIRMATION}"
;;
enable)
test "${CONFIRMATION}" = ENABLE_REGIONAL_REHOMING
test "${RATE_PER_MINUTE}" = 10
[[ "${NOT_BEFORE}" =~ ^[1-9][0-9]*$ ]]
;;
pause)
test "${CONFIRMATION}" = PAUSE_REGIONAL_REHOMING
;;
disable)
test "${CONFIRMATION}" = DISABLE_REGIONAL_REHOMING
;;
esac
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Apply emergency durable pause or disable before diagnostics
if: ${{ inputs.mode == 'pause' || inputs.mode == 'disable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- uses: pnpm/action-setup@v4
if: ${{ inputs.mode == 'enable' }}
with: { package_json_file: cloud/package.json }
- run: pnpm install --frozen-lockfile
if: ${{ inputs.mode == 'enable' }}
- name: Download fresh aggregate safety evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify enable evidence provenance
if: ${{ inputs.mode == 'enable' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" --run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" --mode dry-run \
--required-migration-policy strict
- name: Reject previously consumed enable safety evidence
if: ${{ inputs.mode == 'enable' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = 0
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Verify exact serving and rollback director identities
if: ${{ inputs.mode == 'inspect' || inputs.mode == 'enable' }}
env:
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
SERVING_REVISION="$(jq -er \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName
else error("director does not have one serving revision") end' \
<<< "${SERVICE_JSON}")"
ROLLBACK_REVISION="$(jq -er \
'[.status.traffic[] | select(.tag == "selector-rollback")] |
if length == 1 then .[0].revisionName else error("rollback tag missing") end' \
<<< "${SERVICE_JSON}")"
verify_revision() {
local revision="$1" expected_digest="$2"
local json
json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
test "$(jq -r '.spec.serviceAccountName' <<< "${json}")" = \
"${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '.spec.containers[0].image | split("@") | last' <<< "${json}")" = \
"${expected_digest}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT") | .value] | if length == 1
then .[0] else empty end' <<< "${json}")" = "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_AUDIENCE") | .value] | if length == 1 then .[0]
else empty end' <<< "${json}")" = "${REHOME_AUDIENCE}"
}
verify_revision "${SERVING_REVISION}" "${DIRECTOR_IMAGE_DIGEST}"
verify_revision "${ROLLBACK_REVISION}" "${ROLLBACK_IMAGE_DIGEST}"
- name: Seal 24-hour aggregate region observation evidence
if: ${{ inputs.mode == 'enable' }}
run: |
mkdir -p "${RUNNER_TEMP}/relay-region-observation"
# 6 director instances x 120 samples/hour x 25h = 18000; a clipped
# read empties the oldest hourly buckets and fails the seal.
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND jsonPayload.event="orca_relay_runtime_metrics" AND jsonPayload.role="director"' \
--project "${GCP_PROJECT_ID}" --freshness=25h --limit=30000 --format=json \
| node dev/scripts/relay-region-observation-evidence.mjs create \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}" \
> "${RUNNER_TEMP}/relay-region-observation/evidence.json"
- name: Upload sealed 24-hour aggregate region evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-region-observation-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-region-observation/evidence.json
retention-days: 90
if-no-files-found: error
- name: Verify sealed 24-hour enable authority
if: ${{ inputs.mode == 'enable' }}
run: |
node dev/scripts/relay-region-observation-evidence.mjs verify \
--file "${RUNNER_TEMP}/relay-region-observation/evidence.json" \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}"
- name: Recheck every aggregate safety signal before enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Seal single-use enable safety authority
if: ${{ inputs.mode == 'enable' }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-rehome-enable-authority"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-rehome-enable-authority/${MARKER_NAME}"
- name: Consume enable safety evidence before durable mutation
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-rehome-enable-authority/relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Inspect regional rehome control
if: ${{ inputs.mode == 'inspect' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Apply exact durable regional rehome enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Read fresh aggregate completion and abort evidence
run: |
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND textPayload:"[orca-relay] regional rehome inventory"' \
--project "${GCP_PROJECT_ID}" --freshness=15m --limit=20 --format=json \
| node dev/scripts/relay-rehome-aggregate-evidence.mjs --max-age-ms 900000 \
| tee "${RUNNER_TEMP}/relay-rehome-inventory.json"
- name: Publish aggregate control evidence
run: |
{
echo '### Regional rehome control'
jq -r '"- mode: `\(.mode)`\n- generation: `\(.control.generation)`\n- enabled: `\(.control.enabled)`"' \
"${RUNNER_TEMP}/relay-rehome-control.json"
jq -r '"- active: `\(.active)`\n- awaiting receipt: `\(.awaitingReceipt)`\n- target registered: `\(.targetRegistered)`\n- completed (24h): `\(.completedLast24Hours)`\n- aborted (24h): `\(.abortedLast24Hours)`"' \
"${RUNNER_TEMP}/relay-rehome-inventory.json"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Fail closed after an unsuccessful enable run
if: ${{ failure() && inputs.mode == 'enable' && steps.google-auth.outcome == 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode recover-enable --director-origin "${DIRECTOR_ORIGIN}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--confirmation RECOVER_FAILED_REGIONAL_REHOME_ENABLE \
| tee "${RUNNER_TEMP}/relay-rehome-enable-recovery.json"
jq -e \
'.mode == "recover-enable" and .control.enabled == false' \
"${RUNNER_TEMP}/relay-rehome-enable-recovery.json" >/dev/null
@@ -0,0 +1,106 @@
name: Operate Relay Production Rehome
on:
workflow_dispatch:
inputs:
mode:
description: Inspect or apply the durable regional-rehome switch
required: true
default: inspect
type: choice
options: [inspect, enable, pause, disable]
director-image-digest:
description: Exact immutable serving director digest
required: true
type: string
rollback-image-digest:
description: Exact immutable selector-rollback director digest
required: true
type: string
expected-selector-generation:
description: Exact admission selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-control-generation:
description: Exact durable rehome generation
required: true
type: string
not-before:
description: Exact epoch milliseconds; ignored only by inspect
required: true
default: '0'
type: string
rate-per-minute:
description: Exact global host rate; initial enable is fixed at 10
required: true
default: '10'
type: string
preference-max-age-ms:
description: Maximum fresh preference age
required: true
default: '86400000'
type: string
drain-grace-ms:
description: Per-host source drain grace
required: true
default: '3600000'
type: string
monitor-run-id:
description: Fresh successful aggregate dry-run required only by enable
required: false
type: string
monitor-run-attempt:
description: Exact monitor attempt required only by enable
required: false
type: string
confirmation:
description: ENABLE_REGIONAL_REHOMING, PAUSE_REGIONAL_REHOMING, or DISABLE_REGIONAL_REHOMING
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
operate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-operate-relay-production-rehome-job.yml
with:
mode: ${{ inputs.mode }}
director-image-digest: ${{ inputs.director-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-control-generation: ${{ inputs.expected-control-generation }}
not-before: ${{ inputs.not-before }}
rate-per-minute: ${{ inputs.rate-per-minute }}
preference-max-age-ms: ${{ inputs.preference-max-age-ms }}
drain-grace-ms: ${{ inputs.drain-grace-ms }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
secrets: inherit
@@ -0,0 +1,101 @@
name: Power Relay Staging
on:
schedule:
# A zero-activity guard makes this a no-op when an internal test is still running.
- cron: '0 9 * * *'
workflow_dispatch:
inputs:
mode:
description: Inspect, wake, or sleep the staging Relay data plane
required: true
default: status
type: choice
options:
- status
- wake
- sleep
wake-cells:
description: Wake configured admission cells, or include disabled candidate cells
required: true
default: configured
type: choice
options:
- configured
- all
confirmation:
description: Enter WAKE_STAGING or SLEEP_STAGING for a manual mutation
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
power:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
POWER_MODE: ${{ github.event_name == 'schedule' && 'sleep' || inputs.mode }}
WAKE_CELLS: ${{ inputs.wake-cells || 'configured' }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit manual mutation confirmation
if: ${{ github.event_name == 'workflow_dispatch' && inputs.mode != 'status' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${POWER_MODE}" = "wake" ]]; then
test "${CONFIRMATION}" = "WAKE_STAGING"
else
test "${CONFIRMATION}" = "SLEEP_STAGING"
fi
- name: Read reviewed staging topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
- name: Inspect or change staging power state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/power-staging-relay.mjs \
--mode "${POWER_MODE}" \
--wake-cells "${WAKE_CELLS}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json"
@@ -0,0 +1,331 @@
name: Prove Relay Asia Staging
on:
workflow_dispatch:
inputs:
image-digest:
description: Exact immutable Relay digest deployed on staging C4
required: true
type: string
selector-generation:
description: Exact selector generation with C4 migration-only
required: true
type: string
promote-attempt-id:
description: Durable unique C4 promotion attempt ID
required: true
type: string
rollback-attempt-id:
description: Durable unique C4 rollback attempt ID
required: true
type: string
confirmation:
description: Enter PROVE_ASIA_STAGING
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
prove:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: [self-hosted, linux, x64, relay-asia-east2-load]
timeout-minutes: 75
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
AUTH_ORIGIN: https://auth-staging.onorca.dev
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact staging proof request
shell: bash
run: |
set -euo pipefail
test "${{ inputs.confirmation }}" = PROVE_ASIA_STAGING
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "${INITIAL_SELECTOR_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${PROMOTE_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
[[ "${ROLLBACK_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${PROMOTE_ATTEMPT_ID}" != "${ROLLBACK_ATTEMPT_ID}"
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- name: Require the exact staging director image before promotion
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- name: Install exact load-harness dependencies
run: pnpm install --frozen-lockfile
- name: Build the Relay load-harness contract
run: pnpm --filter @orca-cloud/relay-contract build
- name: Promote only staging C4
id: promote
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode promote \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = general
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run sharded two-horizon and mixed splice proofs
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
mkdir -p "${proof_dir}"
fd_limit="$(ulimit -n)"
if test "${fd_limit}" != unlimited; then
[[ "${fd_limit}" =~ ^[0-9]+$ ]]
test "${fd_limit}" -ge 4096
fi
run_phase() {
phase="$1"
controls="$2"
splices="$3"
pids=()
stop_shards() {
for pid in "${pids[@]}"; do kill "${pid}" 2>/dev/null || true; done
for pid in "${pids[@]}"; do wait "${pid}" 2>/dev/null || true; done
}
trap stop_shards EXIT
for shard in 0 1 2 3; do
slow=0
wedged=0
boundary_args=()
request_unit_args=()
if test "${phase}" = launch && test "${shard}" = 0; then
slow=4
wedged=1
fi
if test "${phase}" = launch && test "${shard}" = 0; then
boundary_args=(
--region-behavior-probes 1
--capacity-cell-id staging-gce-c4
--capacity-cell-origin https://c4.relay-staging.onorca.dev
--capacity-unobserved-bound 60
--rebind-probes 2
--skip-rebind-overflow-check
)
fi
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region asia-east2 \
--relay-asia-load-principals 32 \
--controls "${controls}" \
--splices "${splices}" \
--slow-reader-splices "${slow}" \
--wedged-reader-splices "${wedged}" \
--capacity-hard-cap 3000 \
"${boundary_args[@]}" \
"${request_unit_args[@]}" \
--phase-barrier-dir "${proof_dir}/${phase}-barrier" \
--aggregate-controls "$((controls * 4))" \
--aggregate-splices "$((splices * 4))" \
--aggregate-reader-splices "$([[ "${phase}" = launch ]] && echo 5 || echo 0)" \
--aggregate-reader-bytes "$([[ "${phase}" = launch ]] && echo 12582912 || echo 0)" \
--required-lease-horizons 2 \
--splice-ramp-seconds 120 \
--max-generator-rss-growth-mib 512 \
--ramp-seconds 180 \
--duration-seconds 210 \
--shard-count 4 \
--shard-index "${shard}" \
> "${proof_dir}/${phase}-${shard}.jsonl" &
pids+=("$!")
done
failed=0
for pid in "${pids[@]}"; do
if ! wait "${pid}"; then failed=1; break; fi
done
if test "${failed}" = 1; then
stop_shards
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_progress" or .event == "relay_load_complete") |
{event, shardIndex, active, peakActive, connected, connectionFailures,
rampConnectionFailures, connectionFailuresByReason, unexpectedCloses,
protocolErrors, refreshErrors, socketErrors, elapsedSeconds}' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1 || true
done
trap - EXIT
return 1
fi
trap - EXIT
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_complete")' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1
done | jq -s . > "${proof_dir}/${phase}.json"
}
run_phase launch 5 5
- name: Collect and validate aggregate staging proof evidence
env:
PROOF_STARTED_AT: ${{ steps.promote.outputs.started_at }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
ended_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
sleep 60
gcloud logging read \
"timestamp>=\"${PROOF_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" --limit 20000 --format json \
> "${proof_dir}/runtime-metrics.json"
node dev/scripts/relay-asia-rollout-evidence.mjs create-staging \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.promote.outputs.generation }}" \
--started-at "${PROOF_STARTED_AT}" \
--ended-at "${ended_at}" \
--launch-report "${proof_dir}/launch.json" \
--logs-json "${proof_dir}/runtime-metrics.json" \
--output "${proof_dir}/evidence.json"
- name: Return staging C4 to migration-only
if: ${{ always() && steps.promote.outcome != 'skipped' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
- name: Upload immutable staging readiness evidence
if: ${{ success() }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-staging-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-staging-proof/evidence.json
if-no-files-found: error
retention-days: 7
recover:
if: ${{ always() && github.ref == 'refs/heads/main' }}
needs: prove
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 15
environment: staging
env:
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Recover staging C4 with a fresh identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
@@ -0,0 +1,917 @@
name: Prove Relay Staging Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify or change C3 capacity, restore admission, or refresh empty Asia C4
required: true
default: verify
type: choice
options:
- verify
- apply
- restore-admission
- refresh-asia-c4-image
expected-hard-cap:
description: Exact cap declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '600'
type: choice
options:
- '1000'
- '600'
expected-unobserved-bound:
description: Exact bound declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '60'
type: choice
options:
- '60'
- '0'
confirmation:
description: Exact confirmation required for a mutation
required: false
type: string
expected-selector-generation:
description: Exact staging selector generation for a C4 image refresh
required: false
type: string
predecessor-image-digest:
description: Exact current C4 sha256 digest
required: false
type: string
target-image-digest:
description: Exact desired C4 sha256 digest
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && inputs.mode != 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c3.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c3
FALLBACK_CELL_ORIGIN: https://c2.relay-staging.onorca.dev
FALLBACK_CELL_ID: staging-gce-c2
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
EXPECTED_HARD_CAP: ${{ inputs.expected-hard-cap }}
EXPECTED_UNOBSERVED_BOUND: ${{ inputs.expected-unobserved-bound }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode != 'restore-admission' }}
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
if: ${{ inputs.mode != 'restore-admission' }}
run: node dev/scripts/infra.mjs init --env staging
- name: Require reviewed desired capacity and image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
CAP_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_hard_cap"
BOUND_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_unobserved_bound"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_CAP="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${CAP_EXPRESSION}")"
DESIRED_BOUND="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${BOUND_EXPRESSION}")"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
CELL_ORIGIN="$(jq -r --arg cell "${TARGET_CELL_ID}" \
'.[] | select(.id == $cell) | .url' <<< "${DESIRED_CELLS_JSON}")"
test "${DESIRED_CAP}" = "${EXPECTED_HARD_CAP}"
test "${DESIRED_BOUND}" = "${EXPECTED_UNOBSERVED_BOUND}"
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
[[ "${MIG_NAME}" =~ ^[a-z0-9-]+$ ]]
test "${CELL_ORIGIN}" = "https://c3.relay-staging.onorca.dev"
jq -e \
--arg cell "${TARGET_CELL_ID}" \
--arg fallback "${FALLBACK_CELL_ID}" \
--argjson cap "${EXPECTED_HARD_CAP}" \
--argjson bound "${EXPECTED_UNOBSERVED_BOUND}" \
'(any(.[]; .id == $cell and .connectionHardCap == $cap and .connectionUnobservedBound == $bound)) and
(any(.[]; .id == $fallback and .connectionHardCap == 600 and .connectionUnobservedBound == 60))' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
echo "DESIRED_IMAGE=${DESIRED_IMAGE}" >> "${GITHUB_ENV}"
echo "TARGET_ZONE=${TARGET_ZONE}" >> "${GITHUB_ENV}"
echo "MIG_NAME=${MIG_NAME}" >> "${GITHUB_ENV}"
echo "CELL_ORIGIN=${CELL_ORIGIN}" >> "${GITHUB_ENV}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" >> "${GITHUB_ENV}"
- name: Verify exact compatible director image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_IMAGE="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${ACTIVE_IMAGE}" = "${DESIRED_IMAGE}"
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}" >> "${GITHUB_ENV}"
- name: Require explicit transition confirmation
if: ${{ inputs.mode == 'apply' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "FENCE_AND_TRANSITION_STAGING_C3"
- name: Require explicit admission restore confirmation
if: ${{ inputs.mode == 'restore-admission' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "RESTORE_STAGING_C2_C3_GENERAL"
- name: Require capacity identity and exact predecessor state
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
case "${EXPECTED_HARD_CAP}/${EXPECTED_UNOBSERVED_BOUND}" in
1000/0)
PREDECESSOR_C3_CAP=600
PREDECESSOR_C3_BOUND=60
;;
1000/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=0
;;
600/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=60
;;
*)
echo "Unsupported staging capacity transition" >&2
exit 1
;;
esac
ACTIVE_REVISION="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
PREDECESSOR_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${PREDECESSOR_C3_CAP}" \
--argjson bound "${PREDECESSOR_C3_BOUND}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${DESIRED_CELLS_JSON}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
if jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${DESIRED_CELLS_JSON}" \
'$current == $expected'; then
if node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-active
elif node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-ready
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe
TRANSITION_PHASE=director-ready
fi
else
jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${PREDECESSOR_CELLS_JSON}" \
'$current == $expected'
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${PREDECESSOR_C3_CAP}" \
--unobserved-bound "${PREDECESSOR_C3_BOUND}" \
--heartbeat fresh \
--admission either \
--draining either \
--activity allowed
TRANSITION_PHASE=predecessor
fi
echo "TRANSITION_PHASE=${TRANSITION_PHASE}" >> "${GITHUB_ENV}"
- name: Restore C2 as the safe placement fallback
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
- name: Require a healthy non-draining C3 before restoring it
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
- name: Require a healthy general C2 before isolating C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-active; then
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Reversibly isolate and drain C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Verify restart-safe migration-only target
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe
- name: Verify current capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Deploy reviewed director topology and remove pre-protocol revisions
if: ${{ inputs.mode == 'apply' }}
run: |
if test "${TRANSITION_PHASE}" != predecessor; then
echo "DIRECTOR_CONFIG_CHANGED=false" >> "${GITHUB_ENV}"
exit 0
fi
RELEASE_ID="capacity-compat-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
DEPLOY_RESULT="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--min-instances 0 \
--prune-revisions true \
--release-id "${RELEASE_ID}")"
echo "${DEPLOY_RESULT}"
DIRECTOR_CONFIG_CHANGED="$(jq -r '.topologyChanged' <<< "${DEPLOY_RESULT}")"
[[ "${DIRECTOR_CONFIG_CHANGED}" =~ ^(true|false)$ ]]
echo "DIRECTOR_CONFIG_CHANGED=${DIRECTOR_CONFIG_CHANGED}" >> "${GITHUB_ENV}"
- name: Require fail-closed director transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
exit 0
fi
HEARTBEAT_EXPECTATION=either
if test "${DIRECTOR_CONFIG_CHANGED}" = true; then
HEARTBEAT_EXPECTATION=stale
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat "${HEARTBEAT_EXPECTATION}" \
--admission migration-only \
--draining required \
--activity restart-safe
- name: Plan and apply only the exact empty cell
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
recreate_fixed_one_instance() {
local instance
instance="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("capacity cell does not have one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" \
--instances "${instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--quiet
}
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c3"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c3"]' \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}")"
echo "${PLAN_RESULT}"
CELL_PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${CELL_PLAN_CHANGES}" =~ ^(0|2)$ ]]
if test "${TRANSITION_PHASE}" = cell-ready; then
test "${CELL_PLAN_CHANGES}" = 0
elif test "${TRANSITION_PHASE}" = cell-active; then
test "${CELL_PLAN_CHANGES}" = 0
recreate_fixed_one_instance
elif test "${CELL_PLAN_CHANGES}" = 0; then
recreate_fixed_one_instance
else
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--timeout 900
- name: Verify exact live cap and fresh matching heartbeat
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
- name: Make C3 the only staging placement cell
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the sole general canary after transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Restore the reviewed C2 and C3 placement set
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
- name: Verify restored C3 admission and capacity
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Preserve C2 as the safe fallback after a failed transition
if: ${{ failure() && inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
refresh-asia-c4-image:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && github.ref == 'refs/heads/main' && inputs.mode == 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
APPROVED_PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
steps:
- uses: actions/checkout@v4
- name: Require the exact bounded C4 refresh
env:
CONFIRMATION: ${{ inputs.confirmation }}
shell: bash
run: |
set -euo pipefail
test "${CONFIRMATION}" = REFRESH_STAGING_ASIA_C4_IMAGE
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PREDECESSOR_IMAGE_DIGEST}" != "${TARGET_IMAGE_DIGEST}"
test "${PREDECESSOR_IMAGE_DIGEST}" = "${APPROVED_PREDECESSOR_IMAGE_DIGEST}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- name: Resolve the reviewed C4 image and shape
shell: bash
run: |
set -euo pipefail
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
shape="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${cells}")"
test "$(jq -r '.hostname' <<< "${shape}")" = c4
test "$(jq -r '.region' <<< "${shape}")" = asia-east2
test "$(jq -r '.zone' <<< "${shape}")" = asia-east2-a
test "$(jq -r '.machine_type' <<< "${shape}")" = e2-standard-4
test "$(jq -r '.capacity_requests' <<< "${shape}")" = 6000
test "$(jq -r '.database_pool_max' <<< "${shape}")" = 10
test "$(jq -r '.connection_hard_cap' <<< "${shape}")" = 3000
test "$(jq -r '.connection_unobserved_bound' <<< "${shape}")" = 60
test "$(jq -r '.initially_enabled' <<< "${shape}")" = false
desired_image="$(jq -r '.image' <<< "${shape}")"
test "${desired_image}" = \
"us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${desired_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${TARGET_IMAGE_DIGEST}"
mig_name="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
test "${mig_name}" = orca-cloud-staging-relay-gce-c4
{
echo "DESIRED_IMAGE=${desired_image}"
echo "ROLLBACK_IMAGE=us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${PREDECESSOR_IMAGE_DIGEST}"
echo "TARGET_ZONE=asia-east2-a"
echo "MIG_NAME=${mig_name}"
} >> "${GITHUB_ENV}"
- name: Save, validate, and classify the exact C4 plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
case "$(jq -r '[.changes,.changeKind] | join(":")' <<< "${result}")" in
2:replacement) refresh_phase=predecessor ;;
0:none|*:obsolete-template-delete) refresh_phase=applied ;;
*:manager-convergence|*:replacement-with-obsolete-template) refresh_phase=converging ;;
*) exit 1 ;;
esac
{
echo "PLAN_CHANGES=$(jq -r '.changes' <<< "${result}")"
echo "REFRESH_PHASE=${refresh_phase}"
} >> "${GITHUB_ENV}"
- id: state-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the exact selector and current C4 state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.state-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${inspect}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
expected_digests="${TARGET_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = predecessor; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST}"
elif test "${REFRESH_PHASE}" = converging; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
case ",${expected_digests}," in
*,"${current_digest}",*) ;;
*) exit 1 ;;
esac
source_incarnation=''
draining=forbidden
if test "${REFRESH_PHASE}" != applied; then
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
source_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
[[ "${source_incarnation}" =~ ^[0-9a-f-]{36}$ ]]
if test "$(jq -r '.draining' <<< "${runtime}")" = true; then
draining=required
fi
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining "${draining}" \
--activity quiescent --expected-image-digests "${expected_digests}"
runtime_available=true
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 300000
source_incarnation=''
runtime_available=false
fi
{
echo "MIG_STABLE_AT_MS=0"
echo "MUTATION_STARTED=false"
echo "REPLACEMENT_STARTED_AT_MS=0"
echo "RUNTIME_AVAILABLE=${runtime_available}"
echo "SOURCE_INCARNATION=${source_incarnation}"
} >> "${GITHUB_ENV}"
- id: fence-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 and prove it stayed empty before replacement
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
if test "${REFRESH_PHASE}" = applied; then exit 0; fi
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
if test "${RUNTIME_AVAILABLE}" = false; then exit 0; fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate
fence_digests="${PREDECESSOR_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = converging; then
fence_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent --expected-image-digests "${fence_digests}"
- name: Apply the exact saved C4 plan
shell: bash
run: |
set -euo pipefail
if test "${PLAN_CHANGES}" = 0 && test "${RUNTIME_AVAILABLE}" = true; then exit 0; fi
if test "${REFRESH_PHASE}" = applied && test "${RUNTIME_AVAILABLE}" = false; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
fi
if test "${REFRESH_PHASE}" != applied; then
replacement_started_at_ms="$(date -u +%s%3N)"
echo "REPLACEMENT_STARTED_AT_MS=${replacement_started_at_ms}" >> "${GITHUB_ENV}"
fi
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
if test "${RUNTIME_AVAILABLE}" = false && test "${REFRESH_PHASE}" = applied; then
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
fi
echo "MIG_STABLE_AT_MS=$(date -u +%s%3N)" >> "${GITHUB_ENV}"
- id: post-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new C4 incarnation, image, and unchanged isolation
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${TARGET_IMAGE_DIGEST}" \
--timeout-ms 240000
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${result}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"; then break; fi
sleep 5
done
target_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"
if test "${REFRESH_PHASE}" != applied; then
if test -n "${SOURCE_INCARNATION}"; then
test "${target_incarnation}" != "${SOURCE_INCARNATION}"
fi
test "$(jq -r '.status.runtime.startedAt' <<< "${status}")" \
-ge "${REPLACEMENT_STARTED_AT_MS}"
fi
- name: Require an empty targeted Terraform readback
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
test "$(jq -r '.changes' <<< "${result}")" = 0
@@ -0,0 +1,131 @@
name: Publish Relay Production Image
on:
workflow_dispatch:
inputs:
mode:
description: Publish a new production image or mirror an existing immutable image to staging
required: true
default: publish
type: choice
options: [publish, mirror-staging]
image-digest:
description: Exact existing production digest for mirror-staging mode
required: false
type: string
confirmation:
description: Enter MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING for mirror-staging mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: publish-relay-production
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
publish:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
PUBLISH_MODE: ${{ inputs.mode }}
MIRROR_DIGEST: ${{ inputs.image-digest }}
MIRROR_CONFIRMATION: ${{ inputs.confirmation }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact publish request before authentication
shell: bash
run: |
set -euo pipefail
if test "${PUBLISH_MODE}" = mirror-staging; then
[[ "${MIRROR_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${MIRROR_CONFIRMATION}" = MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING
else
test "${PUBLISH_MODE}" = publish
test -z "${MIRROR_DIGEST}"
test -z "${MIRROR_CONFIRMATION}"
fi
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: docker/setup-buildx-action@v3
- name: Configure Docker auth
run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet
- name: Build and publish immutable image
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}:sha-${GITHUB_SHA}"
docker build -f apps/relay/Dockerfile -t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${DIGEST}"
{
echo '### Terraform candidate image'
echo
echo "\`${IMAGE}\`"
echo
echo 'Declare this digest on a distinct disabled candidate cell in a reviewed Terraform PR.'
} >> "${GITHUB_STEP_SUMMARY}"
- name: Build and publish immutable fence broker
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker:sha-${GITHUB_SHA}"
docker build \
--build-arg "ORCA_RELAY_FENCE_IMAGE_COMMIT=${GITHUB_SHA}" \
-f apps/relay-fence-broker/Dockerfile \
-t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker@${DIGEST}"
{
echo
echo '### Terraform fence broker image'
echo
echo "\`${IMAGE}\`"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Mirror the exact production manifest to staging
if: ${{ inputs.mode == 'mirror-staging' }}
shell: bash
run: |
set -euo pipefail
source_image="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${MIRROR_DIGEST}"
target_tag="${GCP_REGION}-docker.pkg.dev/onorca-cloud-staging/${REPOSITORY_ID}/${IMAGE_NAME}:production-${MIRROR_DIGEST#sha256:}"
source_digest="$(gcloud artifacts docker images describe "${source_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${source_digest}" = "${MIRROR_DIGEST}"
docker pull "${source_image}"
docker tag "${source_image}" "${target_tag}"
docker push "${target_tag}"
target_digest="$(gcloud artifacts docker images describe "${target_tag}" \
--project onorca-cloud-staging --format='value(image_summary.digest)')"
test "${target_digest}" = "${MIRROR_DIGEST}"
{
echo '### Mirrored Relay image'
echo
printf 'Production and staging now resolve the same immutable digest: %s.\n' \
"${MIRROR_DIGEST}"
} >> "${GITHUB_STEP_SUMMARY}"
@@ -0,0 +1,411 @@
name: Recover Relay Staging C4 Image
on:
workflow_run:
workflows: [Prove Relay Staging Capacity]
types: [completed]
workflow_dispatch:
inputs:
confirmation:
description: Enter RECOVER_STAGING_ASIA_C4_IMAGE
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event_name == 'workflow_dispatch' || (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion != 'success')) }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
outputs:
recover: ${{ steps.trigger.outputs.recover }}
steps:
- name: Bind recovery to the exact failed C4 job
working-directory: .
id: trigger
env:
CONFIRMATION: ${{ inputs.confirmation }}
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
SOURCE_RUN_EVENT: ${{ github.event.workflow_run.event }}
shell: bash
run: |
set -euo pipefail
if test "${GITHUB_EVENT_NAME}" = workflow_dispatch; then
test "${CONFIRMATION}" = RECOVER_STAGING_ASIA_C4_IMAGE
echo "recover=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${SOURCE_RUN_EVENT}" = workflow_dispatch
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "refresh-asia-c4-image" and
(.conclusion == "failure" or .conclusion == "cancelled" or
.conclusion == "timed_out"))] | length' <<< "${jobs}")"
if test "${count}" = 0; then
echo "recover=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${count}" = 1
echo "recover=true" >> "${GITHUB_OUTPUT}"
recover:
needs: gate
if: ${{ needs.gate.outputs.recover == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
TARGET_ZONE: asia-east2-a
MIG_NAME: orca-cloud-staging-relay-gce-c4
PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
TARGET_IMAGE_DIGEST: sha256:5aedbca5c86de24c8b4d4bf7e3b444b76c712f281ede916cb9d90f70cad1e563
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- id: preflight-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Inspect the exact C4 recovery state
id: preflight
timeout-minutes: 3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.preflight-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${PREDECESSOR_IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${inspect}")"
[[ "${generation}" =~ ^(0|[1-9][0-9]*)$ ]]
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
echo "selector_generation=${generation}" >> "${GITHUB_OUTPUT}"
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
[[ "${current_digest}" =~ ^sha256:[a-f0-9]{64}$ ]]
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
jq -e '.draining | type == "boolean"' <<< "${runtime}" >/dev/null
{
echo "runtime_available=true"
echo "current_digest=${current_digest}"
echo "draining=$(jq -r '.draining' <<< "${runtime}")"
echo "ready=$(jq -r '.status.runtime.ready == true' <<< "${status}")"
} >> "${GITHUB_OUTPUT}"
else
echo "runtime_available=false" >> "${GITHUB_OUTPUT}"
fi
- name: Classify both exact recovery end states
id: recovery-plan
timeout-minutes: 10
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
predecessor_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
target_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${predecessor_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${PREDECESSOR_IMAGE_DIGEST}"
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].image' <<< "${cells}")" = \
"${target_image}"
target_plan="${RUNNER_TEMP}/relay-c4-image-target.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false -lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${target_plan}"
target_result="$(terraform -chdir=infra/terraform show -json "${target_plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${target_image}" \
--rollback-image "${predecessor_image}")"
target_state="$(jq -r '[.changes,.changeKind] | join(":")' <<< "${target_result}")"
case "${target_state}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${predecessor_image}" '.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-recovery.tfvars.json"
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-recovery.tfvars.json" \
-var manage_artifact_dns=false -lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${predecessor_image}" \
--rollback-image "${target_image}")"
changes="$(jq -r '.changes' <<< "${result}")"
change_kind="$(jq -r '.changeKind' <<< "${result}")"
case "${changes}:${change_kind}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
mig="$(gcloud compute instance-groups managed describe "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
mig_stable="$(jq -r '.status.isStable == true and .status.versionTarget.isReached == true' \
<<< "${mig}")"
instances="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
instance_stable="$(jq -r 'length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"' <<< "${instances}")"
action=rollback-predecessor
recovery_digest="${PREDECESSOR_IMAGE_DIGEST}"
if test "${{ steps.preflight.outputs.runtime_available }}" = true && \
test "${{ steps.preflight.outputs.ready }}" = true && \
test "${{ steps.preflight.outputs.draining }}" = false && \
test "${mig_stable}" = true && test "${instance_stable}" = true; then
if test "${{ steps.preflight.outputs.current_digest }}" = "${TARGET_IMAGE_DIGEST}" && \
test "${target_state}" = 0:none; then
action=verify-target
recovery_digest="${TARGET_IMAGE_DIGEST}"
elif test "${{ steps.preflight.outputs.current_digest }}" = \
"${PREDECESSOR_IMAGE_DIGEST}" && test "${changes}:${change_kind}" = 0:none; then
action=verify-predecessor
fi
fi
{
echo "changes=${changes}"
echo "change_kind=${change_kind}"
echo "action=${action}"
echo "recovery_digest=${recovery_digest}"
} >> "${GITHUB_OUTPUT}"
- id: fence-auth
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 before predecessor recovery
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 6
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
current_digest="${{ steps.preflight.outputs.current_digest }}"
if test -n "${current_digest}" && [[ ",${expected_digests}," != *",${current_digest},"* ]]; then
expected_digests="${expected_digests},${current_digest}"
fi
if curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/drain" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1,"graceMs":0}' \
>/dev/null; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent \
--expected-image-digests "${expected_digests}" \
--timeout-ms 240000
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 240000
fi
- name: Apply and stabilize the saved predecessor plan
id: apply
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
CHANGES: ${{ steps.recovery-plan.outputs.changes }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
if test "${CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Restart only when the plan did not replace C4
id: restore
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
APPLY_STABLE_AT_MS: ${{ steps.apply.outputs.stable_at_ms }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
if [[ "${CHANGE_KIND}" =~ ^(replacement|replacement-with-obsolete-template|manager-convergence)$ ]]; then
echo "stable_at_ms=${APPLY_STABLE_AT_MS}" >> "${GITHUB_OUTPUT}"
exit 0
fi
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Require an empty selected-image recovery readback
timeout-minutes: 8
env:
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
recovery_image="${image_repository}@${RECOVERY_DIGEST}"
other_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
if test "${RECOVERY_DIGEST}" = "${TARGET_IMAGE_DIGEST}"; then
other_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
fi
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars -var manage_artifact_dns=false \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" --arg image "${recovery_image}" \
'.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-readback.tfvars.json"
readback="${RUNNER_TEMP}/relay-c4-image-recovery-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-readback.tfvars.json" \
-var manage_artifact_dns=false -lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${readback}"
readback_result="$(terraform -chdir=infra/terraform show -json "${readback}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${recovery_image}" \
--rollback-image "${other_image}")"
test "$(jq -r '.changes' <<< "${readback_result}")" = 0
- id: verify-auth
if: ${{ always() }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the recovered image and unchanged isolation
if: ${{ always() && steps.verify-auth.outcome == 'success' }}
timeout-minutes: 8
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.verify-auth.outputs.id_token }}
SELECTOR_GENERATION: ${{ steps.preflight.outputs.selector_generation }}
STABLE_AT_MS: ${{ steps.restore.outputs.stable_at_ms }}
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${RECOVERY_DIGEST}" \
--timeout-ms 240000
stable_at_ms="${STABLE_AT_MS:-0}"
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"; then break; fi
sleep 5
done
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${RECOVERY_DIGEST}")"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
@@ -0,0 +1,60 @@
name: Requeue Relay Staging C4 Recovery
on:
workflow_run:
workflows: [Recover Relay Staging C4 Image]
types: [completed]
permissions:
actions: write
contents: read
concurrency:
group: relay-staging-c4-recovery-requeue
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
requeue:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion == 'cancelled') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- name: Requeue only a cancelled protected recovery job
working-directory: .
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
shell: bash
run: |
set -euo pipefail
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "recover" and
.conclusion == "cancelled" and .started_at == null)] | length' <<< "${jobs}")"
if test "${count}" = 0; then exit 0; fi
test "${count}" = 1
runs="$(gh api \
"repos/${GITHUB_REPOSITORY}/actions/workflows/cloud-recover-relay-staging-c4-image.yml/runs?branch=main&per_page=100")"
active=0
while IFS= read -r run_id; do
active_jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${run_id}/jobs?filter=latest")"
if jq -se '
([.[].jobs[] | select(.name == "gate" and .conclusion == "success")] | length) == 1 and
([.[].jobs[] | select(.name == "recover" and .status != "completed")] | length) == 1
' <<< "${active_jobs}" >/dev/null; then
active=1
break
fi
done < <(jq -r --arg source "${SOURCE_RUN_ID}" \
'.workflow_runs[] | select((.id | tostring) != $source and
.status != "completed") | .id' <<< "${runs}")
if test "${active}" != 0; then exit 0; fi
gh workflow run cloud-recover-relay-staging-c4-image.yml \
--repo "${GITHUB_REPOSITORY}" --ref main \
-f confirmation=RECOVER_STAGING_ASIA_C4_IMAGE
+120
View File
@@ -0,0 +1,120 @@
name: Cloud Verify
on:
pull_request:
paths:
- cloud/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
push:
branches: [main]
paths:
- cloud/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
permissions:
contents: read
concurrency:
group: cloud-verify-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
working-directory: cloud
jobs:
security:
name: Secret scan
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Scan Cloud history with Gitleaks
run: >-
docker run --rm
--volume "${GITHUB_WORKSPACE}:/repo:ro"
zricethezav/gitleaks@sha256:cdbb7c955abce02001a9f6c9f602fb195b7fadc1e812065883f695d1eeaba854
git /repo --config /repo/cloud/.gitleaks.toml
--log-opts="--all -- cloud :(glob).github/workflows/cloud-*.yml .github/actions/cloud-sql-rollout-lease"
- name: Scan the single-commit Cloud snapshot with TruffleHog
run: >-
docker run --rm
--volume "${GITHUB_WORKSPACE}:/repo:ro"
trufflesecurity/trufflehog@sha256:5dc064868ba7933601b5cbaea6954954d524ddd5dc6222a9667acea70068bf7d
filesystem /repo --no-verification --fail
--include-paths=/repo/cloud/.trufflehog-include-paths.txt
--exclude-paths=/repo/cloud/.trufflehog-exclude-paths.txt
# Compiles the workspace. No Postgres service: nothing here reaches a
# database, and the service container costs ~13s of startup.
build:
runs-on: blacksmith-4vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- run: pnpm build
- run: pnpm typecheck
# Runs in parallel with build. `pnpm test` compiles the one workspace
# package it needs through the relay pretest hook, so it does not depend on
# `pnpm build` having run.
test:
runs-on: blacksmith-4vcpu-ubuntu-2204
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_DB: orca_relay_test
POSTGRES_PASSWORD: relay_test
POSTGRES_USER: relay_test
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U relay_test -d orca_relay_test"
--health-interval 5s
--health-timeout 5s
--health-retries 10
env:
ORCA_RELAY_TEST_POSTGRES_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- run: pnpm test
# Fork pull requests reach this job, so it never configures a backend, never plans, and never
# holds a credential. Only the relay root ships here; foundation and apps stay private.
terraform:
runs-on: blacksmith-2vcpu-ubuntu-2204
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
- run: terraform -chdir=infra/terraform fmt -check -recursive
- run: terraform -chdir=infra/terraform init -backend=false -input=false
- run: terraform -chdir=infra/terraform validate
+2 -1
View File
@@ -3,5 +3,6 @@
"singleQuote": true,
"semi": false,
"printWidth": 100,
"trailingComma": "none"
"trailingComma": "none",
"ignorePatterns": ["cloud/**", ".github/actions/cloud-sql-rollout-lease/**"]
}
+8 -1
View File
@@ -174,5 +174,12 @@
}
}
],
"ignorePatterns": ["**/node_modules", "**/dist", "**/out", "tests/e2e/.cross-version-checkouts"]
"ignorePatterns": [
"**/node_modules",
"**/dist",
"**/out",
"cloud/**",
".github/actions/cloud-sql-rollout-lease/**",
"tests/e2e/.cross-version-checkouts"
]
}
+3
View File
@@ -252,6 +252,9 @@ Pair with your desktop app to monitor and steer your agents from your phone.
Want to contribute or run locally? See our [CONTRIBUTING.md](.github/CONTRIBUTING.md) guide.
The relay that pairs the mobile app with a desktop host is also in this repository under
[`cloud/`](cloud/README.md), with a separate pnpm workspace and setup guide.
<a href="https://github.com/stablyai/orca/graphs/contributors">
<img src="https://contrib.rocks/image?repo=stablyai/orca" alt="Orca contributors" />
</a>
+15
View File
@@ -0,0 +1,15 @@
.git/
.github/
.local/
.terraform/
node_modules/
dist/
coverage/
.env
.env.local
.env.local.generated
*.log
*.tfplan
*.tfstate
*.tfstate.*
+10
View File
@@ -0,0 +1,10 @@
root = true
[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
+24
View File
@@ -0,0 +1,24 @@
node_modules/
dist/
coverage/
.turbo/
.local/
.env
.env.local
.env.local.generated
*.log
# Terraform/OpenTofu local state and plans must stay out of git.
**/.terraform/
*.tfstate
*.tfstate.*
*.tfplan
crash.log
override.tf
override.tf.json
*_override.tf
*_override.tf.json
# Local dev signing key + SQLite live under data/ — never commit (contains a
# private key and dev PII). Prod supplies the key via ORCA_CLOUD_SIGNING_KEY_PEM.
data/
+15
View File
@@ -0,0 +1,15 @@
[extend]
useDefault = true
[[allowlists]]
description = "Explicit Relay test signing key"
regexTarget = "secret"
regexes = ['''^test-assignment-key-with-at-least-32-bytes$''']
# The Cloud SQL rollout lease records `owner/repo/run_id` as the holder of a lease. The action's
# unit tests build fixture holders from that shape, which the generic key rule reads as a secret.
[[allowlists]]
description = "Cloud SQL rollout lease holder keys in the action's unit tests"
regexTarget = "secret"
paths = ['''\.github/actions/cloud-sql-rollout-lease/[a-z-]+\.test\.mjs$''']
regexes = ['''^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+/[0-9]+$''']
+2
View File
@@ -0,0 +1,2 @@
24
+3
View File
@@ -0,0 +1,3 @@
engine-strict=false
package-manager-strict=true
+1
View File
@@ -0,0 +1 @@
(^|/)cloud/apps/relay/src/postgres-idle-client-error\.test\.ts$
+2
View File
@@ -0,0 +1,2 @@
(^|/)cloud/
(^|/)\.github/workflows/cloud-[^/]+\.yml$
+92
View File
@@ -0,0 +1,92 @@
# Orca Relay
The relay that connects the Orca mobile app to a desktop host. Phones and
desktops never talk to each other directly: each opens an outbound WebSocket
to a relay cell, the relay pairs the two sessions, and it splices frames
between them. A director assigns hosts to cells and coordinates migrations;
cells carry the user connections.
This directory is an independent pnpm workspace inside the Orca monorepo. Run
its commands from `cloud/`, not the repository root. The source is covered by
the repository's root [MIT license](../LICENSE).
## Packages
- `packages/relay-contract`: the wire contract shared by the relay, the
desktop app, and the mobile app (frame shapes, close codes, admission budgets,
splice state machine).
- `apps/relay`: the relay server. The same image runs as a director or a cell
depending on `ORCA_RELAY_ROLE`.
- `apps/relay-fence-broker`: a private, IAM-only service that owns the durable
mutation lease, the Terraform checkout, and the narrow Compute mutation used
when a registered target is superseded. The workflow that calls it holds read
and invoke rights only, never those mutation permissions.
- `apps/relay-ops`: the relay operations console and the incident monitor
behind `pnpm ops:relay`, `pnpm incident:relay`, and
`pnpm incident:relay-preflight`.
## Infrastructure and operations
- `infra/terraform`: the relay Terraform root. It owns the cells, the director,
the shared Cloud SQL instance, DNS, observability, and every GitHub Workload
Identity provider the relay workflows authenticate through. `backend/` holds
the per-environment backend configuration and `environments/` the tfvars.
Drive it through `pnpm infra:init`, `pnpm infra:plan`, and `pnpm infra:apply`.
- `dev/scripts`: the deploy, capacity, admission, rehome, monitoring, and load
scripts the workflows call, plus the contract tests that pin each workflow
and Terraform surface. Run them with `pnpm test`.
- `dev/contracts` and `dev/fixtures`: the checked-in data those contract tests
read, including the Terraform root partition.
- `docs/`: the relay runbooks, capacity-testing guide, incident-monitor
reference, and the workflow variable reference in `docs/relay-workflows.md`.
## Workflows
The 24 `.github/workflows/cloud-*.yml` workflows are the relay's deploy and
operate surface: publish and deploy the director, roll GCE cell capacity,
operate Asia admission and regional rehoming, prove staging capacity, monitor
production, and power staging up and down. `.github/actions/cloud-sql-rollout-lease`
is the compare-and-swap lease that serializes every rollout against the shared
Cloud SQL instance.
Every one of them is inert. Each top-level job is gated on
`vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true'`, a repository variable that is
unset here, so the two scheduled triggers and every manual dispatch skip
without running a step. Only the repository owner, holding the GCP identities
these workflows authenticate as, can turn them on.
`Cloud Verify` is not gated. It builds, typechecks, lints, tests, secret-scans,
and validates the relay Terraform on every change under `cloud/`, and it runs
on fork pull requests, so it configures no backend and holds no credential.
## What is not here
The `terraform-foundation` and `terraform-apps` roots and the API and auth
services live in the private `stablyai/orca-cloud` repository. Scripts and
tests that spanned both trees were narrowed to the relay side rather than
carrying a dangling reference.
## Local development
```sh
cd cloud
pnpm install
pnpm build
pnpm test
```
`pnpm test` runs the SQLite-backed suites. Tests that need PostgreSQL run only
when `ORCA_RELAY_TEST_POSTGRES_URL` points at a disposable PostgreSQL 16 or 17
database, for example:
```sh
docker run --rm -d --name orca-relay-pg -e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=orca_relay_test -p 55440:5432 postgres:16-alpine
ORCA_RELAY_TEST_POSTGRES_URL=postgres://postgres@127.0.0.1:55440/orca_relay_test \
pnpm --filter @orca-cloud/relay test
docker rm -f orca-relay-pg
```
Configuration is read from environment variables validated in
`apps/relay/src/config.ts`. `ORCA_RELAY_ASSIGNMENT_SIGNING_KEY` (at least 32
bytes) is the only required value; everything else has a local default.
+33
View File
@@ -0,0 +1,33 @@
FROM node:24-bookworm-slim AS build
WORKDIR /workspace
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json ./
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
RUN pnpm install --frozen-lockfile
COPY apps/relay-fence-broker apps/relay-fence-broker
RUN pnpm --filter @orca-cloud/relay-fence-broker build
FROM hashicorp/terraform:1.15.8 AS terraform
FROM gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
ARG ORCA_RELAY_FENCE_IMAGE_COMMIT
RUN test "$(printf '%s' "${ORCA_RELAY_FENCE_IMAGE_COMMIT}" | grep -E '^[a-f0-9]{40}$')"
ENV NODE_ENV=production
ENV PORT=8080
ENV ORCA_RELAY_FENCE_IMAGE_COMMIT=${ORCA_RELAY_FENCE_IMAGE_COMMIT}
ENV IAC_TOOL=terraform
WORKDIR /workspace
COPY --from=build /usr/local /usr/local
COPY --from=terraform /bin/terraform /usr/local/bin/terraform
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY apps/relay-fence-broker/package.json apps/relay-fence-broker/package.json
COPY --from=build /workspace/apps/relay-fence-broker/dist apps/relay-fence-broker/dist
COPY dev/scripts dev/scripts
COPY infra/terraform infra/terraform
RUN corepack enable \
&& pnpm install --prod --frozen-lockfile --filter @orca-cloud/relay-fence-broker... \
&& useradd --create-home --uid 10001 broker \
&& chown -R broker:broker /workspace
USER broker
EXPOSE 8080
CMD ["node", "apps/relay-fence-broker/dist/index.js"]
@@ -0,0 +1,27 @@
{
"name": "@orca-cloud/relay-fence-broker",
"private": true,
"version": "0.0.0",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "pnpm clean && tsc -p tsconfig.build.json",
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
"dev": "tsx watch src/index.ts",
"lint": "tsc -p tsconfig.json --noEmit",
"start": "node dist/index.js",
"test": "vitest run",
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"@hono/node-server": "^1.19.14",
"hono": "^4.12.27",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^24.10.0",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vitest": "^4.0.8"
}
}
@@ -0,0 +1,316 @@
import { describe, expect, it, vi } from 'vitest'
import { createApp } from './app.js'
import type { RelayFenceBrokerConfig } from './config.js'
import type {
GoogleStorageMutationLease,
MutationLease
} from './mutation-lease.js'
const commit = 'a'.repeat(40)
const config: RelayFenceBrokerConfig = {
port: 8080,
project: 'onorca-cloud',
stateBucket: 'onorca-cloud-terraform-state',
leaseObject: 'terraform/state/relay-fence-broker/production.lock',
directorOrigin: 'https://relay.onorca.dev',
adminAudience: 'https://relay.onorca.dev/v1/admin/drain',
requesterServiceAccount: 'requester@example.com',
runtimeServiceAccount: 'runtime@example.com',
sourceCellId: 'production-gce-c3',
failedTargetCellId: 'production-gce-c11',
replacementTargetCellId: 'production-gce-c12',
imageCommit: commit,
terraformDir: 'infra/terraform',
unobservedConnectionBound: 10,
connectionCeiling: 600
}
function request(fenceCommit = commit): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit,
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function recoveryRequest(): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: commit,
completedFenceRecovery: {
attemptId: '11111111-1111-4111-8111-111111111111',
fenceCommit: 'b'.repeat(40),
gceOperation: 'operation-1',
terraformStateSerial: 61,
planObjectGeneration: '123',
terraformStateObjectGeneration: '456',
terraformStateObjectSha256: 'c'.repeat(64)
},
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
},
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function leaseTakeoverRequest(): Request {
return new Request('http://broker/v1/supersede-target', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: commit,
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
},
confirmation: 'SUPERSEDE_TARGET'
})
})
}
function sourceFenceRequest(
overrides: Record<string, unknown> = {}
): Request {
return new Request('http://broker/v1/fence-source', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
v: 1,
operationId: 'c3-final-fence',
fenceCommit: commit,
targetCellIds: ['production-gce-c7', 'production-gce-c12'],
confirmation: 'FENCE_SOURCE',
...overrides
})
})
}
describe('relay fence broker', () => {
it('runs only after acquiring the durable lease', async () => {
const events: string[] = []
const lease = {
acquire: vi.fn(async () => {
events.push('acquire')
return {
generation: '7',
record: {}
} as MutationLease
}),
release: vi.fn(async () => {
events.push('release')
})
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
supersede: async () => {
events.push('supersede')
}
})
const response = await app.request(request())
expect(response.status).toBe(200)
expect(events).toEqual(['acquire', 'supersede', 'release'])
})
it('rejects a commit not bound to the immutable image', async () => {
const lease = {
acquire: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const response = await app.request(request('b'.repeat(40)))
expect(response.status).toBe(409)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('retains the lease when supersession fails', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '7', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
supersede: async () => {
throw new Error('stopped safely')
}
})
const response = await app.request(request())
expect(response.status).toBe(500)
expect(lease.release).not.toHaveBeenCalled()
})
it('passes exact completed-attempt and live-lease recovery pins', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '8', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const supersede = vi.fn(async () => {})
const app = createApp(config, { lease, supersede })
const response = await app.request(recoveryRequest())
expect(response.status).toBe(200)
expect(lease.acquire).toHaveBeenCalledWith(
'c11-to-c12-forward',
expect.objectContaining({
completedFenceRecovery: expect.objectContaining({
terraformStateSerial: 61
})
}),
{
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
)
expect(supersede).toHaveBeenCalledWith(
config,
expect.objectContaining({
attemptId: '11111111-1111-4111-8111-111111111111'
})
)
})
it('conditionally resumes the exact live supersession lease', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '8', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const supersede = vi.fn(async () => {})
const app = createApp(config, { lease, supersede })
const response = await app.request(leaseTakeoverRequest())
expect(response.status).toBe(200)
expect(lease.acquire).toHaveBeenCalledWith(
'c11-to-c12-forward',
expect.objectContaining({
expectedLease: {
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
}),
{
generation: '7',
operationId: 'c11-to-c12-forward',
requestDigest: 'd'.repeat(64)
}
)
expect(supersede).toHaveBeenCalledWith(config, undefined)
})
it('rejects a live supersession lease for another operation', async () => {
const lease = {
acquire: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const request = leaseTakeoverRequest()
const body = (await request.json()) as {
expectedLease: { operationId: string }
}
body.expectedLease.operationId = 'another-operation'
const response = await app.request(
new Request(request.url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(body)
})
)
expect(response.status).toBe(400)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('fences the configured source only after acquiring the durable lease', async () => {
const events: string[] = []
const expectedLease = {
generation: '6',
operationId: 'c3-final-fence',
requestDigest: 'd'.repeat(64)
}
const lease = {
acquire: vi.fn(async () => {
events.push('acquire')
return { generation: '7', record: {} } as MutationLease
}),
release: vi.fn(async () => {
events.push('release')
})
} as unknown as GoogleStorageMutationLease
const fenceSource = vi.fn(async () => {
events.push('fence')
})
const app = createApp(config, { lease, fenceSource })
const response = await app.request(sourceFenceRequest({ expectedLease }))
expect(response.status).toBe(200)
expect(events).toEqual(['acquire', 'fence', 'release'])
expect(lease.acquire).toHaveBeenCalledWith(
'c3-final-fence',
expect.objectContaining({ targetCellIds: expect.any(Array) }),
expectedLease
)
expect(fenceSource).toHaveBeenCalledWith(config, [
'production-gce-c7',
'production-gce-c12'
])
})
it('rejects duplicate targets and the configured source', async () => {
const lease = { acquire: vi.fn() } as unknown as GoogleStorageMutationLease
const app = createApp(config, { lease })
const duplicate = await app.request(
sourceFenceRequest({
targetCellIds: ['production-gce-c7', 'production-gce-c7']
})
)
const source = await app.request(
sourceFenceRequest({ targetCellIds: [config.sourceCellId] })
)
expect(duplicate.status).toBe(400)
expect(source.status).toBe(400)
expect(lease.acquire).not.toHaveBeenCalled()
})
it('retains the source-fence lease after a controlled failure', async () => {
const lease = {
acquire: vi.fn(async () => ({ generation: '7', record: {} })),
release: vi.fn()
} as unknown as GoogleStorageMutationLease
const app = createApp(config, {
lease,
fenceSource: async () => {
throw new Error('stopped safely')
}
})
const response = await app.request(sourceFenceRequest())
expect(response.status).toBe(500)
expect(lease.release).not.toHaveBeenCalled()
})
})
+162
View File
@@ -0,0 +1,162 @@
import { Hono } from 'hono'
import { z } from 'zod'
import type { RelayFenceBrokerConfig } from './config.js'
import {
GoogleStorageMutationLease,
MutationLeaseConflict
} from './mutation-lease.js'
import {
runSourceFence,
runTargetSupersession
} from './fence-operation-runner.js'
const expectedLeaseSchema = z
.object({
generation: z.string().regex(/^[1-9][0-9]{0,30}$/),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
requestDigest: z.string().regex(/^[a-f0-9]{64}$/)
})
.strict()
const supersessionRequestSchema = z.object({
v: z.literal(1),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
completedFenceRecovery: z
.object({
attemptId: z.string().uuid(),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
gceOperation: z.string().min(1).max(256),
terraformStateSerial: z.number().int().nonnegative().safe(),
planObjectGeneration: z.string().regex(/^[1-9][0-9]{0,30}$/),
terraformStateObjectGeneration: z.string().regex(/^[1-9][0-9]{0,30}$/),
terraformStateObjectSha256: z.string().regex(/^[a-f0-9]{64}$/)
})
.strict()
.optional(),
expectedLease: expectedLeaseSchema.optional(),
confirmation: z.literal('SUPERSEDE_TARGET')
})
.strict()
.refine(
(value) =>
(!value.completedFenceRecovery || Boolean(value.expectedLease)) &&
(!value.expectedLease || value.expectedLease.operationId === value.operationId)
)
const cellIdSchema = z.string().regex(/^[a-z][a-z0-9-]{0,127}$/)
const sourceFenceRequestSchema = z
.object({
v: z.literal(1),
operationId: z.string().regex(/^[A-Za-z0-9_-]{8,128}$/),
fenceCommit: z.string().regex(/^[a-f0-9]{40}$/),
targetCellIds: z.array(cellIdSchema).min(1).max(16),
expectedLease: expectedLeaseSchema.optional(),
confirmation: z.literal('FENCE_SOURCE')
})
.strict()
.refine(
(value) =>
new Set(value.targetCellIds).size === value.targetCellIds.length &&
(!value.expectedLease || value.expectedLease.operationId === value.operationId)
)
type AppDependencies = {
lease?: GoogleStorageMutationLease
supersede?: (
config: RelayFenceBrokerConfig,
recovery?: z.infer<typeof supersessionRequestSchema>['completedFenceRecovery']
) => Promise<void>
fenceSource?: (
config: RelayFenceBrokerConfig,
targetCellIds: string[]
) => Promise<void>
}
export function createApp(
config: RelayFenceBrokerConfig,
dependencies: AppDependencies = {}
): Hono {
const app = new Hono()
const lease =
dependencies.lease ??
new GoogleStorageMutationLease(
config.stateBucket,
config.leaseObject,
config.imageCommit
)
const supersede = dependencies.supersede ?? runTargetSupersession
const fenceSource = dependencies.fenceSource ?? runSourceFence
app.get('/healthz', (context) =>
context.json({ ok: true, fenceCommit: config.imageCommit })
)
app.post('/v1/supersede-target', async (context) => {
const parsed = supersessionRequestSchema.safeParse(
await context.req.json().catch(() => null)
)
if (!parsed.success) return context.json({ error: 'invalid_request' }, 400)
if (parsed.data.fenceCommit !== config.imageCommit) {
return context.json({ error: 'fence_commit_mismatch' }, 409)
}
let acquired
try {
acquired = await lease.acquire(
parsed.data.operationId,
parsed.data,
parsed.data.expectedLease
)
} catch (error) {
if (error instanceof MutationLeaseConflict) {
return context.json({ error: 'mutation_lease_conflict' }, 409)
}
throw error
}
await supersede(config, parsed.data.completedFenceRecovery)
await lease.release(acquired)
return context.json({
ok: true,
operationId: parsed.data.operationId,
fenceCommit: config.imageCommit
})
})
app.post('/v1/fence-source', async (context) => {
const parsed = sourceFenceRequestSchema.safeParse(
await context.req.json().catch(() => null)
)
if (
!parsed.success ||
parsed.data.targetCellIds.includes(config.sourceCellId)
) {
return context.json({ error: 'invalid_request' }, 400)
}
if (parsed.data.fenceCommit !== config.imageCommit) {
return context.json({ error: 'fence_commit_mismatch' }, 409)
}
let acquired
try {
acquired = await lease.acquire(
parsed.data.operationId,
parsed.data,
parsed.data.expectedLease
)
} catch (error) {
if (error instanceof MutationLeaseConflict) {
return context.json({ error: 'mutation_lease_conflict' }, 409)
}
throw error
}
await fenceSource(config, parsed.data.targetCellIds)
await lease.release(acquired)
return context.json({
ok: true,
operationId: parsed.data.operationId,
fenceCommit: config.imageCommit
})
})
app.onError((error, context) => {
console.error(error)
return context.json({ error: 'broker_operation_failed' }, 500)
})
return app
}
@@ -0,0 +1,92 @@
import { z } from 'zod'
const environmentSchema = z.object({
PORT: z.coerce.number().int().positive().max(65_535).default(8080),
ORCA_RELAY_FENCE_PROJECT: z.string().regex(/^[a-z][a-z0-9-]{4,29}$/),
ORCA_RELAY_FENCE_STATE_BUCKET: z.string().min(3).max(222),
ORCA_RELAY_FENCE_LEASE_OBJECT: z.string().min(1).max(512),
ORCA_RELAY_FENCE_DIRECTOR_ORIGIN: z.string().url(),
ORCA_RELAY_FENCE_ADMIN_AUDIENCE: z.string().url(),
ORCA_RELAY_FENCE_REQUESTER_SERVICE_ACCOUNT: z.string().email(),
ORCA_RELAY_FENCE_RUNTIME_SERVICE_ACCOUNT: z.string().email(),
ORCA_RELAY_FENCE_SOURCE_CELL_ID: z.string().regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID: z
.string()
.regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID: z
.string()
.regex(/^[a-z][a-z0-9-]{0,127}$/),
ORCA_RELAY_FENCE_IMAGE_COMMIT: z.string().regex(/^[a-f0-9]{40}$/),
// Resolved against the broker's working directory, which the image sets to the copied tree root.
ORCA_RELAY_FENCE_TERRAFORM_DIR: z.string().min(1).default('infra/terraform'),
ORCA_RELAY_FENCE_UNOBSERVED_CONNECTION_BOUND: z.coerce
.number()
.int()
.nonnegative()
.max(499),
ORCA_RELAY_FENCE_CONNECTION_CEILING: z.coerce
.number()
.int()
.positive()
.max(600)
.default(600)
})
export type RelayFenceBrokerConfig = {
port: number
project: string
stateBucket: string
leaseObject: string
directorOrigin: string
adminAudience: string
requesterServiceAccount: string
runtimeServiceAccount: string
sourceCellId: string
failedTargetCellId: string
replacementTargetCellId: string
imageCommit: string
terraformDir: string
unobservedConnectionBound: number
connectionCeiling: number
}
export function loadConfig(
environment: NodeJS.ProcessEnv = process.env
): RelayFenceBrokerConfig {
const parsed = environmentSchema.parse(environment)
const director = new URL(parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN)
const audience = new URL(parsed.ORCA_RELAY_FENCE_ADMIN_AUDIENCE)
if (
director.origin !== parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN ||
audience.origin !== director.origin ||
audience.pathname !== '/v1/admin/drain' ||
audience.search ||
audience.hash
) {
throw new Error('broker director and admin audience must use the canonical drain origin')
}
const cells = new Set([
parsed.ORCA_RELAY_FENCE_SOURCE_CELL_ID,
parsed.ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID,
parsed.ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID
])
if (cells.size !== 3) throw new Error('broker cells must be distinct')
return {
port: parsed.PORT,
project: parsed.ORCA_RELAY_FENCE_PROJECT,
stateBucket: parsed.ORCA_RELAY_FENCE_STATE_BUCKET,
leaseObject: parsed.ORCA_RELAY_FENCE_LEASE_OBJECT,
directorOrigin: parsed.ORCA_RELAY_FENCE_DIRECTOR_ORIGIN,
adminAudience: parsed.ORCA_RELAY_FENCE_ADMIN_AUDIENCE,
requesterServiceAccount: parsed.ORCA_RELAY_FENCE_REQUESTER_SERVICE_ACCOUNT,
runtimeServiceAccount: parsed.ORCA_RELAY_FENCE_RUNTIME_SERVICE_ACCOUNT,
sourceCellId: parsed.ORCA_RELAY_FENCE_SOURCE_CELL_ID,
failedTargetCellId: parsed.ORCA_RELAY_FENCE_FAILED_TARGET_CELL_ID,
replacementTargetCellId: parsed.ORCA_RELAY_FENCE_REPLACEMENT_TARGET_CELL_ID,
imageCommit: parsed.ORCA_RELAY_FENCE_IMAGE_COMMIT,
terraformDir: parsed.ORCA_RELAY_FENCE_TERRAFORM_DIR,
unobservedConnectionBound:
parsed.ORCA_RELAY_FENCE_UNOBSERVED_CONNECTION_BOUND,
connectionCeiling: parsed.ORCA_RELAY_FENCE_CONNECTION_CEILING
}
}
@@ -0,0 +1,57 @@
import { describe, expect, it } from 'vitest'
import type { RelayFenceBrokerConfig } from './config.js'
import {
fenceChildEnvironment,
sourceFenceArguments
} from './fence-operation-runner.js'
const config = {
project: 'onorca-cloud',
directorOrigin: 'https://relay.onorca.dev',
adminAudience: 'https://relay.onorca.dev/v1/admin/drain',
sourceCellId: 'production-gce-c3',
runtimeServiceAccount: 'runtime@example.com',
imageCommit: 'a'.repeat(40),
terraformDir: 'infra/terraform',
unobservedConnectionBound: 10,
connectionCeiling: 600
} as RelayFenceBrokerConfig
describe('fence operation runner', () => {
it('passes distinct read and mutation identity tokens', () => {
expect(
fenceChildEnvironment(
config,
'read.token.value',
'mutation.token.value',
{ PRESERVED: 'yes' }
)
).toMatchObject({
PRESERVED: 'yes',
IAC_TOOL: 'terraform',
ORCA_RELAY_ADMIN_ID_TOKEN: 'read.token.value',
ORCA_RELAY_FENCE_MUTATION_ID_TOKEN: 'mutation.token.value',
ORCA_RELAY_FENCE_IMAGE_COMMIT: 'a'.repeat(40)
})
})
it('binds a source fence to deterministic targets and the production var file', () => {
const args = sourceFenceArguments(config, '/tmp/topology.json', [
'production-gce-c12',
'production-gce-c7'
])
expect(args).toEqual(
expect.arrayContaining([
'--source-cell-id',
'production-gce-c3',
'--target-cell-ids',
'production-gce-c12,production-gce-c7',
'--terraform-var-file',
'environments/production.tfvars',
'--mode',
'fence-source'
])
)
})
})
@@ -0,0 +1,185 @@
import { execFile } from 'node:child_process'
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { promisify } from 'node:util'
import type { RelayFenceBrokerConfig } from './config.js'
import {
metadataIdentityToken,
metadataServiceAccountEmail
} from './google-metadata.js'
const exec = promisify(execFile)
const MAX_OUTPUT_BYTES = 10 * 1024 * 1024
type CompletedFenceRecovery = {
attemptId: string
fenceCommit: string
gceOperation: string
terraformStateSerial: number
planObjectGeneration: string
terraformStateObjectGeneration: string
terraformStateObjectSha256: string
}
async function run(file: string, args: string[], environment?: NodeJS.ProcessEnv) {
return await exec(file, args, {
env: environment,
maxBuffer: MAX_OUTPUT_BYTES
})
}
export function fenceChildEnvironment(
config: RelayFenceBrokerConfig,
readToken: string,
mutationToken: string,
environment: NodeJS.ProcessEnv = process.env
): NodeJS.ProcessEnv {
return {
...environment,
IAC_TOOL: 'terraform',
ORCA_RELAY_ADMIN_ID_TOKEN: readToken,
ORCA_RELAY_FENCE_MUTATION_ID_TOKEN: mutationToken,
ORCA_RELAY_FENCE_IMAGE_COMMIT: config.imageCommit
}
}
function relayFenceArguments(
config: RelayFenceBrokerConfig,
topologyFile: string,
targetCellIds: string[]
): string[] {
return [
'dev/scripts/deploy-relay-gce-multi-target.mjs',
'--project',
config.project,
'--director-origin',
config.directorOrigin,
'--admin-audience',
config.adminAudience,
'--topology-file',
topologyFile,
'--source-cell-id',
config.sourceCellId,
'--target-cell-ids',
[...targetCellIds].sort().join(','),
'--unobserved-connection-bound',
String(config.unobservedConnectionBound),
'--runtime-service-account',
config.runtimeServiceAccount,
'--environment',
'production',
'--fence-commit',
config.imageCommit,
'--terraform-dir',
config.terraformDir,
'--terraform-var-file',
'environments/production.tfvars',
'--connection-ceiling',
String(config.connectionCeiling),
'--minimum-lease-remaining-ms',
'600000'
]
}
async function runRelayFenceCommand(
config: RelayFenceBrokerConfig,
argumentsForTopology: (
topologyFile: string,
brokerServiceAccount: string
) => string[]
): Promise<void> {
const directory = await mkdtemp(join(tmpdir(), 'orca-relay-fence-operation-'))
const topologyFile = join(directory, 'topology.json')
try {
await run('terraform', [
`-chdir=${config.terraformDir}`,
'init',
'-input=false',
'-backend-config=backend/production.hcl'
])
const topology = await run('terraform', [
`-chdir=${config.terraformDir}`,
'output',
'-json',
'relay_gce_cell_deployments'
])
await writeFile(topologyFile, topology.stdout, { mode: 0o600 })
const brokerToken = await metadataIdentityToken(config.adminAudience)
const brokerServiceAccount = await metadataServiceAccountEmail()
const environment = fenceChildEnvironment(
config,
brokerToken,
brokerToken
)
await run(
'node',
argumentsForTopology(topologyFile, brokerServiceAccount),
environment
)
} finally {
await rm(directory, { recursive: true, force: true })
}
}
export function sourceFenceArguments(
config: RelayFenceBrokerConfig,
topologyFile: string,
targetCellIds: string[]
): string[] {
return [
...relayFenceArguments(config, topologyFile, targetCellIds),
'--mode',
'fence-source'
]
}
export async function runSourceFence(
config: RelayFenceBrokerConfig,
targetCellIds: string[]
): Promise<void> {
await runRelayFenceCommand(config, (topologyFile) =>
sourceFenceArguments(config, topologyFile, targetCellIds)
)
}
export async function runTargetSupersession(
config: RelayFenceBrokerConfig,
recovery?: CompletedFenceRecovery
): Promise<void> {
await runRelayFenceCommand(config, (topologyFile, brokerServiceAccount) => {
const targets = [
config.failedTargetCellId,
config.replacementTargetCellId
]
const args = [
...relayFenceArguments(config, topologyFile, targets),
'--failed-target-cell-id',
config.failedTargetCellId,
'--replacement-target-cell-id',
config.replacementTargetCellId,
'--mode',
'supersede-target'
]
if (recovery) {
args.push(
'--completed-fence-attempt-id',
recovery.attemptId,
'--completed-fence-commit',
recovery.fenceCommit,
'--completed-fence-operation',
recovery.gceOperation,
'--completed-fence-state-serial',
String(recovery.terraformStateSerial),
'--completed-fence-plan-generation',
recovery.planObjectGeneration,
'--completed-fence-state-generation',
recovery.terraformStateObjectGeneration,
'--completed-fence-state-sha256',
recovery.terraformStateObjectSha256,
'--fence-broker-service-account',
brokerServiceAccount
)
}
return args
})
}
@@ -0,0 +1,44 @@
const METADATA_ROOT =
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default'
async function metadataText(path: string, fetcher: typeof fetch): Promise<string> {
const response = await fetcher(`${METADATA_ROOT}/${path}`, {
headers: { 'Metadata-Flavor': 'Google' }
})
if (!response.ok) throw new Error(`metadata request failed: ${response.status}`)
return await response.text()
}
export async function metadataAccessToken(fetcher: typeof fetch = fetch): Promise<string> {
const body = JSON.parse(await metadataText('token', fetcher)) as {
access_token?: unknown
}
if (typeof body.access_token !== 'string' || body.access_token.length < 20) {
throw new Error('metadata access token is missing')
}
return body.access_token
}
export async function metadataServiceAccountEmail(
fetcher: typeof fetch = fetch
): Promise<string> {
const email = (await metadataText('email', fetcher)).trim()
if (!/^[^@\s]+@[^@\s]+\.gserviceaccount\.com$/.test(email)) {
throw new Error('metadata service account email is invalid')
}
return email
}
export async function metadataIdentityToken(
audience: string,
fetcher: typeof fetch = fetch
): Promise<string> {
const token = await metadataText(
`identity?audience=${encodeURIComponent(audience)}&format=full`,
fetcher
)
if (!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)) {
throw new Error('metadata identity token is invalid')
}
return token
}
@@ -0,0 +1,10 @@
import { serve } from '@hono/node-server'
import { createApp } from './app.js'
import { loadConfig } from './config.js'
const config = loadConfig()
const app = createApp(config)
serve({ fetch: app.fetch, port: config.port }, (info) => {
console.log(`[relay-fence-broker] listening on port ${info.port}`)
})
@@ -0,0 +1,164 @@
import { describe, expect, it, vi } from 'vitest'
import {
GoogleStorageMutationLease,
MutationLeaseConflict
} from './mutation-lease.js'
function json(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { 'Content-Type': 'application/json' }
})
}
const accessToken = () => json({ access_token: 'a'.repeat(40) })
const request = {
v: 1,
operationId: 'c11-to-c12-forward',
fenceCommit: 'a'.repeat(40),
confirmation: 'SUPERSEDE_TARGET'
}
describe('GoogleStorageMutationLease', () => {
it('creates and conditionally releases a new lease', async () => {
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(new Response(null, { status: 404 }))
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(new Response(null, { status: 204 }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request)
await leaseStore.release(lease)
expect(lease.generation).toBe('7')
expect(fetcher.mock.calls[2]?.[0]).toContain('ifGenerationMatch=0')
expect(fetcher.mock.calls[4]?.[0]).toContain('ifGenerationMatch=7')
})
it('rejects a different request while the durable lease is live', async () => {
const existing = {
operationId: 'other-operation',
requestDigest: 'b'.repeat(64),
imageCommit: 'a'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
await expect(
leaseStore.acquire(request.operationId, request)
).rejects.toBeInstanceOf(MutationLeaseConflict)
expect(fetcher).toHaveBeenCalledTimes(3)
})
it('takes over an expired lease with an exact generation precondition', async () => {
const existing = {
operationId: 'other-operation',
requestDigest: 'b'.repeat(64),
imageCommit: 'a'.repeat(40),
acquiredAt: 500,
expiresAt: 999
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
.mockResolvedValueOnce(json({ generation: '8' }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request)
expect(lease.generation).toBe('8')
expect(fetcher.mock.calls[3]?.[0]).toContain('ifGenerationMatch=7')
})
it('conditionally replaces the exact authorized live lease', async () => {
const existing = {
operationId: request.operationId,
requestDigest: 'b'.repeat(64),
imageCommit: 'b'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
.mockResolvedValueOnce(json({ generation: '8' }))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
const lease = await leaseStore.acquire(request.operationId, request, {
generation: '7',
operationId: existing.operationId,
requestDigest: existing.requestDigest
})
expect(lease.generation).toBe('8')
expect(fetcher.mock.calls[3]?.[0]).toContain('ifGenerationMatch=7')
})
it('rejects a live-lease takeover when any expected field differs', async () => {
const existing = {
operationId: request.operationId,
requestDigest: 'b'.repeat(64),
imageCommit: 'b'.repeat(40),
acquiredAt: 500,
expiresAt: 2_000
}
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(accessToken())
.mockResolvedValueOnce(json({ generation: '7' }))
.mockResolvedValueOnce(json(existing))
const leaseStore = new GoogleStorageMutationLease(
'state-bucket',
'fence/production.lock',
'a'.repeat(40),
fetcher,
() => 1_000
)
await expect(
leaseStore.acquire(request.operationId, request, {
generation: '8',
operationId: existing.operationId,
requestDigest: existing.requestDigest
})
).rejects.toBeInstanceOf(MutationLeaseConflict)
expect(fetcher).toHaveBeenCalledTimes(3)
})
})
@@ -0,0 +1,152 @@
import { createHash } from 'node:crypto'
import { metadataAccessToken } from './google-metadata.js'
const LEASE_TTL_MS = 35 * 60 * 1_000
type LeaseRecord = {
operationId: string
requestDigest: string
imageCommit: string
acquiredAt: number
expiresAt: number
}
type ObjectMetadata = {
generation: string
}
export class MutationLeaseConflict extends Error {}
export type MutationLease = {
generation: string
record: LeaseRecord
}
export type ExpectedMutationLease = {
generation: string
operationId: string
requestDigest: string
}
export class GoogleStorageMutationLease {
constructor(
private readonly bucket: string,
private readonly objectName: string,
private readonly imageCommit: string,
private readonly fetcher: typeof fetch = fetch,
private readonly now: () => number = Date.now
) {}
async acquire(
operationId: string,
request: unknown,
expectedExisting?: ExpectedMutationLease
): Promise<MutationLease> {
const token = await metadataAccessToken(this.fetcher)
const existing = await this.read(token)
const requestDigest = createHash('sha256')
.update(JSON.stringify(request))
.digest('hex')
const exactTakeover =
existing &&
expectedExisting?.generation === existing.metadata.generation &&
expectedExisting.operationId === existing.record.operationId &&
expectedExisting.requestDigest === existing.record.requestDigest
if (
(!existing && expectedExisting) ||
(existing &&
existing.record.requestDigest !== requestDigest &&
existing.record.expiresAt > this.now() &&
!exactTakeover)
) {
throw new MutationLeaseConflict('another relay mutation owns the durable lease')
}
const acquiredAt = this.now()
const record: LeaseRecord = {
operationId,
requestDigest,
imageCommit: this.imageCommit,
acquiredAt,
expiresAt: acquiredAt + LEASE_TTL_MS
}
const generation = existing?.metadata.generation ?? '0'
const response = await this.fetcher(
`${this.uploadUrl()}&ifGenerationMatch=${encodeURIComponent(generation)}`,
{
method: 'POST',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json'
},
body: JSON.stringify(record)
}
)
if (response.status === 412) {
throw new MutationLeaseConflict('relay mutation lease changed concurrently')
}
if (!response.ok) throw new Error(`mutation lease acquisition failed: ${response.status}`)
const metadata = (await response.json()) as Partial<ObjectMetadata>
if (!/^[1-9][0-9]{0,30}$/.test(metadata.generation ?? '')) {
throw new Error('mutation lease has no valid generation')
}
return { generation: metadata.generation!, record }
}
async release(lease: MutationLease): Promise<void> {
const token = await metadataAccessToken(this.fetcher)
const response = await this.fetcher(
`${this.metadataUrl()}?ifGenerationMatch=${encodeURIComponent(lease.generation)}`,
{
method: 'DELETE',
headers: { Authorization: `Bearer ${token}` }
}
)
if (!response.ok && response.status !== 404) {
throw new Error(`mutation lease release failed: ${response.status}`)
}
}
private async read(
token: string
): Promise<{ metadata: ObjectMetadata; record: LeaseRecord } | null> {
const metadataResponse = await this.fetcher(this.metadataUrl(), {
headers: { Authorization: `Bearer ${token}` }
})
if (metadataResponse.status === 404) return null
if (!metadataResponse.ok) {
throw new Error(`mutation lease inspection failed: ${metadataResponse.status}`)
}
const metadata = (await metadataResponse.json()) as Partial<ObjectMetadata>
if (!/^[1-9][0-9]{0,30}$/.test(metadata.generation ?? '')) {
throw new Error('existing mutation lease has no valid generation')
}
const bodyResponse = await this.fetcher(`${this.metadataUrl()}?alt=media`, {
headers: { Authorization: `Bearer ${token}` }
})
if (!bodyResponse.ok) {
throw new Error(`mutation lease body read failed: ${bodyResponse.status}`)
}
const record = (await bodyResponse.json()) as Partial<LeaseRecord>
if (
typeof record.operationId !== 'string' ||
!/^[a-f0-9]{64}$/.test(record.requestDigest ?? '') ||
!/^[a-f0-9]{40}$/.test(record.imageCommit ?? '') ||
!Number.isSafeInteger(record.acquiredAt) ||
!Number.isSafeInteger(record.expiresAt)
) {
throw new Error('existing mutation lease is invalid')
}
return {
metadata: { generation: metadata.generation! },
record: record as LeaseRecord
}
}
private metadataUrl(): string {
return `https://storage.googleapis.com/storage/v1/b/${encodeURIComponent(this.bucket)}/o/${encodeURIComponent(this.objectName)}`
}
private uploadUrl(): string {
return `https://storage.googleapis.com/upload/storage/v1/b/${encodeURIComponent(this.bucket)}/o?uploadType=media&name=${encodeURIComponent(this.objectName)}`
}
}
@@ -0,0 +1,10 @@
{
"extends": "./tsconfig.json",
"compilerOptions": {
"declaration": true,
"noEmit": false,
"outDir": "dist",
"rootDir": "src"
},
"exclude": ["src/**/*.test.ts"]
}
@@ -0,0 +1,5 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": { "noEmit": true },
"include": ["src/**/*.ts"]
}
+66
View File
@@ -0,0 +1,66 @@
# Orca Relay Operations
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local `gcloud` and `gh` credentials on the server; credentials and per-user Relay state never enter the browser. One cached `gcloud auth print-access-token` refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
## Run locally
Prerequisites:
- Node 24 and pnpm 10
- `gcloud` authenticated for `onorca-cloud` and `onorca-cloud-staging`
- `gh` authenticated with read access to `stablyai/orca-cloud`
From the repository root:
```sh
pnpm install
pnpm ops:relay
```
Open <http://127.0.0.1:2455>. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
## Share through Tailscale
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
```sh
tailscale serve --bg http://127.0.0.1:2455
tailscale serve status
```
Share the HTTPS URL printed by `tailscale serve status` with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
```sh
tailscale serve reset
```
For a persistent host, run `pnpm --filter @orca-cloud/relay-ops build` and supervise `pnpm --filter @orca-cloud/relay-ops start` with the host's normal process manager. The process needs the same non-interactive `gcloud` and `gh` identities.
## Optional staging controls
Controls are intentionally local-only and disabled unless explicitly enabled:
```sh
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
```
Even in this mode the service never changes GCP directly. It dispatches `.github/workflows/power-relay-staging.yml`, preserves the workflow's typed `WAKE_STAGING` / `SLEEP_STAGING` confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
## Data and security boundaries
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
- Missing cell inventory renders as `Unknown`, never `Sleeping`. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale.
- Responses use `no-store`, a restrictive CSP, frame denial, and no-referrer headers.
## Verification
```sh
pnpm --filter @orca-cloud/relay-ops test
pnpm --filter @orca-cloud/relay-ops typecheck
pnpm --filter @orca-cloud/relay-ops build
```
+29
View File
@@ -0,0 +1,29 @@
{
"name": "@orca-cloud/relay-ops",
"private": true,
"version": "0.0.0",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "pnpm clean && tsc -p tsconfig.build.json && node -e \"require('fs').cpSync('public','dist/public',{recursive:true})\"",
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
"dev": "tsx watch src/index.ts",
"incident:monitor": "tsx src/incident-monitor-cli.ts",
"incident:preflight": "tsx src/incident-live-preflight-cli.ts",
"lint": "tsc -p tsconfig.json --noEmit",
"start": "node dist/index.js",
"test": "vitest run",
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"@hono/node-server": "^1.19.14",
"hono": "^4.12.27",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^24.10.0",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vitest": "^4.0.8"
}
}
+273
View File
@@ -0,0 +1,273 @@
const state = { environment: 'production', window: 360, snapshot: null, config: null, loading: false }
const $ = (selector) => document.querySelector(selector)
const all = (selector) => [...document.querySelectorAll(selector)]
const escapeHtml = (value) => String(value).replace(/[&<>'"]/g, (character) => ({
'&': '&amp;', '<': '&lt;', '>': '&gt;', "'": '&#039;', '"': '&quot;'
})[character])
function formatNumber(value, maximumFractionDigits = 0) {
if (value === null || value === undefined) return '—'
return new Intl.NumberFormat('en-US', { notation: value >= 10_000 ? 'compact' : 'standard', maximumFractionDigits }).format(value)
}
function formatBytes(value) {
if (value === null || value === undefined) return '—'
const units = ['B', 'KB', 'MB', 'GB', 'TB']
let amount = value
let unit = 0
while (Math.abs(amount) >= 1000 && unit < units.length - 1) { amount /= 1000; unit += 1 }
return `${formatNumber(amount, amount < 10 ? 1 : 0)} ${units[unit]}`
}
function formatMetric(metric, value) {
if (value === null || value === undefined) return '—'
if (metric.unit === 'bytes') return formatBytes(value)
if (metric.unit === 'milliseconds') return `${formatNumber(value, 1)} ms`
return formatNumber(value, value < 10 ? 1 : 0)
}
function timeAgo(value) {
const seconds = Math.max(0, Math.round((Date.now() - Date.parse(value)) / 1000))
if (seconds < 60) return `${seconds}s ago`
if (seconds < 3600) return `${Math.floor(seconds / 60)}m ago`
if (seconds < 86400) return `${Math.floor(seconds / 3600)}h ago`
return `${Math.floor(seconds / 86400)}d ago`
}
function shortImage(value) {
const digest = value?.match(/sha256:([a-f0-9]{64})/)?.[1]
if (digest) return digest.slice(0, 10)
const tag = value?.split(':').at(-1)
return tag?.slice(0, 16) ?? '—'
}
function badge(label, healthy) {
return `<span class="badge ${healthy === true ? 'healthy' : healthy === false ? 'unhealthy' : ''}">${escapeHtml(label)}</span>`
}
function renderSummary(snapshot) {
const { summary } = snapshot
const utilization = summary.poweredCapacity === null
? null
: summary.poweredCapacity > 0
? summary.observedConnections / summary.poweredCapacity * 100
: 0
const items = [
['Observed connections', formatNumber(summary.observedConnections, 1), 'Latest 1-minute aggregate mean'],
['Active relay sessions', formatNumber(summary.observedSplices, 1), `${formatNumber(summary.observedControls, 1)} desktop-control mean`],
['Healthy cells', `${summary.activeCells ?? '—'} / ${summary.totalCells}`, summary.poweredCapacity === null ? 'Cell inventory unavailable' : `${formatNumber(summary.poweredCapacity)} powered request units`],
['Capacity signal', utilization === null ? '—' : `${formatNumber(utilization, 2)}%`, `${formatNumber(summary.configuredCapacity)} configured admission units`]
]
$('#summary').innerHTML = items.map(([label, value, caption]) => `
<article class="metric-card">
<p class="eyebrow">${escapeHtml(label)}</p>
<div class="value">${escapeHtml(value)}</div>
<div class="caption">${escapeHtml(caption)}</div>
</article>`).join('')
}
function cellState(cell) {
if (cell.targetSize === null) return ['Unknown', null]
if (cell.targetSize === 0) return ['Sleeping', null]
if (cell.backendHealth === 'healthy' && cell.endpoint.ready) return ['Healthy', true]
if (cell.stable && cell.backendHealth === 'empty') return ['Starting', null]
return ['Attention', false]
}
function renderTopology(snapshot) {
const director = snapshot.resources.director
const sleeping = snapshot.resources.cells.every((cell) => cell.targetSize === 0)
&& snapshot.resources.sql?.activationPolicy === 'NEVER'
const directorHealthy = director?.ready && snapshot.resources.directorEndpoint.health
$('#topology-meta').textContent = `${snapshot.environment.project} · ${snapshot.environment.region}`
const cellNodes = snapshot.resources.cells.map((cell) => {
const [label, healthy] = cellState(cell)
return `<div class="topology-node">
<strong>${escapeHtml(cell.hostname.toUpperCase())} ${badge(label, healthy)}</strong>
<span>${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)} · ${formatNumber(cell.capacityRequests)} units</span>
</div>`
}).join('')
$('#topology').innerHTML = `
<div class="topology-node"><strong>Desktop + phone</strong><span>Encrypted Relay traffic</span></div>
<div class="connector" aria-hidden="true"></div>
<div class="topology-node"><strong>Director ${badge(sleeping ? 'Sleeping' : directorHealthy ? 'Ready' : 'Attention', sleeping ? null : Boolean(directorHealthy))}</strong><span>${escapeHtml(snapshot.environment.directorOrigin)}</span></div>
<div class="connector" aria-hidden="true"></div>
<div class="topology-cells">${cellNodes}</div>`
}
function chartPaths(points, width = 400, height = 112) {
if (points.length === 0) return null
const values = points.map((point) => point.value)
const minimum = Math.min(0, ...values)
const maximum = Math.max(...values)
const spread = maximum - minimum || 1
const coordinates = points.map((point, index) => {
const x = points.length === 1 ? width : index / (points.length - 1) * width
const y = height - ((point.value - minimum) / spread * (height - 10) + 5)
return [x, y]
})
const line = coordinates.map(([x, y], index) => `${index === 0 ? 'M' : 'L'}${x.toFixed(1)},${y.toFixed(1)}`).join(' ')
const area = `${line} L${width},${height} L0,${height} Z`
return { line, area }
}
function renderCharts(snapshot) {
const names = ['controls', 'splices', 'assignment_5xx', 'postgres_retries', 'auth_failures', 'event_loop_ms_p99']
$('#charts').innerHTML = names.map((name) => {
const metric = snapshot.monitoring.metrics[name]
const paths = chartPaths(metric.points)
const graph = paths ? `<svg class="chart" viewBox="0 0 400 112" preserveAspectRatio="none" role="img" aria-label="${escapeHtml(metric.label)} over time">
<line x1="0" y1="111" x2="400" y2="111"></line>
<path class="area" d="${paths.area}"></path><path class="line" d="${paths.line}"></path>
</svg>` : '<div class="chart-empty">No samples in this window</div>'
return `<article class="panel chart-card"><div class="chart-head"><div><p class="eyebrow">${escapeHtml(metric.label)}</p><div class="chart-value">${escapeHtml(formatMetric(metric, metric.latest))}</div></div><span class="chart-unit">${escapeHtml(metric.unit)}</span></div>${graph}</article>`
}).join('')
}
function renderCells(snapshot) {
const controlsByCell = snapshot.monitoring.metrics.controls.latestByCell
const splicesByCell = snapshot.monitoring.metrics.splices.latestByCell
$('#cells').innerHTML = snapshot.resources.cells.map((cell) => {
const [label, healthy] = cellState(cell)
const observed = (controlsByCell[cell.cellId] ?? 0) + (splicesByCell[cell.cellId] ?? 0)
return `<tr>
<td><strong>${escapeHtml(cell.hostname.toUpperCase())}</strong><div class="row-caption">${escapeHtml(cell.region)} · ${escapeHtml(cell.zone)}</div></td>
<td>${badge(label, healthy)}<div class="row-caption">MIG ${cell.runningInstances ?? '—'}/${cell.targetSize ?? '—'}</div></td>
<td>${formatNumber(observed, 1)}<div class="row-caption">1-minute mean</div></td>
<td>${formatNumber(cell.capacityRequests)}<div class="row-caption">DB pool ${formatNumber(cell.databasePoolMax)} · ${cell.configuredAdmission ? 'configured' : 'candidate only'}</div></td>
<td class="mono">${escapeHtml(shortImage(cell.imageDigest))}</td>
</tr>`
}).join('')
}
function serviceRow(name, healthy, detail, suffix = '') {
return `<div class="service-row"><div><div class="row-title">${escapeHtml(name)}</div><div class="row-caption">${escapeHtml(detail)}</div></div>${badge(suffix || (healthy ? 'Ready' : 'Attention'), healthy)}</div>`
}
function renderServices(snapshot) {
const { resources } = snapshot
const sleeping = resources.cells.every((cell) => cell.targetSize === 0)
&& resources.sql?.activationPolicy === 'NEVER'
const certDays = resources.certificate?.expireTime
? Math.floor((Date.parse(resources.certificate.expireTime) - Date.now()) / 86400000)
: null
$('#services').innerHTML = [
serviceRow('Director', sleeping ? null : Boolean(resources.director?.ready && resources.directorEndpoint.health), `${resources.director?.revision ?? 'Revision unavailable'} · ${shortImage(resources.director?.image)}`, sleeping ? 'Sleeping' : ''),
serviceRow('Authentication', sleeping ? null : Boolean(resources.auth?.ready && resources.authEndpoint.health), `${resources.auth?.revision ?? 'Revision unavailable'} · ${shortImage(resources.auth?.image)}`, sleeping ? 'Sleeping' : ''),
serviceRow('Cloud SQL', resources.sql?.state === 'RUNNABLE' || resources.sql?.activationPolicy === 'NEVER', `${resources.sql?.tier ?? 'unknown'} · ${resources.sql?.activationPolicy ?? 'unknown'}`, resources.sql?.state ?? 'Unknown'),
serviceRow('Wildcard TLS', resources.certificate?.state === 'ACTIVE', resources.certificate?.domains.join(', ') ?? 'Certificate unavailable', certDays === null ? 'Unknown' : `${certDays}d`)
].join('')
}
function renderAlerts(snapshot) {
const policies = snapshot.monitoring.alertPolicies
$('#alerts').innerHTML = policies.length ? policies.map((policy) => `<div class="list-row"><div><div class="row-title">${escapeHtml(policy.displayName.replace('Orca Relay: ', ''))}</div><div class="row-caption">Cloud Monitoring policy</div></div>${badge(policy.enabled ? 'Enabled' : 'Disabled', policy.enabled)}</div>`).join('') : '<p class="muted">No Relay alert policies returned.</p>'
}
function renderWorkflows(snapshot) {
$('#workflows').innerHTML = snapshot.workflows.length ? snapshot.workflows.slice(0, 6).map((run) => {
const healthy = run.conclusion === 'success'
const stateLabel = run.status === 'completed' ? (run.conclusion ?? 'completed') : run.status
return `<a class="list-row" href="${escapeHtml(run.url)}" target="_blank" rel="noreferrer"><div><div class="row-title">${escapeHtml(run.name)}</div><div class="row-caption">${escapeHtml(run.headSha)} · ${timeAgo(run.updatedAt)}</div></div>${badge(stateLabel, healthy)}</a>`
}).join('') : '<p class="muted">Workflow history unavailable.</p>'
}
function renderCost(snapshot) {
const cost = snapshot.cost
$('#cost').innerHTML = `<div class="cost-total"><strong>$${formatNumber(cost.monthlyUsd)}</strong><span class="muted">/ month</span></div>
<p class="row-caption">Modeled range $${formatNumber(cost.rangeUsd[0])}$${formatNumber(cost.rangeUsd[1])}. Not billed cost.</p>
<div class="cost-lines">${cost.lines.map((line) => `<div class="cost-line"><span>${escapeHtml(line.label)}</span><strong>$${formatNumber(line.monthlyUsd, 2)}</strong></div>`).join('')}</div>
<p class="cost-note"><strong>Exact billing:</strong> ${escapeHtml(cost.actualBilling.reason)}<br>${escapeHtml(cost.caveats[1])}</p>`
}
function renderWarnings(snapshot) {
const warning = $('#warnings')
warning.classList.toggle('hidden', snapshot.warnings.length === 0)
warning.textContent = snapshot.warnings.length ? `Partial data: ${snapshot.warnings.join(' ')}` : ''
}
function render(snapshot) {
state.snapshot = snapshot
$('#environment-label').textContent = `${snapshot.environment.label} · ${snapshot.environment.region}`
$('#freshness').textContent = snapshot.stale
? `Last good update ${timeAgo(snapshot.generatedAt)} · refresh degraded`
: `Updated ${timeAgo(snapshot.generatedAt)}`
$('#freshness-dot').className = snapshot.stale ? 'status-dot neutral' : 'status-dot healthy'
renderWarnings(snapshot)
renderSummary(snapshot)
renderTopology(snapshot)
renderCharts(snapshot)
renderCells(snapshot)
renderServices(snapshot)
renderAlerts(snapshot)
renderWorkflows(snapshot)
renderCost(snapshot)
$('#power-panel').classList.toggle('hidden', !(state.environment === 'staging' && state.config?.stagingControlsEnabled))
$('#compute-link').href = snapshot.environment.consoleLinks.compute
$('#alert-link').href = snapshot.environment.consoleLinks.alerts
}
async function loadSnapshot() {
if (state.loading) return
state.loading = true
$('#refresh').disabled = true
$('#error').classList.add('hidden')
$('#freshness-dot').className = 'status-dot neutral'
$('#freshness').textContent = 'Refreshing current state…'
try {
const response = await fetch(`/api/snapshot?environment=${state.environment}&window=${state.window}`)
const body = await response.json()
if (!response.ok) throw new Error(body.error ?? 'Snapshot failed')
render(body)
} catch (error) {
$('#freshness-dot').className = 'status-dot unhealthy'
$('#freshness').textContent = 'Refresh failed'
$('#error').textContent = error instanceof Error ? error.message : 'Relay operations data is unavailable.'
$('#error').classList.remove('hidden')
} finally {
state.loading = false
$('#refresh').disabled = false
}
}
async function dispatchPower(mode) {
const confirmation = mode === 'wake' ? 'WAKE_STAGING' : mode === 'sleep' ? 'SLEEP_STAGING' : ''
if (confirmation) {
const entered = window.prompt(`Type ${confirmation} to dispatch the guarded workflow.`) ?? ''
if (entered !== confirmation) return
}
all('[data-power]').forEach((button) => { button.disabled = true })
$('#power-result').textContent = 'Dispatching…'
try {
const response = await fetch('/api/staging/power', {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-csrf-token': state.config.csrfToken },
body: JSON.stringify({ mode, confirmation })
})
const body = await response.json()
if (!response.ok) throw new Error(body.error)
$('#power-result').textContent = 'Workflow accepted. Refresh after it completes.'
} catch (error) {
$('#power-result').textContent = error instanceof Error ? error.message : 'Dispatch failed.'
} finally {
all('[data-power]').forEach((button) => { button.disabled = false })
}
}
all('[data-environment]').forEach((button) => button.addEventListener('click', () => {
state.environment = button.dataset.environment
all('[data-environment]').forEach((candidate) => candidate.setAttribute('aria-pressed', String(candidate === button)))
loadSnapshot()
}))
$('#window').addEventListener('change', (event) => { state.window = Number(event.target.value); loadSnapshot() })
$('#refresh').addEventListener('click', loadSnapshot)
all('[data-power]').forEach((button) => button.addEventListener('click', () => dispatchPower(button.dataset.power)))
async function start() {
try { state.config = await fetch('/api/config').then((response) => response.json()) } catch { state.config = {} }
await loadSnapshot()
window.setInterval(loadSnapshot, 60_000)
}
start()
+115
View File
@@ -0,0 +1,115 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="color-scheme" content="light dark" />
<title>Orca Relay Operations</title>
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='9' fill='%23171717'/%3E%3Ctext x='16' y='22' text-anchor='middle' font-family='sans-serif' font-weight='700' font-size='17' fill='white'%3EO%3C/text%3E%3C/svg%3E" />
<link rel="stylesheet" href="/styles.css" />
<script type="module" src="/app.js"></script>
</head>
<body>
<header class="topbar">
<div class="brand">
<span class="brand-mark" aria-hidden="true">O</span>
<div>
<strong>Orca Relay</strong>
<span>Operations</span>
</div>
</div>
<div class="toolbar">
<div class="segmented" aria-label="Environment">
<button type="button" data-environment="production" aria-pressed="true">Production</button>
<button type="button" data-environment="staging" aria-pressed="false">Staging</button>
</div>
<select id="window" aria-label="Metrics window">
<option value="60">Last hour</option>
<option value="360" selected>Last 6 hours</option>
<option value="1440">Last 24 hours</option>
</select>
<button type="button" id="refresh" class="button secondary">Refresh</button>
</div>
</header>
<main>
<section class="page-heading">
<div>
<p class="eyebrow" id="environment-label">Production · us-central1</p>
<h1>Relay data plane</h1>
<p class="lede">Private, aggregate operations view. No account, host, device, or pairing identifiers.</p>
</div>
<div class="freshness">
<span class="status-dot neutral" id="freshness-dot"></span>
<span id="freshness">Loading current state…</span>
</div>
</section>
<div id="error" class="banner error hidden" role="alert"></div>
<div id="warnings" class="banner warning hidden"></div>
<section class="summary-grid" id="summary" aria-label="Relay summary">
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
<article class="metric-card skeleton"></article>
</section>
<section class="panel topology-panel">
<div class="panel-heading">
<div><p class="eyebrow">Topology</p><h2>Request path</h2></div>
<span class="meta" id="topology-meta"></span>
</div>
<div class="topology" id="topology"></div>
</section>
<section class="section-heading"><div><p class="eyebrow">Traffic</p><h2>Signals in selected window</h2></div></section>
<section class="chart-grid" id="charts"></section>
<section class="two-column">
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Compute</p><h2>Relay cells</h2></div><a class="meta panel-link" id="compute-link" target="_blank" rel="noreferrer">Open in GCP ↗</a></div>
<p class="panel-note">Configured admission is shown below. Live director admission and heartbeat state stays unavailable because this dashboard has no Relay-admin identity.</p>
<div class="table-wrap"><table>
<thead><tr><th>Cell</th><th>State</th><th>Observed</th><th>Capacity</th><th>Image</th></tr></thead>
<tbody id="cells"></tbody>
</table></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Control plane</p><h2>Services</h2></div></div>
<div class="service-list" id="services"></div>
</article>
</section>
<section class="three-column">
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Guardrails</p><h2>Alert policies</h2></div><a class="meta panel-link" id="alert-link" target="_blank" rel="noreferrer">Open incidents ↗</a></div>
<div class="list" id="alerts"></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Delivery</p><h2>Recent workflows</h2></div></div>
<div class="list" id="workflows"></div>
</article>
<article class="panel">
<div class="panel-heading"><div><p class="eyebrow">Planning</p><h2>Monthly run-rate</h2></div></div>
<div id="cost"></div>
</article>
</section>
<section class="panel hidden" id="power-panel">
<div class="panel-heading">
<div><p class="eyebrow">Staging only</p><h2>Power workflow</h2></div>
<span class="badge">GitHub guarded</span>
</div>
<p class="muted">Dispatches the reviewed GitHub workflow. It never changes GCP resources directly from this process.</p>
<div class="power-actions">
<button class="button secondary" type="button" data-power="status">Check status</button>
<button class="button secondary" type="button" data-power="wake">Wake configured cells</button>
<button class="button secondary" type="button" data-power="sleep">Sleep staging</button>
<span id="power-result" class="meta"></span>
</div>
</section>
</main>
<footer>Orca Relay Operations · Server-side credentials · Read-only by default</footer>
</body>
</html>
+165
View File
@@ -0,0 +1,165 @@
:root {
--radius: 10px;
--background: #fff;
--foreground: #0a0a0a;
--card: #fff;
--primary: #171717;
--primary-foreground: #fafafa;
--secondary: #f5f5f5;
--muted: #f5f5f5;
--muted-foreground: #737373;
--accent: #f5f5f5;
--border: #e5e5e5;
--ring: #a1a1a1;
--destructive: #e40014;
--success: #15803d;
--warning: #895503;
--font-mono: 'SF Mono', SFMono-Regular, ui-monospace, Menlo, Consolas, monospace;
}
@media (prefers-color-scheme: dark) {
:root {
--background: #0a0a0a;
--foreground: #fafafa;
--card: #171717;
--primary: #e5e5e5;
--primary-foreground: #171717;
--secondary: #262626;
--muted: #262626;
--muted-foreground: #a1a1a1;
--accent: #262626;
--border: rgb(255 255 255 / 0.07);
--ring: #737373;
--destructive: #ff6568;
--success: #4ade80;
--warning: #fbbf24;
}
}
* { box-sizing: border-box; }
body {
margin: 0;
background: var(--background);
color: var(--foreground);
font-family: 'Geist', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
font-size: 14px;
letter-spacing: .01em;
}
button, select { font: inherit; }
button:focus-visible, select:focus-visible { outline: 2px solid var(--ring); outline-offset: 2px; }
.topbar {
height: 64px;
border-bottom: 1px solid var(--border);
padding: 0 max(24px, calc((100vw - 1440px) / 2));
display: flex;
align-items: center;
justify-content: space-between;
position: sticky;
top: 0;
background: color-mix(in srgb, var(--background) 92%, transparent);
backdrop-filter: blur(16px);
z-index: 10;
}
.brand { display: flex; align-items: center; gap: 10px; }
.brand-mark {
width: 30px; height: 30px; border-radius: 9px; background: var(--primary);
color: var(--primary-foreground); display: grid; place-items: center; font-weight: 700;
}
.brand div { display: flex; align-items: baseline; gap: 7px; }
.brand span:last-child { color: var(--muted-foreground); font-size: 12px; }
.toolbar { display: flex; align-items: center; gap: 8px; }
.segmented { display: flex; padding: 3px; gap: 2px; border-radius: 8px; background: var(--secondary); }
.segmented button { border: 0; background: transparent; color: var(--muted-foreground); padding: 5px 10px; border-radius: 6px; cursor: pointer; }
.segmented button[aria-pressed="true"] { background: var(--card); color: var(--foreground); box-shadow: 0 1px 2px rgb(0 0 0 / .08); }
select, .button { height: 32px; border: 1px solid var(--border); border-radius: 8px; background: var(--card); color: var(--foreground); padding: 0 10px; }
.button { cursor: pointer; font-weight: 500; }
.button:hover { background: var(--accent); }
.button:disabled { opacity: .5; cursor: wait; }
main { max-width: 1440px; margin: 0 auto; padding: 42px 24px 72px; }
.page-heading { display: flex; justify-content: space-between; align-items: end; gap: 24px; margin-bottom: 28px; }
h1, h2, p { margin: 0; }
h1 { font-size: 28px; line-height: 1.2; letter-spacing: -.025em; margin-top: 5px; }
h2 { font-size: 16px; line-height: 1.25; letter-spacing: -.01em; margin-top: 3px; }
.lede { color: var(--muted-foreground); margin-top: 8px; max-width: 680px; }
.eyebrow { color: var(--muted-foreground); font-size: 11px; line-height: 1; font-weight: 600; text-transform: uppercase; letter-spacing: .05em; }
.freshness { color: var(--muted-foreground); display: flex; align-items: center; gap: 8px; font-size: 12px; white-space: nowrap; }
.status-dot { width: 7px; height: 7px; border-radius: 999px; background: var(--ring); display: inline-block; }
.status-dot.healthy { background: var(--success); }
.status-dot.unhealthy { background: var(--destructive); }
.summary-grid { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
.metric-card, .panel { border: 1px solid var(--border); border-radius: 14px; background: var(--card); }
.metric-card { padding: 18px; min-height: 116px; }
.metric-card .value { font-size: 29px; font-weight: 600; letter-spacing: -.035em; margin-top: 16px; }
.metric-card .caption { color: var(--muted-foreground); font-size: 12px; margin-top: 3px; }
.skeleton { background: linear-gradient(90deg, var(--card), var(--muted), var(--card)); background-size: 200% 100%; animation: shimmer 1.6s infinite; }
@keyframes shimmer { to { background-position: -200% 0; } }
.panel { padding: 18px; }
.topology-panel { margin-bottom: 36px; }
.panel-heading, .section-heading { display: flex; align-items: center; justify-content: space-between; gap: 16px; margin-bottom: 18px; }
.section-heading { margin-top: 34px; }
.topology { display: grid; grid-template-columns: 1fr 48px 1fr 48px 2fr; align-items: stretch; gap: 8px; }
.topology-node { border: 1px solid var(--border); background: var(--background); border-radius: 10px; padding: 14px; min-width: 0; }
.topology-node strong { display: block; font-size: 13px; }
.topology-node span { display: block; color: var(--muted-foreground); font-size: 12px; margin-top: 4px; overflow: hidden; text-overflow: ellipsis; }
.topology-cells { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 8px; }
.connector { display: grid; place-items: center; color: var(--muted-foreground); }
.connector::before { content: ''; width: 100%; height: 1px; background: var(--border); }
.chart-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 36px; }
.chart-card { min-height: 220px; }
.chart-head { display: flex; justify-content: space-between; align-items: start; }
.chart-value { font-size: 22px; font-weight: 600; letter-spacing: -.03em; }
.chart-unit { color: var(--muted-foreground); font-size: 11px; }
.chart { width: 100%; height: 122px; margin-top: 18px; overflow: visible; }
.chart path.area { fill: color-mix(in srgb, var(--foreground) 5%, transparent); }
.chart path.line { fill: none; stroke: var(--foreground); stroke-width: 1.5; vector-effect: non-scaling-stroke; }
.chart line { stroke: var(--border); stroke-width: 1; }
.chart-empty { color: var(--muted-foreground); height: 120px; display: grid; place-items: center; font-size: 12px; }
.two-column { display: grid; grid-template-columns: 1.7fr 1fr; gap: 12px; margin-bottom: 12px; }
.three-column { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin-bottom: 12px; }
.two-column > *, .three-column > *, .chart-grid > * { min-width: 0; }
.table-wrap { overflow-x: auto; }
table { width: 100%; border-collapse: collapse; font-size: 12px; }
th { color: var(--muted-foreground); font-weight: 500; text-align: left; padding: 0 10px 10px; }
td { padding: 12px 10px; border-top: 1px solid var(--border); vertical-align: middle; }
td:first-child, th:first-child { padding-left: 0; }
td:last-child, th:last-child { padding-right: 0; }
.mono { font-family: var(--font-mono); font-size: 11px; }
.badge { display: inline-flex; align-items: center; border: 1px solid var(--border); border-radius: 999px; padding: 2px 7px; font-size: 11px; color: var(--muted-foreground); }
.badge.healthy { color: var(--success); border-color: color-mix(in srgb, var(--success) 25%, var(--border)); }
.badge.unhealthy { color: var(--destructive); border-color: color-mix(in srgb, var(--destructive) 25%, var(--border)); }
.service-list, .list { display: grid; }
.service-row, .list-row { padding: 11px 0; border-top: 1px solid var(--border); display: flex; align-items: center; justify-content: space-between; gap: 12px; min-width: 0; }
.service-row:first-child, .list-row:first-child { border-top: 0; padding-top: 0; }
.service-row:last-child, .list-row:last-child { padding-bottom: 0; }
.row-title { font-size: 12px; font-weight: 500; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.row-caption, .meta, .muted { color: var(--muted-foreground); font-size: 11px; }
.panel-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.45; margin: -8px 0 14px; }
.panel-link:hover { color: var(--foreground); text-decoration: underline; }
.row-caption { margin-top: 3px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
a { color: inherit; text-decoration: none; }
a:hover .row-title { text-decoration: underline; }
.cost-total { display: flex; align-items: baseline; gap: 7px; margin-bottom: 12px; }
.cost-total strong { font-size: 28px; letter-spacing: -.035em; }
.cost-line { display: flex; justify-content: space-between; gap: 10px; padding: 7px 0; border-top: 1px solid var(--border); font-size: 12px; }
.cost-note { color: var(--muted-foreground); font-size: 11px; line-height: 1.5; margin-top: 12px; }
.banner { border: 1px solid var(--border); border-radius: 10px; padding: 11px 13px; margin-bottom: 12px; font-size: 12px; }
.banner.error { color: var(--destructive); }
.banner.warning { color: var(--warning); }
.hidden { display: none !important; }
.power-actions { display: flex; align-items: center; gap: 8px; margin-top: 14px; flex-wrap: wrap; }
footer { border-top: 1px solid var(--border); padding: 24px; color: var(--muted-foreground); font-size: 11px; text-align: center; }
@media (max-width: 1050px) {
.summary-grid, .chart-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); }
.three-column { grid-template-columns: 1fr; }
.topology { grid-template-columns: 1fr; }
.connector { height: 20px; }
.connector::before { height: 100%; width: 1px; }
}
@media (max-width: 760px) {
.topbar { height: auto; min-height: 64px; padding: 12px 16px; align-items: stretch; flex-direction: column; gap: 12px; }
.brand div { display: grid; gap: 0; }
.toolbar { flex-wrap: wrap; justify-content: flex-start; }
main { padding: 28px 16px 56px; }
.page-heading { align-items: flex-start; flex-direction: column; }
.summary-grid, .chart-grid, .two-column { grid-template-columns: 1fr; }
.topology-cells { grid-template-columns: 1fr; }
table { min-width: 600px; }
}
@@ -0,0 +1,85 @@
import { describe, expect, it } from 'vitest'
import { buildCostModel } from './cost-model.js'
import {
RELAY_OPS_ENVIRONMENTS,
relayOpsCellsFromTerraform,
type RelayOpsCellConfig
} from './environment-config.js'
import type { ResourceInventory } from './resource-inventory.js'
const regionalCellsSource = `
relay_gce_cells = {
"staging-gce-c3" = {
hostname = "c3"
zone = "us-central1-a"
machine_type = "e2-standard-2"
capacity_requests = 4000
}
"staging-gce-c4" = {
hostname = "c4"
region = "asia-east2"
zone = "asia-east2-a"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
}
`
function inventory(
targetSize: number,
activationPolicy: string,
cells: RelayOpsCellConfig[] = RELAY_OPS_ENVIRONMENTS.staging.cells
): ResourceInventory {
return {
director: null,
auth: null,
sql: { state: targetSize ? 'RUNNABLE' : 'STOPPED', activationPolicy, tier: 'db-custom-1-3840', availabilityType: 'ZONAL', databaseVersion: 'POSTGRES_17' },
certificate: null,
directorEndpoint: { health: null, ready: null, latencyMs: null },
authEndpoint: { health: null, ready: null, latencyMs: null },
cells: cells.map((cell) => ({
...cell, migName: `mig-${cell.hostname}`, targetSize, runningInstances: targetSize,
stable: true, template: 'template', imageDigest: null,
backendHealth: targetSize ? 'healthy' : 'empty',
endpoint: { health: null, ready: null, latencyMs: null }
})),
warnings: []
}
}
describe('buildCostModel', () => {
it('shows the sleeping staging floor without VM or SQL compute', () => {
const result = buildCostModel(RELAY_OPS_ENVIRONMENTS.staging, inventory(0, 'NEVER'))
expect(result.kind).toBe('planning-estimate')
expect(result.monthlyUsd).toBe(34)
expect(result.lines.find((line) => line.label === 'Relay cell VMs')?.monthlyUsd).toBe(0)
expect(result.actualBilling.available).toBe(false)
expect(result.caveats[0]).toContain('not the Cloud Billing invoice')
})
it('prices durable machine inventory and network floors by region', () => {
const cells = relayOpsCellsFromTerraform({
environment: 'staging',
domain: 'relay-staging.onorca.dev',
source: regionalCellsSource
})
const environment = { ...RELAY_OPS_ENVIRONMENTS.staging, cells }
const result = buildCostModel(environment, inventory(1, 'NEVER', cells))
const machines = result.lines.find((line) => line.label === 'Relay cell VMs')
const network = result.lines.find((line) => line.label === 'Load balancer and network floor')
expect(machines).toEqual({
label: 'Relay cell VMs',
monthlyUsd: 185.79,
basis: '2 configured VM cells at regional machine rates × 730 hours'
})
expect(network).toEqual({
label: 'Load balancer and network floor',
monthlyUsd: 29,
basis: 'shared HTTPS foundation plus NAT floor in 2 configured regions'
})
})
})
+115
View File
@@ -0,0 +1,115 @@
import type {
RelayOpsEnvironment,
RelayOpsMachineType,
RelayOpsRegion
} from './environment-config.js'
import type { ResourceInventory } from './resource-inventory.js'
export type CostLine = {
label: string
monthlyUsd: number
basis: string
}
export type CostModel = {
kind: 'planning-estimate'
monthlyUsd: number
rangeUsd: [number, number]
actualBilling: { available: false; reason: string }
lines: CostLine[]
caveats: string[]
}
const HOURS_PER_MONTH = 730
const MACHINE_HOURLY_USD: Record<
RelayOpsRegion,
Record<RelayOpsMachineType, number>
> = {
'us-central1': {
'e2-standard-2': 0.06701142,
'e2-standard-4': 0.13402284
},
'asia-east2': {
'e2-standard-2': 0.0938,
'e2-standard-4': 0.1875
}
}
const SHARED_NETWORK_FLOOR_USD = 19
const REGIONAL_NAT_FLOOR_USD = 5
function round(value: number): number {
return Math.round(value * 100) / 100
}
export function buildCostModel(
environment: RelayOpsEnvironment,
resources: ResourceInventory
): CostModel {
const runningCells = resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0)
const runningCellCount = runningCells.reduce((sum, cell) => sum + (cell.targetSize ?? 0), 0)
const compute = runningCells.reduce(
(sum, cell) =>
sum + (cell.targetSize ?? 0) * MACHINE_HOURLY_USD[cell.region][cell.machineType],
0
) * HOURS_PER_MONTH
const disks = runningCellCount * 30 * 0.1
const sqlRunning = resources.sql?.activationPolicy === 'ALWAYS'
const sql = sqlRunning ? (environment.id === 'production' ? 105 : 52) : 0
const cloudRunMinimums =
(resources.director?.minInstances ?? 0) + (resources.auth?.minInstances ?? 0)
const cloudRun = cloudRunMinimums * 10
const configuredRegions = new Set(
(resources.cells.length > 0 ? resources.cells : environment.cells).map((cell) => cell.region)
)
const networkFoundation =
SHARED_NETWORK_FLOOR_USD + configuredRegions.size * REGIONAL_NAT_FLOOR_USD
const observability = environment.id === 'production' ? 12 : 5
const lines: CostLine[] = [
{
label: 'Relay cell VMs',
monthlyUsd: round(compute),
basis: `${runningCellCount} configured VM cells at regional machine rates × 730 hours`
},
{
label: 'Cell boot disks',
monthlyUsd: round(disks),
basis: `${runningCellCount} × 30 GB balanced persistent disk`
},
{
label: 'Cloud SQL',
monthlyUsd: sql,
basis: sqlRunning ? `${resources.sql?.tier ?? 'configured tier'} active` : 'stopped'
},
{
label: 'Cloud Run minimums',
monthlyUsd: cloudRun,
basis: `${cloudRunMinimums} configured minimum instances`
},
{
label: 'Load balancer and network floor',
monthlyUsd: networkFoundation,
basis: `shared HTTPS foundation plus NAT floor in ${configuredRegions.size} configured region${configuredRegions.size === 1 ? '' : 's'}`
},
{
label: 'Logs and monitoring allowance',
monthlyUsd: observability,
basis: 'planning allowance; varies with traffic and retention'
}
]
const monthlyUsd = round(lines.reduce((sum, line) => sum + line.monthlyUsd, 0))
return {
kind: 'planning-estimate',
monthlyUsd,
rangeUsd: [round(monthlyUsd * 0.85), round(monthlyUsd * 1.3)],
actualBilling: {
available: false,
reason: 'Cloud Billing export is not configured in either Relay project.'
},
lines,
caveats: [
'This is a modeled run-rate, not the Cloud Billing invoice.',
'Network egress, actual request traffic, credits, discounts, taxes, and free tiers are excluded.',
'Use the GCP Billing report for authoritative spend and forecasts.'
]
}
}
@@ -0,0 +1,62 @@
import { describe, expect, it } from 'vitest'
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
import type { DashboardSnapshot } from './dashboard-snapshot.js'
import type { GcloudClient } from './gcloud-client.js'
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
function snapshot(kind: 'good' | 'unavailable'): DashboardSnapshot {
const good = kind === 'good'
return {
generatedAt: '2026-07-15T12:00:00.000Z',
resources: {
director: good ? {} : null,
auth: good ? {} : null,
sql: good ? {} : null,
cells: [{ targetSize: good ? 1 : null }]
},
monitoring: {
warnings: good
? []
: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
},
summary: { observedConnections: good ? 7 : 0 },
warnings: good ? [] : ['Google Cloud credentials are unavailable. Run gcloud auth login.'],
stale: false,
staleReason: null
} as unknown as DashboardSnapshot
}
describe('DashboardSnapshotCache', () => {
it('keeps the last good view when a later credential refresh fails', async () => {
let calls = 0
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
calls += 1
return snapshot(calls === 1 ? 'good' : 'unavailable')
})
const first = await cache.read('production', 30)
const second = await cache.read('production', 31)
expect(first.stale).toBe(false)
expect(second.stale).toBe(true)
expect(second.summary.observedConnections).toBe(7)
expect(second.staleReason).toContain('credentials')
})
it('keeps the last good view when a later collector throws', async () => {
let calls = 0
const cache = new DashboardSnapshotCache(gcloud, 0, async () => {
calls += 1
if (calls > 1) throw new Error('sensitive collector context')
return snapshot('good')
})
await cache.read('production', 30)
const second = await cache.read('production', 31)
expect(second.stale).toBe(true)
expect(second.summary.observedConnections).toBe(7)
expect(JSON.stringify(second)).not.toContain('sensitive collector context')
})
})
@@ -0,0 +1,167 @@
import type { RelayOpsEnvironmentId } from './environment-config.js'
import { relayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { readRelayWorkflowRuns } from './github-runs.js'
import { readMonitoringSnapshot } from './monitoring-snapshot.js'
import { readResourceInventory } from './resource-inventory.js'
import { buildCostModel } from './cost-model.js'
export type DashboardSnapshot = Awaited<ReturnType<typeof buildDashboardSnapshot>>
export async function buildDashboardSnapshot(
environmentId: RelayOpsEnvironmentId,
gcloud: GcloudClient,
options: { windowMinutes?: number; fetchImpl?: typeof fetch; now?: Date } = {}
) {
const generatedAt = (options.now ?? new Date()).toISOString()
const environment = relayOpsEnvironment(environmentId)
const [monitoringResult, resourceResult, workflowResult] = await Promise.allSettled([
readMonitoringSnapshot(environment, gcloud, {
...(options.now === undefined ? {} : { now: options.now }),
...(options.windowMinutes === undefined ? {} : { windowMinutes: options.windowMinutes }),
...(options.fetchImpl === undefined ? {} : { fetchImpl: options.fetchImpl })
}),
readResourceInventory(environment, gcloud, options.fetchImpl),
readRelayWorkflowRuns()
])
if (monitoringResult.status === 'rejected' || resourceResult.status === 'rejected') {
const failed = [
monitoringResult.status === 'rejected' ? 'Monitoring snapshot' : null,
resourceResult.status === 'rejected' ? 'Resource inventory' : null
].filter(Boolean)
throw new Error(`${failed.join(' and ')} unavailable`)
}
const resources = resourceResult.value
const monitoring = monitoringResult.value
const warnings = [...resources.warnings, ...monitoring.warnings]
const poweredDigests = new Set(
resources.cells.filter((cell) => (cell.targetSize ?? 0) > 0).map((cell) => cell.imageDigest)
)
if (poweredDigests.has(null)) warnings.push('A powered cell image digest is unavailable.')
if (poweredDigests.size > 1) warnings.push('Powered cells are not serving one immutable digest.')
const expectedCertificateDomain = `*.${new URL(environment.cells[0]!.origin).hostname
.split('.').slice(1).join('.')}`
if (resources.certificate && !resources.certificate.domains.includes(expectedCertificateDomain)) {
warnings.push('The Relay certificate domain does not match the configured cell domain.')
}
if (workflowResult.status === 'rejected') warnings.push('GitHub workflow history is unavailable.')
const observedConnections = monitoring.metrics.total_connections.latest ?? 0
const observedControls = monitoring.metrics.controls.latest ?? 0
const observedSplices = monitoring.metrics.splices.latest ?? 0
const configuredCapacity = environment.cells
.filter((cell) => cell.configuredAdmission)
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
const cellInventoryAvailable = resources.cells.every((cell) => cell.targetSize !== null)
const poweredCapacity = cellInventoryAvailable
? resources.cells
.filter((cell) => (cell.targetSize ?? 0) > 0)
.reduce((sum, cell) => sum + cell.capacityRequests, 0)
: null
return {
schemaVersion: 1,
generatedAt,
environment: {
id: environment.id,
label: environment.label,
project: environment.project,
region: environment.region,
directorOrigin: environment.directorOrigin,
authOrigin: environment.authOrigin,
consoleLinks: {
project: `https://console.cloud.google.com/home/dashboard?project=${environment.project}`,
alerts: `https://console.cloud.google.com/monitoring/alerting?project=${environment.project}`,
compute: `https://console.cloud.google.com/compute/instanceGroups/list?project=${environment.project}`
}
},
summary: {
observedConnections,
observedControls,
observedSplices,
configuredCapacity,
poweredCapacity,
activeCells: cellInventoryAvailable
? resources.cells.filter(
(cell) => (cell.targetSize ?? 0) > 0 && cell.backendHealth === 'healthy'
).length
: null,
totalCells: resources.cells.length
},
resources,
monitoring,
workflows: workflowResult.status === 'fulfilled' ? workflowResult.value : [],
cost: buildCostModel(environment, resources),
warnings,
stale: false,
staleReason: null as string | null
}
}
type SnapshotCacheEntry = { snapshot: DashboardSnapshot; expiresAt: number }
type SnapshotBuilder = (
environment: RelayOpsEnvironmentId,
gcloud: GcloudClient,
options: { windowMinutes?: number }
) => Promise<DashboardSnapshot>
export class DashboardSnapshotCache {
private readonly entries = new Map<string, SnapshotCacheEntry>()
private readonly pending = new Map<string, Promise<DashboardSnapshot>>()
private readonly lastGood = new Map<RelayOpsEnvironmentId, DashboardSnapshot>()
constructor(
private readonly gcloud: GcloudClient,
private readonly ttlMs = 30_000,
private readonly builder: SnapshotBuilder = buildDashboardSnapshot
) {}
async read(environment: RelayOpsEnvironmentId, windowMinutes: number): Promise<DashboardSnapshot> {
const key = `${environment}:${windowMinutes}`
const cached = this.entries.get(key)
if (cached && cached.expiresAt > Date.now()) return cached.snapshot
const existing = this.pending.get(key)
if (existing) return await existing
const request = this.builder(environment, this.gcloud, { windowMinutes })
.then((snapshot) => {
const coreInventoryUnavailable =
snapshot.resources.director === null &&
snapshot.resources.auth === null &&
snapshot.resources.sql === null &&
snapshot.resources.cells.every((cell) => cell.targetSize === null)
const monitoringCredentialsUnavailable = snapshot.monitoring.warnings.some(
(warning) => warning.includes('credentials are unavailable')
)
const lastGood = this.lastGood.get(environment)
const result = (coreInventoryUnavailable || monitoringCredentialsUnavailable) && lastGood
? {
...lastGood,
stale: true,
staleReason: 'Local Google Cloud credentials are temporarily unavailable.',
warnings: [...new Set([...lastGood.warnings, ...snapshot.warnings])]
}
: snapshot
if (!coreInventoryUnavailable && !monitoringCredentialsUnavailable) {
this.lastGood.set(environment, snapshot)
}
this.entries.set(key, { snapshot: result, expiresAt: Date.now() + this.ttlMs })
return result
})
.catch((error: unknown) => {
const lastGood = this.lastGood.get(environment)
if (!lastGood) throw error
const stale = {
...lastGood,
stale: true,
staleReason: 'The latest operations refresh failed; showing the last good snapshot.',
warnings: [...new Set([
...lastGood.warnings,
'The latest operations refresh failed before a complete snapshot was available.'
])]
}
this.entries.set(key, { snapshot: stale, expiresAt: Date.now() + this.ttlMs })
return stale
})
.finally(() => this.pending.delete(key))
this.pending.set(key, request)
return await request
}
}
@@ -0,0 +1,148 @@
import { readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import {
RELAY_OPS_ENVIRONMENTS,
relayOpsCellsFromTerraform
} from './environment-config.js'
const durableUsCell = `
"production-gce-c26" = {
hostname = "c26"
zone = "us-central1-a"
machine_type = "e2-standard-4"
capacity_requests = 4000
initially_enabled = false
}
`
const durableAsiaCells = `
"production-gce-c27" = {
hostname = "c27"
region = "asia-east2"
zone = "asia-east2-a"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
"production-gce-c28" = {
hostname = "c28"
region = "asia-east2"
zone = "asia-east2-b"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
"production-gce-c29" = {
hostname = "c29"
region = "asia-east2"
zone = "asia-east2-c"
machine_type = "e2-standard-4"
capacity_requests = 6000
database_pool_max = 10
initially_enabled = false
}
`
const durableAsiaSource = `
relay_gce_cells = {${durableUsCell}${durableAsiaCells}}
`
const durableUsOnlySource = `
relay_gce_cells = {${durableUsCell}}
`
// Why: relay-ops sat at 18 cells for four days after C19-C22 shipped, which threw
// `selector membership must contain every configured cell exactly once` and blocked
// every production mutation. Reading Terraform here makes that drift fail the build.
function terraformCells(environment: 'production' | 'staging'): Array<{
cellId: string
region: string
zone: string
machineType: string
capacityRequests: number
databasePoolMax: number
}> {
const tfvars = readFileSync(
fileURLToPath(new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url)),
'utf8'
)
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(tfvars)?.[1] ?? ''
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
.map((match) => ({
cellId: match[1] ?? '',
region: /region\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? 'us-central1',
zone: /zone\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
machineType: /machine_type\s*=\s*"([^"]+)"/.exec(match[2] ?? '')?.[1] ?? '',
capacityRequests: Number(/capacity_requests\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1]),
databasePoolMax: Number(/database_pool_max\s*=\s*(\d+)/.exec(match[2] ?? '')?.[1] ?? 10)
}))
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
}
const cellOrdinal = (cellId: string): number => Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
describe('relay operations environment config', () => {
it.each(['production', 'staging'] as const)(
'matches the %s cells Terraform actually provisions',
(environment) => {
const expected = terraformCells(environment)
expect(expected.length).toBeGreaterThan(0)
expect(
RELAY_OPS_ENVIRONMENTS[environment].cells.map((cell) => ({
cellId: cell.cellId,
region: cell.region,
zone: cell.zone,
machineType: cell.machineType,
capacityRequests: cell.capacityRequests,
databasePoolMax: cell.databasePoolMax
}))
).toEqual(expected)
}
)
it('derives hostname and origin from the cell ordinal', () => {
const cells = RELAY_OPS_ENVIRONMENTS.production.cells
expect(cells.at(-1)).toMatchObject({
hostname: `c${cells.length}`,
origin: `https://c${cells.length}.relay.onorca.dev`
})
})
it('inventories Asia cells only when they exist in durable Terraform', () => {
const cells = relayOpsCellsFromTerraform({
environment: 'production',
domain: 'relay.onorca.dev',
source: durableAsiaSource
})
expect(cells.slice(1)).toEqual([
{
cellId: 'production-gce-c27', hostname: 'c27', origin: 'https://c27.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-a', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
},
{
cellId: 'production-gce-c28', hostname: 'c28', origin: 'https://c28.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-b', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
},
{
cellId: 'production-gce-c29', hostname: 'c29', origin: 'https://c29.relay.onorca.dev',
region: 'asia-east2', zone: 'asia-east2-c', machineType: 'e2-standard-4',
capacityRequests: 6000, databasePoolMax: 10, configuredAdmission: false
}
])
const usOnlyCells = relayOpsCellsFromTerraform({
environment: 'production',
domain: 'relay.onorca.dev',
source: durableUsOnlySource
})
expect(usOnlyCells).toHaveLength(1)
expect(usOnlyCells.some((cell) => cell.region === 'asia-east2')).toBe(false)
expect(usOnlyCells.some((cell) => cell.cellId === 'production-gce-c27')).toBe(false)
})
})
@@ -0,0 +1,125 @@
import { readFileSync } from 'node:fs'
import { z } from 'zod'
export type RelayOpsEnvironmentId = 'production' | 'staging'
export type RelayOpsRegion = 'us-central1' | 'asia-east2'
export type RelayOpsMachineType = 'e2-standard-2' | 'e2-standard-4'
export type RelayOpsCellConfig = {
cellId: string
hostname: string
origin: string
region: RelayOpsRegion
zone: string
machineType: RelayOpsMachineType
capacityRequests: number
databasePoolMax: number
configuredAdmission: boolean
}
export type RelayOpsEnvironment = {
id: RelayOpsEnvironmentId
label: string
project: string
region: string
directorOrigin: string
authOrigin: string
directorService: string
authService: string
sqlInstance: string
migPrefix: string
certificateName: string
cells: RelayOpsCellConfig[]
}
const RegionSchema = z.enum(['us-central1', 'asia-east2'])
const MachineTypeSchema = z.enum(['e2-standard-2', 'e2-standard-4'])
const EnvironmentSchema = z.enum(['production', 'staging'])
function cellOrdinal(cellId: string): number {
return Number(/c(\d+)$/.exec(cellId)?.[1] ?? 0)
}
function required(body: string, pattern: RegExp, label: string): string {
const value = pattern.exec(body)?.[1]
if (!value) throw new Error(`Relay Ops could not read ${label} from durable Terraform config`)
return value
}
export function relayOpsCellsFromTerraform(input: {
environment: RelayOpsEnvironmentId
domain: string
source: string
}): RelayOpsCellConfig[] {
const block = /relay_gce_cells\s*=\s*\{([\s\S]*)\n\}/.exec(input.source)?.[1]
if (!block) throw new Error('Relay Ops could not read durable Relay cells')
return [...block.matchAll(/"([a-z]+-gce-c\d+)"\s*=\s*\{([\s\S]*?)\n {2}\}/g)]
.map((match) => {
const cellId = match[1]!
const body = match[2]!
const hostname = required(body, /\bhostname\s*=\s*"([^"]+)"/, `${cellId} hostname`)
const configuredAdmission = /\binitially_enabled\s*=\s*(true|false)/.exec(body)?.[1]
return {
cellId,
hostname,
origin: `https://${hostname}.${input.domain}`,
region: RegionSchema.parse(
/\bregion\s*=\s*"([^"]+)"/.exec(body)?.[1] ?? 'us-central1'
),
zone: required(body, /\bzone\s*=\s*"([^"]+)"/, `${cellId} zone`),
machineType: MachineTypeSchema.parse(
required(body, /\bmachine_type\s*=\s*"([^"]+)"/, `${cellId} machine type`)
),
capacityRequests: Number(
required(body, /\bcapacity_requests\s*=\s*(\d+)/, `${cellId} capacity`)
),
databasePoolMax: Number(/\bdatabase_pool_max\s*=\s*(\d+)/.exec(body)?.[1] ?? 10),
configuredAdmission: configuredAdmission === undefined || configuredAdmission === 'true'
}
})
.sort((left, right) => cellOrdinal(left.cellId) - cellOrdinal(right.cellId))
}
function durableCells(environment: RelayOpsEnvironmentId, domain: string): RelayOpsCellConfig[] {
const source = readFileSync(
// Repository root; infra/terraform moves with this tree, so the relative depth holds.
new URL(`../../../infra/terraform/environments/${environment}.tfvars`, import.meta.url),
'utf8'
)
return relayOpsCellsFromTerraform({ environment, domain, source })
}
export const RELAY_OPS_ENVIRONMENTS: Record<RelayOpsEnvironmentId, RelayOpsEnvironment> = {
production: {
id: 'production',
label: 'Production',
project: 'onorca-cloud',
region: 'us-central1',
directorOrigin: 'https://relay.onorca.dev',
authOrigin: 'https://login.onorca.dev',
directorService: 'orca-cloud-relay',
authService: 'orca-cloud-auth',
sqlInstance: 'orca-cloud-auth-db',
migPrefix: 'orca-cloud-relay-gce-',
certificateName: 'orca-cloud-relay-gce',
cells: durableCells('production', 'relay.onorca.dev')
},
staging: {
id: 'staging',
label: 'Staging',
project: 'onorca-cloud-staging',
region: 'us-central1',
directorOrigin: 'https://relay-staging.onorca.dev',
authOrigin: 'https://auth-staging.onorca.dev',
directorService: 'orca-cloud-relay-staging',
authService: 'orca-cloud-auth-staging',
sqlInstance: 'orca-cloud-staging-auth-db',
migPrefix: 'orca-cloud-staging-relay-gce-',
certificateName: 'orca-cloud-staging-relay-gce',
cells: durableCells('staging', 'relay-staging.onorca.dev')
}
}
export function relayOpsEnvironment(value: unknown): RelayOpsEnvironment {
return RELAY_OPS_ENVIRONMENTS[EnvironmentSchema.parse(value)]
}
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest'
import { createGcloudClient } from './gcloud-client.js'
describe('createGcloudClient', () => {
it('shares and caches one credential refresh across concurrent readers', async () => {
let calls = 0
const token = 'a'.repeat(40)
const client = createGcloudClient(async () => {
calls += 1
await Promise.resolve()
return token
})
const values = await Promise.all([
client.accessToken(),
client.accessToken(),
client.accessToken()
])
expect(values).toEqual([token, token, token])
expect(await client.accessToken()).toBe(token)
expect(calls).toBe(1)
})
it('caches bounded identity tokens by audience', async () => {
const commands: string[][] = []
const token = 'aaa.bbb.ccc'
const client = createGcloudClient(async (args) => {
commands.push(args)
return token
})
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
await expect(client.identityToken?.('https://relay.example/admin')).resolves.toBe(token)
expect(commands).toEqual([
[
'auth',
'print-identity-token',
'--audiences=https://relay.example/admin',
'--include-email'
]
])
})
})
+77
View File
@@ -0,0 +1,77 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
const execFileAsync = promisify(execFile)
const TOKEN_PATTERN = /^[A-Za-z0-9._~+\/-]{32,8192}$/
export type GcloudClient = {
accessToken(): Promise<string>
identityToken?(audience: string): Promise<string>
}
export class GcloudCommandError extends Error {
constructor(readonly operation: string) {
super(`${operation} is unavailable`)
}
}
async function runGcloud(args: string[]): Promise<string> {
try {
const result = await execFileAsync('gcloud', args, {
encoding: 'utf8',
timeout: 90_000,
maxBuffer: 8 * 1024 * 1024,
env: {
...process.env,
CLOUDSDK_COMPONENT_MANAGER_DISABLE_UPDATE_CHECK: '1',
CLOUDSDK_CORE_DISABLE_PROMPTS: '1',
CLOUDSDK_CORE_DISABLE_USAGE_REPORTING: '1'
}
})
return result.stdout.trim()
} catch {
// Gcloud stderr can echo command context; keep dashboard errors intentionally non-sensitive.
throw new GcloudCommandError(`gcloud ${args.slice(0, 3).join(' ')}`)
}
}
export function createGcloudClient(
tokenCommand: (args: string[]) => Promise<string> = runGcloud
): GcloudClient {
let cachedToken: { value: string; expiresAt: number } | null = null
const identityTokens = new Map<string, { value: string; expiresAt: number }>()
let pendingToken: Promise<string> | null = null
return {
async accessToken(): Promise<string> {
if (cachedToken && cachedToken.expiresAt > Date.now()) return cachedToken.value
if (pendingToken) return await pendingToken
// One refresh avoids concurrent gcloud processes contending on the local credential store.
pendingToken = tokenCommand(['auth', 'print-access-token'])
.then((token) => {
if (!TOKEN_PATTERN.test(token)) throw new GcloudCommandError('gcloud access token')
cachedToken = { value: token, expiresAt: Date.now() + 5 * 60_000 }
return token
})
.finally(() => { pendingToken = null })
return await pendingToken
},
async identityToken(audience: string): Promise<string> {
const cached = identityTokens.get(audience)
if (cached && cached.expiresAt > Date.now()) return cached.value
const token = await tokenCommand([
'auth',
'print-identity-token',
`--audiences=${audience}`,
'--include-email'
])
if (
token.length > 8_192 ||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(token)
) {
throw new GcloudCommandError('gcloud identity token')
}
identityTokens.set(audience, { value: token, expiresAt: Date.now() + 5 * 60_000 })
return token
}
}
}
+77
View File
@@ -0,0 +1,77 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { z } from 'zod'
import { relayRepositoryApiPath } from './relay-repository.js'
const execFileAsync = promisify(execFile)
const WorkflowRunSchema = z.object({
id: z.number().int().positive(),
name: z.string(),
event: z.string(),
status: z.string(),
conclusion: z.string().nullable(),
head_sha: z.string(),
html_url: z.string().url(),
created_at: z.string(),
updated_at: z.string()
})
const WorkflowRunsSchema = z.object({
workflow_runs: z.array(WorkflowRunSchema)
})
export type RelayWorkflowRun = {
id: number
name: string
status: string
conclusion: string | null
headSha: string
url: string
createdAt: string
updatedAt: string
}
const RELAY_WORKFLOW_PATTERN = /(Relay|Auth|Power)/i
export async function readRelayWorkflowRuns(): Promise<RelayWorkflowRun[]> {
let stdout: string
try {
const result = await execFileAsync(
'gh',
[
'api',
'--method',
'GET',
relayRepositoryApiPath('actions/runs'),
'-f',
'per_page=100'
],
{ encoding: 'utf8', timeout: 30_000, maxBuffer: 4 * 1024 * 1024 }
)
stdout = result.stdout
} catch {
throw new Error('GitHub workflow history is unavailable')
}
const runs = WorkflowRunsSchema.parse(JSON.parse(stdout) as unknown).workflow_runs
const counts = new Map<string, number>()
return runs
.filter((run) => {
if (!RELAY_WORKFLOW_PATTERN.test(run.name)) return false
const count = counts.get(run.name) ?? 0
if (count >= 2) return false
counts.set(run.name, count + 1)
return true
})
.slice(0, 12)
.map((run) => ({
id: run.id,
name: run.name,
status: run.status,
conclusion: run.conclusion,
headSha: run.head_sha.slice(0, 8),
url: run.html_url,
createdAt: run.created_at,
updatedAt: run.updated_at
}))
}
@@ -0,0 +1,373 @@
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import {
livePreflightGcloud,
runIncidentLivePreflight
} from './incident-live-preflight-cli.js'
import type { IncidentSample } from './incident-monitor.js'
import type { AdmissionSelector } from './incident-selector.js'
const directories: string[] = []
const now = Date.parse('2026-07-28T12:00:00.000Z')
const selector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
function stateFile(
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition' = 'strict',
overrides: Record<string, unknown> = {}
): string {
const directory = mkdtempSync(join(tmpdir(), 'relay-live-preflight-'))
directories.push(directory)
const path = join(directory, 'state.json')
const expectedSelector = migrationPolicy === 'capacity-transition'
? {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
: selector
writeFileSync(path, JSON.stringify({
schemaVersion: 4,
environment: 'production',
expectedSelector,
migrationPolicy,
recoverySourceCellId:
migrationPolicy === 'recover-forward' ? 'production-gce-c1' : null,
capacityCellId:
migrationPolicy === 'capacity-transition' ? 'production-gce-c1' : null,
preDrainDryRun: true,
startedAt: new Date(now - 17 * 60_000).toISOString(),
windowStartedAt: new Date(now - 16 * 60_000).toISOString(),
durationMinutes: 15,
intervalMs: 60_000,
sampleCount: 16,
lastSampleAt: new Date(now - 60_007).toISOString(),
frozenAt: null,
completedAt: new Date(now - 60_000).toISOString(),
...overrides
}))
return path
}
function sample(): IncidentSample {
const observedAt = new Date(now).toISOString()
const signal = (value: number) => ({ value, observedAt })
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId: 'production-gce-c1',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'existing-only'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1),
'director.ready': signal(1),
'director.latency_ms': signal(1),
'auth.health': signal(1),
'auth.ready': signal(1),
'auth.latency_ms': signal(1),
'cell.production-gce-c1.health': signal(1),
'cell.production-gce-c1.ready': signal(1),
'cell.production-gce-c1.latency_ms': signal(1)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.1),
'cloud_sql.memory': signal(0.1),
'cloud_sql.backends': signal(1),
'cloud_sql.lock_waits': signal(0),
'cloud_sql.deadlocks': signal(0),
'director.instances': signal(5),
'director.cpu': signal(0.1),
'director.memory': signal(0.1),
'director.concurrency': signal(1),
'director.errors': signal(0),
'auth.errors': signal(0)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0),
'relay.pool_wait_ms': signal(0),
'relay.postgres_retries': signal(0),
'relay.postgres_retry_exhausted': signal(0),
'cell.production-gce-c1.connections': signal(1),
'cell.production-gce-c1.queued_bytes': signal(0)
}
},
'director-admin': {
observedAt,
signals: {
'cell.production-gce-c1.admission_state': signal(0),
'cell.production-gce-c1.heartbeat_fresh': signal(1),
'cell.production-gce-c1.heartbeat_age_ms': signal(1),
'cell.production-gce-c1.migration_blocked': signal(0),
'cell.production-gce-c1.migration_target_inactive': signal(0)
}
}
}
}
}
afterEach(() => {
for (const directory of directories.splice(0)) {
rmSync(directory, { recursive: true, force: true })
}
})
describe('relay incident live preflight', () => {
it('accepts the package-manager argument separator', async () => {
await expect(runIncidentLivePreflight(
['--', '--state-file', stateFile()],
{ now: () => now, collect: async () => sample() }
)).resolves.toBeUndefined()
})
it('accepts one complete fresh green sample', async () => {
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => sample() }
)).resolves.toBeUndefined()
})
it('rejects monitor evidence beyond the 25-minute lineage bound', async () => {
const path = stateFile('strict', {
startedAt: new Date(now - 26 * 60_000 - 1).toISOString()
})
await expect(runIncidentLivePreflight(
['--state-file', path],
{ now: () => now, collect: async () => sample() }
)).rejects.toThrow('monitor evidence is incomplete or stale')
})
it('scales the evidence age bound by same-cap wave index', async () => {
const agedState = (ageMs: number) => stateFile('strict', {
startedAt: new Date(now - ageMs - 17 * 60_000).toISOString(),
windowStartedAt: new Date(now - ageMs - 16 * 60_000).toISOString(),
lastSampleAt: new Date(now - ageMs - 7).toISOString(),
completedAt: new Date(now - ageMs).toISOString()
})
const deps = { now: () => now, collect: async () => sample() }
// One predecessor cell roll (~16 min) exceeds wave 0 but fits wave 1.
const oneRollOld = agedState(17 * 60_000)
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld, '--wave-index', '0'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', oneRollOld, '--wave-index', '1'], deps
)).resolves.toBeUndefined()
// Both edges of one predecessor job timeout: 5min + 75min exactly.
await expect(runIncidentLivePreflight(
['--state-file', agedState(80 * 60_000), '--wave-index', '1'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(80 * 60_000 + 1), '--wave-index', '1'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', agedState(155 * 60_000), '--wave-index', '2'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(155 * 60_000 + 1), '--wave-index', '2'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
await expect(runIncidentLivePreflight(
['--state-file', agedState(230 * 60_000), '--wave-index', '3'], deps
)).resolves.toBeUndefined()
await expect(runIncidentLivePreflight(
['--state-file', agedState(230 * 60_000 + 1), '--wave-index', '3'], deps
)).rejects.toThrow('monitor evidence is incomplete or stale')
// The wave index is a strict single-use 0-3 argument.
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', '4'], deps
)).rejects.toThrow('usage:')
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', ''], deps
)).rejects.toThrow('usage:')
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--wave-index', '1', '--wave-index', '1'],
deps
)).rejects.toThrow('usage:')
})
it('expects the wave-adjusted live selector generation', async () => {
const agedPath = stateFile('strict', {
startedAt: new Date(now - 34 * 60_000).toISOString(),
windowStartedAt: new Date(now - 33 * 60_000).toISOString(),
lastSampleAt: new Date(now - 17 * 60_000 - 7).toISOString(),
completedAt: new Date(now - 17 * 60_000).toISOString()
})
const liveAt = (generation: number) =>
async (expectedSelector: AdmissionSelector) => ({
...sample(),
selector: { ...selector, generation },
expectedSelector
})
// One predecessor roll advanced the live selector by exactly 2.
await expect(runIncidentLivePreflight(
['--state-file', agedPath, '--wave-index', '1'],
{ now: () => now, collect: liveAt(selector.generation + 2) }
)).resolves.toBeUndefined()
// The sealed pre-roll generation must no longer satisfy wave 1.
await expect(runIncidentLivePreflight(
['--state-file', agedPath, '--wave-index', '1'],
{ now: () => now, collect: liveAt(selector.generation) }
)).rejects.toThrow('director-admin/selector_mismatch')
// Wave 0 still expects the sealed generation itself.
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: liveAt(selector.generation) }
)).resolves.toBeUndefined()
})
it('fails closed on a live threshold breach', async () => {
const unhealthy = sample()
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => unhealthy }
)).rejects.toThrow('cloud-monitoring/threshold_max')
})
it('enforces the signed migration policy', async () => {
const inactiveTarget = sample()
inactiveTarget.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
]!.value = 30
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => inactiveTarget }
)).rejects.toThrow('director-admin/threshold_max')
await expect(runIncidentLivePreflight(
['--state-file', stateFile('recover-forward')],
{ now: () => now, collect: async () => inactiveTarget }
)).resolves.toBeUndefined()
inactiveTarget.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
]!.value = 1
await expect(runIncidentLivePreflight(
['--state-file', stateFile('recover-forward')],
{ now: () => now, collect: async () => inactiveTarget }
)).rejects.toThrow('director-admin/threshold_max')
})
it('binds capacity-transition evidence to its general cell', async () => {
const capacitySample = sample()
const capacitySelector = {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
capacitySample.selector = capacitySelector
capacitySample.expectedSelector = capacitySelector
capacitySample.cells[0]!.expectedAdmissionState = 'general'
capacitySample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
]!.value = 2
await expect(runIncidentLivePreflight(
['--state-file', stateFile('capacity-transition')],
{ now: () => now, collect: async () => capacitySample }
)).resolves.toBeUndefined()
capacitySample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
]!.value = 1
await expect(runIncidentLivePreflight(
['--state-file', stateFile('capacity-transition')],
{ now: () => now, collect: async () => capacitySample }
)).rejects.toThrow('director-admin/threshold_max')
})
it('rejects stale live evidence', async () => {
const stale = sample()
stale.sources['active-probe']!.observedAt = new Date(now - 60_001).toISOString()
await expect(runIncidentLivePreflight(
['--state-file', stateFile()],
{ now: () => now, collect: async () => stale }
)).rejects.toThrow('active-probe/source_stale')
})
it('retries freshness-only failures when explicitly requested', async () => {
const stale = sample()
stale.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
new Date(now - 180_001).toISOString()
const missing = sample()
delete missing.sources['relay-logs']
const collect = vi.fn()
.mockResolvedValueOnce(stale)
.mockResolvedValueOnce(missing)
.mockResolvedValueOnce(sample())
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).resolves.toBeUndefined()
expect(collect).toHaveBeenCalledTimes(3)
expect(wait).toHaveBeenCalledTimes(2)
expect(wait).toHaveBeenNthCalledWith(1, 15_000)
expect(wait).toHaveBeenNthCalledWith(2, 15_000)
})
it('does not retry a threshold failure', async () => {
const unhealthy = sample()
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.value = 0.9
unhealthy.sources['cloud-monitoring']!.signals['cloud_sql.cpu']!.observedAt =
new Date(now - 180_001).toISOString()
const collect = vi.fn(async () => unhealthy)
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).rejects.toThrow('cloud-monitoring/threshold_max')
expect(collect).toHaveBeenCalledOnce()
expect(wait).not.toHaveBeenCalled()
})
it('fails closed after the bounded freshness retry window', async () => {
const stale = sample()
stale.sources['cloud-monitoring']!.observedAt = new Date(now - 180_001).toISOString()
const collect = vi.fn(async () => stale)
const wait = vi.fn(async () => undefined)
await expect(runIncidentLivePreflight(
['--state-file', stateFile(), '--retry-freshness'],
{ now: () => now, collect, wait }
)).rejects.toThrow('cloud-monitoring/source_stale')
expect(collect).toHaveBeenCalledTimes(5)
expect(wait).toHaveBeenCalledTimes(4)
})
it('uses the supplied admin token without minting through gcloud', async () => {
const identityToken = vi.fn(async () => 'minted.token.value')
const gcloud = livePreflightGcloud(
{ accessToken: async () => 'access-token', identityToken },
{ ORCA_RELAY_ADMIN_ID_TOKEN: 'supplied.token.value' }
)
await expect(gcloud.identityToken!('audience')).resolves.toBe(
'supplied.token.value'
)
expect(identityToken).not.toHaveBeenCalled()
})
})
@@ -0,0 +1,195 @@
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
import { z } from 'zod'
import { createGcloudClient } from './gcloud-client.js'
import { suppliedIdentityToken } from './incident-monitor-cli.js'
import { AdmissionSelectorSchema, type AdmissionSelector } from './incident-selector.js'
import {
evaluateIncidentSample,
preDrainDryRunPassed,
type IncidentSample
} from './incident-monitor.js'
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
const FRESHNESS_RETRY_ATTEMPTS = 5
const FRESHNESS_RETRY_INTERVAL_MS = 15_000
const MONITOR_EVIDENCE_MAX_AGE_MS = 5 * 60_000
// Matches the same-cap cell job timeout-minutes; bounds each predecessor wave.
const WAVE_PREDECESSOR_TIMEOUT_MS = 75 * 60_000
const WAVE_INDEX_PATTERN = /^[0-3]$/
const FRESHNESS_FAILURE_CODES = new Set([
'signal_missing',
'signal_stale',
'source_missing',
'source_stale'
])
export function livePreflightGcloud(
gcloud: ReturnType<typeof createGcloudClient>,
environment: NodeJS.ProcessEnv = process.env
): ReturnType<typeof createGcloudClient> {
const token = suppliedIdentityToken(environment.ORCA_RELAY_ADMIN_ID_TOKEN)
return token ? { ...gcloud, identityToken: async () => token } : gcloud
}
const PreflightStateSchema = z.object({
schemaVersion: z.literal(4),
environment: z.literal('production'),
expectedSelector: AdmissionSelectorSchema,
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
recoverySourceCellId: z.string().nullable(),
capacityCellId: z.string().nullable(),
preDrainDryRun: z.literal(true),
startedAt: z.string(),
windowStartedAt: z.string(),
durationMinutes: z.literal(15),
intervalMs: z.literal(60_000),
sampleCount: z.number().int().min(16),
lastSampleAt: z.string(),
frozenAt: z.null(),
completedAt: z.string()
}).superRefine((state, context) => {
const validRecovery =
state.migrationPolicy === 'recover-forward' &&
state.capacityCellId === null &&
state.recoverySourceCellId !== null &&
state.expectedSelector.membership.existingOnly.includes(
state.recoverySourceCellId
)
const validStrict =
state.migrationPolicy === 'strict' &&
state.recoverySourceCellId === null &&
state.capacityCellId === null
const validCapacity =
state.migrationPolicy === 'capacity-transition' &&
state.recoverySourceCellId === null &&
state.capacityCellId !== null &&
state.expectedSelector.membership.general.includes(state.capacityCellId)
if (!validRecovery && !validStrict && !validCapacity) {
context.addIssue({
code: 'custom',
message: 'relay live preflight migration policy is invalid'
})
}
})
export async function runIncidentLivePreflight(
argv: string[],
dependencies: {
now?: () => number
wait?: (ms: number) => Promise<void>
collect?: (expectedSelector: AdmissionSelector) => Promise<IncidentSample>
gcloud?: ReturnType<typeof createGcloudClient>
environment?: NodeJS.ProcessEnv
} = {}
): Promise<void> {
const args = argv[0] === '--' ? argv.slice(1) : argv
const freshnessRetryCount = args.filter((arg) => arg === '--retry-freshness').length
const rest = args.filter((arg) => arg !== '--retry-freshness')
const stateArgs: string[] = []
let waveIndex = '0'
let waveIndexCount = 0
for (let index = 0; index < rest.length; index += 1) {
if (rest[index] === '--wave-index') {
waveIndexCount += 1
waveIndex = rest[index + 1] ?? ''
index += 1
} else {
stateArgs.push(rest[index] as string)
}
}
if (
freshnessRetryCount > 1 ||
waveIndexCount > 1 ||
!WAVE_INDEX_PATTERN.test(waveIndex) ||
stateArgs.length !== 2 ||
stateArgs[0] !== '--state-file' ||
!stateArgs[1]
) {
throw new Error(
'usage: --state-file <verified-monitor-state> [--wave-index <0-3>] [--retry-freshness]'
)
}
const state = PreflightStateSchema.parse(
JSON.parse(await readFile(resolve(stateArgs[1]), 'utf8'))
)
const now = dependencies.now ?? Date.now
const completedAt = Date.parse(state.completedAt)
const windowStartedAt = Date.parse(state.windowStartedAt)
const lastSampleAt = Date.parse(state.lastSampleAt)
const evidenceAgeMs = now() - completedAt
// Later same-cap waves start after sequential predecessor cell rolls, so the
// freshness bound grows by one cell-job timeout per predecessor; the live
// samples collected below still hold every wave to current health.
const maxEvidenceAgeMs =
MONITOR_EVIDENCE_MAX_AGE_MS + Number(waveIndex) * WAVE_PREDECESSOR_TIMEOUT_MS
if (
!preDrainDryRunPassed(state) ||
!Number.isFinite(windowStartedAt) ||
completedAt - windowStartedAt < 15 * 60_000 ||
!Number.isFinite(lastSampleAt) ||
lastSampleAt > completedAt ||
completedAt - lastSampleAt > state.intervalMs ||
!Number.isFinite(completedAt) ||
evidenceAgeMs < 0 ||
evidenceAgeMs > maxEvidenceAgeMs
) {
throw new Error('relay live preflight monitor evidence is incomplete or stale')
}
const gcloud = livePreflightGcloud(
dependencies.gcloud ?? createGcloudClient(),
dependencies.environment
)
// Each predecessor same-cap apply wave reversibly isolates and restores its
// cell, advancing the selector generation by exactly 2 with membership
// unchanged (rollback is single-cell, so it never reaches a later wave), so
// the live selector comparison must expect the wave-adjusted generation.
const collectOptions = {
environment: state.environment,
expectedSelector: {
...state.expectedSelector,
generation: state.expectedSelector.generation + 2 * Number(waveIndex)
},
...(dependencies.now ? { now: dependencies.now } : {})
}
const injected = dependencies.collect
const collect = injected
? () => injected(collectOptions.expectedSelector)
: createIncidentSampleCollector(gcloud, collectOptions)
const wait = dependencies.wait ?? ((ms: number) => new Promise<void>((resolveWait) => {
setTimeout(resolveWait, ms)
}))
const attempts = freshnessRetryCount === 1 ? FRESHNESS_RETRY_ATTEMPTS : 1
for (let attempt = 1; attempt <= attempts; attempt++) {
const evaluation = evaluateIncidentSample(
await collect(),
now(),
state.migrationPolicy,
state.recoverySourceCellId,
state.capacityCellId
)
if (evaluation.status === 'green') return
const freshnessOnly = evaluation.failures.every((failure) =>
FRESHNESS_FAILURE_CODES.has(failure.code)
)
if (!freshnessOnly || attempt === attempts) {
throw new Error(
`relay live preflight failed: ${evaluation.failures
.map((failure) => `${failure.source}/${failure.code}`)
.join(',')}`
)
}
console.warn(
`relay live preflight awaiting fresh evidence (${attempt}/${attempts - 1})`
)
await wait(FRESHNESS_RETRY_INTERVAL_MS)
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
runIncidentLivePreflight(process.argv.slice(2)).catch((error: unknown) => {
console.error(error instanceof Error ? error.message : 'relay live preflight failed')
process.exitCode = 1
})
}
@@ -0,0 +1,454 @@
import { mkdtempSync, readFileSync, rmSync, statSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import {
parseIncidentMonitorArguments,
runIncidentMonitorCli
} from './incident-monitor-cli.js'
import type { IncidentSample } from './incident-monitor.js'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
const directories: string[] = []
const startedAt = Date.parse('2026-07-28T00:00:00.000Z')
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map((cell) => cell.cellId)
const selector = {
generation: 1,
membership: {
existingOnly: productionCells.slice(1),
migrationOnly: [],
general: [productionCells[0]!]
}
}
const selectorArguments = [
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
productionCells[0]!
]
const signal = (value: number, at: number) => ({
value,
observedAt: new Date(at).toISOString()
})
function sample(at: number): IncidentSample {
const observedAt = new Date(at).toISOString()
const cellId = 'production-gce-c1'
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId,
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1, at),
'director.ready': signal(1, at),
'director.latency_ms': signal(1, at),
'auth.health': signal(1, at),
'auth.ready': signal(1, at),
'auth.latency_ms': signal(1, at),
[`cell.${cellId}.health`]: signal(1, at),
[`cell.${cellId}.ready`]: signal(1, at),
[`cell.${cellId}.latency_ms`]: signal(1, at)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.1, at),
'cloud_sql.memory': signal(0.1, at),
'cloud_sql.backends': signal(1, at),
'cloud_sql.lock_waits': signal(0, at),
'cloud_sql.deadlocks': signal(0, at),
'director.instances': signal(5, at),
'director.cpu': signal(0.1, at),
'director.memory': signal(0.1, at),
'director.concurrency': signal(1, at),
'director.errors': signal(0, at),
'auth.errors': signal(0, at)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0, at),
'relay.pool_wait_ms': signal(0, at),
'relay.postgres_retries': signal(0, at),
'relay.postgres_retry_exhausted': signal(0, at),
[`cell.${cellId}.connections`]: signal(1, at),
[`cell.${cellId}.queued_bytes`]: signal(0, at)
}
},
'director-admin': {
observedAt,
signals: {
[`cell.${cellId}.admission_state`]: signal(2, at),
[`cell.${cellId}.heartbeat_fresh`]: signal(1, at),
[`cell.${cellId}.heartbeat_age_ms`]: signal(1, at),
[`cell.${cellId}.migration_blocked`]: signal(0, at),
[`cell.${cellId}.migration_target_inactive`]: signal(0, at)
}
}
}
}
}
afterEach(() => {
for (const directory of directories.splice(0)) {
rmSync(directory, { recursive: true, force: true })
}
})
describe('incident monitor CLI', () => {
it('requires an exact selector and rejects invalid membership', () => {
expect(() => parseIncidentMonitorArguments([])).toThrow(
'--expected-selector-generation'
)
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
'production-gce-c99'
])
).toThrow('every configured cell exactly once')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--interval-seconds',
'61'
])
).toThrow('between 1 and 60')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'14'
])
).toThrow('between 15 and 90')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
'--expected-selector-generation',
'0',
'--expected-existing-only-cells',
productionCells.slice(1).join(','),
'--expected-migration-only-cells',
productionCells[0]!,
'--expected-general-cells',
'none'
])
).toThrow('generation 0 cannot represent migration-only')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'recover-forward',
'--pre-drain-dry-run'
])
).toThrow('requires an existing-only --recovery-source-cell-id')
expect(() =>
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'capacity-transition',
'--pre-drain-dry-run'
])
).toThrow('requires a general --capacity-cell-id')
expect(
parseIncidentMonitorArguments([
'--incident-id',
'incident-1',
...selectorArguments,
'--migration-policy',
'capacity-transition',
'--capacity-cell-id',
productionCells[0]!,
'--pre-drain-dry-run'
])
).toMatchObject({
migrationPolicy: 'capacity-transition',
capacityCellId: productionCells[0]!,
recoverySourceCellId: null
})
})
it('writes private durable checkpoints for a green pre-drain dry run', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const output: string[] = []
const code = await runIncidentMonitorCli(
[
'--incident-id',
'incident-1',
...selectorArguments,
'--pre-drain-dry-run',
'--output-directory',
directory
],
{
cwd: directory,
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => sample(now),
writeOutput: (value) => output.push(value)
}
)
expect(code).toBe(0)
const statePath = join(directory, 'incident-1.state.json')
const summaryPath = join(directory, 'incident-1.summaries.jsonl')
const markdownPath = join(directory, 'incident-1.summary.md')
expect(statSync(statePath).mode & 0o077).toBe(0)
expect(statSync(summaryPath).mode & 0o077).toBe(0)
expect(statSync(markdownPath).mode & 0o077).toBe(0)
const summaries = readFileSync(summaryPath, 'utf8')
.trim()
.split('\n')
.map((line) => JSON.parse(line))
expect(summaries.map((entry) => entry.checkpointMinute)).toEqual([0, 5, 15])
expect(output.join('')).not.toContain('token')
expect(readFileSync(markdownPath, 'utf8')).toContain(
'| 0 | 15 | green | 16 | none |'
)
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
frozenAt: null,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
sampleCount: 16
})
})
it('runs a recovery dry run without masking blocked migrations', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const recoverySelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: productionCells.slice(1)
}
}
const recoverySample = (): IncidentSample => {
const current = sample(now)
current.selector = recoverySelector
current.expectedSelector = recoverySelector
current.cells[0]!.expectedAdmissionState = 'existing-only'
current.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0, now)
current.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(30, now)
return current
}
const args = [
'--incident-id',
'incident-1',
'--expected-selector-generation',
'1',
'--expected-existing-only-cells',
'production-gce-c1',
'--expected-migration-only-cells',
'none',
'--expected-general-cells',
productionCells.slice(1).join(','),
'--pre-drain-dry-run',
'--migration-policy',
'recover-forward',
'--recovery-source-cell-id',
'production-gce-c1',
'--output-directory',
directory
]
const dependencies = {
cwd: directory,
now: () => now,
wait: async (ms: number) => {
now += ms
},
collect: async () => recoverySample(),
writeOutput: () => {}
}
await expect(runIncidentMonitorCli(args, dependencies)).resolves.toBe(0)
expect(
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
).toMatchObject({
migrationPolicy: 'recover-forward',
recoverySourceCellId: 'production-gce-c1',
capacityCellId: null,
frozenAt: null
})
})
it('fails a frozen pre-drain gate after its first sample', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
let collections = 0
let waits = 0
const code = await runIncidentMonitorCli(
[
'--incident-id',
'incident-1',
...selectorArguments,
'--pre-drain-dry-run',
'--output-directory',
directory
],
{
cwd: directory,
now: () => now,
wait: async (ms) => {
waits++
now += ms
},
collect: async () => {
collections++
const unhealthy = sample(now)
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] =
signal(801, now)
return unhealthy
},
writeOutput: () => {}
}
)
expect(code).toBe(2)
expect(collections).toBe(1)
expect(waits).toBe(0)
expect(
JSON.parse(readFileSync(join(directory, 'incident-1.state.json'), 'utf8'))
).toMatchObject({
completedAt: new Date(startedAt).toISOString(),
frozenAt: new Date(startedAt).toISOString(),
sampleCount: 1
})
})
it('requires --restart and preserves a latched freeze', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const args = [
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'15',
'--output-directory',
directory
]
await runIncidentMonitorCli(args, {
cwd: directory,
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => {
const unhealthy = sample(now)
unhealthy.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(801, now)
return unhealthy
},
writeOutput: () => {}
})
await expect(
runIncidentMonitorCli(args, {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).rejects.toThrow('pass --restart')
const changedAdmission = [...args]
changedAdmission[changedAdmission.indexOf('--expected-selector-generation') + 1] = '2'
await expect(
runIncidentMonitorCli([...changedAdmission, '--restart'], {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).rejects.toThrow('do not match')
await expect(
runIncidentMonitorCli([...args, '--restart'], {
cwd: directory,
now: () => now,
collect: async () => sample(now),
writeOutput: () => {}
})
).resolves.toBe(2)
})
it('resumes a gracefully segmented monitor without resetting continuity', async () => {
const directory = mkdtempSync(join(tmpdir(), 'relay-incident-cli-'))
directories.push(directory)
let now = startedAt
const args = [
'--incident-id',
'incident-1',
...selectorArguments,
'--duration-minutes',
'15',
'--output-directory',
directory
]
const dependencies = {
cwd: directory,
now: () => now,
wait: async (ms: number) => {
now += ms
},
collect: async () => sample(now),
writeOutput: () => {}
}
await expect(
runIncidentMonitorCli([...args, '--max-samples-this-run', '2'], dependencies)
).resolves.toBe(0)
const statePath = join(directory, 'incident-1.state.json')
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: null,
sampleCount: 2,
lastSampleAt: new Date(startedAt + 60_000).toISOString()
})
await expect(
runIncidentMonitorCli([...args, '--restart'], dependencies)
).resolves.toBe(0)
expect(JSON.parse(readFileSync(statePath, 'utf8'))).toMatchObject({
completedAt: new Date(startedAt + 15 * 60_000).toISOString(),
windowSequence: 0,
sampleCount: 17
})
})
})
@@ -0,0 +1,493 @@
import { randomUUID } from 'node:crypto'
import {
appendFile,
chmod,
mkdir,
open,
readFile,
rename,
stat,
writeFile
} from 'node:fs/promises'
import { dirname, resolve } from 'node:path'
import { pathToFileURL } from 'node:url'
import { z } from 'zod'
import { relayOpsEnvironment } from './environment-config.js'
import { createGcloudClient } from './gcloud-client.js'
import {
AdmissionSelectorSchema,
normalizeSelectorMembership,
type AdmissionSelector
} from './incident-selector.js'
import {
initialIncidentMonitorState,
preDrainDryRunPassed,
runIncidentMonitor,
type IncidentCheckpoint,
type IncidentSample,
type IncidentMonitorState
} from './incident-monitor.js'
import { createIncidentSampleCollector } from './incident-monitor-sources.js'
const StateSchema = z.object({
schemaVersion: z.literal(4),
incidentId: z.string(),
environment: z.enum(['production', 'staging']),
expectedSelector: AdmissionSelectorSchema,
preDrainDryRun: z.boolean(),
migrationPolicy: z.enum(['strict', 'recover-forward', 'capacity-transition']),
recoverySourceCellId: z.string().nullable(),
capacityCellId: z.string().nullable(),
startedAt: z.string(),
windowStartedAt: z.string().nullable(),
windowSequence: z.number().int().nonnegative(),
durationMinutes: z.number().int(),
intervalMs: z.number().int(),
nextCheckpointIndex: z.number().int().nonnegative(),
sampleCount: z.number().int().nonnegative(),
totalSampleCount: z.number().int().nonnegative(),
lastSampleAt: z.string().nullable(),
continuityEvents: z.array(z.object({
recordedAt: z.string(),
windowSequence: z.number().int().nonnegative(),
failures: z.array(z.object({
code: z.string(),
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
signal: z.string().optional(),
observed: z.number().optional(),
threshold: z.number().optional()
}))
})),
frozenAt: z.string().nullable(),
failures: z.array(z.object({
code: z.string(),
source: z.enum(['active-probe', 'cloud-monitoring', 'relay-logs', 'director-admin']),
signal: z.string().optional(),
observed: z.number().optional(),
threshold: z.number().optional()
})),
completedAt: z.string().nullable()
})
type CliOptions = {
environment: 'production' | 'staging'
incidentId: string
durationMinutes: number
intervalMs: number
expectedSelector: AdmissionSelector
stateFile: string
summaryFile: string
markdownFile: string
restart: boolean
preDrainDryRun: boolean
migrationPolicy: 'strict' | 'recover-forward' | 'capacity-transition'
recoverySourceCellId: string | null
capacityCellId: string | null
maxSamplesThisRun: number | null
}
function parsePositiveInteger(value: string | undefined, name: string): number {
const parsed = Number(value)
if (!Number.isSafeInteger(parsed) || parsed <= 0) {
throw new Error(`${name} must be a positive integer`)
}
return parsed
}
export function parseIncidentMonitorArguments(argv: string[], cwd = process.cwd()): CliOptions {
const flags = new Set(['restart', 'pre-drain-dry-run'])
const valueArguments = new Set([
'duration-minutes',
'environment',
'expected-selector-generation',
'expected-existing-only-cells',
'expected-migration-only-cells',
'expected-general-cells',
'incident-id',
'interval-seconds',
'max-samples-this-run',
'migration-policy',
'output-directory',
'recovery-source-cell-id',
'capacity-cell-id'
])
const values: Record<string, string> = {}
const enabledFlags = new Set<string>()
for (let index = 0; index < argv.length; index++) {
const argument = argv[index]
if (!argument?.startsWith('--')) throw new Error(`invalid argument ${argument ?? ''}`)
const name = argument.slice(2)
if (flags.has(name)) {
enabledFlags.add(name)
continue
}
if (!valueArguments.has(name)) throw new Error(`unknown argument --${name}`)
const value = argv[++index]
if (!value || value.startsWith('--')) throw new Error(`missing --${name} value`)
values[name] = value
}
const environment = z.enum(['production', 'staging']).parse(
values.environment ?? 'production'
)
const incidentId = values['incident-id'] ?? randomUUID()
if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]{7,127}$/.test(incidentId)) {
throw new Error('--incident-id must be 8-128 safe characters')
}
const selectorGeneration = Number(values['expected-selector-generation'])
if (!Number.isSafeInteger(selectorGeneration) || selectorGeneration < 0) {
throw new Error('--expected-selector-generation must be a nonnegative integer')
}
const configuredCells = new Set(
relayOpsEnvironment(environment).cells.map((cell) => cell.cellId)
)
const cellList = (name: string): string[] => {
const value = values[name]
if (value === undefined) throw new Error(`--${name} is required; use none for an empty set`)
return value === 'none' ? [] : value.split(',').map((cellId) => cellId.trim())
}
const expectedSelector = {
generation: selectorGeneration,
membership: normalizeSelectorMembership(
{
existingOnly: cellList('expected-existing-only-cells'),
migrationOnly: cellList('expected-migration-only-cells'),
general: cellList('expected-general-cells')
},
configuredCells
)
}
if (selectorGeneration === 0 && expectedSelector.membership.migrationOnly.length > 0) {
throw new Error('generation 0 cannot represent migration-only admission')
}
const preDrainDryRun = enabledFlags.has('pre-drain-dry-run')
const migrationPolicy = z.enum([
'strict',
'recover-forward',
'capacity-transition'
]).parse(
values['migration-policy'] ?? 'strict'
)
const recoverySourceCellId =
values['recovery-source-cell-id'] === undefined ||
values['recovery-source-cell-id'] === 'none'
? null
: values['recovery-source-cell-id']
const capacityCellId =
values['capacity-cell-id'] === undefined || values['capacity-cell-id'] === 'none'
? null
: values['capacity-cell-id']
const durationMinutes = parsePositiveInteger(
values['duration-minutes'] ?? (preDrainDryRun ? '15' : '90'),
'--duration-minutes'
)
const intervalMs =
parsePositiveInteger(values['interval-seconds'] ?? '60', '--interval-seconds') * 1_000
if (durationMinutes < 15 || durationMinutes > 90) {
throw new Error('--duration-minutes must be between 15 and 90')
}
if (intervalMs > 60_000) {
throw new Error('--interval-seconds must be between 1 and 60')
}
if (preDrainDryRun && durationMinutes !== 15) {
throw new Error('--pre-drain-dry-run requires --duration-minutes 15')
}
if (migrationPolicy !== 'strict' && !preDrainDryRun) {
throw new Error(`--migration-policy ${migrationPolicy} requires --pre-drain-dry-run`)
}
if (
migrationPolicy === 'recover-forward' &&
(
recoverySourceCellId === null ||
!configuredCells.has(recoverySourceCellId) ||
!expectedSelector.membership.existingOnly.includes(recoverySourceCellId)
)
) {
throw new Error(
'--migration-policy recover-forward requires an existing-only --recovery-source-cell-id'
)
}
if (migrationPolicy !== 'recover-forward' && recoverySourceCellId !== null) {
throw new Error('--recovery-source-cell-id requires --migration-policy recover-forward')
}
if (
migrationPolicy === 'capacity-transition' &&
(
capacityCellId === null ||
!configuredCells.has(capacityCellId) ||
!expectedSelector.membership.general.includes(capacityCellId)
)
) {
throw new Error(
'--migration-policy capacity-transition requires a general --capacity-cell-id'
)
}
if (migrationPolicy !== 'capacity-transition' && capacityCellId !== null) {
throw new Error('--capacity-cell-id requires --migration-policy capacity-transition')
}
const directory = resolve(cwd, values['output-directory'] ?? '.relay-incidents')
const maxSamplesThisRun = values['max-samples-this-run']
? parsePositiveInteger(values['max-samples-this-run'], '--max-samples-this-run')
: null
return {
environment,
incidentId,
durationMinutes,
intervalMs,
expectedSelector,
stateFile: resolve(directory, `${incidentId}.state.json`),
summaryFile: resolve(directory, `${incidentId}.summaries.jsonl`),
markdownFile: resolve(directory, `${incidentId}.summary.md`),
restart: enabledFlags.has('restart'),
preDrainDryRun,
migrationPolicy,
recoverySourceCellId,
capacityCellId,
maxSamplesThisRun
}
}
async function fileExists(path: string): Promise<boolean> {
try {
await stat(path)
return true
} catch {
return false
}
}
async function syncFile(path: string): Promise<void> {
const handle = await open(path, 'r')
try {
await handle.sync()
} finally {
await handle.close()
}
}
async function persistState(path: string, state: IncidentMonitorState): Promise<void> {
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
await chmod(dirname(path), 0o700)
const temporaryPath = `${path}.tmp`
await writeFile(temporaryPath, `${JSON.stringify(state)}\n`, { mode: 0o600 })
await chmod(temporaryPath, 0o600)
await syncFile(temporaryPath)
await rename(temporaryPath, path)
await syncFile(path)
}
async function appendCheckpoint(path: string, checkpoint: IncidentCheckpoint): Promise<void> {
await mkdir(dirname(path), { recursive: true, mode: 0o700 })
await chmod(dirname(path), 0o700)
if (await fileExists(path)) {
const existing = (await readFile(path, 'utf8'))
.trim()
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line) as IncidentCheckpoint)
if (
existing.some((entry) =>
entry.windowSequence === checkpoint.windowSequence &&
entry.checkpointMinute === checkpoint.checkpointMinute
)
) return
}
await appendFile(path, `${JSON.stringify(checkpoint)}\n`, { mode: 0o600 })
await chmod(path, 0o600)
await syncFile(path)
}
function markdownFailure(checkpoint: IncidentCheckpoint): string {
if (checkpoint.failures.length === 0) return 'none'
return checkpoint.failures
.map((failure) => {
const signal = failure.signal ? `/${failure.signal}` : ''
return `${failure.source}/${failure.code}${signal}`
})
.join(', ')
}
async function writeMarkdownSummary(
path: string,
summaryPath: string,
incidentId: string,
environment: string
): Promise<void> {
const checkpoints = (await readFile(summaryPath, 'utf8'))
.trim()
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line) as IncidentCheckpoint)
const rows = checkpoints.map((checkpoint) => [
`| ${checkpoint.windowSequence}`,
checkpoint.checkpointMinute,
checkpoint.status,
checkpoint.sampleCount,
`${markdownFailure(checkpoint)} |`
].join(' | '))
const markdown = [
'# Relay incident monitor',
'',
`Incident: \`${incidentId}\``,
'',
`Environment: \`${environment}\``,
'',
`Expected selector: \`${JSON.stringify(checkpoints[0]?.expectedSelector ?? null)}\``,
'',
`Migration policy: \`${checkpoints[0]?.migrationPolicy ?? 'unknown'}\``,
'',
`Recovery source: \`${checkpoints[0]?.recoverySourceCellId ?? 'none'}\``,
'',
`Capacity cell: \`${checkpoints[0]?.capacityCellId ?? 'none'}\``,
'',
'| Window | Minute | Status | Samples | Failures |',
'| ---: | ---: | --- | ---: | --- |',
...rows,
''
].join('\n')
const temporaryPath = `${path}.tmp`
await writeFile(temporaryPath, markdown, { mode: 0o600 })
await chmod(temporaryPath, 0o600)
await syncFile(temporaryPath)
await rename(temporaryPath, path)
await syncFile(path)
}
export function suppliedIdentityToken(value: string | undefined): string | null {
if (value === undefined) return null
if (
value.length > 8_192 ||
!/^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/.test(value)
) {
throw new Error('ORCA_RELAY_ADMIN_ID_TOKEN is invalid')
}
return value
}
async function readInitialState(
options: CliOptions,
now: () => number = Date.now
): Promise<IncidentMonitorState> {
const exists = await fileExists(options.stateFile)
if (exists && !options.restart) {
throw new Error('incident state already exists; pass --restart to resume it')
}
if (!exists && options.restart) throw new Error('no incident state exists to restart')
if (!exists) {
return initialIncidentMonitorState({
incidentId: options.incidentId,
environment: options.environment,
expectedSelector: options.expectedSelector,
preDrainDryRun: options.preDrainDryRun,
migrationPolicy: options.migrationPolicy,
recoverySourceCellId: options.recoverySourceCellId,
capacityCellId: options.capacityCellId,
startedAt: new Date(now()).toISOString(),
durationMinutes: options.durationMinutes,
intervalMs: options.intervalMs
})
}
const state = StateSchema.parse(
JSON.parse(await readFile(options.stateFile, 'utf8'))
) as IncidentMonitorState
if (
state.incidentId !== options.incidentId ||
state.environment !== options.environment ||
JSON.stringify(state.expectedSelector) !== JSON.stringify(options.expectedSelector) ||
state.preDrainDryRun !== options.preDrainDryRun ||
state.migrationPolicy !== options.migrationPolicy ||
state.recoverySourceCellId !== options.recoverySourceCellId ||
state.capacityCellId !== options.capacityCellId ||
state.durationMinutes !== options.durationMinutes ||
state.intervalMs !== options.intervalMs
) {
throw new Error('restart arguments do not match durable incident state')
}
return state
}
export async function runIncidentMonitorCli(
argv: string[],
dependencies: {
cwd?: string
now?: () => number
wait?: (ms: number) => Promise<void>
gcloud?: ReturnType<typeof createGcloudClient>
collect?: () => Promise<IncidentSample>
writeOutput?: (value: string) => void
environment?: NodeJS.ProcessEnv
} = {}
): Promise<number> {
const options = parseIncidentMonitorArguments(argv, dependencies.cwd)
const state = await readInitialState(options, dependencies.now)
const baseGcloud = dependencies.gcloud ?? createGcloudClient()
const token = suppliedIdentityToken(
(dependencies.environment ?? process.env).ORCA_RELAY_ADMIN_ID_TOKEN
)
await persistState(options.stateFile, state)
const gcloud = token
? { ...baseGcloud, identityToken: async () => token }
: baseGcloud
const collect =
dependencies.collect ??
createIncidentSampleCollector(gcloud, {
environment: options.environment,
expectedSelector: options.expectedSelector,
...(dependencies.now ? { now: dependencies.now } : {})
})
let samplesThisRun = 0
const segmentedCollect = async (): Promise<IncidentSample> => {
try {
return await collect()
} finally {
samplesThisRun++
}
}
const wait =
dependencies.wait ??
((ms: number) => new Promise((resolvePromise) => setTimeout(resolvePromise, ms)))
const segmentComplete = Symbol('segment-complete')
const output = dependencies.writeOutput ?? ((value) => process.stdout.write(`${value}\n`))
let result: IncidentMonitorState
try {
result = await runIncidentMonitor(state, {
now: dependencies.now ?? Date.now,
wait: async (ms) => {
if (
options.maxSamplesThisRun !== null &&
samplesThisRun >= options.maxSamplesThisRun
) {
throw segmentComplete
}
await wait(ms)
},
collect: segmentedCollect,
persist: async (nextState) => await persistState(options.stateFile, nextState),
checkpoint: async (checkpoint) => {
await appendCheckpoint(options.summaryFile, checkpoint)
await writeMarkdownSummary(
options.markdownFile,
options.summaryFile,
options.incidentId,
options.environment
)
output(JSON.stringify(checkpoint))
}
})
} catch (error) {
if (error !== segmentComplete) throw error
return 0
}
if (options.preDrainDryRun && !preDrainDryRunPassed(result)) return 2
return result.frozenAt ? 2 : 0
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
runIncidentMonitorCli(process.argv.slice(2))
.then((code) => {
process.exitCode = code
})
.catch((error: unknown) => {
console.error(error instanceof Error ? error.message : 'incident monitor failed')
process.exitCode = 1
})
}
@@ -0,0 +1,418 @@
import { describe, expect, it } from 'vitest'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { effectiveAdmissionState } from './incident-selector.js'
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
import {
directorSignals,
GOOGLE_METRICS,
readGoogleMetric,
readGoogleMetricWithEmptyRetry,
relayFiveMinuteDeltaSignal
} from './incident-monitor-sources.js'
const now = Date.parse('2026-07-28T10:00:00.000Z')
const startAt = new Date(now - 5 * 60_000).toISOString()
const endAt = new Date(now).toISOString()
const productionCells = RELAY_OPS_ENVIRONMENTS.production.cells.map(
({ cellId }) => cellId
)
describe('incident monitor sources', () => {
it('uses legacy booleans only at selector generation zero', () => {
const membership = {
existingOnly: ['c1'],
migrationOnly: [],
general: ['c2']
}
expect(
effectiveAdmissionState({ generation: 0, membership }, true, 'c1')
).toBe('general')
expect(
effectiveAdmissionState({ generation: 1, membership }, true, 'c1')
).toBe('existing-only')
})
it('sums DELTA metrics across the window and scopes every target exactly', async () => {
const filters: string[] = []
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
filters.push(url.searchParams.get('filter') ?? '')
return Response.json({
timeSeries: [
{
points: [
{
interval: { endTime: new Date(now - 120_000).toISOString() },
value: { int64Value: '2' }
},
{
interval: { endTime: new Date(now - 60_000).toISOString() },
value: { int64Value: '3' }
}
]
}
]
})
}
const environment = RELAY_OPS_ENVIRONMENTS.production
const directorErrors = GOOGLE_METRICS.find(
(definition) => definition.signal === 'director.errors'
)!
const deadlocks = GOOGLE_METRICS.find(
(definition) => definition.signal === 'cloud_sql.deadlocks'
)!
await expect(
readGoogleMetric(
environment,
directorErrors,
'secret-access-token',
startAt,
endAt,
fetchImpl
)
).resolves.toEqual({
value: 5,
observedAt: new Date(now - 60_000).toISOString()
})
await readGoogleMetric(
environment,
deadlocks,
'secret-access-token',
startAt,
endAt,
fetchImpl
)
expect(filters[0]).toContain(
'resource.label."service_name"="orca-cloud-relay"'
)
expect(filters[0]).toContain('metric.label."response_code"!="503"')
expect(filters[1]).toContain(
'resource.label."database_id"="onorca-cloud:orca-cloud-auth-db"'
)
})
it('zero-fills an expired sparse lock-wait point', async () => {
let pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
const fetchImpl: typeof fetch = async () => Response.json({
timeSeries: [{
points: [{
interval: { endTime: new Date(pointAt).toISOString() },
value: { int64Value: '1' }
}]
}]
})
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl
)).resolves.toEqual({
value: 1,
observedAt: new Date(pointAt).toISOString()
})
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => now + 3_000
)).resolves.toEqual({
value: 1,
observedAt: new Date(pointAt).toISOString()
})
pointAt--
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl
)).resolves.toEqual({ value: 0, observedAt: endAt })
})
it('freshens a sparse zero without masking a recent nonzero lock wait', async () => {
let value = 0
const pointAt = now - INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
const readAt = now + 11_879
const fetchImpl: typeof fetch = async () => Response.json({
timeSeries: [{
points: [{
interval: { endTime: new Date(pointAt).toISOString() },
value: { int64Value: String(value) }
}]
}]
})
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
const sparseZero = await readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => readAt
)
expect(sparseZero).toEqual({
value: 0,
observedAt: new Date(readAt).toISOString()
})
expect(readAt - pointAt).toBe(191_879)
expect(readAt - Date.parse(sparseZero!.observedAt)).toBe(0)
value = 20
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
fetchImpl,
() => readAt
)).resolves.toEqual({
value: 20,
observedAt: new Date(pointAt).toISOString()
})
})
it('preserves a recent nonzero lock wait across staggered series', async () => {
const nonzeroAt = now - 179_000
const zeroAt = now - 178_000
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.lock_waits'
)!
const metric = await readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json({
timeSeries: [
{
points: [{
interval: { endTime: new Date(nonzeroAt).toISOString() },
value: { int64Value: '7' }
}]
},
{
points: [{
interval: { endTime: new Date(zeroAt).toISOString() },
value: { int64Value: '0' }
}]
}
]
})
)
expect(metric).toEqual({
value: 7,
observedAt: new Date(nonzeroAt).toISOString()
})
await expect(readGoogleMetric(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json({
timeSeries: [{
points: [
{
interval: { endTime: new Date(nonzeroAt).toISOString() },
value: { int64Value: '7' }
},
{
interval: { endTime: new Date(zeroAt).toISOString() },
value: { int64Value: '0' }
}
]
}]
})
)).resolves.toEqual({ value: 0, observedAt: endAt })
})
it('retries an empty required metric without weakening its value', async () => {
let calls = 0
const waits: number[] = []
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'cloud_sql.cpu'
)!
await expect(readGoogleMetricWithEmptyRetry(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => Response.json(calls++ === 0
? { timeSeries: [] }
: {
timeSeries: [{
points: [{
interval: { endTime: new Date(now - 60_000).toISOString() },
value: { doubleValue: 0.81 }
}]
}]
}),
() => now,
async (ms) => { waits.push(ms) }
)).resolves.toEqual({
value: 0.81,
observedAt: new Date(now - 60_000).toISOString()
})
expect(calls).toBe(2)
expect(waits).toEqual([2_000])
})
it('still reports a required metric missing after bounded retries', async () => {
let calls = 0
const waits: number[] = []
const definition = GOOGLE_METRICS.find(
({ signal }) => signal === 'director.concurrency'
)!
await expect(readGoogleMetricWithEmptyRetry(
RELAY_OPS_ENVIRONMENTS.production,
definition,
'secret-access-token',
startAt,
endAt,
async () => {
calls++
return Response.json({ timeSeries: [] })
},
() => now,
async (ms) => { waits.push(ms) }
)).resolves.toBeNull()
expect(calls).toBe(3)
expect(waits).toEqual([2_000, 2_000])
})
it('timestamps sparse retry aggregates at query completion', () => {
expect(relayFiveMinuteDeltaSignal({
available: true,
points: [
{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 },
{ at: new Date(now - 4 * 60_000).toISOString(), value: 2 }
]
}, endAt)).toEqual({ value: 2, observedAt: endAt })
expect(relayFiveMinuteDeltaSignal({
available: true,
points: [{ at: new Date(now - 22 * 60_000).toISOString(), value: 7 }]
}, endAt)).toEqual({ value: 0, observedAt: endAt })
expect(relayFiveMinuteDeltaSignal({
available: false,
points: []
}, endAt)).toBeNull()
})
it('aggregates admin state without returning tokens or response identities', async () => {
const identityToken = 'secret.header.signature'
const sensitiveIdentity = 'user@example.test'
const gcloud: GcloudClient = {
accessToken: async () => 'unused',
identityToken: async () => identityToken
}
let activeRequests = 0
let maximumActiveRequests = 0
let requestCount = 0
const fetchImpl: typeof fetch = async (_input, init) => {
requestCount++
activeRequests++
maximumActiveRequests = Math.max(maximumActiveRequests, activeRequests)
expect(new Headers(init?.headers).get('authorization')).toBe(
`Bearer ${identityToken}`
)
const body = JSON.parse(String(init?.body)) as {
cellId?: string
sourceCellId?: string
targetCellId?: string
}
await Promise.resolve()
activeRequests--
if (!body.cellId && !body.sourceCellId) {
return Response.json({
selector: {
generation: 1,
membership: {
existingOnly: productionCells.slice(2),
migrationOnly: ['production-gce-c2'],
general: ['production-gce-c1']
}
}
})
}
if (body.cellId) {
return Response.json({
status: {
// Selector-era monitoring must ignore this legacy compatibility bit.
enabled: body.cellId !== 'production-gce-c1',
connectionCapacity: {
hardCap: body.cellId === 'production-gce-c1' ? 1_000 : 600
},
runtime: {
lastHeartbeatAt: now - 1_000,
heartbeatFresh: true
},
userId: sensitiveIdentity
}
})
}
return Response.json({
blocked: 0,
blockedExpiredUnregistered:
body.sourceCellId === 'production-gce-c1' &&
body.targetCellId === 'production-gce-c2'
? 1
: 0,
registeredTargetInactive: 0,
userId: sensitiveIdentity
})
}
const result = await directorSignals(
'production',
{
generation: 1,
membership: {
existingOnly: productionCells.slice(2),
migrationOnly: ['production-gce-c2'],
general: ['production-gce-c1']
}
},
gcloud,
now,
fetchImpl
)
expect(requestCount).toBe(productionCells.length * 2)
expect(maximumActiveRequests).toBe(1)
expect(
result.source.signals['cell.production-gce-c1.migration_blocked']
).toMatchObject({ value: 1 })
expect(result.cells.find((cell) => cell.cellId === 'production-gce-c1')).toMatchObject({
expectedAdmissionState: 'general'
})
expect(
result.source.signals['cell.production-gce-c1.admission_state']
).toMatchObject({ value: 2 })
expect(
result.source.signals['cell.production-gce-c1.connection_hard_cap']
).toMatchObject({ value: 1_000 })
expect(
result.source.signals['cell.production-gce-c2.admission_state']
).toMatchObject({ value: 1 })
const serialized = JSON.stringify(result)
expect(serialized).not.toContain(identityToken)
expect(serialized).not.toContain(sensitiveIdentity)
})
})
@@ -0,0 +1,646 @@
import { z } from 'zod'
import { buildDashboardSnapshot } from './dashboard-snapshot.js'
import type {
RelayOpsEnvironment,
RelayOpsEnvironmentId
} from './environment-config.js'
import { relayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import type { RelayMetricSnapshot } from './monitoring-snapshot.js'
import {
AdmissionSelectorSchema,
effectiveAdmissionState,
normalizeSelectorMembership,
selectorCellState,
type AdmissionSelector,
} from './incident-selector.js'
import {
INCIDENT_MONITOR_THRESHOLDS,
type IncidentSample,
type IncidentSignal,
type IncidentSource
} from './incident-monitor.js'
const NumericSchema = z.union([z.number(), z.string()])
.transform(Number)
.pipe(z.number().finite())
const MonitoringPointSchema = z.object({
interval: z.object({ endTime: z.string() }),
value: z.object({
doubleValue: NumericSchema.optional(),
int64Value: NumericSchema.optional(),
distributionValue: z.object({
mean: NumericSchema.optional(),
range: z.object({ max: NumericSchema.optional() }).optional()
}).optional()
})
})
const MonitoringResponseSchema = z.object({
timeSeries: z.array(z.object({ points: z.array(MonitoringPointSchema) })).default([]),
nextPageToken: z.string().optional()
})
const CellStatusSchema = z.object({
status: z.object({
enabled: z.boolean(),
connectionCapacity: z
.object({ hardCap: z.number().int().positive() })
.nullable(),
runtime: z.object({
lastHeartbeatAt: z.number(),
heartbeatFresh: z.boolean()
}).nullable()
})
})
const SelectorStatusSchema = z.object({
selector: AdmissionSelectorSchema
})
const MigrationStatusSchema = z.object({
blocked: z.number().int().nonnegative(),
registeredTargetInactive: z.number().int().nonnegative(),
blockedExpiredUnregistered: z.number().int().nonnegative()
})
export type GoogleMetricDefinition = {
signal: string
type: string
resourceFilter: string
aggregation: 'latest-max' | 'latest-sum' | 'window-sum'
emptyIsZero?: boolean
zeroAfterMs?: number
}
export const GOOGLE_METRICS: GoogleMetricDefinition[] = [
{
signal: 'cloud_sql.cpu',
type: 'cloudsql.googleapis.com/database/cpu/utilization',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-max'
},
{
signal: 'cloud_sql.memory',
type: 'cloudsql.googleapis.com/database/memory/utilization',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-max'
},
{
signal: 'cloud_sql.backends',
type: 'cloudsql.googleapis.com/database/postgresql/num_backends',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'latest-sum'
},
{
signal: 'cloud_sql.lock_waits',
type: 'cloudsql.googleapis.com/database/postgresql/backends_in_wait',
resourceFilter:
'resource.type="cloudsql_database" AND metric.label."wait_event_type"="Lock"',
aggregation: 'latest-max',
emptyIsZero: true,
zeroAfterMs: INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
},
{
signal: 'cloud_sql.deadlocks',
type: 'cloudsql.googleapis.com/database/postgresql/deadlock_count',
resourceFilter: 'resource.type="cloudsql_database"',
aggregation: 'window-sum',
emptyIsZero: true
},
{
signal: 'director.instances',
type: 'run.googleapis.com/container/instance_count',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-sum'
},
{
signal: 'director.cpu',
type: 'run.googleapis.com/container/cpu/utilizations',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-max'
},
{
signal: 'director.memory',
type: 'run.googleapis.com/container/memory/utilizations',
resourceFilter: 'resource.type="cloud_run_revision"',
aggregation: 'latest-max'
},
{
signal: 'director.concurrency',
type: 'run.googleapis.com/container/max_request_concurrencies',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."state"="active"',
aggregation: 'latest-max'
},
{
signal: 'director.errors',
type: 'run.googleapis.com/request_count',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx" AND metric.label."response_code"!="503"',
aggregation: 'window-sum',
emptyIsZero: true
},
{
signal: 'auth.errors',
type: 'run.googleapis.com/request_count',
resourceFilter:
'resource.type="cloud_run_revision" AND metric.label."response_code_class"="5xx"',
aggregation: 'window-sum',
emptyIsZero: true
}
]
function pointValue(point: z.infer<typeof MonitoringPointSchema>): number {
return (
point.value.doubleValue ??
point.value.int64Value ??
point.value.distributionValue?.range?.max ??
point.value.distributionValue?.mean ??
0
)
}
function serviceFilter(signal: string, directorService: string, authService: string): string {
if (signal.startsWith('director.')) {
return `resource.label."service_name"="${directorService}"`
}
if (signal.startsWith('auth.')) {
return `resource.label."service_name"="${authService}"`
}
return ''
}
function targetFilter(
definition: GoogleMetricDefinition,
environment: RelayOpsEnvironment
): string {
if (definition.signal.startsWith('cloud_sql.')) {
return `resource.label."database_id"="${environment.project}:${environment.sqlInstance}"`
}
return serviceFilter(
definition.signal,
environment.directorService,
environment.authService
)
}
async function googleJson(
fetchImpl: typeof fetch,
token: string,
url: URL | string,
init: RequestInit = {}
): Promise<unknown> {
const response = await fetchImpl(url, {
...init,
headers: {
authorization: `Bearer ${token}`,
...(init.body ? { 'content-type': 'application/json' } : {})
},
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Google telemetry returned ${response.status}`)
return await response.json()
}
export async function readGoogleMetric(
environment: RelayOpsEnvironment,
definition: GoogleMetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch,
now: () => number = () => Date.parse(endAt)
): Promise<IncidentSignal | null> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
)
const filters = [
`metric.type="${definition.type}"`,
definition.resourceFilter,
targetFilter(definition, environment)
].filter(Boolean)
url.searchParams.set('filter', filters.join(' AND '))
url.searchParams.set('interval.startTime', startAt)
url.searchParams.set('interval.endTime', endAt)
url.searchParams.set('view', 'FULL')
url.searchParams.set('pageSize', '1000')
const parsed = MonitoringResponseSchema.parse(
await googleJson(fetchImpl, token, url)
)
if (parsed.nextPageToken) throw new Error('Google metric pagination is incomplete')
const queryEndMs = Date.parse(endAt)
const readAtMs = Math.max(queryEndMs, now())
const readAt = new Date(readAtMs).toISOString()
const points = parsed.timeSeries.flatMap((series) => series.points)
if (points.length === 0) {
return definition.emptyIsZero ? { value: 0, observedAt: readAt } : null
}
const zeroAfterMs = definition.zeroAfterMs
if (definition.emptyIsZero && zeroAfterMs !== undefined) {
const latestSeriesPoints = parsed.timeSeries.flatMap((series) => {
const seriesNewestAt = Math.max(
...series.points.map((point) => Date.parse(point.interval.endTime))
)
return series.points.filter(
(point) => Date.parse(point.interval.endTime) === seriesNewestAt
)
})
const futurePoints = latestSeriesPoints.filter(
(point) => Date.parse(point.interval.endTime) > queryEndMs
)
if (futurePoints.length > 0) {
return {
value: Math.max(...futurePoints.map(pointValue)),
observedAt: new Date(Math.max(
...futurePoints.map((point) => Date.parse(point.interval.endTime))
)).toISOString()
}
}
const recentNonzero = latestSeriesPoints.filter((point) => {
const pointAt = Date.parse(point.interval.endTime)
return pointValue(point) > 0 && queryEndMs - pointAt <= zeroAfterMs
})
if (recentNonzero.length === 0) return { value: 0, observedAt: readAt }
return {
value: Math.max(...recentNonzero.map(pointValue)),
observedAt: new Date(Math.min(
...recentNonzero.map((point) => Date.parse(point.interval.endTime))
)).toISOString()
}
}
const newestAt = Math.max(...points.map((point) => Date.parse(point.interval.endTime)))
const selected = definition.aggregation === 'window-sum'
? points
: points.filter((point) => Date.parse(point.interval.endTime) === newestAt)
const values = selected.map(pointValue)
const value =
definition.aggregation !== 'latest-max'
? values.reduce((total, entry) => total + entry, 0)
: Math.max(...values)
return {
value,
observedAt: new Date(newestAt).toISOString()
}
}
export async function readGoogleMetricWithEmptyRetry(
environment: RelayOpsEnvironment,
definition: GoogleMetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch,
now: () => number = () => Date.parse(endAt),
wait: (ms: number) => Promise<void> = async (ms) =>
await new Promise((resolve) => setTimeout(resolve, ms))
): Promise<IncidentSignal | null> {
for (let attempt = 0; attempt < 3; attempt++) {
const signal = await readGoogleMetric(
environment,
definition,
token,
startAt,
endAt,
fetchImpl,
now
)
if (signal !== null || definition.emptyIsZero) return signal
if (attempt < 2) await wait(2_000)
}
return null
}
function addSignal(
signals: Record<string, IncidentSignal>,
name: string,
value: number | null,
observedAt: string | null
): void {
if (value === null || observedAt === null) return
signals[name] = { value, observedAt }
}
function endpointSignals(
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>,
nowAt: string
): IncidentSource {
const signals: Record<string, IncidentSignal> = {}
const addEndpoint = (
prefix: string,
endpoint: { health: boolean | null; ready: boolean | null; latencyMs: number | null },
unavailableIsZero = false
) => {
addSignal(
signals,
`${prefix}.health`,
endpoint.health === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.health),
nowAt
)
addSignal(
signals,
`${prefix}.ready`,
endpoint.ready === null ? (unavailableIsZero ? 0 : null) : Number(endpoint.ready),
nowAt
)
addSignal(signals, `${prefix}.latency_ms`, endpoint.latencyMs, nowAt)
}
addEndpoint('director', snapshot.resources.directorEndpoint)
addEndpoint('auth', snapshot.resources.authEndpoint)
for (const cell of snapshot.resources.cells) {
addEndpoint(`cell.${cell.cellId}`, cell.endpoint, true)
}
return { observedAt: nowAt, signals }
}
export function relayFiveMinuteDeltaSignal(
metric: Pick<RelayMetricSnapshot, 'available' | 'points'>,
endAt: string
): IncidentSignal | null {
if (!metric.available) return null
const endMs = Date.parse(endAt)
if (!Number.isFinite(endMs)) throw new Error('Relay telemetry end time is invalid')
return {
value: metric.points
.filter((point) => {
const pointMs = Date.parse(point.at)
return pointMs >= endMs - 300_000 && pointMs <= endMs
})
.reduce((total, point) => total + point.value, 0),
observedAt: endAt
}
}
function relaySignals(
snapshot: Awaited<ReturnType<typeof buildDashboardSnapshot>>
): IncidentSource {
const metrics = snapshot.monitoring.metrics
const signals: Record<string, IncidentSignal> = {}
addSignal(
signals,
'relay.pool_waiting',
metrics.db_waiters_max.latest,
metrics.db_waiters_max.latestAt
)
addSignal(
signals,
'relay.pool_wait_ms',
metrics.db_wait_ms_max.latest,
metrics.db_wait_ms_max.latestAt
)
const retries = relayFiveMinuteDeltaSignal(
metrics.postgres_retries,
snapshot.monitoring.endAt
)
if (retries) signals['relay.postgres_retries'] = retries
const retryExhausted = relayFiveMinuteDeltaSignal(
metrics.postgres_retry_exhausted,
snapshot.monitoring.endAt
)
if (retryExhausted) signals['relay.postgres_retry_exhausted'] = retryExhausted
for (const cell of snapshot.resources.cells) {
addSignal(
signals,
`cell.${cell.cellId}.connections`,
metrics.total_connections.latestByCell[cell.cellId] ?? null,
metrics.total_connections.latestAt
)
addSignal(
signals,
`cell.${cell.cellId}.queued_bytes`,
metrics.queued_bytes.latestByCell[cell.cellId] ?? null,
metrics.queued_bytes.latestAt
)
}
const observedTimes = Object.values(signals).map((entry) => Date.parse(entry.observedAt))
if (observedTimes.length === 0) throw new Error('Relay telemetry is unavailable')
const observedAt = new Date(Math.max(...observedTimes)).toISOString()
return { observedAt, signals }
}
async function adminPost(
fetchImpl: typeof fetch,
origin: string,
token: string,
path: string,
body: unknown
): Promise<unknown> {
const response = await fetchImpl(`${origin}${path}`, {
method: 'POST',
headers: { authorization: `Bearer ${token}`, 'content-type': 'application/json' },
body: JSON.stringify(body),
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Relay admin telemetry returned ${response.status}`)
return await response.json()
}
export async function directorSignals(
environmentId: RelayOpsEnvironmentId,
expectedSelector: AdmissionSelector,
gcloud: GcloudClient,
nowMs: number,
fetchImpl: typeof fetch
): Promise<{
source: IncidentSource
selector: AdmissionSelector
cells: IncidentSample['cells']
}> {
const environment = relayOpsEnvironment(environmentId)
if (!gcloud.identityToken) throw new Error('gcloud identity-token support is unavailable')
const token = await gcloud.identityToken(`${environment.directorOrigin}/v1/admin/drain`)
const configuredCellIds = new Set(environment.cells.map((cell) => cell.cellId))
const rawSelector = SelectorStatusSchema.parse(
await adminPost(
fetchImpl,
environment.directorOrigin,
token,
'/v1/admin/admission-selector/status',
{ v: 1 }
)
).selector
const selector = {
generation: rawSelector.generation,
membership: normalizeSelectorMembership(rawSelector.membership, configuredCellIds)
}
const statuses: Array<{
cell: RelayOpsEnvironment['cells'][number]
status: z.infer<typeof CellStatusSchema>['status']
}> = []
for (const cell of environment.cells) {
statuses.push({
cell,
status: CellStatusSchema.parse(
await adminPost(fetchImpl, environment.directorOrigin, token, '/v1/admin/cell-status', {
v: 1,
cellId: cell.cellId
})
).status
})
}
const migrationEntries: Array<{
sourceCellId: string
migration: z.infer<typeof MigrationStatusSchema>
}> = []
const migrationTargets = new Set(expectedSelector.membership.migrationOnly)
for (const source of environment.cells) {
for (const target of environment.cells) {
if (source.cellId === target.cellId || !migrationTargets.has(target.cellId)) continue
migrationEntries.push({
sourceCellId: source.cellId,
migration: MigrationStatusSchema.parse(
await adminPost(
fetchImpl,
environment.directorOrigin,
token,
'/v1/admin/evacuation-status',
{
v: 1,
sourceCellId: source.cellId,
targetCellId: target.cellId,
completeReady: false
}
)
)
})
}
}
const migrationBySource = new Map<string, { blocked: number; targetInactive: number }>()
for (const { sourceCellId, migration } of migrationEntries) {
const aggregate = migrationBySource.get(sourceCellId) ?? { blocked: 0, targetInactive: 0 }
aggregate.blocked += migration.blocked + migration.blockedExpiredUnregistered
aggregate.targetInactive += migration.registeredTargetInactive
migrationBySource.set(sourceCellId, aggregate)
}
const nowAt = new Date(nowMs).toISOString()
const signals: Record<string, IncidentSignal> = {}
for (const { cell, status } of statuses) {
const prefix = `cell.${cell.cellId}`
const admissionState = effectiveAdmissionState(
selector,
status.enabled,
cell.cellId
)
addSignal(
signals,
`${prefix}.admission_state`,
['existing-only', 'migration-only', 'general'].indexOf(admissionState),
nowAt
)
addSignal(
signals,
`${prefix}.connection_hard_cap`,
status.connectionCapacity?.hardCap ??
INCIDENT_MONITOR_THRESHOLDS.cellConnections,
nowAt
)
if (status.runtime) {
addSignal(
signals,
`${prefix}.heartbeat_fresh`,
Number(status.runtime.heartbeatFresh),
nowAt
)
addSignal(
signals,
`${prefix}.heartbeat_age_ms`,
Math.max(0, nowMs - status.runtime.lastHeartbeatAt),
nowAt
)
}
const migration = migrationBySource.get(cell.cellId) ?? { blocked: 0, targetInactive: 0 }
addSignal(signals, `${prefix}.migration_blocked`, migration.blocked, nowAt)
addSignal(
signals,
`${prefix}.migration_target_inactive`,
migration.targetInactive,
nowAt
)
}
return {
source: { observedAt: nowAt, signals },
selector,
cells: statuses.map(({ cell }) => ({
cellId: cell.cellId,
runtimeKnown: true,
powered: true,
expectedAdmissionState: selectorCellState(expectedSelector, cell.cellId)
}))
}
}
export type IncidentSampleCollectorOptions = {
environment: RelayOpsEnvironmentId
expectedSelector: AdmissionSelector
fetchImpl?: typeof fetch
now?: () => number
}
export function createIncidentSampleCollector(
gcloud: GcloudClient,
options: IncidentSampleCollectorOptions
): () => Promise<IncidentSample> {
const fetchImpl = options.fetchImpl ?? fetch
const now = options.now ?? Date.now
return async () => {
const nowMs = now()
const nowAt = new Date(nowMs).toISOString()
const startAt = new Date(nowMs - 5 * 60_000).toISOString()
const environment = relayOpsEnvironment(options.environment)
const accessToken = gcloud.accessToken()
const cloudMetricEntries = accessToken.then(async (token) => await Promise.all(
GOOGLE_METRICS.map(async (definition) => [
definition.signal,
await readGoogleMetricWithEmptyRetry(
environment,
definition,
token,
startAt,
nowAt,
fetchImpl,
now
)
] as const)
))
const [snapshot, director, metricEntries] = await Promise.all([
buildDashboardSnapshot(options.environment, gcloud, {
windowMinutes: 5,
now: new Date(nowMs),
fetchImpl
}),
directorSignals(
options.environment,
options.expectedSelector,
gcloud,
nowMs,
fetchImpl
),
cloudMetricEntries
])
const cloudSignals = Object.fromEntries(
metricEntries.filter((entry): entry is [string, IncidentSignal] => entry[1] !== null)
)
const relay = relaySignals(snapshot)
const poweredByCell = new Map(
snapshot.resources.cells.map((cell) => [
cell.cellId,
{
runtimeKnown: cell.targetSize !== null,
powered: (cell.targetSize ?? 0) > 0
}
])
)
return {
collectedAt: nowAt,
selector: director.selector,
expectedSelector: options.expectedSelector,
sources: {
'active-probe': endpointSignals(snapshot, nowAt),
'cloud-monitoring': { observedAt: nowAt, signals: cloudSignals },
'relay-logs': relay,
'director-admin': director.source
},
cells: director.cells.map((cell) => ({
...cell,
runtimeKnown: poweredByCell.get(cell.cellId)?.runtimeKnown ?? false,
powered: poweredByCell.get(cell.cellId)?.powered ?? false
}))
}
}
}
@@ -0,0 +1,786 @@
import { describe, expect, it } from 'vitest'
import {
evaluateIncidentSample,
INCIDENT_CHECKPOINT_MINUTES,
INCIDENT_MONITOR_THRESHOLDS,
INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS,
initialIncidentMonitorState,
preDrainDryRunPassed,
runIncidentMonitor,
type IncidentSample
} from './incident-monitor.js'
const startedAt = Date.parse('2026-07-28T00:00:00.000Z')
const selector = {
generation: 1,
membership: {
existingOnly: [],
migrationOnly: [],
general: ['production-gce-c1']
}
}
const signal = (value: number, at = startedAt) => ({
value,
observedAt: new Date(at).toISOString()
})
function healthySample(at = startedAt): IncidentSample {
const observedAt = new Date(at).toISOString()
return {
collectedAt: observedAt,
selector,
expectedSelector: selector,
cells: [{
cellId: 'production-gce-c1',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
}],
sources: {
'active-probe': {
observedAt,
signals: {
'director.health': signal(1, at),
'director.ready': signal(1, at),
'director.latency_ms': signal(100, at),
'auth.health': signal(1, at),
'auth.ready': signal(1, at),
'auth.latency_ms': signal(100, at),
'cell.production-gce-c1.health': signal(1, at),
'cell.production-gce-c1.ready': signal(1, at),
'cell.production-gce-c1.latency_ms': signal(100, at)
}
},
'cloud-monitoring': {
observedAt,
signals: {
'cloud_sql.cpu': signal(0.2, at),
'cloud_sql.memory': signal(0.3, at),
'cloud_sql.backends': signal(12, at),
'cloud_sql.lock_waits': signal(0, at),
'cloud_sql.deadlocks': signal(0, at),
'director.instances': signal(5, at),
'director.cpu': signal(0.2, at),
'director.memory': signal(0.3, at),
'director.concurrency': signal(5, at),
'director.errors': signal(0, at),
'auth.errors': signal(0, at)
}
},
'relay-logs': {
observedAt,
signals: {
'relay.pool_waiting': signal(0, at),
'relay.pool_wait_ms': signal(1, at),
'relay.postgres_retries': signal(0, at),
'relay.postgres_retry_exhausted': signal(0, at),
'cell.production-gce-c1.connections': signal(100, at),
'cell.production-gce-c1.queued_bytes': signal(0, at)
}
},
'director-admin': {
observedAt,
signals: {
'cell.production-gce-c1.admission_state': signal(2, at),
'cell.production-gce-c1.heartbeat_fresh': signal(1, at),
'cell.production-gce-c1.heartbeat_age_ms': signal(1_000, at),
'cell.production-gce-c1.migration_blocked': signal(0, at),
'cell.production-gce-c1.migration_target_inactive': signal(0, at)
}
}
}
}
}
describe('incident monitor evaluator', () => {
it('accepts a complete fresh sample at every exact boundary', () => {
const sample = healthySample()
sample.sources['active-probe']!.signals['director.latency_ms'] =
signal(INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs)
sample.sources['cloud-monitoring']!.signals['cloud_sql.cpu'] =
signal(INCIDENT_MONITOR_THRESHOLDS.cloudSqlCpuUtilization)
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPoolWaitMs)
sample.sources['relay-logs']!.signals['relay.postgres_retries'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries)
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] =
signal(INCIDENT_MONITOR_THRESHOLDS.cloudSqlBackends)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('freezes when postgres retries exceed the recalibrated ceiling', () => {
const sample = healthySample()
sample.sources['relay-logs']!.signals['relay.postgres_retries'] =
signal(INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries + 1)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'freeze',
failures: [
expect.objectContaining({ signal: 'relay.postgres_retries', threshold: 300 })
]
})
})
it('allows missing auth readiness and legacy existing-only connections', () => {
const sample = healthySample()
const legacySelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
sample.selector = legacySelector
sample.expectedSelector = legacySelector
sample.cells[0]!.expectedAdmissionState = 'existing-only'
delete sample.sources['active-probe']!.signals['auth.ready']
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(900)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('fails loudly on every missing or stale source', () => {
const missing = healthySample()
delete missing.sources['relay-logs']
expect(evaluateIncidentSample(missing, startedAt).failures).toContainEqual({
code: 'source_missing',
source: 'relay-logs'
})
const stale = healthySample(startedAt - 180_001)
const failures = evaluateIncidentSample(stale, startedAt).failures
expect(failures.some((failure) => failure.source === 'cloud-monitoring')).toBe(true)
expect(failures.some((failure) => failure.source === 'active-probe')).toBe(true)
})
it('freezes on SQL, director, relay pool, heartbeat, and migration breaches', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.cpu'] = signal(0.81)
sample.sources['cloud-monitoring']!.signals['director.instances'] = signal(7)
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(801)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_age_ms'
] = signal(45_001)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
] = signal(1)
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(501)
const evaluation = evaluateIncidentSample(sample, startedAt)
expect(evaluation.status).toBe('freeze')
expect(evaluation.failures.map((failure) => failure.signal)).toEqual(
expect.arrayContaining([
'cloud_sql.cpu',
'director.instances',
'relay.pool_waiting',
'cell.production-gce-c1.connections',
'cell.production-gce-c1.heartbeat_age_ms',
'cell.production-gce-c1.migration_blocked'
])
)
})
it('uses each cell reported physical connection cap', () => {
const sample = healthySample()
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.connection_hard_cap'
] = signal(1_000)
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(999)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
sample.sources['relay-logs']!.signals['cell.production-gce-c1.connections'] =
signal(1_000)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('freeze')
})
it('allows five active directors plus the warm rollback', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['director.instances'] = signal(6)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
})
it('allows bounded relay pool waiting below the latency ceiling', () => {
const sample = healthySample()
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(800)
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] = signal(2_500)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
sample.sources['relay-logs']!.signals['relay.pool_wait_ms'] = signal(2_501)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'relay-logs',
signal: 'relay.pool_wait_ms',
observed: 2_501,
threshold: 2_500
})
})
it('bounds Cloud SQL backends above measured healthy peaks', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] = signal(250)
expect(evaluateIncidentSample(sample, startedAt).status).toBe('green')
sample.sources['cloud-monitoring']!.signals['cloud_sql.backends'] = signal(251)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.backends',
observed: 251,
threshold: 250
})
})
it('bounds SQL lock waiters and keeps deadlocks zero-tolerance', () => {
const sample = healthySample()
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(20)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(21)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.lock_waits',
observed: 21,
threshold: 20
})
sample.sources['cloud-monitoring']!.signals['cloud_sql.lock_waits'] = signal(0)
sample.sources['cloud-monitoring']!.signals['cloud_sql.deadlocks'] = signal(1)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'cloud-monitoring',
signal: 'cloud_sql.deadlocks',
observed: 1,
threshold: 0
})
})
it('allows only registered target inactivity during forward recovery', () => {
const sample = healthySample()
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(40)
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'threshold_max',
source: 'director-admin',
signal: 'cell.production-gce-c1.migration_target_inactive',
observed: 40,
threshold: 0
})
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c1'
)
).toMatchObject({
status: 'green',
failures: []
})
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c2'
).failures
).toContainEqual(expect.objectContaining({
signal: 'cell.production-gce-c1.migration_target_inactive'
}))
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_blocked'
] = signal(1)
expect(
evaluateIncidentSample(
sample,
startedAt,
'recover-forward',
'production-gce-c1'
).failures
).toContainEqual({
code: 'threshold_max',
source: 'director-admin',
signal: 'cell.production-gce-c1.migration_blocked',
observed: 1,
threshold: 0
})
})
it('scopes registered target inactivity to the capacity cell', () => {
const sample = healthySample()
const scopedSelector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: ['production-gce-c2', 'production-gce-c3']
}
}
sample.selector = scopedSelector
sample.expectedSelector = scopedSelector
sample.cells[0]!.expectedAdmissionState = 'existing-only'
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
sample.cells.push({
cellId: 'production-gce-c2',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
})
sample.cells.push({
cellId: 'production-gce-c3',
runtimeKnown: true,
powered: true,
expectedAdmissionState: 'general'
})
for (const sourceName of ['active-probe', 'relay-logs', 'director-admin'] as const) {
const signals = sample.sources[sourceName]!.signals
for (const [name, value] of Object.entries(signals)) {
if (name.includes('production-gce-c1')) {
for (const cellId of ['production-gce-c2', 'production-gce-c3']) {
signals[name.replace('production-gce-c1', cellId)] = value
}
}
}
}
for (const cellId of ['production-gce-c2', 'production-gce-c3']) {
sample.sources['director-admin']!.signals[
`cell.${cellId}.admission_state`
] = signal(2)
}
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.migration_target_inactive'
] = signal(40)
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c2'
)
).toMatchObject({ status: 'green', failures: [] })
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c3'
)
).toMatchObject({ status: 'green', failures: [] })
sample.sources['director-admin']!.signals[
'cell.production-gce-c2.migration_target_inactive'
] = signal(1)
expect(
evaluateIncidentSample(
sample,
startedAt,
'capacity-transition',
null,
'production-gce-c2'
).failures
).toContainEqual(expect.objectContaining({
signal: 'cell.production-gce-c2.migration_target_inactive'
}))
})
it('freezes when expected admission has no powered runtime', () => {
const sample = healthySample()
sample.cells[0]!.powered = false
const evaluation = evaluateIncidentSample(sample, startedAt)
expect(evaluation.failures).toContainEqual({
code: 'expected_admission_without_runtime',
source: 'director-admin',
signal: 'cell.production-gce-c1.powered',
observed: 0,
threshold: 1
})
})
it('ignores stale runtime signals for an expected offline existing-only cell', () => {
const sample = healthySample()
sample.cells[0] = {
...sample.cells[0]!,
powered: false,
expectedAdmissionState: 'existing-only'
}
sample.expectedSelector = {
generation: sample.expectedSelector.generation,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
sample.selector = sample.expectedSelector
sample.sources['active-probe']!.signals['cell.production-gce-c1.health'] = signal(0)
sample.sources['active-probe']!.signals['cell.production-gce-c1.ready'] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.admission_state'
] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_fresh'
] = signal(0)
sample.sources['director-admin']!.signals[
'cell.production-gce-c1.heartbeat_age_ms'
] = signal(9_000_000)
expect(evaluateIncidentSample(sample, startedAt)).toMatchObject({
status: 'green',
failures: []
})
})
it('freezes when cell power inventory is unavailable', () => {
const sample = healthySample()
sample.cells[0]!.runtimeKnown = false
expect(evaluateIncidentSample(sample, startedAt).failures).toContainEqual({
code: 'runtime_power_unknown',
source: 'cloud-monitoring',
signal: 'cell.production-gce-c1.powered'
})
})
it('freezes on selector generation or tri-state membership drift', () => {
const generation = healthySample()
generation.selector = { ...generation.selector, generation: 2 }
expect(evaluateIncidentSample(generation, startedAt).failures).toContainEqual(
expect.objectContaining({ code: 'selector_mismatch' })
)
const membership = healthySample()
membership.selector = {
generation: 1,
membership: {
existingOnly: ['production-gce-c1'],
migrationOnly: [],
general: []
}
}
expect(evaluateIncidentSample(membership, startedAt).failures).toContainEqual(
expect.objectContaining({ code: 'selector_mismatch' })
)
})
})
describe('incident monitor lifecycle', () => {
it('persists the exact 90-minute checkpoints while polling every minute', async () => {
let now = startedAt
const checkpoints: number[] = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: false,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 90,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async (summary) => {
checkpoints.push(summary.checkpointMinute)
}
})
expect(checkpoints).toEqual([...INCIDENT_CHECKPOINT_MINUTES])
expect(result.sampleCount).toBe(91)
expect(result.completedAt).not.toBeNull()
expect(result.frozenAt).toBeNull()
})
it('latches monitor freeze across a restart without rewriting its time', async () => {
let now = startedAt
let unhealthy = true
let persisted = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: false,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const stop = new Error('stop after first persistence')
await expect(
runIncidentMonitor(persisted, {
now: () => now,
wait: async () => {
throw stop
},
collect: async () => {
const sample = healthySample(now)
if (unhealthy) {
sample.sources['relay-logs']!.signals['relay.pool_waiting'] = signal(31, now)
}
return sample
},
persist: async (state) => {
persisted = structuredClone(state)
},
checkpoint: async () => {}
})
).rejects.toThrow('stop after first persistence')
const frozenAt = persisted.frozenAt
unhealthy = false
now += 60_000
const result = await runIncidentMonitor(persisted, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.frozenAt).toBe(frozenAt)
expect(preDrainDryRunPassed(result)).toBe(false)
})
it.each([15, 90])(
'restarts a %i-minute continuous window after stale telemetry',
async (durationMinutes) => {
let now = startedAt
let staleInjected = false
const checkpoints: Array<[number, number]> = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: durationMinutes === 15,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => {
if (!staleInjected && now === startedAt + 5 * 60_000) {
staleInjected = true
return healthySample(now - 180_001)
}
return healthySample(now)
},
persist: async () => {},
checkpoint: async (summary) => {
checkpoints.push([summary.windowSequence, summary.checkpointMinute])
}
})
expect(result.windowSequence).toBe(1)
expect(result.windowStartedAt).toBe(
new Date(startedAt + 6 * 60_000).toISOString()
)
expect(result.completedAt).toBe(
new Date(startedAt + (durationMinutes + 6) * 60_000).toISOString()
)
expect(result.sampleCount).toBe(durationMinutes + 1)
expect(result.continuityEvents).toHaveLength(1)
expect(result.continuityEvents[0]!.failures).toEqual(
expect.arrayContaining([
expect.objectContaining({ code: 'source_stale' })
])
)
expect(checkpoints).toContainEqual([1, durationMinutes])
expect(result.frozenAt).toBeNull()
}
)
it('resets at the next fresh sample after a runner gap', async () => {
let now = startedAt + 10 * 60_000
const state = {
...initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
}),
lastSampleAt: new Date(startedAt).toISOString(),
sampleCount: 1,
totalSampleCount: 1
}
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () => healthySample(now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.windowSequence).toBe(1)
expect(result.windowStartedAt).toBe(new Date(startedAt + 10 * 60_000).toISOString())
expect(result.continuityEvents[0]!.failures[0]!.code).toBe('monitor_gap')
expect(result.sampleCount).toBe(16)
})
it('fails a dry run after 25 total minutes of continuity resets', async () => {
let now = startedAt
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
now += ms
},
collect: async () =>
healthySample(now === startedAt + 10 * 60_000 ? now - 180_001 : now),
persist: async () => {},
checkpoint: async () => {}
})
expect(result.completedAt).toBe(
new Date(startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS).toISOString()
)
expect(result.frozenAt).not.toBeNull()
expect(result.windowSequence).toBe(1)
expect(result.sampleCount).toBe(15)
expect(result.failures).toContainEqual({
code: 'continuity_deadline_exceeded',
source: 'active-probe',
observed: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS,
threshold: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
})
expect(preDrainDryRunPassed(result)).toBe(false)
})
it('fails an overdue resumed dry run before collecting again', async () => {
const now = startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS + 1
let collections = 0
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
const result = await runIncidentMonitor(state, {
now: () => now,
wait: async () => {},
collect: async () => {
collections++
return healthySample(now)
},
persist: async () => {},
checkpoint: async () => {}
})
expect(collections).toBe(0)
expect(result.failures).toContainEqual(expect.objectContaining({
code: 'continuity_deadline_exceeded'
}))
})
it('requires a completed green 15-minute dry run', () => {
const state = {
...initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
}),
sampleCount: 16,
completedAt: new Date(startedAt + 15 * 60_000).toISOString()
}
expect(preDrainDryRunPassed(state)).toBe(true)
expect(preDrainDryRunPassed({ ...state, frozenAt: state.startedAt })).toBe(false)
expect(preDrainDryRunPassed({
...state,
completedAt: new Date(startedAt + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS + 1).toISOString()
})).toBe(false)
})
it('keeps poll starts on the configured cadence after collection time', async () => {
let now = startedAt
const starts: number[] = []
const waits: number[] = []
const state = initialIncidentMonitorState({
incidentId: 'incident-1',
environment: 'production',
expectedSelector: selector,
preDrainDryRun: true,
migrationPolicy: 'strict',
recoverySourceCellId: null,
capacityCellId: null,
startedAt: new Date(startedAt).toISOString(),
durationMinutes: 15,
intervalMs: 60_000
})
await runIncidentMonitor(state, {
now: () => now,
wait: async (ms) => {
waits.push(ms)
now += ms
},
collect: async () => {
starts.push(now)
now += 15_000
return healthySample(now)
},
persist: async () => {},
checkpoint: async () => {}
})
expect(starts.slice(0, 3)).toEqual([
startedAt,
startedAt + 60_000,
startedAt + 120_000
])
expect(waits[0]).toBe(45_000)
})
})
@@ -0,0 +1,800 @@
import {
exactAdmissionSelector,
type AdmissionSelector,
type AdmissionState
} from './incident-selector.js'
export const INCIDENT_MONITOR_THRESHOLDS = {
activeProbeMaxAgeMs: 60_000,
cloudDataMaxAgeMs: 180_000,
relayLogMaxAgeMs: 180_000,
heartbeatMaxAgeMs: 45_000,
endpointLatencyMs: 2_000,
cloudSqlCpuUtilization: 0.8,
cloudSqlMemoryUtilization: 0.9,
// Why: healthy latest-sum backends idle near 100 but spike to 216 in 1-minute
// bursts (~10 min/day exceeded the old bar of 160 on 2026-08-26, freezing a
// pre-drain gate on baseline noise). 250 clears measured healthy peaks while
// firing well before the verified 400-connection ceiling; pool-wait and
// exhausted-retry signals keep their strict thresholds.
cloudSqlBackends: 250,
// Bound the observed recovery load; deadlocks remain zero-tolerance.
cloudSqlLockWaits: 20,
cloudSqlDeadlocks: 0,
// Why: pool amplitude cannot discriminate the 2026-08-23 incident. Healthy
// fleet-wide bursts reach 43 waiters / 2.03s waits several times an hour,
// and a cell roll's reconnect surge peaks at 676 waiters, while the real
// incident peaked at 356 waiters and never crossed 2.5s (waits cap ~2s
// structurally). The old bars of 30/1000 froze pre-drain gates on baseline
// noise (~17% per 15-minute window). Incident-class contention is caught by
// the retry signals below at ~10x separation; these bars now fence only
// genuinely unbounded queueing, which grows past both.
relayPoolWaiting: 800,
relayPoolWaitMs: 2_500,
// Why: successful lock retries are the contention machinery working, not harm.
// Healthy 2026-08-26 baseline bursts to 234/5min (26% of windows crossed the old
// bar of 20, set unmeasured at the monitor's 2026-07-28 birth); the 2026-08-23
// incident ran ~2,200-3,000/5min. 300 clears healthy bursts with ~10x incident
// margin; relayPostgresRetryExhausted below stays at zero tolerance, so any
// transaction that terminally fails still freezes the gate.
relayPostgresRetries: 300,
relayPostgresRetryExhausted: 0,
// Why: public admission is a per-instance semaphore, so fleet assignment capacity is
// concurrency x instances. A floor of 1 let the 2026-08-04 collapse from five instances
// to two pass unnoticed, which is the exact failure this monitor exists to catch. Keep in
// step with relay_min_instances in infra/terraform/environments/production.tfvars.
directorInstancesMin: 5,
// Five serving instances plus one warm scale-to-zero rollback during recovery.
directorInstancesMax: 6,
directorCpuUtilization: 0.8,
directorMemoryUtilization: 0.8,
directorConcurrency: 64,
directorErrors: 0,
authErrors: 0,
// Why: 800 exceeded the 600 hard cap, so this could never trigger on a capped cell. 500 is
// the ordinary admission limit a cell actually stops at (600 cap - 100 control-rebind reserve).
cellConnections: 500,
cellQueuedBytes: 48 * 1024 * 1024,
migrationBlocked: 0
} as const
export const INCIDENT_CHECKPOINT_MINUTES = [0, 5, 15, 30, 45, 60, 75, 90] as const
export const INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS = 25 * 60_000
export type IncidentSourceName =
| 'active-probe'
| 'cloud-monitoring'
| 'relay-logs'
| 'director-admin'
export type IncidentMigrationPolicy =
| 'strict'
| 'recover-forward'
| 'capacity-transition'
export type IncidentSignal = {
value: number
observedAt: string
}
export type IncidentSource = {
observedAt: string
signals: Record<string, IncidentSignal>
}
export type IncidentCellExpectation = {
cellId: string
runtimeKnown: boolean
powered: boolean
expectedAdmissionState: AdmissionState
}
export type IncidentSample = {
collectedAt: string
selector: AdmissionSelector
expectedSelector: AdmissionSelector
sources: Partial<Record<IncidentSourceName, IncidentSource>>
cells: IncidentCellExpectation[]
}
export type IncidentFailure = {
code: string
source: IncidentSourceName
signal?: string
observed?: number
threshold?: number
}
export type IncidentEvaluation = {
status: 'green' | 'freeze'
evaluatedAt: string
failures: IncidentFailure[]
}
export type IncidentCheckpoint = {
schemaVersion: 4
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
windowSequence: number
windowStartedAt: string
checkpointMinute: number
scheduledAt: string
recordedAt: string
status: 'green' | 'freeze'
frozenAt: string | null
sampleCount: number
failures: IncidentFailure[]
thresholds: typeof INCIDENT_MONITOR_THRESHOLDS
}
export type IncidentMonitorState = {
schemaVersion: 4
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
startedAt: string
windowStartedAt: string | null
windowSequence: number
durationMinutes: number
intervalMs: number
nextCheckpointIndex: number
sampleCount: number
totalSampleCount: number
lastSampleAt: string | null
continuityEvents: {
recordedAt: string
windowSequence: number
failures: IncidentFailure[]
}[]
frozenAt: string | null
failures: IncidentFailure[]
completedAt: string | null
}
type NumericRule = {
source: IncidentSourceName
signal: string
comparison: 'max' | 'min' | 'equal'
threshold: number
}
const NUMERIC_RULES: NumericRule[] = [
{ source: 'active-probe', signal: 'director.health', comparison: 'equal', threshold: 1 },
{ source: 'active-probe', signal: 'director.ready', comparison: 'equal', threshold: 1 },
{
source: 'active-probe',
signal: 'director.latency_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
},
{ source: 'active-probe', signal: 'auth.health', comparison: 'equal', threshold: 1 },
{
source: 'active-probe',
signal: 'auth.latency_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.cpu',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlCpuUtilization
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.memory',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlMemoryUtilization
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.backends',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlBackends
},
{
source: 'cloud-monitoring',
signal: 'director.instances',
comparison: 'min',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorInstancesMin
},
{
source: 'cloud-monitoring',
signal: 'director.instances',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorInstancesMax
},
{
source: 'cloud-monitoring',
signal: 'director.cpu',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorCpuUtilization
},
{
source: 'cloud-monitoring',
signal: 'director.memory',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorMemoryUtilization
},
{
source: 'cloud-monitoring',
signal: 'director.concurrency',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorConcurrency
},
{
source: 'cloud-monitoring',
signal: 'director.errors',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.directorErrors
},
{
source: 'cloud-monitoring',
signal: 'auth.errors',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.authErrors
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.lock_waits',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlLockWaits
},
{
source: 'cloud-monitoring',
signal: 'cloud_sql.deadlocks',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.cloudSqlDeadlocks
},
{
source: 'relay-logs',
signal: 'relay.pool_waiting',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPoolWaiting
},
{
source: 'relay-logs',
signal: 'relay.pool_wait_ms',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPoolWaitMs
},
{
source: 'relay-logs',
signal: 'relay.postgres_retries',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetries
},
{
source: 'relay-logs',
signal: 'relay.postgres_retry_exhausted',
comparison: 'max',
threshold: INCIDENT_MONITOR_THRESHOLDS.relayPostgresRetryExhausted
}
]
const SOURCE_MAX_AGE: Record<IncidentSourceName, number> = {
'active-probe': INCIDENT_MONITOR_THRESHOLDS.activeProbeMaxAgeMs,
'cloud-monitoring': INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs,
'relay-logs': INCIDENT_MONITOR_THRESHOLDS.relayLogMaxAgeMs,
'director-admin': INCIDENT_MONITOR_THRESHOLDS.cloudDataMaxAgeMs
}
function ageMs(timestamp: string, nowMs: number): number {
const parsed = Date.parse(timestamp)
return Number.isFinite(parsed) ? nowMs - parsed : Number.POSITIVE_INFINITY
}
function addMissingSignal(
failures: IncidentFailure[],
source: IncidentSourceName,
signal: string
): void {
failures.push({ code: 'signal_missing', source, signal })
}
function checkRule(
failures: IncidentFailure[],
source: IncidentSourceName,
signals: Record<string, IncidentSignal>,
rule: NumericRule
): void {
const signal = signals[rule.signal]
if (!signal) return addMissingSignal(failures, source, rule.signal)
const failed =
(rule.comparison === 'max' && signal.value > rule.threshold) ||
(rule.comparison === 'min' && signal.value < rule.threshold) ||
(rule.comparison === 'equal' && signal.value !== rule.threshold)
if (failed) {
failures.push({
code: `threshold_${rule.comparison}`,
source,
signal: rule.signal,
observed: signal.value,
threshold: rule.threshold
})
}
}
function checkCell(
failures: IncidentFailure[],
sample: IncidentSample,
cell: IncidentCellExpectation,
migrationPolicy: IncidentMigrationPolicy,
recoverySourceCellId: string | null,
capacityCellId: string | null
): void {
const probe = sample.sources['active-probe']?.signals
const relay = sample.sources['relay-logs']?.signals
const admin = sample.sources['director-admin']?.signals
if (!cell.runtimeKnown) {
failures.push({
code: 'runtime_power_unknown',
source: 'cloud-monitoring',
signal: `cell.${cell.cellId}.powered`
})
}
if (
cell.runtimeKnown &&
cell.expectedAdmissionState !== 'existing-only' &&
!cell.powered
) {
failures.push({
code: 'expected_admission_without_runtime',
source: 'director-admin',
signal: `cell.${cell.cellId}.powered`,
observed: 0,
threshold: 1
})
}
const checks = [
['active-probe', probe, `cell.${cell.cellId}.health`, cell.powered ? 1 : 0, 'equal'],
['active-probe', probe, `cell.${cell.cellId}.ready`, cell.powered ? 1 : 0, 'equal'],
[
'active-probe',
probe,
`cell.${cell.cellId}.latency_ms`,
INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.admission_state`,
['existing-only', 'migration-only', 'general'].indexOf(
cell.expectedAdmissionState
),
'equal'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.heartbeat_fresh`,
1,
'equal'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.heartbeat_age_ms`,
INCIDENT_MONITOR_THRESHOLDS.heartbeatMaxAgeMs,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.migration_blocked`,
INCIDENT_MONITOR_THRESHOLDS.migrationBlocked,
'max'
],
[
'director-admin',
admin,
`cell.${cell.cellId}.migration_target_inactive`,
INCIDENT_MONITOR_THRESHOLDS.migrationBlocked,
'max'
],
[
'relay-logs',
relay,
`cell.${cell.cellId}.connections`,
(admin?.[`cell.${cell.cellId}.connection_hard_cap`]?.value ??
INCIDENT_MONITOR_THRESHOLDS.cellConnections + 1) - 1,
'max'
],
[
'relay-logs',
relay,
`cell.${cell.cellId}.queued_bytes`,
INCIDENT_MONITOR_THRESHOLDS.cellQueuedBytes,
'max'
]
] as const
for (const [source, signals, signalName, threshold, comparison] of checks) {
if (
migrationPolicy === 'recover-forward' &&
cell.cellId === recoverySourceCellId &&
signalName.endsWith('.migration_target_inactive')
) {
if (!signals?.[signalName]) addMissingSignal(failures, source, signalName)
continue
}
if (
migrationPolicy === 'capacity-transition' &&
capacityCellId !== null &&
cell.cellId !== capacityCellId &&
cell.expectedAdmissionState === 'existing-only' &&
signalName.endsWith('.migration_target_inactive')
) {
if (!signals?.[signalName]) addMissingSignal(failures, source, signalName)
continue
}
if (
cell.expectedAdmissionState === 'existing-only' &&
signalName.endsWith('.connections')
) {
continue
}
if (
!cell.powered &&
[
'latency_ms',
'heartbeat_fresh',
'heartbeat_age_ms',
'connections',
'queued_bytes'
].some((suffix) => signalName.endsWith(suffix))
) {
continue
}
if (!signals?.[signalName]) {
addMissingSignal(failures, source, signalName)
continue
}
const value = signals[signalName].value
const failed = comparison === 'equal' ? value !== threshold : value > threshold
if (failed) {
failures.push({
code: `threshold_${comparison}`,
source,
signal: signalName,
observed: value,
threshold
})
}
}
}
export function evaluateIncidentSample(
sample: IncidentSample,
nowMs = Date.now(),
migrationPolicy: IncidentMigrationPolicy = 'strict',
recoverySourceCellId: string | null = null,
capacityCellId: string | null = null
): IncidentEvaluation {
const failures: IncidentFailure[] = []
if (!exactAdmissionSelector(sample.selector, sample.expectedSelector)) {
failures.push({
code: 'selector_mismatch',
source: 'director-admin',
signal: 'selector.generation',
observed: sample.selector.generation,
threshold: sample.expectedSelector.generation
})
}
for (const [sourceName, maxAge] of Object.entries(SOURCE_MAX_AGE) as [
IncidentSourceName,
number
][]) {
const source = sample.sources[sourceName]
if (!source) {
failures.push({ code: 'source_missing', source: sourceName })
continue
}
if (ageMs(source.observedAt, nowMs) < 0 || ageMs(source.observedAt, nowMs) > maxAge) {
failures.push({
code: 'source_stale',
source: sourceName,
observed: ageMs(source.observedAt, nowMs),
threshold: maxAge
})
}
for (const [signalName, signal] of Object.entries(source.signals)) {
if (ageMs(signal.observedAt, nowMs) < 0 || ageMs(signal.observedAt, nowMs) > maxAge) {
failures.push({
code: 'signal_stale',
source: sourceName,
signal: signalName,
observed: ageMs(signal.observedAt, nowMs),
threshold: maxAge
})
}
}
}
for (const rule of NUMERIC_RULES) {
const source = sample.sources[rule.source]
if (source) checkRule(failures, rule.source, source.signals, rule)
}
for (const cell of sample.cells) {
checkCell(
failures,
sample,
cell,
migrationPolicy,
recoverySourceCellId,
capacityCellId
)
}
return {
status: failures.length === 0 ? 'green' : 'freeze',
evaluatedAt: new Date(nowMs).toISOString(),
failures
}
}
export function initialIncidentMonitorState(input: {
incidentId: string
environment: 'production' | 'staging'
expectedSelector: AdmissionSelector
preDrainDryRun: boolean
migrationPolicy: IncidentMigrationPolicy
recoverySourceCellId: string | null
capacityCellId: string | null
startedAt: string
durationMinutes: number
intervalMs: number
}): IncidentMonitorState {
if (input.intervalMs < 1_000 || input.intervalMs > 60_000) {
throw new Error('incident monitor interval must be between 1 and 60 seconds')
}
if (input.durationMinutes < 15 || input.durationMinutes > 90) {
throw new Error('incident monitor duration must be between 15 and 90 minutes')
}
return {
schemaVersion: 4,
...input,
windowStartedAt: input.startedAt,
windowSequence: 0,
nextCheckpointIndex: 0,
sampleCount: 0,
totalSampleCount: 0,
lastSampleAt: null,
continuityEvents: [],
frozenAt: null,
failures: [],
completedAt: null
}
}
export type IncidentMonitorDependencies = {
now(): number
wait(ms: number): Promise<void>
collect(): Promise<IncidentSample>
persist(state: IncidentMonitorState): Promise<void>
checkpoint(summary: IncidentCheckpoint): Promise<void>
}
function checkpointMinutes(durationMinutes: number): number[] {
return INCIDENT_CHECKPOINT_MINUTES.filter((minute) => minute <= durationMinutes)
}
const CONTINUITY_FAILURE_CODES = new Set([
'collector_failed',
'monitor_gap',
'signal_stale',
'source_missing',
'source_stale'
])
function resetContinuousWindow(
state: IncidentMonitorState,
recordedAt: string,
failures: IncidentFailure[]
): void {
if (state.windowStartedAt !== null) {
state.windowSequence++
state.windowStartedAt = null
state.nextCheckpointIndex = 0
state.sampleCount = 0
state.completedAt = null
}
state.continuityEvents.push({
recordedAt,
windowSequence: state.windowSequence,
failures
})
}
function completeContinuityDeadline(
state: IncidentMonitorState,
nowMs: number,
lineageStartMs: number
): void {
const recordedAt = new Date(nowMs).toISOString()
state.frozenAt ??= recordedAt
state.failures.push({
code: 'continuity_deadline_exceeded',
source: 'active-probe',
observed: nowMs - lineageStartMs,
threshold: INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
})
state.completedAt = recordedAt
}
export async function runIncidentMonitor(
initialState: IncidentMonitorState,
dependencies: IncidentMonitorDependencies
): Promise<IncidentMonitorState> {
const state = structuredClone(initialState)
const lineageStartMs = Date.parse(state.startedAt)
if (!Number.isFinite(lineageStartMs)) {
throw new Error('incident monitor start time is invalid')
}
const checkpoints = checkpointMinutes(state.durationMinutes)
const lineageDeadlineMs = state.preDrainDryRun
? lineageStartMs + INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS
: Number.POSITIVE_INFINITY
const resumedAt = dependencies.now()
const priorSampleMs = state.lastSampleAt
? Date.parse(state.lastSampleAt)
: lineageStartMs
const gapThreshold = state.intervalMs
if (resumedAt - priorSampleMs > gapThreshold) {
resetContinuousWindow(state, new Date(resumedAt).toISOString(), [{
code: 'monitor_gap',
source: 'active-probe',
observed: resumedAt - priorSampleMs,
threshold: gapThreshold
}])
}
if (state.completedAt !== null) {
await dependencies.persist(state)
return state
}
while (state.completedAt === null) {
if (dependencies.now() > lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
const sampleStartedAt = dependencies.now()
let evaluation: IncidentEvaluation
try {
evaluation = evaluateIncidentSample(
await dependencies.collect(),
dependencies.now(),
state.migrationPolicy,
state.recoverySourceCellId,
state.capacityCellId
)
} catch {
evaluation = {
status: 'freeze',
evaluatedAt: new Date(dependencies.now()).toISOString(),
failures: [{
code: 'collector_failed',
source: 'cloud-monitoring'
}]
}
}
state.totalSampleCount++
state.lastSampleAt = evaluation.evaluatedAt
const continuityFailures = evaluation.failures.filter((failure) =>
CONTINUITY_FAILURE_CODES.has(failure.code)
)
const thresholdFailures = evaluation.failures.filter((failure) =>
!CONTINUITY_FAILURE_CODES.has(failure.code)
)
if (continuityFailures.length > 0) {
resetContinuousWindow(state, evaluation.evaluatedAt, continuityFailures)
} else {
if (state.windowStartedAt === null) {
state.windowStartedAt = evaluation.evaluatedAt
}
state.sampleCount++
}
if (thresholdFailures.length > 0) {
state.frozenAt ??= evaluation.evaluatedAt
state.failures = [...state.failures, ...thresholdFailures]
}
if (state.windowStartedAt === null) {
if (dependencies.now() >= lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
await dependencies.persist(state)
await dependencies.wait(
Math.max(0, Math.min(state.intervalMs, lineageDeadlineMs - dependencies.now()))
)
continue
}
const startMs = Date.parse(state.windowStartedAt)
const endMs = startMs + state.durationMinutes * 60_000
const elapsedMinutes = (dependencies.now() - startMs) / 60_000
while (
state.nextCheckpointIndex < checkpoints.length &&
elapsedMinutes >= checkpoints[state.nextCheckpointIndex]!
) {
const minute = checkpoints[state.nextCheckpointIndex]!
await dependencies.checkpoint({
schemaVersion: 4,
incidentId: state.incidentId,
environment: state.environment,
expectedSelector: state.expectedSelector,
preDrainDryRun: state.preDrainDryRun,
migrationPolicy: state.migrationPolicy,
recoverySourceCellId: state.recoverySourceCellId,
capacityCellId: state.capacityCellId,
windowSequence: state.windowSequence,
windowStartedAt: state.windowStartedAt,
checkpointMinute: minute,
scheduledAt: new Date(startMs + minute * 60_000).toISOString(),
recordedAt: new Date(dependencies.now()).toISOString(),
status: state.frozenAt ? 'freeze' : 'green',
frozenAt: state.frozenAt,
sampleCount: state.sampleCount,
failures: state.failures,
thresholds: INCIDENT_MONITOR_THRESHOLDS
})
state.nextCheckpointIndex++
}
if (state.preDrainDryRun && state.frozenAt !== null) {
state.completedAt = new Date(dependencies.now()).toISOString()
await dependencies.persist(state)
break
}
if (dependencies.now() >= endMs) {
state.completedAt = new Date(dependencies.now()).toISOString()
await dependencies.persist(state)
break
}
if (dependencies.now() >= lineageDeadlineMs) {
completeContinuityDeadline(state, dependencies.now(), lineageStartMs)
await dependencies.persist(state)
break
}
await dependencies.persist(state)
await dependencies.wait(
Math.max(
0,
Math.min(sampleStartedAt + state.intervalMs, endMs, lineageDeadlineMs) - dependencies.now()
)
)
}
return state
}
export function preDrainDryRunPassed(state: Pick<
IncidentMonitorState,
| 'completedAt'
| 'durationMinutes'
| 'frozenAt'
| 'intervalMs'
| 'preDrainDryRun'
| 'sampleCount'
| 'startedAt'
>): boolean {
const minimumSamples =
Math.ceil((state.durationMinutes * 60_000) / state.intervalMs) + 1
const lineageElapsedMs = state.completedAt === null
? Number.POSITIVE_INFINITY
: Date.parse(state.completedAt) - Date.parse(state.startedAt)
return (
state.preDrainDryRun &&
state.durationMinutes === 15 &&
state.completedAt !== null &&
lineageElapsedMs >= 0 &&
lineageElapsedMs <= INCIDENT_PRE_DRAIN_MAX_LINEAGE_MS &&
state.frozenAt === null &&
state.sampleCount >= minimumSamples
)
}
@@ -0,0 +1,77 @@
import { z } from 'zod'
export const AdmissionStateSchema = z.enum([
'existing-only',
'migration-only',
'general'
])
export type AdmissionState = z.infer<typeof AdmissionStateSchema>
export const SelectorMembershipSchema = z.object({
existingOnly: z.array(z.string()),
migrationOnly: z.array(z.string()),
general: z.array(z.string())
})
export type SelectorMembership = z.infer<typeof SelectorMembershipSchema>
export const AdmissionSelectorSchema = z.object({
generation: z.number().int().nonnegative(),
membership: SelectorMembershipSchema
})
export type AdmissionSelector = z.infer<typeof AdmissionSelectorSchema>
export function normalizeSelectorMembership(
membership: SelectorMembership,
configuredCellIds: ReadonlySet<string>
): SelectorMembership {
const normalized = {
existingOnly: [...membership.existingOnly].sort(),
migrationOnly: [...membership.migrationOnly].sort(),
general: [...membership.general].sort()
}
const all = [
...normalized.existingOnly,
...normalized.migrationOnly,
...normalized.general
]
if (
all.length !== configuredCellIds.size ||
new Set(all).size !== all.length ||
all.some((cellId) => !configuredCellIds.has(cellId))
) {
throw new Error('selector membership must contain every configured cell exactly once')
}
return normalized
}
export function selectorCellState(
selector: AdmissionSelector,
cellId: string
): AdmissionState {
if (selector.membership.existingOnly.includes(cellId)) return 'existing-only'
if (selector.membership.migrationOnly.includes(cellId)) return 'migration-only'
if (selector.membership.general.includes(cellId)) return 'general'
throw new Error(`selector does not contain ${cellId}`)
}
export function effectiveAdmissionState(
selector: AdmissionSelector,
legacyEnabled: boolean,
cellId: string
): AdmissionState {
if (selector.generation === 0) return legacyEnabled ? 'general' : 'existing-only'
return selectorCellState(selector, cellId)
}
export function exactAdmissionSelector(
actual: AdmissionSelector,
expected: AdmissionSelector
): boolean {
return (
actual.generation === expected.generation &&
JSON.stringify(actual.membership) === JSON.stringify(expected.membership)
)
}
+108
View File
@@ -0,0 +1,108 @@
import { randomBytes, timingSafeEqual } from 'node:crypto'
import { readFile } from 'node:fs/promises'
import { resolve } from 'node:path'
import { serve } from '@hono/node-server'
import { Hono } from 'hono'
import { z } from 'zod'
import { DashboardSnapshotCache } from './dashboard-snapshot.js'
import { createGcloudClient } from './gcloud-client.js'
import {
dispatchStagingPowerWorkflow,
parseStagingPowerRequest
} from './staging-workflow.js'
const QuerySchema = z.object({
environment: z.enum(['production', 'staging']).default('production'),
window: z.coerce.number().int().min(30).max(1440).default(360)
})
const port = z.coerce.number().int().min(1024).max(65_535).parse(process.env.PORT ?? 2455)
const controlsEnabled = process.env.RELAY_OPS_ENABLE_STAGING_CONTROLS === '1'
const csrfToken = randomBytes(32).toString('base64url')
const publicDirectory = resolve(import.meta.dirname, '../public')
const cache = new DashboardSnapshotCache(createGcloudClient())
const app = new Hono()
function safeEqual(left: string, right: string): boolean {
const leftBuffer = Buffer.from(left)
const rightBuffer = Buffer.from(right)
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer)
}
app.use('*', async (context, next) => {
await next()
context.header('Cache-Control', 'no-store')
context.header('Content-Security-Policy', [
"default-src 'self'",
"script-src 'self'",
"style-src 'self'",
"font-src 'self'",
"connect-src 'self'",
"img-src 'self' data:",
"object-src 'none'",
"base-uri 'none'",
"frame-ancestors 'none'",
"form-action 'self'"
].join('; '))
context.header('Referrer-Policy', 'no-referrer')
context.header('X-Content-Type-Options', 'nosniff')
context.header('X-Frame-Options', 'DENY')
})
app.get('/health', (context) => context.json({ status: 'ok' }))
app.get('/api/config', (context) => context.json({
stagingControlsEnabled: controlsEnabled,
csrfToken: controlsEnabled ? csrfToken : null
}))
app.get('/api/snapshot', async (context) => {
const query = QuerySchema.safeParse(context.req.query())
if (!query.success) return context.json({ error: 'Invalid dashboard query' }, 400)
try {
return context.json(await cache.read(query.data.environment, query.data.window))
} catch {
return context.json({
error: 'Relay operations data is unavailable. Check local gcloud and gh authentication.'
}, 503)
}
})
app.post('/api/staging/power', async (context) => {
if (!controlsEnabled) return context.json({ error: 'Staging controls are disabled' }, 403)
const origin = context.req.header('origin')
const expectedOrigin = `http://127.0.0.1:${port}`
if (origin !== expectedOrigin) return context.json({ error: 'Origin rejected' }, 403)
if (!safeEqual(context.req.header('x-csrf-token') ?? '', csrfToken)) {
return context.json({ error: 'Request token rejected' }, 403)
}
try {
const request = parseStagingPowerRequest(await context.req.json())
await dispatchStagingPowerWorkflow(request)
return context.json({ accepted: true })
} catch {
return context.json({ error: 'Invalid or failed staging workflow dispatch' }, 400)
}
})
const staticTypes: Record<string, string> = {
'/app.js': 'text/javascript; charset=utf-8',
'/styles.css': 'text/css; charset=utf-8'
}
for (const [route, contentType] of Object.entries(staticTypes)) {
app.get(route, async (context) => {
const file = route.slice(1)
try {
const content = await readFile(resolve(publicDirectory, file))
return context.body(content, 200, { 'Content-Type': contentType })
} catch {
return context.notFound()
}
})
}
app.get('/', async (context) => {
const html = await readFile(resolve(publicDirectory, 'index.html'), 'utf8')
return context.html(html)
})
serve({ fetch: app.fetch, hostname: '127.0.0.1', port }, () => {
// Loopback is intentional; operators may add authenticated Tailscale Serve separately.
console.log(`Orca Relay Operations: http://127.0.0.1:${port}`)
console.log(`Staging controls: ${controlsEnabled ? 'enabled through GitHub workflow' : 'read-only'}`)
})
@@ -0,0 +1,85 @@
import { describe, expect, it } from 'vitest'
import { RELAY_METRICS, readMonitoringSnapshot } from './monitoring-snapshot.js'
import type { GcloudClient } from './gcloud-client.js'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
const gcloud: GcloudClient = {
accessToken: async () => 'a'.repeat(40)
}
function distribution(at: string, mean: number | undefined, count = 1) {
return {
interval: { endTime: at },
value: { distributionValue: { count, ...(mean === undefined ? {} : { mean }) } }
}
}
describe('readMonitoringSnapshot', () => {
it('aggregates gauge series per minute and distribution deltas by count', async () => {
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
if (url.pathname.endsWith('/alertPolicies')) {
return Response.json({ alertPolicies: [] })
}
const filter = url.searchParams.get('filter') ?? ''
if (filter.includes('orca_relay_controls')) {
return Response.json({ timeSeries: [
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
points: [
distribution('2026-07-15T12:00:10Z', 1),
distribution('2026-07-15T12:00:50Z', 2)
]
},
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'two' } },
points: [distribution('2026-07-15T12:00:20Z', 3)]
},
{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'stale' } },
points: [distribution('2026-07-15T11:59:20Z', 100)]
}
] })
}
if (filter.includes('orca_relay_forwarded_bytes')) {
return Response.json({ timeSeries: [{
metric: { labels: { cell_id: 'production-gce-c1' } },
resource: { type: 'gce_instance', labels: { instance_id: 'one' } },
points: [distribution('2026-07-15T12:00:20Z', 10, 4)]
}] })
}
return Response.json({ timeSeries: [] })
}
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, gcloud, {
now: new Date('2026-07-15T12:01:00Z'),
windowMinutes: 30,
fetchImpl
})
expect(result.warnings).toEqual([])
expect(result.metrics.controls.points).toEqual([
{ at: '2026-07-15T11:59:00.000Z', value: 100 },
{ at: '2026-07-15T12:00:00.000Z', value: 5 }
])
expect(result.metrics.controls.latestByCell).toEqual({ 'production-gce-c1': 5 })
expect(result.metrics.forwarded_bytes.latest).toBe(40)
expect(result.metrics.postgres_retries.available).toBe(true)
expect(Object.keys(result.metrics)).toHaveLength(RELAY_METRICS.length)
})
it('degrades safely when credentials are unavailable', async () => {
const unavailable: GcloudClient = {
accessToken: async () => { throw new Error('sensitive context') }
}
const result = await readMonitoringSnapshot(RELAY_OPS_ENVIRONMENTS.production, unavailable)
expect(result.warnings).toEqual([
'Cloud Monitoring credentials are unavailable. Run gcloud auth login.'
])
expect(result.metrics.postgres_retries.available).toBe(false)
expect(JSON.stringify(result)).not.toContain('sensitive context')
})
})
@@ -0,0 +1,379 @@
import { z } from 'zod'
import type { RelayOpsEnvironment } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
type MetricMode = 'gauge-sum' | 'delta-sum' | 'maximum'
export type RelayMetricName =
| 'total_connections'
| 'controls'
| 'splices'
| 'pending_splices'
| 'queued_bytes'
| 'http_latency_ms'
| 'sql_latency_ms'
| 'heap_used_bytes'
| 'event_loop_ms_p99'
| 'forwarded_bytes'
| 'auth_successes'
| 'auth_failures'
| 'reconnects'
| 'sql_queries'
| 'sql_failures'
| 'assignment_5xx'
| 'postgres_retries'
| 'postgres_retry_exhausted'
| 'db_pool_total'
| 'db_pool_idle'
| 'db_pool_waiting'
| 'db_waiters_max'
| 'db_oldest_wait_ms'
| 'db_wait_ms_max'
type MetricDefinition = {
name: RelayMetricName
label: string
unit: 'count' | 'bytes' | 'milliseconds'
mode: MetricMode
}
export const RELAY_METRICS: MetricDefinition[] = [
{ name: 'total_connections', label: 'Connections', unit: 'count', mode: 'gauge-sum' },
{ name: 'controls', label: 'Desktop controls', unit: 'count', mode: 'gauge-sum' },
{ name: 'splices', label: 'Phone splices', unit: 'count', mode: 'gauge-sum' },
{ name: 'pending_splices', label: 'Pending splices', unit: 'count', mode: 'gauge-sum' },
{ name: 'queued_bytes', label: 'Queued bytes', unit: 'bytes', mode: 'maximum' },
{ name: 'http_latency_ms', label: 'HTTP latency', unit: 'milliseconds', mode: 'maximum' },
{ name: 'sql_latency_ms', label: 'SQL latency', unit: 'milliseconds', mode: 'maximum' },
{ name: 'heap_used_bytes', label: 'Heap used', unit: 'bytes', mode: 'maximum' },
{
name: 'event_loop_ms_p99',
label: 'Event-loop p99',
unit: 'milliseconds',
mode: 'maximum'
},
{ name: 'forwarded_bytes', label: 'Forwarded bytes', unit: 'bytes', mode: 'delta-sum' },
{ name: 'auth_successes', label: 'Auth successes', unit: 'count', mode: 'delta-sum' },
{ name: 'auth_failures', label: 'Auth failures', unit: 'count', mode: 'delta-sum' },
{ name: 'reconnects', label: 'Reconnects', unit: 'count', mode: 'delta-sum' },
{ name: 'sql_queries', label: 'SQL queries', unit: 'count', mode: 'delta-sum' },
{ name: 'sql_failures', label: 'SQL failures', unit: 'count', mode: 'delta-sum' },
{ name: 'assignment_5xx', label: 'Assignment 5xx', unit: 'count', mode: 'delta-sum' },
{
name: 'postgres_retries',
label: 'PostgreSQL retries',
unit: 'count',
mode: 'delta-sum'
},
{
name: 'postgres_retry_exhausted',
label: 'PostgreSQL retry exhausted',
unit: 'count',
mode: 'delta-sum'
},
{ name: 'db_pool_total', label: 'Database pool total', unit: 'count', mode: 'gauge-sum' },
{ name: 'db_pool_idle', label: 'Database pool idle', unit: 'count', mode: 'gauge-sum' },
{
name: 'db_pool_waiting',
label: 'Database pool waiting',
unit: 'count',
mode: 'gauge-sum'
},
{
name: 'db_waiters_max',
label: 'Database waiters max',
unit: 'count',
mode: 'maximum'
},
{
name: 'db_oldest_wait_ms',
label: 'Database oldest wait',
unit: 'milliseconds',
mode: 'maximum'
},
{
name: 'db_wait_ms_max',
label: 'Database wait max',
unit: 'milliseconds',
mode: 'maximum'
}
]
const NumericSchema = z.union([z.number(), z.string()]).transform((value) => Number(value))
const DistributionSchema = z.object({
count: NumericSchema.default(0),
mean: NumericSchema.optional()
})
const PointSchema = z.object({
interval: z.object({ endTime: z.string() }),
value: z.object({
doubleValue: NumericSchema.optional(),
int64Value: NumericSchema.optional(),
distributionValue: DistributionSchema.optional()
})
})
const TimeSeriesSchema = z.object({
metric: z.object({ labels: z.record(z.string()).default({}) }),
resource: z.object({ type: z.string(), labels: z.record(z.string()).default({}) }),
points: z.array(PointSchema).default([])
})
const TimeSeriesResponseSchema = z.object({
timeSeries: z.array(TimeSeriesSchema).default([])
})
export type MetricPoint = { at: string; value: number }
export type RelayMetricSnapshot = MetricDefinition & {
available: boolean
points: MetricPoint[]
latest: number | null
latestAt: string | null
latestByCell: Record<string, number>
}
export type AlertPolicySnapshot = {
id: string
displayName: string
enabled: boolean
documentation: string | null
}
export type MonitoringSnapshot = {
startAt: string
endAt: string
resolutionSeconds: number
metrics: Record<RelayMetricName, RelayMetricSnapshot>
alertPolicies: AlertPolicySnapshot[]
warnings: string[]
}
type ParsedPoint = {
atMs: number
bucketMs: number
value: number
sampleTotal: number
seriesKey: string
cellId: string
}
function pointValue(point: z.infer<typeof PointSchema>): { value: number; sampleTotal: number } {
if (point.value.distributionValue) {
const count = point.value.distributionValue.count
const value = point.value.distributionValue.mean ?? 0
return { value, sampleTotal: value * count }
}
const value = point.value.doubleValue ?? point.value.int64Value ?? 0
return { value, sampleTotal: value }
}
function parsePoints(series: z.infer<typeof TimeSeriesSchema>[]): ParsedPoint[] {
return series.flatMap((entry) => {
const cellId = entry.metric.labels.cell_id ?? 'unknown'
const resourceId =
entry.resource.labels.instance_id ?? entry.resource.labels.revision_name ?? entry.resource.type
const seriesKey = `${cellId}:${resourceId}`
return entry.points.flatMap((point) => {
const atMs = Date.parse(point.interval.endTime)
if (!Number.isFinite(atMs)) return []
const values = pointValue(point)
return [{
atMs,
bucketMs: Math.floor(atMs / 60_000) * 60_000,
value: values.value,
sampleTotal: values.sampleTotal,
seriesKey,
cellId
}]
})
})
}
function aggregatePoints(points: ParsedPoint[], mode: MetricMode): MetricPoint[] {
if (mode === 'gauge-sum') {
const buckets = new Map<number, Map<string, ParsedPoint>>()
for (const point of points) {
const bySeries = buckets.get(point.bucketMs) ?? new Map<string, ParsedPoint>()
const previous = bySeries.get(point.seriesKey)
if (!previous || previous.atMs < point.atMs) bySeries.set(point.seriesKey, point)
buckets.set(point.bucketMs, bySeries)
}
return [...buckets.entries()]
.sort(([left], [right]) => left - right)
.map(([at, bySeries]) => ({
at: new Date(at).toISOString(),
value: [...bySeries.values()].reduce((total, point) => total + point.value, 0)
}))
}
const buckets = new Map<number, number>()
for (const point of points) {
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
const previous = buckets.get(point.bucketMs)
buckets.set(
point.bucketMs,
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
)
}
return [...buckets.entries()]
.sort(([left], [right]) => left - right)
.map(([at, value]) => ({ at: new Date(at).toISOString(), value }))
}
function latestByCell(points: ParsedPoint[], mode: MetricMode): Record<string, number> {
const newestBucketByCell = new Map<string, number>()
for (const point of points) {
newestBucketByCell.set(
point.cellId,
Math.max(newestBucketByCell.get(point.cellId) ?? 0, point.bucketMs)
)
}
const newestBySeries = new Map<string, ParsedPoint>()
for (const point of points) {
if (point.bucketMs !== newestBucketByCell.get(point.cellId)) continue
const previous = newestBySeries.get(point.seriesKey)
if (!previous || previous.atMs < point.atMs) newestBySeries.set(point.seriesKey, point)
}
const totals = new Map<string, number>()
for (const point of newestBySeries.values()) {
const value = mode === 'delta-sum' ? point.sampleTotal : point.value
const previous = totals.get(point.cellId)
totals.set(
point.cellId,
mode === 'maximum' ? Math.max(previous ?? 0, value) : (previous ?? 0) + value
)
}
return Object.fromEntries(totals)
}
async function monitoringRequest(
fetchImpl: typeof fetch,
token: string,
url: URL
): Promise<unknown> {
const response = await fetchImpl(url, {
headers: { authorization: `Bearer ${token}` },
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Cloud Monitoring returned ${response.status}`)
return await response.json()
}
async function readMetric(
environment: RelayOpsEnvironment,
definition: MetricDefinition,
token: string,
startAt: string,
endAt: string,
fetchImpl: typeof fetch
): Promise<RelayMetricSnapshot> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/timeSeries`
)
url.searchParams.set(
'filter',
`metric.type="logging.googleapis.com/user/orca_relay_${definition.name}"`
)
url.searchParams.set('interval.startTime', startAt)
url.searchParams.set('interval.endTime', endAt)
url.searchParams.set('view', 'FULL')
url.searchParams.set('pageSize', '1000')
const body = TimeSeriesResponseSchema.parse(
await monitoringRequest(fetchImpl, token, url)
)
const parsed = parsePoints(body.timeSeries)
const points = aggregatePoints(parsed, definition.mode)
const latest = points.at(-1) ?? null
return {
...definition,
available: true,
points,
latest: latest?.value ?? null,
latestAt: latest?.at ?? null,
latestByCell: latestByCell(parsed, definition.mode)
}
}
async function readAlertPolicies(
environment: RelayOpsEnvironment,
token: string,
fetchImpl: typeof fetch
): Promise<AlertPolicySnapshot[]> {
const url = new URL(
`https://monitoring.googleapis.com/v3/projects/${environment.project}/alertPolicies`
)
url.searchParams.set('pageSize', '100')
const body = (await monitoringRequest(fetchImpl, token, url)) as {
alertPolicies?: Array<{
name?: string
displayName?: string
enabled?: boolean
documentation?: { content?: string }
}>
}
return (body.alertPolicies ?? [])
.filter((policy) => policy.displayName?.startsWith('Orca Relay:'))
.map((policy) => ({
id: policy.name ?? '',
displayName: policy.displayName ?? 'Orca Relay alert',
enabled: policy.enabled === true,
documentation: policy.documentation?.content ?? null
}))
.sort((left, right) => left.displayName.localeCompare(right.displayName))
}
function emptyMetric(definition: MetricDefinition): RelayMetricSnapshot {
return {
...definition,
available: false,
points: [],
latest: null,
latestAt: null,
latestByCell: {}
}
}
export async function readMonitoringSnapshot(
environment: RelayOpsEnvironment,
gcloud: GcloudClient,
options: { now?: Date; windowMinutes?: number; fetchImpl?: typeof fetch } = {}
): Promise<MonitoringSnapshot> {
const now = options.now ?? new Date()
const windowMinutes = Math.min(24 * 60, Math.max(30, options.windowMinutes ?? 360))
const endAt = now.toISOString()
const startAt = new Date(now.getTime() - windowMinutes * 60_000).toISOString()
const fetchImpl = options.fetchImpl ?? fetch
const warnings: string[] = []
let token: string
try {
token = await gcloud.accessToken()
} catch {
return {
startAt,
endAt,
resolutionSeconds: 60,
metrics: Object.fromEntries(
RELAY_METRICS.map((definition) => [definition.name, emptyMetric(definition)])
) as Record<RelayMetricName, RelayMetricSnapshot>,
alertPolicies: [],
warnings: ['Cloud Monitoring credentials are unavailable. Run gcloud auth login.']
}
}
const settled = await Promise.allSettled(
RELAY_METRICS.map((definition) =>
readMetric(environment, definition, token, startAt, endAt, fetchImpl)
)
)
const metrics = {} as Record<RelayMetricName, RelayMetricSnapshot>
settled.forEach((result, index) => {
const definition = RELAY_METRICS[index]!
if (result.status === 'fulfilled') metrics[definition.name] = result.value
else {
metrics[definition.name] = emptyMetric(definition)
warnings.push(`${definition.label} metric is unavailable.`)
}
})
const alertPolicies = await readAlertPolicies(environment, token, fetchImpl).catch(() => {
warnings.push('Cloud Monitoring alert policies are unavailable.')
return []
})
return { startAt, endAt, resolutionSeconds: 60, metrics, alertPolicies, warnings }
}
@@ -0,0 +1,42 @@
import { readdirSync, readFileSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import {
RELAY_GITHUB_REPOSITORY,
RELAY_WORKFLOW_FILE_PREFIX,
relayRepositoryApiPath,
relayWorkflowFile
} from './relay-repository.js'
const sourceDir = fileURLToPath(new URL('.', import.meta.url))
const sources = readdirSync(sourceDir)
.filter((name) => name.endsWith('.ts') && name !== 'relay-repository.ts')
.map((name) => ({ name, text: readFileSync(`${sourceDir}${name}`, 'utf8') }))
describe('relay repository identity', () => {
it('builds API paths and workflow filenames from the one repository name', () => {
expect(relayRepositoryApiPath('actions/runs')).toBe(`repos/${RELAY_GITHUB_REPOSITORY}/actions/runs`)
expect(relayWorkflowFile('power-relay-staging.yml')).toBe(
`${RELAY_WORKFLOW_FILE_PREFIX}power-relay-staging.yml`
)
})
// Why: the public-repo copy renames the repository and prefixes every workflow file. Both have to
// be one edit, so no other module may restate either.
it('is the only module naming a GitHub repository', () => {
for (const { name, text } of sources) {
expect(text, `${name} restates a GitHub repository`).not.toMatch(/stablyai\//)
}
})
it('is the only module naming a workflow file', () => {
for (const { name, text } of sources) {
for (const match of text.matchAll(/'([^']*\.yml)'/g)) {
const file = match[1] ?? ''
expect(text, `${name} names ${file} outside relayWorkflowFile`).toMatch(
new RegExp(`relayWorkflowFile\\('${file.replaceAll('.', '\\.')}'\\)`)
)
}
}
})
})
@@ -0,0 +1,14 @@
// Single place naming the GitHub repository that holds the Relay workflows. When the Relay tree is
// copied to its public repository, only this file changes: the repository moves and every workflow
// file gains a prefix, while the workflow display names stay as they are.
export const RELAY_GITHUB_REPOSITORY = 'stablyai/orca-cloud'
export const RELAY_WORKFLOW_FILE_PREFIX = ''
export function relayWorkflowFile(name: string): string {
return `${RELAY_WORKFLOW_FILE_PREFIX}${name}`
}
export function relayRepositoryApiPath(resource: string): string {
return `repos/${RELAY_GITHUB_REPOSITORY}/${resource}`
}
@@ -0,0 +1,150 @@
import { describe, expect, it } from 'vitest'
import { RELAY_OPS_ENVIRONMENTS } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { probeEndpointHealth, readResourceInventory } from './resource-inventory.js'
const digest = `sha256:${'a'.repeat(64)}`
const runService = {
template: {
scaling: { minInstanceCount: 0, maxInstanceCount: 2 },
containers: [{ image: 'registry/image:tag' }]
},
conditions: [{ state: 'CONDITION_SUCCEEDED' }],
latestReadyRevision: 'projects/project/revisions/revision-one'
}
describe('readResourceInventory', () => {
it('does not delay a healthy endpoint sample', async () => {
let calls = 0
let waits = 0
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async () => {
calls += 1
return new Response(null, { status: 200 })
},
async () => {
waits += 1
}
)
expect(result.health).toBe(true)
expect(result.ready).toBe(true)
expect(calls).toBe(2)
expect(waits).toBe(0)
})
it('retries one transient endpoint failure within the same sample', async () => {
const calls = new Map<string, number>()
const waits: number[] = []
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async (input) => {
const path = new URL(String(input)).pathname
const call = (calls.get(path) ?? 0) + 1
calls.set(path, call)
return new Response(null, { status: path === '/ready' && call === 1 ? 503 : 200 })
},
async (ms) => {
waits.push(ms)
}
)
expect(result.health).toBe(true)
expect(result.ready).toBe(true)
expect(calls).toEqual(new Map([['/health', 2], ['/ready', 2]]))
expect(waits).toEqual([11_000])
})
it('fails closed when the endpoint retry is also unhealthy', async () => {
let calls = 0
const waits: number[] = []
const result = await probeEndpointHealth(
'https://c9.relay.onorca.dev',
async () => {
calls += 1
return new Response(null, { status: 503 })
},
async (ms) => {
waits.push(ms)
}
)
expect(result.health).toBe(false)
expect(result.ready).toBe(false)
expect(calls).toBe(4)
expect(waits).toEqual([11_000])
})
it('uses aggregate REST inventory without probing sleeping staging endpoints', async () => {
const gcloud: GcloudClient = { accessToken: async () => 'a'.repeat(40) }
let publicProbeCalls = 0
const fetchImpl: typeof fetch = async (input) => {
const url = new URL(String(input))
if (url.hostname.endsWith('onorca.dev')) {
publicProbeCalls += 1
return Response.json({ status: 'ok' })
}
if (url.hostname === 'run.googleapis.com') return Response.json(runService)
if (url.hostname === 'sqladmin.googleapis.com') return Response.json({
state: 'STOPPED',
databaseVersion: 'POSTGRES_17',
settings: {
activationPolicy: 'NEVER',
availabilityType: 'ZONAL',
tier: 'db-custom-1-3840'
}
})
if (url.hostname === 'certificatemanager.googleapis.com') return Response.json({
managed: { domains: ['*.relay-staging.onorca.dev'], state: 'ACTIVE' }
})
if (url.pathname.includes('/instanceGroupManagers/')) {
const name = url.pathname.split('/').at(-1)!
return Response.json({
name,
targetSize: 0,
size: '0',
instanceGroup: `projects/project/zones/zone/instanceGroups/${name}`,
instanceTemplate: `projects/project/global/instanceTemplates/template-${name}`,
status: { isStable: true }
})
}
if (url.pathname.includes('/instanceTemplates/')) return Response.json({
properties: { metadata: { items: [{
key: 'startup-script',
value: `SECRET_TEXT\nORCA_RELAY_IMAGE_DIGEST=%s\\n' '${digest}'`
}] } }
})
if (url.pathname.endsWith('/getHealth')) return Response.json([])
throw new Error(`Unexpected request to ${url.hostname}${url.pathname}`)
}
const result = await readResourceInventory(
RELAY_OPS_ENVIRONMENTS.staging,
gcloud,
fetchImpl
)
expect(publicProbeCalls).toBe(0)
expect(result.cells.every((cell) => cell.targetSize === 0)).toBe(true)
expect(result.cells.every((cell) => cell.endpoint.health === null)).toBe(true)
expect(result.cells.every((cell) => cell.imageDigest === digest)).toBe(true)
expect(JSON.stringify(result)).not.toContain('SECRET_TEXT')
})
it('represents missing credentials as unknown inventory, never sleeping', async () => {
const gcloud: GcloudClient = {
accessToken: async () => { throw new Error('sensitive context') }
}
let fetchCalls = 0
const result = await readResourceInventory(
RELAY_OPS_ENVIRONMENTS.production,
gcloud,
async () => { fetchCalls += 1; return Response.json({}) }
)
expect(fetchCalls).toBe(0)
expect(result.cells.every((cell) => cell.targetSize === null)).toBe(true)
expect(result.cells.every((cell) => cell.backendHealth === 'unknown')).toBe(true)
expect(JSON.stringify(result)).not.toContain('sensitive context')
})
})
@@ -0,0 +1,366 @@
import { z } from 'zod'
import type { RelayOpsEnvironment, RelayOpsCellConfig } from './environment-config.js'
import type { GcloudClient } from './gcloud-client.js'
import { INCIDENT_MONITOR_THRESHOLDS } from './incident-monitor.js'
const RunServiceSchema = z.object({
template: z.object({
scaling: z.object({
minInstanceCount: z.number().optional(),
maxInstanceCount: z.number().optional()
}).optional(),
containers: z.array(z.object({ image: z.string() })).min(1)
}),
conditions: z.array(z.object({ state: z.string() })).default([]),
latestReadyRevision: z.string().optional()
})
const SqlInstanceSchema = z.object({
state: z.string(),
databaseVersion: z.string(),
settings: z.object({
activationPolicy: z.string(),
availabilityType: z.string().optional(),
tier: z.string()
})
})
const MigSchema = z.object({
name: z.string(),
targetSize: z.number(),
size: z.union([z.string(), z.number()]).transform(Number).optional(),
instanceGroup: z.string(),
instanceTemplate: z.string(),
status: z.object({ isStable: z.boolean().default(false) }).default({ isStable: false })
})
const TemplateSchema = z.object({
properties: z.object({
metadata: z.object({
items: z.array(z.object({ key: z.string(), value: z.string().optional() })).default([])
}).optional()
})
})
const BackendHealthGroupSchema = z.object({
healthStatus: z.array(z.object({ healthState: z.string() })).default([])
})
const BackendHealthSchema = z.union([
BackendHealthGroupSchema,
z.array(z.object({ status: BackendHealthGroupSchema }))
])
const CertificateSchema = z.object({
expireTime: z.string().optional(),
managed: z.object({
domains: z.array(z.string()).default([]),
state: z.string()
})
})
export type EndpointHealth = {
health: boolean | null
ready: boolean | null
latencyMs: number | null
}
export type ServiceInventory = {
ready: boolean
revision: string | null
image: string
minInstances: number
maxInstances: number
}
export type CellInventory = RelayOpsCellConfig & {
migName: string
targetSize: number | null
runningInstances: number | null
stable: boolean | null
template: string | null
imageDigest: string | null
backendHealth: 'healthy' | 'unhealthy' | 'empty' | 'unknown'
endpoint: EndpointHealth
}
export type ResourceInventory = {
director: ServiceInventory | null
auth: ServiceInventory | null
sql: {
state: string
activationPolicy: string
tier: string
availabilityType: string
databaseVersion: string
} | null
certificate: { state: string; domains: string[]; expireTime: string | null } | null
directorEndpoint: EndpointHealth
authEndpoint: EndpointHealth
cells: CellInventory[]
warnings: string[]
}
const unavailableEndpoint = (): EndpointHealth => ({ health: null, ready: null, latencyMs: null })
const independentEndpointRetryDelayMs = 11_000
function finalSegment(value: string): string {
return value.split('/').at(-1) ?? value
}
function parseService(value: unknown): ServiceInventory {
const service = RunServiceSchema.parse(value)
return {
ready: service.conditions.length > 0 && service.conditions.every(
(condition) => condition.state === 'CONDITION_SUCCEEDED'
),
revision: service.latestReadyRevision ? finalSegment(service.latestReadyRevision) : null,
image: service.template.containers[0]!.image,
minInstances: service.template.scaling?.minInstanceCount ?? 0,
maxInstances: service.template.scaling?.maxInstanceCount ?? 0
}
}
async function googleRequest(
fetchImpl: typeof fetch,
token: string,
url: string,
init: RequestInit = {}
): Promise<unknown> {
const response = await fetchImpl(url, {
...init,
headers: {
authorization: `Bearer ${token}`,
...(init.body ? { 'content-type': 'application/json' } : {})
},
signal: AbortSignal.timeout(30_000)
})
if (!response.ok) throw new Error(`Google API returned ${response.status}`)
return await response.json()
}
async function endpointProbe(origin: string, fetchImpl: typeof fetch): Promise<EndpointHealth> {
const startedAt = performance.now()
const check = async (path: '/health' | '/ready'): Promise<boolean> => {
try {
const response = await fetchImpl(`${origin}${path}`, {
redirect: 'error',
signal: AbortSignal.timeout(8_000)
})
return response.ok
} catch {
return false
}
}
const [health, ready] = await Promise.all([check('/health'), check('/ready')])
return { health, ready, latencyMs: Math.round(performance.now() - startedAt) }
}
export async function probeEndpointHealth(
origin: string,
fetchImpl: typeof fetch,
wait: (ms: number) => Promise<void> = async (ms) =>
await new Promise((resolvePromise) => setTimeout(resolvePromise, ms))
): Promise<EndpointHealth> {
const first = await endpointProbe(origin, fetchImpl)
if (
first.health &&
first.ready &&
first.latencyMs !== null &&
first.latencyMs <= INCIDENT_MONITOR_THRESHOLDS.endpointLatencyMs
) {
return first
}
// Outwait Relay's ten-second readiness cache before treating the retry as independent.
await wait(independentEndpointRetryDelayMs)
return await endpointProbe(origin, fetchImpl)
}
function imageDigest(template: z.infer<typeof TemplateSchema>): string | null {
const startupScript = template.properties.metadata?.items.find(
(item) => item.key === 'startup-script'
)?.value
// Return only the immutable digest; startup metadata contains secret names and operational detail.
return startupScript?.match(/ORCA_RELAY_IMAGE_DIGEST=%s\\n' '(sha256:[a-f0-9]{64})'/)?.[1] ?? null
}
function backendState(value: unknown): CellInventory['backendHealth'] {
const parsedHealth = BackendHealthSchema.parse(value)
const groups = Array.isArray(parsedHealth)
? parsedHealth.map((group) => group.status)
: [parsedHealth]
const states = groups.flatMap((group) => group.healthStatus.map((status) => status.healthState))
if (states.length === 0) return 'empty'
if (states.every((state) => state === 'HEALTHY')) return 'healthy'
return 'unhealthy'
}
function unavailableCell(cell: RelayOpsCellConfig, migName: string): CellInventory {
return {
...cell,
migName,
targetSize: null,
runningInstances: null,
stable: null,
template: null,
imageDigest: null,
backendHealth: 'unknown',
endpoint: unavailableEndpoint()
}
}
async function readCell(
environment: RelayOpsEnvironment,
cell: RelayOpsCellConfig,
mig: z.infer<typeof MigSchema> | null,
token: string,
fetchImpl: typeof fetch
): Promise<CellInventory> {
const migName = `${environment.migPrefix}${cell.hostname}`
if (!mig) return unavailableCell(cell, migName)
// An empty fixed-one MIG cannot serve and must never be woken by observation.
const endpoint = mig.targetSize > 0
? await probeEndpointHealth(cell.origin, fetchImpl)
: unavailableEndpoint()
const templateName = finalSegment(mig.instanceTemplate)
const [templateResult, healthResult] = await Promise.allSettled([
googleRequest(
fetchImpl,
token,
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/instanceTemplates/${templateName}`
),
googleRequest(
fetchImpl,
token,
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/global/backendServices/${migName}/getHealth`,
{ method: 'POST', body: JSON.stringify({ group: mig.instanceGroup }) }
)
])
return {
...cell,
migName,
targetSize: mig.targetSize,
runningInstances: mig.size ?? (mig.status.isStable ? mig.targetSize : null),
stable: mig.status.isStable,
template: templateName,
imageDigest: templateResult.status === 'fulfilled'
? imageDigest(TemplateSchema.parse(templateResult.value))
: null,
backendHealth: healthResult.status === 'fulfilled'
? backendState(healthResult.value)
: 'unknown',
endpoint
}
}
function parsed<S extends z.ZodTypeAny>(
result: PromiseSettledResult<unknown>,
schema: S,
warning: string,
warnings: string[]
): z.infer<S> | null {
if (result.status === 'rejected') {
warnings.push(warning)
return null
}
const parsedValue = schema.safeParse(result.value)
if (!parsedValue.success) {
warnings.push(warning)
return null
}
return parsedValue.data
}
function unavailableInventory(environment: RelayOpsEnvironment, warning: string): ResourceInventory {
return {
director: null,
auth: null,
sql: null,
certificate: null,
directorEndpoint: unavailableEndpoint(),
authEndpoint: unavailableEndpoint(),
cells: environment.cells.map((cell) =>
unavailableCell(cell, `${environment.migPrefix}${cell.hostname}`)
),
warnings: [warning]
}
}
export async function readResourceInventory(
environment: RelayOpsEnvironment,
gcloud: GcloudClient,
fetchImpl: typeof fetch = fetch
): Promise<ResourceInventory> {
let token: string
try {
token = await gcloud.accessToken()
} catch {
return unavailableInventory(
environment,
'Google Cloud credentials are unavailable. Run gcloud auth login.'
)
}
const runUrl = (service: string) =>
`https://run.googleapis.com/v2/projects/${environment.project}/locations/${environment.region}/services/${service}`
const migUrl = (cell: RelayOpsCellConfig) =>
`https://compute.googleapis.com/compute/v1/projects/${environment.project}/zones/${cell.zone}/instanceGroupManagers/${environment.migPrefix}${cell.hostname}`
const settled = await Promise.allSettled([
googleRequest(fetchImpl, token, runUrl(environment.directorService)),
googleRequest(fetchImpl, token, runUrl(environment.authService)),
googleRequest(
fetchImpl,
token,
`https://sqladmin.googleapis.com/sql/v1beta4/projects/${environment.project}/instances/${environment.sqlInstance}`
),
googleRequest(
fetchImpl,
token,
`https://certificatemanager.googleapis.com/v1/projects/${environment.project}/locations/global/certificates/${environment.certificateName}`
),
...environment.cells.map((cell) => googleRequest(fetchImpl, token, migUrl(cell)))
])
const warnings: string[] = []
const directorValue = parsed(settled[0]!, RunServiceSchema, 'Director service inventory is unavailable.', warnings)
const authValue = parsed(settled[1]!, RunServiceSchema, 'Auth service inventory is unavailable.', warnings)
const sqlValue = parsed(settled[2]!, SqlInstanceSchema, 'Cloud SQL inventory is unavailable.', warnings)
const certificateValue = parsed(
settled[3]!, CertificateSchema, 'TLS certificate inventory is unavailable.', warnings
)
const migValues = environment.cells.map((cell, index) => parsed(
settled[index + 4]!,
MigSchema,
`${cell.hostname.toUpperCase()} MIG inventory is unavailable.`,
warnings
))
const controlPlaneSleeping =
environment.id === 'staging' && sqlValue?.settings.activationPolicy === 'NEVER'
// Health probes would cold-start scale-to-zero Cloud Run services, so sleeping staging is inventory-only.
const [directorEndpoint, authEndpoint] = controlPlaneSleeping
? [unavailableEndpoint(), unavailableEndpoint()]
: await Promise.all([
probeEndpointHealth(environment.directorOrigin, fetchImpl),
probeEndpointHealth(environment.authOrigin, fetchImpl)
])
const cells = await Promise.all(environment.cells.map((cell, index) =>
readCell(environment, cell, migValues[index] ?? null, token, fetchImpl)
))
return {
director: directorValue ? parseService(directorValue) : null,
auth: authValue ? parseService(authValue) : null,
sql: sqlValue ? {
state: sqlValue.state,
activationPolicy: sqlValue.settings.activationPolicy,
tier: sqlValue.settings.tier,
availabilityType: sqlValue.settings.availabilityType ?? 'unknown',
databaseVersion: sqlValue.databaseVersion
} : null,
certificate: certificateValue ? {
state: certificateValue.managed.state,
domains: certificateValue.managed.domains,
expireTime: certificateValue.expireTime ?? null
} : null,
directorEndpoint,
authEndpoint,
cells,
warnings
}
}
@@ -0,0 +1,19 @@
import { describe, expect, it } from 'vitest'
import { parseStagingPowerRequest } from './staging-workflow.js'
describe('parseStagingPowerRequest', () => {
it('accepts the exact reviewed confirmations', () => {
expect(parseStagingPowerRequest({ mode: 'status', confirmation: '' })).toEqual({
mode: 'status', confirmation: ''
})
expect(parseStagingPowerRequest({ mode: 'wake', confirmation: 'WAKE_STAGING' }).mode).toBe('wake')
expect(parseStagingPowerRequest({ mode: 'sleep', confirmation: 'SLEEP_STAGING' }).mode).toBe('sleep')
})
it('rejects missing, swapped, or additional fields', () => {
expect(() => parseStagingPowerRequest({ mode: 'wake', confirmation: '' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'sleep', confirmation: 'WAKE_STAGING' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'production', confirmation: '' })).toThrow()
expect(() => parseStagingPowerRequest({ mode: 'status', confirmation: '', project: 'other' })).toThrow()
})
})
@@ -0,0 +1,36 @@
import { execFile } from 'node:child_process'
import { promisify } from 'node:util'
import { z } from 'zod'
import { RELAY_GITHUB_REPOSITORY, relayWorkflowFile } from './relay-repository.js'
const execFileAsync = promisify(execFile)
const DispatchSchema = z.discriminatedUnion('mode', [
z.object({ mode: z.literal('status'), confirmation: z.literal('') }).strict(),
z.object({ mode: z.literal('wake'), confirmation: z.literal('WAKE_STAGING') }).strict(),
z.object({ mode: z.literal('sleep'), confirmation: z.literal('SLEEP_STAGING') }).strict()
])
export type StagingPowerRequest = z.infer<typeof DispatchSchema>
export function parseStagingPowerRequest(value: unknown): StagingPowerRequest {
return DispatchSchema.parse(value)
}
export async function dispatchStagingPowerWorkflow(request: StagingPowerRequest): Promise<void> {
const args = [
'workflow', 'run', relayWorkflowFile('power-relay-staging.yml'),
'--repo', RELAY_GITHUB_REPOSITORY,
'-f', `mode=${request.mode}`,
'-f', 'wake-cells=configured'
]
if (request.confirmation) args.push('-f', `confirmation=${request.confirmation}`)
try {
await execFileAsync('gh', args, {
encoding: 'utf8',
timeout: 30_000,
maxBuffer: 1024 * 1024
})
} catch {
throw new Error('Staging power workflow dispatch failed')
}
}
+4
View File
@@ -0,0 +1,4 @@
{
"extends": "./tsconfig.json",
"exclude": ["src/**/*.test.ts"]
}
+11
View File
@@ -0,0 +1,11 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "dist",
"types": ["node", "vitest"],
"exactOptionalPropertyTypes": true,
"verbatimModuleSyntax": true
},
"include": ["src/**/*.ts"]
}
+25
View File
@@ -0,0 +1,25 @@
FROM node:24-alpine AS build
WORKDIR /app
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml tsconfig.base.json ./
COPY packages/relay-contract/package.json packages/relay-contract/package.json
COPY apps/relay/package.json apps/relay/package.json
RUN pnpm install --frozen-lockfile
COPY packages/relay-contract packages/relay-contract
COPY apps/relay apps/relay
RUN pnpm --filter @orca-cloud/relay-contract build && pnpm --filter @orca-cloud/relay build
FROM node:24-alpine AS runtime
ENV NODE_ENV=production
ENV PORT=8080
WORKDIR /app
RUN corepack enable
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
COPY packages/relay-contract/package.json packages/relay-contract/package.json
COPY apps/relay/package.json apps/relay/package.json
COPY --from=build /app/packages/relay-contract/dist packages/relay-contract/dist
COPY --from=build /app/apps/relay/dist apps/relay/dist
RUN pnpm install --prod --frozen-lockfile --filter @orca-cloud/relay...
USER node
EXPOSE 8080
CMD ["node", "apps/relay/dist/index.js"]
+35
View File
@@ -0,0 +1,35 @@
{
"name": "@orca-cloud/relay",
"private": true,
"version": "0.0.0",
"type": "module",
"main": "dist/index.js",
"scripts": {
"build": "pnpm clean && tsc -p tsconfig.build.json",
"clean": "node -e \"require('fs').rmSync('dist', { recursive: true, force: true })\"",
"dev": "tsx watch src/index.ts",
"lint": "tsc -p tsconfig.json --noEmit",
"pretest": "pnpm --filter @orca-cloud/relay-contract build",
"start": "node dist/index.js",
"test": "vitest run",
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"dependencies": {
"@hono/node-server": "^1.19.14",
"@orca-cloud/relay-contract": "workspace:*",
"hono": "^4.12.27",
"jose": "^6.1.3",
"pg": "^8.22.0",
"tweetnacl": "^1.0.3",
"ws": "^8.18.3",
"zod": "^3.25.76"
},
"devDependencies": {
"@types/node": "^24.10.0",
"@types/pg": "^8.20.0",
"@types/ws": "^8.18.1",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vitest": "^4.0.8"
}
}
@@ -0,0 +1,98 @@
import { describe, expect, it } from 'vitest'
import {
RELAY_ASIA_PROOF_ADMIN_ROUTES,
RELAY_CAPACITY_ADMIN_ROUTES,
RELAY_FENCE_BROKER_ADMIN_ROUTES,
RELAY_FENCE_ADMIN_ROUTES,
RELAY_MONITOR_ADMIN_ROUTES,
relayAdminIdentityMayAccess
} from './admin-token-verifier.js'
const mutationRoutes = [
'/v1/admin/drain',
'/v1/admin/evacuate',
'/v1/admin/migration-complete',
'/v1/admin/migration-supersede-cell',
'/v1/admin/rebalance-dormant',
'/v1/admin/admission-selector/apply',
'/v1/admin/admission-selector/add-migration-cells',
'/v1/admin/cell-state',
'/v1/admin/cell-fence-adopt-legacy',
'/v1/admin/cell-fence-commit-legacy-adoption',
'/v1/admin/cell-fence-attest',
'/v1/admin/cell-fence-attempt-prepare',
'/v1/admin/cell-fence-attempt-start',
'/v1/admin/cell-fence-attempt-operation',
'/v1/admin/cell-fence-attempt-abort',
'/v1/admin/drain-attempt-prepare',
'/v1/admin/drain-attempt-send',
'/v1/admin/drain-attempt-receipt',
'/v1/admin/drain-attempt-recover-forward',
'/v1/admin/cell-config',
'/v1/admin/evacuate-cell',
'/v1/admin/cell-heartbeat',
'/v1/admin/regional-rehome-control',
'/v1/admin/regional-rehome-trust-probe'
] as const
describe('Relay admin route authorization', () => {
it('allows the staging capacity identity only its transition routes', () => {
for (const route of RELAY_CAPACITY_ADMIN_ROUTES) {
expect(relayAdminIdentityMayAccess('capacity', route)).toBe(true)
}
for (const route of mutationRoutes) {
if ((RELAY_CAPACITY_ADMIN_ROUTES as readonly string[]).includes(route)) continue
expect(relayAdminIdentityMayAccess('capacity', route)).toBe(false)
}
expect(RELAY_CAPACITY_ADMIN_ROUTES).toContain('/v1/admin/cell-state')
expect(relayAdminIdentityMayAccess('capacity', '/v1/admin/evacuation-status')).toBe(false)
})
it('allows the Asia proof identity only its selector and status routes', () => {
for (const route of RELAY_ASIA_PROOF_ADMIN_ROUTES) {
expect(relayAdminIdentityMayAccess('asia-proof', route)).toBe(true)
}
expect(relayAdminIdentityMayAccess('asia-proof', '/v1/admin/drain')).toBe(false)
expect(relayAdminIdentityMayAccess('asia-proof', '/v1/admin/cell-state')).toBe(false)
expect(relayAdminIdentityMayAccess('asia-proof', '/v1/admin/admission-selector/apply')).toBe(false)
expect(relayAdminIdentityMayAccess('asia-proof', '/v1/admin/add-migration-cells')).toBe(false)
})
it('keeps the monitor identity on exact aggregate read routes', () => {
for (const route of RELAY_MONITOR_ADMIN_ROUTES) {
expect(relayAdminIdentityMayAccess('monitor', route)).toBe(true)
}
for (const route of [...mutationRoutes, ...RELAY_FENCE_ADMIN_ROUTES]) {
if ((RELAY_MONITOR_ADMIN_ROUTES as readonly string[]).includes(route)) continue
expect(relayAdminIdentityMayAccess('monitor', route)).toBe(false)
}
expect(RELAY_MONITOR_ADMIN_ROUTES).toContain('/v1/admin/evacuation-status')
})
it('allows only reviewed fence evidence mutations beyond aggregate reads', () => {
for (const route of RELAY_FENCE_ADMIN_ROUTES) {
expect(relayAdminIdentityMayAccess('fence', route)).toBe(true)
}
for (const route of mutationRoutes) {
if ((RELAY_FENCE_ADMIN_ROUTES as readonly string[]).includes(route)) continue
expect(relayAdminIdentityMayAccess('fence', route)).toBe(false)
}
})
it('allows the broker only exact fence inspection and mutation routes', () => {
for (const route of RELAY_FENCE_BROKER_ADMIN_ROUTES) {
expect(relayAdminIdentityMayAccess('fence-broker', route)).toBe(true)
}
for (const route of [...mutationRoutes, ...RELAY_FENCE_ADMIN_ROUTES]) {
if ((RELAY_FENCE_BROKER_ADMIN_ROUTES as readonly string[]).includes(route)) continue
expect(relayAdminIdentityMayAccess('fence-broker', route)).toBe(false)
}
})
it('rejects unknown routes for dedicated identities', () => {
for (const identity of ['capacity', 'asia-proof', 'monitor', 'fence', 'fence-broker'] as const) {
expect(relayAdminIdentityMayAccess(identity, '/v1/admin/future-mutation')).toBe(false)
expect(relayAdminIdentityMayAccess(identity, '/health')).toBe(false)
}
})
})

Some files were not shown because too many files have changed in this diff Show More