Merge upstream main and distinguish SSH endpoint routes

This commit is contained in:
vam
2026-10-28 12:08:02 +08:00
31639 changed files with 1010339 additions and 822002 deletions
+11
View File
@@ -0,0 +1,11 @@
{
"extraKnownMarketplaces": {
"orca-agent-plugins": {
"source": { "source": "directory", "path": "./config/agent-plugins" }
}
},
"enabledPlugins": {
"orca-typescript@orca-agent-plugins": true,
"typescript-lsp@claude-plugins-official": false
}
}
+12 -36
View File
@@ -1,6 +1,4 @@
# A shebang plus CRLF makes vite's SSR transform emit a literal `#!` mid-module,
# so any suite importing the script dies at load with a SyntaxError. Pin the whole
# directory rather than the scripts that happen to have a test today.
# CRLF shebangs break Vite's SSR transform.
/config/scripts/**/*.mjs text eol=lf
/skill-guides/*.md text eol=lf
/skill-stubs/*.md text eol=lf
@@ -15,55 +13,33 @@
*.html text eol=lf
# Bundled plugin trees are byte-hashed; CRLF checkout would break the pinned hash.
/resources/plugins/** text eol=lf
# Relay assets are copied verbatim into the bundle and hashed byte-for-byte into
# .version, which names the immutable remote install dir. A CRLF checkout makes a
# Windows-built client disagree with a mac/Linux-built one on the same release,
# so one host ends up with two relay trees (#17886 review).
# Pin the bytes so a patch reads and diffs identically on every host. It is NOT
# what makes the hash right: pnpm hashes a patch LF-normalized, so a CRLF checkout
# cannot change it. Believing otherwise put a hand-computed raw digest in the
# lockfile twice and broke every install (#17886).
# These files are stored LF, which is not always the encoding they were written
# against -- @vscode/windows-process-tree ships CRLF sources -- so any code that
# runs `git apply` on one must force `-c core.autocrlf=input` rather than trust
# the host's setting. See config/scripts/windows-process-tree-gyp-rebuild.mjs.
# Keep patch diffs stable; pnpm hashes them with LF normalization.
# CRLF upstream sources still need git -c core.autocrlf=input apply (windows-process-tree-gyp-rebuild.mjs).
/config/patches/*.patch -text
# Same reason, and pnpm parses these too: a CRLF checkout makes the mobile
# patches unparseable, so Windows packaging dies on ERR_PNPM_INVALID_PATCH.
# CRLF patches fail pnpm parsing on Windows.
/mobile/patches/*.patch -text
# The xterm bundle hunks also make a diff nobody can read; review the hand-written
# source patch under xterm-src/ instead. The sibling patches stay diffable.
# Review xterm-src/ instead of generated bundle hunks.
/config/patches/@xterm__xterm@*.patch -diff
/config/patches/xterm-src/*.patch text eol=lf
# pnpm parses these unified diffs during Windows installs; keep checkout bytes stable.
/mobile/patches/*.patch -text
# Generated wrapper fixtures: collapse them in the PR diff so they stop drowning
# the reviewable change, and pin LF because they are compared byte-for-byte.
# Not -diff: the shell diff is the review surface when a wrapper does change.
# Collapse generated wrappers but keep diffs available; tests compare LF bytes.
/src/main/__fixtures__/shell-wrapper-snapshots/*.txt linguist-generated=true text eol=lf
# Captured agent PTY transcripts. -text, not `text eol=lf` like the wrapper snapshots above:
# these carry real CR and CRLF bytes as the terminal emitted them, and line-ending
# normalisation on a Windows checkout would rewrite the evidence the fixture exists to be.
# PTY transcripts must retain captured CR and CRLF bytes.
/src/main/runtime/__fixtures__/*.txt -text
/src/main/daemon/__fixtures__/pty-transcripts/*.txt -text
/src/renderer/src/components/terminal-pane/terminal-link-transcripts/*.txt -text whitespace=cr-at-eol
# Generated runtime English subset: compared byte-for-byte by
# verify:localization-runtime-catalog, so a CRLF checkout would fail the gate.
# verify:localization-runtime-catalog compares LF bytes.
/src/renderer/src/i18n/en-runtime-required.json linguist-generated=true text eol=lf
# Generated method->params catalog: compared byte-for-byte by
# verify:rpc-params-catalog, so a CRLF checkout would fail the gate.
# verify:rpc-params-catalog compares LF bytes.
/src/shared/rpc-contract/rpc-params-catalog.generated.ts linguist-generated=true text eol=lf
# Mobile RPC recording goldens: replay requires the file text to equal what rpc:record writes
# (LF), so a CRLF checkout would fail every golden.
# RPC recording goldens must match rpc:record's LF output.
/mobile/rpc-foundation/goldens/*.json text eol=lf
# Mobile web page source. Every text byte here is hashed into an asset digest and from
# there into buildId, so a CRLF Windows checkout would ship a different bundle id for
# identical source (91af2897 vs 9d78435e, measured on the bundle this replaced).
# Mobile build IDs hash source bytes; checkout line endings must match across hosts.
/mobile/src/** text eol=lf
/mobile/app/** text eol=lf
/mobile/web-entry/** text eol=lf
# The blanket pin above would mark a future binary as text; exempt the asset types an
# RN page actually carries.
# Preserve binary assets under the text-pinned mobile directories.
/mobile/src/**/*.png -text
/mobile/src/**/*.jpg -text
/mobile/src/**/*.jpeg -text
-2
View File
@@ -1,10 +1,8 @@
# Product source remains owned by its normal reviewers; localization inputs need focused review.
/src/renderer/src/i18n/locales/ @brennanb2025
/config/scripts/*localization*.mjs @brennanb2025
/config/scripts/*locale*.mjs @brennanb2025
/config/i18next.config.ts @brennanb2025
# The relay's deploy and operate surface: workspace, workflows, and the rollout lease action.
/cloud/ @Jinwoo-H
/.github/workflows/cloud-*.yml @Jinwoo-H
/.github/actions/cloud-sql-rollout-lease/ @Jinwoo-H
+142 -5
View File
@@ -21,11 +21,119 @@ Install Node 24, pnpm, and the Bun version in [`config/.bun-version`](../config/
native PTYs, socket liveness, and V8 memory behavior. `pnpm test:node` runs the same suites
entirely on Node. Builds and dependency installation still use Node and pnpm.
The server, its workers and its packaged dependencies target Node 24. The compatibility
launcher and CLI still target Node 18; the launcher hands off to bundled Node 24 before
loading server code. WSL guest programs and host discovery retain their Node 18 minimum.
Keep Node 24 APIs and server imports out of these host compatibility paths. Running an
unpackaged server directly requires Node 24 or newer.
Use the pnpm version in `packageManager` in [`package.json`](../package.json): `corepack enable`
needs Corepack 0.34.5 or newer to start the pinned pnpm (`npm install -g corepack@latest`). On
Windows it writes shims next to `node.exe` and fails with `EPERM` outside an elevated shell when
Node is in `C:\Program Files\nodejs`; run `npm install -g pnpm@<that version>` instead.
`pnpm install` compiles native modules such as `node-pty` with `node-gyp` and loads them in
Electron, so a fresh machine also needs:
- **macOS**: Xcode Command Line Tools (`xcode-select --install`).
- **Windows**: Python 3 and Visual Studio Build Tools with the "Desktop development with C++"
workload.
- **Linux**: a C++ toolchain, Python 3, and the libraries Electron loads even on a headless
machine, plus `xvfb`, which the `*.electron.test.ts` files in `pnpm test` use when `DISPLAY` is
unset. On Ubuntu 24.04+ and Debian 13+:
```bash
sudo apt-get update && sudo apt-get install -y build-essential python3 xvfb libgtk-3-0 libnss3 libgbm1 libasound2t64
```
Older releases name the last package `libasound2`.
```bash
pnpm install
pnpm dev
```
### Dev Profiles and Parallel Instances
`pnpm dev` in a linked Git worktree uses its own userData profile, renderer port, and remote
debugging (CDP) port, so dev instances in different worktrees run side by side. The main checkout
keeps the shared `orca-dev` profile. Ports are derived from the worktree path and move to the next
free port when taken: renderer ports start at 15200 and CDP ports at 9333.
On startup the runner prints one stdout line and writes the same JSON to `out/dev-instance.json`:
```text
ORCA_DEV_INSTANCE {"userData":".../orca-dev-wt/1d3830d4d3","profile":"worktree","rendererPort":15339,"rendererUrl":"http://localhost:15339","cdpPort":9513,"cdpUrl":"http://127.0.0.1:9513","repoRoot":"...","runnerPid":4242}
```
| Option | Effect |
| ----------------------------------------------------------------------- | ---------------------------------------------------------- |
| `pnpm dev --shared-profile` or `ORCA_DEV_PROFILE=shared` | Use the shared `orca-dev` profile |
| `pnpm dev --worktree-profile` or `ORCA_DEV_PROFILE=worktree` | Use a per-worktree profile, including in the main checkout |
| `ORCA_DEV_USER_DATA_PATH=<dir>` | Use an explicit profile directory |
| `REMOTE_DEBUGGING_PORT=<port>`, `ORCA_DEV_RENDERER_PORT=<port>` | Pin a port |
| `pnpm dev --seed-profile[=<dir>]` or `ORCA_DEV_SEED_PROFILE=<dir or 1>` | Seed an empty profile on first launch |
Per-worktree profiles live in `orca-dev-wt/<hash>` next to `orca-dev` (`~/Library/Application
Support` on macOS, `%APPDATA%` on Windows, `$XDG_CONFIG_HOME` or `~/.config` on Linux). The
automatic storage cleanup removes them once their worktree is gone and nothing still runs from
them. `config/scripts/orca-dev.mjs` and the `orca-dev` terminal wrapper target the profile this
checkout's `pnpm dev` used.
Seeding copies a slim set of state from another profile: projects, workspaces, settings, SSH
targets, and the Orca account session. Terminal sessions, PTY leases, automations, caches, logs,
and lock files are not copied, and the source profile is only read. The default source is
`orca-dev`, or the installed app's profile when `orca-dev` is empty. Seed from `orca-dev` when you
need the signed-in account session: dev builds cannot decrypt secrets the installed app encrypted.
```bash
pnpm dev:seed-profile [--from=<profile dir>] [--to=<profile dir>] [--reset]
pnpm dev:fresh [--keep] [--seed-from[=<profile dir>]] # temporary profile, removed on exit
```
### Targeting Local Tests
Use a test file or directory when you know the relevant suites. For dependency-based
feedback, Vitest can select tests affected by your edits:
```bash
pnpm test:changed src/main/runtime --maxWorkers=2
pnpm test --changed=origin/main --maxWorkers=2
pnpm test src/shared/app-version.test.ts --maxWorkers=2
```
`test:changed` considers staged, unstaged, and untracked files. `--changed=origin/main`
also includes committed changes since the branch's merge base. Update that ref when
you need a newer baseline. Both accept the same path filters and options as `pnpm test`.
When Git reports no changed files at all, the launcher exits before starting Vitest.
Local runs enable Vitest's [dependency recording](https://vitest.dev/config/experimental#experimental-recorddependencies).
Ordinary test runs populate the cache; later changed-file runs read it instead of
transforming every test's import graph. Records are separate for the Bun and Node
projects and live in each checkout's `node_modules/.vite`. A test without a record
runs when at most 100 such tests are in scope, which records them. Beyond that, as in
a new worktree, unrecorded tests are selected with Vitest's static import graph
instead of running all of them. Keep Vite's local test cache between runs. Use
`--experimental.recordDependencies=false` to use the static import graph for every test.
This is local feedback, not a complete regression check. Vitest records loaded
imports, including executed dynamic imports, but cannot infer arbitrary filesystem
reads, subprocess inputs, or externalized module dependencies. Explicitly run tests
for changed fixtures, generated files, CLI entry points, and other indirect inputs.
After changing imports, run the affected suites before committing those changes so
later edits use current records. Configuration, lockfile, patch, and workspace-package
changes rerun every test within the requested scope. CI does not enable recording;
the existing CI selection policy is unchanged. The static graph has the same blind spots
and also misses dynamic imports with computed specifiers, so the first changed-file run
in a new worktree is a little less complete than later ones.
`pnpm build:packages` (which `pnpm test` runs first) skips compiling workspace packages
when their sources, build script, and compiler are unchanged and `dist` is intact.
Vitest also supports [`related --run <source-files>`](https://vitest.dev/guide/cli#vitest-related)
and dependency-aware watch mode. `pnpm test --changed` uses Git to provide the source
inputs automatically and exits after one run.
To reduce main-process rebuild CPU and memory when you do not need its source maps,
set `ORCA_DEV_MAIN_SOURCEMAPS=0` before `pnpm dev`. In Bash/zsh:
@@ -37,6 +145,29 @@ In PowerShell, use `$env:ORCA_DEV_MAIN_SOURCEMAPS='0'` before `pnpm dev`.
Remove the variable when debugging the main process with source maps. Renderer and
preload settings are unaffected; production builds always keep their hidden main maps.
`pnpm dev` serves the renderer with Vite's bundled dev mode: a few bundled chunks instead of
thousands of module requests, so the first render and full reloads are faster. While a rebuild is
pending, a newly opened pop-out, space, or web window may briefly get placeholder HTML. If the
renderer misbehaves, set `ORCA_DEV_BUNDLED=0` (PowerShell: `$env:ORCA_DEV_BUNDLED='0'`) and report it.
Local installs use pnpm's global virtual store: each package version is one slot
under `~/pnpm/store` that every worktree links to, instead of a full `node_modules` per worktree.
An existing worktree converts on its next `pnpm install`, and an older branch without the setting
converts back the same way. CI always uses the normal layout. On Windows, pnpm links with
junctions when you lack the symlink privilege, so no administrator rights or Developer Mode are
needed. Things to know:
- Keep `~/node_modules` (and any other `node_modules` above the store) out of your home
directory. Node resolves upward from the store slot, so a stray one there supplies
dependencies the package never declared and hides missing-dependency bugs; `pnpm install` warns
when it finds one.
- Native addons are never rebuilt in place in a shared slot: each CPU gets its own build
beside it (see [`native-build-entry.cjs`](../config/scripts/native-build-entry.cjs)), so
cross-arch packaging is safe for other worktrees.
- `ORCA_PNPM_GLOBAL_VIRTUAL_STORE=0` opts out. Keep it set
in your shell: `pnpm run` re-checks the install and relinks a worktree whose layout no longer
matches.
Ordinary installs include native optional dependencies for the current OS and CPU only.
Before a cross-architecture build (including `pnpm build:mac`, which produces both x64 and
arm64 artifacts by default), run `pnpm install:release` to add the other CPU's variants.
@@ -71,12 +202,18 @@ pnpm test
pnpm build
```
`pnpm lint` runs independent checks in a bounded pool. Use
`pnpm lint --maxWorkers=2 --timings` to limit concurrent checks and print their durations.
For repeated local typechecks, use `pnpm tc` (or `pnpm tc:node`, `pnpm tc:web`,
and `pnpm tc:cli`). These run `bun check` from the pinned `bun` devDependency —
about a third of tsc's CPU and an eighth of its memory, so several agents can
typecheck at once. Pass `--compiler=native` to use tsc and its incremental
cache instead. CI uses `pnpm typecheck:ci`, the same Bun check with CI's
concurrency. Shared tests are checked by the
and `pnpm tc:cli`). On Apple Silicon macOS and x64/arm64 Linux, these use the pinned
`tsrs` compiler with incremental caches. Keep `.cache/typecheck/` between checks;
delete it to force a fresh check. Caches are separate for each project, compiler
version, and worktree. At most two projects run together, subject to the memory budget.
Windows, Intel macOS, and other unsupported platforms use pinned Bun automatically.
Pass `--compiler=bun` to explicitly use Bun, or `--compiler=native` to use TypeScript.
CI uses `pnpm typecheck:ci`, which explicitly selects Bun and its CI concurrency.
Shared tests are checked by the
Node project; shared production code is checked in all three environments.
Run `pnpm tc:node` after editing shared tests, or `pnpm tc` to check everything.
@@ -137,7 +137,7 @@ runners; only the compare-and-swap algorithm is shared with the fence broker.
## Duplication
This directory is copied verbatim into `stablyai/orca`. It has no `package.json`, no
`node_modules`, and imports nothing outside itself — `action-contract.test.mjs` enforces all three.
`node_modules`, and imports nothing outside itself. Keep it self-contained when updating the action.
Cross-repo consumption via `uses: stablyai/orca/.github/actions/...@<sha>` was rejected: it would
put public-repo code inside private app deploys that hold a production credential, and neither
repository protects `main` today.
@@ -1,11 +1,9 @@
import assert from 'node:assert/strict'
import { readFileSync, readdirSync } from 'node:fs'
import { readFileSync } from 'node:fs'
import { test } from 'node:test'
const here = new URL('./', import.meta.url)
const action = readFileSync(new URL('action.yml', here), 'utf8')
const modules = readdirSync(here).filter((name) => name.endsWith('.mjs'))
const shipped = modules.filter((name) => !name.endsWith('.test.mjs'))
test('is a node24 JavaScript action with an always-run post step', () => {
// A composite action has no `post:`, so the lease could never be released on cancel or failure.
@@ -23,30 +21,3 @@ test('declares the inputs the wave-chain callers depend on', () => {
assert.match(action, /default: \$\{\{ github\.repository \}\}\/\$\{\{ github\.run_id \}\}/)
assert.match(action, /default: 'true'/)
})
test('stays self-contained so it can be duplicated into the public repo', () => {
assert.deepEqual(
readdirSync(here).filter((name) => name === 'package.json' || name === 'node_modules'),
[],
'the action must run with zero installed dependencies'
)
for (const name of modules) {
const source = readFileSync(new URL(name, here), 'utf8')
for (const match of source.matchAll(/^import\b[\s\S]*?from '([^']+)'/gm)) {
const specifier = match[1]
const local = specifier.startsWith('.')
assert.ok(
specifier.startsWith('node:') || (local && !specifier.includes('..')),
`${name} imports ${specifier}; only node: builtins and same-directory modules are allowed`
)
}
}
})
test('never reaches for the GCE metadata server', () => {
// Runners have no metadata.google.internal; the fence broker's token path must not be copied.
for (const name of shipped) {
const source = readFileSync(new URL(name, here), 'utf8')
assert.doesNotMatch(source, /metadata\.google\.internal/, name)
}
})
@@ -1,17 +1,6 @@
import { execFileSync } from 'node:child_process'
// DESIGN CHOICE: shell out to `gcloud auth print-access-token` instead of exchanging the
// external_account credentials file that google-github-actions/auth writes.
//
// Every workflow on the Cloud SQL rollout lease already runs google-github-actions/setup-gcloud
// right after auth (verified across all 11 mutating members), so gcloud is on PATH and already
// bound to the federated identity. Doing the exchange ourselves would mean reimplementing the STS
// token swap plus the service-account impersonation leg, in an action that must stay
// zero-dependency and is duplicated by hand into a second repo. gcloud already handles every ADC
// flavour and refreshes on its own. The metadata server is not an option: it does not exist on
// GitHub or Blacksmith runners.
//
// Consequence, documented in the README: the lease step MUST come after setup-gcloud.
// Use gcloud after setup-gcloud: it handles federated STS/impersonation and ADC refresh without dependencies or repo-local code in this duplicated action. Actions/Blacksmith runners have no metadata server.
const TOKEN_REUSE_MS = 40 * 60 * 1_000 // GCP access tokens live ~60 min; re-mint well before that.
@@ -1,5 +1,4 @@
// Who we claim to be on the lease object. Shared by main and the detached renewer so both agree
// on the holder key without re-deriving it from a different set of environment variables.
// Main and detached renewer must derive the same holder key from one environment contract.
export function holderIdentity(explicitHolderKey) {
const repository = process.env.GITHUB_REPOSITORY ?? 'unknown'
@@ -3,8 +3,7 @@ import { createAccessTokenSource } from './gcloud-access-token.mjs'
import { renewerLogPath } from './runner-state.mjs'
import { CloudSqlRolloutLease, RENEW_INTERVAL_MS } from './storage-lease.mjs'
// Detached renewer. `main` spawns it, `post` kills it. It rewrites expires_at on the same
// generation-matched path as acquisition, and stops the moment the object stops being ours.
// Detached renewer shares acquisition’s generation-matched writes; stop when ownership changes. Main starts it and post kills it.
const MAX_LIFETIME_MS = 6 * 60 * 60 * 1_000 // Backstop if post never runs (runner killed).
@@ -1,7 +1,6 @@
import { appendFileSync } from 'node:fs'
// Action-state and output plumbing via the runner's file protocol, so the action needs no
// @actions/core dependency. Values are single-line by construction; anything else is rejected.
// Use runner files without @actions/core; reject multiline values rather than corrupting their protocol.
/** The detached renewer's stdio is ignored, so it appends here instead and `post` echoes it. */
export function renewerLogPath() {
@@ -1,10 +1,4 @@
// Compare-and-swap lease over a single Cloud Storage object.
//
// Ported from apps/relay-fence-broker/src/mutation-lease.ts rather than imported: this action is
// duplicated verbatim into stablyai/orca, so it must carry no repo-local imports. Only the
// algorithm is shared (read metadata -> write with ifGenerationMatch -> 412 is a conflict ->
// generation-matched delete -> an expired record is free). The broker's token path is NOT shared;
// it reads the GCE metadata server, which does not exist on Actions runners.
// Ported CAS algorithm from relay-fence-broker/mutation-lease.ts: keep repo-local imports out of this duplicated action. Broker metadata-server tokens cannot work on Actions runners; use generation-matched writes/deletes, 412 conflicts and expiry.
export const LEASE_TTL_MS = 35 * 60 * 1_000
export const RENEW_INTERVAL_MS = 5 * 60 * 1_000
@@ -12,10 +12,7 @@ const BUCKET = 'onorca-cloud-terraform-state'
const OBJECT = 'terraform/state/cloud-sql-rollout/production.lock'
const NOW = 1_756_000_000_000
/**
* Enough of the Cloud Storage JSON API to exercise real compare-and-swap semantics: generations
* increment, ifGenerationMatch is enforced, and a mismatch is a 412. `faults` injects failures.
*/
/** Model real GCS generation increments, ifGenerationMatch and 412 conflicts; faults inject failures. */
function fakeStorage({ object = null, faults = [] } = {}) {
const state = { object, requests: [] }
const fetcher = async (rawUrl, init = {}) => {
@@ -4,12 +4,7 @@ description: Frozen pnpm install for the mobile/ project, whose node_modules the
runs:
using: composite
steps:
# Why a separate install: mobile is its own pnpm project, so the root install leaves
# mobile/node_modules empty and every mobile import resolves to nothing.
# Why no --ignore-scripts, unlike the root install: mobile's postinstall generates the
# gitignored terminal/mermaid webview engine modules that tracked source imports.
# The drift guard mirrors the root install so a stale mobile lockfile fails by name --
# mobile's lockfile carries patchedDependencies that a silent rewrite would drop.
# Install the separate mobile project: root install leaves its imports unresolved. Keep lifecycle scripts for generated terminal/mermaid engines; reject lockfile drift that could drop patchedDependencies.
- name: Install mobile dependencies
shell: bash
working-directory: mobile
@@ -46,7 +46,7 @@ outputs:
description: The small pnpm-owned verification record, without registry metadata.
value: ${{ steps.verification-cache.outputs.path }}
verification-cache-key:
description: Exact verification record key, also used by the isolated PR benchmark.
description: Exact verification record cache key.
value: ${{ steps.verification-cache.outputs.key }}
node-version:
description: Resolved Node.js version used for the install.
@@ -67,6 +67,32 @@ outputs:
runs:
using: composite
steps:
# The image resets HOME during login; shell fixtures intentionally use temporary homes.
- name: Preserve explicit login-shell homes on Blacksmith
if: runner.os == 'Linux' && startsWith(runner.name, 'blacksmith-') && job.container.id == ''
shell: bash
run: |
sudo python3 - <<'PYTHON'
from pathlib import Path
profile = Path('/etc/profile.d/blacksmith.sh')
if profile.exists():
original = profile.read_text()
updated = original.replace('export HOME="$RUNNER_HOME"', 'export HOME="${HOME:-$RUNNER_HOME}"')
if updated != original:
profile.write_text(updated)
PYTHON
fixture_home="$(mktemp -d)"
trap 'rm -rf "$fixture_home"' EXIT
printf 'export ORCA_CI_PROFILE_LOADED=1\n' > "$fixture_home/.bash_profile"
HOME="$fixture_home" ORCA_CI_FIXTURE_HOME="$fixture_home" /bin/bash -lc '
test "$HOME" = "$ORCA_CI_FIXTURE_HOME" && test "$ORCA_CI_PROFILE_LOADED" = 1
' || {
echo '::error::Runner login profile overrides fixture HOME'
grep -nH 'HOME' /etc/profile /etc/profile.d/*.sh || true
HOME="$fixture_home" /bin/bash -lc 'printf "Observed HOME=%s\n" "$HOME"'
exit 1
}
- name: Resolve pnpm store mode
id: pnpm-store-mode
if: >-
@@ -86,16 +112,14 @@ runs:
case "$LOOKUP_REQUEST" in
[aA][uU][tT][oO])
# Hosted runners have Node for this manifest-only check before toolchain setup.
lookup_only="$(node -p 'const p = require("./package.json"); p.engines?.node === "24" && typeof p.packageManager === "string" && p.packageManager.split("+")[0] === "pnpm@12.8.1"')"
lookup_only="$(node -p 'const p = require("./package.json"); p.engines?.node === "24" && typeof p.packageManager === "string" && p.packageManager.split("+")[0] === "pnpm@12.14.0"')"
;;
esac
printf 'lookup-only=%s\n' "$lookup_only" >> "$GITHUB_OUTPUT"
# setup-node needs pnpm on PATH to locate and restore its store.
- name: Setup pnpm
uses: pnpm/setup@v2
with:
install: false
uses: ./.github/actions/setup-pnpm
# Desktop-only jobs should not miss their download cache when mobile dependencies change.
- name: Setup Node.js
@@ -104,6 +128,7 @@ runs:
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
cache: ${{ github.event_name != 'pull_request' && inputs.cache-pnpm-store != 'false' && steps.pnpm-store-mode.outputs.lookup-only != 'true' && 'pnpm' || '' }}
cache-dependency-path: ${{ inputs.cache-dependency-path }}
package-manager-cache: false
@@ -114,6 +139,7 @@ runs:
uses: actions/setup-node@v6
with:
node-version: ${{ inputs.node-version }}
check-latest: true
cache: ${{ github.event_name != 'pull_request' && inputs.cache-pnpm-store != 'false' && steps.pnpm-store-mode.outputs.lookup-only != 'true' && 'pnpm' || '' }}
cache-dependency-path: ${{ inputs.cache-dependency-path }}
package-manager-cache: false
@@ -159,9 +185,10 @@ runs:
key: node-cache-${{ runner.os }}-${{ steps.pnpm-store.outputs.arch }}-pnpm-${{ hashFiles(inputs.cache-dependency-path) }}
# Producers can refresh access and publish misses without downloading existing archives.
# Not on GitHub-hosted runners: nothing restores that store, and direct installs beat restoring it.
- name: Keep pnpm download store without restoring
id: pnpm-store-lookup
if: steps.pnpm-store-mode.outputs.lookup-only == 'true'
if: steps.pnpm-store-mode.outputs.lookup-only == 'true' && runner.environment != 'github-hosted'
uses: actions/cache@v5
with:
# Twice-nested composite cleanup loses internal step outputs.
@@ -196,10 +223,7 @@ runs:
rm -rf node_modules
fi
# Why --frozen-lockfile: re-resolving pulls ~62 MB of registry packuments per job
# (measured) to recompute what the lockfile already pins, and the `git diff` guard
# below fails the run whenever that recomputation would have changed anything. The
# guard stays so a stale lockfile still fails by name rather than by resolver error.
# Frozen install avoids ~62MB/job of redundant resolution; retain diff guard to name stale lockfiles explicitly.
- name: Install dependencies
shell: bash
run: |
@@ -250,15 +274,16 @@ runs:
printf 'ELECTRON_CACHE=%s\n' "$cache_root" >> "$GITHUB_ENV"
printf 'version=%s\n' "$(node -p "require('./node_modules/electron/package.json').version")" >> "$GITHUB_OUTPUT"
# Main and the warmers own every OS's archive; PR copies duplicated it per PR.
- name: Cache Electron package archive
if: (github.event_name != 'pull_request' || runner.os != 'Linux') && steps.electron-package-cache.outputs.version != ''
if: github.event_name != 'pull_request' && steps.electron-package-cache.outputs.version != ''
uses: actions/cache@v5
with:
path: ${{ steps.electron-package-cache.outputs.cache-root }}
key: electron-package-${{ runner.os }}-${{ runner.arch }}-${{ steps.electron-package-cache.outputs.version }}
- name: Restore Electron package archive without saving
if: github.event_name == 'pull_request' && runner.os == 'Linux' && steps.electron-package-cache.outputs.version != ''
if: github.event_name == 'pull_request' && steps.electron-package-cache.outputs.version != ''
uses: actions/cache/restore@v5
with:
path: ${{ steps.electron-package-cache.outputs.cache-root }}
@@ -29,13 +29,7 @@ runs:
# Why: force TLS 1.2 so gallery downloads work on older hosted images.
[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
# Why: on some hosted Windows images `Register-PSRepository -Default`
# fails inside the legacy nuget.exe provider with "Missing option value
# for: '-source'", so PSGallery is never registered and the install
# below dies with "No repository with the name 'PSGallery'". PSResourceGet
# (bundled with PowerShell 7.4+) has PSGallery registered by default and
# avoids that code path, so prefer it and fall back to PowerShellGet only
# when it is absent.
# Prefer PSResourceGet (PowerShell 7.4+): legacy nuget Register-PSRepository can fail with “Missing option value for: -source” and leave PSGallery absent.
$useResourceGet = $null -ne (Get-Command -Name Install-PSResource -ErrorAction SilentlyContinue)
try {
@@ -53,9 +47,7 @@ runs:
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
}
} catch {
# Why: repository registration also talks to the gallery, so a gallery
# outage can fail here before a single install is attempted. The CDN
# fallback below does not need a registered repository, so keep going.
# Repository registration can fail during gallery outages; continue so the independent CDN fallback remains reachable.
Write-Warning "PSGallery repository registration failed: $_"
}
@@ -114,13 +106,8 @@ runs:
}
}
# Why: the gallery's package API (OData search + repository metadata) sits
# behind Azure Front Door and has returned 403/502/504 for every install
# attempt during gallery incidents, which hard-failed the whole Windows
# release job. The CDN that serves the nupkg itself is a separate origin
# and stays up through those incidents, so fall back to a pinned version
# fetched straight from it. The hash pin is mandatory: this route skips the
# gallery's package validation, so an unexpected payload must fail loudly.
# Gallery OData/metadata has returned 403/502/504 while the independent nupkg CDN stayed up.
# Fall back to a pinned version and mandatory hash because this bypasses gallery validation.
if (-not $installed) {
$version = $env:SIGNPATH_FALLBACK_VERSION
$expectedHash = $env:SIGNPATH_FALLBACK_SHA256
@@ -131,9 +118,7 @@ runs:
Remove-Item -LiteralPath $nupkg -Force
}
# Why two URLs: the /api/v2/package route 302s to the CDN and can serve
# while the OData search endpoint is failing; the CDN URL is the same
# redirect target reached directly when the api host is down entirely.
# Try the API’s CDN redirect, then the CDN directly if the API host is down.
$sources = @(
"https://www.powershellgallery.com/api/v2/package/SignPath/$version",
"https://cdn.powershellgallery.com/packages/signpath.$version.nupkg"
@@ -173,9 +158,7 @@ runs:
}
Remove-SignPathModuleDirectory
# Why a version-named subdirectory: PowerShell only treats a nested folder
# as a side-by-side module version when the name matches the manifest's
# ModuleVersion, which is what makes `Import-Module SignPath` resolve it.
# PowerShell resolves side-by-side modules only when the directory matches the manifest’s ModuleVersion.
$versionRoot = Join-Path -Path $signPathModulePath -ChildPath $version
New-Item -ItemType Directory -Path $versionRoot -Force | Out-Null
Expand-Archive -LiteralPath $nupkg -DestinationPath $versionRoot -Force
@@ -0,0 +1,28 @@
name: Prepare release dependency audit data
description: Extract only manifests, lockfiles, and patches without candidate executable configuration
inputs:
ref:
required: true
description: Exact release ref to audit
runs:
using: composite
steps:
- shell: bash
env:
RELEASE_REF: ${{ inputs.ref }}
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
git -c credential.helper= -c 'credential.helper=!gh auth git-credential' fetch --no-tags --depth=1 origin "$RELEASE_REF"
mapfile -t files < <(git ls-tree -r FETCH_HEAD -- \
pnpm-lock.yaml pnpm-workspace.yaml mobile/pnpm-lock.yaml mobile/pnpm-workspace.yaml \
native config/patches mobile/patches | \
awk '$1 ~ /^100(644|755)$/ {sub(/^[^\t]*\t/, ""); print}' | \
grep -E '(^|/)(pnpm-lock.yaml|pnpm-workspace.yaml)$|^native/[^/]+/Cargo\.(toml|lock)$|^(config|mobile)/patches/.*\.patch$')
test "${#files[@]}" -gt 0
mkdir candidate
git archive --worktree-attributes FETCH_HEAD -- "${files[@]}" | tar -x -C candidate
for file in pnpm-lock.yaml pnpm-workspace.yaml mobile/pnpm-lock.yaml mobile/pnpm-workspace.yaml; do
test -f "candidate/$file"
test ! -L "candidate/$file"
done
@@ -14,7 +14,7 @@ runs:
- id: identity
shell: bash
env:
COMPILER_POLICY_HASH: ${{ hashFiles('package.json', 'pnpm-lock.yaml', 'pnpm-workspace.yaml', '.npmrc', '.pnpmfile.cjs', 'config/patches/**', '.github/actions/install-node-dependencies/**', '.github/actions/restore-pnpm-verification/**', 'config/scripts/headless-detector-compiler-cache.mjs', '.github/actions/prepare-headless-compiler/action.yml') }}
COMPILER_POLICY_HASH: ${{ hashFiles('package.json', 'pnpm-lock.yaml', 'pnpm-workspace.yaml', '.npmrc', 'config/pnpmfile.cjs', 'config/patches/**', '.github/actions/install-node-dependencies/**', '.github/actions/restore-pnpm-verification/**', 'config/scripts/headless-detector-compiler-cache.mjs', '.github/actions/prepare-headless-compiler/action.yml') }}
run: node config/scripts/headless-detector-compiler-cache.mjs identity
- id: cache
uses: actions/cache/restore@v5
@@ -7,10 +7,16 @@ inputs:
save-cache:
description: Build and save a missing fixture on the default branch.
default: 'false'
load:
description: Load the image here; 'false' only restores, so the caller can load it in a background run step.
default: 'true'
outputs:
image:
description: Loaded image to use as a Docker build cache, or empty on a cache miss.
value: ${{ steps.image.outputs.image }}
cache-hit:
description: Whether the fixture archive was restored under the exact key.
value: ${{ steps.cache.outputs.cache-hit }}
runs:
using: composite
steps:
@@ -20,39 +26,17 @@ runs:
continue-on-error: true
with:
path: ${{ runner.temp }}/orca-package-fixtures/${{ inputs.fixture }}.tar
key: linux-package-fixture-v1-${{ runner.os }}-${{ runner.arch }}-linux-amd64-${{ inputs.fixture }}-${{ hashFiles(format('config/docker/{0}/**', inputs.fixture), '.github/actions/prepare-linux-package-fixture/action.yml') }}
key: linux-package-fixture-v1-${{ runner.os }}-${{ runner.arch }}-linux-amd64-${{ inputs.fixture }}-${{ hashFiles(format('config/docker/{0}/**', inputs.fixture), '.github/actions/prepare-linux-package-fixture/action.yml', '.github/actions/prepare-linux-package-fixture/load-fixture.sh') }}
- name: Load or seed fixture build cache
id: image
if: inputs.load != 'false'
shell: bash
env:
FIXTURE: ${{ inputs.fixture }}
SAVE_CACHE: ${{ inputs.save-cache }}
CACHE_HIT: ${{ steps.cache.outputs.cache-hit }}
run: |
set -euo pipefail
case "$FIXTURE" in
headless-serve-shutdown|cli-launch-contract|daemon-shutdown-descendants) ;;
*) echo "Unsupported package fixture: $FIXTURE" >&2; exit 1 ;;
esac
archive="$RUNNER_TEMP/orca-package-fixtures/$FIXTURE.tar"
image="orca-package-fixture-$FIXTURE:cache"
if [[ "$CACHE_HIT" == true ]] && docker load --input "$archive"; then
if docker image inspect "$image" > /dev/null; then
echo "image=$image" >> "$GITHUB_OUTPUT"
exit 0
fi
fi
if [[ "$SAVE_CACHE" != true ]]; then
echo 'No usable fixture cache; the package runner will build normally.'
exit 0
fi
docker build --platform linux/amd64 --build-arg BUILDKIT_INLINE_CACHE=1 \
--tag "$image" --file "config/docker/$FIXTURE/Dockerfile" "config/docker/$FIXTURE"
mkdir -p "$(dirname "$archive")"
docker save --output "$archive" "$image"
echo "image=$image" >> "$GITHUB_OUTPUT"
echo 'built=true' >> "$GITHUB_OUTPUT"
run: bash "$GITHUB_ACTION_PATH/load-fixture.sh"
- name: Save fixture image
if: inputs.save-cache == 'true' && steps.cache.outputs.cache-hit != 'true' && steps.image.outputs.built == 'true'
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# Loads a restored fixture image; only the default-branch warmer builds and exports a missing one.
set -euo pipefail
case "$FIXTURE" in
headless-serve-shutdown|cli-launch-contract|daemon-shutdown-descendants) ;;
*) echo "Unsupported package fixture: $FIXTURE" >&2; exit 1 ;;
esac
archive="$RUNNER_TEMP/orca-package-fixtures/$FIXTURE.tar"
image="orca-package-fixture-$FIXTURE:cache"
if [[ "${CACHE_HIT:-}" == true ]] && docker load --input "$archive"; then
if docker image inspect "$image" > /dev/null; then
echo "image=$image" >> "$GITHUB_OUTPUT"
exit 0
fi
fi
if [[ "${SAVE_CACHE:-}" != true ]]; then
echo 'No usable fixture cache; the package runner will build normally.'
exit 0
fi
docker build --platform linux/amd64 --build-arg BUILDKIT_INLINE_CACHE=1 \
--tag "$image" --file "config/docker/$FIXTURE/Dockerfile" "config/docker/$FIXTURE"
mkdir -p "$(dirname "$archive")"
docker save --output "$archive" "$image"
echo "image=$image" >> "$GITHUB_OUTPUT"
echo 'built=true' >> "$GITHUB_OUTPUT"
@@ -54,7 +54,7 @@ runs:
env:
NATIVE_RUNTIME: ${{ inputs.native-runtime }}
NODE_VERSION: ${{ inputs.node-version }}
NATIVE_SOURCE_HASH: ${{ hashFiles('pnpm-lock.yaml', 'pnpm-workspace.yaml', '.npmrc', '.pnpmfile.cjs', '.github/actions/prepare-native-runtime/action.yml', 'config/scripts/ensure-native-runtime.mjs', 'config/scripts/initialize-windows-native-toolchain.mjs', 'config/scripts/capture-windows-native-environment.cmd', 'config/scripts/rebuild-native-deps.mjs', 'config/scripts/node-pty-job-ownership.cjs', 'config/scripts/windows-pe-machine.cjs', 'config/scripts/windows-process-tree-gyp-rebuild.mjs', 'config/scripts/windows-process-tree-creation-time.cjs', 'config/scripts/install-electron-package-binary.mjs', 'config/scripts/electron-platform-path.mjs', 'config/scripts/zip-extractor-command.mjs', 'src/shared/zip-extractor-command.ts', 'config/scripts/shared-electron-dist-cache.mjs', 'config/scripts/space-sharing-copy.mjs', 'config/patches/node-pty@1.1.0.patch', 'config/patches/@vscode__windows-process-tree@0.8.0.patch', 'native/windows-registry/src/addon.cc', 'native/windows-registry/binding.gyp', 'native/windows-registry/package.json', 'native/windows-registry/index.js', 'native/windows-security/src/addon.cc', 'native/windows-security/binding.gyp', 'native/windows-security/package.json', 'native/windows-security/index.js') }}
NATIVE_SOURCE_HASH: ${{ hashFiles('.npmrc', 'config/pnpmfile.cjs', '.github/actions/prepare-native-runtime/action.yml', 'config/scripts/ensure-native-runtime.mjs', 'config/scripts/initialize-windows-native-toolchain.mjs', 'config/scripts/capture-windows-native-environment.cmd', 'config/scripts/rebuild-native-deps.mjs', 'config/scripts/node-pty-job-ownership.cjs', 'config/scripts/windows-pe-machine.cjs', 'config/scripts/windows-process-tree-gyp-rebuild.mjs', 'config/scripts/windows-process-tree-creation-time.cjs', 'config/scripts/install-electron-package-binary.mjs', 'config/scripts/electron-platform-path.mjs', 'config/scripts/zip-extractor-command.mjs', 'src/shared/zip-extractor-command.ts', 'config/scripts/shared-electron-dist-cache.mjs', 'config/scripts/space-sharing-copy.mjs', 'config/scripts/shared-tree-publish.mjs', 'config/scripts/native-build-entry.cjs', 'config/scripts/native-build-entry-probe.cjs', 'config/scripts/shared-native-build.mjs', 'config/scripts/orcad-prebuild-node-gyp.mjs', 'config/scripts/msbuild-file-tracking.mjs', 'config/scripts/native-dependency-lock-digest.mjs', 'config/patches/node-pty@1.1.0.patch', 'config/patches/@vscode__windows-process-tree@0.8.0.patch', 'native/windows-registry/src/addon.cc', 'native/windows-registry/binding.gyp', 'native/windows-registry/package.json', 'native/windows-registry/index.js', 'native/windows-security/src/addon.cc', 'native/windows-security/binding.gyp', 'native/windows-security/package.json', 'native/windows-security/index.js') }}
run: |
scope="${ImageOS:-$RUNNER_OS}"
if [ -r /etc/os-release ]; then
@@ -63,8 +63,11 @@ runs:
fi
pnpm_version="$(pnpm --version)"
test -n "$pnpm_version"
# Locked native closures and the Electron release, not the whole lockfile.
lock_digest="$(node config/scripts/native-dependency-lock-digest.mjs)"
test -n "$lock_digest"
printf 'scope=%s\n' "$scope" >> "$GITHUB_OUTPUT"
printf 'key=native-modules-%s-%s-%s-%s-node%s-pnpm%s-%s\n' "$RUNNER_OS" "$scope" "$RUNNER_ARCH" "$NATIVE_RUNTIME" "$NODE_VERSION" "$pnpm_version" "$NATIVE_SOURCE_HASH" >> "$GITHUB_OUTPUT"
printf 'key=native-modules-%s-%s-%s-%s-node%s-pnpm%s-%s-%s\n' "$RUNNER_OS" "$scope" "$RUNNER_ARCH" "$NATIVE_RUNTIME" "$NODE_VERSION" "$pnpm_version" "$NATIVE_SOURCE_HASH" "$lock_digest" >> "$GITHUB_OUTPUT"
cat >> "$GITHUB_OUTPUT" <<'ORCA_NATIVE_CACHE_PATHS'
path<<ORCA_NATIVE_BUILD_DIRECTORIES
node_modules/.pnpm/node-pty@*/node_modules/node-pty/build
@@ -74,9 +77,13 @@ runs:
ORCA_NATIVE_BUILD_DIRECTORIES
ORCA_NATIVE_CACHE_PATHS
# The warmers seed every Windows image and runtime, and hosted Linux Electron, on main; PR copies were single-use uploads.
- name: Restore compiled native modules
id: native-cache-restore
if: inputs.native-runtime != 'none' && inputs.persist-native-cache != 'false'
if: >-
inputs.native-runtime != 'none' && inputs.persist-native-cache != 'false' &&
!(github.event_name == 'pull_request' && (runner.os == 'Windows' ||
runner.os == 'Linux' && inputs.native-runtime == 'electron' && runner.environment == 'github-hosted'))
uses: actions/cache@v5
with:
# Post-job saving cannot resolve nested composite step outputs.
@@ -89,7 +96,10 @@ runs:
- name: Restore compiled native modules without saving
id: native-cache-restore-only
if: inputs.native-runtime != 'none' && inputs.persist-native-cache == 'false'
if: >-
inputs.native-runtime != 'none' && (inputs.persist-native-cache == 'false' ||
github.event_name == 'pull_request' && (runner.os == 'Windows' ||
runner.os == 'Linux' && inputs.native-runtime == 'electron' && runner.environment == 'github-hosted'))
uses: actions/cache/restore@v5
with:
path: |
@@ -5,6 +5,14 @@ inputs:
save-cache:
description: Publish a validated helper from the default branch.
default: 'false'
build:
description: Build here; 'false' only restores, so the caller can build in a background run step.
default: 'true'
outputs:
cache-hit:
description: Whether the helper was restored under the exact key.
value: ${{ steps.cache.outputs.cache-hit }}
runs:
using: composite
@@ -30,22 +38,14 @@ runs:
key: ${{ steps.identity.outputs.key }}
- name: Build or validate Wayland clipboard helper
if: inputs.build != 'false'
shell: bash
env:
CACHE_HIT: ${{ steps.cache.outputs.cache-hit }}
run: |
if node config/scripts/build-wayland-clipboard.mjs; then
exit 0
fi
if [ "$CACHE_HIT" != true ]; then
exit 1
fi
echo 'Discarding an unusable Wayland clipboard cache and rebuilding from source.'
rm -rf -- native/wayland-clipboard/.build
node config/scripts/build-wayland-clipboard.mjs
run: bash "$GITHUB_ACTION_PATH/build-or-validate.sh"
- name: Save validated Wayland clipboard helper from main
if: inputs.save-cache == 'true' && steps.cache.outputs.cache-hit != 'true' && github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
if: inputs.build != 'false' && inputs.save-cache == 'true' && steps.cache.outputs.cache-hit != 'true' && github.ref == 'refs/heads/main' && github.event_name != 'pull_request'
uses: actions/cache/save@v5
continue-on-error: true
with:
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
# Validates a restored helper and rebuilds once from source when the restored copy is unusable.
set -e
if node config/scripts/build-wayland-clipboard.mjs; then
exit 0
fi
if [ "${CACHE_HIT:-}" != true ]; then
exit 1
fi
echo 'Discarding an unusable Wayland clipboard cache and rebuilding from source.'
rm -rf -- native/wayland-clipboard/.build
node config/scripts/build-wayland-clipboard.mjs
@@ -4,6 +4,14 @@ description: Compiles native/windows-agent-hook into native/windows-agent-hook/.
runs:
using: composite
steps:
- name: Resolve the Windows Rust build scope
id: rust-scope
if: runner.os == 'Windows'
shell: bash
run: node config/scripts/windows-rust-build-scope.mjs >> "$GITHUB_OUTPUT"
# Source globs, not crate/**: target/ and .build/ can already exist when this key is computed.
# Real inputs only: older restores are safe because the build re-checks the exe's fingerprint.
- name: Cache the Windows agent hook build
if: runner.os == 'Windows'
uses: actions/cache@v5
@@ -12,16 +20,13 @@ runs:
native/windows-agent-hook/.build
native/windows-agent-hook/target
~/.cargo/registry
key: windows-agent-hook-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('native/windows-agent-hook/**', 'config/scripts/build-windows-agent-hook.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'resources/build/icon.ico', 'package.json') }}
key: windows-agent-hook-v2-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-scope.outputs.toolchain }}-${{ steps.rust-scope.outputs.app-version }}-${{ hashFiles('native/windows-agent-hook/src/**', 'native/windows-agent-hook/build.rs', 'native/windows-agent-hook/Cargo.toml', 'native/windows-agent-hook/Cargo.lock', 'native/windows-agent-hook/app.manifest', 'native/windows-agent-hook/.cargo/**', 'config/scripts/build-windows-agent-hook.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'config/scripts/windows-pe-imports.mjs', 'resources/build/icon.ico') }}
restore-keys: |
windows-agent-hook-v2-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-scope.outputs.toolchain }}-
# Why explicit: a runner image without cargo would otherwise fail inside electron-builder's verify.
- name: Build the Windows agent hook
if: runner.os == 'Windows'
shell: pwsh
run: |
if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
rustup toolchain install stable --profile minimal
rustup default stable
}
node config/scripts/build-windows-agent-hook.mjs
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
@@ -9,6 +9,14 @@ inputs:
runs:
using: composite
steps:
- name: Resolve the Windows Rust build scope
id: rust-scope
if: runner.os == 'Windows'
shell: bash
run: node config/scripts/windows-rust-build-scope.mjs >> "$GITHUB_OUTPUT"
# Source globs, not crate/**: target/ and .build/ can already exist when this key is computed.
# Real inputs only: older restores are safe because the build re-checks each exe's fingerprint.
- name: Cache the Windows CLI launcher builds
if: runner.os == 'Windows'
uses: actions/cache@v5
@@ -17,19 +25,16 @@ runs:
.build/windows-cli-launcher
native/windows-cli-launcher/target
~/.cargo/registry
key: orcad-windows-cli-launcher-${{ runner.arch }}-${{ inputs.arches }}-${{ hashFiles('native/windows-cli-launcher/**', 'config/scripts/build-windows-cli-launcher.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'resources/build/icon.ico', 'package.json') }}
key: orcad-windows-cli-launcher-v2-${{ runner.arch }}-${{ inputs.arches }}-${{ steps.rust-scope.outputs.toolchain }}-${{ steps.rust-scope.outputs.app-version }}-${{ hashFiles('native/windows-cli-launcher/src/**', 'native/windows-cli-launcher/build.rs', 'native/windows-cli-launcher/Cargo.toml', 'native/windows-cli-launcher/Cargo.lock', 'native/windows-cli-launcher/app.manifest', 'native/windows-cli-launcher/.cargo/**', 'config/scripts/build-windows-cli-launcher.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'config/scripts/windows-pe-imports.mjs', 'resources/build/icon.ico') }}
restore-keys: |
orcad-windows-cli-launcher-v2-${{ runner.arch }}-${{ inputs.arches }}-${{ steps.rust-scope.outputs.toolchain }}-
# Why explicit: a runner image without cargo would otherwise fail inside the orcad build.
- name: Build the launcher for each architecture
if: runner.os == 'Windows'
shell: pwsh
env:
ARCHES: ${{ inputs.arches }}
run: |
if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
rustup toolchain install stable --profile minimal
rustup default stable
}
$triples = @{ x64 = 'x86_64-pc-windows-msvc'; arm64 = 'aarch64-pc-windows-msvc' }
foreach ($arch in ($env:ARCHES -split ' ' | Where-Object { $_ })) {
rustup target add $triples[$arch]
+57
View File
@@ -0,0 +1,57 @@
name: Setup verified pnpm
description: Install the pinned pnpm, with a signed npm fallback when version metadata is unavailable.
inputs:
package-json-file:
description: Repository-relative manifest declaring the pinned packageManager.
required: false
default: package.json
runs:
using: composite
steps:
- name: Setup pnpm
id: primary
continue-on-error: true
uses: pnpm/setup@v2
with:
package-json-file: ${{ inputs.package-json-file }}
install: false
- name: Setup Node.js for the npm fallback
if: steps.primary.outcome == 'failure'
uses: actions/setup-node@v6
with:
node-version: '24'
package-manager-cache: false
# npm resolves full packuments; pnpm/setup's version endpoint can stall independently.
- name: Bootstrap and verify pnpm through npm
if: steps.primary.outcome == 'failure'
shell: bash
env:
PNPM_MANIFEST: ${{ inputs.package-json-file }}
run: |
set -euo pipefail
bootstrap_dir="$(mktemp -d "$RUNNER_TEMP/orca-pnpm.XXXXXX")"
pnpm_version="$(node -p 'const p = require(require("node:path").resolve(process.env.GITHUB_WORKSPACE, process.env.PNPM_MANIFEST)); const match = /^pnpm@(\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?)(?:\+sha\d+\.[0-9a-f]+)?$/.exec(p.packageManager); if (!match) throw new Error("packageManager must pin an exact pnpm version"); match[1]')"
npm install --prefix "$bootstrap_dir" --ignore-scripts --no-audit --no-fund --save-exact --package-lock=true --registry=https://registry.npmjs.org "pnpm@$pnpm_version"
npm audit signatures --prefix "$bootstrap_dir" --registry=https://registry.npmjs.org
PNPM_BOOTSTRAP_DIR="$bootstrap_dir" node <<'NODE'
const { readFileSync } = require('node:fs')
const { join, resolve } = require('node:path')
const manifest = JSON.parse(readFileSync(resolve(process.env.GITHUB_WORKSPACE, process.env.PNPM_MANIFEST), 'utf8'))
const [pin, checksum] = manifest.packageManager.split('+')
if (checksum) {
const [algorithm, expected] = checksum.split('.')
const lock = JSON.parse(readFileSync(join(process.env.PNPM_BOOTSTRAP_DIR, 'package-lock.json'), 'utf8'))
const version = pin.slice('pnpm@'.length)
const entry = Object.values(lock.packages).find((entry) => entry.version === version && entry.resolved === `https://registry.npmjs.org/pnpm/-/pnpm-${version}.tgz`)
const actual = `${algorithm}-${Buffer.from(expected, 'hex').toString('base64')}`
if (!entry || entry.integrity !== actual) throw new Error('Installed pnpm does not match packageManager integrity')
}
NODE
npm rebuild --prefix "$bootstrap_dir" pnpm
actual_version="$("$bootstrap_dir/node_modules/.bin/pnpm" --version)"
test "$actual_version" = "$pnpm_version"
printf '%s\n' "$bootstrap_dir/node_modules/.bin" >> "$GITHUB_PATH"
@@ -20,10 +20,10 @@ $installer = Start-Process msiexec.exe -ArgumentList @('/i', $kernelMsi, '/quiet
if ($installer.ExitCode -ne 0) { throw "WSL kernel installation failed: $($installer.ExitCode)" }
wsl.exe --status
if ($LASTEXITCODE -ne 0) { throw "WSL status failed: $LASTEXITCODE" }
wsl.exe --distribution Ubuntu --user root --exec /usr/bin/curl --fail --silent --show-error --location https://nodejs.org/dist/v22.14.0/node-v22.14.0-linux-x64.tar.xz --output /tmp/orca-node.tar.xz
wsl.exe --distribution Ubuntu --user root --exec /usr/bin/curl --fail --silent --show-error --location https://nodejs.org/dist/v22.23.4/node-v22.23.4-linux-x64.tar.xz --output /tmp/orca-node.tar.xz
if ($LASTEXITCODE -ne 0) { throw 'Node download failed' }
$nodeHash = wsl.exe --distribution Ubuntu --user root --exec /usr/bin/sha256sum /tmp/orca-node.tar.xz
if ($LASTEXITCODE -ne 0 -or -not ($nodeHash -match '^69b09dba5c8dcb05c4e4273a4340db1005abeafe3927efda2bc5b249e80437ec')) { throw 'Node checksum mismatch' }
if ($LASTEXITCODE -ne 0 -or -not ($nodeHash -match '^1805bcd35e6815559bd2eb97db438d3fdb0eb3a5e6a977b7c1badca76337e9ce')) { throw 'Node checksum mismatch' }
wsl.exe --distribution Ubuntu --user root --exec /usr/bin/tar -xJf /tmp/orca-node.tar.xz -C /usr/local --strip-components=1
if ($LASTEXITCODE -ne 0) { throw 'Node extraction failed' }
wsl.exe --distribution Ubuntu --user root --exec /usr/local/bin/node --version
+1 -2
View File
@@ -1,7 +1,6 @@
version: 2
updates:
# Why only ripgrep: the bundled rg ships in every artifact and to SSH remotes, and a bump is a
# one-line pin change (the SSH cache keys on the binary's hash). Other dependencies stay manual.
# Automate only bundled ripgrep’s single pin; its hash invalidates SSH cache. Other dependency updates stay manual.
- package-ecosystem: npm
directory: /
schedule:
+3 -3
View File
@@ -12,7 +12,7 @@
## Linked Issue
_If you do not have one and are an outside contributors, your PR **wiil** be ignored. Refs is not sufficient. Link an actual issue_
_If you do not have one and are an outside contributor, your PR **will** be ignored. Refs is not sufficient. Link an actual issue_
<!-- Link the issue this PR addresses, there should ALWAYS be one (for outside contributors) -->
<!-- SPECIAL CASE: If you are a maintainer (member of stablyai org) AVOID opening needless issues. Only attach pre-existing ones -->
@@ -20,7 +20,7 @@ Fixes #
## Visual Proof
<!-- REQUIRED for UI / behavior changes. Please attach a BEFORE and AFTER that can easily tabbed/switched. Use videos for when appropriate over screenshots -->
<!-- REQUIRED for UI / behavior changes. Please attach a BEFORE and AFTER that can easily be tabbed/switched. Use videos for when appropriate over screenshots -->
<!-- If there is truly no visual or interaction change, write exactly: `N/A` and briefly say why. -->
<!-- For attachments NEVER add directly to the PR files (do not commit to files), use `gh image` extension or drag + drop (works for any attachment) -->
@@ -44,7 +44,7 @@ Fixes #
## Notes
Ensure no issues in: Security, Cross-platoform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance
Ensure no issues in: Security, Cross-platform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance
## Checklist
@@ -2,10 +2,7 @@ import { execFileSync } from 'node:child_process'
import { randomUUID } from 'node:crypto'
function readRootEntries(sha) {
// Why: a git pathname is arbitrary bytes, and 'utf8' folds every invalid
// sequence to U+FFFD — that mangles the reported name and makes two different
// entries compare equal, so a genuinely new one can slip past the Set below.
// latin1 maps each byte to one code unit, so the bytes survive the round trip.
// Git paths are arbitrary bytes; latin1 round-trips them while utf8 merges invalid sequences into U+FFFD, hiding distinct entries.
const stdout = execFileSync('git', ['ls-tree', '-z', '--name-only', sha], {
encoding: 'latin1',
stdio: ['ignore', 'pipe', 'inherit']
@@ -14,7 +11,7 @@ function readRootEntries(sha) {
}
// These reviewed additions stay listed until the base tree carries them.
const REVIEWED_ROOT_ENTRIES = new Set(['cloud', 'opencode.json'])
const REVIEWED_ROOT_ENTRIES = new Set(['.claude', '.mcp.json', 'cloud', 'opencode.json'])
function checkRootDirectoryEntries(argv) {
if (argv.length !== 2) {
@@ -42,10 +39,7 @@ function checkRootDirectoryEntries(argv) {
)
console.log('Move each new entry under an existing top-level directory.')
console.log('Blocked entries:')
// Why: an entry name is attacker-controlled and may start with '::' (the runner
// trims leading spaces before matching) or embed a newline, so printing it bare
// lets a PR forge annotations. Fence the untrusted list with an unguessable
// stop-commands token, and write the raw bytes rather than a re-encoded string.
// Untrusted names can forge runner commands/newlines; fence them with a random stop-commands token and preserve raw bytes.
const resumeToken = randomUUID()
console.log(`::stop-commands::${resumeToken}`)
for (const entry of blockedEntries) {
@@ -56,12 +50,10 @@ function checkRootDirectoryEntries(argv) {
}
try {
// Why: process.exit truncates a piped write part-way through on macOS, so set
// exitCode and let node flush the blocked-entry list before it exits.
// Let piped output flush; process.exit can truncate it on macOS.
process.exitCode = checkRootDirectoryEntries(process.argv.slice(2))
} catch (error) {
// Why: git already reported the failure on the inherited stderr, so surface its
// status rather than a node stack trace. Anything else is a real bug — rethrow.
// Git already reported failures; preserve its exit status and rethrow other bugs.
if (typeof error.status !== 'number') {
throw error
}
+1 -3
View File
@@ -3,9 +3,7 @@ export const LOC_BLOCK_END = '<!-- /orca-pr-loc -->'
export const LOC_HANDS_OFF_COMMENT =
'<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->'
// Why the __-wrapped names: they are unambiguous test-data markers, so a dir
// cannot be one by accident. Bare `fixtures/` is deliberately absent — it reads
// as a plausible prod module name, and this classifier bills every PR.
// __-wrapped segments unambiguously mark test data; bare fixtures/ could be production, and this classifier bills every PR.
const TEST_DIR_SEGMENT = /(?:^|\/)(?:__tests__|__fixtures__|__snapshots__|e2e|tests)(?:\/|$)/i
const TEST_FILENAME = /\.(?:test|spec|e2e)\.[^/]+$/i
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
set -euo pipefail
# GitHub's mirror+file sources resolve through apt-mirrors.txt, not just sources.list.
for source in /etc/apt/apt-mirrors.txt /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
if [ -f "$source" ]; then
sudo sed -i 's|https*://azure\.archive\.ubuntu\.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' "$source"
fi
done
sudo tee /etc/apt/apt.conf.d/99-orca-ci >/dev/null <<'APTCONF'
Acquire::http::Timeout "15";
Acquire::https::Timeout "15";
Acquire::Retries "1";
APTCONF
+420 -385
View File
@@ -1,58 +1,12 @@
name: Adhoc macOS + Windows Dev Build
# Why: lets anyone cut installable macOS and Windows builds of a trusted repo ref
# so the team can run an experimental feature instead of reasoning from a diff.
# Hourly covers current main; this covers experiments and on-demand validation.
#
# Deliberately narrow scope:
# - macOS and Windows desktop installers. Linux keeps using RC/stable.
# - No tests, no lint, no e2e. PR CI and release-cut remain the gates. The one
# exception is the orcad template: like release-cut, it is merged from the
# node-server lanes that build and qualify each SSH target's slot.
# - macOS is signed and notarized so TCC grants survive updates.
# - Windows is unsigned; the published release notes explain the one-time
# SmartScreen/manual-install requirement.
#
# Artifacts publish to stablyai/orca-adhoc — separate from both orca and
# orca-hourly. Separate from orca because the main repo's releases atom feed
# exposes only its 10 newest entries. Separate from orca-hourly because a build
# from someone's branch must never be picked up by a developer who only meant to
# ride main; the two are different levels of "unvetted".
#
# From the Actions tab: pick this workflow, "Run workflow", leave "Use workflow
# from" on main, and type your branch in the first field. Or from the CLI:
#
# gh workflow run adhoc-mac-build.yml --ref main -f ref=my-branch -f label=wasm-terminal
#
# Leaving the ref field empty builds whatever "Use workflow from" is set to, which
# is what someone who only touched that picker means. Naming the branch explicitly
# is still better: the workflow file is always read from the dispatch ref, so a
# branch carrying a stale copy of this file would otherwise run that copy.
#
# GITHUB_TOKEN is scoped to this repo and cannot publish there, so writes use the
# same GitHub App as hourly, additionally installed on orca-adhoc with
# Contents: Read and write. The secret names below are historical — one App, one
# private key, both dev-channel repos — and rotating it stays a single operation.
# Provision with `bash config/scripts/setup-hourly-release-token.sh`.
#
# The requested ref is vetted before checkout: it must be a branch or tag of this
# repo, or a commit reachable from one. PR refs are refused outright — this
# workflow runs the checked-out code next to MAC_CERTS and the notary password,
# so "just build that community PR" must not become a way to hand fork code the
# release identity. A branch here always belongs to someone with write access,
# which is the same trust the dispatch button itself already requires.
# Build trusted repo refs on demand: signed/notarized macOS and unsigned Windows, with qualified orcad slots; PR/release pipelines own other gates. Publish to orca-adhoc so branch experiments cannot enter main’s hourly channel or crowd stable/RC release feeds. Dispatch main’s workflow with -f ref=my-branch to avoid stale YAML. Writes use the hourly GitHub App installed on orca-adhoc (setup-hourly-release-token.sh); GITHUB_TOKEN cannot publish cross-repo. Ref vetting rejects PR refs and unreachable SHAs before source runs beside signing secrets.
on:
workflow_dispatch:
inputs:
ref:
# Why optional: the Actions UI already shows its own "Use workflow from"
# branch picker directly above this field, and picking a branch there is
# what most people will read as "build this". Defaulting to that branch
# makes the obvious action correct. Fill this in only to build a ref other
# than the one the workflow file itself is read from — normally leave the
# picker on main and name your branch here, so a stale copy of this
# workflow on an old branch is not what runs.
# Default empty ref to the dispatch picker; prefer main’s workflow plus explicit branch input to build old branches with current YAML.
description: 'Branch, tag, or SHA to build — must live in stablyai/orca; PR refs are refused (default: the branch selected above)'
required: false
default: ''
@@ -67,29 +21,21 @@ permissions:
contents: read
concurrency:
# Why keyed on the ref rather than global: two people cutting builds from two
# different branches at the same time is the ordinary case here, and serialising
# them would make each wait out the other's notary queue. Re-dispatching the
# *same* branch still queues, so a push mid-build cannot race itself.
# Serialize per ref: queue repeat builds while independent branches avoid each other’s notary wait.
group: adhoc-mac-build-${{ inputs.ref || github.ref_name }}
cancel-in-progress: false
env:
ORCA_BACKGROUND_LAUNCH: '1'
ADHOC_REPO: stablyai/orca-adhoc
# Why age and not a count like hourly: this channel is low-volume and bursty, so
# a count would either hold one week's experiments forever or evict a build
# someone is still running after a busy afternoon. A month is well past the "few
# days" these exist for, and by then the branch has landed or been abandoned.
# Retain by month rather than count so bursts cannot evict recent experiments or sparse periods keep them forever.
ADHOC_RETAIN_DAYS: 30
jobs:
# Why: every job below checks out its own copy, and a branch name read per job builds whatever
# the branch points at when that job starts. A push mid-run then mixes commits, e.g. slot lanes
# from one commit merged by a template step from the next. Resolving once pins the whole run.
# The mac job still vets this commit's reachability before anything is signed.
# Resolve once before source execution so concurrent pushes cannot mix slot/template/desktop commits.
resolve-ref:
if: github.repository == 'stablyai/orca'
runs-on: ubuntu-latest
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 5
permissions:
contents: read
@@ -102,48 +48,69 @@ jobs:
env:
REQUESTED_REF: ${{ inputs.ref || github.ref_name }}
REPO_URL: https://github.com/${{ github.repository }}
run: |
run: &vet-repository-ref |
set -euo pipefail
case "$REQUESTED_REF" in
refs/pull/*|pull/*)
echo "::error::Refusing to build PR ref '$REQUESTED_REF'; push the code to a branch of stablyai/orca instead."
echo "::error::Refusing to build PR ref '$REQUESTED_REF': this workflow signs with release credentials, so it only builds branches, tags, or commits of stablyai/orca. Push the code to a branch of this repo instead."
exit 1
;;
esac
if [[ "$REQUESTED_REF" =~ ^[0-9a-f]{40}$ ]]; then
sha="$REQUESTED_REF"
else
# Branch first, as the mac job's vet does; a peeled annotated tag names its commit.
refs="$(git ls-remote "$REPO_URL" "refs/heads/$REQUESTED_REF" "refs/tags/$REQUESTED_REF" "refs/tags/$REQUESTED_REF^{}")"
sha=""
for name in "refs/heads/$REQUESTED_REF" "refs/tags/$REQUESTED_REF^{}" "refs/tags/$REQUESTED_REF"; do
sha="$(awk -v ref="$name" '$2 == ref { print $1; exit }' <<<"$refs")"
[[ -n "$sha" ]] && break
done
# Bare avoids fetching over a checked-out branch; tree:0 fetches only the commit graph.
# Reftable preserves case-only branch names that make the files backend fail on case-insensitive disks.
scratch="$RUNNER_TEMP/vet-requested-ref"
git init -q --bare --ref-format=reftable "$scratch"
git -C "$scratch" fetch -q --filter=tree:0 "$REPO_URL" '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'
# Resolve branches before tags, matching actions/checkout’s tie-break.
sha=""
case "$REQUESTED_REF" in
refs/heads/*|refs/tags/*) candidates=("$REQUESTED_REF") ;;
*) candidates=("refs/heads/$REQUESTED_REF" "refs/tags/$REQUESTED_REF") ;;
esac
if [[ "$REQUESTED_REF" =~ ^[0-9a-fA-F]{40}$ ]]; then
candidates+=("$REQUESTED_REF")
fi
for cand in "${candidates[@]}"; do
if ! [[ "$cand" =~ ^[0-9a-fA-F]{40}$ ]] && ! git check-ref-format "$cand"; then
continue
fi
if sha="$(git -C "$scratch" rev-parse --verify --quiet --end-of-options "$cand^{commit}")"; then
break
fi
sha=""
done
if [[ -z "$sha" ]]; then
echo "::error::'$REQUESTED_REF' is not a branch, tag, or full commit SHA of stablyai/orca."
echo "::error::'$REQUESTED_REF' does not resolve to a branch, tag, or commit of stablyai/orca."
exit 1
fi
echo "Resolved $REQUESTED_REF -> $sha"
# Lazy-fetchable SHAs can be PR-only; require reachability from a trusted branch/tag, not local object presence.
if [[ -z "$(git -C "$scratch" for-each-ref --count=1 --format='%(refname)' --contains "$sha" refs/heads refs/tags)" ]]; then
echo "::error::Commit $REQUESTED_REF is not reachable from any branch or tag of stablyai/orca; refusing to build it with release credentials."
exit 1
fi
echo "Vetted $REQUESTED_REF -> $sha"
echo "sha=$sha" >>"$GITHUB_OUTPUT"
# Why: a branch cut before the orcad template landed has none to build or ship.
# Resolve shared build identity while detecting whether this ref ships a template.
orcad-template-support:
needs: resolve-ref
runs-on: ubuntu-latest
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 5
permissions:
contents: read
outputs:
ships: ${{ steps.detect.outputs.ships }}
version: ${{ steps.adhoc.outputs.version }}
commit: ${{ steps.adhoc.outputs.commit }}
name: ${{ steps.adhoc.outputs.name }}
steps:
- name: Checkout the template packager only
- name: Checkout template detection and version scripts
uses: actions/checkout@v6
with:
ref: ${{ needs.resolve-ref.outputs.sha }}
sparse-checkout: |
/config/scripts/packaged-orcad-template.cjs
/config/scripts/
/package.json
/config/tsconfig.relay.json
sparse-checkout-cone-mode: false
persist-credentials: false
@@ -166,182 +133,12 @@ jobs:
echo "::notice::This ref predates the orcad template; the build ships without it."
fi
# Design D2, as release-cut does: every desktop build ships the orcad template (server JS plus
# every target's addons), merged from the node-server lanes that qualify each slot at this ref.
# Without it an adhoc build cannot deploy managed orcad to an SSH host. No secrets, and the mac
# job vets the ref before anything is signed.
orcad-template:
needs: [resolve-ref, orcad-template-support]
if: needs.orcad-template-support.outputs.ships == 'true'
permissions:
contents: read
uses: ./.github/workflows/node-server-tests.yml
with:
ref: ${{ needs.resolve-ref.outputs.sha }}
build_template: true
build-adhoc-mac:
needs: [resolve-ref, orcad-template-support, orcad-template]
# Why not the implicit success(): a ref without the orcad template skips that job on purpose.
if: >-
!cancelled() && github.repository == 'stablyai/orca' &&
needs.orcad-template-support.result == 'success' &&
(needs.orcad-template.result == 'success' ||
(needs.orcad-template.result == 'skipped' &&
needs.orcad-template-support.outputs.ships == 'false'))
# Why an environment: it gives the signing/notary/App secrets somewhere to
# live that a stale copy of this workflow on an old branch cannot reach.
# Referencing it is a no-op until repo settings give it teeth; the intended
# follow-up is to move MAC_CERTS, MAC_CERTS_PASSWORD, APPLE_ID,
# APPLE_APP_SPECIFIC_PASSWORD, APPLE_TEAM_ID, HOURLY_RELEASE_APP_ID, and
# HOURLY_RELEASE_APP_PRIVATE_KEY into it, then pin its deployment branch
# policy to main so only main's copy of this file can read them.
environment: adhoc-mac-build
outputs:
tag: ${{ steps.release.outputs.tag }}
version: ${{ steps.adhoc.outputs.version }}
head_sha: ${{ steps.adhoc.outputs.head_sha }}
published: ${{ steps.publish_live.outcome == 'success' && 'true' || 'false' }}
ships_orcad_template: ${{ needs.orcad-template-support.outputs.ships }}
runs-on: blacksmith-6vcpu-macos-15
# Why 150: it must exceed the worst case the retry budgets below can produce
# (install 3x10 + publish 2x45 = 120, plus ~25 for checkout/build/verify), or
# the job is killed mid-retry and no cleanup step runs at all.
timeout-minutes: 150
env:
NODE_OPTIONS: --max-old-space-size=4096
steps:
# Why vet before checkout: everything after this step runs the checked-out
# code with release signing credentials in reach. Branches and tags of this
# repo are the intended audience; refs/pull/* would smuggle in fork code,
# and a raw SHA is only accepted when some branch or tag of this repo can
# actually reach it. Resolving to a pinned SHA here also means the commit
# that was vetted is the commit that gets checked out — a push to the
# branch between the two steps cannot swap it.
- name: Vet the requested ref
id: vetted
shell: bash
env:
# The commit resolve-ref pinned for every job; vetting it keeps the reachability test.
REQUESTED_REF: ${{ needs.resolve-ref.outputs.sha }}
REPO_URL: https://github.com/${{ github.repository }}
run: |
set -euo pipefail
case "$REQUESTED_REF" in
refs/pull/*|pull/*)
echo "::error::Refusing to build PR ref '$REQUESTED_REF': this workflow signs with release credentials, so it only builds branches, tags, or commits of stablyai/orca. Push the code to a branch of this repo instead."
exit 1
;;
esac
# Bare: a work-tree repo refuses to fetch over its own checked-out
# branch. tree:0 keeps the fetch to the commit graph — no trees, no
# blobs — so this stays cheap next to the build it fronts. reftable
# because this repo has branches that differ only in casing, and the
# files backend cannot store both on a case-insensitive runner disk —
# it fails the entire fetch, not just the one ref.
scratch="$RUNNER_TEMP/vet-requested-ref"
git init -q --bare --ref-format=reftable "$scratch"
git -C "$scratch" fetch -q --filter=tree:0 "$REPO_URL" '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'
# Branch first to keep actions/checkout's old tie-break: bare
# rev-parse would prefer the tag when a branch shares its name.
sha=""
for cand in "refs/heads/$REQUESTED_REF" "refs/tags/$REQUESTED_REF" "$REQUESTED_REF"; do
if sha="$(git -C "$scratch" rev-parse --verify --quiet "$cand^{commit}")"; then
break
fi
sha=""
done
if [[ -z "$sha" ]]; then
echo "::error::'$REQUESTED_REF' does not resolve to a branch, tag, or commit of stablyai/orca."
exit 1
fi
# The object resolving locally is not proof a branch or tag reaches
# it: a partial clone can lazily fetch a bare SHA on demand, and
# GitHub serves PR-only commits by SHA. Reachability is the actual
# trust test.
if [[ -z "$(git -C "$scratch" for-each-ref --contains "$sha" refs/heads refs/tags | head -1)" ]]; then
echo "::error::Commit $REQUESTED_REF is not reachable from any branch or tag of stablyai/orca; refusing to build it with release credentials."
exit 1
fi
echo "Vetted $REQUESTED_REF -> $sha"
echo "sha=$sha" >>"$GITHUB_OUTPUT"
- name: Checkout the requested ref
uses: actions/checkout@v6
with:
# Why an input at all rather than just github.ref: the whole point is to
# build code that has not landed, and the workflow definition itself
# always comes from the dispatch ref — naming the branch here instead
# applies main's current copy of this file to an arbitrary branch.
ref: ${{ steps.vetted.outputs.sha }}
# Version helpers only read HEAD; published versions come from the release API.
fetch-depth: 1
# This job only reads stablyai/orca and never pushes; every write goes
# to the adhoc repo through a minted App token passed by env. Not
# persisting the checkout credential shrinks the blast radius if a build
# step is compromised (zizmor: artipacked).
persist-credentials: false
- name: Setup pnpm
uses: pnpm/setup@v2
with:
install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
cache: pnpm
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
# Why: packaging bundles the pinned Node into the macOS terminal host. Keyed on the pin alone
# with no restore-keys, so a pin bump starts empty instead of carrying old archives forward.
- name: Locate the pinned Node runtime cache
shell: bash
run: echo "ORCA_NODE_RUNTIME_CACHE_DIR=$HOME/Library/Caches/orca-node-runtime" >>"$GITHUB_ENV"
- name: Cache the pinned Node runtime
uses: actions/cache@v5
with:
path: ~/Library/Caches/orca-node-runtime
key: orca-node-runtime-mac-${{ hashFiles('src/shared/node-runtime-pin.ts') }}
- name: Cache electron-builder downloads
uses: actions/cache@v5
with:
path: |
~/Library/Caches/electron
~/Library/Caches/electron-builder
key: electron-builder-mac-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: |
electron-builder-mac-
# Why both CPUs: the mac config packages x64 and arm64 from this arm64
# runner, so the install must carry both variants of the native optional deps.
- name: Install dependencies
uses: nick-fields/retry@v4
with:
timeout_minutes: 10
max_attempts: 3
retry_wait_seconds: 30
command: pnpm install --frozen-lockfile --cpu=current,x64,arm64
# Why here: electron-builder's beforePack requires out/mobile-web, and the bundle
# build resolves React Native and Expo from mobile/node_modules.
- uses: ./.github/actions/install-mobile-dependencies
# Why: signing is what makes an adhoc build installable over an existing
# Orca, so a missing cert must fail here rather than after a 20-minute build.
- name: Verify macOS signing environment
run: node config/scripts/verify-macos-release-env.mjs
env:
CSC_LINK: ${{ secrets.MAC_CERTS }}
CSC_KEY_PASSWORD: ${{ secrets.MAC_CERTS_PASSWORD }}
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
check-latest: true
package-manager-cache: false
- name: Compute adhoc version
id: adhoc
@@ -352,21 +149,15 @@ jobs:
MAIN_REPO_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# Why this check: the version scripts are read from the branch being built,
# not from main, so a branch cut before the adhoc channel landed has no
# copy of them. Say that plainly instead of failing with a module-not-found.
# Old branch refs may predate adhoc version scripts; report that explicitly.
for script in adhoc-build-version dev-channel-base-version; do
if [[ ! -f "config/scripts/$script.mjs" ]]; then
echo "::error::$REF has no config/scripts/$script.mjs; rebase it onto a main that has the adhoc channel."
exit 1
fi
done
echo "head_sha=$(git rev-parse HEAD)" >>"$GITHUB_OUTPUT"
# Why git tags, not GitHub releases: unpublishing a buggy cut deletes the
# GitHub release and leaves the tag, which still owns that number.
# Releases-only let adhoc sit on a number already taken, so the updater
# would not install it. Empty on failure — the script then falls back
# to package.json.
# Unpublishing deletes releases but leaves version-owning tags; use tags to prevent updater regression.
# On lookup failure the script falls back to package.json.
published="$(GH_TOKEN="$MAIN_REPO_TOKEN" gh api \
"repos/$GITHUB_REPOSITORY/git/matching-refs/tags/v" \
--jq '.[].ref | sub("^refs/tags/"; "")' || true)"
@@ -379,27 +170,40 @@ jobs:
fi
cat "$RUNNER_TEMP/adhoc-identity.txt" >>"$GITHUB_OUTPUT"
- name: Build app
run: pnpm build:release
env:
NODE_OPTIONS: --max-old-space-size=4096
# Why: adhoc builds are not an official channel — telemetry's transport
# gate accepts only 'stable' or 'rc', so leaving this unset keeps them
# silent, which is correct for unvetted branch artifacts.
ORCA_DIAGNOSTICS_TOKEN_URL: https://www.onorca.dev/diagnostics/token
# Ship managed-SSH orcad template from qualified per-target lanes at the vetted ref, as release-cut does.
orcad-template:
needs: [resolve-ref, orcad-template-support]
if: needs.orcad-template-support.outputs.ships == 'true'
permissions:
contents: read
actions: read
uses: ./.github/workflows/qualified-orcad-template.yml
with:
ref: ${{ needs.resolve-ref.outputs.sha }}
fast_runners: true
# After the app build so nothing that cleans out/ can drop it; electron-builder ships it.
- name: Download the orcad deployment template
if: needs.orcad-template-support.outputs.ships == 'true'
uses: actions/download-artifact@v8
with:
name: orcad-template
path: out/orcad-template
release-javascript:
needs: [resolve-ref, orcad-template-support]
uses: ./.github/workflows/release-javascript.yml
with:
ref: ${{ needs.resolve-ref.outputs.sha }}
dev_build: true
runner: blacksmith-8vcpu-ubuntu-2404
# Why the token is minted here and not at the top: installation tokens live
# one hour, everything before this point writes nothing, and the notary round
# trip inside the publish step can be tens of minutes. Minting after the build
# starts the clock at the first call that actually uses it.
prepare-adhoc-mac-release:
needs: [resolve-ref, orcad-template-support, release-javascript]
if: >-
!cancelled() && github.repository == 'stablyai/orca' &&
needs.orcad-template-support.result == 'success' &&
needs.release-javascript.result == 'success'
environment: adhoc-mac-build
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 10
permissions:
contents: read
outputs:
tag: ${{ steps.release.outputs.tag }}
steps:
- name: Mint adhoc repo token
id: app_token
uses: actions/create-github-app-token@v2
@@ -414,26 +218,17 @@ jobs:
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
TAG: v${{ steps.adhoc.outputs.version }}
NAME: ${{ steps.adhoc.outputs.name }}
SHA: ${{ steps.adhoc.outputs.head_sha }}
TAG: v${{ needs.orcad-template-support.outputs.version }}
NAME: ${{ needs.orcad-template-support.outputs.name }}
SHA: ${{ needs.resolve-ref.outputs.sha }}
REF: ${{ inputs.ref || github.ref_name }}
# Via env, not inline `${{ }}`: both land inside a shell string, and an
# expression expanded there is substituted before bash parses the line
# (zizmor: template-injection).
# Pass expressions via env so bash parses data rather than substituted script text (zizmor: template-injection).
ACTOR: ${{ github.actor }}
run: |
set -euo pipefail
short_sha="${SHA:0:12}"
# Why create it up front: electron-builder then uploads into a known tag
# rather than inferring one from package.json.
#
# Why --draft: everything between here and the manifest check is a window
# where the release exists but has no installable assets. A draft is
# absent from the releases list and from listReleaseBuilds, so a job that
# dies in that window — including a hard kill by the job timeout, which
# runs no cleanup step at all — leaves something invisible rather than a
# tag the picker offers and the download 404s on.
# Precreate a known tag as draft so builder cannot infer it from package.json.
# Keep incomplete assets invisible until manifest verification, even if timeout kills the job without cleanup.
gh release create "$TAG" \
--repo "$ADHOC_REPO" \
--title "$NAME" \
@@ -454,91 +249,329 @@ jobs:
including back to Stable, works in-app from there."
echo "tag=$TAG" >>"$GITHUB_OUTPUT"
- name: Publish adhoc macOS artifacts
build-adhoc-mac-arch:
needs:
[
resolve-ref,
orcad-template-support,
orcad-template,
release-javascript,
prepare-adhoc-mac-release
]
# Start after template qualification: overlapping ~2.5 min of preparation left both Macs idle ~7 min.
if: >-
!cancelled() && github.repository == 'stablyai/orca' &&
needs.orcad-template-support.result == 'success' &&
(needs.orcad-template.result == 'success' || needs.orcad-template-support.outputs.ships != 'true') &&
needs.release-javascript.result == 'success' &&
needs.prepare-adhoc-mac-release.result == 'success'
# Signing environment must restrict deployments to main after MAC_CERTS/password, APPLE_ID/app password/team and hourly App secrets move there; referencing it alone enforces nothing.
environment: adhoc-mac-build
permissions:
contents: read
actions: read
strategy:
fail-fast: false
matrix:
arch: [x64, arm64]
# Notary waits dominate; 12 vCPUs packaged no faster than the hourly build’s 6.
runs-on: blacksmith-6vcpu-macos-15
# Allow 150min for install 3×10 + publish 2×45 + ~25 setup/build/verify; outer timeout would prevent cleanup.
timeout-minutes: 150
env:
JOBS: max
NODE_OPTIONS: --max-old-space-size=4096
steps:
# Vet branch/tag reachability and reject refs/pull before checkout can access signing credentials; checkout the resolved SHA to prevent ref-swap races.
- name: Vet the requested ref
id: vetted
shell: bash
env:
# The commit resolve-ref pinned for every job; vetting it keeps the reachability test.
REQUESTED_REF: ${{ needs.resolve-ref.outputs.sha }}
REPO_URL: https://github.com/${{ github.repository }}
run: *vet-repository-ref
- name: Checkout the requested ref
uses: actions/checkout@v6
with:
# Explicit source input lets main’s workflow build an unmerged branch rather than run that branch’s stale workflow.
ref: ${{ steps.vetted.outputs.sha }}
# Version helpers only read HEAD; published versions come from the release API.
fetch-depth: 1
# Read this repo without persisted checkout credentials; all writes use an env-passed App token for orca-adhoc (zizmor: artipacked).
persist-credentials: false
- name: Setup pnpm
uses: pnpm/setup@v2
with:
install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
cache: pnpm
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
# Cache bundled host Node by exact pin, without restore keys, so bumps cannot restore old archives.
- name: Locate the pinned Node runtime cache
shell: bash
run: echo "ORCA_NODE_RUNTIME_CACHE_DIR=$HOME/Library/Caches/orca-node-runtime" >>"$GITHUB_ENV"
- name: Cache the pinned Node runtime
uses: actions/cache@v5
with:
path: ~/Library/Caches/orca-node-runtime
key: orca-node-runtime-mac-${{ hashFiles('src/shared/node-runtime-pin.ts') }}
- name: Cache electron-builder downloads
uses: actions/cache@v5
with:
path: |
~/Library/Caches/electron
~/Library/Caches/electron-builder
key: electron-builder-mac-${{ hashFiles('pnpm-lock.yaml') }}
restore-keys: |
electron-builder-mac-
# Install both x64/arm64 optional natives because this arm64 runner packages both.
- name: Install dependencies
uses: nick-fields/retry@v4
with:
# Why 45: an attempt is pack + notarize + upload, and the notary queue is
# the unbounded part. Two attempts, because a failed adhoc build has a
# person waiting on it who can simply dispatch again.
timeout_minutes: 45
max_attempts: 2
timeout_minutes: 10
max_attempts: 3
retry_wait_seconds: 30
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && ORCA_MAC_ADHOC=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac --publish always
command: pnpm install --frozen-lockfile --cpu=current,x64,arm64
# Build required out/mobile-web after installing its React Native/Expo dependencies.
- uses: ./.github/actions/install-mobile-dependencies
if: needs.release-javascript.outputs.supported != 'true'
# Require signing cert before expensive build; unsigned macOS cannot update an existing install correctly.
- name: Verify macOS signing environment
run: node config/scripts/verify-macos-release-env.mjs
env:
# Why: electron-builder's github publisher targets the repo named in the
# config; the token must therefore carry write access to orca-adhoc.
GH_TOKEN: ${{ steps.app_token.outputs.token }}
ORCA_ADHOC_BUILD_VERSION: ${{ steps.adhoc.outputs.version }}
ORCA_BUILD_COMMIT: ${{ steps.adhoc.outputs.commit }}
# beforePack and afterPack fail the package when the template is absent.
ORCA_REQUIRE_ORCAD_TEMPLATE: ${{ needs.orcad-template-support.outputs.ships == 'true' && '1' || '' }}
CSC_LINK: ${{ secrets.MAC_CERTS }}
CSC_KEY_PASSWORD: ${{ secrets.MAC_CERTS_PASSWORD }}
# Why all three: electron-builder's notarize step authenticates to the
# Apple notary service with the app-specific password, not with the
# signing cert. Omitting them fails the build rather than skipping it.
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
# Why: the updater resolves a tag, then fetches latest-mac.yml from it. A
# release missing that manifest is a tag the picker offers and the download
# 404s on, so fail loudly instead of leaving a broken entry.
- name: Verify update manifest published
- name: Download shared JavaScript
if: needs.release-javascript.outputs.supported == 'true'
uses: actions/download-artifact@v8
with:
name: release-javascript
path: .build/release-javascript
- name: Build app
run: |
if [[ "$ORCA_RELEASE_JAVASCRIPT_SOURCE_SHA" != '' ]]; then
node config/scripts/release-javascript-artifact.mjs restore
pnpm run build:release:host
else
pnpm build:release
fi
env:
ORCA_RELEASE_JAVASCRIPT_SOURCE_SHA: ${{ needs.release-javascript.outputs.supported == 'true' && needs.release-javascript.outputs.source_sha || '' }}
ORCA_RELEASE_JAVASCRIPT_BUILD_IDENTITY: dev
NODE_OPTIONS: --max-old-space-size=4096
# Leave build identity unset: telemetry accepts only official stable/rc, so branch artifacts stay silent.
ORCA_DIAGNOSTICS_TOKEN_URL: https://www.onorca.dev/diagnostics/token
# After the app build so nothing that cleans out/ can drop it; electron-builder ships it.
- name: Download the orcad deployment template
if: needs.orcad-template-support.outputs.ships == 'true'
uses: actions/download-artifact@v8
with:
name: orcad-template
path: out/orcad-template
# Mint the one-hour write token after build, just before potentially long notarization/publish.
- name: Checkout workflow packaging scripts
uses: actions/checkout@v6
with:
ref: ${{ github.workflow_sha }}
path: .workflow-tools
sparse-checkout: config/scripts/adhoc-mac-manifests.mjs
sparse-checkout-cone-mode: false
persist-credentials: false
- name: Package and notarize this Mac architecture
uses: nick-fields/retry@v4
with:
# Allow two 45min pack/notarize attempts; an adhoc user can redispatch rather than pay for a third.
timeout_minutes: 45
max_attempts: 2
retry_wait_seconds: 30
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && ORCA_MAC_ADHOC=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac dmg zip --${{ matrix.arch }} --publish never
env:
ORCA_ADHOC_BUILD_VERSION: ${{ needs.orcad-template-support.outputs.version }}
ORCA_BUILD_COMMIT: ${{ needs.orcad-template-support.outputs.commit }}
# beforePack and afterPack fail the package when the template is absent.
ORCA_REQUIRE_ORCAD_TEMPLATE: ${{ needs.orcad-template-support.outputs.ships == 'true' && '1' || '' }}
CSC_LINK: ${{ secrets.MAC_CERTS }}
CSC_KEY_PASSWORD: ${{ secrets.MAC_CERTS_PASSWORD }}
# Require Apple id/app-specific password/team: notarization authenticates separately from signing cert.
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
- name: Verify signed and stapled app
shell: bash
env:
MAC_ARCH: ${{ matrix.arch }}
run: |
set -euo pipefail
app_dir=dist/mac
if [[ "$MAC_ARCH" == arm64 ]]; then app_dir=dist/mac-arm64; fi
codesign --verify --deep --strict "$app_dir/Orca.app"
xcrun stapler validate "$app_dir/Orca.app"
- name: Verify this architecture's updater files
env:
MAC_ARCH: ${{ matrix.arch }}
MAC_VERSION: ${{ needs.orcad-template-support.outputs.version }}
run: node .workflow-tools/config/scripts/adhoc-mac-manifests.mjs stage "$MAC_ARCH" "$MAC_VERSION" dist .build/mac-manifests
- name: Mint adhoc repo token
id: app_token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.HOURLY_RELEASE_APP_ID }}
private-key: ${{ secrets.HOURLY_RELEASE_APP_PRIVATE_KEY }}
owner: stablyai
repositories: orca-adhoc
- name: Upload verified architecture assets to the draft
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
TAG: ${{ steps.release.outputs.tag }}
TAG: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
MAC_ARCH: ${{ matrix.arch }}
run: |
set -euo pipefail
assets="$(gh release view "$TAG" --repo "$ADHOC_REPO" --json assets --jq '.assets[].name')"
echo "Published assets:"
echo "$assets"
# Why exit 1 without deleting here: the release is still a draft, so it is
# already invisible to users, and the failure handler below owns cleanup.
for required in latest-mac.yml; do
if ! grep -qx "$required" <<<"$assets"; then
echo "::error::Adhoc draft $TAG is missing $required; the updater could not install it."
exit 1
fi
done
if ! grep -q '\.zip$' <<<"$assets"; then
echo "::error::Adhoc draft $TAG has no ZIP artifact for the updater to download."
exit 1
fi
files=()
while IFS= read -r file; do files+=("dist/$file"); done < ".build/mac-manifests/$MAC_ARCH-assets.txt"
gh release upload "$TAG" "${files[@]}" --repo "$ADHOC_REPO" --clobber
# Why this is the last mutating step: publishing the draft is what makes the
# build visible to listReleaseBuilds. Doing it only after the manifest check
# means the picker can never offer a release whose assets are incomplete.
- name: Stage verified architecture manifest
uses: actions/upload-artifact@v7
with:
name: adhoc-mac-manifest-${{ matrix.arch }}
path: .build/mac-manifests/${{ matrix.arch }}.json
if-no-files-found: error
retention-days: 1
build-adhoc-mac:
needs: [resolve-ref, orcad-template-support, prepare-adhoc-mac-release, build-adhoc-mac-arch]
if: always() && needs.prepare-adhoc-mac-release.result == 'success'
environment: adhoc-mac-build
runs-on: blacksmith-2vcpu-ubuntu-2404
timeout-minutes: 35
permissions:
contents: read
actions: read
outputs:
tag: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
version: ${{ needs.orcad-template-support.outputs.version }}
head_sha: ${{ needs.resolve-ref.outputs.sha }}
published: ${{ steps.publish_live.outcome == 'success' && 'true' || 'false' }}
ships_orcad_template: ${{ needs.orcad-template-support.outputs.ships }}
steps:
- name: Mint adhoc repo token
id: app_token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.HOURLY_RELEASE_APP_ID }}
private-key: ${{ secrets.HOURLY_RELEASE_APP_PRIVATE_KEY }}
owner: stablyai
repositories: orca-adhoc
- name: Require both architecture jobs
env:
ARCH_RESULT: ${{ needs.build-adhoc-mac-arch.result }}
run: test "$ARCH_RESULT" = success
- name: Checkout manifest merger
uses: actions/checkout@v6
with:
ref: ${{ github.workflow_sha }}
sparse-checkout: config/scripts/adhoc-mac-manifests.mjs
sparse-checkout-cone-mode: false
persist-credentials: false
- name: Download architecture manifests
uses: actions/download-artifact@v8
with:
pattern: adhoc-mac-manifest-*
merge-multiple: true
path: .build/mac-manifests
- name: Merge and verify the updater manifest
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
TAG: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
MAC_VERSION: ${{ needs.orcad-template-support.outputs.version }}
run: |
set -euo pipefail
gh release view "$TAG" --repo "$ADHOC_REPO" --json assets > .build/mac-manifests/assets.json
node config/scripts/adhoc-mac-manifests.mjs merge "$MAC_VERSION" .build/mac-manifests
gh release upload "$TAG" .build/mac-manifests/latest-mac.yml --repo "$ADHOC_REPO" --clobber
- name: Confirm final template qualification before publication
if: needs.orcad-template-support.outputs.ships == 'true'
uses: actions/github-script@v8
env:
WAIT_FOR_TEMPLATE_COMPLETION: '1'
with:
script: |
const deadline = Date.now() + 30 * 60_000
while (Date.now() < deadline) {
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
...context.repo, run_id: context.runId, filter: 'latest', per_page: 100
})
const lanes = jobs.filter(job => job.name.startsWith('orcad-template / '))
const failed = lanes.find(job => job.status === 'completed' &&
['failure', 'cancelled', 'timed_out', 'action_required', 'startup_failure'].includes(job.conclusion))
if (failed) throw new Error(`Template qualification failed: ${failed.name}`)
const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts, {
...context.repo, run_id: context.runId, per_page: 100
})
const ready = artifacts.some(artifact => artifact.name === 'orcad-template' && !artifact.expired)
const completed = lanes.length > 0 && lanes.every(job => job.status === 'completed')
if (ready && (process.env.WAIT_FOR_TEMPLATE_COMPLETION !== '1' || completed)) return
await new Promise(resolve => setTimeout(resolve, 15_000))
}
throw new Error('Timed out waiting for the qualified orcad template')
# Publish only after manifest validation so the picker never offers incomplete assets.
- name: Publish the verified release
id: publish_live
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
TAG: ${{ steps.release.outputs.tag }}
NAME: ${{ steps.adhoc.outputs.name }}
TAG: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
NAME: ${{ needs.orcad-template-support.outputs.name }}
run: |
set -euo pipefail
# --title again: electron-builder resolves this draft by tag and may
# rewrite its title on upload. Re-asserting here means the name the
# picker reads is the one composed above, whatever it did in between.
# Reassert the picker title after electron-builder may rewrite the tag’s draft on upload.
gh release edit "$TAG" --repo "$ADHOC_REPO" --draft=false --prerelease --title "$NAME"
echo "Published $TAG as \"$NAME\""
# Why: a draft left behind by a failed publish is invisible to users but still
# holds its tag name. Gated on publish_live not having succeeded so a later
# failure (the prune step) cannot delete a release that already went live and
# that people may already be installing. Why cancelled() too: a run stopped
# from the Actions UI is not a failure(), so without it a manual cancel
# mid-publish would strand the draft.
# Delete stranded drafts on failure/cancel, but never after publish_live: later prune failure must not remove an installable release.
- name: Discard the draft release on failure
if: >-
(failure() || cancelled()) && steps.release.outputs.tag != '' &&
(failure() || cancelled()) && needs.prepare-adhoc-mac-release.outputs.tag != '' &&
steps.publish_live.outcome != 'success'
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
TAG: ${{ steps.release.outputs.tag }}
TAG: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
run: |
set -uo pipefail
# No --cleanup-tag: an unpublished draft never created a git tag.
@@ -547,32 +580,21 @@ jobs:
echo "::warning::Could not discard draft $TAG; remove it manually."
- name: Prune expired adhoc releases
# Only after a live publish: $TAG is then a non-draft this step must not
# delete, and a run that failed before publishing has nothing to retire.
# Retire only after live publication; preserve this non-draft tag and leave failed runs to draft cleanup.
if: steps.publish_live.outcome == 'success'
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
# Protect the tag this run just shipped, so no filter mistake can delete
# a build minutes after the person who cut it was told it exists.
TAG: ${{ steps.release.outputs.tag }}
# Exclude the just-published tag from pruning even if ordering/filter logic regresses.
TAG: ${{ needs.prepare-adhoc-mac-release.outputs.tag }}
run: |
set -euo pipefail
# Why compute the cutoff in bash rather than with jq's `now`: this runs
# once per dispatch, and a fixed epoch makes the threshold visible in the
# log when someone asks where their build went.
# Fix/log the cutoff epoch once per dispatch so retention decisions remain auditable.
cutoff=$(( $(date -u +%s) - ADHOC_RETAIN_DAYS * 86400 ))
echo "Pruning adhoc releases published before $(date -u -r "$cutoff" '+%Y-%m-%dT%H:%M:%SZ')"
# --cleanup-tag so pruning does not leave orphan tags with no release or
# assets attached. Drafts are excluded: a stale draft is the failure
# path's business, not the retention window's.
#
# Age comes from publishedAt, never createdAt. GitHub reports createdAt
# as the date of the *commit* a release's tag points at, and every tag
# here is cut from this repo's one seed commit — so all of them carry
# that same createdAt, and the day the window rolled past it the entire
# channel expired at once and a single run deleted it. A release with no
# publishedAt is kept rather than aged by guesswork.
echo "Pruning adhoc releases published before $(date -u -d "@$cutoff" '+%Y-%m-%dT%H:%M:%SZ')"
# --cleanup-tag removes release-less tags; failure cleanup owns drafts, which retention excludes.
# Use publishedAt: GitHub’s createdAt is the seed commit’s date for every tag. Aging by it once
# expired the entire channel in one run; retain releases with unknown publishedAt.
jq_filter='map(select(.isDraft | not))'
if [[ -n "${TAG:-}" ]]; then
jq_filter+=" | map(select(.tagName != \"${TAG//\"/\\\"}\"))"
@@ -597,27 +619,40 @@ jobs:
echo "::warning::Could not prune $tag"
done <<<"$stale"
# Why this runs after the mac leg rather than beside it: the tag and version
# are computed inside that job, so until it has run nothing else can name the
# release to upload into. The cost is small enough not to matter — the Windows
# leg measures ~7.5 min (install 2m45, build 35s, NSIS package 3m) against a
# ~9.5 min mac run, which keeps a adhoc run far inside its interval.
#
# Why `./` rather than a pinned `@main`: `uses:` resolves against the ref this
# file itself came from, which for an ordinary dispatch is main. Someone who
# deliberately points the Actions "Use workflow from" picker at a branch
# already gets that branch's copy of this entire file, so this follows the same
# rule instead of inventing a second one.
# Build unsigned Windows artifacts alongside Mac; publication waits below.
build-adhoc-win:
needs: build-adhoc-mac
# Only once the mac release is actually live: there is no release to upload
# into otherwise, and the Windows workflow refuses to create one.
if: needs.build-adhoc-mac.outputs.published == 'true'
needs: [resolve-ref, orcad-template-support, release-javascript]
if: >-
!cancelled() && github.repository == 'stablyai/orca' &&
needs.orcad-template-support.result == 'success' &&
needs.release-javascript.result == 'success'
permissions:
contents: read
actions: read
uses: ./.github/workflows/dev-channel-win-build.yml
secrets: inherit
with:
channel: adhoc
tag: ${{ needs.build-adhoc-mac.outputs.tag }}
ref: ${{ needs.build-adhoc-mac.outputs.head_sha }}
version: ${{ needs.build-adhoc-mac.outputs.version }}
orcad_template: ${{ needs.build-adhoc-mac.outputs.ships_orcad_template == 'true' }}
tag: v${{ needs.orcad-template-support.outputs.version }}
ref: ${{ needs.resolve-ref.outputs.sha }}
version: ${{ needs.orcad-template-support.outputs.version }}
orcad_template: ${{ needs.orcad-template-support.outputs.ships == 'true' }}
stage_only: true
fast_release: true
javascript_source_sha: ${{ needs.release-javascript.outputs.supported == 'true' && needs.release-javascript.outputs.source_sha || '' }}
# Prefetch the installer during notarization; the publisher still waits for Mac success.
publish-adhoc-win:
needs: [orcad-template-support, build-adhoc-win]
if: >-
!cancelled() && needs.orcad-template-support.result == 'success' &&
needs.build-adhoc-win.result == 'success'
permissions:
contents: read
actions: read
uses: ./.github/workflows/dev-channel-win-publish.yml
secrets: inherit
with:
channel: adhoc
tag: v${{ needs.orcad-template-support.outputs.version }}
wait_for_mac_job: build-adhoc-mac
@@ -1,13 +1,6 @@
name: Publish agent state rules
# The only publisher of agent state rules releases. Merging never publishes: a maintainer runs
# this from main, first to `next` (RC and dev builds), then promotes the identical file to
# `stable` after a soak. Every job runs on the dispatched commit, so the publish job runs exactly
# the script and rules the gate tested.
#
# Why GITHUB_TOKEN: release-policy.yml deletes an agent-state-rules-* release any other author
# publishes. Its releases start no workflow, so release-policy.yml sees them only when someone
# edits one by hand, and then accepts them as bot-authored prereleases.
# Maintainer-only publisher: dispatch main to next, soak, then promote identical bytes to stable. All jobs use the dispatched commit. Use GITHUB_TOKEN so release-policy accepts bot-authored prereleases; token-created releases do not trigger workflows until manual edits.
on:
workflow_dispatch:
inputs:
@@ -0,0 +1,62 @@
name: Sweep idle branch caches
# Branch-scoped caches serve only their branch, and nothing else removes them before LRU eviction
# reaches main's shared entries. PR merge refs belong to ci-closed-pr-caches.yml; tag refs to releases.
on:
schedule:
- cron: '23 */3 * * *'
workflow_dispatch:
inputs:
dry_run:
description: List what would be deleted without deleting it.
type: boolean
default: true
permissions:
actions: write
concurrency:
group: ci-branch-cache-sweep
cancel-in-progress: false
jobs:
sweep:
runs-on: ubuntu-24.04
timeout-minutes: 10
steps:
# No checkout: only this script runs, with the cache-write token.
- uses: actions/github-script@v8
env:
DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
with:
script: |
const keep = new Set(['refs/heads/main'])
const idleMs = 6 * 60 * 60 * 1000
const dryRun = process.env.DRY_RUN === 'true'
const now = Date.now()
const caches = await github.paginate(github.rest.actions.getActionsCacheList, {
...context.repo, per_page: 100
})
let count = 0
let bytes = 0
for (const cache of caches) {
if (typeof cache.ref !== 'string' || !cache.ref.startsWith('refs/heads/') ||
keep.has(cache.ref)) continue
const idle = now - Date.parse(cache.last_accessed_at)
// An unparseable access time compares false, so the cache is kept.
if (!(idle > idleMs)) continue
core.info(`${dryRun ? 'Would delete' : 'Deleting'} ${cache.ref} ${cache.key} ` +
`(${cache.size_in_bytes} B, idle ${Math.round(idle / 3600000)} h)`)
count += 1
bytes += cache.size_in_bytes ?? 0
if (dryRun) continue
try {
await github.rest.actions.deleteActionsCacheById({
...context.repo, cache_id: cache.id
})
} catch (error) {
if (error.status !== 404) throw error
}
}
core.info(`${dryRun ? 'Would remove' : 'Removed'} ${count} idle branch caches ` +
`(${(bytes / 1e9).toFixed(2)} GB)`)
@@ -15,7 +15,14 @@ jobs:
strategy:
fail-fast: false
matrix:
runner: [ubuntu-24.04-arm, blacksmith-4vcpu-ubuntu-2404]
# Separate cache stores per pool; native keys are image-scoped, and ci-cache-warmup.yml warms hosted 24.04 x64.
runner:
[
blacksmith-4vcpu-ubuntu-2404-arm,
blacksmith-8vcpu-ubuntu-2404,
ubuntu-24.04-arm,
ubuntu-22.04
]
runs-on: ${{ matrix.runner }}
timeout-minutes: 10
env:
@@ -28,8 +35,37 @@ jobs:
with:
native-runtime: node
node-version: '24'
cache-electron-package: 'true'
# Hosted consumers of these images restore neither the Electron archive nor the root store.
cache-electron-package: ${{ startsWith(matrix.runner, 'blacksmith-') }}
cache-pnpm-store: ${{ startsWith(matrix.runner, 'blacksmith-') }}
cache-pnpm-store-lookup-only: 'true'
- name: Populate shared Electron archive
if: startsWith(matrix.runner, 'blacksmith-')
run: node config/scripts/install-electron-package-binary.mjs
- name: Verify native cache is usable
run: node config/scripts/ensure-native-runtime.mjs --check-only
# node-server's hosted macOS lane needs nothing; this serves hosted macOS PR contracts and updater tests.
warm-macos-runtime:
if: github.event_name != 'pull_request'
concurrency:
group: ci-cache-warmup-macos-runtime-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: macos-15
timeout-minutes: 15
env:
ORCA_BACKGROUND_LAUNCH: '1'
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: node
node-version: '24'
# A main-scoped archive stops each macOS PR from saving its own copy.
cache-electron-package: 'true'
cache-pnpm-store: 'false'
- name: Populate shared Electron archive
run: node config/scripts/install-electron-package-binary.mjs
- name: Verify native cache is usable
@@ -42,7 +78,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [windows-2022, windows-11-arm]
os: [windows-2022, windows-11-arm, windows-2025]
runs-on: ${{ matrix.os }}
timeout-minutes: 20
env:
@@ -59,8 +95,14 @@ jobs:
- name: Verify native cache is usable
run: node config/scripts/ensure-native-runtime.mjs --check-only
- name: Resolve the Windows Rust build scope
id: rust-scope
if: runner.arch == 'X64' && matrix.os != 'windows-2025'
shell: bash
run: node config/scripts/windows-rust-build-scope.mjs >> "$GITHUB_OUTPUT"
- name: Cache Windows CLI launcher
if: runner.arch == 'X64'
if: runner.arch == 'X64' && matrix.os != 'windows-2025'
uses: actions/cache@v5
with:
path: |
@@ -69,16 +111,14 @@ jobs:
native/windows-agent-hook/.build
native/windows-agent-hook/target
~/.cargo/registry
key: windows-cli-launcher-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('native/windows-cli-launcher/**', 'native/windows-agent-hook/**', 'config/scripts/build-windows-cli-launcher.mjs', 'config/scripts/build-windows-agent-hook.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'resources/build/icon.ico', 'package.json') }}
key: windows-cli-launcher-v2-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-scope.outputs.toolchain }}-${{ steps.rust-scope.outputs.app-version }}-${{ hashFiles('native/windows-cli-launcher/src/**', 'native/windows-cli-launcher/build.rs', 'native/windows-cli-launcher/Cargo.toml', 'native/windows-cli-launcher/Cargo.lock', 'native/windows-cli-launcher/app.manifest', 'native/windows-cli-launcher/.cargo/**', 'native/windows-agent-hook/src/**', 'native/windows-agent-hook/build.rs', 'native/windows-agent-hook/Cargo.toml', 'native/windows-agent-hook/Cargo.lock', 'native/windows-agent-hook/app.manifest', 'native/windows-agent-hook/.cargo/**', 'config/scripts/build-windows-cli-launcher.mjs', 'config/scripts/build-windows-agent-hook.mjs', 'config/scripts/windows-rust-executable-build.mjs', 'config/scripts/windows-pe-imports.mjs', 'resources/build/icon.ico') }}
restore-keys: |
windows-cli-launcher-v2-${{ runner.os }}-${{ runner.arch }}-${{ steps.rust-scope.outputs.toolchain }}-
- name: Build the Windows native executables
if: runner.arch == 'X64'
if: runner.arch == 'X64' && matrix.os != 'windows-2025'
shell: pwsh
run: |
if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
rustup toolchain install stable --profile minimal
rustup default stable
}
node config/scripts/build-windows-agent-hook.mjs
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
node config/scripts/build-windows-cli-launcher.mjs
@@ -90,12 +130,151 @@ jobs:
with:
arches: arm64
# node-server, Windows E2E and the survival lanes restore these exact keys.
- name: Warm the Windows x64 server launcher
if: matrix.os == 'windows-2022'
uses: ./.github/actions/prepare-windows-cli-launcher
with:
arches: x64
- name: Warm the Windows agent hook
if: matrix.os == 'windows-2022'
uses: ./.github/actions/prepare-windows-agent-hook
- name: Warm the Windows server prebuild
id: orcad-prebuild
if: matrix.os != 'windows-2025'
uses: ./.github/actions/prepare-orcad-prebuilds
with:
resolve-windows-cache: 'true'
restore-windows-cache: 'true'
# Same check a consumer runs before trusting a restored slot.
- name: Validate the Windows server prebuild before saving
id: orcad-prebuild-validate
if: >-
github.event_name != 'pull_request' && matrix.os != 'windows-2025' &&
steps.orcad-prebuild.outputs.cache-hit != 'true' && steps.orcad-prebuild.outputs.cache-key != ''
run: node config/scripts/orcad-windows-prebuild-cache.mjs --validate
- name: Save the Windows server prebuild
if: steps.orcad-prebuild-validate.outcome == 'success'
continue-on-error: true
uses: actions/cache/save@v5
with:
path: ${{ steps.orcad-prebuild.outputs.cache-path }}
key: ${{ steps.orcad-prebuild.outputs.cache-key }}
# Windows packaging and E2E restore these; PRs no longer save their own copies.
warm-windows-electron:
concurrency:
group: ci-cache-warmup-windows-electron-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: windows-2022
timeout-minutes: 20
env:
ORCA_BACKGROUND_LAUNCH: '1'
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: electron
cache-pnpm-store: 'false'
# Hosted Linux packaging and packaged-browser E2E restore these; PRs no longer save their own copies.
warm-linux-electron:
concurrency:
group: ci-cache-warmup-linux-electron-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: ubuntu-24.04
timeout-minutes: 15
env:
ORCA_BACKGROUND_LAUNCH: '1'
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: electron
cache-pnpm-store: 'false'
- name: Resolve electron-builder download versions
id: eb-versions
if: github.event_name != 'pull_request'
shell: bash
run: node -p "const d = require('./package.json').devDependencies; 'versions=electron' + d.electron + '-builder' + d['electron-builder']" >> "$GITHUB_OUTPUT"
# pr.yml package restores this exact path and key.
- name: Seed shared Linux packaging downloads
id: eb-cache
if: github.event_name != 'pull_request'
uses: actions/cache@v5
with:
path: ~/.cache/electron-builder
key: electron-builder-linux-${{ steps.eb-versions.outputs.versions }}
lookup-only: true
# The same toolsets electron-builder fetches for AppImage, deb and rpm.
- name: Download Linux packaging tools
if: github.event_name != 'pull_request' && steps.eb-cache.outputs.cache-hit != 'true'
run: >-
node -e "const linux = require('app-builder-lib/out/toolsets/linux.js');
const { getIconsToolsetPath } = require('app-builder-lib/out/toolsets/icons.js');
const { Arch } = require('builder-util');
const { toolsets } = require('./config/electron-builder-pr-linux.config.cjs');
(async () => { await getIconsToolsetPath(); await linux.getFpmPath();
await linux.getAppImageTools(toolsets.appimage, Arch.x64) })().catch((error) => { console.error(error); process.exit(1) })"
# Mobile verify restores this bundle on pull requests without saving.
warm-mobile-bundle:
if: github.event_name != 'pull_request'
concurrency:
group: ci-cache-warmup-mobile-bundle-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: ubuntu-24.04-arm
timeout-minutes: 15
env:
BUNDLE_FROZEN: 'true'
defaults:
run:
working-directory: mobile
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ruby/setup-ruby@v1
with:
ruby-version: '3.3'
working-directory: mobile
- name: Resolve fastlane bundle cache
id: ruby-bundle
env:
GEMFILE_HASH: ${{ hashFiles('mobile/Gemfile', 'mobile/Gemfile.lock') }}
run: |
. /etc/os-release
printf 'key=mobile-bundle-v1-%s-%s-%s-%s\n' "$ID-$VERSION_ID" "$(ruby -e 'print RUBY_PLATFORM')" "$(ruby -e 'print RbConfig::CONFIG["ruby_version"]')" "$GEMFILE_HASH" >> "$GITHUB_OUTPUT"
- name: Seed fastlane bundle
id: ruby-bundle-cache
uses: actions/cache@v5
with:
path: mobile/vendor/bundle
key: ${{ steps.ruby-bundle.outputs.key }}
lookup-only: true
- name: Install fastlane
if: steps.ruby-bundle-cache.outputs.cache-hit != 'true'
run: |
bundle config set --local path "$PWD/vendor/bundle"
bundle config set --local deployment true
bundle install --jobs 4
warm-linux-package-fixtures:
if: github.event_name != 'pull_request'
concurrency:
group: ci-cache-warmup-linux-package-fixtures-${{ github.event.pull_request.number || github.ref }}
group: ci-cache-warmup-linux-package-fixtures-${{ matrix.runner }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
runner: [blacksmith-4vcpu-ubuntu-2404, ubuntu-24.04]
runs-on: ${{ matrix.runner }}
timeout-minutes: 10
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -11,6 +11,24 @@ on:
- '.github/actions/restore-pnpm-verification/**'
- '.github/actions/prepare-native-runtime/**'
- '.github/actions/prepare-windows-cli-launcher/**'
- '.github/actions/prepare-windows-agent-hook/**'
- 'config/scripts/windows-rust-build-scope.mjs'
- '.github/actions/prepare-orcad-prebuilds/**'
- 'config/scripts/native-dependency-lock-digest.mjs'
- 'config/scripts/orcad-windows-prebuild-cache.mjs'
- 'config/scripts/shared-native-build.mjs'
- 'config/scripts/shared-tree-publish.mjs'
- 'config/scripts/native-build-entry*.cjs'
- 'config/scripts/orcad-prebuild-node-gyp.mjs'
- 'config/scripts/msbuild-file-tracking.mjs'
- 'config/scripts/windows-pe-imports.mjs'
- 'config/scripts/build-orcad-prebuilds.mjs'
- 'config/scripts/orcad-prebuild-slot-contents.mjs'
- 'config/scripts/pinned-node-downloads.mjs'
- 'config/scripts/update-node-runtime-pin.mjs'
- 'config/scripts/build-workspace-packages.mjs'
- 'src/shared/node-runtime-pin.ts'
- 'src/packages/process-host/**'
- '.github/actions/prepare-git-compatibility/**'
- '.github/actions/prepare-linux-package-fixture/**'
- 'config/docker/headless-serve-shutdown/**'
@@ -20,7 +38,7 @@ on:
- 'pnpm-lock.yaml'
- 'pnpm-workspace.yaml'
- '.npmrc'
- '.pnpmfile.cjs'
- 'config/pnpmfile.cjs'
- 'config/scripts/ensure-native-runtime.mjs'
- 'config/scripts/initialize-windows-native-toolchain.mjs'
- 'config/scripts/capture-windows-native-environment.cmd'
@@ -44,6 +62,8 @@ on:
- 'config/scripts/build-windows-agent-hook.mjs'
- 'config/scripts/windows-rust-executable-build.mjs'
- 'resources/build/icon.ico'
- 'mobile/Gemfile'
- 'mobile/Gemfile.lock'
pull_request:
types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
paths:
+10 -3
View File
@@ -29,7 +29,7 @@ on:
- 'pnpm-lock.yaml'
- 'pnpm-workspace.yaml'
- '.npmrc'
- '.pnpmfile.cjs'
- 'config/pnpmfile.cjs'
- 'config/scripts/ensure-native-runtime.mjs'
- 'config/scripts/initialize-windows-native-toolchain.mjs'
- 'config/scripts/capture-windows-native-environment.cmd'
@@ -66,9 +66,14 @@ jobs:
warm:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
concurrency:
group: ci-cache-warmup-linux-x64-${{ github.event.pull_request.number || github.ref }}
group: ci-cache-warmup-linux-x64-${{ matrix.runner }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
runs-on: ubuntu-latest
# Blacksmith and GitHub-hosted jobs restore from separate cache stores; warm both.
strategy:
fail-fast: false
matrix:
runner: [blacksmith-4vcpu-ubuntu-2404, ubuntu-24.04]
runs-on: ${{ matrix.runner }}
timeout-minutes: 10
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -83,6 +88,8 @@ jobs:
native-runtime: node
node-version: '24'
cache-electron-package: 'true'
# No hosted job restores the root store, so keep it out of GitHub's cache quota.
cache-pnpm-store: ${{ startsWith(matrix.runner, 'blacksmith-') }}
cache-pnpm-store-lookup-only: 'true'
- uses: ./.github/actions/prepare-headless-compiler
+3 -3
View File
@@ -10,7 +10,7 @@ permissions:
jobs:
clean:
runs-on: blacksmith-2vcpu-ubuntu-2204
runs-on: ubuntu-22.04
timeout-minutes: 5
steps:
- name: Cancel obsolete PR checks
@@ -22,11 +22,13 @@ jobs:
const superseded = context.payload.action === 'synchronize'
const targets = [
['pr-checks', 'pr.yml'],
['dependency-security', 'dependency-security.yml'],
['node-server', 'node-server-tests.yml'],
['ssh-windows-hosts', 'ssh-windows-hosts.yml'],
['ssh-hostile-hosts', 'ssh-hostile-hosts.yml'],
['mobile', 'mobile.yml'],
['computer-e2e', 'computer-e2e.yml'],
['computer-macos-native', 'computer-macos-native.yml'],
['wsl-orcad-e2e', 'wsl-orcad-e2e.yml'],
['performance-contracts', 'performance-contracts.yml'],
['pr-test-loc', 'pr-test-loc.yml'],
@@ -39,7 +41,6 @@ jobs:
['release-ref-validation', 'release-ref-validation.yml'],
['docs-pr', 'docs.yml'],
['cloud-verify-refs/pull', 'cloud-verify.yml', '/merge'],
['pnpm-verification-pilot', 'ci-pnpm-verification-pilot.yml'],
['win-orcad-serve-switch-e2e', 'win-orcad-serve-switch-e2e.yml'],
['pi-provider-runtime', 'pi-provider-runtime.yml'],
['coding-plan-contract', 'zcode-credential-contract.yml', '-credentials'],
@@ -47,7 +48,6 @@ jobs:
['ci-cache-warmup-platforms', 'ci-cache-warmup-platforms.yml'],
['pi-owner-runtime', 'pi-owner-runtime.yml'],
['git-command-termination', 'git-command-termination-runtime.yml'],
['skill-roundtrip-pull_request', 'skill-update-roundtrip.yml'],
['ci-cache-warmup', 'ci-cache-warmup.yml']
]
const stillCurrent = async () => {
@@ -1,67 +0,0 @@
name: pnpm verification cache pilot
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
paths: ['.github/workflows/ci-pnpm-verification-pilot.yml']
workflow_dispatch:
permissions:
contents: read
concurrency:
group: pnpm-verification-pilot-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
ORCA_BACKGROUND_LAUNCH: '1'
jobs:
seed:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
strategy:
matrix:
runner: [ubuntu-latest, ubuntu-24.04-arm]
runs-on: ${{ matrix.runner }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
id: deps
# PR caches are scoped to the PR merge ref; main never restores them.
- uses: actions/cache/save@v5
if: steps.deps.outputs.verification-cache-hit != 'true'
with:
path: ${{ steps.deps.outputs.verification-cache-path }}
key: ${{ steps.deps.outputs.verification-cache-key }}
measure:
needs: seed
name: measure ${{ matrix.runner }} sample ${{ matrix.sample }} cache ${{ matrix.cache }}
strategy:
fail-fast: false
max-parallel: 4
matrix:
runner: [ubuntu-latest, ubuntu-24.04-arm]
sample: [1, 2, 3]
cache: ['false', 'true']
runs-on: ${{ matrix.runner }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
id: deps
with:
cache-pnpm-verification: ${{ matrix.cache }}
- name: Validate treatment and frozen install
env:
CACHE_ENABLED: ${{ matrix.cache }}
CACHE_HIT: ${{ steps.deps.outputs.verification-cache-hit }}
run: |
if [ "$CACHE_ENABLED" = true ]; then test "$CACHE_HIT" = true; fi
git diff --exit-code -- package.json pnpm-lock.yaml pnpm-workspace.yaml
node -e "require.resolve('vitest'); require.resolve('electron')"
+1 -1
View File
@@ -15,7 +15,7 @@ concurrency:
jobs:
report:
runs-on: ubuntu-slim
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
+10 -1
View File
@@ -23,7 +23,15 @@ env:
jobs:
seed:
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04-arm
# Blacksmith and GitHub keep separate cache stores; seed the same key in both for either consumer pool.
strategy:
fail-fast: false
matrix:
runner: [blacksmith-4vcpu-ubuntu-2404-arm, ubuntu-24.04-arm]
concurrency:
group: xterm-cache-seed-${{ github.ref }}-${{ matrix.runner }}
cancel-in-progress: true
runs-on: ${{ matrix.runner }}
timeout-minutes: 10
steps:
- uses: actions/checkout@v6
@@ -32,6 +40,7 @@ jobs:
- uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
package-manager-cache: false
- uses: ./.github/actions/prepare-xterm-dependencies
id: cache
@@ -1,677 +0,0 @@
name: Bootstrap Relay Staging Capacity
on:
workflow_dispatch:
inputs:
confirmation:
description: Enter BOOTSTRAP_STAGING_CAPACITY
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
bootstrap:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: us-central1
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
LEGACY_C3_IMAGE_DIGEST: sha256:2d0f6e6db2b0eb9d6aba188698de8330f8c30b4e76badfcf0fac3f3eb9508a87
steps:
- uses: actions/checkout@v4
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: capacity-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: access_token
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit bootstrap confirmation
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "BOOTSTRAP_STAGING_CAPACITY"
- name: Read and verify reviewed 600/60 topology
shell: bash
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
> "${RUNNER_TEMP}/relay-gce-state.json"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
DESIRED_IMAGES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.image })' \
| jq -r '.')"
DESIRED_ZONES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode({ for cell_id, cell in var.relay_gce_cells : cell_id => cell.zone })' \
| jq -r '.')"
DESIRED_TARGET_SIZES_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(local.relay_gce_cell_target_sizes)' | jq -r '.')"
jq -e \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson target_sizes "${DESIRED_TARGET_SIZES_JSON}" \
'([.[] | select(.id == "staging-gce-c2")] | length == 1) and
([.[] | select(.id == "staging-gce-c3")] | length == 1) and
(any(.[]; .id == "staging-gce-c2" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
(any(.[]; .id == "staging-gce-c3" and .connectionHardCap == 600 and .connectionUnobservedBound == 60)) and
($images["staging-gce-c2"] == $images["staging-gce-c3"]) and
($target_sizes["staging-gce-c2"] == 1) and
($target_sizes["staging-gce-c3"] == 1)' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
jq \
--argjson desired "${DESIRED_CELLS_JSON}" \
--argjson images "${DESIRED_IMAGES_JSON}" \
--argjson zones "${DESIRED_ZONES_JSON}" \
'($desired | map({key: .id, value: .}) | from_entries) as $cells |
to_entries | map(. as $entry | {
key: $entry.key,
value: ($entry.value + {
origin: $cells[$entry.key].url,
image: $images[$entry.key],
zone: $zones[$entry.key],
connection_hard_cap: $cells[$entry.key].connectionHardCap,
connection_unobserved_bound: $cells[$entry.key].connectionUnobservedBound
})
}) | from_entries' \
"${RUNNER_TEMP}/relay-gce-state.json" \
> "${RUNNER_TEMP}/relay-gce-topology.json"
ACTIVE_REVISION="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
DESIRED_IMAGE="$(jq -r '.["staging-gce-c2"]' <<< "${DESIRED_IMAGES_JSON}")"
gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region us-central1 \
--format=json \
| jq -e \
--arg image "${DESIRED_IMAGE}" \
--arg capacity_service_account "${CAPACITY_SERVICE_ACCOUNT}" \
'(.spec.containers[0].image == $image) and
any(.spec.containers[0].env[]?;
.name == "ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT" and
.value == $capacity_service_account)' >/dev/null
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
jq -e --argjson desired "${DESIRED_CELLS_JSON}" '
def without_bootstrap_capacity:
map(if .id == "staging-gce-c2" or .id == "staging-gce-c3"
then del(.connectionHardCap, .connectionUnobservedBound)
else . end);
without_bootstrap_capacity == ($desired | without_bootstrap_capacity)
' <<< "${CURRENT_CELLS_JSON}" >/dev/null
- name: Bootstrap C2 then C3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
topology="${RUNNER_TEMP}/relay-gce-topology.json"
fixed_one_instance_name() {
local cell_id="$1"
local mig_name zone
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instance-groups managed list-instances \
"${mig_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("legacy cell does not have one stable running instance") end'
}
fixed_one_instance_id() {
local cell_id="$1"
local instance_name="$2"
local zone
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
gcloud compute instances describe "${instance_name}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--format='value(id)'
}
write_legacy_metrics() {
local cell_id="$1"
local instance_id="$2"
local after="$3"
local output="$4"
gcloud logging read \
"resource.type=\"gce_instance\" AND
resource.labels.instance_id=\"${instance_id}\" AND
jsonPayload.event=\"orca_relay_runtime_metrics\" AND
jsonPayload.cellId=\"${cell_id}\" AND
timestamp>=\"${after}\"" \
--project "${GCP_PROJECT_ID}" \
--limit 10 \
--order desc \
--format json \
| jq '[.[] | {
timestamp,
cellId: .jsonPayload.cellId,
metricVersion: .jsonPayload.metricVersion,
totalConnections: .jsonPayload.totalConnections,
preAuthConnections: .jsonPayload.preAuthConnections,
controls: .jsonPayload.controls,
splices: .jsonPayload.splices,
pendingSplices: .jsonPayload.pendingSplices,
queuedBytes: .jsonPayload.queuedBytes
}]' > "${output}"
}
verify_legacy_cell() {
local cell_id="$1"
local admission="$2"
local after="$3"
local expected_instance_id="$4"
local runtime_started_after="${5:-}"
local previous_incarnation_digest="${6:-}"
local hard_cap="${7:-}"
local unobserved_bound="${8:-}"
local capacity_state="${9:-}"
local current_instance current_instance_id metrics origin result
local runtime_start_args=() incarnation_args=() capacity_args=()
current_instance="$(fixed_one_instance_name "${cell_id}")"
current_instance_id="$(fixed_one_instance_id "${cell_id}" "${current_instance}")"
test "${current_instance_id}" = "${expected_instance_id}"
metrics="${RUNNER_TEMP}/${cell_id}-legacy-runtime-metrics.json"
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
if test -n "${runtime_started_after}"; then
runtime_start_args=(--runtime-started-after "${runtime_started_after}")
fi
if test -n "${previous_incarnation_digest}"; then
incarnation_args=(--previous-incarnation-digest "${previous_incarnation_digest}")
fi
if test -n "${hard_cap}"; then
capacity_args=(--hard-cap "${hard_cap}" --unobserved-bound "${unobserved_bound}")
fi
if test -n "${capacity_state}"; then
capacity_args+=(--capacity-state "${capacity_state}")
fi
for _attempt in $(seq 1 18); do
write_legacy_metrics \
"${cell_id}" "${current_instance_id}" "${after}" "${metrics}"
if result="$(node dev/scripts/verify-relay-legacy-bootstrap.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--admission "${admission}" \
--expected-image-digest "${LEGACY_C3_IMAGE_DIGEST}" \
--metrics-after "${after}" \
--metrics-file "${metrics}" \
"${runtime_start_args[@]}" \
"${incarnation_args[@]}" \
"${capacity_args[@]}")"; then
echo "${result}"
return 0
fi
sleep 10
done
return 1
}
post_admin() {
local origin="$1"
local path="$2"
local body="$3"
curl --fail --silent --show-error \
--request POST \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${body}" \
"${origin}${path}"
}
runtime_kind() {
local cell_id="$1"
local origin desired_digest digest
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
desired_digest="$(jq -r --arg cell "${cell_id}" \
'.[$cell].image | split("@") | last' "${topology}")"
digest="$(post_admin "${origin}" /v1/admin/runtime-status '{"v":1}' \
| jq -er '.imageDigest')"
if test "${digest}" = "${LEGACY_C3_IMAGE_DIGEST}"; then
echo legacy
elif test "${digest}" = "${desired_digest}"; then
echo modern
else
echo 'bootstrap cell image is neither legacy nor reviewed' >&2
return 1
fi
}
cell_admission() {
local cell_id="$1"
post_admin "${DIRECTOR_ORIGIN}" /v1/admin/cell-status \
"$(jq -cn --arg cell "${cell_id}" '{v:1, cellId:$cell}')" \
| jq -er '.status.admissionState |
if . == "general" or . == "migration-only" then .
else error("bootstrap admission is not recoverable") end'
}
verify_modern_cell() {
local cell_id="$1"
local admission="$2"
local origin
origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission "${admission}" \
--draining forbidden \
--activity allowed
}
ensure_modern_general() {
local cell_id="$1"
local admission="$2"
verify_modern_cell "${cell_id}" "${admission}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids "${cell_id}"
verify_modern_cell "${cell_id}" general
}
prepare_legacy_c3_fallback() {
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c3.relay-staging.onorca.dev
legacy_c3_restart_started_after=
legacy_c3_old_incarnation=
}
normalize_legacy_c3() {
legacy_pre_boundary="$(node -e \
'process.stdout.write(new Date(Date.now() - 120_000).toISOString())')"
legacy_c2_instance="$(fixed_one_instance_name staging-gce-c2)"
legacy_c2_instance_id="$(fixed_one_instance_id \
staging-gce-c2 "${legacy_c2_instance}")"
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/probe-relay-legacy-admission.mjs \
--cell-origin https://c2.relay-staging.onorca.dev
legacy_c3_isolated=false
restore_legacy_c3_fallback() {
if test "${legacy_c3_isolated}" = true; then
verify_legacy_cell \
staging-gce-c2 general "${legacy_pre_boundary}" "${legacy_c2_instance_id}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore-fallback \
--general-cell-ids staging-gce-c2
fi
}
trap restore_legacy_c3_fallback EXIT
legacy_c3_isolated=true
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin https://c3.relay-staging.onorca.dev \
--cell-id staging-gce-c3 \
--mode isolate
legacy_c3_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_drained="$(verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_drain_boundary}" "${legacy_c3_instance_id}")"
echo "${legacy_c3_drained}"
legacy_c3_old_incarnation="$(jq -er '.incarnationDigest' \
<<< "${legacy_c3_drained}")"
legacy_c3_restart_started_after="$(node -e \
'process.stdout.write(new Date().toISOString())')"
gcloud compute instance-groups managed recreate-instances \
orca-cloud-staging-relay-gce-c3 \
--instances "${legacy_c3_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--quiet
gcloud compute instance-groups managed wait-until \
orca-cloud-staging-relay-gce-c3 \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone us-central1-a \
--timeout 900
legacy_c3_instance="$(fixed_one_instance_name staging-gce-c3)"
legacy_c3_instance_id="$(fixed_one_instance_id \
staging-gce-c3 "${legacy_c3_instance}")"
legacy_c3_metrics_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
verify_legacy_cell \
staging-gce-c3 migration-only \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id staging-gce-c3 \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
verify_legacy_cell \
staging-gce-c3 general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
legacy_c3_isolated=false
trap - EXIT
}
roll_cell() (
local cell_id="$1"
local fallback_cell_id="$2"
local target_kind="$3"
local fallback_kind="$4"
local active_revision cell_origin current_cells_json desired_bound desired_cap
local desired_cells_json director_result image mig_name plan
local fallback_origin plan_changes plan_result restored target_drain_boundary
local target_instance target_instance_id zone
cell_origin="$(jq -r --arg cell "${cell_id}" '.[$cell].origin' "${topology}")"
fallback_origin="$(jq -r --arg cell "${fallback_cell_id}" \
'.[$cell].origin' "${topology}")"
zone="$(jq -r --arg cell "${cell_id}" '.[$cell].zone' "${topology}")"
mig_name="$(jq -r --arg cell "${cell_id}" '.[$cell].mig_name' "${topology}")"
image="$(jq -r --arg cell "${cell_id}" '.[$cell].image' "${topology}")"
desired_cap="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_hard_cap' "${topology}")"
desired_bound="$(jq -r --arg cell "${cell_id}" \
'.[$cell].connection_unobserved_bound' "${topology}")"
plan="${RUNNER_TEMP}/${cell_id}-capacity-bootstrap.tfplan"
restored=false
restore_fallback() {
if test "${restored}" = false; then
verify_fallback
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore-fallback \
--general-cell-ids "${fallback_cell_id}"
fi
}
verify_fallback() {
if test "${fallback_kind}" = legacy; then
verify_legacy_cell \
"${fallback_cell_id}" general \
"${legacy_c3_metrics_boundary}" "${legacy_c3_instance_id}" \
"${legacy_c3_restart_started_after}" "${legacy_c3_old_incarnation}"
return
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${fallback_origin}" \
--cell-id "${fallback_cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
}
verify_fallback
trap restore_fallback EXIT
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--mode isolate
target_drain_boundary="$(node -e \
'process.stdout.write(new Date().toISOString())')"
if test "${target_kind}" = legacy; then
target_instance="$(fixed_one_instance_name "${cell_id}")"
target_instance_id="$(fixed_one_instance_id "${cell_id}" "${target_instance}")"
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}" absent-or-stale
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
active_revision="$(gcloud run services describe orca-cloud-relay-staging \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${active_revision}"
current_cells_json="$(gcloud run revisions describe "${active_revision}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
desired_cells_json="$(jq -ce \
--arg cell "${cell_id}" \
--argjson cap "${desired_cap}" \
--argjson bound "${desired_bound}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${current_cells_json}")"
director_result="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service orca-cloud-relay-staging \
--image "${image}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${cell_id}" \
--director-cells-json "${desired_cells_json}" \
--prune-revisions true \
--release-id "bootstrap-${cell_id}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}")"
echo "${director_result}"
if test "${target_kind}" = legacy; then
verify_legacy_cell \
"${cell_id}" migration-only \
"${target_drain_boundary}" "${target_instance_id}" \
'' '' "${desired_cap}" "${desired_bound}"
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap "${desired_cap}" \
--unobserved-bound "${desired_bound}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity quiescent
fi
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
-out="${plan}"
plan_result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--image "${image}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${plan_result}"
plan_changes="$(jq -r '.changes' <<< "${plan_result}")"
[[ "${plan_changes}" =~ ^(0|2)$ ]]
if test "${plan_changes}" = 2; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
else
target_instance="$(fixed_one_instance_name "${cell_id}")"
gcloud compute instance-groups managed recreate-instances "${mig_name}" \
--instances "${target_instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--quiet
fi
gcloud compute instance-groups managed wait-until "${mig_name}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${zone}" \
--timeout 900
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${cell_origin}" \
--cell-id "${cell_id}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${cell_id}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
restored=true
trap - EXIT
)
c2_kind="$(runtime_kind staging-gce-c2)"
c3_kind="$(runtime_kind staging-gce-c3)"
c2_admission="$(cell_admission staging-gce-c2)"
c3_admission="$(cell_admission staging-gce-c3)"
bootstrap_phase="$(node dev/scripts/classify-relay-staging-bootstrap.mjs \
--c2-kind "${c2_kind}" \
--c2-admission "${c2_admission}" \
--c3-kind "${c3_kind}" \
--c3-admission "${c3_admission}")"
jq -cn --arg phase "${bootstrap_phase}" \
'{event:"relay_staging_bootstrap_phase", phase:$phase}'
case "${bootstrap_phase}" in
normalize-and-roll-both)
normalize_legacy_c3
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
resume-c2-then-c3)
prepare_legacy_c3_fallback
roll_cell staging-gce-c2 staging-gce-c3 legacy legacy
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
roll-c2)
ensure_modern_general staging-gce-c3 "${c3_admission}"
roll_cell staging-gce-c2 staging-gce-c3 legacy modern
;;
roll-c3)
ensure_modern_general staging-gce-c2 "${c2_admission}"
roll_cell staging-gce-c3 staging-gce-c2 legacy modern
;;
complete)
ensure_modern_general staging-gce-c2 "${c2_admission}"
ensure_modern_general staging-gce-c3 "${c3_admission}"
;;
*)
echo 'unsupported staging bootstrap phase' >&2
exit 1
;;
esac
- name: Verify both bootstrapped cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
for number in 2 3; do
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://c${number}.relay-staging.onorca.dev" \
--cell-id "staging-gce-c${number}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
done
@@ -1,288 +0,0 @@
name: Deploy Relay Asia Topology
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Validate a saved plan or apply that exact plan
required: true
default: plan
type: choice
options: [plan, apply]
cell-ids:
description: Exact reviewed comma-separated cell set (Asia, or the US cells at the Asia shape)
required: true
type: string
image:
description: Full environment Relay image pinned by sha256 digest
required: true
type: string
confirmation:
description: Enter APPLY_RELAY_ASIA_TOPOLOGY for apply mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
topology:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
TARGET_IMAGE: ${{ inputs.image }}
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
CLOUD_SQL_INSTANCE: ${{ inputs.environment == 'production' && 'orca-cloud-auth-db' || 'orca-cloud-staging-auth-db' }}
VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER: db-custom-4-15360
VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION: POSTGRES_17
# SHOW max_connections on the live instance, 2026-09-16; no flag is set.
VERIFIED_DEFAULT_MAX_CONNECTIONS: '500'
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TF_VARS: ${{ inputs.environment == 'production' && 'environments/production.tfvars' || 'environments/staging.tfvars' }}
TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
TOPOLOGY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_ASIA_TOPOLOGY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate the reviewed request before authentication
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
test -n "${TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${TOPOLOGY_SERVICE_ACCOUNT}"
target_region=asia-east2
case "${TARGET_ENVIRONMENT}:${TARGET_CELL_IDS}" in
staging:staging-gce-c4) ;;
production:production-gce-c27,production-gce-c28,production-gce-c29) ;;
production:production-gce-c30) ;;
production:production-gce-c31) ;;
production:production-gce-c32,production-gce-c33) target_region=us-central1 ;;
production:production-gce-c34) ;;
production:production-gce-c35) ;;
*) echo "cell-ids do not match the reviewed environment topology" >&2; exit 1 ;;
esac
echo "TARGET_REGION=${target_region}" >> "${GITHUB_ENV}"
[[ "${TARGET_IMAGE}" =~ ^us-central1-docker\.pkg\.dev/${GCP_PROJECT_ID}/orca-cloud/relay@sha256:[0-9a-f]{64}$ ]]
if test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = APPLY_RELAY_ASIA_TOPOLOGY
else
test "${DEPLOY_MODE}" = plan
test -z "${CONFIRMATION}"
fi
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.TOPOLOGY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.TOPOLOGY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
- name: Require the checked Cloud SQL connection budget
shell: bash
run: |
set -euo pipefail
budget="$(node dev/scripts/relay-cloud-sql-connection-budget.mjs)"
checked_max="$(jq -er '.maxConnections' <<< "${budget}")"
jq -e '.withinBudget == true' <<< "${budget}" >/dev/null
if test "${TARGET_ENVIRONMENT}" = production; then
instance="$(gcloud sql instances describe "${CLOUD_SQL_INSTANCE}" \
--project "${GCP_PROJECT_ID}" --format=json)"
live_flag="$(jq -er '[.settings.databaseFlags[]? |
select(.name == "max_connections") | .value] |
if length <= 1 then (.[0] // "") else error("duplicate max_connections flags") end' \
<<< "${instance}")"
if test -n "${live_flag}"; then
live_max="${live_flag}"
live_source=explicit-flag
else
# No flag: the ceiling is the tier default measured for this exact
# shape; fail closed if the shape changes.
test "$(jq -er '.settings.tier' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_TIER}"
test "$(jq -er '.databaseVersion' <<< "${instance}")" = \
"${VERIFIED_DEFAULT_MAX_CONNECTIONS_DATABASE_VERSION}"
live_max="${VERIFIED_DEFAULT_MAX_CONNECTIONS}"
live_source=verified-shape-default
fi
test "${live_max}" = "${checked_max}"
else
live_max="not-read-for-staging"
live_source=not-read-for-staging
fi
{
echo "### Relay Cloud SQL connection budget"
echo "- Checked maximum: ${checked_max}"
echo "- Configured maximum: $(jq -er '.configuredMaximum' <<< "${budget}")"
echo "- Rollout operating maximum: $(jq -er '.operatingMaximum' <<< "${budget}")"
echo "- Explicit reserve: $(jq -er '.explicitReserve' <<< "${budget}")"
echo "- Production live max_connections: ${live_max}"
echo "- Production live maximum source: ${live_source}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Initialize the exact environment state
run: terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
- id: targets
name: Build the exact additive target set
shell: bash
run: |
set -euo pipefail
file="${RUNNER_TEMP}/relay-asia-targets"
: > "${file}"
# The root region's network predates every cell; only asia-east2's is additive.
if test "${TARGET_REGION}" = asia-east2; then
printf '%s\n' \
'-target=google_compute_subnetwork.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router.relay_gce_additional["asia-east2"]' \
'-target=google_compute_router_nat.relay_gce_additional["asia-east2"]' >> "${file}"
fi
printf '%s\n' '-target=google_compute_url_map.relay_gce[0]' >> "${file}"
IFS=, read -ra cells <<< "${TARGET_CELL_IDS}"
for cell_id in "${cells[@]}"; do
printf '%s\n' \
"-target=google_compute_instance_template.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${cell_id}\"]" \
"-target=google_compute_backend_service.relay_gce_cell[\"${cell_id}\"]" >> "${file}"
done
echo "file=${file}" >> "${GITHUB_OUTPUT}"
- id: live-images
name: Plan every non-target cell at the image it serves
shell: bash
run: |
set -euo pipefail
# The URL map target pulls every cell's template into the plan, and same-cap rolls
# leave committed images behind the served ones; read only templates out of state.
cells="${RUNNER_TEMP}/relay-committed-cells.json"
live="${RUNNER_TEMP}/relay-live-cell-templates.json"
overlay="${RUNNER_TEMP}/relay-asia-live-images.tfvars.json"
committed_plan="${RUNNER_TEMP}/relay-committed-cells.tfplan"
mapfile -t targets < "${{ steps.targets.outputs.file }}"
# Not console: it evaluates every output against state, where a new cell has no MIG yet.
terraform -chdir=infra/terraform plan -input=false -refresh=false -lock=false \
-var-file="${TF_VARS}" "${targets[@]}" -out="${committed_plan}" > /dev/null
terraform -chdir=infra/terraform show -json "${committed_plan}" \
| jq -ce '.variables.relay_gce_cells.value | objects' > "${cells}"
terraform -chdir=infra/terraform show -json | jq -ce '[
.values.root_module.resources[]?
| select(.mode == "managed" and
.type == "google_compute_instance_template" and .name == "relay_gce_cell")
| { index, metadata_startup_script: .values.metadata_startup_script }
]' > "${live}"
summary="$(node dev/scripts/relay-live-cell-image-overlay.mjs \
--cells-json "${cells}" --live-templates-json "${live}" \
--cell-ids "${TARGET_CELL_IDS}" --output "${overlay}")"
echo "file=${overlay}" >> "${GITHUB_OUTPUT}"
{
echo "### Live images held for non-target cells"
echo "- Cells whose committed image differs from the served one: $(jq -r '.drifted | join(", ")' <<< "${summary}")"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Create and validate the saved topology plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-asia-topology.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology.json"
mapfile -t targets < "${{ steps.targets.outputs.file }}"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" -var-file="${{ steps.live-images.outputs.file }}" \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
committed="${RUNNER_TEMP}/relay-committed-asia-topology.json"
jq -e '{
relay_gce_cells: .variables.relay_gce_cells.value,
relay_gce_additional_region_subnetwork_cidrs:
.variables.relay_gce_additional_region_subnetwork_cidrs.value
}' "${plan_json}" > "${committed}"
node dev/scripts/prepare-relay-asia-topology-input.mjs \
--existing-json "${committed}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image "${TARGET_IMAGE}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
changes="$(jq -er '.changes' <<< "${result}")"
digest="$(sha256sum "${plan}" | awk '{print $1}')"
echo "plan=${plan}" >> "${GITHUB_OUTPUT}"
echo "changes=${changes}" >> "${GITHUB_OUTPUT}"
{
echo "### Relay Asia topology saved plan"
echo "- Environment: ${TARGET_ENVIRONMENT}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Region: ${TARGET_REGION}"
echo "- Mutating resources: ${changes}"
echo "- Saved-plan SHA-256: ${digest}"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Apply only the validated saved plan
if: ${{ inputs.mode == 'apply' }}
run: terraform -chdir=infra/terraform apply -input=false -auto-approve "${{ steps.plan.outputs.plan }}"
- name: Prove the exact topology targets converged
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
set -euo pipefail
mapfile -t targets < "${{ steps.targets.outputs.file }}"
plan="${RUNNER_TEMP}/relay-asia-topology-readback.tfplan"
plan_json="${RUNNER_TEMP}/relay-asia-topology-readback.json"
terraform -chdir=infra/terraform plan -input=false -lock-timeout=30s \
-var-file="${TF_VARS}" -var-file="${{ steps.live-images.outputs.file }}" \
"${targets[@]}" -out="${plan}"
terraform -chdir=infra/terraform show -json "${plan}" > "${plan_json}"
result="$(node dev/scripts/validate-relay-asia-topology-plan.mjs \
--plan-json "${plan_json}" \
--environment "${TARGET_ENVIRONMENT}" \
--cell-ids "${TARGET_CELL_IDS}" \
--region "${TARGET_REGION}" \
--image "${TARGET_IMAGE}")"
test "$(jq -er '.changes' <<< "${result}")" = 0
- name: Record the required selector-safe next step
if: ${{ inputs.mode == 'apply' }}
run: |
{
echo "### Required next step"
echo "The VMs are not eligible for ordinary placement yet."
echo "Register the exact new cells atomically as migration-only before any director configuration lists them."
echo "Rollback is migration-only admission; do not destroy the Asia network on rollout day."
} >> "${GITHUB_STEP_SUMMARY}"
@@ -1,92 +0,0 @@
name: Deploy Relay Fence Broker
on:
workflow_dispatch:
inputs:
image-digest:
description: Immutable broker image digest built from this main commit
required: true
type: string
permissions:
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
SERVICE_NAME: orca-cloud-relay-fence
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay-fence-broker
IMAGE_DIGEST: ${{ inputs.image-digest }}
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Resolve exact-commit broker image
run: |
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
TAGS="$(gcloud artifacts docker tags list "${IMAGE_REPOSITORY}" \
--filter="version:${IMAGE_DIGEST}" \
--format=json)"
jq -e --arg tag "/tags/sha-${GITHUB_SHA}" \
'any(.[]; .tag | endswith($tag))' <<< "${TAGS}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
- name: Deploy broker image only
run: |
gcloud run services update "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--image "${IMAGE}" \
--quiet
- name: Verify ready singleton revision
run: |
SERVICE="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
jq -e '.status.conditions[] | select(.type == "Ready" and .status == "True")' \
<<< "${SERVICE}"
REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) == 100)] |
if length == 1 then .[0].revisionName // empty else empty end' \
<<< "${SERVICE}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
@@ -1,824 +0,0 @@
name: Deploy Relay Production Capacity Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
target-cell-id:
required: true
type: string
confirmation:
required: true
type: string
monitor-run-id:
required: true
type: string
monitor-run-attempt:
required: true
type: string
evidence-mode:
required: true
type: string
wave-cell-ids:
required: true
type: string
wave-index:
required: true
type: string
source-wave-run-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 75
environment: production
env:
# These drains are unpaced, but restart-safe still waits out the production pace window
# from the drain, the same one the same-cap drain uses.
RESTART_SAFE_PACE_WINDOW_MS: '300000'
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
CAPACITY_CELL_IDS: production-gce-c7,production-gce-c8,production-gce-c9,production-gce-c10,production-gce-c13,production-gce-c14,production-gce-c15,production-gce-c16,production-gce-c19,production-gce-c20,production-gce-c21,production-gce-c22,production-gce-c23,production-gce-c24,production-gce-c25,production-gce-c26
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_DIRECTOR_IMAGE_DIGEST: sha256:01b7fc3e6dce66180034f268a2dc92c05458706c5b3a0dc4450dcdd6161f6e73
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DEPLOY_MODE: ${{ inputs.mode }}
EVIDENCE_MODE: ${{ inputs.evidence-mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
WAVE_INDEX: ${{ inputs.wave-index }}
SOURCE_WAVE_RUN_ID: ${{ inputs.source-wave-run-id }}
steps:
- name: Require exact reusable-workflow invocation
working-directory: .
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
if test "${EVIDENCE_MODE}" = continuation; then
test "${DEPLOY_MODE}" = apply
[[ "${WAVE_INDEX}" =~ ^[0-3]$ ]]
test "${WAVE_CELL_IDS}" != none
test "${SOURCE_WAVE_RUN_ID}" = none
elif test "${EVIDENCE_MODE}" = resume; then
test "${DEPLOY_MODE}" = apply
test "${WAVE_INDEX}" = resume
test "${WAVE_CELL_IDS}" != none
[[ "${SOURCE_WAVE_RUN_ID}" =~ ^[0-9]+$ ]]
else
test "${EVIDENCE_MODE}" = single
test "${WAVE_CELL_IDS}" = none
test "${WAVE_INDEX}" = 0
test "${SOURCE_WAVE_RUN_ID}" = none
fi
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
run: |
test -n "${GCP_REGION}"
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- id: resume-provenance
if: ${{ inputs.evidence-mode == 'resume' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
test "${MONITOR_RUN_ATTEMPT}" = 1
case "${MONITOR_RUN_ID}:${SOURCE_WAVE_RUN_ID}:${WAVE_CELL_IDS}:${TARGET_CELL_ID}" in
31554591366:31555510376:production-gce-c16,production-gce-c15,production-gce-c14,production-gce-c13:production-gce-c13)
EXPECTED_SHA=a917e8e1fc1a2654e8cb81ba39b57733ec56be9c
EXPECTED_SOURCE_ATTEMPT=1
;;
31562760783:31563664692:production-gce-c10,production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c10)
EXPECTED_SHA=6082e9ca89a918ca51f0c87db003f5e8805b64b7
EXPECTED_SOURCE_ATTEMPT=1
;;
31571019947:31572080665:production-gce-c9,production-gce-c8,production-gce-c7:production-gce-c8)
EXPECTED_SHA=e59958130c9d9b7a6cd805df2678d08997842c7c
EXPECTED_SOURCE_ATTEMPT=2
;;
*) exit 1 ;;
esac
MONITOR_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${MONITOR_RUN_ID}" \
--jq 'select(.name == "Monitor Relay Production" and
.path == ".github/workflows/cloud-monitor-relay-production.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "success" and .run_attempt == 1) |
.head_sha')"
SOURCE_SHA="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq 'select(.name == "Deploy Relay Production Capacity" and
.path == ".github/workflows/cloud-deploy-relay-production-capacity.yml" and
.head_branch == "main" and .head_repository.full_name == env.GITHUB_REPOSITORY and
.event == "workflow_dispatch" and .conclusion == "failure") |
.head_sha')"
SOURCE_ATTEMPT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_WAVE_RUN_ID}" \
--jq '.run_attempt')"
[[ "${MONITOR_SHA}" =~ ^[0-9a-f]{40}$ ]]
test "${MONITOR_SHA}" = "${EXPECTED_SHA}"
test "${SOURCE_SHA}" = "${MONITOR_SHA}"
test "${SOURCE_ATTEMPT}" = "${EXPECTED_SOURCE_ATTEMPT}"
echo "commit-sha=${MONITOR_SHA}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode == 'apply' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode == 'apply' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode == 'apply' }}
run: |
EVIDENCE_COMMIT_SHA="${GITHUB_SHA}"
if test "${EVIDENCE_MODE:-single}" = resume; then
EVIDENCE_COMMIT_SHA="${{ steps.resume-provenance.outputs.commit-sha }}"
fi
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${EVIDENCE_COMMIT_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- name: Download this workflow's wave authority
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Download the failed wave authority for resume
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-wave-authority
github-token: ${{ github.token }}
run-id: ${{ inputs.source-wave-run-id }}
- name: Require wave evidence consumed by this workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${GITHUB_RUN_ID}"
- name: Require wave evidence consumed by the failed source workflow
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
test "$(< "${RUNNER_TEMP}/relay-wave-authority/${MARKER_NAME}")" = "${SOURCE_WAVE_RUN_ID}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Require exact mutation confirmation
if: ${{ inputs.mode != 'verify' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if test "${EVIDENCE_MODE:-single}" = resume; then
test "${CONFIRMATION}" = "RESUME_SELECTED_CELL_TO_1000 ${TARGET_CELL_ID}"
elif test "${DEPLOY_MODE}" = apply; then
test "${CONFIRMATION}" = "RAISE_SELECTED_CELL_TO_1000"
else
test "${CONFIRMATION}" = "ROLL_BACK_SELECTED_CELL_TO_600 ${TARGET_CELL_ID}"
fi
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Build the exact selected-cell configuration
shell: bash
run: |
if test "${DEPLOY_MODE}" = rollback; then
TARGET_HARD_CAP=600
else
TARGET_HARD_CAP=1000
fi
TARGET_UNOBSERVED_BOUND=60
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
[[ "${TARGET_HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
OVERRIDE_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${TARGET_HARD_CAP}" \
--argjson bound "${TARGET_UNOBSERVED_BOUND}" \
'.[$cell].connection_hard_cap = $cap |
.[$cell].connection_unobserved_bound = $bound' \
<<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-capacity.tfvars.json"
BASE_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'local.relay_director_cells_json' | jq -er '.')"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
<<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
<<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
jq -e --arg cell "${TARGET_CELL_ID}" \
'any(.[]; .id == $cell and .connectionHardCap == 1000 and
.connectionUnobservedBound == 60)' \
<<< "${BASE_CELLS_JSON}" >/dev/null
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
DESIRED_IMAGE_DIGEST="${DESIRED_IMAGE##*@}"
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
test "${MIG_NAME}" = "orca-cloud-relay-gce-${TARGET_HOSTNAME}"
{
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "TARGET_HARD_CAP=${TARGET_HARD_CAP}"
echo "TARGET_UNOBSERVED_BOUND=${TARGET_UNOBSERVED_BOUND}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "BASE_CELLS_JSON=${BASE_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact compatible production image and topology
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_REVISION_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_IMAGE="$(jq -er '.spec.containers[0].image' <<< "${ACTIVE_REVISION_JSON}")"
ACTIVE_IMAGE_DIGEST="${ACTIVE_IMAGE##*@}"
if test "${ACTIVE_IMAGE}" != "${DESIRED_IMAGE}"; then
test "${ACTIVE_IMAGE_DIGEST}" = "${COMPATIBLE_DIRECTOR_IMAGE_DIGEST}"
test "${DESIRED_IMAGE_DIGEST}" = "${COMPATIBLE_CELL_IMAGE_DIGEST}"
fi
CURRENT_CELLS_JSON="$(jq -cer '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end' \
<<< "${ACTIVE_REVISION_JSON}")"
CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON="$(
node dev/scripts/read-relay-production-capacity-identity.mjs \
<<< "${ACTIVE_REVISION_JSON}"
)"
CLASSIFICATION="$(jq -nc \
--argjson baseCells "${BASE_CELLS_JSON}" \
--argjson currentCells "${CURRENT_CELLS_JSON}" \
--arg capacityCellIds "${CAPACITY_CELL_IDS}" \
--arg targetCellId "${TARGET_CELL_ID}" \
--argjson targetHardCap "${TARGET_HARD_CAP}" \
--argjson currentCapacityServiceAccount \
"${CURRENT_CAPACITY_SERVICE_ACCOUNT_JSON}" \
'{baseCells:$baseCells, currentCells:$currentCells,
capacityCellIds:($capacityCellIds | split(",")),
targetCellId:$targetCellId, targetHardCap:$targetHardCap,
currentCapacityServiceAccount:$currentCapacityServiceAccount}' \
| node dev/scripts/classify-relay-production-capacity-director.mjs \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
TOPOLOGY_PHASE="$(jq -er '.topologyPhase' <<< "${CLASSIFICATION}")"
DESIRED_CELLS_JSON="$(jq -cer '.desiredCells' <<< "${CLASSIFICATION}")"
DIRECTOR_READY="$(jq -er \
'if (.directorReady | type) == "boolean" then
(.directorReady | tostring)
else error("invalid directorReady classification") end' \
<<< "${CLASSIFICATION}")"
{
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}"
echo "TOPOLOGY_PHASE=${TOPOLOGY_PHASE}"
echo "DIRECTOR_READY=${DIRECTOR_READY}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}"
} >> "${GITHUB_ENV}"
- name: Require the exact wave predecessor topology
if: ${{ inputs.mode == 'apply' && (inputs.evidence-mode == 'continuation' || inputs.evidence-mode == 'resume') }}
run: test "${TOPOLOGY_PHASE}" = predecessor
- name: Verify fresh dry-run evidence against the live selector
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${TARGET_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Recheck exact wave state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'continuation' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--wave-index "${WAVE_INDEX}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
RETRY_ARGS=()
if test "${WAVE_INDEX}" != 0; then RETRY_ARGS=(--retry-freshness); fi
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
"${RETRY_ARGS[@]}"
- name: Recheck exact isolated resume state and every live safety signal
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'resume' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/relay-production-capacity-wave.mjs build-resume-preflight \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json" \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--target-cell-id "${TARGET_CELL_ID}" \
--output-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json"
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-capacity-wave-preflight.json" \
--retry-freshness
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission migration-only \
--draining required \
--activity allowed \
--runtime required \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
- name: Consume the single-use dry-run evidence
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
if: ${{ inputs.mode == 'apply' && inputs.evidence-mode == 'single' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Verify current selected-cell capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
CURRENT_CAP="${TARGET_HARD_CAP}"
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
if test "${TOPOLOGY_PHASE}" = predecessor; then
CURRENT_CAP=600
CURRENT_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${CURRENT_IMAGE_DIGEST}"
- name: Arm fail-closed mutation cleanup
if: ${{ inputs.mode != 'verify' }}
run: echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
- name: Reversibly isolate only the selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Drain the selected cell or prove an offline rollback
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
if node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain; then
echo "OFFLINE_ROLLBACK=false" >> "${GITHUB_ENV}"
elif test "${DEPLOY_MODE}" = rollback; then
echo "OFFLINE_ROLLBACK=true" >> "${GITHUB_ENV}"
else
exit 1
fi
- id: restart-auth-one
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: restart-gate-one
name: Require restart-safe selected-cell activity
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-one.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
GATE_LOG="${RUNNER_TEMP}/relay-capacity-restart-gate-one.log"
set +e
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}" \
2> "${GATE_LOG}"
GATE_EXIT=$?
set -e
cat "${GATE_LOG}" >&2
if test "${GATE_EXIT}" = 0; then
echo "settled=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
if test "$(wc -l < "${GATE_LOG}" | tr -d ' ')" = 1 &&
grep -Eq '^capacity transition verification timed out: \{.*\}$' "${GATE_LOG}"; then
echo "settled=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
exit "${GATE_EXIT}"
- id: restart-auth-two
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require extended restart-safe selected-cell activity
if: ${{ steps.restart-gate-one.outputs.settled == 'false' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.restart-auth-two.outputs.id_token }}
run: |
CURRENT_CAP=600
if test "${TOPOLOGY_PHASE}" = desired; then
CURRENT_CAP="${TARGET_HARD_CAP}"
elif test "${TARGET_HARD_CAP}" = 600; then
CURRENT_CAP=1000
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${CURRENT_CAP}" \
--unobserved-bound 60 \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" \
--timeout-ms 450000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- name: Deploy only the reviewed director topology
if: ${{ inputs.mode != 'verify' }}
run: |
if test "${DIRECTOR_READY}" = true; then exit 0; fi
RELEASE_ID="capacity-${TARGET_HOSTNAME}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--max-instances 5 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--min-instances 5 \
--prune-revisions false \
--release-id "${RELEASE_ID}"
- id: director-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Require fail-closed director transition
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.director-transition-auth.outputs.id_token }}
run: |
if test "${OFFLINE_ROLLBACK:-false}" = true; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat stale \
--admission migration-only \
--draining either \
--activity restart-safe \
--runtime unavailable
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--runtime required \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" \
--timeout-ms 480000 \
--expected-image-digests \
"${DESIRED_IMAGE_DIGEST},${PREDECESSOR_IMAGE_DIGEST}"
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Plan and apply only the empty selected cell
if: ${{ inputs.mode != 'verify' }}
shell: bash
run: |
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-capacity.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode bootstrap-cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}")"
echo "${PLAN_RESULT}"
PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${PLAN_CHANGES}" =~ ^(0|1|2)$ ]]
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
else
INSTANCE="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--format=json | jq -er 'if length == 1 and
.[0].instanceStatus == "RUNNING" and .[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("selected cell is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" --instances "${INSTANCE}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" --stable --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --timeout 900
- id: capacity-transition-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify fresh exact selected-cell heartbeat before admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- name: Restore only the selected cell to general admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the live general selected cell
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.capacity-transition-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${TARGET_HARD_CAP}" \
--unobserved-bound "${TARGET_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}"
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep the selected cell isolated after a failed mutation
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
CLEANUP_STATUS=0
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate || CLEANUP_STATUS=$?
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode drain || CLEANUP_STATUS=$?
exit "${CLEANUP_STATUS}"
@@ -1,355 +0,0 @@
name: Deploy Relay Production Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify, change one cell, or raise a sequential wave
required: true
default: verify
type: choice
options:
- verify
- apply
- rollback
- wave-apply
- wave-resume
target-cell-id:
description: Exact serving cell for verify, apply, or rollback
required: true
default: production-gce-c26
type: choice
options:
- production-gce-c7
- production-gce-c8
- production-gce-c9
- production-gce-c10
- production-gce-c13
- production-gce-c14
- production-gce-c15
- production-gce-c16
- production-gce-c19
- production-gce-c20
- production-gce-c21
- production-gce-c22
- production-gce-c23
- production-gce-c24
- production-gce-c25
- production-gce-c26
wave-cell-ids:
description: Ordered comma-separated wave of two to four serving cells
required: false
default: none
type: string
confirmation:
description: Enter the exact single-cell or wave confirmation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID for apply
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt for apply
required: false
type: string
source-wave-run-id:
description: Failed wave run that isolated the resume target
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
single_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode != 'wave-apply' && inputs.mode != 'wave-resume') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: ${{ inputs.mode }}
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: single
wave-cell-ids: none
wave-index: '0'
source-wave-run-id: none
secrets: inherit
resume_cell:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-resume' && github.ref == 'refs/heads/main') }}
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ inputs.target-cell-id }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: resume
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: resume
source-wave-run-id: ${{ inputs.source-wave-run-id }}
secrets: inherit
wave_gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (inputs.mode == 'wave-apply' && github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
env:
DIRECTOR_ORIGIN: https://relay.onorca.dev
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
PREDECESSOR_IMAGE_DIGEST: sha256:0e83408b0dc08531f1e8182019dc151afc38d63ddde4ad5cc01e40247ef3681d
COMPATIBLE_CELL_IMAGE_DIGEST: sha256:c77ec7aef565009fdb645b0989806859bfa40a7aa14e4a57ab55ac92fee6c34f
WAVE_CELL_IDS: ${{ inputs.wave-cell-ids }}
steps:
- name: Require production workflow configuration
working-directory: .
env:
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
run: |
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- id: wave
name: Validate the exact wave request
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
CELLS="$(node dev/scripts/relay-production-capacity-wave.mjs validate \
--wave-cell-ids "${WAVE_CELL_IDS}" \
--confirmation "${CONFIRMATION}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
- name: Require fresh dry-run evidence reference
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify dry-run artifact before cloud authentication
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Verify wave evidence against the live selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
FIRST_CELL="$(jq -er '.[0]' <<< '${{ steps.wave.outputs.cells }}')"
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode capacity-transition \
--source-cell-id "${FIRST_CELL}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck every live safety signal
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Require exact 600/60 predecessor wave cells
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
while read -r CELL_ID; do
HOSTNAME="${CELL_ID#production-gce-}"
[[ "${HOSTNAME}" =~ ^c(7|8|9|10|13|14|15|16|19|20|21|22|23|24|25|26)$ ]]
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "https://${HOSTNAME}.relay.onorca.dev" \
--cell-id "${CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed \
--expected-image-digests \
"${PREDECESSOR_IMAGE_DIGEST},${COMPATIBLE_CELL_IMAGE_DIGEST}"
done < <(jq -r '.[]' <<< '${{ steps.wave.outputs.cells }}')
- name: Consume the single-use dry-run evidence
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Publish the consumed-evidence marker
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
wave_cell_1:
needs: wave_gate
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[0] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '0'
source-wave-run-id: none
secrets: inherit
wave_cell_2:
needs: [wave_gate, wave_cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[1] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '1'
source-wave-run-id: none
secrets: inherit
wave_cell_3:
if: ${{ needs.wave_cell_2.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[2] != null }}
needs: [wave_gate, wave_cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[2] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '2'
source-wave-run-id: none
secrets: inherit
wave_cell_4:
if: ${{ needs.wave_cell_3.result == 'success' && fromJSON(needs.wave_gate.outputs.cells)[3] != null }}
needs: [wave_gate, wave_cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-capacity-job.yml
with:
mode: apply
target-cell-id: ${{ fromJSON(needs.wave_gate.outputs.cells)[3] }}
confirmation: RAISE_SELECTED_CELL_TO_1000
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
evidence-mode: continuation
wave-cell-ids: ${{ inputs.wave-cell-ids }}
wave-index: '3'
source-wave-run-id: none
secrets: inherit
# Every wave job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- single_cell
- resume_cell
- wave_gate
- wave_cell_1
- wave_cell_2
- wave_cell_3
- wave_cell_4
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -1,310 +0,0 @@
name: Deploy Relay Production Director
on:
workflow_dispatch:
inputs:
image-digest:
description: 'Immutable relay image digest (sha256: plus 64 lowercase hex characters)'
required: true
type: string
regional-placement-mode:
description: Preserve the live switch, explicitly enable Asia preference, or force US-first
required: true
default: preserve
type: choice
options: [preserve, enable, disable]
region-correction-cohort-percent:
description: 'Preserve the measured-correction cohort, or set an integer 0–100; durable rehome stays disabled'
required: true
default: preserve
type: string
reserve-placement:
description: Preserve step-5 reserve placement, or set it; turning it off is refused while any cell is reserve
required: true
default: preserve
type: choice
options: [preserve, 'off', dry-run, 'on']
shadow-seat-feed-cells:
description: Preserve the step-3 shadow seat-feed cells, or set all or a comma list of cell ids; dropping a reserve cell is refused
required: true
default: preserve
type: string
prune-incompatible-revisions:
description: Retain only the newly verified serving and rollback revisions
required: true
default: false
type: boolean
confirmation:
description: Enter the exact confirmation required by a destructive option
required: false
type: string
expected-rehome-generation:
description: Exact durable regional-rehome generation; it must remain disabled
required: true
type: string
bootstrap-runtime-identity:
description: One-time move from the stamped-cell identity to the director identity
required: true
default: false
type: boolean
predecessor-image-digest:
description: Exact immutable serving predecessor digest for the one-time identity bootstrap
required: true
type: string
shape-baseline-revision:
description: Only after a Terraform apply changed the director template. The apply's 0% revision, whose shape the deploy adopts
required: false
type: string
permissions:
contents: read
id-token: write
# Director updates and candidate operations both mutate production relay control state.
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
IMAGE_DIGEST: ${{ inputs.image-digest }}
REGIONAL_PLACEMENT_MODE: ${{ inputs.regional-placement-mode }}
REGION_CORRECTION_COHORT_PERCENT: ${{ inputs.region-correction-cohort-percent }}
RESERVE_PLACEMENT: ${{ inputs.reserve-placement }}
SHADOW_SEAT_FEED_CELLS: ${{ inputs.shadow-seat-feed-cells }}
PRUNE_INCOMPATIBLE_REVISIONS: ${{ inputs.prune-incompatible-revisions }}
# Floor the served revision must keep, matching relay_min_instances in
# environments/production.tfvars. This gate only fails a bad deploy; Terraform
# still owns the value, and the candidate inherits it from the serving revision.
DIRECTOR_MIN_INSTANCES: 5
DIRECTOR_MAX_INSTANCES: 5
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
PREDECESSOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_RUNTIME_SERVICE_ACCOUNT }}
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
SHAPE_BASELINE_REVISION: ${{ inputs.shape-baseline-revision }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Resolve immutable production image
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ ! "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]; then
echo "image-digest must be an immutable lowercase sha256 digest" >&2
exit 1
fi
if test "${REGION_CORRECTION_COHORT_PERCENT}" != preserve; then
[[ "${REGION_CORRECTION_COHORT_PERCENT}" =~ ^([0-9]|[1-9][0-9]|100)$ ]]
fi
IMAGE="${IMAGE_REPOSITORY}@${IMAGE_DIGEST}"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" --project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${IMAGE_DIGEST}"
[[ "${PRUNE_INCOMPATIBLE_REVISIONS}" =~ ^(true|false)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" =~ ^[a-z][a-z0-9-]+@${GCP_PROJECT_ID}[.]iam[.]gserviceaccount[.]com$ ]]
[[ "${BOOTSTRAP_RUNTIME_IDENTITY}" =~ ^(true|false)$ ]]
if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test -z "${SHAPE_BASELINE_REVISION}"
test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY
elif test -n "${SHAPE_BASELINE_REVISION}"; then
[[ "${SHAPE_BASELINE_REVISION}" =~ ^orca-cloud-relay-[0-9]{5}-[a-z0-9]{3}$ ]]
test "${PRUNE_INCOMPATIBLE_REVISIONS}" = false
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE
elif test "${PRUNE_INCOMPATIBLE_REVISIONS}" = true; then
test "${REGIONAL_PLACEMENT_MODE}" = preserve
test "${CONFIRMATION}" = PRUNE_INCOMPATIBLE_RELAY_DIRECTOR_REVISIONS
elif test "${REGIONAL_PLACEMENT_MODE}" = disable; then
test "${CONFIRMATION}" = FORCE_RELAY_US_FIRST
else
test -z "${CONFIRMATION}"
fi
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
# Why: the deploy INHERITS the serving revision's floor, so when that revision has
# already lost it the candidate inherits zero, the in-script gate compares zero against
# zero and passes, and the post-deploy check below only notices after traffic moved.
# The documented rollback target is created at minimum instances zero, so promoting it
# arms exactly that. Refuse to inherit a degraded floor rather than latch it.
- name: Require a healthy serving floor before deploying
shell: bash
run: |
SERVING="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${SERVING}"
FLOOR="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${FLOOR:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "serving revision ${SERVING} holds ${FLOOR:-0} minimum instances," \
"below ${DIRECTOR_MIN_INSTANCES}; deploying would inherit and latch it." >&2
echo "Restore the floor first: gcloud run services update ${DIRECTOR_SERVICE_NAME}" \
"--min-instances=${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
CEILING="$(gcloud run revisions describe "${SERVING}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${CEILING}" = "${DIRECTOR_MAX_INSTANCES}"
echo "serving revision ${SERVING} holds ${FLOOR} minimum instances"
echo "SERVING_REVISION=${SERVING}" >> "${GITHUB_ENV}"
# Why: no --min-instances here. The candidate inherits the Terraform-owned
# scaling, and this step ends with 100% traffic on it. Pinning 1 rebuilt the
# per-instance admission shortage that took placement failures to ~70%.
- name: Deploy director blue/green
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
served_version="$(gcloud run revisions describe "${SERVING_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
(.version // .key // empty)] |
if length == 1 then .[0] else empty end')"
if [[ "${served_version}" =~ ^[1-9][0-9]*$ ]]; then
current_version="${served_version}"
else
test "${REGIONAL_PLACEMENT_MODE}" = preserve
current_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${current_version}" =~ ^[1-9][0-9]*$ ]]
fi
current="$(gcloud secrets versions access "${current_version}" \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
case "${REGIONAL_PLACEMENT_MODE}" in
preserve) desired="${current}" ;;
enable) desired=true ;;
disable) desired=false ;;
*) echo "regional-placement-mode is invalid" >&2; exit 1 ;;
esac
if test "${current}" != "${desired}"; then
target_version="$(printf '%s' "${desired}" | gcloud secrets versions add \
"${REGIONAL_PLACEMENT_SECRET}" --project "${GCP_PROJECT_ID}" --data-file=- \
--format='value(name)' --quiet | awk -F/ '{print $NF}')"
else
target_version="${current_version}"
fi
[[ "${target_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
shape_args=()
if test -n "${SHAPE_BASELINE_REVISION}"; then
shape_args=(--shape-baseline-revision "${SHAPE_BASELINE_REVISION}")
fi
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
"${shape_args[@]}" \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}" \
--bootstrap-runtime-identity "${BOOTSTRAP_RUNTIME_IDENTITY}" \
--predecessor-image-digest "${PREDECESSOR_IMAGE_DIGEST}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience "${REHOME_AUDIENCE}" \
--rehome-control-origin https://relay.onorca.dev \
--admin-audience https://relay.onorca.dev/v1/admin/drain \
--expected-rehome-generation "${EXPECTED_REHOME_GENERATION}" \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--prune-revisions "${PRUNE_INCOMPATIBLE_REVISIONS}" \
--release-id "${RELEASE_ID}" \
--regional-placement-secret-version "${target_version}" \
--region-correction-cohort-percent "${REGION_CORRECTION_COHORT_PERCENT}" \
--reserve-placement "${RESERVE_PLACEMENT}" \
--shadow-seat-feed-cells "${SHADOW_SEAT_FEED_CELLS}"
echo "REGIONAL_PLACEMENT_ENABLED=${desired}" >> "${GITHUB_ENV}"
echo "REGIONAL_PLACEMENT_VERSION=${target_version}" >> "${GITHUB_ENV}"
- name: Verify served revision and native health
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format=json)"
REVISION="$(jq -r '[.status.traffic[] | select((.percent // 0) > 0)] | if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' <<< "${SERVICE_JSON}")"
test -n "${REVISION}"
SERVED_IMAGE="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${SERVED_IMAGE}" = "${IMAGE}"
SERVED_REGIONAL_PLACEMENT_SECRET="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)}] |
if length == 1 then .[0] else error("regional placement secret missing") end')"
test "$(jq -r '.secret' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_SECRET}"
test "$(jq -r '.version' <<< "${SERVED_REGIONAL_PLACEMENT_SECRET}")" = \
"${REGIONAL_PLACEMENT_VERSION}"
test "$(gcloud secrets versions access "${REGIONAL_PLACEMENT_VERSION}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" = "${REGIONAL_PLACEMENT_ENABLED}"
# Why: a served revision with no warm-instance floor still passes health and digest
# checks while quietly shrinking per-instance admission capacity.
SERVED_MIN_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${SERVED_MIN_INSTANCES:-0}" -lt "${DIRECTOR_MIN_INSTANCES}" ]]; then
echo "served revision ${REVISION} holds ${SERVED_MIN_INSTANCES:-0} minimum instances, expected at least ${DIRECTOR_MIN_INSTANCES}" >&2
exit 1
fi
SERVED_MAX_INSTANCES="$(gcloud run revisions describe "${REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${SERVED_MAX_INSTANCES}" = "${DIRECTOR_MAX_INSTANCES}"
SERVICE_URL="$(jq -r '.status.url' <<< "${SERVICE_JSON}")"
node dev/scripts/smoke-relay.mjs "${SERVICE_URL}"
@@ -1,499 +0,0 @@
name: Deploy Relay Production Multi-Target
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform source cell ID
required: true
type: string
target-cell-ids:
description: Comma-separated distinct Terraform target cell IDs
required: true
type: string
general-cell-ids:
description: Comma-separated proven cells that remain eligible for ordinary placement
required: false
type: string
unobserved-connection-bound:
description: Exact worst-case unobserved connection bound proven by the passing load gate
required: false
type: string
failed-target-cell-id:
description: Registered failed target to fence and supersede
required: false
type: string
replacement-target-cell-id:
description: Healthy replacement for registered failed target
required: false
type: string
mode:
description: Preflight/audit are read-only; other modes mutate production
required: true
default: preflight
type: choice
options:
- audit
- preflight
- cutover-admission
- add-migration-cells
- promote-general-cell
- retire-migration-cell
- execute
- recover-forward
- fence-source
- supersede-target
confirmation:
description: Enter CUTOVER_SELECTOR, ADD_MIGRATION_CELLS, PROMOTE_GENERAL_CELL, RETIRE_MIGRATION_CELL, EVACUATE_MULTI, RECOVER_FORWARD, or FENCE_SOURCE
required: false
type: string
selector-attempt-id:
description: Exact durable selector attempt ID for admission mutations
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
broker-operation-id:
description: Stable durable broker operation ID for target supersession
required: false
type: string
completed-fence-attempt-id:
description: Exact older completed fence attempt to recover without replay
required: false
type: string
completed-fence-commit:
description: Exact older fence commit bound to the completed attempt
required: false
type: string
completed-fence-operation:
description: Exact DONE Compute resize operation to adopt
required: false
type: string
completed-fence-state-serial:
description: Exact Terraform serial before the completed fence
required: false
type: string
completed-fence-plan-generation:
description: Exact saved-plan object generation
required: false
type: string
completed-fence-state-generation:
description: Exact current Terraform state object generation
required: false
type: string
completed-fence-state-sha256:
description: Exact current Terraform state object SHA-256
required: false
type: string
expected-lease-generation:
description: Exact live lease generation authorized for conditional takeover
required: false
type: string
expected-lease-operation-id:
description: Exact live lease operation ID authorized for takeover
required: false
type: string
expected-lease-request-digest:
description: Exact live lease request digest authorized for takeover
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_IDS: ${{ inputs.target-cell-ids }}
GENERAL_CELL_IDS: ${{ inputs.general-cell-ids }}
UNOBSERVED_CONNECTION_BOUND: ${{ inputs.unobserved-connection-bound }}
FAILED_TARGET_CELL_ID: ${{ inputs.failed-target-cell-id }}
REPLACEMENT_TARGET_CELL_ID: ${{ inputs.replacement-target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
BROKER_OPERATION_ID: ${{ inputs.broker-operation-id }}
COMPLETED_FENCE_ATTEMPT_ID: ${{ inputs.completed-fence-attempt-id }}
COMPLETED_FENCE_COMMIT: ${{ inputs.completed-fence-commit }}
COMPLETED_FENCE_OPERATION: ${{ inputs.completed-fence-operation }}
COMPLETED_FENCE_STATE_SERIAL: ${{ inputs.completed-fence-state-serial }}
COMPLETED_FENCE_PLAN_GENERATION: ${{ inputs.completed-fence-plan-generation }}
COMPLETED_FENCE_STATE_GENERATION: ${{ inputs.completed-fence-state-generation }}
COMPLETED_FENCE_STATE_SHA256: ${{ inputs.completed-fence-state-sha256 }}
EXPECTED_LEASE_GENERATION: ${{ inputs.expected-lease-generation }}
EXPECTED_LEASE_OPERATION_ID: ${{ inputs.expected-lease-operation-id }}
EXPECTED_LEASE_REQUEST_DIGEST: ${{ inputs.expected-lease-request-digest }}
steps:
- uses: actions/checkout@v4
- name: Require private fence-broker environment
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
env:
FENCE_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
FENCE_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
FENCE_BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
test -n "${FENCE_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${FENCE_SERVICE_ACCOUNT}"
test -n "${FENCE_BROKER_URI}"
- name: Reject direct-runner Terraform fence aborts
if: ${{ inputs.mode == 'abort-fence-source' }}
run: |
echo "Terraform fence aborts require a reviewed private-broker recovery path." >&2
exit 1
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
if: ${{ inputs.mode != 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "cutover-admission" ]]; then
test "${CONFIRMATION}" = "CUTOVER_SELECTOR"
elif [[ "${DEPLOY_MODE}" = "add-migration-cells" ]]; then
test "${CONFIRMATION}" = "ADD_MIGRATION_CELLS"
elif [[ "${DEPLOY_MODE}" = "promote-general-cell" ]]; then
test "${CONFIRMATION}" = "PROMOTE_GENERAL_CELL"
elif [[ "${DEPLOY_MODE}" = "retire-migration-cell" ]]; then
test "${CONFIRMATION}" = "RETIRE_MIGRATION_CELL"
elif [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE_MULTI"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "fence-source" ]]; then
test "${CONFIRMATION}" = "FENCE_SOURCE"
elif [[ "${DEPLOY_MODE}" = "supersede-target" ]]; then
test "${CONFIRMATION}" = "SUPERSEDE_TARGET"
elif [[ "${DEPLOY_MODE}" = "abort-fence-source" ]]; then
test "${CONFIRMATION}" = "ABORT_FENCE"
else
test "${CONFIRMATION}" = "FENCE_SOURCE"
fi
- name: Require exact source-fence broker contract
if: ${{ inputs.mode == 'fence-source' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${TARGET_CELL_IDS}" = "production-gce-c7,production-gce-c8,production-gce-c10,production-gce-c13,production-gce-c17,production-gce-c18"
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Require exact broker cell contract
if: ${{ inputs.mode == 'supersede-target' }}
run: |
test "${SOURCE_CELL_ID}" = "production-gce-c3"
test "${FAILED_TARGET_CELL_ID}" = "production-gce-c12"
test "${REPLACEMENT_TARGET_CELL_ID}" = "production-gce-c13"
test "${TARGET_CELL_IDS}" = "production-gce-c12,production-gce-c13"
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
[[ "${COMPLETED_FENCE_ATTEMPT_ID}" =~ ^[0-9a-f-]{36}$ ]]
[[ "${COMPLETED_FENCE_COMMIT}" =~ ^[0-9a-f]{40}$ ]]
[[ "${COMPLETED_FENCE_OPERATION}" =~ ^[A-Za-z0-9._-]{1,256}$ ]]
[[ "${COMPLETED_FENCE_STATE_SERIAL}" =~ ^[0-9]+$ ]]
[[ "${COMPLETED_FENCE_PLAN_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${COMPLETED_FENCE_STATE_SHA256}" =~ ^[0-9a-f]{64}$ ]]
test -n "${EXPECTED_LEASE_GENERATION}"
fi
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
[[ "${EXPECTED_LEASE_GENERATION}" =~ ^[1-9][0-9]*$ ]]
test "${EXPECTED_LEASE_OPERATION_ID}" = "${BROKER_OPERATION_ID}"
[[ "${EXPECTED_LEASE_REQUEST_DIGEST}" =~ ^[0-9a-f]{64}$ ]]
else
test -z "${EXPECTED_LEASE_OPERATION_ID}"
test -z "${EXPECTED_LEASE_REQUEST_DIGEST}"
fi
- name: Read reviewed Terraform topology
if: ${{ inputs.mode != 'supersede-target' }}
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
DIRECTOR_MIN_INSTANCES="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars <<< 'var.relay_min_instances')"
[[ "${DIRECTOR_MIN_INSTANCES}" =~ ^[1-9][0-9]*$ ]]
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
echo "DIRECTOR_MIN_INSTANCES=${DIRECTOR_MIN_INSTANCES}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
SCOPED_RECOVERY_ARGS=()
if [[ ("${DEPLOY_MODE}" = "execute" ||
"${DEPLOY_MODE}" = "recover-forward") &&
"${SOURCE_CELL_ID}" = "production-gce-c12" ]]; then
SCOPED_RECOVERY_ARGS=(
--scoped-recovery-source-cell-id
production-gce-c3
)
fi
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
"${SCOPED_RECOVERY_ARGS[@]}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' && inputs.mode != 'cutover-admission' && inputs.mode != 'add-migration-cells' && inputs.mode != 'promote-general-cell' && inputs.mode != 'retire-migration-cell' && inputs.mode != 'supersede-target' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- id: google-fence-broker-auth
if: >-
${{ inputs.mode == 'fence-source' ||
inputs.mode == 'supersede-target' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
id_token_include_email: true
- name: Invoke private target-supersession broker
if: ${{ inputs.mode == 'supersede-target' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
[[ "${BROKER_OPERATION_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
if [[ -n "${COMPLETED_FENCE_ATTEMPT_ID}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg attemptId "${COMPLETED_FENCE_ATTEMPT_ID}" \
--arg completedCommit "${COMPLETED_FENCE_COMMIT}" \
--arg gceOperation "${COMPLETED_FENCE_OPERATION}" \
--arg stateSerial "${COMPLETED_FENCE_STATE_SERIAL}" \
--arg planGeneration "${COMPLETED_FENCE_PLAN_GENERATION}" \
--arg stateGeneration "${COMPLETED_FENCE_STATE_GENERATION}" \
--arg stateSha256 "${COMPLETED_FENCE_STATE_SHA256}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
completedFenceRecovery:{attemptId:$attemptId,fenceCommit:$completedCommit,
gceOperation:$gceOperation,terraformStateSerial:($stateSerial|tonumber),
planObjectGeneration:$planGeneration,
terraformStateObjectGeneration:$stateGeneration,
terraformStateObjectSha256:$stateSha256},
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
elif [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"SUPERSEDE_TARGET"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,confirmation:"SUPERSEDE_TARGET"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/supersede-target" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Invoke private source-fence broker
if: ${{ inputs.mode == 'fence-source' }}
env:
BROKER_ID_TOKEN: ${{ steps.google-fence-broker-auth.outputs.id_token }}
BROKER_URI: ${{ vars.PRODUCTION_GCP_RELAY_FENCE_BROKER_URI }}
run: |
if [[ -n "${EXPECTED_LEASE_GENERATION}" ]]; then
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
--arg leaseGeneration "${EXPECTED_LEASE_GENERATION}" \
--arg leaseOperationId "${EXPECTED_LEASE_OPERATION_ID}" \
--arg leaseRequestDigest "${EXPECTED_LEASE_REQUEST_DIGEST}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),
expectedLease:{generation:$leaseGeneration,operationId:$leaseOperationId,
requestDigest:$leaseRequestDigest},confirmation:"FENCE_SOURCE"}')"
else
REQUEST="$(jq -cn \
--arg operationId "${BROKER_OPERATION_ID}" \
--arg fenceCommit "${GITHUB_SHA}" \
--arg targetCellIds "${TARGET_CELL_IDS}" \
'{v:1,operationId:$operationId,fenceCommit:$fenceCommit,
targetCellIds:($targetCellIds|split(",")),confirmation:"FENCE_SOURCE"}')"
fi
curl --fail-with-body --max-time 1790 \
--request POST "${BROKER_URI}/v1/fence-source" \
--header "Authorization: Bearer ${BROKER_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "${REQUEST}"
- name: Preflight or run multi-target evacuation
if: >-
${{ inputs.mode != 'fence-source' &&
inputs.mode != 'supersede-target' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-multi-target.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-ids "${TARGET_CELL_IDS}" \
--general-cell-ids "${GENERAL_CELL_IDS}" \
--unobserved-connection-bound "${UNOBSERVED_CONNECTION_BOUND}" \
--director-region "${{ vars.PRODUCTION_GCP_REGION }}" \
--director-service "orca-cloud-relay" \
--director-min-instances "${DIRECTOR_MIN_INSTANCES}" \
--selector-attempt-id "${SELECTOR_ATTEMPT_ID}" \
--failed-target-cell-id "${FAILED_TARGET_CELL_ID}" \
--replacement-target-cell-id "${REPLACEMENT_TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--environment production \
--fence-commit "${GITHUB_SHA}" \
--terraform-dir infra/terraform \
--terraform-var-file environments/production.tfvars \
--mode "${DEPLOY_MODE}" \
--connection-ceiling 1000 \
--minimum-lease-remaining-ms 600000
@@ -1,923 +0,0 @@
name: Deploy Relay Production Same-Cap Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
target-cell-id: { required: true, type: string }
target-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
target-rehome-protocol: { required: true, type: string }
rollback-rehome-protocol: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-rehome-generation: { required: true, type: string }
wave-index: { required: true, type: string }
drain-pace-window-ms: { required: true, type: string }
permissions:
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
rollout:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
# The pre-drain sample adds 3-8 min (20 at most); a timeout cancels the job and skips the failsafe.
# A 1200000 drain step alone may take 50 min.
timeout-minutes: 120
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay.onorca.dev
IMAGE_REPOSITORY: us-central1-docker.pkg.dev/onorca-cloud/orca-cloud/relay
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
TARGET_REHOME_PROTOCOL: ${{ inputs.target-rehome-protocol }}
ROLLBACK_REHOME_PROTOCOL: ${{ inputs.rollback-rehome-protocol }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
WAVE_INDEX: ${{ inputs.wave-index }}
# The cell spreads its drain sends over this window, so each drained host re-dials
# /v1/assign at hosts / window and is re-placed through the director's drain-return
# lane. The wave validator admits only its closed set, and anything below the 300000
# default only on US general cells: Asia drains are bound by their targets' accept
# rate. The transition wait below is the 15-min migration lease plus this window. A
# cell on an older image drains at once, or refuses a window above 300000.
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
CAPACITY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${DEPLOY_MODE}" =~ ^(verify|apply|rollback)$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${TARGET_IMAGE_DIGEST}" != "${ROLLBACK_IMAGE_DIGEST}"
[[ "${TARGET_REHOME_PROTOCOL}" =~ ^(0|1|3)$ ]]
[[ "${ROLLBACK_REHOME_PROTOCOL}" =~ ^(0|1|3)$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
# cell_1..cell_10 in the calling wave; the chain is static, so this range is too.
[[ "${WAVE_INDEX}" =~ ^[0-9]$ ]]
if test "${DEPLOY_MODE}" != verify && test "${GITHUB_RUN_ATTEMPT}" != 1; then
echo "mutations are single-dispatch: recover each remaining cell" >&2
echo "with its own fresh dispatch instead of a re-run" >&2
exit 1
fi
test -n "${GCP_REGION}"
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
test -n "${CAPACITY_WIF}"
test -n "${CAPACITY_SERVICE_ACCOUNT}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with: { package_json_file: cloud/package.json }
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with: { terraform_wrapper: false }
# One approved-cell table, in the wave validator the dispatch gate already uses, so
# a cell's class and its wave's selector delta cannot drift apart between the two.
# Also re-checks the drain pace against this cell, before anything reads production.
- name: Resolve this cell's admission class and wave selector delta
run: |
CELL_CLASS="$(node dev/scripts/relay-production-same-cap-wave.mjs cell-class \
--cell-id "${TARGET_CELL_ID}" --drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}")"
ENTRY_ADMISSION="$(jq -er '.entryAdmission' <<< "${CELL_CLASS}")"
SELECTOR_WAVE_DELTA="$(jq -er '.selectorWaveDelta' <<< "${CELL_CLASS}")"
if test "${DEPLOY_MODE}" = verify; then
EFFECTIVE_SELECTOR_GENERATION="${EXPECTED_SELECTOR_GENERATION}"
else
EFFECTIVE_SELECTOR_GENERATION="$((EXPECTED_SELECTOR_GENERATION \
+ (SELECTOR_WAVE_DELTA * WAVE_INDEX)))"
fi
{
echo "ENTRY_ADMISSION=${ENTRY_ADMISSION}"
echo "SELECTOR_WAVE_DELTA=${SELECTOR_WAVE_DELTA}"
echo "EFFECTIVE_SELECTOR_GENERATION=${EFFECTIVE_SELECTOR_GENERATION}"
} >> "${GITHUB_ENV}"
- id: deploy-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'false'
- name: Recheck aggregate SQL, pool, reconnect, migration, and selector safety
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
# Freshness-only failures are publish lag, not health, so the CLI retries
# them a bounded number of times. The expected selector comes from the
# dispatch inputs, which the rehome inspect below verifies against the
# live director; the pre-drain sample later reads the same file.
jq -n \
--arg existingOnly "${EXPECTED_EXISTING_ONLY_CELLS/none/}" \
--arg migrationOnly "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg general "${EXPECTED_GENERAL_CELLS/none/}" \
'{existingOnly:$existingOnly,migrationOnly:$migrationOnly,general:$general}
| map_values(split(",") | map(select(length > 0)))' \
> "${RUNNER_TEMP}/relay-same-cap-selector.json"
pnpm incident:relay-preflight -- \
--no-monitor-state \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--selector-membership-file "${RUNNER_TEMP}/relay-same-cap-selector.json" \
--wave-index "${WAVE_INDEX}" \
--selector-wave-delta "${SELECTOR_WAVE_DELTA}" --retry-freshness
- name: Require durable rehome disabled and exact selector
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect \
--director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EFFECTIVE_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
- name: Initialize the exact production backend
run: node dev/scripts/infra.mjs init --env production
- name: Resolve immutable same-cap cell configuration
shell: bash
run: |
TARGET_HOSTNAME="${TARGET_CELL_ID#production-gce-}"
case "${TARGET_HOSTNAME}" in
c7|c8|c9|c10|c13|c14|c15|c16|c19|c20|c21|c22|c23|c24|c25|c26)
EXPECTED_HARD_CAP=1000
EXPECTED_REGION=us-central1
EXPECTED_DATABASE_POOL_MAX=
;;
c17|c18)
EXPECTED_HARD_CAP=600
EXPECTED_REGION=us-central1
EXPECTED_DATABASE_POOL_MAX=
;;
c27|c28|c29|c30|c31|c34|c35)
EXPECTED_HARD_CAP=3000
EXPECTED_REGION=asia-east2
EXPECTED_DATABASE_POOL_MAX=16
;;
c32|c33)
EXPECTED_HARD_CAP=3000
EXPECTED_REGION=us-central1
EXPECTED_DATABASE_POOL_MAX=
;;
*) exit 1 ;;
esac
EXPECTED_UNOBSERVED_BOUND=60
CELL_ORIGIN="https://${TARGET_HOSTNAME}.relay.onorca.dev"
CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
SOURCE_CELLS="$(terraform -chdir=infra/terraform console \
-var-file=environments/production.tfvars \
<<< 'jsonencode(var.relay_region_rehome_source_cell_ids)' | jq -er '.')"
CURRENT_SHAPE="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${CELLS_JSON}")"
test "$(jq -r '.connection_hard_cap' <<< "${CURRENT_SHAPE}")" = "${EXPECTED_HARD_CAP}"
test "$(jq -r '.connection_unobserved_bound' <<< "${CURRENT_SHAPE}")" = \
"${EXPECTED_UNOBSERVED_BOUND}"
# The startup script emits a pool line only off the root default, so an unpinned cell
# must still be on that default or its plan would carry a line nothing reviews.
test "$(jq -r '.database_pool_max' <<< "${CURRENT_SHAPE}")" = \
"${EXPECTED_DATABASE_POOL_MAX:-10}"
TARGET_ZONE="$(jq -r '.zone' <<< "${CURRENT_SHAPE}")"
MIG_NAME="orca-cloud-relay-gce-${TARGET_HOSTNAME}"
if test "${DEPLOY_MODE}" = rollback; then
DESIRED_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
else
DESIRED_IMAGE_DIGEST="${TARGET_IMAGE_DIGEST}"
CURRENT_IMAGE_DIGEST="${ROLLBACK_IMAGE_DIGEST}"
DESIRED_REHOME_PROTOCOL="${TARGET_REHOME_PROTOCOL}"
CURRENT_REHOME_PROTOCOL="${ROLLBACK_REHOME_PROTOCOL}"
fi
# The startup template emits rehome trust lines only for a declared source cell, so
# require membership exactly when either side of this roll expects those lines.
if test "${EXPECTED_REGION}" = us-central1 && {
test "${DESIRED_REHOME_PROTOCOL}" != 0 || test "${CURRENT_REHOME_PROTOCOL}" != 0
}; then
jq -e --arg cell "${TARGET_CELL_ID}" 'index($cell) != null' \
<<< "${SOURCE_CELLS}" >/dev/null
fi
DESIRED_IMAGE="${IMAGE_REPOSITORY}@${DESIRED_IMAGE_DIGEST}"
OVERRIDE_CELLS_JSON="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${DESIRED_IMAGE}" '.[$cell].image = $image' <<< "${CELLS_JSON}")"
jq -n --argjson cells "${OVERRIDE_CELLS_JSON}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-same-cap.tfvars.json"
SERVED_DIGEST="$(gcloud artifacts docker images describe "${DESIRED_IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${SERVED_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"
{
echo "TARGET_HOSTNAME=${TARGET_HOSTNAME}"
echo "CELL_ORIGIN=${CELL_ORIGIN}"
echo "TARGET_ZONE=${TARGET_ZONE}"
echo "MIG_NAME=${MIG_NAME}"
echo "EXPECTED_HARD_CAP=${EXPECTED_HARD_CAP}"
echo "EXPECTED_UNOBSERVED_BOUND=${EXPECTED_UNOBSERVED_BOUND}"
echo "EXPECTED_DATABASE_POOL_MAX=${EXPECTED_DATABASE_POOL_MAX}"
echo "EXPECTED_REGION=${EXPECTED_REGION}"
echo "DESIRED_IMAGE=${DESIRED_IMAGE}"
echo "DESIRED_IMAGE_DIGEST=${DESIRED_IMAGE_DIGEST}"
echo "CURRENT_IMAGE_DIGEST=${CURRENT_IMAGE_DIGEST}"
echo "DESIRED_REHOME_PROTOCOL=${DESIRED_REHOME_PROTOCOL}"
echo "CURRENT_REHOME_PROTOCOL=${CURRENT_REHOME_PROTOCOL}"
} >> "${GITHUB_ENV}"
- name: Verify exact current generation, digest, cap, and rollback point
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
# A single transient 5xx (LB warm-up behind a fresh instance) must not
# fail a canary; 4xx (auth, generation mismatch) still fails fast.
admin_post() {
local out="${RUNNER_TEMP}/$1.json"
if ! curl --fail-with-body --max-time 30 \
--retry 3 --retry-delay 2 --retry-connrefused --output "${out}" \
--request POST "$2" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data "$3"; then
cat "${out}" >&2
return 1
fi
cat "${out}"
}
CURRENT_RUNTIME="$(admin_post current-runtime \
"${CELL_ORIGIN}/v1/admin/runtime-status" '{"v":1}')"
# Two different failures leave the cell on the rollback image, and the image
# alone cannot tell them apart. A rollback that failed between its template
# apply and its admission restore restarted the cell, so that cell is not
# draining and resumes. A wave that stopped after its drain and before its
# template apply never restarted anything, so its cell is still draining and
# is stranded: the drain flag only clears on a restart, so it has to be rolled.
LIVE_IMAGE_DIGEST="$(jq -r '.imageDigest' <<< "${CURRENT_RUNTIME}")"
LIVE_DRAINING="$(jq -r '.draining' <<< "${CURRENT_RUNTIME}")"
if test "${DEPLOY_MODE}" = rollback \
&& test "${LIVE_IMAGE_DIGEST}" = "${DESIRED_IMAGE_DIGEST}"; then
if test "${LIVE_DRAINING}" = true; then
ROLLBACK_STAGE=stranded
else
ROLLBACK_STAGE=resume
fi
PREDECESSOR_IMAGE_DIGEST="${DESIRED_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${DESIRED_REHOME_PROTOCOL}"
else
ROLLBACK_STAGE=roll
PREDECESSOR_IMAGE_DIGEST="${CURRENT_IMAGE_DIGEST}"
PREDECESSOR_REHOME_PROTOCOL="${CURRENT_REHOME_PROTOCOL}"
fi
if test "${ROLLBACK_STAGE}" = resume; then
ROLLBACK_RESUME=true
else
ROLLBACK_RESUME=false
fi
# A stranded cell's template still carries the image the cell is serving, so that
# is the predecessor its plan is reviewed against. A template already moved on to
# the target is refused here rather than rolled backwards under a stale review.
if test "${ROLLBACK_STAGE}" = stranded; then
PLAN_ROLLBACK_IMAGE="${DESIRED_IMAGE}"
else
PLAN_ROLLBACK_IMAGE="${IMAGE_REPOSITORY}@${CURRENT_IMAGE_DIGEST}"
fi
# Rollback is the documented recovery from a failed canary, which
# leaves the cell migration-only (and possibly still marked
# draining); apply and verify still require the cell pristine in the
# class it is declared to serve in.
if test "${DEPLOY_MODE}" = rollback; then
PRECHECK_ADMISSION=general-or-migration-only
else
PRECHECK_ADMISSION="${ENTRY_ADMISSION}"
fi
# Draining sheds connections, and a migration-only cell holds none, so the flag
# carries no precondition there. It also outlives a failed wave, because the drain
# that set it is followed by no restart, which is the state a failed canary leaves.
if test "${DEPLOY_MODE}" = rollback \
|| test "${ENTRY_ADMISSION}" = migration-only; then
PRECHECK_DRAINING=either
else
PRECHECK_DRAINING=forbidden
fi
# A resumed rollback already restarted, so its cell has to come back not draining;
# that is what separates it from a wave that stopped before its template apply.
if test "${PRECHECK_DRAINING}" = either && test "${ROLLBACK_RESUME}" != true; then
PREDECESSOR_DRAINING_OK=true
else
PREDECESSOR_DRAINING_OK=false
fi
RESTORED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
RESTORED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
test -n "${RESTORED_MIGRATION_CELLS}" || RESTORED_MIGRATION_CELLS=none
test -n "${RESTORED_GENERAL_CELLS}" || RESTORED_GENERAL_CELLS=none
ISOLATED_MIGRATION_CELLS="$(jq -rn \
--arg value "${EXPECTED_MIGRATION_ONLY_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0)) + [$target] | unique | join(",")')"
ISOLATED_GENERAL_CELLS="$(jq -rn \
--arg value "${EXPECTED_GENERAL_CELLS/none/}" \
--arg target "${TARGET_CELL_ID}" \
'$value | split(",") | map(select(length > 0 and . != $target)) | unique | join(",")')"
test -n "${ISOLATED_MIGRATION_CELLS}" || ISOLATED_MIGRATION_CELLS=none
test -n "${ISOLATED_GENERAL_CELLS}" || ISOLATED_GENERAL_CELLS=none
# A migration-only cell is already isolated and is handed back isolated, so both
# halves of its wave see exactly the membership it entered with.
if test "${ENTRY_ADMISSION}" = migration-only; then
RESTORED_MIGRATION_CELLS="${ISOLATED_MIGRATION_CELLS}"
RESTORED_GENERAL_CELLS="${ISOLATED_GENERAL_CELLS}"
fi
{
echo "ROLLBACK_RESUME=${ROLLBACK_RESUME}"
echo "ROLLBACK_STAGE=${ROLLBACK_STAGE}"
echo "PLAN_ROLLBACK_IMAGE=${PLAN_ROLLBACK_IMAGE}"
# The drain wait and the plan review both read the image this cell actually
# serves, which is the rollback image on a stranded cell and not the current one.
echo "PREDECESSOR_IMAGE_DIGEST=${PREDECESSOR_IMAGE_DIGEST}"
# The failsafe consumes these; deriving them here keeps them
# defined for a failure in any later step.
echo "ISOLATED_MIGRATION_CELLS=${ISOLATED_MIGRATION_CELLS}"
echo "ISOLATED_GENERAL_CELLS=${ISOLATED_GENERAL_CELLS}"
# No restart happens on resume, so isolate below is skipped and
# cannot advance the selector generation.
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${EFFECTIVE_SELECTOR_GENERATION}"
# A failed-canary rollback enters with the target migration-only,
# so the restore inspect cannot reuse the entry membership inputs.
echo "RESTORED_MIGRATION_CELLS=${RESTORED_MIGRATION_CELLS}"
echo "RESTORED_GENERAL_CELLS=${RESTORED_GENERAL_CELLS}"
} >> "${GITHUB_ENV}"
if ! jq -e --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "${PREDECESSOR_DRAINING_OK}" \
'.role == "cell" and .cellId == $cell and .cellUrl == $origin and
(.region == $region or
($region == "us-central1" and $protocol == 0 and .region == null)) and
.imageDigest == $digest and
.connectionCapacity.hardCap == $hardCap and
.connectionCapacity.unobservedBound == $unobservedBound and
(.draining == false or $drainingOk) and
(.regionalRehomeProtocol // 0) == $protocol' <<< "${CURRENT_RUNTIME}" >/dev/null
then
jq -r --arg cell "${TARGET_CELL_ID}" --arg origin "${CELL_ORIGIN}" \
--arg digest "${PREDECESSOR_IMAGE_DIGEST}" \
--arg region "${EXPECTED_REGION}" \
--argjson hardCap "${EXPECTED_HARD_CAP}" \
--argjson unobservedBound "${EXPECTED_UNOBSERVED_BOUND}" \
--argjson protocol "${PREDECESSOR_REHOME_PROTOCOL}" \
--argjson drainingOk "${PREDECESSOR_DRAINING_OK}" \
'[
if .role != "cell" then "role" else empty end,
if .cellId != $cell then "cellId" else empty end,
if .cellUrl != $origin then "cellUrl" else empty end,
if (.region != $region and
($region != "us-central1" or $protocol != 0 or .region != null))
then "region" else empty end,
if .imageDigest != $digest then "imageDigest" else empty end,
if .connectionCapacity.hardCap != $hardCap then "hardCap" else empty end,
if .connectionCapacity.unobservedBound != $unobservedBound then "unobservedBound" else empty end,
if (.draining != false and ($drainingOk | not)) then "draining" else empty end,
if (.regionalRehomeProtocol // 0) != $protocol then "regionalRehomeProtocol" else empty end
] | "runtime predecessor mismatch fields=" + join(",")' \
<<< "${CURRENT_RUNTIME}" >&2
exit 1
fi
# The exact legacy digest binds omitted pre-region fields to US and protocol 0.
jq -r '[
if .region == null then "region" else empty end,
if .regionalRehomeProtocol == null then "regionalRehomeProtocol" else empty end
] | if length > 0 then "runtime predecessor normalized legacy fields=" + join(",") else empty end' \
<<< "${CURRENT_RUNTIME}"
CURRENT_DIRECTOR_STATUS="$(admin_post current-cell-status \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
"$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
SOURCE_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${CURRENT_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true && ! jq -e \
'.status.admissionState == "migration-only"' \
<<< "${CURRENT_DIRECTOR_STATUS}" >/dev/null; then
echo 'resume requires the isolated migration-only cell a failed rollback leaves' >&2
exit 1
fi
[[ "${SOURCE_INCARNATION}" =~ ^[0-9a-f-]{36}$ ]]
echo "SOURCE_INCARNATION=${SOURCE_INCARNATION}" >> "${GITHUB_ENV}"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission "${PRECHECK_ADMISSION}" \
--draining "${PRECHECK_DRAINING}" --activity allowed \
--expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}"
# Hosts drained with nowhere to go keep redialling and pin the cell empty (c28, 2026-10-01).
# Every run that drains (apply, rollback) checks first; verify reads the same answer; a resume drains nothing.
- name: Require free general-cell slots for the selected cell's hosts
if: ${{ env.ROLLBACK_RESUME != 'true' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
HEADROOM_STATUS=0
HEADROOM="$(node dev/scripts/check-relay-same-cap-headroom.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --general-cells "${EXPECTED_GENERAL_CELLS}")" \
|| HEADROOM_STATUS=$?
echo "${HEADROOM}"
test "${HEADROOM_STATUS}" = 0
# The pre-drain sample sizes its window from this count.
TARGET_HOSTS="$(jq -er '.targetHosts' <<< "${HEADROOM}")"
echo "TARGET_HOSTS=${TARGET_HOSTS}" >> "${GITHUB_ENV}"
# The last read before isolation, so a roll drains on health measured minutes ago, not on a
# separate monitor run's verdict. Its window grows with the hosts the drain will re-place;
# every later wave samples again. Any trip fails here, before anything is changed. Rollback
# skips it: getting off a crash-looping image must not wait for the crashes to stop.
- name: Sample fleet health for a window sized to this drain
if: ${{ inputs.mode == 'apply' }}
timeout-minutes: 20
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.deploy-auth.outputs.id_token }}
run: |
pnpm incident:relay-pre-drain-sample -- \
--target-cell-id "${TARGET_CELL_ID}" \
--target-hosts "${TARGET_HOSTS}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--selector-membership-file "${RUNNER_TEMP}/relay-same-cap-selector.json" \
--wave-index "${WAVE_INDEX}" \
--selector-wave-delta "${SELECTOR_WAVE_DELTA}"
- name: Finish read-only verification
if: ${{ inputs.mode == 'verify' }}
run: echo 'Exact same-cap rollback point verified.'
# The sample can age the first one-hour token by up to 20 min; the drain's restart-safe
# wait (20 min at the default pace, 35 at the slowest) gets a fresh one.
- id: drain-auth
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Reversibly isolate and drain only the selected cell
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
id: drain
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.drain-auth.outputs.id_token }}
run: |
# Before any mutation: a cell whose image caps the pace below this one stops untouched.
node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode pace-check \
--pace-window-ms "${DRAIN_PACE_WINDOW_MS}"
# Opens the window the report-only shadow health gate below judges this cell over.
echo "drain-started-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# A cell isolated by a failed canary is already migration-only, so
# isolate is a no-op there that does not advance the selector; the
# result's generation is authoritative either way.
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode isolate)"
echo "${ISOLATE_RESULT}"
# Isolating a migration-only cell must be a read-only no-op; a change here would
# mean the live class is not the one this wave planned around.
if test "${ENTRY_ADMISSION}" = migration-only; then
jq -e '.changed == false' <<< "${ISOLATE_RESULT}" >/dev/null
fi
# The director re-places hosts off a cell only when the isolate stamped it, so a
# script too old to ask for the stamp produces today's behaviour and the canary
# reads as "the fix did nothing" with nothing to tell that from a wrong premise.
jq -e '.rollIsolated == true' <<< "${ISOLATE_RESULT}" >/dev/null
ISOLATE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
echo "SELECTOR_GENERATION_AFTER_ISOLATE=${ISOLATE_GENERATION}" >> "${GITHUB_ENV}"
DRAIN_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode drain \
--pace-window-ms "${DRAIN_PACE_WINDOW_MS}")"
echo "${DRAIN_RESULT}"
# What the cell accepted, which is 0 when an older image fell back to an unpaced drain.
echo "drain-applied-pace-window-ms=$(jq -er '.paceWindowMs' <<< "${DRAIN_RESULT}")" \
>> "${GITHUB_OUTPUT}"
# Restart-safe means an empty runtime that lasts past the drain's pace window; leases
# held by unplaceable hosts are reported, not waited on (c28, 2026-10-01).
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat either --admission migration-only --draining required \
--activity restart-safe --expected-image-digests "${PREDECESSOR_IMAGE_DIGEST}" \
--pace-window-ms "${DRAIN_PACE_WINDOW_MS}" \
--timeout-ms "$((900000 + DRAIN_PACE_WINDOW_MS))"
echo "drain-settled-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
- id: capacity-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- name: Require converged Terraform state and a stable MIG on resume
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME == 'true' }}
shell: bash
env:
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
# A cell on the root pool default emits no pool line, so pin one only where it exists.
POOL_ARGUMENTS=()
if test -n "${EXPECTED_DATABASE_POOL_MAX}"; then
POOL_ARGUMENTS=(--database-pool-max "${EXPECTED_DATABASE_POOL_MAX}")
fi
# Zero resource changes prove the prior run's apply completed and no
# restart will follow, keeping the incarnation check honest. Root
# outputs may lag a targeted apply, so judge resource_changes only.
# The cell backend service is deliberately not targeted: the capacity role has no
# compute.backendServices.update, so naming it fails the apply after the MIG has
# rolled. Its drain and log_config settings ship in a root apply by a principal
# that holds the permission.
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap-resume.tfplan"
if ! terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -e '[.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))]
| length == 0' >/dev/null
then
# An apply that failed before its template apply also resumes here
# (the cell still serves the rollback image), and repo drift since
# the cell's last roll (for example newly added rehome trust
# config) then legitimately replaces the template. Nothing is
# applied on resume either way, so accept exactly the drift the
# reviewed validator would let a real apply ship for the image the
# cell already serves: the template leaves and re-enters the
# rollback image, as exactly the template-and-MIG change pair.
terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| jq -r '"resume found unconverged resources: " +
([.resource_changes[]?
| select(.change.actions | any(. != "no-op" and . != "read"))
| .address] | join(","))'
echo 'requiring reviewed rollback-image drift'
RESUME_REVIEW="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap-resume.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}" \
--rollback-image "${DESIRED_IMAGE}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
"${POOL_ARGUMENTS[@]}")"
echo "${RESUME_REVIEW}"
jq -e '.changes == 2' <<< "${RESUME_REVIEW}" >/dev/null
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
- name: Apply only the selected same-cap template and MIG
if: ${{ inputs.mode != 'verify' && env.ROLLBACK_RESUME != 'true' }}
id: apply
shell: bash
env:
CAPACITY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
# A cell on the root pool default emits no pool line, so pin one only where it exists.
POOL_ARGUMENTS=()
if test -n "${EXPECTED_DATABASE_POOL_MAX}"; then
POOL_ARGUMENTS=(--database-pool-max "${EXPECTED_DATABASE_POOL_MAX}")
fi
terraform -chdir=infra/terraform plan \
-var-file=environments/production.tfvars \
-var-file="${RUNNER_TEMP}/relay-same-cap.tfvars.json" \
"-target=google_compute_instance_template.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
"-target=google_compute_instance_group_manager.relay_gce_cell[\"${TARGET_CELL_ID}\"]" \
-out="${RUNNER_TEMP}/relay-same-cap.tfplan"
PLAN_REVIEW="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-same-cap.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-cell --cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" --image "${DESIRED_IMAGE}" \
--rollback-image "${PLAN_ROLLBACK_IMAGE}" \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience https://relay.onorca.dev/v1/admin/host-drain \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" \
"${POOL_ARGUMENTS[@]}")"
echo "${PLAN_REVIEW}"
# Stamped before the apply, not after it: the new container announces its listener while
# the MIG is still converging, so a bound taken at the end of this step is already past
# the announcement the shadow gate looks for.
echo "apply-started-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-same-cap.tfplan"
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
# A stranded cell already runs the reviewed template, so the apply above replaces
# no instance and the drain flag, which only a restart clears, would survive the
# whole wave. Fewer than the template-and-MIG pair means the template stayed put,
# including a MIG-only reconciliation. Recreate its one instance from the MIG's
# current template; a rolling action would rewrite the MIG's version name outside
# Terraform, and every later targeted plan would carry that revert.
if test "${ROLLBACK_STAGE}" = stranded \
&& jq -e '.changes < 2' <<< "${PLAN_REVIEW}" >/dev/null; then
STRANDED_INSTANCE="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" --project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" \
--format=json \
| jq -er 'if length == 1 then .[0].instance | split("/") | last
else error("stranded cell MIG does not have exactly one instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${STRANDED_INSTANCE}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
fi
# Recorded for the operator comparing verdicts; the gate's boot search opens at the
# apply-started-at stamp above, not here.
echo "apply-completed-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
- id: post-auth
if: ${{ inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new incarnation, exact image, protocol, and durable safety
if: ${{ inputs.mode != 'verify' }}
id: verify-target
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
# Rides out one DB stall: 5-7 s, then readiness grace and two 10 s LB
# health checks (c18, 10-07 18:49Z: 503 "no healthy upstream", healthy
# 16 s later). A cell still down after 60 s fails; other 4xx fail fast.
# Retried in this loop: the runner's 7.81 client (Ubuntu 22.04) never
# retries under --fail-with-body, so its --retry flags were inert.
admin_post() {
local out="${RUNNER_TEMP}/$1.json" window=60 delay=5 status
local deadline=$((SECONDS + window))
while true; do
# A refused or timed-out attempt writes no body; never report the last one's.
rm -f "${out}"
status="$(curl --silent --show-error --max-time 30 --output "${out}" \
--write-out '%{http_code}' --request POST "$2" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data "$3")" || status=000
case "${status}" in
2??) cat "${out}"; return 0 ;;
000|408|429|5??) ((SECONDS + delay < deadline)) || break ;;
*) break ;;
esac
sleep "${delay}"
done
echo "admin_post $1: HTTP ${status}" >&2
cat "${out}" >&2 2>/dev/null || true
return 1
}
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity allowed --expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}" --timeout-ms 900000
TARGET_RUNTIME="$(admin_post target-runtime \
"${CELL_ORIGIN}/v1/admin/runtime-status" '{"v":1}')"
jq -e --arg digest "${DESIRED_IMAGE_DIGEST}" \
--argjson protocol "${DESIRED_REHOME_PROTOCOL}" \
'.imageDigest == $digest and (.regionalRehomeProtocol // 0) == $protocol' \
<<< "${TARGET_RUNTIME}" >/dev/null
TARGET_DIRECTOR_STATUS="$(admin_post target-cell-status \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
"$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
TARGET_INCARNATION="$(jq -er '.status.runtime.cellIncarnation' \
<<< "${TARGET_DIRECTOR_STATUS}")"
if test "${ROLLBACK_RESUME}" = true; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# No restart happened; the incarnation legitimately stays put.
test "${TARGET_INCARNATION}" = "${SOURCE_INCARNATION}"
else
test "${TARGET_INCARNATION}" != "${SOURCE_INCARNATION}"
fi
echo "TARGET_INCARNATION=${TARGET_INCARNATION}" >> "${GITHUB_ENV}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_ISOLATE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
echo "verify-ended-at=$(date -u +%FT%TZ)" >> "${GITHUB_OUTPUT}"
- name: Prove exact per-host trust and idempotent no-neighbor behavior
if: ${{ inputs.mode != 'verify' && ((inputs.mode == 'rollback' && inputs.rollback-rehome-protocol != '0') || (inputs.mode != 'rollback' && inputs.target-rehome-protocol != '0')) }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
node dev/scripts/probe-relay-rehome-trust.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-id "${TARGET_CELL_ID}" \
--cell-incarnation "${TARGET_INCARNATION}"
- name: Restore only the verified selected cell to its entry admission
if: ${{ inputs.mode != 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
run: |
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
# Activating a migration-only cell would promote it to a serving cell for good, so
# restore it with the idempotent isolate that reports the authoritative generation.
if test "${ENTRY_ADMISSION}" = migration-only; then
RESTORE_MODE=isolate
else
RESTORE_MODE=activate
fi
RESTORE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode "${RESTORE_MODE}")"
echo "${RESTORE_RESULT}"
SELECTOR_GENERATION_AFTER_RESTORE="$(jq -er '.generation' \
<<< "${RESTORE_RESULT}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh --admission "${ENTRY_ADMISSION}" \
--draining forbidden --activity allowed \
--expected-image-digests "${DESIRED_IMAGE_DIGEST}" \
--regional-rehome-protocol "${DESIRED_REHOME_PROTOCOL}"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${SELECTOR_GENERATION_AFTER_RESTORE}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${RESTORED_MIGRATION_CELLS}" \
--expected-general-cells "${RESTORED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}" \
| jq -e '.control.enabled == false' >/dev/null
# Report only: this evaluates the oracles an operator reads by hand after a cell and records
# what it would have called, so its verdicts can be compared with the operator's over a full
# roll before it is ever allowed to block. Two independent guarantees keep it inert: the
# script exits 0 on every verdict, and continue-on-error keeps even a crash off the job's
# outcome. The failsafe below therefore cannot fire on anything this step observes.
#
# It runs after the restore, not before it, for two reasons: the cell goes back into
# admission on exactly today's schedule rather than waiting out a minute of log reads, and
# the window it judges has closed by then, so Cloud Logging's ingestion lag is behind it.
# These are fleet oracles anyway; when this does gate, what it gates is the next cell.
- name: Shadow health gate (report only)
id: shadow-gate
if: ${{ inputs.mode != 'verify' }}
continue-on-error: true
# continue-on-error bounds this step's contribution to the job outcome, not its clock,
# and its reads are serialised. A timed-out step is a failed step, which continue-on-error
# absorbs; without this bound a Logging 429 storm or an expired credential makes every
# read cost its full retry budget and can push the job past timeout-minutes, and a
# cancelled job takes the whole wave with it. A healthy gate is already minutes of
# serial reads, so both bounds sit above that: the script settles at seven minutes and
# reaching this eight is the pathological case. Eight on top of a cell of ~14 minutes
# (~45 at the slowest pace) still leaves the job's 120 minutes intact.
timeout-minutes: 8
env:
DRAIN_STARTED_AT: ${{ steps.drain.outputs.drain-started-at }}
DRAIN_SETTLED_AT: ${{ steps.drain.outputs.drain-settled-at }}
DRAIN_APPLIED_PACE_WINDOW_MS: ${{ steps.drain.outputs.drain-applied-pace-window-ms }}
APPLY_STARTED_AT: ${{ steps.apply.outputs.apply-started-at }}
APPLY_COMPLETED_AT: ${{ steps.apply.outputs.apply-completed-at }}
VERIFY_ENDED_AT: ${{ steps.verify-target.outputs.verify-ended-at }}
SHADOW_GATE_DIRECTORY: ${{ runner.temp }}/relay-same-cap-shadow-gate
SHADOW_GATE_NAME: relay-same-cap-shadow-gate-${{ inputs.target-cell-id }}-${{ github.run_id }}.json
run: |
mkdir -p "${SHADOW_GATE_DIRECTORY}"
node dev/scripts/relay-same-cap-shadow-gate.mjs \
--cell-id "${TARGET_CELL_ID}" \
--cell-host "${TARGET_HOSTNAME}.relay.onorca.dev" \
--project-id "${GCP_PROJECT_ID}" \
--director-service orca-cloud-relay \
--drain-started-at "${DRAIN_STARTED_AT}" \
--apply-started-at "${APPLY_STARTED_AT}" \
--apply-completed-at "${APPLY_COMPLETED_AT}" \
--verify-ended-at "${VERIFY_ENDED_AT}" \
--drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}" \
--drain-applied-pace-window-ms "${DRAIN_APPLIED_PACE_WINDOW_MS}" \
--drain-settled-at "${DRAIN_SETTLED_AT}" \
--target-hosts "${TARGET_HOSTS:-}" \
--summary-file "${GITHUB_STEP_SUMMARY}" \
--output-file "${SHADOW_GATE_DIRECTORY}/${SHADOW_GATE_NAME}"
- name: Publish the shadow health gate verdict
if: ${{ inputs.mode != 'verify' }}
continue-on-error: true
# One small JSON file; a retrying upload must not spend the wave's remaining minutes either.
timeout-minutes: 2
uses: actions/upload-artifact@v4
with:
name: relay-same-cap-shadow-gate-${{ inputs.target-cell-id }}-${{ github.run_id }}.json
path: ${{ runner.temp }}/relay-same-cap-shadow-gate
if-no-files-found: warn
retention-days: 14
overwrite: true
- id: cleanup-auth
if: ${{ failure() && inputs.mode != 'verify' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Keep a failed cell isolated and rehome disabled
if: ${{ failure() && inputs.mode != 'verify' }}
continue-on-error: true
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.cleanup-auth.outputs.id_token }}
run: |
test "${MUTATION_STARTED:-false}" = true || exit 0
ISOLATE_RESULT="$(node dev/scripts/prepare-relay-production-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --approved-cells same-cap --mode isolate)"
echo "${ISOLATE_RESULT}"
# Same reason as the isolate step above: a failed wave leaves this cell isolated
# deliberately, and the stamp is what lets its hosts leave.
jq -e '.rollIsolated == true' <<< "${ISOLATE_RESULT}" >/dev/null
# The isolate result carries the authoritative post-isolate generation;
# fixed offsets are wrong whenever an earlier isolate was a no-op.
FAILSAFE_GENERATION="$(jq -er '.generation' <<< "${ISOLATE_RESULT}")"
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${FAILSAFE_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${ISOLATED_MIGRATION_CELLS}" \
--expected-general-cells "${ISOLATED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_REHOME_GENERATION}"
@@ -1,433 +0,0 @@
name: Deploy Relay Production Same-Cap
on:
workflow_dispatch:
inputs:
mode:
description: Verify, roll one canary, roll a bounded batch, or roll back
required: true
default: verify
type: choice
options: [verify, canary-apply, batch-apply, rollback]
cell-ids:
description: Ordered comma-separated serving cells; one canary or two to ten batch cells
required: true
type: string
target-image-digest:
description: Exact immutable compatibility image digest
required: true
type: string
rollback-image-digest:
description: Exact immutable currently serving rollback digest
required: true
type: string
target-rehome-protocol:
description: Exact target regional-rehome protocol
required: true
default: '1'
type: choice
options: ['0', '1', '3']
rollback-rehome-protocol:
description: Exact rollback regional-rehome protocol
required: true
default: '0'
type: choice
options: ['0', '1', '3']
expected-selector-generation:
description: Exact selector generation before the first cell
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-rehome-generation:
description: Exact durable regional-rehome control generation; it must be disabled
required: true
type: string
drain-pace-window-ms:
description: Window the cell spreads its drain over; below 300000 only for US general cells; any other value named in the confirmation
required: true
default: '300000'
type: choice
options: ['300000', '60000', '30000', '900000', '1200000']
canary-run-id:
description: Successful same-code canary in this rehome control generation; reusable across batches
required: false
type: string
confirmation:
description: Exact digest-and-cell-bound mutation confirmation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
# Headroom for the full-history checkout the canary provenance check needs.
timeout-minutes: 15
environment: production
outputs:
cells: ${{ steps.wave.outputs.cells }}
job-mode: ${{ steps.wave.outputs.job-mode }}
steps:
# Full history: the canary authority a batch verifies is sealed at an ancestor commit, and
# the provenance check fails closed on a commit a shallow clone left out.
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with: { node-version: 24 }
- id: wave
env:
MODE: ${{ inputs.mode }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
CONFIRMATION: ${{ inputs.confirmation }}
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
run: |
CELLS="$(node dev/scripts/relay-production-same-cap-wave.mjs validate \
--mode "${MODE}" --cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" --rollback-digest "${ROLLBACK_DIGEST}" \
--confirmation "${CONFIRMATION}" --canary-run-id "${CANARY_RUN_ID}" \
--drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}")"
echo "cells=${CELLS}" >> "${GITHUB_OUTPUT}"
if [[ "${MODE}" =~ ^(canary-apply|batch-apply)$ ]]; then
echo 'job-mode=apply' >> "${GITHUB_OUTPUT}"
else
echo "job-mode=${MODE}" >> "${GITHUB_OUTPUT}"
fi
- name: Download exact prior canary authority
if: ${{ inputs.mode == 'batch-apply' }}
uses: actions/download-artifact@v4
with:
name: relay-same-cap-canary-${{ inputs.canary-run-id }}
path: ${{ runner.temp }}/relay-same-cap-canary
github-token: ${{ github.token }}
run-id: ${{ inputs.canary-run-id }}
- name: Verify canary authority against this batch
if: ${{ inputs.mode == 'batch-apply' }}
env:
CANARY_RUN_ID: ${{ inputs.canary-run-id }}
CELL_IDS: ${{ inputs.cell-ids }}
TARGET_DIGEST: ${{ inputs.target-image-digest }}
ROLLBACK_DIGEST: ${{ inputs.rollback-image-digest }}
SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
DRAIN_PACE_WINDOW_MS: ${{ inputs.drain-pace-window-ms }}
run: |
node dev/scripts/relay-production-same-cap-wave.mjs verify-canary \
--file "${RUNNER_TEMP}/relay-same-cap-canary/authority.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${CANARY_RUN_ID}" \
--cell-ids "${CELL_IDS}" \
--target-digest "${TARGET_DIGEST}" \
--rollback-digest "${ROLLBACK_DIGEST}" \
--drain-pace-window-ms "${DRAIN_PACE_WINDOW_MS}" \
--selector-generation "${SELECTOR_GENERATION}" \
--rehome-generation "${REHOME_GENERATION}"
cell_1:
needs: gate
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[0] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '0'
secrets: inherit
cell_2:
if: ${{ needs.cell_1.result == 'success' && fromJSON(needs.gate.outputs.cells)[1] != null }}
needs: [gate, cell_1]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[1] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '1'
secrets: inherit
cell_3:
if: ${{ needs.cell_2.result == 'success' && fromJSON(needs.gate.outputs.cells)[2] != null }}
needs: [gate, cell_2]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[2] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '2'
secrets: inherit
cell_4:
if: ${{ needs.cell_3.result == 'success' && fromJSON(needs.gate.outputs.cells)[3] != null }}
needs: [gate, cell_3]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[3] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '3'
secrets: inherit
cell_5:
if: ${{ needs.cell_4.result == 'success' && fromJSON(needs.gate.outputs.cells)[4] != null }}
needs: [gate, cell_4]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[4] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '4'
secrets: inherit
cell_6:
if: ${{ needs.cell_5.result == 'success' && fromJSON(needs.gate.outputs.cells)[5] != null }}
needs: [gate, cell_5]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[5] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '5'
secrets: inherit
cell_7:
if: ${{ needs.cell_6.result == 'success' && fromJSON(needs.gate.outputs.cells)[6] != null }}
needs: [gate, cell_6]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[6] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '6'
secrets: inherit
cell_8:
if: ${{ needs.cell_7.result == 'success' && fromJSON(needs.gate.outputs.cells)[7] != null }}
needs: [gate, cell_7]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[7] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '7'
secrets: inherit
cell_9:
if: ${{ needs.cell_8.result == 'success' && fromJSON(needs.gate.outputs.cells)[8] != null }}
needs: [gate, cell_8]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[8] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '8'
secrets: inherit
cell_10:
if: ${{ needs.cell_9.result == 'success' && fromJSON(needs.gate.outputs.cells)[9] != null }}
needs: [gate, cell_9]
uses: ./.github/workflows/cloud-deploy-relay-production-same-cap-job.yml
with:
mode: ${{ needs.gate.outputs.job-mode }}
target-cell-id: ${{ fromJSON(needs.gate.outputs.cells)[9] }}
target-image-digest: ${{ inputs.target-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
target-rehome-protocol: ${{ inputs.target-rehome-protocol }}
rollback-rehome-protocol: ${{ inputs.rollback-rehome-protocol }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-rehome-generation: ${{ inputs.expected-rehome-generation }}
drain-pace-window-ms: ${{ inputs.drain-pace-window-ms }}
wave-index: '9'
secrets: inherit
seal_canary:
if: ${{ inputs.mode == 'canary-apply' }}
needs: [gate, cell_1]
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 24 }
# Its pace checks are what let a batch drain faster than the default; a missing artifact
# seals UNVERIFIED, which only the default accepts. Named from the gate's normalized cell,
# as cell_1 uploaded it, so a padded cell-ids input cannot miss it.
- name: Download the canary cell's shadow health verdict
continue-on-error: true
uses: actions/download-artifact@v4
with:
name: relay-same-cap-shadow-gate-${{ fromJSON(needs.gate.outputs.cells)[0] }}-${{ github.run_id }}.json
path: ${{ runner.temp }}/relay-same-cap-shadow-gate
- name: Seal exact successful canary authority
run: |
mkdir -p "${RUNNER_TEMP}/relay-same-cap-canary"
node dev/scripts/relay-production-same-cap-wave.mjs create-canary \
--cell-id "${{ fromJSON(needs.gate.outputs.cells)[0] }}" \
--target-digest "${{ inputs.target-image-digest }}" \
--rollback-digest "${{ inputs.rollback-image-digest }}" \
--confirmation "${{ inputs.confirmation }}" \
--drain-pace-window-ms "${{ inputs.drain-pace-window-ms }}" \
--shadow-report "${RUNNER_TEMP}/relay-same-cap-shadow-gate/relay-same-cap-shadow-gate-${{ fromJSON(needs.gate.outputs.cells)[0] }}-${GITHUB_RUN_ID}.json" \
--commit-sha "${GITHUB_SHA}" --run-id "${GITHUB_RUN_ID}" \
--selector-generation "${{ inputs.expected-selector-generation }}" \
--rehome-generation "${{ inputs.expected-rehome-generation }}" \
> "${RUNNER_TEMP}/relay-same-cap-canary/authority.json"
- uses: actions/upload-artifact@v4
with:
name: relay-same-cap-canary-${{ github.run_id }}
path: ${{ runner.temp }}/relay-same-cap-canary/authority.json
retention-days: 30
if-no-files-found: error
# Every cell job re-enters the run's lease with release: 'false'; only this job frees it.
release_lease:
if: always()
needs:
- gate
- cell_1
- cell_2
- cell_3
- cell_4
- cell_5
- cell_6
- cell_7
- cell_8
- cell_9
- cell_10
- seal_canary
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 10
environment: production
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
release: 'true'
@@ -1,219 +0,0 @@
name: Deploy Relay Production Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Audit/preflight are read-only; recover/continue resume committed work; disable/enable/reset/execute mutate admission
required: true
default: preflight
type: choice
options:
- audit
- preflight
- recover-forward
- continue-evacuation
- disable-cell
- enable-empty-cell
- reset-empty-candidate
- execute
confirmation:
description: Enter RECOVER_FORWARD, CONTINUE_EVACUATION, DISABLE_CELL, ENABLE_CELL, RESET_CANDIDATE, or EVACUATE for the matching mutation
required: false
type: string
monitor-run-id:
description: Successful fresh dry-run monitor workflow run ID
required: false
type: string
monitor-run-attempt:
description: Exact dry-run monitor workflow attempt
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
DIRECTOR_ORIGIN: https://relay.onorca.dev
ADMIN_AUDIENCE: https://relay.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
steps:
- uses: actions/checkout@v4
- name: Require fresh dry-run evidence reference
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[0-9]+$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
- name: Download private dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- name: Verify dry-run artifact before cloud authentication
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run
- name: Reject previously consumed dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api \
"/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = "0"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
elif [[ "${DEPLOY_MODE}" = "recover-forward" ]]; then
test "${CONFIRMATION}" = "RECOVER_FORWARD"
elif [[ "${DEPLOY_MODE}" = "continue-evacuation" ]]; then
test "${CONFIRMATION}" = "CONTINUE_EVACUATION"
elif [[ "${DEPLOY_MODE}" = "disable-cell" ]]; then
test "${CONFIRMATION}" = "DISABLE_CELL"
elif [[ "${DEPLOY_MODE}" = "enable-empty-cell" ]]; then
test "${CONFIRMATION}" = "ENABLE_CELL"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env production
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Verify fresh dry-run evidence against live selector
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-mutation \
--directory "${RUNNER_TEMP}/relay-monitor-evidence" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" \
--run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode dry-run \
--mutation-mode "${DEPLOY_MODE}" \
--source-cell-id "${SOURCE_CELL_ID}" \
--director-origin "${DIRECTOR_ORIGIN}"
- name: Recheck all live safety signals
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${RUNNER_TEMP}/relay-monitor-evidence/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Create single-use dry-run marker
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
run: |
MARKER_NAME="relay-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-monitor-consumption"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-monitor-consumption/${MARKER_NAME}"
- name: Consume dry-run evidence
if: ${{ inputs.mode != 'audit' && inputs.mode != 'preflight' }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-monitor-consumption/relay-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -1,111 +0,0 @@
name: Deploy Relay Staging GCE Candidate
on:
workflow_dispatch:
inputs:
source-cell-id:
description: Existing Terraform cell ID to evacuate
required: true
type: string
target-cell-id:
description: Distinct Terraform candidate cell ID
required: true
type: string
mode:
description: Preflight is read-only; reset repairs an empty candidate; execute evacuates
required: true
default: preflight
type: choice
options:
- preflight
- reset-empty-candidate
- execute
confirmation:
description: Enter RESET_CANDIDATE for reset or EVACUATE for execute
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
candidate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
ADMIN_AUDIENCE: https://relay-staging.onorca.dev/v1/admin/drain
SOURCE_CELL_ID: ${{ inputs.source-cell-id }}
TARGET_CELL_ID: ${{ inputs.target-cell-id }}
DEPLOY_MODE: ${{ inputs.mode }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit mutation confirmation
if: ${{ inputs.mode != 'preflight' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${DEPLOY_MODE}" = "execute" ]]; then
test "${CONFIRMATION}" = "EVACUATE"
else
test "${CONFIRMATION}" = "RESET_CANDIDATE"
fi
- name: Read reviewed Terraform topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
RUNTIME_SERVICE_ACCOUNT="$(terraform -chdir=infra/terraform output -raw relay_runtime_service_account)"
echo "RUNTIME_SERVICE_ACCOUNT=${RUNTIME_SERVICE_ACCOUNT}" >> "${GITHUB_ENV}"
- name: Preflight or evacuate exact GCE candidate
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/deploy-relay-gce-candidate.mjs \
--project "${GCP_PROJECT_ID}" \
--director-origin "${DIRECTOR_ORIGIN}" \
--admin-audience "${ADMIN_AUDIENCE}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json" \
--source-cell-id "${SOURCE_CELL_ID}" \
--target-cell-id "${TARGET_CELL_ID}" \
--runtime-service-account "${RUNTIME_SERVICE_ACCOUNT}" \
--mode "${DEPLOY_MODE}"
@@ -1,241 +0,0 @@
name: Deploy Relay Staging
on:
workflow_dispatch:
inputs:
expected-image-digest:
description: Exact checked-in production Relay sha256 digest to deploy
required: true
type: string
reserve-placement:
description: Preserve step-5 reserve placement, or set it; turning it off is refused while any cell is reserve
required: true
default: preserve
type: choice
options: [preserve, 'off', dry-run, 'on']
expected-rehome-generation:
description: Exact durable regional-rehome generation; it must remain disabled
required: true
type: string
bootstrap-runtime-identity:
description: One-time move from the stamped-cell identity to the director identity
required: true
default: false
type: boolean
predecessor-image-digest:
description: Bootstrap only. Exact immutable serving predecessor digest
required: false
type: string
capacity-cell-id:
description: Register this cell's reviewed staging.tfvars cap and bound with the director
required: true
default: none
type: choice
options: [none, staging-gce-c1, staging-gce-c2, staging-gce-c3, staging-gce-c4]
min-instances:
description: Director floor; preserve keeps the serving revision's, so a reserve run's 2 survives
required: true
default: preserve
type: choice
options: [preserve, '0', '2']
confirmation:
description: BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY for a bootstrap, ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE with a shape baseline
required: false
type: string
shape-baseline-revision:
description: Only after a Terraform apply changed the director template. The apply's 0% revision, whose shape the deploy adopts
required: false
type: string
shadow-seat-feed-cells:
description: Preserve the step-3 shadow seat-feed cells, or set all or a comma list of cell ids; dropping a reserve cell is refused
required: true
default: preserve
type: string
permissions:
contents: read
id-token: write
concurrency:
# Staging deploy, candidate, auth, and power operations must never overlap.
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
EXPECTED_IMAGE_DIGEST: ${{ inputs.expected-image-digest }}
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
# The director must trust the identity every staging Relay workflow authenticates as.
DEPLOY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
ASIA_PROOF_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: orca-cloud-staging-relay-dir@onorca-cloud-staging.iam.gserviceaccount.com
PREDECESSOR_RUNTIME_SERVICE_ACCOUNT: orca-cloud-staging-relay@onorca-cloud-staging.iam.gserviceaccount.com
REHOME_AUDIENCE: https://relay-staging.onorca.dev/v1/admin/host-drain
EXPECTED_REHOME_GENERATION: ${{ inputs.expected-rehome-generation }}
BOOTSTRAP_RUNTIME_IDENTITY: ${{ inputs.bootstrap-runtime-identity }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }}
MIN_INSTANCES: ${{ inputs.min-instances }}
SHAPE_BASELINE_REVISION: ${{ inputs.shape-baseline-revision }}
steps:
- uses: actions/checkout@v4
- name: Require the expected immutable image and identity request
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
[[ "${EXPECTED_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${EXPECTED_REHOME_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${BOOTSTRAP_RUNTIME_IDENTITY}" =~ ^(true|false)$ ]]
[[ "${MIN_INSTANCES}" =~ ^(preserve|0|2)$ ]]
if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${CAPACITY_CELL_ID}" = none
test "${CONFIRMATION}" = BOOTSTRAP_RELAY_DIRECTOR_REHOME_IDENTITY
else
test -z "${PREDECESSOR_IMAGE_DIGEST}"
fi
if test -n "${SHAPE_BASELINE_REVISION}"; then
[[ "${SHAPE_BASELINE_REVISION}" =~ ^orca-cloud-relay-staging-[0-9]{5}-[a-z0-9]{3}$ ]]
test "${BOOTSTRAP_RUNTIME_IDENTITY}" = false
test "${CAPACITY_CELL_ID}" = none
test "${CONFIRMATION}" = ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE
fi
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Bind the request to the checked-in staging C4 image
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure \
-backend-config=backend/staging.hcl -input=false
IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'var.relay_gce_cells["staging-gce-c4"].image' | jq -er '.')"
test "${IMAGE}" = \
"${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${EXPECTED_IMAGE_DIGEST}"
echo "IMAGE=${IMAGE}" >> "${GITHUB_ENV}"
if test "${CAPACITY_CELL_ID}" != none; then
# The deploy script refuses any topology change beyond this one cell's cap and bound.
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -er '.')"
jq -e --arg cell "${CAPACITY_CELL_ID}" \
'any(.[]; .id == $cell and .connectionHardCap != null)' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" >> "${GITHUB_ENV}"
fi
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require the mirrored immutable image
run: |
DIGEST="$(gcloud artifacts docker images describe "${IMAGE}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${DIGEST}" = "${EXPECTED_IMAGE_DIGEST}"
- name: Deploy director blue/green
env:
# An image change reads every cell's admit mode before the traffic move.
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
RESERVE_PLACEMENT: ${{ inputs.reserve-placement }}
SHADOW_SEAT_FEED_CELLS: ${{ inputs.shadow-seat-feed-cells }}
run: |
regional_version="$(gcloud secrets versions describe latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}" \
--format='value(name)' | awk -F/ '{print $NF}')"
[[ "${regional_version}" =~ ^[1-9][0-9]*$ ]]
RELEASE_ID="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
identity_args=()
if test "${BOOTSTRAP_RUNTIME_IDENTITY}" = true; then
identity_args=(
--bootstrap-runtime-identity true
--predecessor-runtime-service-account "${PREDECESSOR_RUNTIME_SERVICE_ACCOUNT}"
--predecessor-image-digest "${PREDECESSOR_IMAGE_DIGEST}"
)
fi
# Omitted, the deploy script inherits the serving revision's floor.
floor_args=()
if test "${MIN_INSTANCES}" != preserve; then
floor_args=(--min-instances "${MIN_INSTANCES}")
fi
shape_args=()
if test -n "${SHAPE_BASELINE_REVISION}"; then
shape_args=(--shape-baseline-revision "${SHAPE_BASELINE_REVISION}")
fi
topology_args=()
if test "${CAPACITY_CELL_ID}" != none; then
topology_args=(
--capacity-cell-id "${CAPACITY_CELL_ID}"
--director-cells-json "${DESIRED_CELLS_JSON}"
)
fi
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${IMAGE}" \
--role director \
--max-instances 2 \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--deploy-service-account "${DEPLOY_SERVICE_ACCOUNT}" \
--asia-proof-service-account "${ASIA_PROOF_SERVICE_ACCOUNT}" \
--regional-placement-secret-version "${regional_version}" \
"${floor_args[@]}" \
--runtime-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
"${identity_args[@]}" \
"${shape_args[@]}" \
--rehome-director-service-account "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}" \
--rehome-audience "${REHOME_AUDIENCE}" \
--rehome-control-origin https://relay-staging.onorca.dev \
--expected-rehome-generation "${EXPECTED_REHOME_GENERATION}" \
"${topology_args[@]}" \
--admin-audience https://relay-staging.onorca.dev/v1/admin/drain \
--reserve-placement "${RESERVE_PLACEMENT}" \
--shadow-seat-feed-cells "${SHADOW_SEAT_FEED_CELLS}" \
--release-id "${RELEASE_ID}"
- name: Smoke director health
run: |
URL="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" --project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(status.url)')"
node dev/scripts/smoke-relay.mjs "${URL}"
@@ -1,90 +0,0 @@
name: Monitor Relay Cell Clock Skew
# Why: the 2026-08-01 sustained HOST_OFFLINE incident traced to one cell's
# clock running ~100ms ahead, which deterministically failed every host
# challenge under a zero-tolerance freshness check. /health and /ready cannot
# see clock skew; this monitor alarms before drift reaches the (now 2s)
# client tolerance.
on:
workflow_dispatch:
schedule:
- cron: '17 * * * *'
permissions:
contents: read
defaults:
run:
working-directory: cloud
jobs:
skew:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
sparse-checkout: cloud/infra/terraform/environments/production.tfvars
sparse-checkout-cone-mode: false
persist-credentials: false
- name: Measure Date-header skew for every relay cell
run: |
set -u
# Date headers carry whole seconds; compare floored seconds on both
# sides so perfect sync reads 0/±1 and never flaps. An absolute
# floor-skew of >= 2 means real drift of at least ~1s — approaching
# the client's 2s challenge tolerance. Millisecond-precision deltas
# come from the desktop's named-check logging when activations fail.
ALARM_S=2
failures=0
reachable=0
unserved=0
# Every relay_gce_cells hostname (the block relayOpsCellsFromTerraform reads), fenced
# cells included, so a newly declared cell is probed from its first run.
cells="$(awk '
/^relay_gce_cells[ \t]*=[ \t]*\{/ { inside = 1; next }
inside && /^\}/ { exit }
inside && /^[ \t]*hostname[ \t]*=/ { split($0, quoted, "\""); print quoted[2] }
' infra/terraform/environments/production.tfvars || true)"
if [ -z "${cells}" ]; then
echo "::error::read no relay cells from production.tfvars; monitor blind"
exit 1
fi
for cell in ${cells}; do
url="https://${cell}.relay.onorca.dev/health"
# Why: *.relay.onorca.dev is a wildcard, so the load balancer answers with its own
# accurate Date for a fenced, dead, or never-provisioned cell. Timing that reads as
# perfect sync. Only an HTTP 200 is the relay process itself answering, so only a
# 200 carries a clock worth judging.
response="$(curl -sS -D - -o /dev/null --max-time 8 "${url}" 2>/dev/null || true)"
status="$(printf '%s' "${response}" | awk 'NR==1 {print $2}')"
header="$(printf '%s' "${response}" | tr -d '\r' | grep -i '^date:' || true)"
if [ "${status:-000}" != "200" ] || [ -z "${header}" ]; then
# Expected for the fenced cells; a dead unfenced cell is caught by the heartbeat
# and readiness alerts, not here. Named either way so it is never invisible.
echo "${cell}: not serving (status ${status:-none}); no relay clock to judge"
unserved=$((unserved + 1))
continue
fi
reachable=$((reachable + 1))
server_s="$(date -d "${header#*: }" +%s)"
local_s="$(date +%s)"
skew=$((server_s - local_s))
abs=${skew#-}
if [ "${abs}" -ge "${ALARM_S}" ]; then
echo "::error::${cell}: clock skew ${skew}s reaches ±${ALARM_S}s alarm"
failures=$((failures + 1))
else
echo "${cell}: skew ${skew}s"
fi
done
echo "cells serving: ${reachable}, not serving: ${unserved}, alarms: ${failures}"
if [ "${reachable}" -eq 0 ]; then
# Now meaningful: previously the load balancer answered for every name, so this
# could never fire and a total fleet outage reported "all healthy".
echo "::error::no relay cell is serving; monitor blind"
exit 1
fi
exit "$((failures > 0 ? 1 : 0))"
@@ -1,215 +0,0 @@
name: Monitor Relay Production Job
on:
workflow_call:
inputs:
mode:
required: true
type: string
expected-selector-generation:
required: true
type: string
expected-existing-only-cells:
required: true
type: string
expected-migration-only-cells:
required: true
type: string
expected-general-cells:
required: true
type: string
migration-policy:
required: true
type: string
recovery-source-cell-id:
required: true
type: string
capacity-cell-id:
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 100
environment: production
env:
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
MIGRATION_POLICY: ${{ inputs.migration-policy }}
RECOVERY_SOURCE_CELL_ID: ${{ inputs.recovery-source-cell-id }}
CAPACITY_CELL_ID: ${{ inputs.capacity-cell-id }}
INCIDENT_ID: relay-${{ github.run_id }}-${{ inputs.mode }}
MONITOR_MODE: ${{ inputs.mode }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-incident
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
- run: pnpm install --frozen-lockfile
- id: prior-attempt
if: ${{ github.run_attempt > 1 }}
run: echo "value=$((GITHUB_RUN_ATTEMPT - 1))" >> "${GITHUB_OUTPUT}"
- name: Restore prior private monitor state
if: ${{ github.run_attempt > 1 }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ steps.prior-attempt.outputs.value }}
path: ${{ github.workspace }}/relay-incident
github-token: ${{ github.token }}
run-id: ${{ github.run_id }}
- name: Verify restored state provenance
if: ${{ github.run_attempt > 1 }}
run: |
node dev/scripts/relay-monitor-evidence.mjs verify-restore \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${{ steps.prior-attempt.outputs.value }}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
echo "RESTART_FLAG=--restart" >> "${GITHUB_ENV}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- name: Verify exact-audience admin identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
- name: Run read-only relay dry-run
if: ${{ inputs.mode == 'dry-run' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 15 \
--pre-drain-dry-run \
${RESTART_FLAG:-}
- name: Run first relay monitor segment
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--max-samples-this-run 45 \
${RESTART_FLAG:-}
- id: google-auth-refresh
if: ${{ inputs.mode == 'monitor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_MONITOR_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Run remaining relay monitor window
if: ${{ inputs.mode == 'monitor' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-refresh.outputs.id_token }}
run: |
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
pnpm --filter @orca-cloud/relay-ops incident:monitor \
--environment production \
--incident-id "${INCIDENT_ID}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--migration-policy "${MIGRATION_POLICY}" \
--recovery-source-cell-id "${RECOVERY_SOURCE_CELL_ID}" \
--capacity-cell-id "${CAPACITY_CELL_ID}" \
--interval-seconds 60 \
--output-directory "${OUTPUT_DIRECTORY}" \
--duration-minutes 90 \
--restart
- name: Seal private evidence provenance
if: ${{ always() }}
run: |
node dev/scripts/relay-monitor-evidence.mjs create \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "${INCIDENT_ID}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--mode "${MONITOR_MODE}"
- name: Publish aggregate job summary
if: ${{ always() }}
run: |
if [[ -f "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" ]]; then
cat "${OUTPUT_DIRECTORY}/${INCIDENT_ID}.summary.md" >> "${GITHUB_STEP_SUMMARY}"
else
echo "Relay monitor failed before its first aggregate checkpoint." \
>> "${GITHUB_STEP_SUMMARY}"
fi
- name: Upload private aggregate evidence
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: relay-monitor-${{ inputs.mode }}-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ github.workspace }}/relay-incident
if-no-files-found: error
retention-days: 14
@@ -1,75 +0,0 @@
name: Monitor Relay Production
on:
workflow_dispatch:
inputs:
mode:
description: Run the required 15-minute pre-drain gate or a 90-minute incident watch
required: true
default: dry-run
type: choice
options:
- dry-run
- monitor
expected-selector-generation:
description: Exact durable admission-selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact comma-separated existing-only cells, or none
required: true
type: string
expected-migration-only-cells:
description: Exact comma-separated migration-only cells, or none
required: true
type: string
expected-general-cells:
description: Exact comma-separated general cells, or none
required: true
type: string
migration-policy:
description: Migration checks matched to the intended mutation
required: true
default: strict
type: choice
options:
- strict
- recover-forward
- capacity-transition
recovery-source-cell-id:
description: Existing-only recovery source cell, or none for strict monitoring
required: true
default: none
type: string
capacity-cell-id:
description: General cell for a capacity-transition monitor, or none
required: true
default: none
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
monitor:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-monitor-relay-production-job.yml
with:
mode: ${{ inputs.mode }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
migration-policy: ${{ inputs.migration-policy }}
recovery-source-cell-id: ${{ inputs.recovery-source-cell-id }}
capacity-cell-id: ${{ inputs.capacity-cell-id }}
@@ -1,557 +0,0 @@
name: Operate Relay Asia Admission
on:
workflow_dispatch:
inputs:
environment:
description: Target Relay environment
required: true
type: choice
options: [staging, production]
mode:
description: Inspect, initialize, verify, atomically register, promote, or roll back admission
required: true
default: verify
type: choice
options: [inspect, initialize, verify, register, configure, promote, rollback]
cell-ids:
description: Exact reviewed comma-separated cell wave (Asia, or a US cell at the Asia shape)
required: true
type: string
selector-generation:
description: Exact live selector generation; leave empty only for inspect
required: false
type: string
selector-membership-sha256:
description: Exact fingerprint printed by inspect; required only for initialize
required: false
type: string
selector-attempt-id:
description: Durable unique attempt ID; empty for inspect, verify, and configure
required: false
type: string
image-digest:
description: Expected compatible Relay sha256 digest
required: true
type: string
director-image-digest:
description: Director sha256 digest; required only for configure
required: false
type: string
evidence-run-id:
description: Staging evidence run ID for C27, C27 canary run ID for C28/C29; C30-C35 take none and each proves itself by its own canary
required: false
type: string
evidence-run-attempt:
description: Exact evidence workflow run attempt; required with an evidence run ID
required: false
type: string
confirmation:
description: Exact typed confirmation for a mutation
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: ${{ inputs.environment == 'production' && 'production-cloud-sql-rollout' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
admission:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: ${{ inputs.environment }}
env:
DIRECTOR_ORIGIN: ${{ inputs.environment == 'production' && 'https://relay.onorca.dev' || 'https://relay-staging.onorca.dev' }}
AUTH_ORIGIN: ${{ inputs.environment == 'production' && 'https://login.onorca.dev' || 'https://auth-staging.onorca.dev' }}
DIRECTOR_SERVICE: ${{ inputs.environment == 'production' && 'orca-cloud-relay' || 'orca-cloud-relay-staging' }}
REGIONAL_PLACEMENT_SECRET: orca-cloud-relay-regional-placement-enabled
GCP_PROJECT_ID: ${{ inputs.environment == 'production' && 'onorca-cloud' || 'onorca-cloud-staging' }}
GCP_REGION: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_REGION || vars.STAGING_GCP_REGION }}
DIRECTOR_MAX_INSTANCES: ${{ inputs.environment == 'production' && '5' || '2' }}
TF_BACKEND: ${{ inputs.environment == 'production' && 'backend/production.hcl' || 'backend/staging.hcl' }}
TARGET_ENVIRONMENT: ${{ inputs.environment }}
OPERATION_MODE: ${{ inputs.mode }}
TARGET_CELL_IDS: ${{ inputs.cell-ids }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
EXPECTED_SELECTOR_MEMBERSHIP_SHA256: ${{ inputs.selector-membership-sha256 }}
SELECTOR_ATTEMPT_ID: ${{ inputs.selector-attempt-id }}
IMAGE_DIGEST: ${{ inputs.image-digest }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
EVIDENCE_RUN_ID: ${{ inputs.evidence-run-id }}
EVIDENCE_RUN_ATTEMPT: ${{ inputs.evidence-run-attempt }}
OPERATION_CONFIRMATION: ${{ inputs.confirmation }}
DEPLOY_WORKLOAD_IDENTITY_PROVIDER: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER || vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ inputs.environment == 'production' && vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT || vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
steps:
- uses: actions/checkout@v4
- name: Validate exact operation inputs before authentication
id: inputs
shell: bash
run: |
set -euo pipefail
test -n "${DEPLOY_WORKLOAD_IDENTITY_PROVIDER}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
evidence_kind=none
canary_cell=none
canary_region=asia-east2
artifact_name=none
if test "${OPERATION_MODE}" = inspect; then
test -z "${EXPECTED_SELECTOR_GENERATION}"
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
else
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
fi
if test "${OPERATION_MODE}" = initialize; then
test "${EXPECTED_SELECTOR_GENERATION}" = 0
[[ "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" =~ ^[a-f0-9]{64}$ ]]
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${OPERATION_CONFIRMATION}" = INITIALIZE_ADMISSION_SELECTOR
elif test "${OPERATION_MODE}" = verify; then
test -z "${SELECTOR_ATTEMPT_ID}"
test -z "${OPERATION_CONFIRMATION}"
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
elif test "${OPERATION_MODE}" = inspect; then
:
elif test "${OPERATION_MODE}" = configure; then
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${SELECTOR_ATTEMPT_ID}"
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
test "${OPERATION_CONFIRMATION}" = CONFIGURE_ASIA_DIRECTOR
else
test -z "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}"
test -z "${DIRECTOR_IMAGE_DIGEST}"
[[ "${SELECTOR_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
case "${OPERATION_MODE}:${OPERATION_CONFIRMATION}" in
register:REGISTER_ASIA_MIGRATION_ONLY) ;;
promote:PROMOTE_ASIA_GENERAL) ;;
rollback:ROLLBACK_ASIA_MIGRATION_ONLY) ;;
*) echo "typed confirmation does not match the requested mutation" >&2; exit 1 ;;
esac
fi
if test "${TARGET_ENVIRONMENT}:${OPERATION_MODE}" = production:promote; then
case "${TARGET_CELL_IDS}" in
production-gce-c27)
evidence_kind=staging
artifact_name="relay-asia-staging-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
canary_cell=production-gce-c27
;;
production-gce-c28,production-gce-c29)
evidence_kind=c27
artifact_name="relay-asia-c27-canary-${EVIDENCE_RUN_ID}-${EVIDENCE_RUN_ATTEMPT}"
;;
production-gce-c30|production-gce-c31|production-gce-c34|production-gce-c35)
# No earlier proof binds a later cell's generation; its own canary below rolls it back on failure.
canary_cell="${TARGET_CELL_IDS}"
;;
production-gce-c32|production-gce-c33)
# A US cell at the Asia shape proves itself the same way, with load aimed at its own region.
canary_cell="${TARGET_CELL_IDS}"
canary_region=us-central1
;;
*) echo "production promotion wave is not reviewed" >&2; exit 1 ;;
esac
fi
if test "${evidence_kind}" = none; then
test -z "${EVIDENCE_RUN_ID}"
test -z "${EVIDENCE_RUN_ATTEMPT}"
else
[[ "${EVIDENCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${EVIDENCE_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
fi
canary=false
if test "${canary_cell}" != none; then
canary=true
# Leaves room for the rollback attempt's -rollback suffix within 128 characters.
test "${#SELECTOR_ATTEMPT_ID}" -le 119
fi
{
echo "evidence_kind=${evidence_kind}"
echo "artifact_name=${artifact_name}"
echo "canary=${canary}"
echo "canary_cell=${canary_cell}"
echo "canary_region=${canary_region}"
echo "canary_hostname=${canary_cell##*-}"
} >> "${GITHUB_OUTPUT}"
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
if: ${{ steps.inputs.outputs.canary == 'true' }}
- name: Install exact canary dependencies
if: ${{ steps.inputs.outputs.canary == 'true' }}
run: pnpm install --frozen-lockfile
- name: Build the canary Relay contract
if: ${{ steps.inputs.outputs.canary == 'true' }}
run: pnpm --filter @orca-cloud/relay-contract build
- name: Download immutable rollout evidence
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
uses: actions/download-artifact@v4
with:
name: ${{ steps.inputs.outputs.artifact_name }}
path: ${{ runner.temp }}/relay-asia-input-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.evidence-run-id }}
- name: Verify evidence provenance and rollout binding before authentication
if: ${{ steps.inputs.outputs.evidence_kind != 'none' }}
env:
GH_TOKEN: ${{ github.token }}
EVIDENCE_KIND: ${{ steps.inputs.outputs.evidence_kind }}
shell: bash
run: |
set -euo pipefail
run_json="${RUNNER_TEMP}/relay-asia-evidence-run.json"
verified="${RUNNER_TEMP}/relay-asia-evidence-verified"
gh api "/repos/${GITHUB_REPOSITORY}/actions/runs/${EVIDENCE_RUN_ID}/attempts/${EVIDENCE_RUN_ATTEMPT}" > "${run_json}"
evidence_commit_sha="$(
jq -er '.head_sha | select(type == "string" and test("^[a-f0-9]{40}$"))' "${run_json}"
)"
command=(node dev/scripts/relay-asia-rollout-evidence.mjs "verify-${EVIDENCE_KIND}"
--evidence "${RUNNER_TEMP}/relay-asia-input-evidence/evidence.json"
--run-json "${run_json}"
--commit-sha "${evidence_commit_sha}"
--image-digest "${IMAGE_DIGEST}"
--now "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
--output "${verified}")
if test "${EVIDENCE_KIND}" = c27; then
command+=(--selector-generation "${EXPECTED_SELECTOR_GENERATION}")
fi
"${command[@]}"
test "$(< "${verified}")" = verified
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ env.DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ env.DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: ${{ env.DIRECTOR_ORIGIN }}/v1/admin/drain
id_token_include_email: true
- name: Require the exact director image before promotion
if: ${{ inputs.mode == 'promote' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: ${{ inputs.environment == 'production' && 'onorca-cloud-terraform-state' || 'onorca-cloud-staging-terraform-state' }}
object: ${{ inputs.environment == 'production' && 'terraform/state/cloud-sql-rollout/production.lock' || 'terraform/state/cloud-sql-rollout/staging.lock' }}
if: ${{ inputs.mode == 'configure' || steps.inputs.outputs.canary == 'true' }}
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode == 'configure' }}
with:
terraform_version: 1.15.8
terraform_wrapper: false
- name: Run the exact generation-bound admission operation
id: admission-operation
if: ${{ inputs.mode != 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode "${OPERATION_MODE}" \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 "${EXPECTED_SELECTOR_MEMBERSHIP_SHA256}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
states="$(jq -cS '.states // {}' <<< "${result}")"
membership="$(jq -cS '.membership // empty' <<< "${result}")"
membership_sha256="$(jq -r '.membershipSha256 // empty' <<< "${result}")"
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
result_dir="${RUNNER_TEMP}/relay-asia-admission-result"
mkdir -p "${result_dir}"
node dev/scripts/sanitize-relay-asia-admission-result.mjs \
<<< "${result}" > "${result_dir}/result.json"
{
echo "### Relay Asia admission"
echo "- Mode: ${OPERATION_MODE}"
echo "- Cells: ${TARGET_CELL_IDS}"
echo "- Result generation: ${generation}"
echo "- States: \`${states}\`"
if test -n "${membership}"; then echo "- Membership: \`${membership}\`"; fi
if test -n "${membership_sha256}"; then
echo "- Membership SHA-256: \`${membership_sha256}\`"
fi
} >> "${GITHUB_STEP_SUMMARY}"
- name: Verify the canary cell state and start the timed canary
id: canary-start
if: ${{ steps.inputs.outputs.canary == 'true' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
CANARY_CELL: ${{ steps.inputs.outputs.canary_cell }}
shell: bash
run: |
set -euo pipefail
# A later cell's launch cells were checked general by the promotion; verify reads only its digest.
case "${CANARY_CELL}" in
production-gce-c27)
verify_cells=production-gce-c27,production-gce-c28,production-gce-c29
expected_states='{"production-gce-c27":"general","production-gce-c28":"migration-only","production-gce-c29":"migration-only"}'
;;
production-gce-c30|production-gce-c31|production-gce-c32|production-gce-c33|production-gce-c34|production-gce-c35)
verify_cells="${CANARY_CELL}"
expected_states="{\"${CANARY_CELL}\":\"general\"}"
;;
*) exit 1 ;;
esac
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids "${verify_cells}" \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -cS '.states' <<< "${result}")" = "$(jq -cS '.' <<< "${expected_states}")"
echo "verify_cells=${verify_cells}" >> "${GITHUB_OUTPUT}"
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run a real five-minute canary control and splice
id: canary-load
if: ${{ steps.inputs.outputs.canary == 'true' }}
env:
CANARY_REGION: ${{ steps.inputs.outputs.canary_region }}
shell: bash
run: |
set -euo pipefail
log="${RUNNER_TEMP}/relay-asia-canary-load.jsonl"
report="${RUNNER_TEMP}/relay-asia-canary-load.json"
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region "${CANARY_REGION}" \
--relay-asia-load-principals 1 \
--controls 1 \
--splices 1 \
--capacity-hard-cap 3000 \
--ramp-seconds 0 \
--duration-seconds 300 \
--splice-hold-seconds 60 \
--required-lease-horizons 2 > "${log}"
jq -cer 'select(.event == "relay_load_complete")' "${log}" | tail -n 1 > "${report}"
echo "ended_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Collect regional, Relay SQL, and Cloud SQL canary evidence
if: ${{ steps.inputs.outputs.canary == 'true' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
CANARY_CELL: ${{ steps.inputs.outputs.canary_cell }}
CANARY_STARTED_AT: ${{ steps.canary-start.outputs.started_at }}
CANARY_VERIFY_CELLS: ${{ steps.canary-start.outputs.verify_cells }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode verify \
--cell-ids "${CANARY_VERIFY_CELLS}" \
--expected-generation "${{ steps.admission-operation.outputs.generation }}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r --arg cell "${CANARY_CELL}" '.states[$cell]' <<< "${result}")" = general
ended_at="${{ steps.canary-load.outputs.ended_at }}"
sleep 60
output="${RUNNER_TEMP}/relay-asia-output-evidence"
logs="${RUNNER_TEMP}/relay-asia-canary-runtime-metrics.json"
mkdir -p "${output}"
gcloud logging read \
"timestamp>=\"${CANARY_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" \
--limit 20000 \
--format json > "${logs}"
node dev/scripts/relay-asia-rollout-evidence.mjs create-canary \
--cell-id "${CANARY_CELL}" \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.admission-operation.outputs.generation }}" \
--started-at "${CANARY_STARTED_AT}" \
--ended-at "${ended_at}" \
--load-report "${RUNNER_TEMP}/relay-asia-canary-load.json" \
--logs-json "${logs}" \
--output "${output}/evidence.json"
jq -r '.metrics | to_entries[] | "- \(.key): \(.value)"' \
"${output}/evidence.json" >> "${GITHUB_STEP_SUMMARY}"
- name: Upload immutable canary evidence
id: canary-evidence-upload
if: ${{ steps.inputs.outputs.canary == 'true' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-${{ steps.inputs.outputs.canary_hostname }}-canary-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-output-evidence/evidence.json
if-no-files-found: error
retention-days: 7
- name: Upload sanitized admission result
if: ${{ inputs.mode != 'configure' && steps.admission-operation.outcome == 'success' }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-admission-result-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-admission-result/result.json
if-no-files-found: error
retention-days: 7
- name: Return an unproven canary cell to migration-only
if: ${{ always() && steps.inputs.outputs.canary == 'true' && steps.admission-operation.outcome != 'skipped' && steps.canary-evidence-upload.outcome != 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
CANARY_CELL: ${{ steps.inputs.outputs.canary_cell }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode recover-promotion \
--cell-ids "${CANARY_CELL}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment production \
--mode rollback \
--cell-ids "${CANARY_CELL}" \
--expected-generation "${promoted_generation}" \
--attempt-id "${SELECTOR_ATTEMPT_ID}-rollback" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r --arg cell "${CANARY_CELL}" '.states[$cell]' <<< "${result}")" = migration-only
- name: Require registered migration-only cells before director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode registered \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
- name: Build the additive director cell configuration
if: ${{ inputs.mode == 'configure' }}
id: director-config
shell: bash
run: |
set -euo pipefail
terraform -chdir=infra/terraform init -reconfigure -input=false -backend-config="${TF_BACKEND}"
topology="${RUNNER_TEMP}/relay-asia-state-topology.json"
current="${RUNNER_TEMP}/relay-current-director-cells.json"
desired="${RUNNER_TEMP}/relay-asia-director-cells.json"
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${topology}"
service="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
revision="$(jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end' \
<<< "${service}")"
gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '.spec.containers[0].env[] | select(.name == "ORCA_RELAY_CELLS_JSON") | .value | fromjson' \
> "${current}"
node dev/scripts/prepare-relay-asia-director-cells.mjs \
--current-json "${current}" \
--topology-json "${topology}" \
--output "${desired}" \
--cell-ids "${TARGET_CELL_IDS}" \
--image-digest "${IMAGE_DIGEST}"
echo "file=${desired}" >> "${GITHUB_OUTPUT}"
- name: Deploy the registered additive director topology
if: ${{ inputs.mode == 'configure' }}
env:
DIRECTOR_CELLS_FILE: ${{ steps.director-config.outputs.file }}
run: |
current="$(gcloud secrets versions access latest \
--project "${GCP_PROJECT_ID}" --secret "${REGIONAL_PLACEMENT_SECRET}")"
[[ "${current}" =~ ^(true|false)$ ]]
image="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}"
release_id="asia-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE}" \
--image "${image}" \
--role director \
--max-instances "${DIRECTOR_MAX_INSTANCES}" \
--release-id "${release_id}" \
--director-cells-json "$(< "${DIRECTOR_CELLS_FILE}")" \
--prune-revisions false
- name: Verify selector and heartbeats after director configuration
if: ${{ inputs.mode == 'configure' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
run: |
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment "${TARGET_ENVIRONMENT}" \
--mode verify \
--cell-ids "${TARGET_CELL_IDS}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '[.states[] == "migration-only"] | all' <<< "${result}")" = true
revision="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else error("split traffic") end')"
revision_json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
jq -e --arg image "us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${DIRECTOR_IMAGE_DIGEST}" \
'.spec.containers[0].image == $image' <<< "${revision_json}" > /dev/null
jq -er --arg secret "${REGIONAL_PLACEMENT_SECRET}" \
'[.spec.containers[0].env[] |
select(.name == "ORCA_RELAY_REGIONAL_PLACEMENT_ENABLED") |
(.valueSource.secretKeyRef // .valueFrom.secretKeyRef // {}) |
{secret: (.secret // .name), version: (.version // .key)} |
select(.secret == $secret and (.version | test("^[1-9][0-9]*$")))] |
if length == 1 then .[0].version else error("regional switch version missing") end' \
<<< "${revision_json}" > "${RUNNER_TEMP}/relay-regional-placement-version"
regional_version="$(< "${RUNNER_TEMP}/relay-regional-placement-version")"
[[ "$(gcloud secrets versions access "${regional_version}" --project "${GCP_PROJECT_ID}" \
--secret "${REGIONAL_PLACEMENT_SECRET}")" =~ ^(true|false)$ ]]
echo "Director configuration now lists the registered migration-only Asia cells." >> "${GITHUB_STEP_SUMMARY}"
@@ -1,125 +0,0 @@
name: Operate Relay Production Cell Flags
# Writes one cell's runtime switch object and waits for the cell to report it applied.
# One cell per run; never fleet-wide. dry-run writes nothing.
on:
workflow_dispatch:
inputs:
mode:
description: dry-run prints the change; write applies it
required: true
default: dry-run
type: choice
options: [dry-run, write]
cell-id:
description: Exact cell id, e.g. production-gce-c26
required: true
type: string
break-glass:
description: >-
Only for a dead or unreachable reserve cell, with set admitMode=default. Typed as
BREAK GLASS <cell-id>; records db without the cell's read-back
required: false
default: ''
type: string
confirm-reserve:
description: Required for admitMode=reserve, typed as RESERVE <cell-id>
required: false
default: ''
type: string
set:
description: >-
Switches to change, e.g. readinessLocal=true,admitMode=reserve (key=default removes a
key). Every other switch keeps its current value. Known: readinessLocal, ticketCheck,
admitMode, intakePerSec, reserveDryRun, rejectionFence, readTimeoutMarginMs,
reregisterInFlight
required: true
type: string
expected-generation:
description: Exact current object generation from a dry run, or 0 if none exists
required: true
type: string
confirmation:
description: Required for write, exactly "WRITE_RELAY_CELL_FLAGS <cell-id>"
required: false
type: string
permissions:
contents: read
id-token: write
# One write at a time across the fleet keeps the audit trail in order.
concurrency:
group: relay-production-cell-flags
cancel-in-progress: false
defaults:
run:
# Explicit bash runs with pipefail, so the script's exit code survives the tee.
shell: bash
working-directory: cloud
jobs:
operate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
# A flip back waits for the cell to lease every control: 8-50 minutes, scaled to its controls.
timeout-minutes: 60
environment: production
env:
MODE: ${{ inputs.mode }}
CELL_ID: ${{ inputs.cell-id }}
FLAG_CHANGES: ${{ inputs.set }}
EXPECTED_GENERATION: ${{ inputs.expected-generation }}
CONFIRMATION: ${{ inputs.confirmation }}
CONFIRM_RESERVE: ${{ inputs.confirm-reserve }}
BREAK_GLASS: ${{ inputs.break-glass }}
GCP_PROJECT_ID: onorca-cloud
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- name: Write or preview the cell's switch object
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
[[ "${CELL_ID}" =~ ^production-gce-c[0-9]{1,3}$ ]]
ORCA_RELAY_CONTROL_ACCESS_TOKEN="$(gcloud auth print-access-token)"
echo "::add-mask::${ORCA_RELAY_CONTROL_ACCESS_TOKEN}"
export ORCA_RELAY_CONTROL_ACCESS_TOKEN
node dev/scripts/operate-relay-cell-flags.mjs \
--mode "${MODE}" --cell-id "${CELL_ID}" \
--cell-origin "https://${CELL_ID#production-gce-}.relay.onorca.dev" \
--project-id "${GCP_PROJECT_ID}" \
--set "${FLAG_CHANGES}" \
--director-origin "https://relay.onorca.dev" --confirm-reserve "${CONFIRM_RESERVE}" \
--break-glass "${BREAK_GLASS}" \
--expected-generation "${EXPECTED_GENERATION}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-cell-flags.jsonl"
- name: Audit summary
if: ${{ always() }}
run: |
{
echo "### Relay cell flags: ${MODE} ${CELL_ID}"
echo "Triggered by: ${GITHUB_TRIGGERING_ACTOR}, run ${GITHUB_RUN_ID} attempt ${GITHUB_RUN_ATTEMPT}"
echo '```json'
cat "${RUNNER_TEMP}/relay-cell-flags.jsonl" 2>/dev/null || echo '(no output)'
echo '```'
} >> "${GITHUB_STEP_SUMMARY}"
@@ -1,341 +0,0 @@
name: Operate Relay Production Rehome Job
on:
workflow_call:
inputs:
mode: { required: true, type: string }
director-image-digest: { required: true, type: string }
rollback-image-digest: { required: true, type: string }
expected-selector-generation: { required: true, type: string }
expected-existing-only-cells: { required: true, type: string }
expected-migration-only-cells: { required: true, type: string }
expected-general-cells: { required: true, type: string }
expected-control-generation: { required: true, type: string }
not-before: { required: true, type: string }
rate-per-minute: { required: true, type: string }
preference-max-age-ms: { required: true, type: string }
host-cooldown-ms: { required: true, type: string }
drain-grace-ms: { required: true, type: string }
confirmation: { required: true, type: string }
monitor-run-id: { required: true, type: string }
monitor-run-attempt: { required: true, type: string }
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
# `shell: bash` adds pipefail; without it `node ... | tee` reports tee's exit code and a
# thrown inspect/apply passed green (Aug 28-29 and Sep 3 2026 runs).
shell: bash
working-directory: cloud
jobs:
control:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 30
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
DIRECTOR_SERVICE: orca-cloud-relay
DIRECTOR_ORIGIN: https://relay.onorca.dev
REHOME_AUDIENCE: https://relay.onorca.dev/v1/admin/host-drain
MODE: ${{ inputs.mode }}
DIRECTOR_IMAGE_DIGEST: ${{ inputs.director-image-digest }}
ROLLBACK_IMAGE_DIGEST: ${{ inputs.rollback-image-digest }}
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
EXPECTED_EXISTING_ONLY_CELLS: ${{ inputs.expected-existing-only-cells }}
EXPECTED_MIGRATION_ONLY_CELLS: ${{ inputs.expected-migration-only-cells }}
EXPECTED_GENERAL_CELLS: ${{ inputs.expected-general-cells }}
EXPECTED_CONTROL_GENERATION: ${{ inputs.expected-control-generation }}
NOT_BEFORE: ${{ inputs.not-before }}
RATE_PER_MINUTE: ${{ inputs.rate-per-minute }}
PREFERENCE_MAX_AGE_MS: ${{ inputs.preference-max-age-ms }}
HOST_COOLDOWN_MS: ${{ inputs.host-cooldown-ms }}
DRAIN_GRACE_MS: ${{ inputs.drain-grace-ms }}
CONFIRMATION: ${{ inputs.confirmation }}
MONITOR_RUN_ID: ${{ inputs.monitor-run-id }}
MONITOR_RUN_ATTEMPT: ${{ inputs.monitor-run-attempt }}
OUTPUT_DIRECTORY: ${{ github.workspace }}/relay-monitor-evidence
steps:
- name: Require exact reusable-workflow configuration
working-directory: .
env:
DEPLOY_WIF: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
DEPLOY_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
[[ "${MODE}" =~ ^(inspect|enable|pause|disable)$ ]]
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${EXPECTED_CONTROL_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
test -n "${DEPLOY_WIF}"
test -n "${DEPLOY_SERVICE_ACCOUNT}"
if [[ "${MODE}" =~ ^(inspect|enable)$ ]]; then
[[ "${DIRECTOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${ROLLBACK_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test -n "${GCP_REGION}"
test -n "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
fi
case "${MODE}" in
inspect)
test -z "${CONFIRMATION}"
;;
enable)
test "${CONFIRMATION}" = ENABLE_REGIONAL_REHOMING
test "${RATE_PER_MINUTE}" = 10
[[ "${NOT_BEFORE}" =~ ^[1-9][0-9]*$ ]]
;;
pause)
test "${CONFIRMATION}" = PAUSE_REGIONAL_REHOMING
;;
disable)
test "${CONFIRMATION}" = DISABLE_REGIONAL_REHOMING
;;
esac
# Full history: the monitor evidence this job verifies is sealed at an ancestor commit,
# and the provenance check fails closed on a commit a shallow clone left out.
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Apply emergency durable pause or disable before diagnostics
if: ${{ inputs.mode == 'pause' || inputs.mode == 'disable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--host-cooldown-ms "${HOST_COOLDOWN_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- uses: pnpm/action-setup@v4
if: ${{ inputs.mode == 'enable' }}
with: { package_json_file: cloud/package.json }
- run: pnpm install --frozen-lockfile
if: ${{ inputs.mode == 'enable' }}
- name: Download fresh aggregate safety evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/download-artifact@v4
with:
name: relay-monitor-dry-run-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ github.workspace }}/relay-monitor-evidence
github-token: ${{ github.token }}
run-id: ${{ inputs.monitor-run-id }}
- name: Verify enable evidence provenance
if: ${{ inputs.mode == 'enable' }}
run: |
[[ "${MONITOR_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${MONITOR_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
node dev/scripts/relay-monitor-evidence.mjs verify-authority \
--directory "${OUTPUT_DIRECTORY}" \
--incident-id "relay-${MONITOR_RUN_ID}-dry-run" \
--run-id "${MONITOR_RUN_ID}" --run-attempt "${MONITOR_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" --mode dry-run \
--required-migration-policy strict
- name: Reject previously consumed enable safety evidence
if: ${{ inputs.mode == 'enable' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
COUNT="$(gh api "/repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${MARKER_NAME}&per_page=1" \
--jq '.total_count')"
test "${COUNT}" = 0
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/cloud-sql-rollout/production.lock
- name: Verify exact serving and rollback director identities
if: ${{ inputs.mode == 'inspect' || inputs.mode == 'enable' }}
env:
DIRECTOR_RUNTIME_SERVICE_ACCOUNT: ${{ vars.PRODUCTION_GCP_RELAY_DIRECTOR_RUNTIME_SERVICE_ACCOUNT }}
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
SERVING_REVISION="$(jq -er \
'[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName
else error("director does not have one serving revision") end' \
<<< "${SERVICE_JSON}")"
ROLLBACK_REVISION="$(jq -er \
'[.status.traffic[] | select(.tag == "selector-rollback")] |
if length == 1 then .[0].revisionName else error("rollback tag missing") end' \
<<< "${SERVICE_JSON}")"
verify_revision() {
local revision="$1" expected_digest="$2"
local json
json="$(gcloud run revisions describe "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
test "$(jq -r '.spec.serviceAccountName' <<< "${json}")" = \
"${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '.spec.containers[0].image | split("@") | last' <<< "${json}")" = \
"${expected_digest}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT") | .value] | if length == 1
then .[0] else empty end' <<< "${json}")" = "${DIRECTOR_RUNTIME_SERVICE_ACCOUNT}"
test "$(jq -r '[.spec.containers[0].env[] | select(.name ==
"ORCA_RELAY_REHOME_AUDIENCE") | .value] | if length == 1 then .[0]
else empty end' <<< "${json}")" = "${REHOME_AUDIENCE}"
}
verify_revision "${SERVING_REVISION}" "${DIRECTOR_IMAGE_DIGEST}"
verify_revision "${ROLLBACK_REVISION}" "${ROLLBACK_IMAGE_DIGEST}"
- name: Seal 24-hour aggregate region observation evidence
if: ${{ inputs.mode == 'enable' }}
run: |
mkdir -p "${RUNNER_TEMP}/relay-region-observation"
# 6 director instances x 120 samples/hour x 25h = 18000; a clipped
# read empties the oldest hourly buckets and fails the seal.
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND jsonPayload.event="orca_relay_runtime_metrics" AND jsonPayload.role="director"' \
--project "${GCP_PROJECT_ID}" --freshness=25h --limit=30000 --format=json \
| node dev/scripts/relay-region-observation-evidence.mjs create \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}" \
> "${RUNNER_TEMP}/relay-region-observation/evidence.json"
- name: Upload sealed 24-hour aggregate region evidence
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-region-observation-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-region-observation/evidence.json
retention-days: 90
if-no-files-found: error
- name: Verify sealed 24-hour enable authority
if: ${{ inputs.mode == 'enable' }}
run: |
node dev/scripts/relay-region-observation-evidence.mjs verify \
--file "${RUNNER_TEMP}/relay-region-observation/evidence.json" \
--commit-sha "${GITHUB_SHA}" \
--director-image-digest "${DIRECTOR_IMAGE_DIGEST}" \
--selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--control-generation "${EXPECTED_CONTROL_GENERATION}"
- name: Recheck every aggregate safety signal before enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
pnpm incident:relay-preflight -- \
--state-file "${OUTPUT_DIRECTORY}/relay-${MONITOR_RUN_ID}-dry-run.state.json"
- name: Seal single-use enable safety authority
if: ${{ inputs.mode == 'enable' }}
run: |
MARKER_NAME="relay-rehome-enable-monitor-consumed-${MONITOR_RUN_ID}-${MONITOR_RUN_ATTEMPT}"
mkdir -p "${RUNNER_TEMP}/relay-rehome-enable-authority"
printf '%s\n' "${GITHUB_RUN_ID}" \
> "${RUNNER_TEMP}/relay-rehome-enable-authority/${MARKER_NAME}"
- name: Consume enable safety evidence before durable mutation
if: ${{ inputs.mode == 'enable' }}
uses: actions/upload-artifact@v4
with:
name: relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
path: ${{ runner.temp }}/relay-rehome-enable-authority/relay-rehome-enable-monitor-consumed-${{ inputs.monitor-run-id }}-${{ inputs.monitor-run-attempt }}
retention-days: 90
if-no-files-found: error
- name: Inspect regional rehome control
if: ${{ inputs.mode == 'inspect' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode inspect --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Apply exact durable regional rehome enable
if: ${{ inputs.mode == 'enable' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode "${MODE}" --director-origin "${DIRECTOR_ORIGIN}" \
--expected-selector-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-existing-only-cells "${EXPECTED_EXISTING_ONLY_CELLS}" \
--expected-migration-only-cells "${EXPECTED_MIGRATION_ONLY_CELLS}" \
--expected-general-cells "${EXPECTED_GENERAL_CELLS}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--not-before "${NOT_BEFORE}" --rate-per-minute "${RATE_PER_MINUTE}" \
--preference-max-age-ms "${PREFERENCE_MAX_AGE_MS}" \
--host-cooldown-ms "${HOST_COOLDOWN_MS}" \
--drain-grace-ms "${DRAIN_GRACE_MS}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-rehome-control.json"
- name: Read fresh aggregate completion and abort evidence
run: |
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="orca-cloud-relay" AND textPayload:"[orca-relay] regional rehome inventory"' \
--project "${GCP_PROJECT_ID}" --freshness=15m --limit=20 --format=json \
| node dev/scripts/relay-rehome-aggregate-evidence.mjs --max-age-ms 900000 \
| tee "${RUNNER_TEMP}/relay-rehome-inventory.json"
- name: Publish aggregate control evidence
run: |
{
echo '### Regional rehome control'
jq -r '"- mode: `\(.mode)`\n- generation: `\(.control.generation)`\n- enabled: `\(.control.enabled)`"' \
"${RUNNER_TEMP}/relay-rehome-control.json"
jq -r '"- active: `\(.active)`\n- awaiting receipt: `\(.awaitingReceipt)`\n- target registered: `\(.targetRegistered)`\n- completed (24h): `\(.completedLast24Hours)`\n- aborted (24h): `\(.abortedLast24Hours)`\n- host not arrived (24h): `\(.hostNotArrivedLast24Hours // "not reported")`\n- oldest active age (ms): `\(.oldestActiveAgeMs // "none")`"' \
"${RUNNER_TEMP}/relay-rehome-inventory.json"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Fail closed after an unsuccessful enable run
if: ${{ failure() && inputs.mode == 'enable' && steps.google-auth.outcome == 'success' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/operate-relay-regional-rehome.mjs \
--mode recover-enable --director-origin "${DIRECTOR_ORIGIN}" \
--expected-control-generation "${EXPECTED_CONTROL_GENERATION}" \
--confirmation RECOVER_FAILED_REGIONAL_REHOME_ENABLE \
| tee "${RUNNER_TEMP}/relay-rehome-enable-recovery.json"
jq -e \
'.mode == "recover-enable" and .control.enabled == false' \
"${RUNNER_TEMP}/relay-rehome-enable-recovery.json" >/dev/null
@@ -1,112 +0,0 @@
name: Operate Relay Production Rehome
on:
workflow_dispatch:
inputs:
mode:
description: Inspect or apply the durable regional-rehome switch
required: true
default: inspect
type: choice
options: [inspect, enable, pause, disable]
director-image-digest:
description: Exact immutable serving director digest
required: true
type: string
rollback-image-digest:
description: Exact immutable selector-rollback director digest
required: true
type: string
expected-selector-generation:
description: Exact admission selector generation
required: true
type: string
expected-existing-only-cells:
description: Exact existing-only membership, or none
required: true
type: string
expected-migration-only-cells:
description: Exact migration-only membership, or none
required: true
type: string
expected-general-cells:
description: Exact general membership, or none
required: true
type: string
expected-control-generation:
description: Exact durable rehome generation
required: true
type: string
not-before:
description: Exact epoch milliseconds; ignored only by inspect
required: true
default: '0'
type: string
rate-per-minute:
description: Exact global host rate; initial enable is fixed at 10
required: true
default: '10'
type: string
preference-max-age-ms:
description: Maximum fresh preference age
required: true
default: '86400000'
type: string
host-cooldown-ms:
description: Minimum gap between two rehomes of the same host
required: true
default: '604800000'
type: string
drain-grace-ms:
description: Per-host source drain grace
required: true
default: '3600000'
type: string
monitor-run-id:
description: Fresh successful aggregate dry-run required only by enable
required: false
type: string
monitor-run-attempt:
description: Exact monitor attempt required only by enable
required: false
type: string
confirmation:
description: ENABLE_REGIONAL_REHOMING, PAUSE_REGIONAL_REHOMING, or DISABLE_REGIONAL_REHOMING
required: false
type: string
permissions:
actions: read
contents: read
id-token: write
concurrency:
group: production-cloud-sql-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
operate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
uses: ./.github/workflows/cloud-operate-relay-production-rehome-job.yml
with:
mode: ${{ inputs.mode }}
director-image-digest: ${{ inputs.director-image-digest }}
rollback-image-digest: ${{ inputs.rollback-image-digest }}
expected-selector-generation: ${{ inputs.expected-selector-generation }}
expected-existing-only-cells: ${{ inputs.expected-existing-only-cells }}
expected-migration-only-cells: ${{ inputs.expected-migration-only-cells }}
expected-general-cells: ${{ inputs.expected-general-cells }}
expected-control-generation: ${{ inputs.expected-control-generation }}
not-before: ${{ inputs.not-before }}
rate-per-minute: ${{ inputs.rate-per-minute }}
preference-max-age-ms: ${{ inputs.preference-max-age-ms }}
host-cooldown-ms: ${{ inputs.host-cooldown-ms }}
drain-grace-ms: ${{ inputs.drain-grace-ms }}
confirmation: ${{ inputs.confirmation }}
monitor-run-id: ${{ inputs.monitor-run-id }}
monitor-run-attempt: ${{ inputs.monitor-run-attempt }}
secrets: inherit
@@ -1,327 +0,0 @@
name: Operate Relay Production Reserve Watch
# Watches one cell already switched to reserve mode and flips it back to db on the first stop
# rule it trips (step 5 design §9). Manual only. dry-run evaluates and logs, writes nothing.
# An operator watches the run. A refused watch or a flip back that did not land raises an ::error::
# annotation, as the clock-skew monitor does, and in write mode logs the line the reserve-watch
# alert pages on. Refuses to start while a rollout holds the Cloud SQL rollout lease.
on:
workflow_dispatch:
inputs:
mode:
description: dry-run logs what a trip would write; write flips the cell back on a trip
required: true
default: dry-run
type: choice
options: [dry-run, write]
cell-id:
description: Exact cell id already in reserve mode, e.g. production-gce-c26
required: true
type: string
duration-minutes:
description: How long to watch, 5 to 300
required: true
default: '120'
type: string
flipped-at:
description: >-
When the reserve write applied, ISO with a zone, e.g. 2026-10-14T12:00:00Z. The 503
background is the median of the 10 minutes before it
required: false
default: ''
type: string
background-503-per-minute:
description: >-
Director 503s per minute before the flip; overrides the one measured before flipped-at.
One of the two is required
required: false
default: ''
type: string
unverifiable-limit:
description: Consecutive samples with any unreadable signal that count as a trip
required: true
default: '3'
type: string
confirmation:
description: Required for write, exactly "WRITE_RELAY_CELL_FLAGS <cell-id>"
required: false
type: string
permissions:
contents: read
id-token: write
# One watch per cell; why not the flag workflow's group: cloud/docs/relay-workflows.md.
concurrency:
group: relay-production-reserve-watch-${{ inputs.cell-id }}
cancel-in-progress: false
defaults:
run:
# Explicit bash runs with pipefail, so the script's exit code survives the tee.
shell: bash
working-directory: cloud
env:
MODE: ${{ inputs.mode }}
CELL_ID: ${{ inputs.cell-id }}
CONFIRMATION: ${{ inputs.confirmation }}
GCP_PROJECT_ID: onorca-cloud
jobs:
watch:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
# Six 55-minute segments cover the longest watch; under the 360-minute job limit.
timeout-minutes: 350
environment: production
outputs:
state: ${{ steps.outcome.outputs.state }}
verdict: ${{ steps.outcome.outputs.verdict }}
env:
DURATION_MINUTES: ${{ inputs.duration-minutes }}
FLIPPED_AT: ${{ inputs.flipped-at }}
BACKGROUND_503_PER_MINUTE: ${{ inputs.background-503-per-minute }}
UNVERIFIABLE_LIMIT: ${{ inputs.unverifiable-limit }}
# One segment, after the auth step before it: fixed text, the inputs arrive as env vars.
WATCH_SEGMENT: |
set -euo pipefail
[[ "${CELL_ID}" =~ ^production-gce-c[0-9]{1,3}$ ]]
node -e "if (!/^[^.]+[.][^.]+[.][^.]+$/.test(process.env.ORCA_RELAY_ADMIN_ID_TOKEN ?? '')) process.exit(1)"
node dev/scripts/watch-relay-reserve-cell.mjs watch \
--mode "${MODE}" --cell-id "${CELL_ID}" \
--cell-origin "https://${CELL_ID#production-gce-}.relay.onorca.dev" \
--project-id "${GCP_PROJECT_ID}" --director-origin "https://relay.onorca.dev" \
--director-service orca-cloud-relay --confirmation "${CONFIRMATION}" \
--duration-minutes "${DURATION_MINUTES}" --segment-minutes 55 \
--flipped-at "${FLIPPED_AT}" --background-503-per-minute "${BACKGROUND_503_PER_MINUTE}" \
--unverifiable-limit "${UNVERIFIABLE_LIMIT}" \
--state-file "${RUNNER_TEMP}/relay-reserve-watch.json" --github-output "${GITHUB_OUTPUT}" \
| tee -a "${RUNNER_TEMP}/relay-reserve-watch.jsonl"
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth-1
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- id: watch-1
name: Watch segment 1 of 6
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-1.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: google-auth-2
if: ${{ steps.watch-1.outputs.state == 'watching' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: watch-2
name: Watch segment 2 of 6
if: ${{ steps.watch-1.outputs.state == 'watching' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-2.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: google-auth-3
if: ${{ steps.watch-2.outputs.state == 'watching' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: watch-3
name: Watch segment 3 of 6
if: ${{ steps.watch-2.outputs.state == 'watching' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-3.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: google-auth-4
if: ${{ steps.watch-3.outputs.state == 'watching' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: watch-4
name: Watch segment 4 of 6
if: ${{ steps.watch-3.outputs.state == 'watching' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-4.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: google-auth-5
if: ${{ steps.watch-4.outputs.state == 'watching' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: watch-5
name: Watch segment 5 of 6
if: ${{ steps.watch-4.outputs.state == 'watching' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-5.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: google-auth-6
if: ${{ steps.watch-5.outputs.state == 'watching' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- id: watch-6
name: Watch segment 6 of 6
if: ${{ steps.watch-5.outputs.state == 'watching' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth-6.outputs.id_token }}
run: bash -c "${WATCH_SEGMENT}"
- id: outcome
name: Record the watch outcome
if: ${{ always() }}
run: |
node dev/scripts/watch-relay-reserve-cell.mjs outcome \
--state-file "${RUNNER_TEMP}/relay-reserve-watch.json" --github-output "${GITHUB_OUTPUT}"
# The documented flow writes reserve before dispatching, so a refusal can leave it unwatched.
- name: Alarm on a refused watch
if: ${{ always() && steps.outcome.outputs.state == 'refused' }}
env:
WATCH_VERDICT: ${{ steps.outcome.outputs.verdict }}
run: |
node dev/scripts/watch-relay-reserve-cell.mjs alarm --kind refused \
--mode "${MODE}" --cell-id "${CELL_ID}" --project-id "${GCP_PROJECT_ID}" --verdict "${WATCH_VERDICT}"
- name: Audit summary
if: ${{ always() }}
env:
WATCH_STATE: ${{ steps.outcome.outputs.state }}
WATCH_VERDICT: ${{ steps.outcome.outputs.verdict }}
run: |
{
echo "### Relay reserve watch: ${MODE} ${CELL_ID}"
echo "Triggered by: ${GITHUB_TRIGGERING_ACTOR}, run ${GITHUB_RUN_ID} attempt ${GITHUB_RUN_ATTEMPT}"
echo "Outcome: ${WATCH_STATE}: ${WATCH_VERDICT}"
echo '```json'
tail -n 20 "${RUNNER_TEMP}/relay-reserve-watch.jsonl" 2>/dev/null || echo '(no output)'
echo '```'
} >> "${GITHUB_STEP_SUMMARY}"
flip-back:
needs: watch
# Missing data is never healthy: a watch that crashed, timed out or was cancelled (by an
# operator too) flips back like a trip. In dry-run this only plans the write.
if: ${{ always() && vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && needs.watch.result != 'skipped' && !contains(fromJSON('["completed","stopped","refused"]'), needs.watch.outputs.state) }}
runs-on: ubuntu-latest
# A flip back waits for the cell to lease every control: 8-50 minutes, scaled to its controls.
timeout-minutes: 60
environment: production
# Only the run's per-cell group above, which a running run never yields (cancel-in-progress off).
# Every step runs on always() too, so an ordinary cancel cannot skip the switch-file write; a force-cancel
# can, and leaves the cell in reserve for the operator to flip back by hand. A cancel during the
# lease wait can still kill the process before Postgres records db: the file says db, Postgres reserve.
env:
VERDICT: ${{ needs.watch.outputs.verdict || format('the watch ended {0} without a verdict', needs.watch.result) }}
steps:
- uses: actions/checkout@v4
if: ${{ always() }}
with:
persist-credentials: false
- uses: actions/setup-node@v4
if: ${{ always() }}
with:
node-version: 24
- id: google-auth
if: ${{ always() }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
if: ${{ always() }}
with:
skip_install: true
- name: Flip the cell back to db, or preview it
if: ${{ always() }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
[[ "${CELL_ID}" =~ ^production-gce-c[0-9]{1,3}$ ]]
ORCA_RELAY_CONTROL_ACCESS_TOKEN="$(gcloud auth print-access-token)"
echo "::add-mask::${ORCA_RELAY_CONTROL_ACCESS_TOKEN}"
export ORCA_RELAY_CONTROL_ACCESS_TOKEN
node dev/scripts/watch-relay-reserve-cell.mjs flip-back \
--mode "${MODE}" --cell-id "${CELL_ID}" \
--cell-origin "https://${CELL_ID#production-gce-}.relay.onorca.dev" \
--project-id "${GCP_PROJECT_ID}" --director-origin "https://relay.onorca.dev" \
--confirmation "${CONFIRMATION}" --verdict "${VERDICT}" \
| tee "${RUNNER_TEMP}/relay-reserve-flip-back.jsonl"
# The flip-back line is logged only once the write (or its dry-run plan) finished; without
# it a missing token, a failed login or a silent cell left the cell in reserve, unwatched.
- name: Alarm when the flip back did not land
if: ${{ always() }}
run: |
if grep -qs '"event":"orca_relay_reserve_watch_flip_back"' "${RUNNER_TEMP}/relay-reserve-flip-back.jsonl"; then
exit 0
fi
{
echo "## FLIP BACK FAILED: ${CELL_ID} may still be in reserve mode"
echo "Flip it to db by hand with Operate Relay Production Cell Flags."
} >> "${GITHUB_STEP_SUMMARY}"
node dev/scripts/watch-relay-reserve-cell.mjs alarm --kind flip-back-failed \
--mode "${MODE}" --cell-id "${CELL_ID}" --project-id "${GCP_PROJECT_ID}" --verdict "${VERDICT}"
- name: Audit summary
if: ${{ always() }}
run: |
{
echo "### Relay reserve watch flip back: ${MODE} ${CELL_ID}"
echo "Triggered by: ${GITHUB_TRIGGERING_ACTOR}, run ${GITHUB_RUN_ID} attempt ${GITHUB_RUN_ATTEMPT}"
echo "Verdict: ${VERDICT}"
echo '```json'
cat "${RUNNER_TEMP}/relay-reserve-flip-back.jsonl" 2>/dev/null || echo '(no output)'
echo '```'
} >> "${GITHUB_STEP_SUMMARY}"
@@ -1,127 +0,0 @@
name: Operate Relay Staging Cell Flags
# The staging twin of Operate Relay Production Cell Flags: writes one staging cell's runtime
# switch object and waits for the cell to report it applied. One cell per run. dry-run writes
# nothing.
on:
workflow_dispatch:
inputs:
mode:
description: dry-run prints the change; write applies it
required: true
default: dry-run
type: choice
options: [dry-run, write]
cell-id:
description: Exact cell id, e.g. staging-gce-c2
required: true
type: string
break-glass:
description: >-
Only for a dead or unreachable reserve cell, with set admitMode=default. Typed as
BREAK GLASS <cell-id>; records db without the cell's read-back
required: false
default: ''
type: string
confirm-reserve:
description: Required for admitMode=reserve, typed as RESERVE <cell-id>
required: false
default: ''
type: string
set:
description: >-
Switches to change, e.g. readinessLocal=true,admitMode=reserve (key=default removes a
key). Every other switch keeps its current value. Known: readinessLocal, ticketCheck,
admitMode, intakePerSec, reserveDryRun, rejectionFence, readTimeoutMarginMs,
reregisterInFlight
required: true
type: string
expected-generation:
description: Exact current object generation from a dry run, or 0 if none exists
required: true
type: string
confirmation:
description: Required for write, exactly "WRITE_RELAY_CELL_FLAGS <cell-id>"
required: false
type: string
permissions:
contents: read
id-token: write
# Not relay-staging-mutation: a switch write must run while a staging deploy or load run is
# in flight, and it touches neither Cloud SQL nor a revision.
concurrency:
group: relay-staging-cell-flags
cancel-in-progress: false
defaults:
run:
# Explicit bash runs with pipefail, so the script's exit code survives the tee.
shell: bash
working-directory: cloud
jobs:
operate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: ubuntu-latest
# A flip back waits for the cell to lease every control: 8-50 minutes, scaled to its controls.
timeout-minutes: 60
environment: staging
env:
MODE: ${{ inputs.mode }}
CELL_ID: ${{ inputs.cell-id }}
FLAG_CHANGES: ${{ inputs.set }}
EXPECTED_GENERATION: ${{ inputs.expected-generation }}
CONFIRMATION: ${{ inputs.confirmation }}
CONFIRM_RESERVE: ${{ inputs.confirm-reserve }}
BREAK_GLASS: ${{ inputs.break-glass }}
GCP_PROJECT_ID: onorca-cloud-staging
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 24
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- name: Write or preview the cell's switch object
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
[[ "${CELL_ID}" =~ ^staging-gce-c[0-9]{1,3}$ ]]
ORCA_RELAY_CONTROL_ACCESS_TOKEN="$(gcloud auth print-access-token)"
echo "::add-mask::${ORCA_RELAY_CONTROL_ACCESS_TOKEN}"
export ORCA_RELAY_CONTROL_ACCESS_TOKEN
node dev/scripts/operate-relay-cell-flags.mjs \
--mode "${MODE}" --cell-id "${CELL_ID}" \
--cell-origin "https://${CELL_ID#staging-gce-}.relay-staging.onorca.dev" \
--project-id "${GCP_PROJECT_ID}" \
--set "${FLAG_CHANGES}" \
--director-origin "https://relay-staging.onorca.dev" --confirm-reserve "${CONFIRM_RESERVE}" \
--break-glass "${BREAK_GLASS}" \
--expected-generation "${EXPECTED_GENERATION}" --confirmation "${CONFIRMATION}" \
| tee "${RUNNER_TEMP}/relay-cell-flags.jsonl"
- name: Audit summary
if: ${{ always() }}
run: |
{
echo "### Relay staging cell flags: ${MODE} ${CELL_ID}"
echo "Triggered by: ${GITHUB_TRIGGERING_ACTOR}, run ${GITHUB_RUN_ID} attempt ${GITHUB_RUN_ATTEMPT}"
echo '```json'
cat "${RUNNER_TEMP}/relay-cell-flags.jsonl" 2>/dev/null || echo '(no output)'
echo '```'
} >> "${GITHUB_STEP_SUMMARY}"
@@ -1,103 +0,0 @@
name: Power Relay Staging
on:
schedule:
# A zero-activity guard makes this a no-op when an internal test is still running.
- cron: '0 9 * * *'
workflow_dispatch:
inputs:
mode:
description: Inspect, wake, or sleep the staging Relay data plane
required: true
default: status
type: choice
options:
- status
- wake
- sleep
wake-cells:
description: Wake configured admission cells, or include disabled candidate cells
required: true
default: configured
type: choice
options:
- configured
- all
confirmation:
description: Enter WAKE_STAGING or SLEEP_STAGING for a manual mutation
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
power:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
POWER_MODE: ${{ github.event_name == 'schedule' && 'sleep' || inputs.mode }}
WAKE_CELLS: ${{ inputs.wake-cells || 'configured' }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Require explicit manual mutation confirmation
if: ${{ github.event_name == 'workflow_dispatch' && inputs.mode != 'status' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
if [[ "${POWER_MODE}" = "wake" ]]; then
test "${CONFIRMATION}" = "WAKE_STAGING"
else
test "${CONFIRMATION}" = "SLEEP_STAGING"
fi
- name: Read reviewed staging topology
run: |
node dev/scripts/infra.mjs init --env staging
terraform -chdir=infra/terraform output -json relay_gce_cell_deployments > "${RUNNER_TEMP}/relay-gce-topology.json"
- name: Inspect or change staging power state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/power-staging-relay.mjs \
--mode "${POWER_MODE}" \
--wake-cells "${WAKE_CELLS}" \
--topology-file "${RUNNER_TEMP}/relay-gce-topology.json"
@@ -1,497 +0,0 @@
name: Prove Relay Asia Staging
on:
workflow_dispatch:
inputs:
mode:
description: prove runs the fixed C4 proof; load runs the generator with free-form arguments
required: true
default: prove
type: choice
options: [prove, load]
image-digest:
description: Prove only. Exact immutable Relay digest deployed on staging C4
required: false
type: string
selector-generation:
description: Prove only. Exact selector generation with C4 migration-only
required: false
type: string
promote-attempt-id:
description: Prove only. Durable unique C4 promotion attempt ID
required: false
type: string
rollback-attempt-id:
description: Prove only. Durable unique C4 rollback attempt ID
required: false
type: string
load-region:
description: >-
Load only. asia-east2 runs on the Asia load runner; asia-east2-hosted asks for asia-east2
from a hosted runner; us-central1 runs on a hosted runner
required: false
default: asia-east2
type: choice
options: [asia-east2, asia-east2-hosted, us-central1]
load-shards:
description: Load only. Generator processes, each with its own shard of the 32 load principals
required: false
default: '4'
type: choice
options: ['1', '2', '3', '4']
load-runners:
description: >-
Load only, hosted runners only. Runner machines, each with load-shards generators; one
hosted runner holds about 2,000 connections to one cell
required: false
default: '1'
type: choice
options: ['1', '2']
load-args:
description: >-
Load only. Per-shard generator arguments, e.g. --controls 150 --splices 15
--reconnect-max-seconds 30 --duration-seconds 900. Origins, region, principals and
sharding are set by the workflow
required: false
default: ''
type: string
confirmation:
description: Enter PROVE_ASIA_STAGING (prove) or LOAD_RELAY_STAGING (load)
required: true
type: string
permissions:
contents: read
id-token: write
# A load run drives traffic only, so it must not queue behind (or hold up) the staging deploys
# and switch writes it is there to exercise.
concurrency:
group: ${{ inputs.mode == 'load' && 'relay-staging-load' || 'relay-staging-mutation' }}
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
prove:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') && inputs.mode == 'prove' }}
runs-on: [self-hosted, linux, x64, relay-asia-east2-load]
timeout-minutes: 75
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
AUTH_ORIGIN: https://auth-staging.onorca.dev
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact staging proof request
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
test "${CONFIRMATION}" = PROVE_ASIA_STAGING
[[ "${IMAGE_DIGEST}" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "${INITIAL_SELECTOR_GENERATION}" =~ ^[1-9][0-9]*$ ]]
[[ "${PROMOTE_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
[[ "${ROLLBACK_ATTEMPT_ID}" =~ ^[A-Za-z0-9_-]{8,128}$ ]]
test "${PROMOTE_ATTEMPT_ID}" != "${ROLLBACK_ATTEMPT_ID}"
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- name: Require the exact staging director image before promotion
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
runtime="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"
test "$(jq -r '.role' <<< "${runtime}")" = director
test "$(jq -r '.imageDigest' <<< "${runtime}")" = "${IMAGE_DIGEST}"
- name: Install exact load-harness dependencies
run: pnpm install --frozen-lockfile
- name: Build the Relay load-harness contract
run: pnpm --filter @orca-cloud/relay-contract build
- name: Promote only staging C4
id: promote
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode promote \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${result}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = general
echo "generation=${generation}" >> "${GITHUB_OUTPUT}"
echo "started_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "${GITHUB_OUTPUT}"
- name: Run sharded two-horizon and mixed splice proofs
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
mkdir -p "${proof_dir}"
fd_limit="$(ulimit -n)"
if test "${fd_limit}" != unlimited; then
[[ "${fd_limit}" =~ ^[0-9]+$ ]]
test "${fd_limit}" -ge 4096
fi
run_phase() {
phase="$1"
controls="$2"
splices="$3"
pids=()
stop_shards() {
for pid in "${pids[@]}"; do kill "${pid}" 2>/dev/null || true; done
for pid in "${pids[@]}"; do wait "${pid}" 2>/dev/null || true; done
}
trap stop_shards EXIT
for shard in 0 1 2 3; do
slow=0
wedged=0
boundary_args=()
request_unit_args=()
if test "${phase}" = launch && test "${shard}" = 0; then
slow=4
wedged=1
fi
if test "${phase}" = launch && test "${shard}" = 0; then
boundary_args=(
--region-behavior-probes 1
--capacity-cell-id staging-gce-c4
--capacity-cell-origin https://c4.relay-staging.onorca.dev
--capacity-unobserved-bound 60
--rebind-probes 2
--skip-rebind-overflow-check
)
fi
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region asia-east2 \
--relay-asia-load-principals 32 \
--controls "${controls}" \
--splices "${splices}" \
--slow-reader-splices "${slow}" \
--wedged-reader-splices "${wedged}" \
--capacity-hard-cap 3000 \
"${boundary_args[@]}" \
"${request_unit_args[@]}" \
--phase-barrier-dir "${proof_dir}/${phase}-barrier" \
--aggregate-controls "$((controls * 4))" \
--aggregate-splices "$((splices * 4))" \
--aggregate-reader-splices "$([[ "${phase}" = launch ]] && echo 5 || echo 0)" \
--aggregate-reader-bytes "$([[ "${phase}" = launch ]] && echo 12582912 || echo 0)" \
--required-lease-horizons 2 \
--splice-ramp-seconds 120 \
--max-generator-rss-growth-mib 512 \
--ramp-seconds 180 \
--duration-seconds 210 \
--shard-count 4 \
--shard-index "${shard}" \
> "${proof_dir}/${phase}-${shard}.jsonl" &
pids+=("$!")
done
failed=0
for pid in "${pids[@]}"; do
if ! wait "${pid}"; then failed=1; break; fi
done
if test "${failed}" = 1; then
stop_shards
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_progress" or .event == "relay_load_complete") |
{event, shardIndex, active, peakActive, connected, connectionFailures,
rampConnectionFailures, connectionFailuresByReason, unexpectedCloses,
protocolErrors, refreshErrors, socketErrors, elapsedSeconds}' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1 || true
done
trap - EXIT
return 1
fi
trap - EXIT
for shard in 0 1 2 3; do
jq -cer 'select(.event == "relay_load_complete")' \
"${proof_dir}/${phase}-${shard}.jsonl" | tail -n 1
done | jq -s . > "${proof_dir}/${phase}.json"
}
run_phase launch 5 5
- name: Collect and validate aggregate staging proof evidence
env:
PROOF_STARTED_AT: ${{ steps.promote.outputs.started_at }}
shell: bash
run: |
set -euo pipefail
proof_dir="${RUNNER_TEMP}/relay-asia-staging-proof"
ended_at="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
sleep 60
gcloud logging read \
"timestamp>=\"${PROOF_STARTED_AT}\" AND timestamp<=\"${ended_at}\" AND jsonPayload.event=\"orca_relay_runtime_metrics\"" \
--project "${GCP_PROJECT_ID}" --limit 20000 --format json \
> "${proof_dir}/runtime-metrics.json"
node dev/scripts/relay-asia-rollout-evidence.mjs create-staging \
--repository "${GITHUB_REPOSITORY}" \
--run-id "${GITHUB_RUN_ID}" \
--run-attempt "${GITHUB_RUN_ATTEMPT}" \
--commit-sha "${GITHUB_SHA}" \
--image-digest "${IMAGE_DIGEST}" \
--selector-generation "${{ steps.promote.outputs.generation }}" \
--started-at "${PROOF_STARTED_AT}" \
--ended-at "${ended_at}" \
--launch-report "${proof_dir}/launch.json" \
--logs-json "${proof_dir}/runtime-metrics.json" \
--output "${proof_dir}/evidence.json"
- name: Return staging C4 to migration-only
if: ${{ always() && steps.promote.outcome != 'skipped' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
- name: Upload immutable staging readiness evidence
if: ${{ success() }}
uses: actions/upload-artifact@v4
with:
name: relay-asia-staging-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/relay-asia-staging-proof/evidence.json
if-no-files-found: error
retention-days: 7
recover:
if: ${{ always() && github.ref == 'refs/heads/main' && inputs.mode == 'prove' }}
needs: prove
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 15
environment: staging
env:
IMAGE_DIGEST: ${{ inputs.image-digest }}
INITIAL_SELECTOR_GENERATION: ${{ inputs.selector-generation }}
PROMOTE_ATTEMPT_ID: ${{ inputs.promote-attempt-id }}
ROLLBACK_ATTEMPT_ID: ${{ inputs.rollback-attempt-id }}
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Recover staging C4 with a fresh identity
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
promoted="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode recover-promotion \
--cell-ids staging-gce-c4 \
--expected-generation "${INITIAL_SELECTOR_GENERATION}" \
--attempt-id "${PROMOTE_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
if test "$(jq -r '.promoted' <<< "${promoted}")" = false; then exit 0; fi
promoted_generation="$(jq -er '.generation' <<< "${promoted}")"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging \
--mode rollback \
--cell-ids staging-gce-c4 \
--expected-generation "${promoted_generation}" \
--attempt-id "${ROLLBACK_ATTEMPT_ID}" \
--image-digest "${IMAGE_DIGEST}")"
test "$(jq -r '.states["staging-gce-c4"]' <<< "${result}")" = migration-only
# The staging auth service mints relay load principals only for this workflow file, so the
# free-form staging load lives here rather than in a workflow of its own.
load:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.ref == 'refs/heads/main') && inputs.mode == 'load' }}
runs-on: ${{ inputs.load-region == 'asia-east2' && fromJSON('["self-hosted","linux","x64","relay-asia-east2-load"]') || 'ubuntu-latest' }}
strategy:
# One runner's failure must not drop the other half of the load mid-run.
fail-fast: false
matrix:
runner: ${{ fromJSON(inputs.load-runners == '2' && '[0, 1]' || '[0]') }}
timeout-minutes: 180
environment: staging
env:
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
AUTH_ORIGIN: https://auth-staging.onorca.dev
LOAD_REGION: ${{ inputs.load-region }}
LOAD_SHARDS: ${{ inputs.load-shards }}
LOAD_RUNNERS: ${{ inputs.load-runners }}
LOAD_RUNNER_INDEX: ${{ matrix.runner }}
LOAD_ARGS: ${{ inputs.load-args }}
steps:
- uses: actions/checkout@v4
- name: Validate the staging load request
shell: bash
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: |
set -euo pipefail
test "${CONFIRMATION}" = LOAD_RELAY_STAGING
[[ "${LOAD_REGION}" =~ ^(asia-east2|asia-east2-hosted|us-central1)$ ]]
[[ "${LOAD_SHARDS}" =~ ^[1-4]$ ]]
[[ "${LOAD_RUNNERS}" =~ ^[12]$ ]]
# The Asia runner is one machine; a second matrix job would only queue behind the first.
if test "${LOAD_RUNNERS}" != 1 && test "${LOAD_REGION}" = asia-east2; then
echo "load-runners 2 needs a hosted runner" >&2
exit 1
fi
read -r -a args <<< "${LOAD_ARGS}"
for arg in "${args[@]}"; do
[[ "${arg}" =~ ^[A-Za-z0-9=:/._,-]+$ ]] || { echo "unsupported characters in load-args" >&2; exit 1; }
case "${arg}" in
--target-origin|--director-origin|--auth-origin|--signing-key-file|--preferred-region|\
--relay-asia-load-principals|--shard-count|--shard-index|--phase-barrier-dir)
echo "${arg} is set by the workflow" >&2
exit 1
;;
esac
done
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_ASIA_PROOF_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: actions/setup-node@v4
with:
node-version: 24
- uses: pnpm/action-setup@v4
with:
package_json_file: cloud/package.json
- name: Install exact load-harness dependencies
run: pnpm install --frozen-lockfile
- name: Build the Relay load-harness contract
run: pnpm --filter @orca-cloud/relay-contract build
- name: Run the sharded staging load
env:
# Probes that read director capacity need an admin token; it lasts about an hour.
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
load_dir="${RUNNER_TEMP}/relay-staging-load"
mkdir -p "${load_dir}"
fd_limit="$(ulimit -n)"
if test "${fd_limit}" != unlimited; then
[[ "${fd_limit}" =~ ^[0-9]+$ ]]
test "${fd_limit}" -ge 4096
fi
read -r -a args <<< "${LOAD_ARGS}"
pids=()
stop_shards() {
for pid in "${pids[@]}"; do kill "${pid}" 2>/dev/null || true; done
for pid in "${pids[@]}"; do wait "${pid}" 2>/dev/null || true; done
}
trap stop_shards EXIT
for ((shard = 0; shard < LOAD_SHARDS; shard++)); do
node dev/scripts/load-relay-controls.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--auth-origin "${AUTH_ORIGIN}" \
--preferred-region "${LOAD_REGION%-hosted}" \
--relay-asia-load-principals 32 \
--shard-count "$((LOAD_SHARDS * LOAD_RUNNERS))" \
--shard-index "$((LOAD_RUNNER_INDEX * LOAD_SHARDS + shard))" \
"${args[@]}" \
> "${load_dir}/shard-${shard}.jsonl" &
pids+=("$!")
done
failed=0
for pid in "${pids[@]}"; do
if ! wait "${pid}"; then failed=1; fi
done
trap - EXIT
for ((shard = 0; shard < LOAD_SHARDS; shard++)); do
jq -c 'select(.event == "relay_load_complete" or .event == "relay_load_progress")' \
"${load_dir}/shard-${shard}.jsonl" | tail -n 1 || true
done
exit "${failed}"
- name: Upload the load reports
if: ${{ always() }}
uses: actions/upload-artifact@v4
with:
name: relay-staging-load-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.runner }}
path: ${{ runner.temp }}/relay-staging-load/
if-no-files-found: warn
retention-days: 7
@@ -1,926 +0,0 @@
name: Prove Relay Staging Capacity
on:
workflow_dispatch:
inputs:
mode:
description: Verify or change C3 capacity, restore admission, or refresh empty Asia C4
required: true
default: verify
type: choice
options:
- verify
- apply
- restore-admission
- refresh-asia-c4-image
expected-hard-cap:
description: Exact cap declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '600'
type: choice
options:
- '1000'
- '600'
expected-unobserved-bound:
description: Exact bound declared for staging-gce-c3 in the reviewed staging tfvars
required: true
default: '60'
type: choice
options:
- '60'
- '0'
confirmation:
description: Exact confirmation required for a mutation
required: false
type: string
expected-selector-generation:
description: Exact staging selector generation for a C4 image refresh
required: false
type: string
predecessor-image-digest:
description: Exact current C4 sha256 digest
required: false
type: string
target-image-digest:
description: Exact desired C4 sha256 digest
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
capacity:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && inputs.mode != 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_SERVICE_NAME: orca-cloud-relay-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c3.relay-staging.onorca.dev
# These drains are unpaced, but restart-safe still waits out the production pace window
# from the drain, the same one the same-cap drain uses.
RESTART_SAFE_PACE_WINDOW_MS: '300000'
TARGET_CELL_ID: staging-gce-c3
FALLBACK_CELL_ORIGIN: https://c2.relay-staging.onorca.dev
FALLBACK_CELL_ID: staging-gce-c2
CAPACITY_SERVICE_ACCOUNT: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
EXPECTED_HARD_CAP: ${{ inputs.expected-hard-cap }}
EXPECTED_UNOBSERVED_BOUND: ${{ inputs.expected-unobserved-bound }}
steps:
- uses: actions/checkout@v4
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
if: ${{ inputs.mode != 'restore-admission' }}
with:
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
if: ${{ inputs.mode != 'restore-admission' }}
run: node dev/scripts/infra.mjs init --env staging
- name: Require reviewed desired capacity and image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
CAP_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_hard_cap"
BOUND_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].connection_unobserved_bound"
IMAGE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].image"
ZONE_EXPRESSION="var.relay_gce_cells[\"${TARGET_CELL_ID}\"].zone"
DESIRED_CAP="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${CAP_EXPRESSION}")"
DESIRED_BOUND="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${BOUND_EXPRESSION}")"
DESIRED_IMAGE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${IMAGE_EXPRESSION}" | jq -r '.')"
TARGET_ZONE="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars <<< "${ZONE_EXPRESSION}" | jq -r '.')"
MIG_NAME="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
DESIRED_CELLS_JSON="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'local.relay_director_cells_json' | jq -r '.')"
CELL_ORIGIN="$(jq -r --arg cell "${TARGET_CELL_ID}" \
'.[] | select(.id == $cell) | .url' <<< "${DESIRED_CELLS_JSON}")"
test "${DESIRED_CAP}" = "${EXPECTED_HARD_CAP}"
test "${DESIRED_BOUND}" = "${EXPECTED_UNOBSERVED_BOUND}"
[[ "${DESIRED_IMAGE}" =~ @sha256:[0-9a-f]{64}$ ]]
[[ "${TARGET_ZONE}" =~ ^[a-z0-9-]+$ ]]
[[ "${MIG_NAME}" =~ ^[a-z0-9-]+$ ]]
test "${CELL_ORIGIN}" = "https://c3.relay-staging.onorca.dev"
jq -e \
--arg cell "${TARGET_CELL_ID}" \
--arg fallback "${FALLBACK_CELL_ID}" \
--argjson cap "${EXPECTED_HARD_CAP}" \
--argjson bound "${EXPECTED_UNOBSERVED_BOUND}" \
'(any(.[]; .id == $cell and .connectionHardCap == $cap and .connectionUnobservedBound == $bound)) and
(any(.[]; .id == $fallback and .connectionHardCap == 600 and .connectionUnobservedBound == 60))' \
<<< "${DESIRED_CELLS_JSON}" >/dev/null
echo "DESIRED_IMAGE=${DESIRED_IMAGE}" >> "${GITHUB_ENV}"
echo "TARGET_ZONE=${TARGET_ZONE}" >> "${GITHUB_ENV}"
echo "MIG_NAME=${MIG_NAME}" >> "${GITHUB_ENV}"
echo "CELL_ORIGIN=${CELL_ORIGIN}" >> "${GITHUB_ENV}"
echo "DESIRED_CELLS_JSON=${DESIRED_CELLS_JSON}" >> "${GITHUB_ENV}"
- name: Verify exact compatible director image
if: ${{ inputs.mode != 'restore-admission' }}
shell: bash
run: |
SERVICE_JSON="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
ACTIVE_REVISION="$(jq -r \
'[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end' \
<<< "${SERVICE_JSON}")"
test -n "${ACTIVE_REVISION}"
ACTIVE_IMAGE="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${ACTIVE_IMAGE}" = "${DESIRED_IMAGE}"
echo "ACTIVE_IMAGE=${ACTIVE_IMAGE}" >> "${GITHUB_ENV}"
- name: Require explicit transition confirmation
if: ${{ inputs.mode == 'apply' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "FENCE_AND_TRANSITION_STAGING_C3"
- name: Require explicit admission restore confirmation
if: ${{ inputs.mode == 'restore-admission' }}
env:
CONFIRMATION: ${{ inputs.confirmation }}
run: test "${CONFIRMATION}" = "RESTORE_STAGING_C2_C3_GENERAL"
- name: Require capacity identity and exact predecessor state
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
case "${EXPECTED_HARD_CAP}/${EXPECTED_UNOBSERVED_BOUND}" in
1000/0)
PREDECESSOR_C3_CAP=600
PREDECESSOR_C3_BOUND=60
;;
1000/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=0
;;
600/60)
PREDECESSOR_C3_CAP=1000
PREDECESSOR_C3_BOUND=60
;;
*)
echo "Unsupported staging capacity transition" >&2
exit 1
;;
esac
ACTIVE_REVISION="$(gcloud run services describe "${DIRECTOR_SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '
[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${ACTIVE_REVISION}"
CURRENT_CELLS_JSON="$(gcloud run revisions describe "${ACTIVE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -cer '
[.spec.containers[0].env[]? |
select(.name == "ORCA_RELAY_CELLS_JSON") | .value] |
if length == 1 then .[0] | fromjson else error("missing director topology") end')"
PREDECESSOR_CELLS_JSON="$(jq -ce \
--arg cell "${TARGET_CELL_ID}" \
--argjson cap "${PREDECESSOR_C3_CAP}" \
--argjson bound "${PREDECESSOR_C3_BOUND}" \
'map(if .id == $cell then . + {
connectionHardCap: $cap,
connectionUnobservedBound: $bound
} else . end)' <<< "${DESIRED_CELLS_JSON}")"
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
if jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${DESIRED_CELLS_JSON}" \
'$current == $expected'; then
if node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-active
elif node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed; then
TRANSITION_PHASE=cell-ready
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" --timeout-ms 480000
TRANSITION_PHASE=director-ready
fi
else
jq -ne \
--argjson current "${CURRENT_CELLS_JSON}" \
--argjson expected "${PREDECESSOR_CELLS_JSON}" \
'$current == $expected'
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${PREDECESSOR_C3_CAP}" \
--unobserved-bound "${PREDECESSOR_C3_BOUND}" \
--heartbeat fresh \
--admission either \
--draining either \
--activity allowed
TRANSITION_PHASE=predecessor
fi
echo "TRANSITION_PHASE=${TRANSITION_PHASE}" >> "${GITHUB_ENV}"
- name: Restore C2 as the safe placement fallback
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
- name: Require a healthy non-draining C3 before restoring it
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission either \
--draining forbidden \
--activity allowed
- name: Require a healthy general C2 before isolating C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-active; then
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${FALLBACK_CELL_ORIGIN}" \
--cell-id "${FALLBACK_CELL_ID}" \
--hard-cap 600 \
--unobserved-bound 60 \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Reversibly isolate and drain C3
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode isolate
- name: Verify restart-safe migration-only target
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
exit 0
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--heartbeat either \
--admission migration-only \
--draining required \
--activity restart-safe \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" --timeout-ms 480000
- name: Verify current capacity
if: ${{ inputs.mode == 'verify' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Deploy reviewed director topology and remove pre-protocol revisions
if: ${{ inputs.mode == 'apply' }}
run: |
if test "${TRANSITION_PHASE}" != predecessor; then
echo "DIRECTOR_CONFIG_CHANGED=false" >> "${GITHUB_ENV}"
exit 0
fi
RELEASE_ID="capacity-compat-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_SHA:0:8}"
DEPLOY_RESULT="$(node dev/scripts/deploy-relay-blue-green.mjs \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--service "${DIRECTOR_SERVICE_NAME}" \
--image "${ACTIVE_IMAGE}" \
--role director \
--capacity-service-account "${CAPACITY_SERVICE_ACCOUNT}" \
--capacity-cell-id "${TARGET_CELL_ID}" \
--director-cells-json "${DESIRED_CELLS_JSON}" \
--prune-revisions true \
--release-id "${RELEASE_ID}")"
echo "${DEPLOY_RESULT}"
DIRECTOR_CONFIG_CHANGED="$(jq -r '.topologyChanged' <<< "${DEPLOY_RESULT}")"
[[ "${DIRECTOR_CONFIG_CHANGED}" =~ ^(true|false)$ ]]
echo "DIRECTOR_CONFIG_CHANGED=${DIRECTOR_CONFIG_CHANGED}" >> "${GITHUB_ENV}"
- name: Require fail-closed director transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
if test "${TRANSITION_PHASE}" = cell-ready; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
exit 0
fi
HEARTBEAT_EXPECTATION=either
if test "${DIRECTOR_CONFIG_CHANGED}" = true; then
HEARTBEAT_EXPECTATION=stale
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat "${HEARTBEAT_EXPECTATION}" \
--admission migration-only \
--draining required \
--activity restart-safe \
--pace-window-ms "${RESTART_SAFE_PACE_WINDOW_MS}" --timeout-ms 480000
- name: Plan and apply only the exact empty cell
if: ${{ inputs.mode == 'apply' }}
shell: bash
run: |
recreate_fixed_one_instance() {
local instance
instance="$(gcloud compute instance-groups managed list-instances \
"${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--format=json \
| jq -er '
if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"
then .[0].instance | split("/") | last
else error("capacity cell does not have one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances \
"${MIG_NAME}" \
--instances "${instance}" \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--quiet
}
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c3"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c3"]' \
-out="${RUNNER_TEMP}/relay-capacity-cell.tfplan"
PLAN_RESULT="$(terraform -chdir=infra/terraform show -json \
"${RUNNER_TEMP}/relay-capacity-cell.tfplan" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode cell \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--image "${DESIRED_IMAGE}")"
echo "${PLAN_RESULT}"
CELL_PLAN_CHANGES="$(jq -r '.changes' <<< "${PLAN_RESULT}")"
[[ "${CELL_PLAN_CHANGES}" =~ ^(0|2)$ ]]
if test "${TRANSITION_PHASE}" = cell-ready; then
test "${CELL_PLAN_CHANGES}" = 0
elif test "${TRANSITION_PHASE}" = cell-active; then
test "${CELL_PLAN_CHANGES}" = 0
recreate_fixed_one_instance
elif test "${CELL_PLAN_CHANGES}" = 0; then
recreate_fixed_one_instance
else
terraform -chdir=infra/terraform apply \
-auto-approve "${RUNNER_TEMP}/relay-capacity-cell.tfplan"
fi
gcloud compute instance-groups managed wait-until \
"${MIG_NAME}" \
--stable \
--project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" \
--timeout 900
- name: Verify exact live cap and fresh matching heartbeat
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission migration-only \
--draining forbidden \
--activity allowed
- name: Make C3 the only staging placement cell
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode activate
- name: Verify the sole general canary after transition
if: ${{ inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Restore the reviewed C2 and C3 placement set
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore \
--general-cell-ids staging-gce-c2,staging-gce-c3
- name: Verify restored C3 admission and capacity
if: ${{ inputs.mode == 'restore-admission' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--hard-cap "${EXPECTED_HARD_CAP}" \
--unobserved-bound "${EXPECTED_UNOBSERVED_BOUND}" \
--heartbeat fresh \
--admission general \
--draining forbidden \
--activity allowed
- name: Preserve C2 as the safe fallback after a failed transition
if: ${{ failure() && inputs.mode == 'apply' }}
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.google-auth.outputs.id_token }}
run: |
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" \
--mode restore-fallback \
--general-cell-ids "${FALLBACK_CELL_ID}"
refresh-asia-c4-image:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && github.ref == 'refs/heads/main' && inputs.mode == 'refresh-asia-c4-image' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
env:
GCP_PROJECT_ID: onorca-cloud-staging
GCP_REGION: ${{ vars.STAGING_GCP_REGION }}
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
EXPECTED_SELECTOR_GENERATION: ${{ inputs.expected-selector-generation }}
PREDECESSOR_IMAGE_DIGEST: ${{ inputs.predecessor-image-digest }}
TARGET_IMAGE_DIGEST: ${{ inputs.target-image-digest }}
APPROVED_PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
steps:
- uses: actions/checkout@v4
- name: Require the exact bounded C4 refresh
env:
CONFIRMATION: ${{ inputs.confirmation }}
shell: bash
run: |
set -euo pipefail
test "${CONFIRMATION}" = REFRESH_STAGING_ASIA_C4_IMAGE
[[ "${EXPECTED_SELECTOR_GENERATION}" =~ ^(0|[1-9][0-9]*)$ ]]
[[ "${PREDECESSOR_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
[[ "${TARGET_IMAGE_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${PREDECESSOR_IMAGE_DIGEST}" != "${TARGET_IMAGE_DIGEST}"
test "${PREDECESSOR_IMAGE_DIGEST}" = "${APPROVED_PREDECESSOR_IMAGE_DIGEST}"
- id: google-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- name: Resolve the reviewed C4 image and shape
shell: bash
run: |
set -euo pipefail
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
shape="$(jq -cer --arg cell "${TARGET_CELL_ID}" '.[$cell]' <<< "${cells}")"
test "$(jq -r '.hostname' <<< "${shape}")" = c4
test "$(jq -r '.region' <<< "${shape}")" = asia-east2
test "$(jq -r '.zone' <<< "${shape}")" = asia-east2-a
test "$(jq -r '.machine_type' <<< "${shape}")" = e2-standard-4
test "$(jq -r '.capacity_requests' <<< "${shape}")" = 6000
test "$(jq -r '.database_pool_max' <<< "${shape}")" = 10
test "$(jq -r '.connection_hard_cap' <<< "${shape}")" = 3000
test "$(jq -r '.connection_unobserved_bound' <<< "${shape}")" = 60
test "$(jq -r '.initially_enabled' <<< "${shape}")" = false
desired_image="$(jq -r '.image' <<< "${shape}")"
test "${desired_image}" = \
"us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${desired_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${TARGET_IMAGE_DIGEST}"
mig_name="$(terraform -chdir=infra/terraform output -json relay_gce_cell_deployments \
| jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].mig_name')"
test "${mig_name}" = orca-cloud-staging-relay-gce-c4
{
echo "DESIRED_IMAGE=${desired_image}"
echo "ROLLBACK_IMAGE=us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay@${PREDECESSOR_IMAGE_DIGEST}"
echo "TARGET_ZONE=asia-east2-a"
echo "MIG_NAME=${mig_name}"
} >> "${GITHUB_ENV}"
- name: Save, validate, and classify the exact C4 plan
id: plan
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
case "$(jq -r '[.changes,.changeKind] | join(":")' <<< "${result}")" in
2:replacement) refresh_phase=predecessor ;;
0:none|*:obsolete-template-delete) refresh_phase=applied ;;
*:manager-convergence|*:replacement-with-obsolete-template) refresh_phase=converging ;;
*) exit 1 ;;
esac
{
echo "PLAN_CHANGES=$(jq -r '.changes' <<< "${result}")"
echo "REFRESH_PHASE=${refresh_phase}"
} >> "${GITHUB_ENV}"
- id: state-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the exact selector and current C4 state
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.state-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${inspect}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
expected_digests="${TARGET_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = predecessor; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST}"
elif test "${REFRESH_PHASE}" = converging; then
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
case ",${expected_digests}," in
*,"${current_digest}",*) ;;
*) exit 1 ;;
esac
source_incarnation=''
draining=forbidden
if test "${REFRESH_PHASE}" != applied; then
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
source_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
[[ "${source_incarnation}" =~ ^[0-9a-f-]{36}$ ]]
if test "$(jq -r '.draining' <<< "${runtime}")" = true; then
draining=required
fi
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining "${draining}" \
--activity quiescent --expected-image-digests "${expected_digests}"
runtime_available=true
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 300000
source_incarnation=''
runtime_available=false
fi
{
echo "MIG_STABLE_AT_MS=0"
echo "MUTATION_STARTED=false"
echo "REPLACEMENT_STARTED_AT_MS=0"
echo "RUNTIME_AVAILABLE=${runtime_available}"
echo "SOURCE_INCARNATION=${source_incarnation}"
} >> "${GITHUB_ENV}"
- id: fence-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 and prove it stayed empty before replacement
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
if test "${REFRESH_PHASE}" = applied; then exit 0; fi
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
if test "${RUNTIME_AVAILABLE}" = false; then exit 0; fi
node dev/scripts/prepare-relay-capacity-canary.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --mode isolate
fence_digests="${PREDECESSOR_IMAGE_DIGEST}"
if test "${REFRESH_PHASE}" = converging; then
fence_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
fi
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent --expected-image-digests "${fence_digests}"
- name: Apply the exact saved C4 plan
shell: bash
run: |
set -euo pipefail
if test "${PLAN_CHANGES}" = 0 && test "${RUNTIME_AVAILABLE}" = true; then exit 0; fi
if test "${REFRESH_PHASE}" = applied && test "${RUNTIME_AVAILABLE}" = false; then
echo "MUTATION_STARTED=true" >> "${GITHUB_ENV}"
fi
if test "${REFRESH_PHASE}" != applied; then
replacement_started_at_ms="$(date -u +%s%3N)"
echo "REPLACEMENT_STARTED_AT_MS=${replacement_started_at_ms}" >> "${GITHUB_ENV}"
fi
if test "${PLAN_CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve \
"${RUNNER_TEMP}/relay-c4-image-refresh.tfplan"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
if test "${RUNTIME_AVAILABLE}" = false && test "${REFRESH_PHASE}" = applied; then
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
fi
echo "MIG_STABLE_AT_MS=$(date -u +%s%3N)" >> "${GITHUB_ENV}"
- id: post-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify new C4 incarnation, image, and unchanged isolation
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.post-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${TARGET_IMAGE_DIGEST}" \
--timeout-ms 240000
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${EXPECTED_SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${TARGET_IMAGE_DIGEST}")"
test "$(jq -r '.generation' <<< "${result}")" = "${EXPECTED_SELECTOR_GENERATION}"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"; then break; fi
sleep 5
done
target_incarnation="$(jq -er '.status.runtime.cellIncarnation' <<< "${status}")"
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${MIG_STABLE_AT_MS}"
if test "${REFRESH_PHASE}" != applied; then
if test -n "${SOURCE_INCARNATION}"; then
test "${target_incarnation}" != "${SOURCE_INCARNATION}"
fi
test "$(jq -r '.status.runtime.startedAt' <<< "${status}")" \
-ge "${REPLACEMENT_STARTED_AT_MS}"
fi
- name: Require an empty targeted Terraform readback
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${DESIRED_IMAGE}" \
--rollback-image "${ROLLBACK_IMAGE}")"
test "$(jq -r '.changes' <<< "${result}")" = 0
@@ -1,133 +0,0 @@
name: Publish Relay Production Image
on:
workflow_dispatch:
inputs:
mode:
description: Publish a new production image or mirror an existing immutable image to staging
required: true
default: publish
type: choice
options: [publish, mirror-staging]
image-digest:
description: Exact existing production digest for mirror-staging mode
required: false
type: string
confirmation:
description: Enter MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING for mirror-staging mode
required: false
type: string
permissions:
contents: read
id-token: write
concurrency:
group: publish-relay-production
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
publish:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
REPOSITORY_ID: orca-cloud
IMAGE_NAME: relay
PUBLISH_MODE: ${{ inputs.mode }}
MIRROR_DIGEST: ${{ inputs.image-digest }}
MIRROR_CONFIRMATION: ${{ inputs.confirmation }}
steps:
- uses: actions/checkout@v4
- name: Validate the exact publish request before authentication
shell: bash
run: |
set -euo pipefail
if test "${PUBLISH_MODE}" = mirror-staging; then
[[ "${MIRROR_DIGEST}" =~ ^sha256:[a-f0-9]{64}$ ]]
test "${MIRROR_CONFIRMATION}" = MIRROR_RELAY_PRODUCTION_IMAGE_TO_STAGING
else
test "${PUBLISH_MODE}" = publish
test -z "${MIRROR_DIGEST}"
test -z "${MIRROR_CONFIRMATION}"
fi
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_RELAY_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: docker/setup-buildx-action@v3
- name: Configure Docker auth
run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet
- name: Build and publish immutable image
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}:sha-${GITHUB_SHA}"
docker build -f apps/relay/Dockerfile -t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${DIGEST}"
{
echo '### Terraform candidate image'
echo
echo "\`${IMAGE}\`"
echo
echo 'Declare this digest on a distinct disabled candidate cell in a reviewed Terraform PR.'
} >> "${GITHUB_STEP_SUMMARY}"
- name: Build and publish immutable fence broker
if: ${{ inputs.mode == 'publish' }}
run: |
IMAGE_TAG="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker:sha-${GITHUB_SHA}"
docker build \
--build-arg "ORCA_RELAY_FENCE_IMAGE_COMMIT=${GITHUB_SHA}" \
-f apps/relay-fence-broker/Dockerfile \
-t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"
DIGEST="$(gcloud artifacts docker images describe "${IMAGE_TAG}" --format='value(image_summary.digest)')"
test -n "${DIGEST}"
IMAGE="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/relay-fence-broker@${DIGEST}"
{
echo
echo '### Terraform fence broker image'
echo
echo "\`${IMAGE}\`"
} >> "${GITHUB_STEP_SUMMARY}"
- name: Mirror the exact production manifest to staging
if: ${{ inputs.mode == 'mirror-staging' }}
shell: bash
run: |
set -euo pipefail
source_image="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${MIRROR_DIGEST}"
target_tag="${GCP_REGION}-docker.pkg.dev/onorca-cloud-staging/${REPOSITORY_ID}/${IMAGE_NAME}:production-${MIRROR_DIGEST#sha256:}"
source_digest="$(gcloud artifacts docker images describe "${source_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${source_digest}" = "${MIRROR_DIGEST}"
docker pull "${source_image}"
docker tag "${source_image}" "${target_tag}"
docker push "${target_tag}"
target_digest="$(gcloud artifacts docker images describe "${target_tag}" \
--project onorca-cloud-staging --format='value(image_summary.digest)')"
test "${target_digest}" = "${MIRROR_DIGEST}"
{
echo '### Mirrored Relay image'
echo
printf 'Production and staging now resolve the same immutable digest: %s.\n' \
"${MIRROR_DIGEST}"
} >> "${GITHUB_STEP_SUMMARY}"
-568
View File
@@ -1,568 +0,0 @@
name: Deploy Push Gateway Production
on:
workflow_dispatch:
inputs:
source_sha:
description: Full reviewed commit SHA to build (feature may remain unmerged)
required: true
type: string
confirmation:
description: Enter DEPLOY_PUSH_GATEWAY to shift production traffic
required: true
type: string
permissions:
contents: read
id-token: write
# Serialize push traffic changes independently of Relay and the shared database.
concurrency:
group: production-push-rollout
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
deploy:
if: >-
${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' &&
github.ref == 'refs/heads/main' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
environment: production
env:
GCP_PROJECT_ID: onorca-cloud
GCP_REGION: ${{ vars.PRODUCTION_GCP_REGION }}
SERVICE_NAME: orca-cloud-push
REPOSITORY_ID: orca-cloud
IMAGE_NAME: push
PUSH_ORIGIN: https://push.onorca.dev
PUSH_RUNTIME_SERVICE_ACCOUNT: orca-cloud-push@onorca-cloud.iam.gserviceaccount.com
# Scaling the serving revision must already hold, matching push_min_instances and
# push_max_instances. Terraform owns both, and the candidate inherits them from the
# service, so this deploy never passes a scaling flag: doing so would write a
# Terraform-owned field that `lifecycle.ignore_changes` does not cover, and a later
# `push_max_instances` raise would then be reverted by every deploy. These two values
# are the expected shape, asserted before the candidate is created and again on the
# candidate itself, so a deploy that would change the gateway's Cloud SQL draw fails.
PUSH_MIN_INSTANCES: 1
PUSH_MAX_INSTANCES: 2
CONFIRMATION: ${{ inputs.confirmation }}
SOURCE_SHA: ${{ inputs.source_sha }}
steps:
- uses: actions/checkout@v4
- name: Require the explicit deploy confirmation
shell: bash
run: |
set -euo pipefail
test "${CONFIRMATION}" = DEPLOY_PUSH_GATEWAY
[[ "${SOURCE_SHA}" =~ ^[a-f0-9]{40}$ ]]
# Keep the workflow and rollout lease on main; only the Docker build uses candidate code.
- name: Fetch the immutable gateway source
shell: bash
run: |
set -euo pipefail
git fetch --no-tags origin "${SOURCE_SHA}"
test "$(git rev-parse FETCH_HEAD)" = "${SOURCE_SHA}"
mkdir -p "${RUNNER_TEMP}/push-source"
git -C "${GITHUB_WORKSPACE}" archive "${SOURCE_SHA}" cloud \
| tar -x -C "${RUNNER_TEMP}/push-source"
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.PRODUCTION_GCP_PUSH_DEPLOY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.PRODUCTION_GCP_PUSH_DEPLOY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: docker/setup-buildx-action@v3
- name: Configure Docker auth
run: gcloud auth configure-docker "${GCP_REGION}-docker.pkg.dev" --quiet
# Building an image does not need the deployment lease.
- name: Build and publish the immutable gateway image
shell: bash
run: |
set -euo pipefail
image_tag="${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}:sha-${SOURCE_SHA}"
docker buildx build --push --platform linux/amd64 --provenance=false --metadata-file "${RUNNER_TEMP}/push-image.json" \
-f "${RUNNER_TEMP}/push-source/cloud/apps/push/Dockerfile" \
-t "${image_tag}" "${RUNNER_TEMP}/push-source/cloud"
digest="$(jq -er '."containerimage.digest"' "${RUNNER_TEMP}/push-image.json")"
[[ "${digest}" =~ ^sha256:[a-f0-9]{64}$ ]]
echo "IMAGE=${GCP_REGION}-docker.pkg.dev/${GCP_PROJECT_ID}/${REPOSITORY_ID}/${IMAGE_NAME}@${digest}" \
>> "${GITHUB_ENV}"
echo "IMAGE_DIGEST=${digest}" >> "${GITHUB_ENV}"
# Refuse older images before they ever boot against production.
- name: Require image support for inert validation
shell: bash
run: |
set -euo pipefail
docker run --rm --network none --entrypoint node "${IMAGE}" --input-type=module -e '
import { loadPushConfig } from "./apps/push/dist/config.js";
const env = { ORCA_PUSH_PUBLIC_URL: "https://push.onorca.dev", ORCA_PUSH_MODE: "validation", ORCA_PUSH_FCM_PROJECT_ID: "onorca-cloud" };
if (loadPushConfig(env).mode !== "validation") throw new Error("validation_mode_unsupported");
let rejected = false;
try { loadPushConfig({ ...env, ORCA_PUSH_MODE: "invalid" }); } catch { rejected = true; }
if (!rejected) throw new Error("validation_mode_not_fail_closed");
'
# Held across the deploy, not just a separate schema step: the gateway opens its pool and
# applies its schema while the new revision starts, so the revision is the schema step.
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-terraform-state
object: terraform/state/push-rollout/production.lock
# Why: the candidate inherits the serving revision's scaling. A serving revision that has
# drifted below the floor would hand the candidate a cold start on every notification, and
# one that has drifted above the ceiling would hand it a larger Cloud SQL draw than the
# rollout lease was taken for. Refuse to inherit either rather than latch it.
- name: Record the serving revision and require its Terraform-owned scaling
shell: bash
run: |
set -euo pipefail
serving="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test -n "${serving}"
revisions="$(gcloud run revisions list --service "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(metadata.name)')"
if test "${revisions}" != "${serving}"; then
echo 'Retire leftover revisions under the rollout lease before deploying; three pools are the limit.' >&2
exit 1
fi
floor="$(gcloud run revisions describe "${serving}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/minScale'])")"
if [[ "${floor:-0}" -lt "${PUSH_MIN_INSTANCES}" ]]; then
echo "serving revision ${serving} holds ${floor:-0} minimum instances," \
"below ${PUSH_MIN_INSTANCES}; deploying would inherit and latch it." >&2
echo "Restore the floor first: gcloud run services update ${SERVICE_NAME}" \
"--region ${GCP_REGION} --min-instances=${PUSH_MIN_INSTANCES}" >&2
exit 1
fi
ceiling="$(gcloud run revisions describe "${serving}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${ceiling}" = "${PUSH_MAX_INSTANCES}"
echo "serving revision ${serving} holds ${floor} minimum and ${ceiling} maximum instances"
echo "ROLLBACK_REVISION=${serving}" >> "${GITHUB_ENV}"
gcloud run revisions describe "${serving}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
> "${RUNNER_TEMP}/push-rollback-revision.json"
image="$(jq -er '.status.imageDigest' "${RUNNER_TEMP}/push-rollback-revision.json")"
[[ "${image}" =~ @sha256:[a-f0-9]{64}$ ]]
echo "ROLLBACK_IMAGE=${image}" >> "${GITHUB_ENV}"
jq -e 'all(.spec.containers[0].env[]?; .name != "ORCA_PUSH_MODE" or .value == "active")' \
"${RUNNER_TEMP}/push-rollback-revision.json" > /dev/null
# Validation has no schema writes, HTTP mutations, worker, or pruners; tags alone do not
# isolate background consumers from production.
- name: Deploy the candidate revision with no traffic
shell: bash
run: |
set -euo pipefail
tag="c${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
{
echo "VALIDATION_DEPLOY_ATTEMPTED=true"
echo "VALIDATION_REVISION=${SERVICE_NAME}-${tag}"
echo "VALIDATION_TAG=${tag}"
echo "CANDIDATE_TAG=${tag}"
echo "CANDIDATE_REVISION=${SERVICE_NAME}-${tag}"
} >> "${GITHUB_ENV}"
gcloud run deploy "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--image "${IMAGE}" \
--tag "${tag}" \
--revision-suffix "${tag}" \
--no-traffic \
--update-env-vars ORCA_PUSH_MODE=validation \
--quiet
candidate="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er --arg tag "${tag}" \
'[.status.traffic[] | select(.tag == $tag)]
| if length == 1 then .[0] else error("tagged candidate is not unique") end')"
test "$(jq -r '.revisionName' <<< "${candidate}")" = "${SERVICE_NAME}-${tag}"
echo "CANDIDATE_URL=$(jq -r '.url' <<< "${candidate}")" >> "${GITHUB_ENV}"
# A tagged revision is directly addressable and sits outside the service-wide cap, so the
# candidate and the serving revision each draw up to the ceiling during the probe window.
# Successor creation later requires three revision pools; assert the inherited ceiling.
- name: Require the candidate to serve the exact image and inherited scaling
shell: bash
run: |
set -euo pipefail
served="$(gcloud run revisions describe "${CANDIDATE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format='value(spec.containers[0].image)')"
test "${served}" = "${IMAGE}"
test "${CANDIDATE_REVISION}" != "${ROLLBACK_REVISION}"
candidate_ceiling="$(gcloud run revisions describe "${CANDIDATE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format="value(metadata.annotations['autoscaling.knative.dev/maxScale'])")"
test "${candidate_ceiling}" = "${PUSH_MAX_INSTANCES}"
- name: Probe the candidate readiness endpoint
shell: bash
run: |
set -euo pipefail
[[ "${CANDIDATE_URL}" =~ ^https://[^/]+$ ]]
for attempt in $(seq 1 30); do
code="$(curl -sS -o "${RUNNER_TEMP}/push-ready.json" -w '%{http_code}' \
--max-time 10 "${CANDIDATE_URL}/ready" || true)"
if test "${code}" = 200; then
jq -e . < "${RUNNER_TEMP}/push-ready.json" > /dev/null
curl --fail --silent --show-error --max-time 10 "${CANDIDATE_URL}/health" \
| jq -e '.ok == true and .deliveryProtocol == 2 and .mode == "validation"' > /dev/null
echo "candidate ${CANDIDATE_REVISION} is ready after ${attempt} attempt(s)"
exit 0
fi
echo "attempt ${attempt}: /ready returned ${code}"
sleep 5
done
echo "candidate ${CANDIDATE_REVISION} never reported ready" >&2
exit 1
# Why: a gateway that boots and answers /ready can still be unable to send. This proves the
# runtime account's FCM grant end to end without delivering anything: validate_only stops
# Google before any push, and the deliberately invalid token means a healthy credential
# answers INVALID_ARGUMENT. PERMISSION_DENIED is the failure this step exists to catch.
#
# Only the four verdicts below are conclusive. A 429, a 5xx, or a transport failure says
# nothing about the credential, so it is retried rather than treated as either answer; a
# denied credential still fails on the first attempt, without burning the retries.
- name: Prove the runtime identity can reach FCM
shell: bash
run: |
set -euo pipefail
token="$(gcloud auth print-access-token \
--impersonate-service-account "${PUSH_RUNTIME_SERVICE_ACCOUNT}")"
test -n "${token}"
echo "::add-mask::${token}"
body='{"validate_only":true,"message":{"token":"orca-push-deploy-probe-invalid-token","notification":{"title":"Orca","body":"deploy probe"}}}'
for attempt in $(seq 1 5); do
code="$(curl -sS -o "${RUNNER_TEMP}/push-fcm.json" -w '%{http_code}' --max-time 20 \
-X POST "https://fcm.googleapis.com/v1/projects/${GCP_PROJECT_ID}/messages:send" \
-H "Authorization: Bearer ${token}" \
-H 'Content-Type: application/json' \
--data "${body}" || true)"
status="$(jq -r '.error.status // empty' < "${RUNNER_TEMP}/push-fcm.json" || true)"
echo "attempt ${attempt}: FCM validate-only send returned HTTP ${code} status ${status:-OK}"
if test "${status}" = PERMISSION_DENIED || test "${status}" = INVALID_ARGUMENT ||
test "${code}" = 401 || test "${code}" = 403; then
break
fi
sleep 5
done
if test "${status}" = PERMISSION_DENIED || test "${code}" = 401 || test "${code}" = 403; then
echo "the push runtime identity cannot send through FCM" >&2
exit 1
fi
test "${status}" = INVALID_ARGUMENT
# Cloud Run requires a successor before the latest revision can be deleted.
- name: Retire inert validation and activate the verified image
shell: bash
run: |
set -euo pipefail
tag="a${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
{
echo "CANDIDATE_TAG=${tag}"
echo "CANDIDATE_REVISION=${SERVICE_NAME}-${tag}"
echo "ACTIVATION_ATTEMPTED=true"
} >> "${GITHUB_ENV}"
# This is the production-effect boundary: schema, pruners and workers start here.
gcloud run deploy "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--image "${IMAGE}" --tag "${tag}" --revision-suffix "${tag}" \
--remove-env-vars ORCA_PUSH_MODE --no-traffic --quiet
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--remove-tags "${VALIDATION_TAG}" --quiet
gcloud run revisions delete "${VALIDATION_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --quiet
echo "VALIDATION_RETIRED=true" >> "${GITHUB_ENV}"
revision="$(gcloud run revisions describe "${SERVICE_NAME}-${tag}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json)"
jq -e --arg image "${IMAGE}" --arg account "${PUSH_RUNTIME_SERVICE_ACCOUNT}" \
--arg ceiling "${PUSH_MAX_INSTANCES}" --arg floor "${PUSH_MIN_INSTANCES}" \
--slurpfile prior "${RUNNER_TEMP}/push-rollback-revision.json" '
def shape: del(.containers[0].image) |
.containers[0].env = ((.containers[0].env // []) |
map(select(.name != "ORCA_PUSH_MODE")) | sort_by(.name));
.spec.containers[0].image == $image and .spec.serviceAccountName == $account and
all(.spec.containers[0].env[]?; .name != "ORCA_PUSH_MODE") and
(.spec | shape) == ($prior[0].spec | shape) and
.metadata.annotations["autoscaling.knative.dev/maxScale"] == $ceiling and
(.metadata.annotations["autoscaling.knative.dev/minScale"] | tonumber) >= ($floor | tonumber)' \
<<< "${revision}" > /dev/null
candidate="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er --arg tag "${tag}" '[.status.traffic[] | select(.tag == $tag)]
| if length == 1 then .[0] else error("active candidate is not unique") end')"
test "$(jq -r '.revisionName' <<< "${candidate}")" = "${SERVICE_NAME}-${tag}"
url="$(jq -er '.url' <<< "${candidate}")"
[[ "${url}" =~ ^https://[^/]+$ ]]
curl --fail --silent --show-error --max-time 10 "${url}/ready" | jq -e '.ok == true'
curl --fail --silent --show-error --max-time 10 "${url}/health" \
| jq -e '.ok == true and .deliveryProtocol == 2 and .mode == "active"'
- name: Shift all traffic to the verified candidate
shell: bash
run: |
set -euo pipefail
echo "TRAFFIC_SHIFT_ATTEMPTED=true" >> "${GITHUB_ENV}"
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--to-revisions "${CANDIDATE_REVISION}=100" \
--quiet
serving="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test "${serving}" = "${CANDIDATE_REVISION}"
echo "TRAFFIC_SHIFTED=true" >> "${GITHUB_ENV}"
# Why: the summary is written before the origin check, not after it. Once traffic has
# moved, the rollback target is the single thing an operator needs, and a summary that only
# appeared on success would be missing in exactly the run that needs it.
- name: Publish the rollout summary
if: ${{ always() && env.CANDIDATE_REVISION != '' && env.ROLLBACK_REVISION != '' }}
shell: bash
run: |
set -euo pipefail
{
echo '### Push gateway rollout'
echo
echo "Source: ${SOURCE_SHA}"
echo
echo "Revision: \`${CANDIDATE_REVISION}\`"
echo
echo "Image: \`${IMAGE_DIGEST}\`"
echo "Known-good image: \`${ROLLBACK_IMAGE}\`"
echo
echo "Recovery: deploy \`${ROLLBACK_IMAGE}\` as a new revision with" \
"\`--remove-env-vars ORCA_PUSH_MODE --no-traffic --tag <unique-recovery-tag> --revision-suffix <unique-recovery-suffix>\`."
echo 'Verify its exact digest, configuration, readiness and active mode, then promote and check the public origin.'
echo 'Only then remove obsolete tags and delete rejected/previous revisions; never delete the latest revision.'
echo 'The previous revision is retired after public checks; retain this immutable image for recovery under the rollout lease.'
echo "Activation attempted: ${ACTIVATION_ATTEMPTED:-false}; traffic rollback cannot undo schema or deliveries."
} >> "${GITHUB_STEP_SUMMARY}"
- name: Verify the public origin after the shift
shell: bash
run: |
set -euo pipefail
for attempt in $(seq 1 30); do
code="$(curl -sS -o /dev/null -w '%{http_code}' --max-time 10 \
"${PUSH_ORIGIN}/ready" || true)"
if test "${code}" = 200; then
curl --fail --silent --show-error --max-time 10 "${PUSH_ORIGIN}/health" \
| jq -e '.ok == true and .deliveryProtocol == 2 and .mode == "active"' > /dev/null
echo "ROLLOUT_VERIFIED=true" >> "${GITHUB_ENV}"
echo "${PUSH_ORIGIN} is ready after ${attempt} attempt(s)"
exit 0
fi
echo "attempt ${attempt}: ${PUSH_ORIGIN}/ready returned ${code}"
sleep 5
done
echo "${PUSH_ORIGIN} never reported ready after the shift" >&2
exit 1
# Why: everything after the shift runs with production on the candidate. A failure there
# is not a failure to deploy, it is a live gateway that has to go back, so the traffic move
# is undone here rather than left to whoever reads the run.
- name: Roll traffic back to the previous revision
if: ${{ (failure() || cancelled()) && env.TRAFFIC_SHIFT_ATTEMPTED == 'true' && env.ROLLOUT_VERIFIED != 'true' }}
shell: bash
run: |
set -euo pipefail
test -n "${ROLLBACK_REVISION:-}"
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--to-revisions "${ROLLBACK_REVISION}=100" \
--quiet
serving="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -r '[.status.traffic[] | select((.percent // 0) > 0)]
| if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test "${serving}" = "${ROLLBACK_REVISION}"
echo "TRAFFIC_ROLLED_BACK=true" >> "${GITHUB_ENV}"
{
echo
echo '### Push gateway rolled back'
echo
echo "Traffic returned to \`${ROLLBACK_REVISION}\`; the candidate" \
"\`${CANDIDATE_REVISION}\` no longer serves HTTP; deletion below must stop its workers."
} >> "${GITHUB_STEP_SUMMARY}"
# Deleting a revision does not restore the service template that Terraform reconciles.
- name: Restore the known-good service template
if: ${{ (failure() || cancelled()) && env.VALIDATION_DEPLOY_ATTEMPTED == 'true' && env.ROLLOUT_VERIFIED != 'true' }}
shell: bash
run: |
set -euo pipefail
test "${TRAFFIC_SHIFT_ATTEMPTED:-false}" != true || test "${TRAFFIC_ROLLED_BACK:-false}" = true
test -n "${ROLLBACK_IMAGE}"
# A partial activation may leave validation plus active; free one slot before recovery.
latest="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--format='value(status.latestCreatedRevisionName)')"
test -n "${latest}"
if test "${VALIDATION_RETIRED:-false}" != true && test "${latest}" != "${VALIDATION_REVISION}"; then
existing="$(gcloud run revisions list --service "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--filter "metadata.name=${VALIDATION_REVISION}" --format='value(metadata.name)')"
if test -n "${existing}"; then
test "${existing}" = "${VALIDATION_REVISION}"
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--remove-tags "${VALIDATION_TAG}" --quiet
gcloud run revisions delete "${VALIDATION_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --quiet
fi
fi
tag="r${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
echo "RECOVERY_TAG=${tag}" >> "${GITHUB_ENV}"
echo "TEMPLATE_RECOVERY_REVISION=${SERVICE_NAME}-${tag}" >> "${GITHUB_ENV}"
echo "Template recovery attempted: ${SERVICE_NAME}-${tag}, image ${ROLLBACK_IMAGE}." \
>> "${GITHUB_STEP_SUMMARY}"
# Known-good schema/workers can run here even though HTTP stays on the old revision.
gcloud run deploy "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--image "${ROLLBACK_IMAGE}" --revision-suffix "${tag}" --tag "${tag}" \
--remove-env-vars ORCA_PUSH_MODE --no-traffic --quiet
gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
> "${RUNNER_TEMP}/push-recovered-service.json"
gcloud run revisions describe "${SERVICE_NAME}-${tag}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
> "${RUNNER_TEMP}/push-recovery-revision.json"
jq -e --arg image "${ROLLBACK_IMAGE}" --arg serving "${ROLLBACK_REVISION}" \
--arg floor "${PUSH_MIN_INSTANCES}" --arg ceiling "${PUSH_MAX_INSTANCES}" \
--slurpfile prior "${RUNNER_TEMP}/push-rollback-revision.json" \
--slurpfile recovered "${RUNNER_TEMP}/push-recovery-revision.json" '
def shape: del(.containers[0].image) |
.containers[0].env = ((.containers[0].env // []) |
map(select(.name != "ORCA_PUSH_MODE")) | sort_by(.name));
.spec.template.spec.containers[0].image == $image and
all(.spec.template.spec.containers[0].env[]?; .name != "ORCA_PUSH_MODE") and
$recovered[0].spec.containers[0].image == $image and
all($recovered[0].spec.containers[0].env[]?; .name != "ORCA_PUSH_MODE") and
($recovered[0].spec | shape) == ($prior[0].spec | shape) and
.spec.template.metadata.annotations["autoscaling.knative.dev/maxScale"] == $ceiling and
(.spec.template.metadata.annotations["autoscaling.knative.dev/minScale"] | tonumber) >= ($floor | tonumber) and
([.status.traffic[] | select((.percent // 0) > 0)] |
length == 1 and .[0].revisionName == $serving and .[0].percent == 100)
' "${RUNNER_TEMP}/push-recovered-service.json" > /dev/null
echo "TEMPLATE_RESTORED=true" >> "${GITHUB_ENV}"
echo 'Known-good image and normal mode restored; previous revision still serves HTTP.' \
>> "${GITHUB_STEP_SUMMARY}"
- name: Promote and verify the known-good recovery revision
if: ${{ always() && env.TEMPLATE_RESTORED == 'true' }}
shell: bash
run: |
set -euo pipefail
url="$(jq -er --arg tag "${RECOVERY_TAG}" \
'.status.traffic[] | select(.tag == $tag) | .url' "${RUNNER_TEMP}/push-recovered-service.json")"
[[ "${url}" =~ ^https://[^/]+$ ]]
curl --fail --silent --show-error --max-time 10 "${url}/ready" | jq -e '.ok == true'
curl --fail --silent --show-error --max-time 10 "${url}/health" \
| jq -e '.ok == true and .deliveryProtocol == 2 and .mode == "active"'
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--to-revisions "${TEMPLATE_RECOVERY_REVISION}=100" --quiet
serving="$(gcloud run services describe "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format=json \
| jq -er '[.status.traffic[] | select((.percent // 0) > 0)] |
if length == 1 and .[0].percent == 100 then .[0].revisionName else empty end')"
test "${serving}" = "${TEMPLATE_RECOVERY_REVISION}"
curl --fail --silent --show-error --max-time 10 "${PUSH_ORIGIN}/ready" | jq -e '.ok == true'
curl --fail --silent --show-error --max-time 10 "${PUSH_ORIGIN}/health" \
| jq -e '.ok == true and .deliveryProtocol == 2 and .mode == "active"'
echo "RECOVERY_VERIFIED=true" >> "${GITHUB_ENV}"
echo "Recovery revision ${TEMPLATE_RECOVERY_REVISION} now serves the known-good image." \
>> "${GITHUB_STEP_SUMMARY}"
- name: Delete the rejected candidate revision
if: ${{ always() && env.RECOVERY_VERIFIED == 'true' }}
shell: bash
run: |
set -euo pipefail
if test -z "${CANDIDATE_REVISION:-}"; then
echo "CANDIDATE_DELETED=true" >> "${GITHUB_ENV}"
exit 0
fi
if test -n "${CANDIDATE_TAG:-}"; then
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--remove-tags "${CANDIDATE_TAG}" \
--quiet
echo "CANDIDATE_TAG=" >> "${GITHUB_ENV}"
fi
existing="$(gcloud run revisions list --service "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" \
--filter "metadata.name=${CANDIDATE_REVISION}" --format='value(metadata.name)')"
if test -n "${existing}"; then
test "${existing}" = "${CANDIDATE_REVISION}"
gcloud run revisions delete "${CANDIDATE_REVISION}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --quiet
fi
echo "CANDIDATE_DELETED=true" >> "${GITHUB_ENV}"
echo "candidate revision ${CANDIDATE_REVISION} is absent"
# Public checks commit the serving revision; cleanup failures must not roll it back.
- name: Retire previous consumers after public checks
if: ${{ always() && (env.ROLLOUT_VERIFIED == 'true' || (env.RECOVERY_VERIFIED == 'true' && env.CANDIDATE_DELETED == 'true')) }}
shell: bash
run: |
set -euo pipefail
serving="${CANDIDATE_REVISION}"
if test "${RECOVERY_VERIFIED:-false}" = true; then
serving="${TEMPLATE_RECOVERY_REVISION}"
fi
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --clear-tags --quiet
revisions="$(gcloud run revisions list --service "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --format='value(metadata.name)')"
while IFS= read -r revision; do
test -n "${revision}" || continue
test "${revision}" != "${serving}" || continue
case "${revision}" in
"${ROLLBACK_REVISION}"|"${VALIDATION_REVISION}"|"${CANDIDATE_REVISION}") ;;
*) echo "Unexpected revision ${revision}; manual retirement required." >&2; exit 1 ;;
esac
gcloud run revisions delete "${revision}" \
--project "${GCP_PROJECT_ID}" --region "${GCP_REGION}" --quiet
done <<< "${revisions}"
echo "CANDIDATE_TAG=" >> "${GITHUB_ENV}"
echo 'Obsolete revision resources retired; verify actual SQL session drain operationally.' \
>> "${GITHUB_STEP_SUMMARY}"
- name: Drop the candidate traffic tag
if: always()
shell: bash
run: |
set -euo pipefail
test -n "${CANDIDATE_TAG:-}" || exit 0
gcloud run services update-traffic "${SERVICE_NAME}" \
--project "${GCP_PROJECT_ID}" \
--region "${GCP_REGION}" \
--remove-tags "${CANDIDATE_TAG}" \
--quiet
@@ -1,413 +0,0 @@
name: Recover Relay Staging C4 Image
on:
workflow_run:
workflows: [Prove Relay Staging Capacity]
types: [completed]
workflow_dispatch:
inputs:
confirmation:
description: Enter RECOVER_STAGING_ASIA_C4_IMAGE
required: true
type: string
permissions:
actions: read
contents: read
id-token: write
defaults:
run:
working-directory: cloud
jobs:
gate:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event_name == 'workflow_dispatch' || (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion != 'success')) }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
outputs:
recover: ${{ steps.trigger.outputs.recover }}
steps:
- name: Bind recovery to the exact failed C4 job
working-directory: .
id: trigger
env:
CONFIRMATION: ${{ inputs.confirmation }}
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
SOURCE_RUN_EVENT: ${{ github.event.workflow_run.event }}
shell: bash
run: |
set -euo pipefail
if test "${GITHUB_EVENT_NAME}" = workflow_dispatch; then
test "${CONFIRMATION}" = RECOVER_STAGING_ASIA_C4_IMAGE
echo "recover=true" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${SOURCE_RUN_EVENT}" = workflow_dispatch
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "refresh-asia-c4-image" and
(.conclusion == "failure" or .conclusion == "cancelled" or
.conclusion == "timed_out"))] | length' <<< "${jobs}")"
if test "${count}" = 0; then
echo "recover=false" >> "${GITHUB_OUTPUT}"
exit 0
fi
test "${count}" = 1
echo "recover=true" >> "${GITHUB_OUTPUT}"
recover:
needs: gate
if: ${{ needs.gate.outputs.recover == 'true' }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 90
environment: staging
concurrency:
group: relay-staging-mutation
cancel-in-progress: false
env:
GCP_PROJECT_ID: onorca-cloud-staging
DIRECTOR_ORIGIN: https://relay-staging.onorca.dev
CELL_ORIGIN: https://c4.relay-staging.onorca.dev
TARGET_CELL_ID: staging-gce-c4
TARGET_ZONE: asia-east2-a
MIG_NAME: orca-cloud-staging-relay-gce-c4
PREDECESSOR_IMAGE_DIGEST: sha256:ce16d13ce6b633c6fbb1a2afdd6cdb8369645a329d42a8355efa7ad1e60a44f7
TARGET_IMAGE_DIGEST: sha256:5aedbca5c86de24c8b4d4bf7e3b444b76c712f281ede916cb9d90f70cad1e563
steps:
- uses: actions/checkout@v4
- id: auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
- uses: google-github-actions/setup-gcloud@v2
with:
skip_install: true
- uses: ./.github/actions/cloud-sql-rollout-lease
with:
bucket: onorca-cloud-staging-terraform-state
object: terraform/state/cloud-sql-rollout/staging.lock
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
terraform_wrapper: false
- uses: actions/setup-node@v4
with:
node-version: 24
- name: Initialize the exact staging backend
run: node dev/scripts/infra.mjs init --env staging
- id: preflight-auth
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Inspect the exact C4 recovery state
id: preflight
timeout-minutes: 3
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.preflight-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
inspect="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode inspect --cell-ids "${TARGET_CELL_ID}" \
--expected-generation '' --expected-membership-sha256 '' --attempt-id '' \
--image-digest "${PREDECESSOR_IMAGE_DIGEST}")"
generation="$(jq -er '.generation' <<< "${inspect}")"
[[ "${generation}" =~ ^(0|[1-9][0-9]*)$ ]]
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${inspect}")" = \
migration-only
echo "selector_generation=${generation}" >> "${GITHUB_OUTPUT}"
if runtime="$(curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/runtime-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1}')"; then
current_digest="$(jq -er '.imageDigest' <<< "${runtime}")"
[[ "${current_digest}" =~ ^sha256:[a-f0-9]{64}$ ]]
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
jq -e '.draining | type == "boolean"' <<< "${runtime}" >/dev/null
{
echo "runtime_available=true"
echo "current_digest=${current_digest}"
echo "draining=$(jq -r '.draining' <<< "${runtime}")"
echo "ready=$(jq -r '.status.runtime.ready == true' <<< "${status}")"
} >> "${GITHUB_OUTPUT}"
else
echo "runtime_available=false" >> "${GITHUB_OUTPUT}"
fi
- name: Classify both exact recovery end states
id: recovery-plan
timeout-minutes: 10
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
predecessor_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
target_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
served_digest="$(gcloud artifacts docker images describe "${predecessor_image}" \
--project "${GCP_PROJECT_ID}" --format='value(image_summary.digest)')"
test "${served_digest}" = "${PREDECESSOR_IMAGE_DIGEST}"
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.[$cell].image' <<< "${cells}")" = \
"${target_image}"
target_plan="${RUNNER_TEMP}/relay-c4-image-target.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars -lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${target_plan}"
target_result="$(terraform -chdir=infra/terraform show -json "${target_plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${target_image}" \
--rollback-image "${predecessor_image}")"
target_state="$(jq -r '[.changes,.changeKind] | join(":")' <<< "${target_result}")"
case "${target_state}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" \
--arg image "${predecessor_image}" '.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-recovery.tfvars.json"
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-recovery.tfvars.json" \
-lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${plan}"
result="$(terraform -chdir=infra/terraform show -json "${plan}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${predecessor_image}" \
--rollback-image "${target_image}")"
changes="$(jq -r '.changes' <<< "${result}")"
change_kind="$(jq -r '.changeKind' <<< "${result}")"
case "${changes}:${change_kind}" in
0:none|2:replacement|*:obsolete-template-delete|*:manager-convergence|*:replacement-with-obsolete-template) ;;
*) exit 1 ;;
esac
mig="$(gcloud compute instance-groups managed describe "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
mig_stable="$(jq -r '.status.isStable == true and .status.versionTarget.isReached == true' \
<<< "${mig}")"
instances="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json)"
instance_stable="$(jq -r 'length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE"' <<< "${instances}")"
action=rollback-predecessor
recovery_digest="${PREDECESSOR_IMAGE_DIGEST}"
if test "${{ steps.preflight.outputs.runtime_available }}" = true && \
test "${{ steps.preflight.outputs.ready }}" = true && \
test "${{ steps.preflight.outputs.draining }}" = false && \
test "${mig_stable}" = true && test "${instance_stable}" = true; then
if test "${{ steps.preflight.outputs.current_digest }}" = "${TARGET_IMAGE_DIGEST}" && \
test "${target_state}" = 0:none; then
action=verify-target
recovery_digest="${TARGET_IMAGE_DIGEST}"
elif test "${{ steps.preflight.outputs.current_digest }}" = \
"${PREDECESSOR_IMAGE_DIGEST}" && test "${changes}:${change_kind}" = 0:none; then
action=verify-predecessor
fi
fi
{
echo "changes=${changes}"
echo "change_kind=${change_kind}"
echo "action=${action}"
echo "recovery_digest=${recovery_digest}"
} >> "${GITHUB_OUTPUT}"
- id: fence-auth
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Fence C4 before predecessor recovery
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 6
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.fence-auth.outputs.id_token }}
shell: bash
run: |
set -euo pipefail
expected_digests="${PREDECESSOR_IMAGE_DIGEST},${TARGET_IMAGE_DIGEST}"
current_digest="${{ steps.preflight.outputs.current_digest }}"
if test -n "${current_digest}" && [[ ",${expected_digests}," != *",${current_digest},"* ]]; then
expected_digests="${expected_digests},${current_digest}"
fi
if curl --silent --show-error --fail-with-body --max-time 30 --request POST \
"${CELL_ORIGIN}/v1/admin/drain" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' --data '{"v":1,"graceMs":0}' \
>/dev/null; then
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining required \
--activity quiescent \
--expected-image-digests "${expected_digests}" \
--timeout-ms 240000
else
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --runtime unavailable --heartbeat stale \
--admission migration-only --draining either --activity restart-safe \
--timeout-ms 240000
fi
- name: Apply and stabilize the saved predecessor plan
id: apply
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
CHANGES: ${{ steps.recovery-plan.outputs.changes }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
plan="${RUNNER_TEMP}/relay-c4-image-recovery.tfplan"
if test "${CHANGES}" != 0; then
terraform -chdir=infra/terraform apply -auto-approve "${plan}"
fi
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Restart only when the plan did not replace C4
id: restore
if: ${{ steps.recovery-plan.outputs.action == 'rollback-predecessor' }}
timeout-minutes: 20
env:
APPLY_STABLE_AT_MS: ${{ steps.apply.outputs.stable_at_ms }}
CHANGE_KIND: ${{ steps.recovery-plan.outputs.change_kind }}
shell: bash
run: |
set -euo pipefail
if [[ "${CHANGE_KIND}" =~ ^(replacement|replacement-with-obsolete-template|manager-convergence)$ ]]; then
echo "stable_at_ms=${APPLY_STABLE_AT_MS}" >> "${GITHUB_OUTPUT}"
exit 0
fi
instance="$(gcloud compute instance-groups managed list-instances "${MIG_NAME}" \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --format=json \
| jq -er 'if length == 1 and .[0].instanceStatus == "RUNNING" and
.[0].currentAction == "NONE" then .[0].instance | split("/") | last
else error("C4 is not one stable running instance") end')"
gcloud compute instance-groups managed recreate-instances "${MIG_NAME}" \
--instances "${instance}" --project "${GCP_PROJECT_ID}" \
--zone "${TARGET_ZONE}" --quiet
gcloud compute instance-groups managed wait-until "${MIG_NAME}" --stable \
--project "${GCP_PROJECT_ID}" --zone "${TARGET_ZONE}" --timeout 900
echo "stable_at_ms=$(date -u +%s%3N)" >> "${GITHUB_OUTPUT}"
- name: Require an empty selected-image recovery readback
timeout-minutes: 8
env:
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
image_repository="us-central1-docker.pkg.dev/${GCP_PROJECT_ID}/orca-cloud/relay"
recovery_image="${image_repository}@${RECOVERY_DIGEST}"
other_image="${image_repository}@${TARGET_IMAGE_DIGEST}"
if test "${RECOVERY_DIGEST}" = "${TARGET_IMAGE_DIGEST}"; then
other_image="${image_repository}@${PREDECESSOR_IMAGE_DIGEST}"
fi
cells="$(terraform -chdir=infra/terraform console \
-var-file=environments/staging.tfvars \
<<< 'jsonencode(var.relay_gce_cells)' | jq -er '.')"
recovery_cells="$(jq -ce --arg cell "${TARGET_CELL_ID}" --arg image "${recovery_image}" \
'.[$cell].image = $image' <<< "${cells}")"
jq -n --argjson cells "${recovery_cells}" \
'{relay_gce_cells:$cells}' > "${RUNNER_TEMP}/relay-c4-readback.tfvars.json"
readback="${RUNNER_TEMP}/relay-c4-image-recovery-readback.tfplan"
terraform -chdir=infra/terraform plan \
-var-file=environments/staging.tfvars \
-var-file="${RUNNER_TEMP}/relay-c4-readback.tfvars.json" \
-lock-timeout=5m \
'-target=google_compute_instance_template.relay_gce_cell["staging-gce-c4"]' \
'-target=google_compute_instance_group_manager.relay_gce_cell["staging-gce-c4"]' \
-out="${readback}"
readback_result="$(terraform -chdir=infra/terraform show -json "${readback}" \
| node dev/scripts/validate-relay-capacity-plan.mjs \
--mode same-cap-image --cell-id "${TARGET_CELL_ID}" --hard-cap 3000 \
--unobserved-bound 60 --image "${recovery_image}" \
--rollback-image "${other_image}")"
test "$(jq -r '.changes' <<< "${readback_result}")" = 0
- id: verify-auth
if: ${{ always() }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ vars.STAGING_GCP_RELAY_CAPACITY_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ vars.STAGING_GCP_RELAY_CAPACITY_SERVICE_ACCOUNT }}
token_format: id_token
id_token_audience: https://relay-staging.onorca.dev/v1/admin/drain
id_token_include_email: true
- name: Verify the recovered image and unchanged isolation
if: ${{ always() && steps.verify-auth.outcome == 'success' }}
timeout-minutes: 8
env:
ORCA_RELAY_ADMIN_ID_TOKEN: ${{ steps.verify-auth.outputs.id_token }}
SELECTOR_GENERATION: ${{ steps.preflight.outputs.selector_generation }}
STABLE_AT_MS: ${{ steps.restore.outputs.stable_at_ms }}
RECOVERY_DIGEST: ${{ steps.recovery-plan.outputs.recovery_digest }}
shell: bash
run: |
set -euo pipefail
node dev/scripts/verify-relay-capacity-transition.mjs \
--director-origin "${DIRECTOR_ORIGIN}" --cell-origin "${CELL_ORIGIN}" \
--cell-id "${TARGET_CELL_ID}" --hard-cap 3000 --unobserved-bound 60 \
--heartbeat fresh --admission migration-only --draining forbidden \
--activity quiescent --expected-image-digests "${RECOVERY_DIGEST}" \
--timeout-ms 240000
stable_at_ms="${STABLE_AT_MS:-0}"
for _ in $(seq 1 36); do
status="$(curl --fail-with-body --max-time 30 --request POST \
"${DIRECTOR_ORIGIN}/v1/admin/cell-status" \
--header "Authorization: Bearer ${ORCA_RELAY_ADMIN_ID_TOKEN}" \
--header 'Content-Type: application/json' \
--data "$(jq -cn --arg cell "${TARGET_CELL_ID}" '{v:1,cellId:$cell}')")"
if test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true && \
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"; then break; fi
sleep 5
done
test "$(jq -r '.status.runtime.ready' <<< "${status}")" = true
test "$(jq -r '.status.runtime.lastHeartbeatAt' <<< "${status}")" \
-ge "${stable_at_ms}"
result="$(node dev/scripts/operate-relay-asia-admission.mjs \
--environment staging --mode verify --cell-ids "${TARGET_CELL_ID}" \
--expected-generation "${SELECTOR_GENERATION}" \
--expected-membership-sha256 '' --attempt-id '' \
--image-digest "${RECOVERY_DIGEST}")"
test "$(jq -r --arg cell "${TARGET_CELL_ID}" '.states[$cell]' <<< "${result}")" = \
migration-only
@@ -1,60 +0,0 @@
name: Requeue Relay Staging C4 Recovery
on:
workflow_run:
workflows: [Recover Relay Staging C4 Image]
types: [completed]
permissions:
actions: write
contents: read
concurrency:
group: relay-staging-c4-recovery-requeue
cancel-in-progress: false
defaults:
run:
working-directory: cloud
jobs:
requeue:
if: ${{ vars.ORCA_CLOUD_OPERATIONS_ENABLED == 'true' && (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.conclusion == 'cancelled') }}
runs-on: blacksmith-2vcpu-ubuntu-2204
timeout-minutes: 5
steps:
- name: Requeue only a cancelled protected recovery job
working-directory: .
env:
GH_TOKEN: ${{ github.token }}
SOURCE_RUN_ID: ${{ github.event.workflow_run.id }}
shell: bash
run: |
set -euo pipefail
[[ "${SOURCE_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${SOURCE_RUN_ID}/jobs?filter=latest")"
count="$(jq -s '[.[].jobs[] | select(.name == "recover" and
.conclusion == "cancelled" and .started_at == null)] | length' <<< "${jobs}")"
if test "${count}" = 0; then exit 0; fi
test "${count}" = 1
runs="$(gh api \
"repos/${GITHUB_REPOSITORY}/actions/workflows/cloud-recover-relay-staging-c4-image.yml/runs?branch=main&per_page=100")"
active=0
while IFS= read -r run_id; do
active_jobs="$(gh api --paginate \
"repos/${GITHUB_REPOSITORY}/actions/runs/${run_id}/jobs?filter=latest")"
if jq -se '
([.[].jobs[] | select(.name == "gate" and .conclusion == "success")] | length) == 1 and
([.[].jobs[] | select(.name == "recover" and .status != "completed")] | length) == 1
' <<< "${active_jobs}" >/dev/null; then
active=1
break
fi
done < <(jq -r --arg source "${SOURCE_RUN_ID}" \
'.workflow_runs[] | select((.id | tostring) != $source and
.status != "completed") | .id' <<< "${runs}")
if test "${active}" != 0; then exit 0; fi
gh workflow run cloud-recover-relay-staging-c4-image.yml \
--repo "${GITHUB_REPOSITORY}" --ref main \
-f confirmation=RECOVER_STAGING_ASIA_C4_IMAGE
+79 -25
View File
@@ -8,6 +8,7 @@ on:
- src/packages/process-host/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
- .github/actions/setup-pnpm/**
push:
branches: [main]
paths:
@@ -15,6 +16,7 @@ on:
- src/packages/process-host/**
- .github/workflows/cloud-*.yml
- .github/actions/cloud-sql-rollout-lease/**
- .github/actions/setup-pnpm/**
permissions:
contents: read
@@ -78,29 +80,94 @@ jobs:
- wait: history-scan
# Compiles the workspace. No Postgres service: nothing here reaches a
# database, and the service container costs ~13s of startup.
# Compile without Postgres: no database access occurs here, avoiding ~13s service startup.
build:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: ./.github/actions/setup-pnpm
with:
package_json_file: cloud/package.json
package-json-file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
check-latest: true
- run: pnpm install --frozen-lockfile
- name: Audit production and development dependencies
run: pnpm audit --json > security-dependencies.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: cloud-dependency-audit
path: cloud/security-dependencies.json
retention-days: 30
- run: pnpm build
- run: pnpm typecheck
# Runs in parallel with build. `pnpm test` compiles the workspace
# packages it needs through pretest hooks, so it does not depend on
# `pnpm build` having run.
containers:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
name: Container security (${{ matrix.app }})
runs-on: ubuntu-22.04
strategy:
fail-fast: false
matrix:
app: [relay, push, connector]
env:
APP: ${{ matrix.app }}
TRIVY_IMAGE: aquasec/trivy:0.76.1@sha256:790fd66d1697395d8693d1acaa1ea702406a20f109b6fc7bac8db623eca6c06c
steps:
- uses: actions/checkout@v4
- name: Build or resolve the runtime image
run: |
set -euo pipefail
IMAGE="orca-cloud-security:$APP"
docker build --platform linux/amd64 \
--file "apps/$APP/Dockerfile" --tag "$IMAGE" .
echo "IMAGE=$IMAGE" >> "$GITHUB_ENV"
- name: Smoke test the nonroot Node runtime
run: |
docker run --rm --platform linux/amd64 --entrypoint sh "$IMAGE" -ec '
test "$(id -u)" -ne 0
! command -v npm
! command -v pnpm
module=app
if test "$1" = push; then module=push-server; fi
if test "$1" = connector; then module=connector-server; fi
node --input-type=module -e "await import(\"./apps/$1/dist/$module.js\")"
' sh "$APP"
- name: Scan every severity and retain the complete inventory
run: |
mkdir -p security-reports
docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock \
--volume "$PWD/security-reports:/reports" "$TRIVY_IMAGE" image \
--platform linux/amd64 --scanners vuln --ignorefile '' \
--format json --output /reports/vulnerabilities.json "$IMAGE"
docker run --rm --volume "$PWD/security-reports:/reports" "$TRIVY_IMAGE" convert \
--format cyclonedx --output /reports/sbom.cdx.json /reports/vulnerabilities.json
- name: Reject available fixes
run: |
jq '(.Results[]?.Vulnerabilities) |= ((. // []) | map(select(.FixedVersion != null and .FixedVersion != "")))' \
security-reports/vulnerabilities.json > security-reports/fixable.json
docker run --rm --volume "$PWD:/repo:ro" "$TRIVY_IMAGE" convert \
--ignorefile '' --show-suppressed --exit-code 1 \
/repo/security-reports/fixable.json
- uses: actions/upload-artifact@v4
if: always()
with:
name: cloud-container-security-${{ matrix.app }}
path: cloud/security-reports/
retention-days: 30
# Run tests beside build; pretest hooks compile their own workspace dependencies.
test:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ubuntu-22.04
@@ -121,33 +188,20 @@ jobs:
env:
ORCA_BACKGROUND_LAUNCH: '1'
ORCA_PUSH_TEST_DATABASE_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
ORCA_CONNECTOR_TEST_DATABASE_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
ORCA_CLOUD_DATABASE_TEST_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
ORCA_RELAY_TEST_POSTGRES_URL: postgres://relay_test:relay_test@127.0.0.1:5432/orca_relay_test
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: ./.github/actions/setup-pnpm
with:
package_json_file: cloud/package.json
package-json-file: cloud/package.json
- uses: actions/setup-node@v4
with:
node-version: 24
check-latest: true
- run: pnpm install --frozen-lockfile
- run: pnpm test
# Fork pull requests reach this job, so it never configures a backend, never plans, and never
# holds a credential. Only the relay root ships here; foundation and apps stay private.
terraform:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: 1.15.8
- run: terraform -chdir=infra/terraform fmt -check -recursive
- run: terraform -chdir=infra/terraform init -backend=false -input=false
- run: terraform -chdir=infra/terraform validate
+9 -48
View File
@@ -25,9 +25,7 @@ on:
- 'config/scripts/daemon-endpoint-handover-smoke.mjs'
- 'config/scripts/windows-daemon-workspace-close-repro.mjs'
- 'config/scripts/verify-computer-native.mjs'
# Why: the native-smoke job boots the built terminal daemon under plain
# Node, so any change to the daemon bundle graph or the main build must
# re-run it (the v1.4.129-rc.1 daemon outage shipped with green CI).
# Re-smoke daemon bundle/build changes under plain Node; v1.4.129-rc.1 shipped a module-load outage despite green CI.
- 'electron.vite.config.ts'
- 'config/build-plugins/**'
- 'src/main/daemon/**'
@@ -68,6 +66,9 @@ concurrency:
group: computer-e2e-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
native-smoke:
if: github.event_name == 'pull_request' && github.event.pull_request.draft != true
@@ -87,17 +88,13 @@ jobs:
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: node
# Why: without --config, bare vitest ignores config/vitest.config.ts (there
# is no root config) and falls back to the 5s default timeout with no
# Windows worker cap, so the real csc.exe launcher-compile tests time out
# on hosted Windows. Use the shared config so this job matches pnpm test.
# Use shared Vitest config: bare invocation misses config/ and its timeout/Windows worker cap, breaking real csc.exe tests.
- run: >-
pnpm vitest run --config config/vitest.config.ts
config/scripts/macos-computer-helper-owner-loss-group-recovery.test.mjs
config/scripts/macos-computer-helper-owner-loss-processes.test.mjs
config/scripts/computer-use-smoke.test.mjs
src/main/computer/computer-provider-lifecycle.test.ts
src/main/computer/computer-provider-unavailable-message.test.ts
src/main/computer/sidecar-client.test.ts
src/main/computer/macos-native-provider-client.test.ts
src/main/computer/macos-native-provider-socket.test.ts
@@ -123,56 +120,20 @@ jobs:
src/main/runtime/rpc/methods/computer.test.ts
src/main/runtime/rpc/methods/computer-actions.test.ts
src/cli/runtime/envelope-schema.test.ts
src/shared/remote-runtime-client.test.ts
src/shared/remote-runtime/remote-runtime-client.test.ts
- run: pnpm verify:computer-native
- run: pnpm build:cli
- run: pnpm run build:electron-vite:parallel
# Why: boot the BUILT daemon-entry under plain Node the way production
# forks it. v1.4.129-rc.1 shipped a daemon that exited at module load
# (leaked electron require) while every other check passed; this fails
# the PR when the built daemon cannot start on ubuntu-22.04 / windows.
# Boot built daemon-entry under plain Node on Ubuntu/Windows, catching leaked electron requires like v1.4.129-rc.1.
- name: Daemon boot smoke
run: node config/scripts/daemon-boot-smoke.mjs
# Why: a daemon that lost its endpoint name used to delete the replacement's
# socket on exit, stranding a live daemon that hosts PTYs nothing can reach.
# Only two real processes racing the same endpoint reproduce it.
# Race real daemon processes on one endpoint; a loser must not unlink the replacement’s socket.
- name: Daemon endpoint handover smoke
run: node config/scripts/daemon-endpoint-handover-smoke.mjs
# Why: workspace removal overlaps graceful renderer teardown with the
# forced main-process sweep; exercise that exact pair against real ConPTY.
# Exercise workspace removal’s renderer teardown plus forced main sweep against real ConPTY.
- name: Windows daemon workspace-close repro
if: runner.os == 'Windows'
run: node config/scripts/windows-daemon-workspace-close-repro.mjs
# Hosted macOS runners cannot receive persistent Accessibility/Screen Recording
# grants. Keep the real native build/owner-loss checks on every trigger instead.
mac-native-owner-smoke:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: macos-15
permissions:
contents: read
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: electron
- name: Owner-loss benchmark process cleanup
run: >-
pnpm vitest run
config/scripts/macos-computer-helper-owner-loss-group-recovery.test.mjs
config/scripts/macos-computer-helper-owner-loss-processes.test.mjs
# Why: the parallel launcher's cancellation tests are darwin-only and no
# other PR job runs on macOS.
- name: Parallel native build launcher cancellation
run: >-
pnpm vitest run --config config/vitest.config.ts
config/scripts/build-native-for-platform.test.mjs
- name: Authenticated helper owner-loss smoke
run: pnpm bench:macos-computer-helper-owner-loss --expect reaped --trials 1
- name: Swift tests and signed universal helper verification
run: pnpm verify:computer-native
linux:
if: github.event_name != 'pull_request'
runs-on: ubuntu-22.04
@@ -0,0 +1,64 @@
name: Computer-use macOS native
# Split from computer-e2e.yml so CLI/daemon-only PRs skip the macOS runner: this job builds the
# Swift helper and forks out/main/computer-sidecar.js, whose imports never reach src/cli or src/main/daemon.
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
paths:
- '.github/workflows/computer-macos-native.yml'
- 'electron.vite.config.ts'
- 'config/build-plugins/**'
- 'config/scripts/build-computer-macos.mjs'
- 'config/scripts/build-native-for-platform.mjs'
- 'config/scripts/build-native-for-platform.test.mjs'
- 'config/scripts/pnpm-cli-invocation.mjs'
- 'config/scripts/macos-computer-helper-owner-loss-*'
- 'config/scripts/verify-computer-native.mjs'
- 'native/computer-use-macos/**'
- 'native/computer-use-linux/**'
- 'native/computer-use-windows/**'
- 'skills/computer-use/SKILL.md'
- 'src/main/computer/**'
- 'src/shared/computer-use-*.ts'
workflow_dispatch:
schedule:
- cron: '0 6 * * *'
concurrency:
group: computer-macos-native-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
# Hosted macOS runners cannot receive persistent Accessibility/Screen Recording
# grants. Keep the real native build/owner-loss checks on every matching trigger instead.
mac-native-owner-smoke:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: macos-15
permissions:
contents: read
steps:
- uses: actions/checkout@v6
with:
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: electron
- name: Owner-loss benchmark process cleanup
run: >-
pnpm vitest run
config/scripts/macos-computer-helper-owner-loss-group-recovery.test.mjs
config/scripts/macos-computer-helper-owner-loss-processes.test.mjs
# Why: the parallel launcher's cancellation tests are darwin-only and no
# other PR job runs on macOS.
- name: Parallel native build launcher cancellation
run: >-
pnpm vitest run --config config/vitest.config.ts
config/scripts/build-native-for-platform.test.mjs
- name: Authenticated helper owner-loss smoke
run: pnpm bench:macos-computer-helper-owner-loss --expect reaped --trials 1
- name: Swift tests and signed universal helper verification
run: pnpm verify:computer-native
@@ -1,118 +0,0 @@
name: Daemon Relocation Spike
# Why: Phase 1 of the Windows update-survival work relocates the terminal
# daemon's file closure out of the install dir so it survives an update. Before
# implementing it in the app, this spike finds the MINIMAL set of packaged files
# that a copied Orca.exe (run as node) needs to start the daemon and drive a
# ConPTY session from a relocated directory, holding no install-dir file locks.
# Runs from the feature branch via push (workflow_dispatch only works on the
# default branch); main is never touched.
on:
push:
branches:
- Jinwoo-H/windows-update-survival
paths:
- 'tests/tools/daemon-relocation-spike/**'
- '.github/workflows/daemon-relocation-spike.yml'
workflow_dispatch: {}
permissions:
contents: read
concurrency:
group: daemon-relocation-spike-${{ github.ref }}
cancel-in-progress: true
jobs:
spike:
name: relocation file-set spike
runs-on: windows-2022
timeout-minutes: 40
steps:
- name: Checkout
uses: actions/checkout@v6
with:
# This job only reads the repo to build/run the spike; it never pushes.
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
- name: Setup pnpm
uses: pnpm/setup@v2
with:
install: false
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Why: cache the unpacked build keyed on the inputs that affect it, so
# spike-script-only edits skip the ~15 min electron-builder build.
- name: Cache unpacked build
id: cache-unpacked
uses: actions/cache@v4
with:
path: dist/win-unpacked
# mobile/ is in the key because beforePack requires out/mobile-web, whose bytes are the
# page built from the mobile install and the page trees below; a mobile-only change must
# miss this cache, not reuse a stale installer. config/** already covers the builder, the
# verifier and the manifest writer.
key: >-
win-unpacked-${{ hashFiles(
'src/**',
'config/**',
'package.json',
'pnpm-lock.yaml',
'mobile/package.json',
'mobile/pnpm-lock.yaml',
'mobile/app/**',
'mobile/src/**',
'mobile/web-entry/**'
) }}
# Why here: electron-builder's beforePack requires out/mobile-web, and the bundle
# build resolves React Native and Expo from mobile/node_modules. Gated with the
# build it feeds, so a cache hit does not pay for an install nothing consumes.
- uses: ./.github/actions/install-mobile-dependencies
if: steps.cache-unpacked.outputs.cache-hit != 'true'
- name: Build unpacked app
if: steps.cache-unpacked.outputs.cache-hit != 'true'
run: pnpm run build:unpack
- name: Run relocation spike (all tiers)
shell: pwsh
run: |
New-Item -ItemType Directory -Force artifacts | Out-Null
$log = "artifacts/spike-output.log"
$tiers = @('full', 'no-gpu', 'minimal')
$any = $false
foreach ($tier in $tiers) {
"===== TIER: $tier =====" | Tee-Object -FilePath $log -Append
$work = Join-Path $env:RUNNER_TEMP "spike-$tier"
# --keep-work-dir so the per-tier daemon stdout/stderr logs survive
# for the artifact upload (the spike otherwise removes work-dir),
# which is what diagnoses a tier that fails to reach ready.
node tests/tools/daemon-relocation-spike/spike.mjs `
--app-dir dist/win-unpacked `
--work-dir "$work" `
--tier $tier `
--keep-work-dir 2>&1 | Tee-Object -FilePath $log -Append
if ($LASTEXITCODE -eq 0) { $any = $true }
}
if (-not $any) { throw "No tier passed the relocation spike" }
- name: Upload spike output
if: always()
uses: actions/upload-artifact@v7
with:
name: daemon-relocation-spike-output
path: |
artifacts/spike-output.log
${{ runner.temp }}/spike-*/**/*.log
retention-days: 7
if-no-files-found: warn
+46 -189
View File
@@ -1,49 +1,6 @@
name: Daily macOS Dev Build
# Why: gives developers a signed macOS build of main once a day that the in-app
# updater can install directly, without waiting for an RC cut — less noise than
# hourly.
#
# Schedule is a single UTC cron (GH Actions has no timezone-aware schedules).
# 18:15 UTC is late morning Pacific year-round (10:15am PST / 11:15am PDT). Minute
# 15 avoids stacking with hourly-mac-build, which fires at minute 0 every hour.
#
# Deliberately narrow scope:
# - macOS only. Other platforms keep using RC/stable.
# - No tests or lint on the build job. After a live publish we fire-and-forget
# the full E2E workflow at the cut SHA (same detached dispatch as
# release-cut). A red suite must not fail or delay the daily.
# - PR CI and release-cut remain the gates that matter.
# - Signed AND notarized, exactly like a release. The notary round trip is the
# one slow step kept: macOS anchors a notarized app's TCC grants on identifier
# + team rather than on its cdhash, so those grants survive an update. Without
# a ticket every daily reads as a new client and silently loses file access
# under Documents/Desktop/Downloads.
#
# Artifacts publish to stablyai/orca-daily, never to stablyai/orca: the main
# repo's releases atom feed exposes only its 10 newest entries, so high-volume
# dev tags would evict every stable/RC entry and break updates for real users.
# Separate from orca-hourly so someone riding main's hourlies never sees the
# sparse daily series mixed into that list.
#
# GITHUB_TOKEN is scoped to this repo and cannot publish there, so writes use the
# same GitHub App as hourly (installed on orca-daily with Contents: Read and
# write). Provision the App secrets with
# `bash config/scripts/setup-hourly-release-token.sh`, then grant the App access
# to orca-daily with `bash config/scripts/setup-daily-release-repo.sh`:
# HOURLY_RELEASE_APP_ID the App's numeric id (shared with hourly/adhoc)
# HOURLY_RELEASE_APP_PRIVATE_KEY the App's .pem private key
#
# Installation tokens live one hour, which is why this mints three times. Install
# and build need no token at all, and notarization can hold the publish step for
# tens of minutes; minting again once the build is done starts the clock at the
# first call that actually uses it rather than burning a third of it on
# `pnpm install`. The upload step's own retry budget (2x45m) can outlive that
# second token, so a third is minted after it for verify/publish/prune/cleanup —
# without it a stuck notary run would strand a draft that cleanup gets a 401 on.
# The release stays an unpublished draft until the manifest check passes, so the
# worst case is still an invisible draft — and the build-number query counts
# drafts, so it holds its number and the next run does not reuse it.
# Publish daily main builds to orca-daily, separate from hourly and stable/RC feeds. Sign/notarize for update-preserved macOS TCC grants; PR/release pipelines own blocking validation, with E2E detached after publish. Cron 18:15 UTC means 10:15 PST/11:15 PDT, offset from hourly. Use the shared hourly App with write access to orca-daily (setup-hourly-release-token.sh/setup-daily-release-repo.sh). Mint tokens again after build and upload because 2×45min retry can exceed their 1h life. Keep releases draft until manifests verify; drafts retain build numbers.
on:
schedule:
@@ -94,10 +51,12 @@ jobs:
needs: resolve-main
permissions:
contents: read
actions: read
uses: ./.github/workflows/node-server-tests.yml
with:
ref: ${{ needs.resolve-main.outputs.head_sha }}
build_template: true
fast_runners: false
build-daily-mac:
needs:
@@ -109,11 +68,9 @@ jobs:
version: ${{ steps.daily.outputs.version }}
head_sha: ${{ steps.freshness.outputs.head_sha }}
published: ${{ steps.publish_live.outcome == 'success' && 'true' || 'false' }}
# Hosted macos-15 cannot boot the packaged x64 terminal host that afterPack verifies.
runs-on: blacksmith-6vcpu-macos-15
# Why 150: it must exceed the worst case the retry budgets below can produce
# (install 3x10 + publish 2x45 = 120, plus ~25 for checkout/build/verify), or
# the job is killed mid-retry and no cleanup step runs at all. A typical run
# is far shorter — this is the notary queue's tail, not its median.
# Allow 150min for install 3×10 + publish 2×45 + ~25 setup/build/verify, so outer timeout cannot strand cleanup mid-retry.
timeout-minutes: 150
env:
NODE_OPTIONS: --max-old-space-size=4096
@@ -124,10 +81,7 @@ jobs:
ref: ${{ needs.resolve-main.outputs.head_sha }}
# Version helpers only read HEAD; published versions come from git tags.
fetch-depth: 1
# Why: this job only reads stablyai/orca and never pushes; every write
# goes to the daily repo through a minted App token passed by env.
# Not persisting the checkout credential shrinks the blast radius if a
# build step is compromised (zizmor: artipacked).
# Read this repo without persisted credentials; write orca-daily through env-passed App token (zizmor: artipacked).
persist-credentials: false
- name: Mint daily repo token
@@ -139,8 +93,7 @@ jobs:
owner: stablyai
repositories: orca-daily
# Why: main is often idle. Rebuilding an unchanged commit burns a runner
# hour and adds a redundant tag to the retention window.
# Skip unchanged main to avoid duplicate builds/tags and runner cost.
- name: Check whether main moved since the last daily
id: freshness
shell: bash
@@ -156,9 +109,7 @@ jobs:
echo "Forced dispatch; building $head_sha."
exit 0
fi
# The previous daily records its source commit in the release body.
# Drafts are excluded: an unpublished leftover never shipped, so treating
# it as "the last build" would skip a build that never actually happened.
# Drafts never shipped; exclude them when comparing the previous daily’s source commit.
last_body="$(gh release list --repo "$DAILY_REPO" --limit 20 --json tagName,isDraft \
--jq 'map(select(.isDraft | not)) | .[0].tagName // empty' 2>/dev/null || true)"
if [[ -z "$last_body" ]]; then
@@ -187,13 +138,13 @@ jobs:
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
cache: pnpm
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
# Why: packaging bundles the pinned Node into the macOS terminal host. Keyed on the pin alone
# with no restore-keys, so a pin bump starts empty instead of carrying old archives forward.
# Cache bundled host Node by exact pin without restore keys.
- name: Locate the pinned Node runtime cache
if: steps.freshness.outputs.should_build == 'true'
shell: bash
@@ -217,8 +168,7 @@ jobs:
restore-keys: |
electron-builder-mac-
# Why both CPUs: the mac config packages x64 and arm64 from this arm64
# runner, so the install must carry both variants of the native optional deps.
# Install x64 and arm64 optional natives for both packaged macOS targets.
- name: Install dependencies
if: steps.freshness.outputs.should_build == 'true'
uses: nick-fields/retry@v4
@@ -228,8 +178,7 @@ jobs:
retry_wait_seconds: 30
command: pnpm install --frozen-lockfile --cpu=current,x64,arm64
# Why here: electron-builder's beforePack requires out/mobile-web, and the bundle
# build resolves React Native and Expo from mobile/node_modules.
# Build required out/mobile-web using mobile’s React Native/Expo install.
- uses: ./.github/actions/install-mobile-dependencies
if: steps.freshness.outputs.should_build == 'true'
@@ -254,21 +203,12 @@ jobs:
MAIN_REPO_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# Existing titles, which carry the build number this series continues
# from. The script picks the number, because it restarts per base version
# and only the script knows which base this build resolved to.
#
# Why drafts count here but not in the freshness check: that check asks
# "did this commit ship", where a draft is a no. This one asks "is the
# number free", where a stranded draft still holds one.
# The script continues numbering for its resolved base version; stranded drafts still reserve numbers,
# although freshness checks exclude them because they never shipped.
names="$(gh release list --repo "$DAILY_REPO" --limit 200 --json name \
--jq '.[].name // empty')"
# Why git tags, not GitHub releases: unpublishing a buggy cut deletes the
# GitHub release and leaves the tag. That dragged hourlies backwards so
# electron-updater stopped offering them; dailies would do the same. A
# separate token because GH_TOKEN above is the App's, scoped to the
# daily repo. Empty on failure — the script then falls back to
# package.json, which is stale but never wrong enough to fail a build.
# Tags survive release deletion, preventing updater version regression. Use a separate token because
# GH_TOKEN’s App grant is scoped to the channel repo; lookup failure falls back to package.json.
main_tags="$(GH_TOKEN="$MAIN_REPO_TOKEN" gh api \
"repos/$GITHUB_REPOSITORY/git/matching-refs/tags/v" \
--jq '.[].ref | sub("^refs/tags/"; "")' || true)"
@@ -282,11 +222,7 @@ jobs:
node config/scripts/daily-build-version.mjs \
>"$RUNNER_TEMP/daily-identity.txt"
grep -E '^(version|build_number)=' "$RUNNER_TEMP/daily-identity.txt"
# Why check rather than trust: the checkout above pins the resolved main commit, but a
# workflow_dispatch runs this file from whatever branch was dispatched. A
# branch that edits this step while main still has the old script yields
# an empty name and an untitled release — silent, and only visible once
# someone opens the releases page. Fail here instead.
# Dispatch YAML can be newer than the pinned source’s version script; reject missing output before publishing an untitled release.
if ! grep -q '^name=' "$RUNNER_TEMP/daily-identity.txt"; then
echo "::error::daily-build-version.mjs emitted no release name; this workflow and main's copy of the script are out of sync."
exit 1
@@ -298,9 +234,7 @@ jobs:
run: pnpm build:release
env:
NODE_OPTIONS: --max-old-space-size=4096
# Why: daily builds are not an official channel — telemetry's transport
# gate accepts only 'stable' or 'rc', so leaving this unset keeps them
# silent, which is correct for unvetted dev artifacts.
# Leave dev identity unset so official-only stable/rc telemetry stays silent.
ORCA_DIAGNOSTICS_TOKEN_URL: https://www.onorca.dev/diagnostics/token
# After the app build so nothing that cleans out/ can drop it; electron-builder ships it.
@@ -311,11 +245,7 @@ jobs:
name: orcad-template
path: out/orcad-template
# Why a second mint: everything from here on writes to the daily repo, and
# the notary round trip inside the publish step can be tens of minutes. The
# token minted at the top has already spent install + build of its one hour
# on steps that never touched it; restarting the clock here gives the slow
# part the full budget.
# Refresh the one-hour token after read-only install/build, before long publish/notary work.
- name: Re-mint daily repo token for publish
id: app_token_publish
if: steps.freshness.outputs.should_build == 'true'
@@ -337,12 +267,9 @@ jobs:
SHA: ${{ steps.freshness.outputs.head_sha }}
run: |
set -euo pipefail
# Kept at 12 even though the title shows 7: the freshness check above
# parses this back out of the body to decide whether main has moved.
# Freshness parses the 12-character source SHA from the body even though titles show seven.
short_sha="${SHA:0:12}"
# Why a file rather than an inline string: the publish step re-asserts
# this same body, and the freshness check only works if the two agree
# exactly. One source, written once, read twice.
# Write one body file for creation and reassertion; freshness parsing requires exact agreement.
notes_file="$RUNNER_TEMP/daily-release-notes.md"
cat >"$notes_file" <<EOF
Automated daily macOS and Windows dev build from commit \`$short_sha\`.
@@ -359,16 +286,8 @@ jobs:
(SmartScreen will warn about an unknown publisher). Every later switch,
including back to Stable, works in-app from there.
EOF
# Why create it up front: electron-builder then uploads into a known tag
# rather than inferring one from package.json.
#
# Why --draft: everything between here and the manifest check is a window
# where the release exists but has no installable assets. A draft is
# absent from the releases list and from listReleaseBuilds, so a job that
# dies in that window — including a hard kill by the job timeout, which
# runs no cleanup step at all — leaves something invisible rather than a
# tag the picker offers and the download 404s on. It is flipped live only
# after the manifest is verified.
# Precreate a known tag as draft so builder cannot infer it from package.json.
# Keep incomplete assets invisible until manifest verification, even if timeout kills the job without cleanup.
gh release create "$TAG" \
--repo "$DAILY_REPO" \
--title "$NAME" \
@@ -381,17 +300,13 @@ jobs:
if: steps.freshness.outputs.should_build == 'true'
uses: nick-fields/retry@v4
with:
# Why 45 like the release pipeline: an attempt is pack + notarize +
# upload, and the notary queue is the unbounded part. Why 2 attempts and
# not 3: a missed daily costs a day, and a third attempt buys less than
# it costs in runner time once the notary is that stuck.
# Allow two 45min pack/notarize/upload attempts; a third costs more runner time than a missed daily.
timeout_minutes: 45
max_attempts: 2
retry_wait_seconds: 30
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && ORCA_MAC_DAILY=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac --publish always
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && if [ -f config/scripts/build-mac-notarization-overlap.mjs ]; then ORCA_MAC_DAILY=1 node config/scripts/build-mac-notarization-overlap.mjs --publish always; else ORCA_MAC_DAILY=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac --publish always; fi
env:
# Why: electron-builder's github publisher targets the repo named in the
# config; the token must therefore carry write access to orca-daily.
# Give the publisher App write access to its configured orca-daily target.
GH_TOKEN: ${{ steps.app_token_publish.outputs.token }}
ORCA_DAILY_BUILD_VERSION: ${{ steps.daily.outputs.version }}
ORCA_BUILD_COMMIT: ${{ steps.daily.outputs.commit }}
@@ -399,21 +314,12 @@ jobs:
ORCA_REQUIRE_ORCAD_TEMPLATE: '1'
CSC_LINK: ${{ secrets.MAC_CERTS }}
CSC_KEY_PASSWORD: ${{ secrets.MAC_CERTS_PASSWORD }}
# Why all three: electron-builder's notarize step authenticates to the
# Apple notary service with the app-specific password, not with the
# signing cert. Omitting them fails the build rather than skipping it,
# since `notarize` is now on for this path.
# Require Apple id/app-specific password/team for notary authentication, separate from signing cert.
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
# Why a third mint: the upload step above allows two 45-minute attempts, so
# it can outlive the one-hour token minted before it. Every remaining step
# writes to the daily repo, including the failure path that discards the
# draft — a 401 there is exactly the stranded draft nobody can clean up.
#
# Why always(): a failed or cancelled upload is the case that needs this
# most, since the cleanup step below runs only on that path.
# Always refresh after upload: 2×45min can outlive the earlier token, including failed/cancelled uploads needing draft cleanup.
- name: Re-mint daily repo token for verify and cleanup
id: app_token_final
if: always() && steps.freshness.outputs.should_build == 'true'
@@ -424,9 +330,7 @@ jobs:
owner: stablyai
repositories: orca-daily
# Why: the updater resolves a tag, then fetches latest-mac.yml from it. A
# release missing that manifest is a tag the picker offers and the download
# 404s on, so fail loudly instead of leaving a broken entry.
# Verify latest-mac.yml before exposing its release to prevent picker downloads returning 404.
- name: Verify update manifest published
if: steps.freshness.outputs.should_build == 'true'
shell: bash
@@ -438,10 +342,7 @@ jobs:
assets="$(gh release view "$TAG" --repo "$DAILY_REPO" --json assets --jq '.assets[].name')"
echo "Published assets:"
echo "$assets"
# Why exit 1 without deleting here: the release is still a draft, so it is
# already invisible to users, and the failure handler below owns cleanup.
# Deleting inline under `set -e` would also let the delete's exit code
# preempt this explicit failure.
# Drafts are already invisible; let failure cleanup delete them so set -e cannot mask this explicit failure.
for required in latest-mac.yml; do
if ! grep -qx "$required" <<<"$assets"; then
echo "::error::Daily draft $TAG is missing $required; the updater could not install it."
@@ -453,9 +354,7 @@ jobs:
exit 1
fi
# Why this is the last mutating step: publishing the draft is what makes the
# build visible to listReleaseBuilds. Doing it only after the manifest check
# means the picker can never offer a release whose assets are incomplete.
# Publish only after manifest validation so the picker cannot offer incomplete assets.
- name: Publish the verified release
id: publish_live
if: steps.freshness.outputs.should_build == 'true'
@@ -467,33 +366,19 @@ jobs:
NOTES_FILE: ${{ steps.release.outputs.notes_file }}
run: |
set -euo pipefail
# --title again: electron-builder resolves this draft by tag and may
# rewrite its title on upload. Re-asserting here means the name the
# picker reads is the one composed above, whatever it did in between.
# --notes-file for the same reason, and it matters more: the next run's
# freshness check parses the source commit back out of this body, so a
# body the publisher overwrote would rebuild an unchanged main daily.
# Builder may rewrite title/body on upload; reassert both so the picker name and next run’s source-SHA freshness check stay correct.
gh release edit "$TAG" --repo "$DAILY_REPO" --draft=false --prerelease \
--title "$NAME" --notes-file "$NOTES_FILE"
echo "Published $TAG as \"$NAME\""
# Why: a draft left behind by a failed publish is invisible to users but still
# holds its tag name, so the next run for the same minute would collide.
#
# Why it is gated on publish_live not having succeeded: a later failure (the
# prune step) must not delete a release that already went live and that users
# may already be installing. A job killed by the outer timeout runs no steps
# at all — which is exactly why the release stays a draft until verified.
# Why cancelled() too: a run stopped from the Actions UI is not a failure(),
# so without it a manual cancel mid-publish would strand the draft.
# Remove failed/cancelled stranded drafts, never releases already published by publish_live, including after prune failure. Outer timeout skips cleanup, so retain draft status until verified.
- name: Discard the draft release on failure
if: >-
(failure() || cancelled()) && steps.release.outputs.tag != '' &&
steps.publish_live.outcome != 'success'
shell: bash
env:
# Fall back to the publish token: if the re-mint itself is what failed,
# the older token is the only one left and may still have time on it.
# If token refresh fails, let cleanup try the older token while it may still be valid.
GH_TOKEN: ${{ steps.app_token_final.outputs.token || steps.app_token_publish.outputs.token }}
TAG: ${{ steps.release.outputs.tag }}
run: |
@@ -504,33 +389,18 @@ jobs:
echo "::warning::Could not discard draft $TAG; remove it manually."
- name: Prune old daily releases
# Only after a live publish: $TAG is then a non-draft we must not delete,
# and failed runs should not reshuffle retention around a draft that the
# failure path is about to discard.
# Prune only after publication; failed runs leave draft disposal to cleanup.
if: steps.publish_live.outcome == 'success'
shell: bash
env:
GH_TOKEN: ${{ steps.app_token_final.outputs.token }}
# Protect the tag this run just shipped; at the retain cap, a bad sort
# can otherwise mark the newest release as stale and delete it.
# Never prune this run’s just-published tag at the retention cap.
TAG: ${{ steps.release.outputs.tag }}
run: |
set -euo pipefail
# Why: --cleanup-tag so pruning does not leave orphan tags behind that
# keep showing up in tag lists with no release or assets attached.
# Drafts are excluded so retention counts shipped builds only; a stale
# draft is handled by the failure path, not by the retention window.
#
# Sort by publishedAt (not createdAt). Many non-draft dailies can share
# one createdAt (bulk import / re-create), so createdAt ranking is
# unstable. publishedAt is real recency; tagName (...YYYYMMDDHHMM) is
# the deterministic tie-break.
#
# Why force $TAG to the front before slicing: a hard retain-window seat
# for this run's release. Dropping $TAG from the list *before* the slice
# would permanently keep retain+1 releases; skipping it only in the
# delete loop would under-prune when the sort is still wrong. Partition
# keeps relative order of every other tag.
# Delete pruned tags too; exclude drafts owned by failure cleanup. Rank by publishedAt/tagName:
# createdAt ties (bulk import/recreation) can evict the just-published release. Partition $TAG to
# the front before slicing so it owns exactly one retained seat without under-pruning or keeping retain+1.
jq_filter='map(select(.isDraft | not)) | sort_by(.publishedAt // "", .tagName) | reverse'
if [[ -n "${TAG:-}" ]]; then
jq_filter+=" | (map(select(.tagName == \"${TAG//\"/\\\"}\")) + map(select(.tagName != \"${TAG//\"/\\\"}\")))"
@@ -554,13 +424,7 @@ jobs:
echo "::warning::Could not prune $tag"
done <<<"$stale"
# Why detached dispatch, not a reusable workflow in this graph: the full suite
# is currently red on main. Inlining it here would fail the daily after the
# signed build already published. Same pattern as release-cut's post-release-e2e.
#
# Why --ref main plus the SHA input: daily tags live on orca-daily, not this
# repo, so there is no cut tag for `gh workflow run --ref`. The workflow file
# comes from main; checkout uses the exact commit this daily built.
# Dispatch E2E separately so main’s red full suite cannot fail an already-published daily. Use main’s workflow plus exact built SHA because daily tags live in another repo.
post-daily-e2e:
needs: build-daily-mac
if: ${{ needs.build-daily-mac.outputs.published == 'true' && needs.build-daily-mac.outputs.head_sha != '' }}
@@ -585,22 +449,14 @@ jobs:
done
echo "::warning::Failed to dispatch post-daily E2E for $SHA after 3 attempts."
# Why this runs after the mac leg rather than beside it: the tag and version
# are computed inside that job, so until it has run nothing else can name the
# release to upload into. The cost is small enough not to matter — the Windows
# leg measures ~7.5 min (install 2m45, build 35s, NSIS package 3m) against a
# ~9.5 min mac run, which keeps a daily run far inside its interval.
#
# Why `./` rather than a pinned `@main`: `uses:` resolves against the ref this
# file itself came from, which for an ordinary dispatch is main. Someone who
# deliberately points the Actions "Use workflow from" picker at a branch
# already gets that branch's copy of this entire file, so this follows the same
# rule instead of inventing a second one.
# Wait for mac release identity before Windows uploads; local ./ workflow follows the dispatch ref consistently.
build-daily-win:
needs: build-daily-mac
# Only once the mac release is actually live: there is no release to upload
# into otherwise, and the Windows workflow refuses to create one.
# Start Windows publication only after the mac release is live; its workflow refuses to create a missing release.
if: needs.build-daily-mac.outputs.published == 'true'
permissions:
contents: read
actions: read
uses: ./.github/workflows/dev-channel-win-build.yml
secrets: inherit
with:
@@ -609,3 +465,4 @@ jobs:
ref: ${{ needs.build-daily-mac.outputs.head_sha }}
version: ${{ needs.build-daily-mac.outputs.version }}
orcad_template: true
fast_release: false
+178
View File
@@ -0,0 +1,178 @@
name: Dependency security
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
paths:
- '**/package.json'
- '**/pnpm-lock.yaml'
- '**/pnpm-workspace.yaml'
- '**/.npmrc'
- 'native/**/Cargo.toml'
- 'native/**/Cargo.lock'
- 'config/dependency-security-exceptions.json'
- 'config/cargo-deny.toml'
- 'config/scripts/*dependency-security*'
- 'config/scripts/pnpm-cli-invocation.mjs'
- 'config/pnpmfile.cjs'
- '**/patches/**'
- '.github/workflows/dependency-security.yml'
- '.github/workflows/release-cut.yml'
- '.github/actions/install-node-dependencies/**'
- '.github/actions/prepare-dependency-audit/**'
- 'src/packages/process-host/**'
workflow_call:
inputs:
ref:
description: Exact release tag to audit using the workflow commit's tooling
required: true
type: string
runner:
description: Runner override for GitHub-hosted signing provenance.
type: string
default: ubuntu-24.04
secrets:
SLACK_WEBHOOK_URL:
description: Existing orca-releases incoming webhook
required: false
workflow_dispatch:
permissions:
contents: read
concurrency:
group: dependency-security-${{ inputs.ref && github.run_id || github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
ORCA_BACKGROUND_LAUNCH: '1'
jobs:
javascript:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ${{ inputs.runner || 'ubuntu-24.04' }}
timeout-minutes: 15
outputs:
audit_result: ${{ steps.audit.outcome }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
- uses: ./.github/actions/prepare-dependency-audit
if: inputs.ref != ''
with:
ref: ${{ inputs.ref }}
- name: Audit JavaScript lockfiles
id: audit
continue-on-error: true
shell: bash
env:
RELEASE_REF: ${{ inputs.ref }}
run: |
if [ -n "$RELEASE_REF" ]; then
pnpm check:dependencies candidate . mobile
else
pnpm check:dependencies
fi
- uses: actions/upload-artifact@v7
if: always()
with:
name: dependency-security-javascript
path: tmp/dependency-security/*.json
if-no-files-found: error
rust:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ${{ inputs.runner || 'ubuntu-24.04' }}
timeout-minutes: 20
outputs:
audit_result: ${{ steps.audit.outcome }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ github.workflow_sha }}
persist-credentials: false
- uses: actions/cache@v5
id: scanner
with:
path: ~/.cargo/bin/cargo-deny
key: cargo-deny-0.20.2-${{ runner.os }}-${{ runner.arch }}
- if: steps.scanner.outputs.cache-hit != 'true'
run: cargo install cargo-deny --locked --version 0.20.2
- uses: ./.github/actions/prepare-dependency-audit
if: inputs.ref != ''
with:
ref: ${{ inputs.ref }}
- name: Audit native Rust lockfiles
id: audit
continue-on-error: true
shell: bash
working-directory: ${{ inputs.ref != '' && 'candidate' || '.' }}
run: |
set -euo pipefail
report_dir="$GITHUB_WORKSPACE/tmp/dependency-security"
mkdir -p "$report_dir"
status=0
for manifest in native/*/Cargo.toml; do
name="$(basename "$(dirname "$manifest")")"
cargo deny --manifest-path "$manifest" --locked --config "$GITHUB_WORKSPACE/config/cargo-deny.toml" check advisories 2>&1 | tee "$report_dir/$name.log" || status=1
done
exit "$status"
- uses: actions/upload-artifact@v7
if: always()
with:
name: dependency-security-rust
path: tmp/dependency-security/*.log
if-no-files-found: error
notify:
needs: [javascript, rust]
if: >-
!cancelled() && github.repository == 'stablyai/orca' &&
(needs.javascript.outputs.audit_result == 'failure' ||
needs.rust.outputs.audit_result == 'failure' ||
needs.javascript.result == 'failure' || needs.rust.result == 'failure')
runs-on: ${{ inputs.runner || 'ubuntu-24.04' }}
timeout-minutes: 5
steps:
- name: Send dependency review to orca-releases
uses: actions/github-script@v8
continue-on-error: true
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
RELEASE_REF: ${{ inputs.ref }}
AUDIT_RESULTS: ${{ toJSON(needs) }}
with:
script: |
if (!process.env.SLACK_WEBHOOK_URL) {
core.notice('Slack notification unavailable for this run; see the dependency reports in Actions.')
return
}
const results = JSON.parse(process.env.AUDIT_RESULTS)
const checks = [['javascript', 'JavaScript'], ['rust', 'Rust']].map(([job, label]) => {
const result = results[job]
const status = result.result !== 'success' ? 'scan incomplete'
: result.outputs.audit_result === 'success' ? 'passed' : 'review suggested'
return `${label}: ${status}`
})
const target = context.payload.pull_request
? `PR #${context.payload.pull_request.number}`
: process.env.RELEASE_REF || context.ref
const runUrl = `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
const text = `Orca dependency review for ${target}\n${checks.join(' · ')}\nReports and logs: ${runUrl}`
let response
try {
response = await fetch(process.env.SLACK_WEBHOOK_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, mrkdwn: false, unfurl_links: false, unfurl_media: false }),
signal: AbortSignal.timeout(15000)
})
} catch {
throw new Error('Slack dependency review notification could not be sent')
}
if (!response.ok) {
throw new Error(`Slack dependency review notification returned HTTP ${response.status}`)
}
+147 -103
View File
@@ -1,45 +1,9 @@
name: Dev Channel Windows Build
# Why its own file rather than steps inlined into hourly/daily/adhoc: one copy of
# the Windows leg instead of three, and it stays dispatchable on its own so a
# Windows artifact can be rebuilt for an existing tag without paying for the mac
# leg's packaging and notarization again.
#
# Each mac workflow calls this as a `needs:`-gated job once its release is live,
# passing the tag it created. Both legs land in that one release, so a tag
# carries every platform it managed to build. Measured cost of the Windows leg is
# ~7.5 min (install 2m45, build 35s, NSIS package 3m) against a mac run of ~9.5
# min, so running it after the mac job keeps an hourly well inside its cron.
#
# Why unsigned: Windows release installers are signed by SignPath *after*
# packaging, and release-cut budgets 1h + 4h for those approval waits. That does
# not fit an hourly cadence and it does not fit "dispatch an adhoc build and go
# get coffee". So dev-channel Windows builds ship unsigned, which has one real
# consequence, handled in `src/shared/release-channel.ts`:
#
# electron-updater Authenticode-verifies every installer it downloads against
# the publisherName baked into the *installed* app's app-update.yml. Stable and
# RC carry 'SignPath Foundation', so they reject an unsigned dev installer and
# no future build can fix the copies already installed. Dev builds omit the
# name (config/electron-builder.config.cjs), so verification is skipped there.
#
# Net effect: the way *into* a dev channel on Windows is a one-time manual
# installer run. Every way out — to another dev build, or back to Stable — works
# through the in-app updater. The picker offers a download for exactly that jump.
#
# Called as a job by each mac workflow once its release is live, and separately
# dispatchable by hand to rebuild a Windows artifact for an existing tag without
# re-running the mac leg's twenty minutes of packaging and notarization:
#
# gh workflow run dev-channel-win-build.yml --ref main \
# -f channel=adhoc -f tag=v1.4.178-adhoc.20260819010203 \
# -f ref=<sha> -f version=1.4.178-adhoc.20260819010203
# Share one independently dispatchable Windows dev build across hourly/daily/adhoc. Adhoc may stage beside Mac and publish after its success. Dev installers are unsigned because SignPath approval waits exceed this cadence. Installed stable/RC publisherName rejects them, requiring one manual entry install; dev builds omit publisherName so later in-app channel changes work. Manual rebuild: dispatch main with channel/tag/ref/version for an existing release.
on:
# Why the inputs are duplicated: workflow_call does not accept `type: choice`,
# and workflow_dispatch wants it so the Actions UI offers a menu instead of a
# free-text box. The channel allowlist below is what actually enforces the set,
# since a workflow_call caller can pass any string.
# Duplicate inputs because workflow_call lacks choice UI; validate its free-string channels with the allowlist.
workflow_call:
inputs:
channel:
@@ -47,7 +11,7 @@ on:
required: true
type: string
tag:
description: Existing release tag in the channel repo
description: Release tag for staged or published artifacts
required: true
type: string
ref:
@@ -58,6 +22,19 @@ on:
description: Version to package, without the leading v
required: true
type: string
stage_only:
description: Package unsigned artifacts for a later gated publish job
required: false
type: boolean
default: false
fast_release:
description: Use the faster unsigned-build runner outside signing provenance
type: boolean
default: true
javascript_source_sha:
description: Restore this run's shared dev JavaScript instead of rebuilding it
type: string
default: ''
orcad_template:
description: Ship the orcad-template artifact the calling run built
required: false
@@ -86,39 +63,40 @@ on:
required: true
type: string
fast_release:
description: Use Blacksmith for unsigned Windows builds
type: boolean
default: true
permissions:
contents: read
concurrency:
# Keyed on the tag: re-dispatching the same tag must not race two uploads into
# one release, but two different channels (or two adhoc branches) are the
# ordinary case and must not wait on each other.
# Serialize per tag to prevent competing uploads while independent channels/branches proceed.
group: dev-channel-win-build-${{ inputs.tag }}
cancel-in-progress: false
jobs:
build-win:
if: github.repository == 'stablyai/orca'
# Why the same environment as the mac workflows: this needs the App token
# that can write to the dev-channel repos, and it should be reachable from
# exactly the same place those secrets already live.
# Use mac workflows’ same environment for cross-repo App secrets.
environment: adhoc-mac-build
# Why windows-2022 and not windows-latest: windows-latest moved to the
# Windows 2025 / VS 2026 image before node-gyp could detect VS 18, breaking
# native dependency install. release-cut pins the same image.
runs-on: windows-2022
# The fast image has VS Build Tools 2022; signed release provenance stays separate.
runs-on: ${{ inputs.fast_release && 'blacksmith-8vcpu-windows-2025' || 'windows-2022' }}
permissions:
contents: read
actions: read
timeout-minutes: 90
env:
ORCA_BACKGROUND_LAUNCH: '1'
JOBS: ${{ inputs.fast_release && 'max' || '' }}
NODE_OPTIONS: --max-old-space-size=4096
CHANNEL: ${{ inputs.channel }}
TAG: ${{ inputs.tag }}
VERSION: ${{ inputs.version }}
steps:
# Why vet before checkout: everything after this runs the checked-out code
# with a token that can write to a release repo. The mac leg already vetted
# the ref it resolved, but this workflow is dispatchable on its own, so it
# re-derives the same guarantee rather than trusting its caller.
# Re-vet source before checkout because standalone dispatch also exposes a release-write token.
- name: Vet the requested inputs
id: vetted
shell: bash
@@ -127,8 +105,7 @@ jobs:
REPO_URL: https://github.com/${{ github.repository }}
run: |
set -euo pipefail
# workflow_call takes channel as a free-text string, so the set is
# enforced here rather than by the input type.
# workflow_call inputs are free text; validate the channel set here.
case "$CHANNEL" in
hourly|daily|adhoc) ;;
*)
@@ -140,24 +117,18 @@ jobs:
echo "::error::ref must be a full 40-character commit SHA, got '$REQUESTED_SHA'. The dispatching workflow passes the commit it resolved."
exit 1
fi
# The tag must name the version being packaged, or the artifacts would
# land in a release describing a different build.
# Bind the tag to the packaged version so assets cannot describe a different build.
if [[ "$TAG" != "v$VERSION" ]]; then
echo "::error::tag '$TAG' does not match version '$VERSION'."
exit 1
fi
# And the version must carry the channel's own prerelease identifier,
# so an hourly artifact can never be uploaded into an adhoc release.
# Require the channel’s prerelease identifier so hourly assets cannot enter adhoc releases.
if [[ "$VERSION" != *"-$CHANNEL."* ]]; then
echo "::error::version '$VERSION' is not a $CHANNEL version."
exit 1
fi
# Reachability is the trust test: GitHub serves PR-only commits by SHA,
# so resolving the object is not proof a branch or tag of this repo
# reaches it. Bare + tree:0 keeps this to the commit graph; reftable
# because branches that differ only in casing cannot both be stored by
# the files backend on a case-insensitive runner disk, which fails the
# entire fetch rather than the one ref.
# Trust requires branch/tag reachability; GitHub serves PR-only SHAs too. Bare/tree:0 limits fetch to
# the commit graph; reftable avoids case-only refs failing the entire fetch on case-insensitive disks.
scratch="$RUNNER_TEMP/vet-requested-ref"
git init -q --bare --ref-format=reftable "$scratch"
git -C "$scratch" fetch -q --filter=tree:0 "$REPO_URL" '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'
@@ -171,13 +142,19 @@ jobs:
fi
echo "Vetted $CHANNEL $TAG at $REQUESTED_SHA"
- name: Enable parallel checkout
if: inputs.fast_release
shell: bash
run: |
git config --global checkout.workers 0
# These fresh native builds do not need MSBuild's incremental file tracking.
echo 'TrackFileAccess=false' >> "$GITHUB_ENV"
- name: Checkout the built commit
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
# This job only reads stablyai/orca and never pushes; every write goes
# to the dev-channel repo through a minted App token passed by env
# (zizmor: artipacked).
# Read this repo without persisted credentials; writes use the dev-repo env App token (zizmor: artipacked).
persist-credentials: false
# Why: a relay-era ref (before #27215) needs the Windows relay addon this build no longer compiles.
@@ -189,11 +166,7 @@ jobs:
exit 1
fi
# Why a guard and not just a build: the workflow file comes from the
# dispatch ref, but the packaging config comes from the *built* commit. A
# branch cut before Windows dev builds landed has a config that ignores
# ORCA_WIN_*, which would resolve publish.repo to the main repo. Say that
# in one sentence here rather than failing deep inside electron-builder.
# Reject old packaging configs that ignore ORCA_WIN_* and would publish to the main repo.
- name: Resolve the dev-channel packaging identity
shell: bash
run: |
@@ -216,28 +189,30 @@ jobs:
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
cache: pnpm
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
# Caches the Electron binary and electron-builder's tool downloads (nsis,
# winCodeSign). Same key shape as release-cut's Windows leg.
# Electron and builder releases fix what this downloads; the lockfile churns far more often.
- name: Resolve electron-builder download versions
id: eb-versions
shell: bash
run: node -p "const d = require('./package.json').devDependencies; 'versions=electron' + d.electron + '-builder' + d['electron-builder']" >> "$GITHUB_OUTPUT"
# Cache Electron/NSIS/winCodeSign tools with release-cut’s key shape.
- name: Cache electron-builder downloads
uses: actions/cache@v5
with:
path: |
~\AppData\Local\electron\Cache
~\AppData\Local\electron-builder\Cache
key: electron-builder-win-${{ hashFiles('pnpm-lock.yaml') }}
key: electron-builder-win-${{ steps.eb-versions.outputs.versions }}
restore-keys: |
electron-builder-win-
# Why retried: pnpm install triggers electron's postinstall, which pulls the
# Electron binary from GitHub release assets, and that CDN returns transient
# 504s often enough to lose a build to it.
# Why host-only: this job packages only for its own runner OS and
# architecture, so the default host-scoped install is deliberate.
# Retry transient Electron CDN 504s during install; keep host-scoped natives for this runner’s only target.
- name: Install dependencies
uses: nick-fields/retry@v4
with:
@@ -246,25 +221,62 @@ jobs:
retry_wait_seconds: 30
command: pnpm install --frozen-lockfile
# Why here: electron-builder's beforePack requires out/mobile-web, and the bundle
# build resolves React Native and Expo from mobile/node_modules.
# Build required out/mobile-web from mobile’s React Native/Expo install.
- uses: ./.github/actions/install-mobile-dependencies
if: inputs.javascript_source_sha == ''
# Why the packaging check runs before the 20-minute build: it only needs
# node_modules, and a stale config should cost seconds rather than a build.
# Validate packaging config after install, before the expensive build.
- name: Verify dev-channel packaging identity
shell: bash
run: node config/scripts/verify-dev-channel-packaging.mjs --channel="$CHANNEL" --platform=win32
- name: Download shared JavaScript
if: inputs.javascript_source_sha != ''
uses: actions/download-artifact@v8
with:
name: release-javascript
path: .build/release-javascript
- name: Build app
shell: bash
run: pnpm build:release
run: |
if [[ "$ORCA_RELEASE_JAVASCRIPT_SOURCE_SHA" != '' ]]; then
node config/scripts/release-javascript-artifact.mjs restore
pnpm run build:release:host
else
pnpm build:release
fi
env:
# Why unset ORCA_BUILD_IDENTITY: telemetry's transport gate accepts only
# 'stable' or 'rc', so leaving it unset keeps dev builds silent — which
# is correct for unvetted artifacts. Same as the mac dev channels.
ORCA_RELEASE_JAVASCRIPT_SOURCE_SHA: ${{ inputs.javascript_source_sha }}
ORCA_RELEASE_JAVASCRIPT_BUILD_IDENTITY: dev
# Unset dev build identity so official stable/rc telemetry stays silent.
ORCA_DIAGNOSTICS_TOKEN_URL: https://www.onorca.dev/diagnostics/token
# Inline so refs predating the wait can still use the trusted dispatch workflow.
- name: Wait for the qualified orcad template
if: inputs.fast_release && inputs.stage_only && inputs.orcad_template
uses: actions/github-script@v8
with:
script: |
const deadline = Date.now() + 30 * 60_000
while (Date.now() < deadline) {
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
...context.repo, run_id: context.runId, filter: 'latest', per_page: 100
})
const lanes = jobs.filter(job => job.name.startsWith('orcad-template / '))
const failed = lanes.find(job => job.status === 'completed' &&
['failure', 'cancelled', 'timed_out', 'action_required', 'startup_failure'].includes(job.conclusion))
if (failed) throw new Error(`Template qualification failed: ${failed.name}`)
const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts, {
...context.repo, run_id: context.runId, per_page: 100
})
const ready = artifacts.some(artifact => artifact.name === 'orcad-template' && !artifact.expired)
const completed = lanes.length > 0 && lanes.every(job => job.status === 'completed')
if (ready && (process.env.WAIT_FOR_TEMPLATE_COMPLETION !== '1' || completed)) return
await new Promise(resolve => setTimeout(resolve, 15_000))
}
throw new Error('Timed out waiting for the qualified orcad template')
# After the app build so nothing that cleans out/ can drop it; electron-builder ships it.
- name: Download the orcad deployment template
if: inputs.orcad_template
@@ -273,9 +285,9 @@ jobs:
name: orcad-template
path: out/orcad-template
# Why the token is minted here and not at the top: installation tokens live
# one hour and nothing before this point writes anything.
# Mint the one-hour write token after read-only setup/build.
- name: Mint dev channel repo token
if: ${{ !inputs.stage_only }}
id: app_token
uses: actions/create-github-app-token@v2
with:
@@ -284,11 +296,9 @@ jobs:
owner: stablyai
repositories: orca-${{ inputs.channel }}
# Why: electron-builder's publisher creates a release when it cannot find
# the tag ("publish: always"). If the mac leg failed and discarded its draft
# while this was building, that would mint a fresh, untitled, Windows-only
# release. Check first and fail instead.
# Require existing release: publish:always would recreate a discarded mac draft as an untitled Windows-only release.
- name: Confirm the target release still exists
if: ${{ !inputs.stage_only }}
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
@@ -300,6 +310,7 @@ jobs:
fi
- name: Publish Windows artifacts
if: ${{ !inputs.stage_only }}
uses: nick-fields/retry@v4
with:
# 30 is the pack + upload budget; there is no notary queue on this leg.
@@ -312,17 +323,12 @@ jobs:
ORCA_BUILD_COMMIT: ${{ inputs.ref }}
# beforePack and afterPack fail the package when the template is absent.
ORCA_REQUIRE_ORCAD_TEMPLATE: ${{ inputs.orcad_template && '1' || '' }}
# Why: electron-publish refuses to upload into a release published more
# than two hours ago (gitHubPublisher.getOrCreateRelease). The mac leg
# publishes the draft live as soon as *it* finishes, so a slow notary
# queue plus a slow Windows build can cross that line and silently drop
# every Windows asset. This is the documented escape hatch.
# Allow upload beyond publisher’s 2h release-age guard; slow mac notarization/Windows build can otherwise silently drop every asset.
EP_GH_IGNORE_TIME: 'true'
# Why: the updater resolves a tag, then fetches latest.yml from it. A
# release carrying the installer but not the manifest is one the picker
# offers and the update 404s on, so assert both.
# Require installer and latest.yml, preventing picker-visible releases with missing updater manifests.
- name: Verify Windows update manifest published
if: ${{ !inputs.stage_only }}
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
@@ -338,3 +344,41 @@ jobs:
fi
done
echo "Windows artifacts verified on $TAG."
- name: Package Windows artifacts without publishing
if: inputs.stage_only
uses: nick-fields/retry@v4
with:
timeout_minutes: 30
max_attempts: 2
retry_wait_seconds: 30
command: node config/scripts/ensure-native-runtime.mjs --runtime=electron; if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }; pnpm exec electron-builder --config config/electron-builder.config.cjs --win --publish never
env:
ORCA_BUILD_COMMIT: ${{ inputs.ref }}
ORCA_REQUIRE_ORCAD_TEMPLATE: ${{ inputs.orcad_template && '1' || '' }}
- name: Validate staged Windows artifacts
if: inputs.stage_only
shell: bash
run: |
set -euo pipefail
for required in orca-windows-setup.exe orca-windows-setup.exe.blockmap latest.yml; do
if [[ ! -s "dist/$required" ]]; then
echo "::error::Missing or empty staged Windows artifact: $required"
exit 1
fi
done
- name: Stage Windows artifacts for gated publishing
if: inputs.stage_only
uses: actions/upload-artifact@v7
with:
name: dev-channel-windows-${{ inputs.tag }}
path: |
dist/orca-windows-setup.exe
dist/orca-windows-setup.exe.blockmap
dist/latest.yml
if-no-files-found: error
retention-days: 3
compression-level: 0
overwrite: true
@@ -0,0 +1,132 @@
name: Dev Channel Windows Publish
on:
workflow_call:
inputs:
wait_for_mac_job:
description: Prefetch artifacts, then wait for this run's macOS job to succeed
type: string
default: ''
channel:
description: Dev channel whose live release receives the staged artifacts
required: true
type: string
tag:
description: Release tag used when staging the Windows artifacts
required: true
type: string
permissions:
contents: read
actions: read
concurrency:
group: dev-channel-win-publish-${{ inputs.tag }}
cancel-in-progress: false
jobs:
publish-win:
if: github.repository == 'stablyai/orca'
environment: adhoc-mac-build
runs-on: ubuntu-latest
timeout-minutes: 165
env:
CHANNEL: ${{ inputs.channel }}
TAG: ${{ inputs.tag }}
steps:
- name: Vet the requested inputs
shell: bash
run: |
set -euo pipefail
case "$CHANNEL" in
hourly|daily|adhoc) ;;
*) echo "::error::Unknown dev channel '$CHANNEL'."; exit 1 ;;
esac
if [[ ! "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+-${CHANNEL}\.[0-9]+$ ]]; then
echo "::error::Tag '$TAG' is not a $CHANNEL release tag."
exit 1
fi
# No run-id: artifacts must belong to this run, never another dispatch.
- name: Download the staged Windows artifacts
uses: actions/download-artifact@v8
with:
name: dev-channel-windows-${{ inputs.tag }}
path: staged-windows
- name: Validate staged Windows artifacts
shell: bash
run: |
set -euo pipefail
for required in orca-windows-setup.exe orca-windows-setup.exe.blockmap latest.yml; do
if [[ ! -s "staged-windows/$required" ]]; then
echo "::error::Missing or empty staged Windows artifact: $required"
exit 1
fi
done
- name: Wait for successful macOS publication
if: inputs.wait_for_mac_job != ''
uses: actions/github-script@v8
env:
MAC_JOB: ${{ inputs.wait_for_mac_job }}
with:
script: |
const deadline = Date.now() + 150 * 60_000
while (Date.now() < deadline) {
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
...context.repo, run_id: context.runId, filter: 'latest', per_page: 100
})
const mac = jobs.find(job => job.name === process.env.MAC_JOB)
if (mac?.status === 'completed') {
if (mac.conclusion !== 'success') throw new Error('macOS publication did not succeed')
return
}
await new Promise(resolve => setTimeout(resolve, 15_000))
}
throw new Error('Timed out waiting for macOS publication')
- name: Mint dev channel repo token
id: app_token
uses: actions/create-github-app-token@v2
with:
app-id: ${{ secrets.HOURLY_RELEASE_APP_ID }}
private-key: ${{ secrets.HOURLY_RELEASE_APP_PRIVATE_KEY }}
owner: stablyai
repositories: orca-${{ inputs.channel }}
- name: Confirm the target release is live
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
run: |
set -euo pipefail
draft="$(gh release view "$TAG" --repo "stablyai/orca-$CHANNEL" --json isDraft --jq '.isDraft')"
if [[ "$draft" != false ]]; then
echo "::error::Release $TAG is not live; refusing to publish Windows artifacts."
exit 1
fi
- name: Publish Windows artifacts
uses: nick-fields/retry@v4
with:
timeout_minutes: 5
max_attempts: 2
retry_wait_seconds: 10
command: gh release upload "$TAG" staged-windows/orca-windows-setup.exe staged-windows/orca-windows-setup.exe.blockmap staged-windows/latest.yml --clobber --repo "stablyai/orca-$CHANNEL"
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
- name: Verify Windows update manifest published
shell: bash
env:
GH_TOKEN: ${{ steps.app_token.outputs.token }}
run: |
set -euo pipefail
assets="$(gh release view "$TAG" --repo "stablyai/orca-$CHANNEL" --json assets --jq '.assets[].name')"
for required in latest.yml orca-windows-setup.exe orca-windows-setup.exe.blockmap; do
if ! grep -qx "$required" <<<"$assets"; then
echo "::error::$TAG is missing $required; Windows could not install this build."
exit 1
fi
done
+29 -37
View File
@@ -38,7 +38,7 @@ jobs:
check:
name: Build and test
if: github.event_name == 'pull_request' && github.event.pull_request.draft != true
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
- name: Checkout pull request
@@ -47,42 +47,40 @@ jobs:
persist-credentials: false
- name: Setup pnpm
# v5 avoids the v6 bootstrap/shim regression when pinning pnpm 10.
uses: pnpm/action-setup@v5
uses: pnpm/setup@v2
with:
version: 10.34.6
package_json_file: docs/site/package.json
run_install: false
package-json-file: docs/site/package.json
install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 22
# The repository root pins pnpm 12 while this isolated app pins 10;
# setup-node's cache probe runs at the root and would auto-switch.
node-version-file: docs/site/package.json
check-latest: true
# setup-node probes the store from the repository root, not this app.
package-manager-cache: false
- name: Install site dependencies
run: pnpm --ignore-workspace install --frozen-lockfile
run: pnpm install --frozen-lockfile
- name: Run package tests
run: pnpm --ignore-workspace test
run: pnpm test
- name: Lint site
run: pnpm --ignore-workspace lint
run: pnpm lint
- name: Typecheck site
run: pnpm --ignore-workspace exec tsc --noEmit --incremental false
run: pnpm exec tsc --noEmit --incremental false
- name: Build site
run: pnpm --ignore-workspace build
run: pnpm build
release_gate:
name: Authorize release
if: >-
github.repository == 'stablyai/orca' &&
(github.event_name == 'release' || github.event_name == 'workflow_dispatch')
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 5
outputs:
deploy: ${{ steps.validate.outputs.deploy }}
@@ -107,8 +105,7 @@ jobs:
tag="$INPUT_TAG"
[[ "$EVENT_NAME" == "release" ]] && tag="$RELEASE_TAG"
# Match the stable desktop format used by release-policy.yml. This
# intentionally rejects mobile-* and all -rc.* tags.
# Use release-policy’s stable desktop tag shape, excluding mobile-* and RC tags.
stable_tag=false
[[ "$tag" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]] && stable_tag=true
@@ -149,9 +146,7 @@ jobs:
fi
fi
# Resolve the tag to an immutable commit before handing it to the
# deployment job. This prevents a force-moved tag from changing the
# source between authorization and checkout.
# Pin the immutable commit before deployment so force-moving a tag cannot swap authorized source.
tag_ref_json=''
for attempt in 1 2 3; do
if tag_ref_json="$(gh api "repos/$GITHUB_REPOSITORY/git/ref/tags/$tag" 2>/dev/null)"; then
@@ -175,9 +170,7 @@ jobs:
fi
echo "commit_sha=$commit_sha" >> "$GITHUB_OUTPUT"
# Stable tags cut before the docs package was merged are valid Orca
# releases, but cannot produce a docs deployment. Skip them before
# requesting the protected production environment.
# Old stable tags may predate docs; skip before requesting the protected production environment.
docs_source_available=false
for attempt in 1 2 3; do
if gh api "repos/$GITHUB_REPOSITORY/contents/docs/site/package.json?ref=$commit_sha" >/dev/null 2>&1; then
@@ -207,7 +200,7 @@ jobs:
needs.release_gate.result == 'success' &&
needs.release_gate.outputs.deploy == 'true'
needs: release_gate
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 15
environment:
name: docs-production
@@ -222,21 +215,20 @@ jobs:
persist-credentials: false
- name: Setup pnpm
# v5 avoids the v6 bootstrap/shim regression when pinning pnpm 10.
uses: pnpm/action-setup@v5
uses: pnpm/setup@v2
with:
version: 10.34.6
package_json_file: docs/site/package.json
run_install: false
package-json-file: docs/site/package.json
install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version: 22
node-version-file: docs/site/package.json
check-latest: true
package-manager-cache: false
- name: Install site dependencies
run: pnpm --ignore-workspace install --frozen-lockfile
run: pnpm install --frozen-lockfile
- name: Verify Vercel credentials
env:
@@ -254,23 +246,23 @@ jobs:
VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
run: pnpm --ignore-workspace exec vercel pull --yes --non-interactive --environment=production
run: pnpm exec vercel pull --yes --non-interactive --environment=production
- name: Run package tests
run: pnpm --ignore-workspace test
run: pnpm test
- name: Lint site
run: pnpm --ignore-workspace lint
run: pnpm lint
- name: Typecheck site
run: pnpm --ignore-workspace exec tsc --noEmit --incremental false
run: pnpm exec tsc --noEmit --incremental false
- name: Build production site
run: pnpm --ignore-workspace exec vercel build --prod --non-interactive
run: pnpm exec vercel build --prod --non-interactive
- name: Deploy production site
env:
VERCEL_ORG_ID: ${{ secrets.VERCEL_ORG_ID }}
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
run: pnpm --ignore-workspace exec vercel deploy --prebuilt --prod --yes --non-interactive
run: pnpm exec vercel deploy --prebuilt --prod --yes --non-interactive
+238 -45
View File
@@ -27,12 +27,29 @@ on:
required: false
type: boolean
default: true
managed_ssh_shards:
description: JSON shard assignments for the selected managed SSH specs
required: false
type: string
default: '["1/2","2/2"]'
# Why: whole-file balancing cannot split one long spec; Playwright's native shards split its tests.
changed_e2e_test_shards:
description: JSON test-level shard assignments for the changed-spec lane; empty sizes shards by file count
required: false
type: string
default: ''
# Why: artifact names are unique per workflow run, and pr.yml calls this workflow twice.
artifact_suffix:
description: Appended to the build and trace artifact names so two calls in one run do not collide
required: false
type: string
default: ''
# Why: the app, orcad and its test template (--dark-test-template) build with it; releases refuse it.
layout_owner:
description: Set to main to build with ORCA_LAYOUT_OWNER=main and record save-divergence reports
required: false
type: string
default: ''
workflow_dispatch:
inputs:
ref:
@@ -43,6 +60,12 @@ on:
description: JSON array of specs to run; empty runs the full suite
required: false
type: string
managed_ssh_shards:
description: Managed SSH shard assignments (use one for small selections)
required: false
type: choice
default: '["1/2","2/2"]'
options: ['["1/2","2/2"]', '["1/1"]']
schedule:
# One complete daily reference run; targeted PR coverage remains unchanged.
- cron: '0 17 * * *'
@@ -51,8 +74,10 @@ jobs:
build:
name: build e2e app
if: inputs.test_files == '' || github.event_name != 'pull_request' || inputs.needs_build
runs-on: ubuntu-latest
runs-on: blacksmith-8vcpu-ubuntu-2404
timeout-minutes: 15
outputs:
changed_e2e_shards: ${{ steps.changed-e2e-shards.outputs.shards }}
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -62,6 +87,24 @@ jobs:
with:
ref: ${{ inputs.ref || github.ref }}
- name: Plan changed E2E allocation
id: changed-e2e-shards
env:
TEST_FILES_JSON: ${{ inputs.test_files }}
TEST_SHARDS_JSON: ${{ inputs.changed_e2e_test_shards }}
run: |
if [ -n "$TEST_SHARDS_JSON" ]; then
echo "shards=$(jq -c . <<<"$TEST_SHARDS_JSON")" >> "$GITHUB_OUTPUT"
exit 0
fi
node --input-type=module <<'NODE'
import { appendFileSync, existsSync } from 'node:fs'
const modulePath = './config/scripts/ci-e2e-job-selection.mjs'
const selection = existsSync(modulePath) ? await import(modulePath) : {}
const shards = selection.generalE2eShards?.(process.env.TEST_FILES_JSON) ?? ['1/1']
appendFileSync(process.env.GITHUB_OUTPUT, `shards=${JSON.stringify(shards)}\n`)
NODE
- name: Install native build tools
env:
ORCA_E2E_APT_PACKAGES: build-essential python3
@@ -71,17 +114,21 @@ jobs:
exit 0
fi
read -r -a packages <<< "$ORCA_E2E_APT_PACKAGES"
# Why apt-mirrors.txt: runner images resolve mirror+file: sources through it, so editing the lists alone leaves Azure in use.
for source in /etc/apt/apt-mirrors.txt /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
if [ -f "$source" ]; then
sudo sed -i 's|https*://azure\.archive\.ubuntu\.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' "$source"
fi
done
sudo tee /etc/apt/apt.conf.d/99-orca-e2e >/dev/null <<'APTCONF'
if [ -f .github/scripts/prepare-ubuntu-apt.sh ]; then
bash .github/scripts/prepare-ubuntu-apt.sh
else
# Dispatched refs can predate the shared preparation script.
for source in /etc/apt/apt-mirrors.txt /etc/apt/sources.list /etc/apt/sources.list.d/*.list /etc/apt/sources.list.d/*.sources; do
if [ -f "$source" ]; then
sudo sed -i 's|https*://azure\.archive\.ubuntu\.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' "$source"
fi
done
sudo tee /etc/apt/apt.conf.d/99-orca-ci >/dev/null <<'APTCONF'
Acquire::http::Timeout "15";
Acquire::https::Timeout "15";
Acquire::Retries "1";
APTCONF
fi
timeout 120 sudo apt-get update
timeout 300 sudo apt-get install -y "${packages[@]}" || timeout 300 sudo apt-get install -y --fix-missing "${packages[@]}"
@@ -107,7 +154,12 @@ jobs:
env:
VITE_EXPOSE_STORE: 'true'
ORCA_CI_MAIN_SOURCEMAPS: '0'
run: pnpm run build:electron-vite:parallel --mode e2e
LAYOUT_OWNER: ${{ inputs.layout_owner }}
run: |
if [ -n "$LAYOUT_OWNER" ]; then
export ORCA_LAYOUT_OWNER="$LAYOUT_OWNER"
fi
pnpm run build:electron-vite:parallel --mode e2e
# The CLI emits into out/main too; start only after Electron finishes clearing that tree.
- name: Build shared E2E CLI
@@ -125,6 +177,11 @@ jobs:
- wait: e2e-cli
# The switch-on specs skip, rather than fail, when the build left no marker.
- name: Check layout-owner build marker
if: inputs.layout_owner == 'main'
run: test -f out/main/ORCA_LAYOUT_OWNER_MAIN
- name: Check shared native runtime action
id: native-runtime-action
run: |
@@ -138,10 +195,8 @@ jobs:
- uses: ./.github/actions/build-mobile-web-bundle
if: hashFiles('.github/actions/build-mobile-web-bundle/action.yml') != ''
# Only the completed output upload overlaps the change from Node to Electron ABI.
# Foreground: a background JavaScript action reads the event payload while later steps rewrite it.
- name: Upload E2E build output
id: e2e-build-upload
background: true
uses: actions/upload-artifact@v7
with:
name: e2e-build-out${{ inputs.artifact_suffix }}
@@ -164,21 +219,17 @@ jobs:
with:
native-runtime: electron
- wait: e2e-build-upload
e2e:
name: e2e ${{ matrix.shard_name }}
needs: [build]
if: inputs.test_files == ''
runs-on: ubuntu-latest
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
# Fourteen scheduled runs averaged 24.6 minutes per shard; shards 4
# and 9 repeatedly hit the 30-minute cap. A 12-way trial still left
# one 30-minute shard, so 14 gives the suite enough failure headroom.
# Use 14 shards: scheduled average 24.6min and repeated 30min caps remained even in a 12-way trial.
- shard: '1/14'
shard_name: 1-of-14
- shard: '2/14'
@@ -233,14 +284,7 @@ jobs:
name: e2e-build-out${{ inputs.artifact_suffix }}
path: out/
# Why: the Electron suite is wall-clock constrained on OSS runners, but
# multiple Electron apps on one Xvfb VM contend on git/Chromium resources.
# Sharding keeps each VM at one Playwright worker while splitting the
# headless suite across separate runners.
# SKIP_BUILD makes Playwright globalSetup reuse the single build job's
# artifact instead of starting five concurrent electron-vite builds.
# ORCA_E2E_FORWARD_APP_LOGS keeps startup failures visible when Electron
# launches but never creates a BrowserWindow.
# Shard one worker per VM to avoid Electron/git/Xvfb contention. Reuse one SKIP_BUILD artifact and forward app logs for pre-window failures.
- name: Balance E2E shard from timing evidence
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -281,9 +325,7 @@ jobs:
if: matrix.shard == '1/14'
run: xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm exec playwright test tests/e2e/worktree-switch-first-paint.spec.ts --config tests/playwright.config.ts --project=electron-headful --workers=1
# Why: Playwright retains traces/screenshots only on failure. Uploading
# them as an artifact makes post-mortem debugging on CI possible without
# re-running locally.
# Upload failure traces/screenshots so CI failures are diagnosable without reproduction.
- name: Upload Playwright traces
if: failure() || cancelled()
uses: actions/upload-artifact@v7
@@ -294,11 +336,15 @@ jobs:
if-no-files-found: ignore
changed-e2e:
name: changed e2e specs
name: changed e2e specs (${{ matrix.shard }})
needs: [build]
if: inputs.test_files != '' && (github.event_name != 'pull_request' || inputs.run_changed_e2e)
runs-on: ubuntu-latest
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 45
strategy:
fail-fast: false
matrix:
shard: ${{ fromJSON(needs.build.outputs.changed_e2e_shards || '["1/1"]') }}
steps:
- name: Checkout
@@ -307,8 +353,7 @@ jobs:
ref: ${{ inputs.ref || github.ref }}
- name: Install native build and headless UI tools
# Why ripgrep: Quick Open's bounded host-side search requires rg instead of an
# unbounded inventory fallback; the paired fixture exercises that real boundary.
# Install rg for bounded Quick Open searches exercised by the paired fixture.
env: *e2e_tool_packages
run: *install_e2e_tools
@@ -325,6 +370,9 @@ jobs:
- name: Run changed E2E specs
env:
TEST_FILES_JSON: ${{ inputs.test_files }}
LAYOUT_OWNER: ${{ inputs.layout_owner }}
SHARD: ${{ matrix.shard }}
TEST_LEVEL_SHARDS: ${{ inputs.changed_e2e_test_shards != '' }}
run: |
if [ -f config/scripts/ci-e2e-job-selection.mjs ]; then
printf '%s\n' "$TEST_FILES_JSON" | node config/scripts/ci-e2e-job-selection.mjs > "$RUNNER_TEMP/general-e2e-specs"
@@ -346,18 +394,75 @@ jobs:
exit 0
fi
E2E_ENV=(SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1)
if printf '%s\n' "${TEST_FILES[@]}" | grep -Fxq 'tests/e2e/browser-image-link-focus.spec.ts'; then
E2E_ENV+=(ORCA_BACKGROUND_LAUNCH=1 ORCA_E2E_BROWSER_NATIVE_XVFB=1)
fi
E2E_PROJECT_ARGS=()
if [ -n "$LAYOUT_OWNER" ]; then
test -f out/main/ORCA_LAYOUT_OWNER_MAIN
mkdir -p layout-divergence
E2E_ENV+=(ORCA_LAYOUT_SAVE_DIVERGENCE_LOG="$PWD/layout-divergence/window-specs.jsonl")
E2E_ENV+=(PLAYWRIGHT_JSON_OUTPUT_FILE="$PWD/layout-divergence/window-specs-results.json")
E2E_PROJECT_ARGS+=('--reporter=list,json')
fi
if grep -l '@headful' "${TEST_FILES[@]}" >/dev/null; then
E2E_PROJECT_ARGS+=(--project=electron-headful)
fi
E2E_SELECTION_ARGS=(--shard="$SHARD")
if [ "$TEST_LEVEL_SHARDS" = true ]; then
mkdir -p ci-shards-changed
pnpm exec playwright test "${TEST_FILES[@]}" --config tests/playwright.config.ts --project=electron-headless "${E2E_PROJECT_ARGS[@]}" --shard="$SHARD" --list --reporter=json > ci-shards-changed/selected-discovery.json
elif [ -f config/scripts/ci-e2e-shard-plan.mjs ]; then
mkdir -p ci-shards-changed
pnpm exec playwright test "${TEST_FILES[@]}" --config tests/playwright.config.ts --project=electron-headless "${E2E_PROJECT_ARGS[@]}" --list --reporter=json > ci-shards-changed/discovery.json
export ORCA_SHARD_SOURCE_SHA="$(git rev-parse HEAD)"
node config/scripts/ci-e2e-shard-plan.mjs ci-shards-changed/discovery.json "$SHARD" ci-shards-changed
pnpm exec playwright test "${TEST_FILES[@]}" --config tests/playwright.config.ts --project=electron-headless "${E2E_PROJECT_ARGS[@]}" --test-list=ci-shards-changed/selected.txt --list --reporter=json > ci-shards-changed/selected-discovery.json
node config/scripts/ci-e2e-shard-plan.mjs --verify ci-shards-changed/assignment.json ci-shards-changed/selected-discovery.json
E2E_SELECTION_ARGS=(--test-list=ci-shards-changed/selected.txt)
fi
xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh env "${E2E_ENV[@]}" \
pnpm run test:e2e "${TEST_FILES[@]}" --workers=1 "${E2E_PROJECT_ARGS[@]}"
pnpm run test:e2e "${TEST_FILES[@]}" "${E2E_SELECTION_ARGS[@]}" --workers=1 "${E2E_PROJECT_ARGS[@]}"
- name: Upload changed E2E shard assignment
if: always()
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: changed-e2e-shard-${{ strategy.job-index }}${{ inputs.artifact_suffix }}-attempt-${{ github.run_attempt }}
path: ci-shards-changed/
retention-days: 14
if-no-files-found: warn
- name: Summarize layout save divergence
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
run: &summarize_layout_divergence |
# Either file names a run: a log with no report is unattributed, a report with no log is zero.
shopt -s nullglob
runs=(layout-divergence/*.jsonl layout-divergence/*-results.json)
for run in $(printf '%s\n' "${runs[@]%.jsonl}" | sed 's/-results\.json$//' | sort -u); do
node config/scripts/layout-save-divergence-summary.mjs "$run.jsonl" "$run-summary.json" "$run-results.json"
if [ -f "$run-results.json" ]; then
node config/scripts/ci-e2e-failure-summary.mjs "$run-results.json"
fi
done
- name: Upload layout save divergence report
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: layout-divergence-changed-${{ strategy.job-index }}${{ inputs.artifact_suffix }}-attempt-${{ github.run_attempt }}
path: layout-divergence/
retention-days: 14
if-no-files-found: warn
- name: Upload Playwright traces
if: failure() || cancelled()
uses: actions/upload-artifact@v7
with:
name: playwright-traces-changed${{ inputs.artifact_suffix }}
name: playwright-traces-changed-${{ strategy.job-index }}${{ inputs.artifact_suffix }}
path: test-results/
retention-days: 7
if-no-files-found: ignore
@@ -365,7 +470,7 @@ jobs:
ssh-browser-network-route:
name: ssh browser network route
if: inputs.test_files == '' || contains(inputs.test_files, 'tests/e2e/ssh-browser-network-execution-route.docker.unit.test.ts')
runs-on: ubuntu-latest
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
@@ -389,7 +494,7 @@ jobs:
inputs.test_files == '' ||
contains(inputs.test_files, 'tests/e2e/orcad-serve-mode-switch.spec.ts') ||
contains(inputs.test_files, 'tests/e2e/workspace-layout-oracle-headless.spec.ts')
runs-on: ubuntu-latest
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 45
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -410,12 +515,23 @@ jobs:
# The packaged orcad slot the T6-11 launcher runs: its own node-pty under the pinned Node.
# The template is what `orca serve`'s default selection materializes that slot from.
- name: Build this runner's orcad slot and template
env:
LAYOUT_OWNER: ${{ inputs.layout_owner }}
run: |
slot="$(node config/scripts/build-orcad-prebuilds.mjs --print-slot)"
pnpm build:orcad-prebuilds
pnpm build:orcad-prebuilds --require-slots "$slot"
pnpm build:orcad
node config/scripts/build-orcad-template.mjs --targets "$slot"
# The orcad host and its materialized template must use the Electron host’s layout owner.
if [ -n "$LAYOUT_OWNER" ]; then
export ORCA_LAYOUT_OWNER="$LAYOUT_OWNER"
pnpm build:orcad
node config/scripts/build-orcad-template.mjs --targets "$slot" --dark-test-template
test -f out/orcad/ORCA_LAYOUT_OWNER_MAIN
test -f out/orcad-template/ORCA_LAYOUT_OWNER_MAIN
else
pnpm build:orcad
node config/scripts/build-orcad-template.mjs --targets "$slot"
fi
- name: Switch serve hosts on one profile
if: inputs.test_files == '' || contains(inputs.test_files, 'tests/e2e/orcad-serve-mode-switch.spec.ts')
env:
@@ -423,6 +539,15 @@ jobs:
ORCA_E2E_ORCAD_SERVE: '1'
ORCA_E2E_FORWARD_APP_LOGS: '1'
run: xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts tests/e2e/orcad-serve-mode-switch.spec.ts --project=electron-headless --workers=1
# Only this job builds out/orcad, so the spec's orcad case runs here; the e2e shards run its Electron case.
- name: Phone Markdown on headless orcad serve
if: >-
!cancelled() &&
(inputs.test_files == '' || contains(inputs.test_files, 'tests/e2e/headless-serve-mobile-markdown.spec.ts'))
env:
SKIP_BUILD: '1'
ORCA_E2E_FORWARD_APP_LOGS: '1'
run: xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts tests/e2e/headless-serve-mobile-markdown.spec.ts --grep "headless orcad server" --project=electron-headless --workers=1
# The layout oracle's headless scenarios run on both serve hosts, so they need this job's orcad.
- name: Layout oracle on headless serve (orcad and Electron)
if: >-
@@ -431,7 +556,30 @@ jobs:
env:
SKIP_BUILD: '1'
ORCA_E2E_FORWARD_APP_LOGS: '1'
run: xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts tests/e2e/workspace-layout-oracle-headless.spec.ts --project=electron-headless --workers=1
LAYOUT_OWNER: ${{ inputs.layout_owner }}
run: |
REPORT_ARGS=()
if [ -n "$LAYOUT_OWNER" ]; then
test -f out/main/ORCA_LAYOUT_OWNER_MAIN
mkdir -p layout-divergence
export ORCA_LAYOUT_SAVE_DIVERGENCE_LOG="$PWD/layout-divergence/headless-oracle.jsonl"
export PLAYWRIGHT_JSON_OUTPUT_FILE="$PWD/layout-divergence/headless-oracle-results.json"
REPORT_ARGS=('--reporter=list,json')
fi
xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts tests/e2e/workspace-layout-oracle-headless.spec.ts --project=electron-headless --workers=1 "${REPORT_ARGS[@]}"
- name: Summarize layout save divergence
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
run: *summarize_layout_divergence
- name: Upload layout save divergence report
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: layout-divergence-headless${{ inputs.artifact_suffix }}-attempt-${{ github.run_attempt }}
path: layout-divergence/
retention-days: 14
if-no-files-found: warn
- uses: actions/upload-artifact@v7
if: failure()
with:
@@ -440,18 +588,18 @@ jobs:
retention-days: 7
if-no-files-found: ignore
# Every managed-orcad SSH spec on a Docker host, including #24979's relay-era conversion. Needs the
# Selected managed-orcad SSH specs on a Docker host; daily runs keep the full suite. Needs the
# orcad template for the fixture's target (Debian, linux-x64-glibc), which only this job builds.
orcad-ssh-docker:
name: ssh on managed orcad (Docker ${{ matrix.shard }})
needs: [build]
if: inputs.test_files == '' || contains(inputs.test_files, 'tests/e2e/ssh-orcad-')
runs-on: ubuntu-latest
runs-on: blacksmith-4vcpu-ubuntu-2404
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
shard: ['1/2', '2/2']
shard: ${{ fromJSON(inputs.managed_ssh_shards || '["1/2","2/2"]') }}
env:
ORCA_BACKGROUND_LAUNCH: '1'
steps:
@@ -472,23 +620,68 @@ jobs:
name: e2e-build-out${{ inputs.artifact_suffix }}
path: out/
# Built to a path the app does not look at by default, so the relay-era phase has no template.
# The switch-on leg deploys a switch-on server, so both sides of the host run one layout owner.
- name: Build the linux-x64-glibc orcad template
env:
LAYOUT_OWNER: ${{ inputs.layout_owner }}
run: |
pnpm build:orcad-prebuilds
pnpm build:orcad-prebuilds --require-slots linux-x64-glibc
node config/scripts/build-orcad-template.mjs --targets linux-x64-glibc
if [ -n "$LAYOUT_OWNER" ]; then
ORCA_LAYOUT_OWNER="$LAYOUT_OWNER" node config/scripts/build-orcad-template.mjs --targets linux-x64-glibc --dark-test-template
test -f out/orcad-template/ORCA_LAYOUT_OWNER_MAIN
else
node config/scripts/build-orcad-template.mjs --targets linux-x64-glibc
fi
mv out/orcad-template "$RUNNER_TEMP/orcad-convert-template"
# The Windows CLI matrix collects here too and skips: it needs a Windows host descriptor.
- name: Run managed SSH specs on a Docker host
env:
TEST_FILES_JSON: ${{ inputs.test_files }}
SKIP_BUILD: '1'
ORCA_E2E_SSH_DOCKER: '1'
ORCA_E2E_ORCAD_CONVERT_HOST: docker
ORCA_E2E_FORWARD_APP_LOGS: '1'
SHARD: ${{ matrix.shard }}
LAYOUT_OWNER: ${{ inputs.layout_owner }}
SHARD_INDEX: ${{ strategy.job-index }}
run: |
node --input-type=module <<'NODE' > "$RUNNER_TEMP/managed-ssh-specs"
import { existsSync } from 'node:fs'
const modulePath = './config/scripts/ci-e2e-job-selection.mjs'
const selection = existsSync(modulePath) ? await import(modulePath) : {}
// Older checkout refs retain their full-suite behavior.
const specs = selection.selectManagedSshE2eSpecs
? selection.selectManagedSshE2eSpecs(process.env.TEST_FILES_JSON)
: ['tests/e2e/ssh-orcad-']
for (const spec of specs) {
console.log(spec.endsWith('.spec.ts') ? spec.replace(/[.*+?^${}()|[\]\\]/g, '\\$&') + '$' : spec)
}
NODE
mapfile -t TEST_FILES < "$RUNNER_TEMP/managed-ssh-specs"
export ORCA_E2E_ORCAD_CONVERT_TEMPLATE="$RUNNER_TEMP/orcad-convert-template"
xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts tests/e2e/ssh-orcad- --project=electron-headless --workers=1 --shard="$SHARD"
REPORT_ARGS=()
if [ -n "$LAYOUT_OWNER" ]; then
test -f out/main/ORCA_LAYOUT_OWNER_MAIN
mkdir -p layout-divergence
export ORCA_LAYOUT_SAVE_DIVERGENCE_LOG="$PWD/layout-divergence/managed-ssh-$SHARD_INDEX.jsonl"
export PLAYWRIGHT_JSON_OUTPUT_FILE="$PWD/layout-divergence/managed-ssh-$SHARD_INDEX-results.json"
REPORT_ARGS=('--reporter=list,json')
fi
xvfb-run --auto-servernum bash .github/scripts/e2e-with-window-manager.sh pnpm exec playwright test --config tests/playwright.config.ts "${TEST_FILES[@]}" --project=electron-headless --workers=1 --shard="$SHARD" "${REPORT_ARGS[@]}"
- name: Summarize layout save divergence
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
run: *summarize_layout_divergence
- name: Upload layout save divergence report
if: always() && inputs.layout_owner == 'main'
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: layout-divergence-managed-ssh-${{ strategy.job-index }}${{ inputs.artifact_suffix }}-attempt-${{ github.run_attempt }}
path: layout-divergence/
retention-days: 14
if-no-files-found: warn
- uses: actions/upload-artifact@v7
if: failure()
with:
@@ -8,6 +8,9 @@ on:
required: false
type: string
permissions:
contents: read
jobs:
golden-e2e:
name: golden e2e ${{ matrix.platform }} experiment
@@ -41,6 +44,7 @@ jobs:
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
- name: Setup pnpm
uses: pnpm/setup@v2
+5 -29
View File
@@ -1,22 +1,6 @@
name: Homebrew Cask Bump
# Why: keep the stablyai/homebrew-orca tap in sync with GitHub Releases.
# Triggers:
# - workflow_call → release-cut.yml finished publishing a stable tag
# - workflow_dispatch → manual backfill for a specific tag
#
# The job computes the sha256 of orca-macos-arm64.dmg + orca-macos-x64.dmg
# from the release, templates the channel-specific cask in this repo with the
# new version and hashes, then PRs the result into stablyai/homebrew-orca.
# Stable tags update Casks/orca.rb; RC tags update Casks/orca@rc.rb.
#
# Why no `release.published` trigger: other products shipped from this repo
# (e.g. mobile under `mobile-v*`) also publish GitHub Releases. A blanket
# `release.published` trigger would fire this workflow for any of them and
# attempt to download `orca-macos-*.dmg` from a non-desktop release, at
# best 404ing and at worst (if the filter were weaker) rewriting the cask
# to a non-desktop version. The tap is only ever bumped by the desktop
# release pipeline calling this workflow directly.
# Update desktop stable/RC casks in stablyai/homebrew-orca with release DMG hashes through release-cut workflow_call or manual backfill. Avoid release.published because mobile/non-desktop releases lack those DMGs and must never rewrite casks.
on:
workflow_call:
@@ -132,10 +116,7 @@ jobs:
print(src)
PY
# Why: reuse the existing buf0-bot GitHub App instead of minting a new
# PAT. The app is installed org-wide, so it has access to
# stablyai/homebrew-orca automatically. GITHUB_TOKEN cannot push
# cross-repo; a short-lived installation token can.
# Use org-installed buf0-bot’s short-lived App token for cross-repo pushes; GITHUB_TOKEN is repo-scoped.
- name: Generate buf0-bot token
id: app-token
uses: actions/create-github-app-token@v3
@@ -166,8 +147,7 @@ jobs:
cp "$CASK_PATH" "tap/$CASK_PATH"
cd tap
# Why: commit author must match the buf0-bot app so commits are
# attributed to the bot in the tap's history / PR UI.
# Match the buf0-bot App’s commit author for tap history/PR attribution.
git config user.name "buf0-bot[bot]"
git config user.email "252831055+buf0-bot[bot]@users.noreply.github.com"
@@ -182,8 +162,7 @@ jobs:
git commit -m "$TITLE"
git push -u origin "$branch" --force
# Why: repeated backfills can race an existing PR branch, so edit
# the PR title if creation reports that one already exists.
# Backfills can race an existing branch; update its PR title when creation finds one.
gh pr create \
--title "$TITLE" \
--body "Automated bump of $CASK_PATH from stablyai/orca release $TAG." \
@@ -191,8 +170,5 @@ jobs:
--head "$branch" \
|| gh pr edit "$branch" --title "$TITLE"
# Why: squash-merge immediately rather than --auto. The tap has no
# required checks or branch protection, and `gh pr merge --auto`
# only activates when there's something to wait on — on a repo
# with no gates it silently no-ops, leaving the PR open forever.
# Squash immediately: this tap has no gates, so gh pr merge --auto silently no-ops.
gh pr merge "$branch" --squash --delete-branch
+45 -167
View File
@@ -1,39 +1,6 @@
name: Hourly macOS Dev Build
# Why: gives developers a signed macOS build of main every hour that the in-app
# updater can install directly, without waiting for an RC cut.
#
# Deliberately narrow scope:
# - macOS only. Other platforms keep using RC/stable.
# - No tests, no lint, no e2e. This channel trades safety for latency; PR CI
# and release-cut remain the gates that matter.
# - Signed AND notarized, exactly like a release. The notary round trip is the
# one slow step kept: macOS anchors a notarized app's TCC grants on identifier
# + team rather than on its cdhash, so those grants survive an update. Without
# a ticket every hourly reads as a new client and silently loses file access
# under Documents/Desktop/Downloads — 24 times a day.
#
# Artifacts publish to stablyai/orca-hourly, never to stablyai/orca: the main
# repo's releases atom feed exposes only its 10 newest entries, so 24 hourly
# tags a day would evict every stable/RC entry and break updates for real users.
#
# GITHUB_TOKEN is scoped to this repo and cannot publish there, so writes use a
# GitHub App installed on orca-hourly with Contents: Read and write. Its private
# key does not expire, unlike a PAT — nothing here needs yearly rotation, and the
# credential belongs to the org rather than to whoever created it.
#
# Provision the two secrets with `bash config/scripts/setup-hourly-release-token.sh`:
# HOURLY_RELEASE_APP_ID the App's numeric id
# HOURLY_RELEASE_APP_PRIVATE_KEY the App's .pem private key
#
# Installation tokens live one hour, so the build job mints twice. Install and
# build need no token at all, and notarization can hold the publish step for tens
# of minutes; minting again once the build is done starts the clock at the first
# call that actually uses it rather than burning a third of it on `pnpm install`.
# A pathological retry can still outrun the second token, but the release is an
# unpublished draft until the manifest check passes, so the damage is a stranded
# invisible draft — and the build-number query counts drafts, so it holds its
# number and the next run does not reuse it.
# Publish signed/notarized main builds to orca-hourly; notarization preserves TCC grants across updates. PR/release gates own validation. Separate repo avoids hourly tags crowding stable/RC Atom entries. Provision shared HOURLY_RELEASE_APP_ID/private key with setup-hourly-release-token.sh and App write access there. Refresh one-hour token after read-only build; a retry may still outlive it, leaving only an invisible draft whose build number remains reserved until manifests validate.
on:
schedule:
@@ -96,9 +63,7 @@ jobs:
echo "Forced dispatch; building $head_sha."
exit 0
fi
# The previous hourly records its source commit in the release body.
# Drafts are excluded: an unpublished leftover never shipped, so treating
# it as "the last build" would skip a build that never actually happened.
# Drafts never shipped; exclude them when comparing the previous hourly’s source commit.
last_body="$(gh release list --repo "$HOURLY_REPO" --limit 20 --json tagName,isDraft \
--jq 'map(select(.isDraft | not)) | .[0].tagName // empty' 2>/dev/null || true)"
if [[ -z "$last_body" ]]; then
@@ -122,10 +87,12 @@ jobs:
if: needs.preflight.outputs.should_build == 'true'
permissions:
contents: read
actions: read
uses: ./.github/workflows/node-server-tests.yml
with:
ref: ${{ needs.preflight.outputs.head_sha }}
build_template: true
fast_runners: false
build-hourly-mac:
needs:
@@ -137,11 +104,9 @@ jobs:
version: ${{ steps.hourly.outputs.version }}
head_sha: ${{ needs.preflight.outputs.head_sha }}
published: ${{ steps.publish_live.outcome == 'success' && 'true' || 'false' }}
# Hosted macos-15 cannot boot the packaged x64 terminal host that afterPack verifies.
runs-on: blacksmith-6vcpu-macos-15
# Why 150: it must exceed the worst case the retry budgets below can produce
# (install 3x10 + publish 2x45 = 120, plus ~25 for checkout/build/verify), or
# the job is killed mid-retry and no cleanup step runs at all. A typical run
# is far shorter — this is the notary queue's tail, not its median.
# Allow 150min for install 3×10 + publish 2×45 + ~25 setup/build/verify; outer timeout otherwise interrupts cleanup.
timeout-minutes: 150
env:
NODE_OPTIONS: --max-old-space-size=4096
@@ -152,10 +117,7 @@ jobs:
ref: ${{ needs.preflight.outputs.head_sha }}
# Version helpers only read HEAD; published versions come from git tags.
fetch-depth: 1
# Why: this job only reads stablyai/orca and never pushes; every write
# goes to the hourly repo through a minted App token passed by env.
# Not persisting the checkout credential shrinks the blast radius if a
# build step is compromised (zizmor: artipacked).
# Read this repo without persisted credentials; write orca-hourly via env App token (zizmor: artipacked).
persist-credentials: false
- name: Mint hourly repo token
@@ -168,21 +130,19 @@ jobs:
repositories: orca-hourly
- name: Setup pnpm
uses: pnpm/setup@v2
with:
install: false
uses: ./.github/actions/setup-pnpm
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
cache: pnpm
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
# Why: packaging bundles the pinned Node into the macOS terminal host. Keyed on the pin alone
# with no restore-keys, so a pin bump starts empty instead of carrying old archives forward.
# Cache bundled host Node by exact pin without restore keys.
- name: Locate the pinned Node runtime cache
shell: bash
run: echo "ORCA_NODE_RUNTIME_CACHE_DIR=$HOME/Library/Caches/orca-node-runtime" >>"$GITHUB_ENV"
@@ -203,8 +163,7 @@ jobs:
restore-keys: |
electron-builder-mac-
# Why both CPUs: the mac config packages x64 and arm64 from this arm64
# runner, so the install must carry both variants of the native optional deps.
# Install both x64/arm64 optional natives for both macOS packages.
- name: Install dependencies
uses: nick-fields/retry@v4
with:
@@ -213,12 +172,10 @@ jobs:
retry_wait_seconds: 30
command: pnpm install --frozen-lockfile --cpu=current,x64,arm64
# Why here: electron-builder's beforePack requires out/mobile-web, and the bundle
# build resolves React Native and Expo from mobile/node_modules.
# Build required out/mobile-web from mobile’s React Native/Expo install.
- uses: ./.github/actions/install-mobile-dependencies
# Why: signing is what makes an hourly installable over an existing Orca, so
# a missing cert must fail here rather than after a 20-minute build.
# Require cert before expensive build; signing enables installation over existing Orca.
- name: Verify macOS signing environment
run: node config/scripts/verify-macos-release-env.mjs
env:
@@ -236,30 +193,17 @@ jobs:
MAIN_REPO_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
# Existing titles, which carry the build number this series continues
# from. The script picks the number, because it restarts per base version
# and only the script knows which base this build resolved to.
#
# Why drafts count here but not in the freshness check: that check asks
# "did this commit ship", where a draft is a no. This one asks "is the
# number free", where a stranded draft still holds one.
# The script continues numbering for its resolved base version; stranded drafts still reserve numbers,
# although freshness checks exclude them because they never shipped.
names="$(gh release list --repo "$HOURLY_REPO" --limit 200 --json name \
--jq '.[].name // empty')"
# Why git tags, not GitHub releases: unpublishing a buggy cut deletes the
# GitHub release and leaves the tag. On 2026-09-14 we deleted v1.4.202's
# release for a bug; hourlies had already climbed to 1.4.203, then
# `gh release list` fell back to v1.4.201 and the next hourlies shipped
# as 1.4.202-hourly — which electron-updater will not install over
# 1.4.203-hourly or over the still-tagged 1.4.202. A separate token
# because GH_TOKEN above is the App's, scoped to the hourly repo. Empty
# on failure — the script then falls back to package.json, which is
# stale but never wrong enough to fail a build.
# Tags survive release deletion, preventing updater version regression. Use a separate token because
# GH_TOKEN’s App grant is scoped to the channel repo; lookup failure falls back to package.json.
# 2026-09-14: deleting v1.4.202 made release-list fallback ship 1.4.202-hourly below installed 1.4.203-hourly.
main_tags="$(GH_TOKEN="$MAIN_REPO_TOKEN" gh api \
"repos/$GITHUB_REPOSITORY/git/matching-refs/tags/v" \
--jq '.[].ref | sub("^refs/tags/"; "")' || true)"
# Already-shipped channel tags are a second floor: even if main's tag
# list is empty this run, a 1.4.203-hourly already out must not be
# followed by a 1.4.202-hourly.
# Already-shipped channel tags also floor versions: never follow installed 1.4.203-hourly with 1.4.202-hourly.
channel_tags="$(gh release list --repo "$HOURLY_REPO" --limit 200 --json tagName \
--jq '.[].tagName' || true)"
published="$main_tags"$'\n'"$channel_tags"
@@ -268,11 +212,7 @@ jobs:
node config/scripts/hourly-build-version.mjs \
>"$RUNNER_TEMP/hourly-identity.txt"
grep -E '^(version|build_number)=' "$RUNNER_TEMP/hourly-identity.txt"
# Why check rather than trust: the checkout above pins the resolved main commit, but a
# workflow_dispatch runs this file from whatever branch was dispatched. A
# branch that edits this step while main still has the old script yields
# an empty name and an untitled release — silent, and only visible once
# someone opens the releases page. Fail here instead.
# Dispatch YAML can be newer than the pinned source’s version script; reject missing output before publishing an untitled release.
if ! grep -q '^name=' "$RUNNER_TEMP/hourly-identity.txt"; then
echo "::error::hourly-build-version.mjs emitted no release name; this workflow and main's copy of the script are out of sync."
exit 1
@@ -283,9 +223,7 @@ jobs:
run: pnpm build:release
env:
NODE_OPTIONS: --max-old-space-size=4096
# Why: hourly builds are not an official channel — telemetry's transport
# gate accepts only 'stable' or 'rc', so leaving this unset keeps them
# silent, which is correct for unvetted dev artifacts.
# Unset dev identity so official-only stable/rc telemetry stays silent.
ORCA_DIAGNOSTICS_TOKEN_URL: https://www.onorca.dev/diagnostics/token
# After the app build so nothing that cleans out/ can drop it; electron-builder ships it.
@@ -295,11 +233,7 @@ jobs:
name: orcad-template
path: out/orcad-template
# Why a second mint: everything from here on writes to the hourly repo, and
# the notary round trip inside the publish step can be tens of minutes. The
# token minted at the top has already spent install + build of its one hour
# on steps that never touched it; restarting the clock here gives the slow
# part the full budget.
# Refresh write token after build to give long notary/upload work a full hour.
- name: Re-mint hourly repo token for publish
id: app_token_publish
uses: actions/create-github-app-token@v2
@@ -319,19 +253,10 @@ jobs:
SHA: ${{ needs.preflight.outputs.head_sha }}
run: |
set -euo pipefail
# Kept at 12 even though the title shows 7: the freshness check above
# parses this back out of the body to decide whether main has moved.
# Freshness parses the 12-character source SHA from the body even though titles show seven.
short_sha="${SHA:0:12}"
# Why create it up front: electron-builder then uploads into a known tag
# rather than inferring one from package.json.
#
# Why --draft: everything between here and the manifest check is a window
# where the release exists but has no installable assets. A draft is
# absent from the releases list and from listReleaseBuilds, so a job that
# dies in that window — including a hard kill by the job timeout, which
# runs no cleanup step at all — leaves something invisible rather than a
# tag the picker offers and the download 404s on. It is flipped live only
# after the manifest is verified.
# Precreate a known tag as draft so builder cannot infer it from package.json.
# Keep incomplete assets invisible until manifest verification, even if timeout kills the job without cleanup.
gh release create "$TAG" \
--repo "$HOURLY_REPO" \
--title "$NAME" \
@@ -354,17 +279,13 @@ jobs:
- name: Publish hourly macOS artifacts
uses: nick-fields/retry@v4
with:
# Why 45 like the release pipeline: an attempt is pack + notarize +
# upload, and the notary queue is the unbounded part. Why 2 attempts and
# not 3: a missed hourly costs an hour, and the next cron picks the same
# commit up, so a third attempt buys less than it costs in runner time.
# Allow two 45min pack/notarize/upload attempts; next hourly picks the same commit rather than paying for a third.
timeout_minutes: 45
max_attempts: 2
retry_wait_seconds: 30
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && ORCA_MAC_HOURLY=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac --publish always
command: node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON config/scripts/pinned-node-downloads.mjs darwin-arm64 darwin-x64 && node config/scripts/ensure-native-runtime.mjs --runtime=electron && if [ -f config/scripts/build-mac-notarization-overlap.mjs ]; then ORCA_MAC_HOURLY=1 node config/scripts/build-mac-notarization-overlap.mjs --publish always; else ORCA_MAC_HOURLY=1 pnpm exec electron-builder --config config/electron-builder.config.cjs --mac --publish always; fi
env:
# Why: electron-builder's github publisher targets the repo named in the
# config; the token must therefore carry write access to orca-hourly.
# Give publisher token write access to configured orca-hourly repo.
GH_TOKEN: ${{ steps.app_token_publish.outputs.token }}
ORCA_HOURLY_BUILD_VERSION: ${{ steps.hourly.outputs.version }}
ORCA_BUILD_COMMIT: ${{ steps.hourly.outputs.commit }}
@@ -372,17 +293,12 @@ jobs:
ORCA_REQUIRE_ORCAD_TEMPLATE: '1'
CSC_LINK: ${{ secrets.MAC_CERTS }}
CSC_KEY_PASSWORD: ${{ secrets.MAC_CERTS_PASSWORD }}
# Why all three: electron-builder's notarize step authenticates to the
# Apple notary service with the app-specific password, not with the
# signing cert. Omitting them fails the build rather than skipping it,
# since `notarize` is now on for this path.
# Require Apple id/app-specific password/team for notarization, separate from signing cert.
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
# Why: the updater resolves a tag, then fetches latest-mac.yml from it. A
# release missing that manifest is a tag the picker offers and the download
# 404s on, so fail loudly instead of leaving a broken entry.
# Verify latest-mac.yml before exposing release; missing manifests cause picker download 404s.
- name: Verify update manifest published
shell: bash
env:
@@ -393,10 +309,7 @@ jobs:
assets="$(gh release view "$TAG" --repo "$HOURLY_REPO" --json assets --jq '.assets[].name')"
echo "Published assets:"
echo "$assets"
# Why exit 1 without deleting here: the release is still a draft, so it is
# already invisible to users, and the failure handler below owns cleanup.
# Deleting inline under `set -e` would also let the delete's exit code
# preempt this explicit failure.
# Drafts are already invisible; let failure cleanup delete them so set -e cannot mask this explicit failure.
for required in latest-mac.yml; do
if ! grep -qx "$required" <<<"$assets"; then
echo "::error::Hourly draft $TAG is missing $required; the updater could not install it."
@@ -408,9 +321,7 @@ jobs:
exit 1
fi
# Why this is the last mutating step: publishing the draft is what makes the
# build visible to listReleaseBuilds. Doing it only after the manifest check
# means the picker can never offer a release whose assets are incomplete.
# Publish only after manifests verify so picker never offers incomplete assets.
- name: Publish the verified release
id: publish_live
shell: bash
@@ -420,21 +331,11 @@ jobs:
NAME: ${{ steps.hourly.outputs.name }}
run: |
set -euo pipefail
# --title again: electron-builder resolves this draft by tag and may
# rewrite its title on upload. Re-asserting here means the name the
# picker reads is the one composed above, whatever it did in between.
# Reassert the picker title after electron-builder may rewrite the tag’s draft on upload.
gh release edit "$TAG" --repo "$HOURLY_REPO" --draft=false --prerelease --title "$NAME"
echo "Published $TAG as \"$NAME\""
# Why: a draft left behind by a failed publish is invisible to users but still
# holds its tag name, so the next run for the same minute would collide.
#
# Why it is gated on publish_live not having succeeded: a later failure (the
# prune step) must not delete a release that already went live and that users
# may already be installing. A job killed by the outer timeout runs no steps
# at all — which is exactly why the release stays a draft until verified.
# Why cancelled() too: a run stopped from the Actions UI is not a failure(),
# so without it a manual cancel mid-publish would strand the draft.
# Delete failed/cancelled drafts but never publish_live releases, including after prune failure; outer timeouts run no cleanup.
- name: Discard the draft release on failure
if: >-
(failure() || cancelled()) && steps.release.outputs.tag != '' &&
@@ -451,34 +352,18 @@ jobs:
echo "::warning::Could not discard draft $TAG; remove it manually."
- name: Prune old hourly releases
# Only after a live publish: $TAG is then a non-draft we must not delete,
# and failed runs should not reshuffle retention around a draft that the
# failure path is about to discard.
# Prune only after live publish; failed runs leave draft disposal to cleanup.
if: steps.publish_live.outcome == 'success'
shell: bash
env:
GH_TOKEN: ${{ steps.app_token_publish.outputs.token }}
# Protect the tag this run just shipped; at the retain cap, a bad sort
# can otherwise mark the newest release as stale and delete it.
# Exclude the just-published tag from retention pruning.
TAG: ${{ steps.release.outputs.tag }}
run: |
set -euo pipefail
# Why: --cleanup-tag so pruning does not leave orphan tags behind that
# keep showing up in tag lists with no release or assets attached.
# Drafts are excluded so retention counts shipped builds only; a stale
# draft is handled by the failure path, not by the retention window.
#
# Sort by publishedAt (not createdAt). Many non-draft hourlies share one
# createdAt (bulk import / re-create), so createdAt ranking is unstable:
# reverse of a stable sort then drops the just-published tag past the
# retain window. publishedAt is real recency; tagName (...YYYYMMDDHHMM)
# is the deterministic tie-break.
#
# Why force $TAG to the front before slicing: a hard retain-window seat
# for this run's release. Dropping $TAG from the list *before* the slice
# would permanently keep retain+1 releases; skipping it only in the
# delete loop would under-prune when the sort is still wrong. Partition
# keeps relative order of every other tag.
# Delete pruned tags too; exclude drafts owned by failure cleanup. Rank by publishedAt/tagName:
# createdAt ties (bulk import/recreation) can evict the just-published release. Partition $TAG to
# the front before slicing so it owns exactly one retained seat without under-pruning or keeping retain+1.
jq_filter='map(select(.isDraft | not)) | sort_by(.publishedAt // "", .tagName) | reverse'
if [[ -n "${TAG:-}" ]]; then
jq_filter+=" | (map(select(.tagName == \"${TAG//\"/\\\"}\")) + map(select(.tagName != \"${TAG//\"/\\\"}\")))"
@@ -502,22 +387,14 @@ jobs:
echo "::warning::Could not prune $tag"
done <<<"$stale"
# Why this runs after the mac leg rather than beside it: the tag and version
# are computed inside that job, so until it has run nothing else can name the
# release to upload into. The cost is small enough not to matter — the Windows
# leg measures ~7.5 min (install 2m45, build 35s, NSIS package 3m) against a
# ~9.5 min mac run, which keeps a hourly run far inside its interval.
#
# Why `./` rather than a pinned `@main`: `uses:` resolves against the ref this
# file itself came from, which for an ordinary dispatch is main. Someone who
# deliberately points the Actions "Use workflow from" picker at a branch
# already gets that branch's copy of this entire file, so this follows the same
# rule instead of inventing a second one.
# Wait for mac tag/version before Windows upload; local ./ workflow consistently follows the dispatch ref.
build-hourly-win:
needs: build-hourly-mac
# Only once the mac release is actually live: there is no release to upload
# into otherwise, and the Windows workflow refuses to create one.
# Publish Windows only into the existing live mac release.
if: needs.build-hourly-mac.outputs.published == 'true'
permissions:
contents: read
actions: read
uses: ./.github/workflows/dev-channel-win-build.yml
secrets: inherit
with:
@@ -526,3 +403,4 @@ jobs:
ref: ${{ needs.build-hourly-mac.outputs.head_sha }}
version: ${{ needs.build-hourly-mac.outputs.version }}
orcad_template: true
fast_release: false
+32
View File
@@ -0,0 +1,32 @@
name: Layout owner E2E (manual)
run-name: Layout owner E2E ${{ inputs.ref || github.ref }}
on:
workflow_dispatch:
inputs:
ref:
description: Commit or ref to test (defaults to the workflow commit)
required: false
type: string
permissions:
contents: read
# This experiment costs about an hour of runner time; keep one suite active repository-wide.
concurrency:
group: layout-owner-e2e
cancel-in-progress: false
jobs:
layout_owner_switch_on:
name: layout owner (switch on)
permissions:
contents: read
uses: ./.github/workflows/e2e.yml
with:
ref: ${{ inputs.ref || github.sha }}
test_files: '["tests/e2e/workspace-layout-oracle-headless.spec.ts","tests/e2e/workspace-layout-oracle.spec.ts","tests/e2e/cli-file-open-position.spec.ts","tests/e2e/relaunch-first-terminal.spec.ts","tests/e2e/relaunch-sleeping-agents-resume-on-view.spec.ts","tests/e2e/terminal-cold-activation-deferral.spec.ts","tests/e2e/cli-first-worktree-single-terminal.spec.ts","tests/e2e/unopened-workspace-agent-single-resume.spec.ts","tests/e2e/terminal-pane-close-layout-consistency.spec.ts","tests/e2e/golden-file-open-edit-save.spec.ts","tests/e2e/golden-source-control-open-diff.spec.ts","tests/e2e/paired-cli-terminal-graph-sync-tab-retention.spec.ts","tests/e2e/floating-workspace-pane-height.spec.ts","tests/e2e/editor-model-host-isolation.spec.ts","tests/e2e/editor-tab-selection-restore.spec.ts","tests/e2e/finished-agent-ghost-resume.spec.ts","tests/e2e/live-background-terminal-mount-authority.spec.ts","tests/e2e/workspace-back-forward-navigation.spec.ts","tests/e2e/worktree-lineage-toggle-shortcut.spec.ts","tests/e2e/local-cli-terminal-retention.spec.ts","tests/e2e/default-tabs-mark-crash-relaunch.spec.ts","tests/e2e/terminal-cli-split-closed-before-mount.spec.ts","tests/e2e/layout-stream-loss-reopen.spec.ts","tests/e2e/terminal-tab-close-restart-persistence.spec.ts","tests/e2e/artificial-opencode-terminal-load.spec.ts","tests/e2e/headless-paired-remote-terminal-retention-memory.spec.ts","tests/e2e/headless-paired-remote-terminal-reveal-grid.spec.ts","tests/e2e/headless-paired-remote-terminal-stall-recovery.spec.ts","tests/e2e/headless-paired-remote-terminal-viewer-size.spec.ts","tests/e2e/headless-serve-browser-tab-restart.spec.ts","tests/e2e/headless-serve-cli-terminal-retention-parity.spec.ts","tests/e2e/headless-serve-desktop-activation.spec.ts","tests/e2e/headless-serve-focused-terminal-create.spec.ts","tests/e2e/headless-serve-chat-tab-placement.spec.ts","tests/e2e/headless-serve-mobile-chat-tab-order.spec.ts","tests/e2e/headless-serve-mobile-native-chat-identity.spec.ts","tests/e2e/headless-serve-session-history-naming-requests.spec.ts","tests/e2e/layout-compat-old-phone-on-new-server.spec.ts","tests/e2e/layout-compat-serve-restart-keeps-at-at-terminals.spec.ts","tests/e2e/paired-mobile-serve-restart-terminal-tabs.spec.ts","tests/e2e/paired-mobile-desktop-tab-composition.spec.ts","tests/e2e/paired-remote-terminal-serve-restart-binding.spec.ts","tests/e2e/paired-desktop-server-layout-frames.spec.ts","tests/e2e/paired-desktop-server-held-pane-reopen.spec.ts","tests/e2e/paired-desktop-server-file-open-group.spec.ts","tests/e2e/paired-desktop-server-layout-intents.spec.ts","tests/e2e/paired-desktop-server-layout-writes.spec.ts","tests/e2e/paired-desktop-server-editor-handover.spec.ts","tests/e2e/ssh-orcad-"]'
run_changed_e2e: true
needs_build: true
artifact_suffix: -layout-owner-switch-on
layout_owner: main
@@ -3,6 +3,9 @@ name: Linux Wayland GPU Sandbox
on:
workflow_dispatch:
permissions:
contents: read
jobs:
verify:
name: wayland terminal input
@@ -22,6 +25,7 @@ jobs:
uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
- name: Setup pnpm
uses: pnpm/setup@v2
+16
View File
@@ -8,10 +8,17 @@ on:
- 'src/main/macos-update-running-instances*'
- 'src/main/updater*'
- 'src/main/updater/**'
- 'native/macos-shipit-scheduling/**'
- 'config/scripts/macos-shipit-scheduling*'
- 'config/electron-builder.config.cjs'
- 'src/main/startup/main-process-quit*'
- 'src/main/window/main-window-state-lifecycle*'
- 'src/main/window/dashboard-popout-window*'
- 'src/packages/process-host/**'
- 'src/main/runtime/structured-agent-cli-writers*'
- 'src/main/providers/macos-tcc-login-shell*'
- 'src/shared/process-table-snapshot*'
- 'src/shared/darwin-process-table-capture*'
- 'src/shared/update-status-types.ts'
- 'pnpm-lock.yaml'
workflow_dispatch:
@@ -35,11 +42,15 @@ jobs:
- uses: ./.github/actions/install-node-dependencies
with:
native-runtime: node
cache-electron-package: 'true'
- name: Install the pinned Electron binary for native updater probes
run: node config/scripts/install-electron-package-binary.mjs
- name: Exercise native application registry and update shutdown
env:
ORCA_BACKGROUND_LAUNCH: '1'
run: >-
pnpm exec vitest run --config config/vitest.config.ts
config/scripts/macos-shipit-scheduling.test.mjs
src/main/macos-update-running-instances.test.ts
src/main/macos-update-running-instances.integration.test.ts
src/main/updater.mac-install.test.ts
@@ -48,3 +59,8 @@ jobs:
src/main/startup/main-process-quit-update-veto.test.ts
src/main/window/main-window-state-lifecycle.test.ts
src/main/window/dashboard-popout-window.test.ts
- name: Prove login-backed terminal writer census on the disposable runner
env:
ORCA_MACOS_LOGIN_CENSUS_CI: '1'
ORCA_BACKGROUND_LAUNCH: '1'
run: pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/structured-agent-cli-writers.test.ts
+14 -34
View File
@@ -1,8 +1,6 @@
name: Mobile Android Release
# Why a separate workflow from iOS: iOS releases go through App Store review,
# which can take days. Decoupling the triggers lets an Android release ship
# immediately without waiting on iOS, and vice versa.
# Release Android independently so App Store review cannot block its shipping cadence.
on:
push:
tags:
@@ -23,10 +21,7 @@ jobs:
android-build:
runs-on: ubuntu-latest
# Why: the "Create GitHub Release" step below uses the default GITHUB_TOKEN
# to call `gh release create`, which requires contents:write. Without this,
# the job builds the APK fine but fails at release time with
# "HTTP 403: Resource not accessible by integration".
# Grant contents:write for gh release create; build success alone does not authorize publication.
permissions:
contents: write
@@ -46,6 +41,7 @@ jobs:
uses: actions/setup-node@v6
with:
node-version: 24
check-latest: true
- name: Setup pnpm
uses: pnpm/setup@v2
@@ -74,10 +70,7 @@ jobs:
- name: Build Android release APK
run: cd android && ./gradlew assembleRelease
# Why: an install that fails with a missing certificate or a package-parse error is
# usually a download that died near the end, and the signature block sits in the last
# ~100 KB. Publishing the size and digest is what lets a reporter tell a truncated
# download apart from a bad build without anyone re-deriving them from the asset.
# Publish size/digest to distinguish truncated downloads (missing tail signature block) from bad APK builds.
- name: Checksum the APK
id: apk
run: |
@@ -92,10 +85,8 @@ jobs:
apk="${apks[0]}"
bytes="$(wc -c < "$apk" | tr -d ' ')"
sha256="$(sha256sum "$apk" | cut -d ' ' -f 1)"
# A sibling file in `sha256sum -c` format, so verifying a download is one command
# and no retyped digest. The asset globs below match *.apk and skip it. The ` *`
# marker is binary mode: Git Bash's sha256sum reads text mode as a licence to
# translate line endings while hashing, which would fail on a valid APK.
# Publish sha256sum -c sidecars, excluded by *.apk globs. Use binary “ *” mode:
# Git Bash text-mode newline translation can reject a valid APK.
printf '%s *%s\n' "$sha256" "$(basename "$apk")" > "$apk.sha256"
{
@@ -143,18 +134,13 @@ jobs:
'${{ steps.apk.outputs.bytes }}' '${{ steps.apk.outputs.sha256 }}'
}
# The section is always last, so dropping from its heading to EOF leaves the
# generated notes intact. \r* because a body round-tripped through the API has CRLFs.
# Verification is last; replace from heading to EOF, accepting API-roundtripped CRLFs.
drop_verification_section() {
sed '/^### Verify your download\r*$/,$d'
}
# Why: reuse the desktop release path's character-safe truncation so a multi-byte
# character cannot be split at the cap. The section's own length is reserved out of
# that cap, because appending after truncating would push a near-limit body past
# GitHub's API limit and fail the call — on the upload path, after --clobber has
# already replaced the assets. `wc -c` counts bytes, so the section's multi-byte
# characters over-reserve, which errs toward a shorter body.
# Truncate by characters like desktop releases and reserve the appended section inside GitHub’s cap,
# or --clobber could replace assets before notes fail. wc -c over-reserves multibyte section text safely.
write_notes_with_verification() {
NOTES_FILE="$notes_file" \
NOTES_RESERVE="$(verification_section | wc -c | tr -d ' ')" \
@@ -178,26 +164,21 @@ jobs:
--clobber \
android/app/build/outputs/apk/release/*.apk \
"${{ steps.apk.outputs.checksum }}"
# Why the body is rewritten too: --clobber replaced the APK, so a digest left
# over from the previous build now describes a file nobody can download, and a
# reader comparing against it would reject a good APK.
# --clobber replaced the APK; rewrite its digest so users cannot compare against obsolete bytes.
gh release view "$tag" --repo "$GITHUB_REPOSITORY" --json body --jq .body \
| drop_verification_section > "$notes_file"
write_notes_with_verification
gh release edit "$tag" --repo "$GITHUB_REPOSITORY" --notes-file "$notes_file"
else
# Why: release tags live on side branches, so GitHub's automatic
# previous-tag detection reaches back several releases; that body
# already exceeds the 125000-character API limit and grows each
# release. Pin the comparison base and cap the size.
# Side-branch release tags make automatic previous-tag detection span multiple releases and exceed
# the 125000-character API cap; pin the comparison base and truncate.
previous_tag="$(
gh release list --repo "$GITHUB_REPOSITORY" --limit 200 --json tagName --jq '.[].tagName' \
| grep '^mobile-android-v' | grep -Fxv "$tag" | sort -V | tail -1 || true
)"
if [ -n "$previous_tag" ]; then
# Why: gh writes the JSON error body to stdout on an HTTP error, so a
# non-empty file is not proof of success — gate on exit status.
# gh writes HTTP error JSON to stdout too; require exit success, not merely a nonempty file.
if ! gh api "repos/$GITHUB_REPOSITORY/releases/generate-notes" -X POST \
-f tag_name="$tag" \
-f target_commitish="$GITHUB_SHA" \
@@ -221,8 +202,7 @@ jobs:
"${{ steps.apk.outputs.checksum }}"
fi
# Why a separate job: the release is already published here, so a failed bump only leaves
# the site's APK download on the previous release and must not fail the release run.
# Keep site bump advisory after publication; failure only leaves its download link on the prior APK.
bump-apk-download:
needs: android-build
if: needs.android-build.outputs.publish_release == 'true'
+8 -17
View File
@@ -1,8 +1,6 @@
name: Mobile iOS Release
# Why a separate workflow from Android: iOS releases go through App Store
# review, which can take days. Decoupling the triggers lets an Android release
# ship immediately without waiting on iOS, and vice versa.
# Release iOS independently so App Store review cannot block Android.
on:
push:
tags:
@@ -23,21 +21,18 @@ on:
required: false
type: string
permissions:
contents: read
jobs:
ios-build:
# GitHub-hosted macOS runner: required for Xcode. Expo SDK 55's
# expo-modules-core declares swift_version 6.0 and uses Swift 6 syntax
# (@MainActor isolation). Xcode 16.x (macos-15) can't even parse it
# ("unknown attribute 'MainActor'"), so we need Xcode 26.x → macos-26.
# Use macos-26/Xcode 26 for Expo SDK 55’s Swift 6 syntax/isolation; Xcode 16 runners cannot parse it.
runs-on: macos-26
# Archive + upload is ~30–40m when healthy; 90m leaves margin for setup.
timeout-minutes: 90
env:
# Why: this job and ios-distribute resolve gems ~25 minutes apart, so both
# must install the committed Gemfile.lock exactly. Frozen turns a lockfile
# drift into a setup failure instead of two different fastlane versions in
# one release.
# Freeze Gemfile.lock in both jobs; installs ~25min apart must not resolve different fastlane versions.
BUNDLE_FROZEN: 'true'
outputs:
@@ -62,6 +57,7 @@ jobs:
uses: actions/setup-node@v6
with:
node-version: 24
check-latest: true
- name: Setup pnpm
uses: pnpm/setup@v2
@@ -102,12 +98,7 @@ jobs:
working-directory: mobile/ios
run: bundle exec pod install
# Why: `-allowProvisioningUpdates` + the App Store Connect API key can
# create/refresh provisioning profiles, but it cannot recreate the
# distribution certificate's PRIVATE KEY across runs. So we import a
# pre-exported distribution .p12 (created once via Apple Developer) into a
# throwaway keychain. The keychain is ephemeral to the runner and torn
# down with the VM; nothing secret is written to the repo.
# Import exported distribution .p12 into ephemeral runner keychain: provisioning updates/API key cannot recreate its private key.
- name: Import distribution certificate
env:
IOS_DIST_CERT_P12: ${{ secrets.IOS_DIST_CERT_P12 }}
+108 -54
View File
@@ -10,9 +10,7 @@ on:
- converted_to_draft
paths:
- 'mobile/**'
# Why all of src/shared: mobile imports hundreds of its modules, directly and through the RPC
# recordings and the host params check, so any shared edit can move a golden or break mobile's
# typecheck. Why the root lockfile: those modules resolve their packages from the root install.
# Route shared edits to mobile’s imports/RPC recordings/typecheck; root lockfile changes affect those modules’ root-resolved packages.
- 'src/shared/**'
# Mobile's Node RPC recording tools import this package.
- 'src/packages/process-host/**'
@@ -27,14 +25,18 @@ on:
- 'src/main/runtime/agent-session-*.ts'
- 'src/main/native-chat/agent-session-wire/**'
- 'src/main/runtime/structured-attention-original-delivery.test-fixture.ts'
# Why: this job holds the only checks that load the Fastfile, so edits to
# it or to the release workflow it guards must re-run them.
# Rerun the only Fastfile-loading checks for release-workflow/Fastfile edits.
- '.github/workflows/mobile.yml'
- 'config/mobile-ci.vitest.config.mjs'
- 'config/scripts/run-mobile-ci-tests.mjs'
# Mobile typechecks run through the root compiler selection.
- 'config/scripts/run-typecheck-projects-in-parallel.mjs'
- 'config/scripts/tsrs-typecheck-compiler.mjs'
- 'config/scripts/pinned-bun-executable.mjs'
- 'config/oxlint-code-quality-native-plugins.json'
- 'config/.bun-version'
- '.github/actions/install-node-dependencies/**'
- '.github/actions/setup-pnpm/**'
- '.github/actions/restore-pnpm-verification/**'
- '.github/workflows/mobile-ios-release.yml'
- 'config/scripts/mobile-release-check-scope*'
@@ -52,9 +54,7 @@ on:
- '!src/main/**/*.test.[jt]s'
- '!src/main/**/*.test.[jt]sx'
- '!src/main/**/*.test.[cm][jt]s'
# Why main too: two pull requests can each pass against their own base and disagree once both
# land, and `verify` never runs on main. The same source paths as above, less the CI inputs only
# `verify` reads.
# Check matching source inputs on main too: independently passing PRs can conflict after merge, and verify is PR-only.
push:
branches:
- main
@@ -75,7 +75,12 @@ on:
- '.github/workflows/mobile.yml'
- 'config/mobile-ci.vitest.config.mjs'
- 'config/scripts/run-mobile-ci-tests.mjs'
- 'config/scripts/run-typecheck-projects-in-parallel.mjs'
- 'config/scripts/tsrs-typecheck-compiler.mjs'
- 'config/scripts/pinned-bun-executable.mjs'
- 'config/.bun-version'
- '.github/actions/install-node-dependencies/**'
- '.github/actions/setup-pnpm/**'
# Desktop suites do not feed mobile; keep fixtures and test/toolchain config inputs.
- '!src/shared/**/*.test.[jt]s'
- '!src/shared/**/*.test.[jt]sx'
@@ -88,27 +93,24 @@ on:
- '!src/main/**/*.test.[cm][jt]s'
concurrency:
# Per commit on main, not per branch. GitHub cancels any PENDING run in a group when a new one
# queues, whatever `cancel-in-progress` says, so one shared main group drops the middle merge of
# three -- and a merge that breaks main there is exactly what `main-tests` checks for.
# Key main concurrency by commit: GitHub replaces pending runs even with cancel-in-progress:false, hiding middle-merge regressions.
group: mobile-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
permissions:
contents: read
jobs:
verify:
name: Mobile verify
if: github.event_name == 'pull_request' && github.event.pull_request.draft != true
# Why ARM: 209s of this job is Vitest and nothing here needs x86: no Android SDK, emulator, gradle,
# Hermes or Watchman, no docker, and no artifacts. The Gemfile.lock lists the generic `ruby`
# platform, so frozen bundler installs without an aarch64-linux entry.
# Use ARM for pure Vitest; no x86 native toolchain is needed and Gemfile.lock’s ruby platform supports frozen install.
runs-on: ubuntu-24.04-arm
env:
# Why: an unfrozen bundler silently re-resolves when Gemfile.lock drifts
# from the Gemfile, which is how the release jobs could land on different
# fastlane versions in the first place. Fail here instead.
# Freeze bundler to reject Gemfile drift before release jobs resolve different fastlane versions.
BUNDLE_FROZEN: 'true'
ORCA_MOBILE_TYPECHECK_COMPILER: 'native'
ORCA_MOBILE_TYPECHECK_COMPILER: 'bun'
defaults:
run:
@@ -139,24 +141,48 @@ jobs:
echo 'should_run=true' >> "$GITHUB_OUTPUT"
fi
# bundler-cache installs mobile/Gemfile.lock, so this job is also what
# proves the pinned fastlane the release workflow depends on still
# resolves — before a release run finds out.
- name: Setup Ruby and fastlane
id: ruby-tools
- name: Setup Ruby
id: ruby-setup
background: true
if: steps.ruby-scope.outputs.should_run != 'false'
uses: ruby/setup-ruby@v1
with:
ruby-version: '3.3'
bundler-cache: true
working-directory: mobile
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Why here: pr.yml runs these import rules before mobile's install, when mobile's tsconfig
# (which extends expo's) can't load, so its import cycles were never seen. Zero tolerance.
- wait: ruby-setup
# ruby/setup-ruby's bundler-cache saves on every miss; the platform warmup saves this key on main.
- name: Resolve fastlane bundle cache
id: ruby-bundle
if: steps.ruby-scope.outputs.should_run != 'false'
env:
GEMFILE_HASH: ${{ hashFiles('mobile/Gemfile', 'mobile/Gemfile.lock') }}
run: |
. /etc/os-release
printf 'key=mobile-bundle-v1-%s-%s-%s-%s\n' "$ID-$VERSION_ID" "$(ruby -e 'print RUBY_PLATFORM')" "$(ruby -e 'print RbConfig::CONFIG["ruby_version"]')" "$GEMFILE_HASH" >> "$GITHUB_OUTPUT"
- name: Restore fastlane bundle
if: steps.ruby-scope.outputs.should_run != 'false'
uses: actions/cache/restore@v5
with:
path: mobile/vendor/bundle
key: ${{ steps.ruby-bundle.outputs.key }}
# Installing proves the release’s pinned fastlane still resolves before a release needs it.
- name: Install fastlane
id: ruby-tools
background: true
if: steps.ruby-scope.outputs.should_run != 'false'
run: |
bundle config set --local path "$PWD/vendor/bundle"
bundle config set --local deployment true
bundle install --jobs 4
# Check mobile import cycles after Expo deps install; pre-install pr.yml cannot load its tsconfig.
- name: Lint mobile imports with mobile dependencies installed
working-directory: .
run: node node_modules/oxlint/bin/oxlint --config config/oxlint-code-quality-native-plugins.json mobile --deny-warnings
@@ -165,9 +191,7 @@ jobs:
- name: Typecheck
id: production-types
background: true
env:
GOMAXPROCS: '2'
run: node scripts/typecheck-mobile.mjs
run: node ../config/scripts/run-typecheck-projects-in-parallel.mjs --compiler=bun --bun-threads=2 --project=mobile/tsconfig.json
# Why a ratchet and not the raw typecheck: mobile/tsconfig.json excludes test files, so until
# tsconfig.test.json existed nothing checked them, and at introduction 127 of the 632 had
@@ -176,16 +200,9 @@ jobs:
- name: Typecheck tests (ratchet)
id: test-types
background: true
env:
GOMAXPROCS: '2'
run: node scripts/check-tests-typecheck-ratchet.mjs
# This includes the bridged replay of the whole recording corpus, which used to be a second
# step of its own behind RPC_FOUNDATION_BRIDGE=1. A gate nobody can forget to set is the point:
# it fails when a divergence class grows, when a divergence lands in no class at all, or when
# one of the 103 goldens inside the C1 page closure changes the verdict it is pinned to. It is
# ~3 min of test time on its own, and Vitest runs it on a worker beside the rest of the suite,
# so folding it in costs a fraction of that in wall time and one step less to skip.
# Run bridged recording corpus unconditionally: reject growing/unclassified divergence and changed verdicts among 103 C1 goldens. Share worker time instead of a separate skippable env-gated step.
- name: Detect mobile test inputs
id: test-scope
shell: bash
@@ -202,6 +219,7 @@ jobs:
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Test
if: steps.test-scope.outputs.should_run != 'false'
@@ -212,9 +230,7 @@ jobs:
# All checks read the completed install; compilers need not hold up test workers.
- wait: [production-types, test-types, ruby-tools]
# Reports, never gates: the goldens are content-addressed JSON, so their raw diff is hashes.
# This decodes which recorded behaviour the pull request moves into the job summary, and runs
# after a red Test too, when a reviewer most wants it.
# Decode hash-addressed golden behavior changes for review, including after failed tests; advisory only.
- name: Summarize RPC recording changes
if: ${{ !cancelled() }}
continue-on-error: true
@@ -228,11 +244,7 @@ jobs:
if: steps.ruby-scope.outputs.should_run != 'false'
run: ruby fastlane/fastfile_testflight_arguments_test.rb
# Why: nothing else in CI loads the Fastfile, so a syntax error, a broken
# require, or an undefined constant only surfaces mid-release — the
# ios-distribute job failed every run for six days that way. `lanes` just
# loads and lists, so it needs no App Store Connect credentials and makes
# no network calls to Apple.
# Load/list Fastfile lanes without Apple credentials/network to catch syntax/require/constants before release; ios-distribute failed six days without this guard.
- name: Smoke-check the Fastfile
if: steps.ruby-scope.outputs.should_run != 'false'
env:
@@ -250,10 +262,55 @@ jobs:
- name: Check formatting
run: pnpm format:check
# Reports, never gates: flags in the job summary when a pull request changes the Expo fingerprint,
# i.e. dev clients need a native rebuild. Only mobile's install, without scripts: the fingerprint
# hashes project sources and dependency paths, never installed package contents.
native-fingerprint:
name: Mobile native fingerprint
if: github.event_name == 'pull_request' && github.event.pull_request.draft != true
runs-on: ubuntu-24.04
timeout-minutes: 15
continue-on-error: true
defaults:
run:
working-directory: mobile
steps:
- name: Checkout
uses: actions/checkout@v6
with:
fetch-depth: 2
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version-file: package.json
package-manager-cache: false
- name: Setup pnpm
uses: ./.github/actions/setup-pnpm
# The base commit may predate the fingerprint tools, so run the head's copy against both.
- name: Compare native fingerprint with the base
shell: bash
run: |
set -euo pipefail
tools="$RUNNER_TEMP/native-fingerprint"
mkdir -p "$tools"
cp scripts/native-client.mjs scripts/native-build-cache.cjs "$tools/"
pnpm install --frozen-lockfile --ignore-scripts
node "$tools/native-client.mjs" fingerprint --project . --out "$tools/head.json"
git checkout --quiet --detach HEAD^1
pnpm install --frozen-lockfile --ignore-scripts
node "$tools/native-client.mjs" fingerprint --project . --out "$tools/base.json"
node "$tools/native-client.mjs" compare "$tools/base.json" "$tools/head.json" \
--project . --summary "$GITHUB_STEP_SUMMARY"
main-tests:
name: Mobile tests on main
env:
ORCA_MOBILE_TYPECHECK_COMPILER: 'native'
ORCA_MOBILE_TYPECHECK_COMPILER: 'bun'
if: github.event_name == 'push'
# Why ARM: the same pure-Node Vitest run `verify` makes on a pull request.
runs-on: ubuntu-24.04-arm
@@ -268,8 +325,10 @@ jobs:
with:
persist-credentials: false
# Restoring the 650 MB store took longer than downloading these packages.
- uses: ./.github/actions/install-node-dependencies
with:
cache-pnpm-store: 'false'
cache-dependency-path: |
pnpm-lock.yaml
mobile/pnpm-lock.yaml
@@ -280,24 +339,19 @@ jobs:
- name: Typecheck
id: production-types
background: true
env:
GOMAXPROCS: '2'
run: node scripts/typecheck-mobile.mjs
run: node ../config/scripts/run-typecheck-projects-in-parallel.mjs --compiler=bun --bun-threads=2 --project=mobile/tsconfig.json
- name: Typecheck tests (ratchet)
id: test-types
background: true
env:
GOMAXPROCS: '2'
run: node scripts/check-tests-typecheck-ratchet.mjs
# The whole suite, not only the recordings: the mutant and page-bridge suites read the goldens
# too. A red run here names each golden and the `rpc:record` command; the author of the merge
# that turned it red re-records in a follow-up.
# Run all golden consumers, including mutant/page-bridge suites; failures name the recording and rpc:record command.
- name: Setup Bun for tests
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Test
env:
+3 -1
View File
@@ -21,7 +21,7 @@ jobs:
fail-fast: false
matrix:
node: ['24', '26']
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- name: Checkout
@@ -42,4 +42,6 @@ jobs:
uses: ./.github/workflows/unit-tests.yml
with:
node_versions: '["24", "26"]'
# Shards restore the native cache primed above, so both use the same hosted cache.
runner: ubuntu-24.04
shards: ${{ needs.unit_plan.outputs.shards }}
+133 -45
View File
@@ -14,7 +14,7 @@ on:
- 'pnpm-workspace.yaml'
- 'tsconfig.json'
- '.npmrc'
- '.pnpmfile.cjs'
- 'config/pnpmfile.cjs'
- '.github/actions/install-node-dependencies/**'
- '.github/actions/restore-pnpm-verification/**'
- '.github/actions/prepare-headless-compiler/**'
@@ -27,6 +27,10 @@ on:
description: Also merge every lane's slot into the desktop orcad template artifact
type: boolean
default: false
fast_runners:
description: Use Blacksmith where the platform and architecture are available
type: boolean
default: true
# Release and dev builds qualify every platform for the template they ship.
workflow_call:
inputs:
@@ -38,6 +42,10 @@ on:
description: Upload each lane's release slot and merge them into the orcad-template artifact
type: boolean
default: false
fast_runners:
description: Use Blacksmith where the platform and architecture are available
type: boolean
default: true
# Requalify fresh builds daily; publish missing immutable Windows cache keys.
schedule:
- cron: '30 11 * * *'
@@ -55,7 +63,7 @@ jobs:
if: >-
github.event_name == 'push' ||
(github.event_name == 'pull_request' && github.event.pull_request.draft != true)
runs-on: ubuntu-latest
runs-on: ${{ inputs.fast_runners && 'blacksmith-2vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 5
outputs:
should_run: ${{ steps.scope.outputs.should_run || steps.graph.outputs.should_run }}
@@ -70,6 +78,7 @@ jobs:
- uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
package-manager-cache: false
- name: Detect headless-server build and test inputs
id: scope
@@ -128,7 +137,9 @@ jobs:
fi
persistence:
needs: changes
needs: [changes, mobile_web_bundle]
permissions:
contents: read
concurrency:
group: node-server-persistence-${{ matrix.os }}-${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' }}
@@ -136,16 +147,28 @@ jobs:
# Drafts re-trigger at readiness; inherited caller events must not depend on PR fields.
if: >-
${{ !cancelled() && needs.changes.outputs.should_run != 'false' &&
(needs.changes.outputs.qualification == 'false' || needs.mobile_web_bundle.result == 'success') &&
(github.event_name != 'pull_request' || github.event.pull_request.draft != true) }}
strategy:
fail-fast: false
matrix:
os: ${{ fromJSON(needs.changes.outputs.runners || '["ubuntu-22.04","ubuntu-24.04-arm","macos-15","macos-15-intel","windows-2022","windows-11-arm"]') }}
runs-on: ${{ matrix.os }}
# Keep platform keys stable for slot names and Windows cross-compilation. PR and scheduled runs use the
# free hosted label; only explicit fast_runners callers rent Blacksmith where it has the platform.
runs-on: >-
${{ inputs.fast_runners && fromJSON('{"ubuntu-22.04":"blacksmith-8vcpu-ubuntu-2204","ubuntu-24.04-arm":"blacksmith-8vcpu-ubuntu-2404-arm","macos-15":"blacksmith-6vcpu-macos-15"}')[matrix.os] || matrix.os }}
timeout-minutes: 20
env:
ORCA_BACKGROUND_LAUNCH: '1'
JOBS: ${{ (inputs.fast_runners || github.event_name == 'pull_request' || github.event_name == 'schedule') && 'max' || '' }}
steps:
- name: Parallelize Windows checkout
if: (inputs.fast_runners || github.event_name == 'pull_request' || github.event_name == 'schedule') && runner.os == 'Windows'
shell: bash
run: |
git config --global checkout.workers 0
# These fresh native builds do not need MSBuild's incremental file tracking.
echo 'TrackFileAccess=false' >> "$GITHUB_ENV"
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
@@ -154,7 +177,8 @@ jobs:
id: dependencies
with:
native-runtime: ${{ runner.os == 'Windows' && 'node' || 'none' }}
cache-pnpm-store: ${{ runner.os != 'Windows' }}
# Template qualification must not wait for large download-cache uploads.
cache-pnpm-store: ${{ !inputs.build_template && runner.os != 'Windows' }}
cache-pnpm-store-lookup-only: 'true'
# Design D7 upgrade and rollback: the last Bun orcad, built from a main commit that shipped
# it, using the current Bun pin beside this checkout's Node slot; hand-over skips once PROTOCOL_VERSION
@@ -163,6 +187,7 @@ jobs:
if: runner.os == 'Linux'
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Build the last Bun orcad for the cross-runtime tests
id: bun-orcad
background: true
@@ -182,19 +207,26 @@ jobs:
id: orcad-prebuild
with:
resolve-windows-cache: >-
${{ !inputs.build_template && inputs.ref == '' &&
${{ inputs.fast_runners || (!inputs.build_template && inputs.ref == '' &&
(github.event_name == 'pull_request' ||
(github.ref == 'refs/heads/main' && contains(fromJSON('["push","schedule","workflow_dispatch"]'), github.event_name))) }}
restore-windows-cache: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
(github.ref == 'refs/heads/main' && contains(fromJSON('["push","schedule","workflow_dispatch"]'), github.event_name)))) }}
restore-windows-cache: ${{ inputs.fast_runners || github.event_name == 'pull_request' || github.event_name == 'push' }}
# The template lane cross-compiles arm64 too, as it does for the process-table addon.
- uses: ./.github/actions/prepare-windows-cli-launcher
if: runner.os == 'Windows'
with:
arches: ${{ (inputs.build_template && matrix.os == 'windows-2022') && 'x64 arm64' || (runner.arch == 'ARM64' && 'arm64' || 'x64') }}
- uses: actions/download-artifact@v8
if: needs.changes.outputs.qualification != 'false'
with:
name: orcad-mobile-web-bundle
path: out/mobile-web
# Narrow test-only runs skip the shared producer but still need a page.
- uses: ./.github/actions/build-mobile-web-bundle
if: needs.changes.outputs.qualification == 'false'
- run: pnpm build:orcad
env:
ORCA_REUSE_PREPARED_WINDOWS_SECURITY: ${{ github.event_name == 'pull_request' && runner.os == 'Windows' && steps.dependencies.outcome == 'success' }}
ORCA_REUSE_PREPARED_WINDOWS_SECURITY: ${{ runner.os == 'Windows' && steps.dependencies.outcome == 'success' }}
- wait: bun-orcad
- run: pnpm test:node-server --artifact ${{ runner.os == 'Linux' && '--cross-runtime' || '' }}
env:
@@ -274,11 +306,12 @@ jobs:
node out/orcad/orcad.js --orcad-smoke-load-check
node out/orcad/orcad.js --orcad-profile-state-preflight 00000000-0000-4000-8000-000000000018 | tee "$RUNNER_TEMP/preflight.json"
node -e "const r=JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'));if(r.runtime!=='node'||!/^24\./.test(r.runtimeVersion))process.exit(1)" "$RUNNER_TEMP/preflight.json"
- name: Save the freshly qualified Windows server prebuild on main
- name: Save the freshly qualified Windows server prebuild
if: >-
success() && runner.os == 'Windows' && (runner.arch == 'X64' || runner.arch == 'ARM64') &&
!inputs.build_template && inputs.ref == '' && github.ref == 'refs/heads/main' &&
contains(fromJSON('["push","schedule","workflow_dispatch"]'), github.event_name) &&
((inputs.fast_runners && github.event_name == 'workflow_dispatch') ||
(!inputs.build_template && inputs.ref == '' && github.ref == 'refs/heads/main' &&
contains(fromJSON('["push","schedule","workflow_dispatch"]'), github.event_name))) &&
steps.orcad-prebuild.outputs.cache-identity-outcome == 'success' &&
steps.orcad-prebuild.outputs.cache-key != ''
continue-on-error: true
@@ -298,15 +331,14 @@ jobs:
# A rerun attempt re-uploads under the same name, which v4 otherwise refuses.
overwrite: true
# The container lanes take the page from here: it is platform-independent, and building it needs
# mobile/node_modules, which those images never install.
# Every qualification lane tests the same platform-independent page.
mobile_web_bundle:
needs: changes
if: >-
${{ !cancelled() && needs.changes.outputs.should_run != 'false' &&
needs.changes.outputs.qualification != 'false' &&
(github.event_name != 'pull_request' || github.event.pull_request.draft != true) }}
runs-on: ubuntu-latest
runs-on: ${{ inputs.fast_runners && 'blacksmith-8vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 15
steps:
- uses: actions/checkout@v6
@@ -324,16 +356,19 @@ jobs:
overwrite: true
linux_glibc_floor:
needs: [changes, persistence, mobile_web_bundle]
needs: [changes, mobile_web_bundle]
permissions:
contents: read
concurrency:
group: node-server-linux_glibc_floor-${{ matrix.os }}-${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' }}
# A failed smoke already blocks qualification; missing scope still selects every platform.
# Each Linux lane owns its inputs; the template still requires every lane to pass.
if: >-
${{ !cancelled() && needs.persistence.result == 'success' &&
needs.mobile_web_bundle.result == 'success' &&
${{ !cancelled() &&
needs.changes.outputs.should_run != 'false' &&
needs.changes.outputs.qualification != 'false' }}
needs.changes.outputs.qualification != 'false' &&
needs.mobile_web_bundle.result == 'success' &&
(github.event_name != 'pull_request' || github.event.pull_request.draft != true) }}
strategy:
fail-fast: false
matrix:
@@ -346,7 +381,7 @@ jobs:
image: quay.io/pypa/manylinux_2_28_x86_64@sha256:407f771c51a2c3e83ebe5a7970b4289ead3a6db21d9b9c089168775cad11d328
- os: ubuntu-24.04-arm
image: quay.io/pypa/manylinux_2_28_aarch64@sha256:c22ffd129ac99a8a42d1f2c2f4e88a9089288dd9ee987a7092da1c7dc48f27a9
runs-on: ${{ matrix.os }}
runs-on: ${{ inputs.fast_runners && (matrix.os == 'ubuntu-22.04' && 'blacksmith-8vcpu-ubuntu-2204' || 'blacksmith-8vcpu-ubuntu-2404-arm') || matrix.os }}
container: ${{ matrix.image }}
timeout-minutes: 25
env:
@@ -374,7 +409,10 @@ jobs:
persist-credentials: false
- name: Trust the checked-out workspace
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
# A direct install (6-9s) beats restoring the 417 MB store plus a linking install.
- uses: ./.github/actions/install-node-dependencies
with:
cache-pnpm-store: 'false'
- name: Build and smoke this runner's node-pty prebuild slot under the pinned Node
run: |
pnpm build:orcad-prebuilds
@@ -397,18 +435,18 @@ jobs:
overwrite: true
linux_glibc217_compat:
needs: [changes, persistence]
needs: changes
concurrency:
group: node-server-linux_glibc217_compat-${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' }}
# A failed smoke already blocks qualification; missing scope still selects every platform.
if: >-
${{ !cancelled() && needs.persistence.result == 'success' &&
${{ !cancelled() &&
needs.changes.outputs.should_run != 'false' &&
needs.changes.outputs.qualification != 'false' &&
(github.event_name != 'pull_request' || github.event.pull_request.draft != true) &&
(github.event_name != 'pull_request' || inputs.build_template || inputs.ref != '' ||
needs.changes.outputs.native_compatibility != 'false') }}
runs-on: ubuntu-22.04
runs-on: ${{ inputs.fast_runners && 'blacksmith-8vcpu-ubuntu-2204' || 'ubuntu-22.04' }}
timeout-minutes: 20
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -454,21 +492,23 @@ jobs:
overwrite: true
linux_musl:
needs: [changes, persistence, mobile_web_bundle]
needs: [changes, mobile_web_bundle]
permissions:
contents: read
concurrency:
group: node-server-linux_musl-${{ matrix.os }}-${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' && github.ref || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'push' && !inputs.build_template && inputs.ref == '' }}
# A failed smoke already blocks qualification; missing scope still selects every platform.
if: >-
${{ !cancelled() && needs.persistence.result == 'success' &&
needs.mobile_web_bundle.result == 'success' &&
${{ !cancelled() &&
needs.changes.outputs.should_run != 'false' &&
needs.changes.outputs.qualification != 'false' }}
needs.changes.outputs.qualification != 'false' &&
needs.mobile_web_bundle.result == 'success' &&
(github.event_name != 'pull_request' || github.event.pull_request.draft != true) }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-22.04, ubuntu-24.04-arm]
runs-on: ${{ matrix.os }}
runs-on: ${{ inputs.fast_runners && (matrix.os == 'ubuntu-22.04' && 'blacksmith-8vcpu-ubuntu-2204' || 'blacksmith-8vcpu-ubuntu-2404-arm') || matrix.os }}
timeout-minutes: 20
env:
ORCA_BACKGROUND_LAUNCH: '1'
@@ -481,21 +521,19 @@ jobs:
id: pnpm-verification
with:
container-toolchain: 'true'
- uses: actions/download-artifact@v8
with:
name: orcad-mobile-web-bundle
path: out/mobile-web
- name: Verify native Alpine artifact and persistence
- name: Prepare native Alpine artifact
env:
VERIFICATION_CACHE_PATH: ${{ steps.pnpm-verification.outputs.path }}
run: |
touch "$VERIFICATION_CACHE_PATH"
mkdir -p out/mobile-web
# Multi-arch index digest of the tag; re-resolve it whenever NODE_RUNTIME_PIN moves.
docker run --rm --init -i \
docker run --rm --init --detach --name orcad-alpine \
-e ORCA_BACKGROUND_LAUNCH=1 \
-v "$VERIFICATION_CACHE_PATH:/root/.cache/pnpm/lockfile-verified.jsonl" \
-v "$GITHUB_WORKSPACE:/work" -w /work \
node:24.21.0-alpine@sha256:ebfe2f90462722a7a4de65e91990e97fe0d401c70e0e762c5b53302f905ec1c1 sh -s <<'NODE_SERVER_QUALIFICATION'
node:24.22.1-alpine@sha256:6a1c190e17215d844926b0677f4bf87273ff7d5e54db06d3b5b5863ab0b09a3d sleep infinity
docker exec -i orcad-alpine sh -s <<'NODE_SERVER_PREPARATION'
set -eu
apk add --no-cache bash git libstdc++ python3 make g++
git config --global --add safe.directory /work
@@ -504,9 +542,21 @@ jobs:
pnpm build:orcad-prebuilds
pnpm build:orcad-prebuilds --require-slots "$(node config/scripts/build-orcad-prebuilds.mjs --print-slot)"
pnpm build:orcad-prebuilds --smoke
NODE_SERVER_PREPARATION
- uses: actions/download-artifact@v8
with:
name: orcad-mobile-web-bundle
path: out/mobile-web
- name: Verify native Alpine artifact and persistence
run: |
docker exec -i orcad-alpine sh -s <<'NODE_SERVER_QUALIFICATION'
set -eu
pnpm build:orcad
pnpm test:node-server --artifact
NODE_SERVER_QUALIFICATION
- name: Clean up the Alpine qualification container
if: always()
run: docker rm --force orcad-alpine
- name: Keep this runner's musl slot for the desktop template
if: inputs.build_template
uses: actions/upload-artifact@v7
@@ -519,13 +569,12 @@ jobs:
# Design D2: the desktop ships every target's addons, merged from the lanes that qualified them.
desktop_template:
needs: [persistence, mobile_web_bundle, linux_glibc_floor, linux_glibc217_compat, linux_musl]
if: >-
${{ !cancelled() && inputs.build_template &&
needs.persistence.result == 'success' && needs.mobile_web_bundle.result == 'success' &&
needs.linux_glibc_floor.result == 'success' &&
needs.linux_glibc217_compat.result == 'success' && needs.linux_musl.result == 'success' }}
runs-on: ubuntu-latest
needs: changes
if: ${{ !cancelled() && inputs.build_template }}
permissions:
contents: read
actions: read
runs-on: ${{ inputs.fast_runners && 'blacksmith-4vcpu-ubuntu-2404' || 'ubuntu-latest' }}
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
@@ -533,6 +582,45 @@ jobs:
ref: ${{ inputs.ref }}
persist-credentials: false
- uses: ./.github/actions/install-node-dependencies
- name: Wait for every template qualification lane
uses: actions/github-script@v8
with:
script: |
const deadline = Date.now() + 25 * 60_000
const expected = {
persistence: 6,
mobile_web_bundle: 1,
linux_glibc_floor: 2,
linux_glibc217_compat: 1,
linux_musl: 2
}
while (Date.now() < deadline) {
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
...context.repo, run_id: context.runId, filter: 'latest', per_page: 100
})
const assemblers = jobs.filter(job =>
job.name === 'desktop_template' || job.name.endsWith(' / desktop_template'))
if (assemblers.length > 1) throw new Error('Ambiguous desktop template qualification')
if (assemblers.length === 1) {
const prefix = assemblers[0].name.slice(0, -'desktop_template'.length)
let ready = true
for (const [name, count] of Object.entries(expected)) {
const lanes = jobs.filter(job => job.name === `${prefix}${name}` ||
job.name.startsWith(`${prefix}${name} (`))
if (lanes.length > count) throw new Error(`Unexpected qualification lanes: ${name}`)
for (const lane of lanes) {
if (lane.status === 'completed' && lane.conclusion !== 'success') {
throw new Error(`Template qualification did not succeed: ${lane.name} (${lane.conclusion})`)
}
}
if (lanes.length !== count || lanes.some(lane => lane.status !== 'completed')) ready = false
}
if (ready) return
}
await new Promise(resolve => setTimeout(resolve, 15_000))
}
throw new Error('Timed out waiting for every template qualification lane')
- name: Collect every lane's slot
uses: actions/download-artifact@v8
with:
+5 -2
View File
@@ -18,9 +18,12 @@ on:
required: false
permissions:
contents: read
# Candidate source must not write caches in the dispatch branch's scope.
cache-mode: read
jobs:
compatibility:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
- uses: actions/checkout@v6
@@ -64,7 +67,7 @@ jobs:
path: test-results/
retention-days: 3
layout-compat:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v6
+10 -4
View File
@@ -14,13 +14,18 @@ on:
- 'config/oxlint-plugins/quadratic-buffer-concat.mjs'
- 'config/scripts/*-plugin.test.mjs'
- '!config/scripts/renderer-scrollbar-style-plugin.test.mjs'
- '!config/scripts/mobile-pairing-qrcode-import-plugin.test.mjs'
# Keep in sync with the contract list in config/vitest.performance.config.ts;
# without these a rename lands green and only breaks the next nightly.
- 'src/main/ipc/pty-spawn-scaling.performance.test.ts'
- 'src/main/ipc/pty/runtime/spawn-preflight-saved-resume.test.ts'
- 'src/main/native-chat/agent-session-wire/structured-agent-session-terminal-writer-acquire.test.ts'
- 'src/main/runtime/hook-status-resume-sleep-ordering.test.ts'
- 'src/renderer/src/hooks/ipc-events/agent-status-refused-resume.performance.test.ts'
- 'src/renderer/src/components/terminal-pane/pty-connection/saved-resume-cancellation.test.ts'
- 'src/main/sqlite/sync-database.test.ts'
- 'src/main/runtime/orchestration/db/row-column-lists.test.ts'
- 'src/shared/native-chat-tool-attribution-allocation.test.ts'
- 'src/shared/native-chat-tool-pairing.performance.test.ts'
- 'src/shared/native-chat/tools/native-chat-tool-attribution-allocation.test.ts'
- 'src/shared/native-chat/tools/native-chat-tool-pairing.performance.test.ts'
- 'src/main/codex-usage/codex-rollout-file-lifetime.test.ts'
- 'src/main/codex-usage/scanner-incremental-append.test.ts'
- 'src/main/usage/jsonl-file-reader.test.ts'
@@ -46,7 +51,8 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
runs-on: ${{ matrix.os }}
# Operation-count and retention contracts, not wall-clock budgets: use the hosted images the cache warmers seed.
runs-on: ${{ fromJSON('{"ubuntu-latest":"ubuntu-24.04","macos-latest":"macos-15","windows-latest":"windows-2022"}')[matrix.os] || matrix.os }}
timeout-minutes: 20
steps:
- uses: actions/checkout@v6
+2 -1
View File
@@ -5,6 +5,7 @@ on:
paths:
- 'src/shared/commit-message-agent-specs-primary.ts'
- 'tests/tools/pi-provider-runtime-smoke.mjs'
- 'src/main/pi/rpc-history-*.ts'
- '.github/workflows/pi-provider-runtime.yml'
permissions:
contents: read
@@ -29,5 +30,5 @@ jobs:
- uses: ./.github/actions/install-node-dependencies
- name: Install pinned Pi runtime
run: npm install --prefix .cache/pi-provider --ignore-scripts --no-audit --no-fund @earendil-works/pi-coding-agent@0.84.2
- name: Verify extension model generation before and after
- name: Verify extension model generation and session entry ids
run: node tests/tools/pi-provider-runtime-smoke.mjs .cache/pi-provider/node_modules/@earendil-works/pi-coding-agent/dist/cli.js
+1 -4
View File
@@ -22,10 +22,7 @@ jobs:
runs-on: ubuntu-slim
timeout-minutes: 2
steps:
# Why no checkout: the Files API already has per-file additions/deletions.
# Why the default branch and never pull/<n>/head: this job holds a write-scoped
# GITHUB_TOKEN, so it may only execute reviewed code. A PR that edits these
# scripts takes effect once merged.
# Read file counts through API without checkout; execute only default-branch scripts because GITHUB_TOKEN has write scope. PR edits take effect after merge.
- name: Count test vs non-test LoC
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+326 -358
View File
File diff suppressed because it is too large Load Diff
+6 -1
View File
@@ -17,7 +17,8 @@ env:
jobs:
compilers:
if: github.event_name != 'pull_request' || github.event.pull_request.draft != true
runs-on: ubuntu-24.04-arm
# Native compiler peaks can exhaust the standard ARM runner before samples are saved.
runs-on: blacksmith-8vcpu-ubuntu-2404-arm
timeout-minutes: 25
steps:
- uses: actions/checkout@v6
@@ -30,6 +31,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Compare native and Bun typechecking
run: node config/scripts/benchmark-preflight-compilers.mjs
- uses: actions/upload-artifact@v7
@@ -53,6 +55,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Compare anti-slop patch and script runtimes
run: node config/scripts/benchmark-preflight-static-analysis.mjs
- uses: actions/upload-artifact@v7
@@ -77,6 +80,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Compare compatibility-import scan and runtimes
run: node config/scripts/benchmark-preflight-compat-boundary.mjs
- uses: actions/upload-artifact@v7
@@ -100,6 +104,7 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- name: Compare localization extraction runtimes and complete catalogs
run: node config/scripts/benchmark-preflight-localization-extraction.mjs
- uses: actions/upload-artifact@v7
@@ -34,6 +34,9 @@ jobs:
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version-file: config/.bun-version
no-cache: true
- uses: ./.github/actions/install-mobile-dependencies
if: matrix.comparison == 'smoke'
- name: Compare checks and verify their results
run: node config/scripts/benchmark-ci-preflight-critical-path.mjs ${{ matrix.comparison }}
- uses: actions/upload-artifact@v7
@@ -0,0 +1,106 @@
name: Qualified server template
on:
workflow_call:
inputs:
ref:
required: true
type: string
fast_runners:
description: Qualify on Blacksmith; disable inside the SignPath signing workflow.
type: boolean
default: true
permissions:
contents: read
env:
ORCA_BACKGROUND_LAUNCH: '1'
jobs:
cache:
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
key: ${{ steps.identity.outputs.key }}
hit: ${{ steps.validate.outputs.valid }}
steps:
- uses: actions/checkout@v6
with:
ref: ${{ inputs.ref }}
persist-credentials: false
- uses: actions/setup-node@v6
with:
node-version-file: package.json
check-latest: true
package-manager-cache: false
- name: Checkout the workflow policy
uses: actions/checkout@v6
with:
ref: ${{ github.workflow_sha }}
path: .build/workflow-policy
sparse-checkout: .github
persist-credentials: false
# The CLI embeds its source commit; tree equality alone cannot identify its bytes.
- name: Identify today's exact source tree
id: identity
shell: bash
env:
RUNNER_PROFILE: ${{ inputs.fast_runners && 'fast' || 'github-hosted' }}
# Production must never reuse qualification from the adhoc runner pool.
run: echo "key=orcad-qualified-template-v2-$RUNNER_PROFILE-$(git -C .build/workflow-policy rev-parse HEAD:.github)-$(git rev-parse HEAD)-$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@v5
id: restore
continue-on-error: true
with:
path: out/orcad-template
key: ${{ steps.identity.outputs.key }}
- name: Validate the restored template
id: validate
if: steps.restore.outputs.cache-hit == 'true' && steps.restore.outcome == 'success'
shell: bash
run: |
if node -e "require('./config/dark-build-switches.cjs').assertNotDarkBuild(['out/orcad-template']); require('./config/scripts/verify-packaged-orcad-template.cjs').verifyPackagedOrcadTemplate('out')"; then
echo 'valid=true' >> "$GITHUB_OUTPUT"
else
rm -rf out/orcad-template
echo '::warning::Cached template failed validation; qualifying a fresh build.'
fi
- name: Reuse the qualified template in this run
if: steps.validate.outputs.valid == 'true'
uses: actions/upload-artifact@v7
with:
name: orcad-template
path: out/orcad-template/
include-hidden-files: true
if-no-files-found: error
retention-days: 7
qualify:
permissions:
contents: read
actions: read
needs: cache
if: needs.cache.outputs.hit != 'true'
uses: ./.github/workflows/node-server-tests.yml
with:
ref: ${{ inputs.ref }}
build_template: true
fast_runners: ${{ inputs.fast_runners }}
save:
needs: [cache, qualify]
if: needs.qualify.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/download-artifact@v8
with:
name: orcad-template
path: out/orcad-template
# Only the full successful qualification can produce a reusable cache entry.
- uses: actions/cache/save@v5
continue-on-error: true
with:
path: out/orcad-template
key: ${{ needs.cache.outputs.key }}
+43 -6
View File
@@ -11,7 +11,7 @@ on:
- deleted
permissions:
contents: write
contents: read
concurrency:
group: readme-downloads-badge
@@ -19,21 +19,35 @@ concurrency:
jobs:
update:
# Why: this workflow commits to main, so forks should not create divergent badge commits.
# Badge updates belong to the upstream repository.
if: github.repository == 'stablyai/orca'
runs-on: ubuntu-latest
steps:
- name: Generate badge update token
id: app-token
uses: actions/create-github-app-token@v3
with:
app-id: 2590194
private-key: ${{ secrets.BUFO_BOT_PRIVATE_KEY }}
owner: stablyai
repositories: orca
permission-contents: write
permission-pull-requests: write
- name: Checkout main
uses: actions/checkout@v6
with:
ref: main
token: ${{ steps.app-token.outputs.token }}
- name: Render downloads badge
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: node .github/scripts/render-readme-downloads-badge.mjs
- name: Commit badge update
- name: Propose badge update
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
set -euo pipefail
if git diff --quiet -- docs/assets/readme-downloads.svg; then
@@ -41,8 +55,31 @@ jobs:
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
branch="automation/readme-downloads-badge"
expected="$(git ls-remote --heads origin "refs/heads/$branch" | cut -f1)"
git checkout -b "$branch"
git config user.name "buf0-bot[bot]"
git config user.email "252831055+buf0-bot[bot]@users.noreply.github.com"
git add docs/assets/readme-downloads.svg
git commit -m "Update README downloads badge"
git push origin main
git push --force-with-lease="refs/heads/$branch:$expected" origin "HEAD:refs/heads/$branch"
existing="$(gh pr list --repo "$GITHUB_REPOSITORY" --state open --base main --head "$branch" --json url --jq '.[0].url // empty')"
if [ -n "$existing" ]; then
echo "Updated $existing"
exit 0
fi
body="$RUNNER_TEMP/readme-downloads-pr.md"
cat > "$body" <<'BODY'
## ELI5
Refresh the README downloads badge with the current release download count.
## Visual Proof
N/A — generated badge count only; no application UI change.
## Testing
Generated from published release assets by the README Downloads Badge workflow.
BODY
gh pr create --repo "$GITHUB_REPOSITORY" --base main --head "$branch" \
--title "Update README downloads badge" --body-file "$body"
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More