Jinwoo Hong
09622f0c28
* feat(relay): add a break-glass override for the same-cap monitor gate Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor dry-run. When a chronic fault is what the gate freezes on, waiting for a green window means waiting for the condition the wave removes: the gate froze 44 consecutive times on the recurring Cloud SQL stall the rolling image fixes. Add `gate-override-reason` and `gate-override-confirmation` (`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A valid pair skips only the aggregate evidence download, provenance verification, and single-use marker. A partial or mismatched override fails closed before any mutation, in both the caller and the reusable job. Record the actor, reason, and confirmation in the gate run summary and, for a canary, in the sealed artifact. The live per-wave preflight still runs. Give it a `--no-monitor-state` source that takes the expected selector from the dispatch inputs and pins the migration policy to `strict`, rather than synthesising a state file that would claim a dry-run it never ran. Also give `director.instances` the two-consecutive-sample tolerance the cell probes have: Cloud Run replaces an instance in place, so the count leaves the [5, 6] band for one sample roughly twice a day, and a deploy overlap raises it the same way. Min and max share one streak so an alternating count still freezes. * fix(relay): canonicalise the break-glass preflight membership The override path parsed the operator's membership with a bare schema parse, while the live selector read from the director is normalised and the comparison is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as selector drift on a healthy fleet, and the every-configured-cell-exactly-once check was lost with it. Normalise through the same `normalizeSelectorMembership` call the monitor CLI uses when it seals evidence, against the same durable Terraform cell set. Tests use a collect stub that returns the director's canonical selector rather than echoing the expected one, so the ordering is actually exercised: unsorted input must canonicalise, and a duplicated, missing, or unknown cell must be rejected.
Orca Relay Operations
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local gcloud and gh credentials on the server; credentials and per-user Relay state never enter the browser. One cached gcloud auth print-access-token refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
Run locally
Prerequisites:
- Node 24 and pnpm 10
gcloudauthenticated foronorca-cloudandonorca-cloud-stagingghauthenticated with read access tostablyai/orca-cloud
From the repository root:
pnpm install
pnpm ops:relay
Open http://127.0.0.1:2455. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
Share through Tailscale
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
tailscale serve --bg http://127.0.0.1:2455
tailscale serve status
Share the HTTPS URL printed by tailscale serve status with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
tailscale serve reset
For a persistent host, run pnpm --filter @orca-cloud/relay-ops build and supervise pnpm --filter @orca-cloud/relay-ops start with the host's normal process manager. The process needs the same non-interactive gcloud and gh identities.
Optional staging controls
Controls are intentionally local-only and disabled unless explicitly enabled:
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
Even in this mode the service never changes GCP directly. It dispatches .github/workflows/power-relay-staging.yml, preserves the workflow's typed WAKE_STAGING / SLEEP_STAGING confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
Data and security boundaries
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
- Missing cell inventory renders as
Unknown, neverSleeping. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale. - Responses use
no-store, a restrictive CSP, frame denial, and no-referrer headers.
Verification
pnpm --filter @orca-cloud/relay-ops test
pnpm --filter @orca-cloud/relay-ops typecheck
pnpm --filter @orca-cloud/relay-ops build