Jinwoo Hong
7184b1dc5b
* fix(relay-ops): let the pre-roll gate ride out chronic production noise The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions that have nothing to do with the roll it gates: - A cell probe is one HTTP round trip from one runner. When the Asia cells' readiness SQL probe times out behind a saturated pool, the load balancer answers "no healthy upstream" for ~30 s and the gate froze on a single sample. Cell probe signals now need more than cellProbeToleranceSamples consecutive failing samples to freeze; absorbed blips are recorded in the state artifact. Director and auth probes keep zero tolerance. - directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old bar sat on the p90 and froze 29% of gates. - cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h: p95 212 / p99 262 / max 282. The old bar sat under the observed peak and froze 22% of gates. Failure codes are unchanged so downstream matchers keep working, and the trusted evidence scripts are untouched. * fix(relay-ops): key probe tolerance by cell and extend it to live preflight Three review findings on the cell-probe tolerance: - The streak was keyed per signal, so a cell alternating between slow (latency over bar) and down (health/ready 0) held every individual streak at one and never reached the tolerance. A continuously unhealthy cell passed the gate. The streak is now keyed by cell id, so one cell's health, ready and latency readings share it. - The live preflight runs one sample before every mutating wave and retried only on freshness codes, so the same Asia blip could still fail a wave there. It now re-samples per-cell probe breaches on the same tolerance, spaced the existing interval. Director and auth probes still fail the wave on the first bad sample, as does any non-probe threshold. - docs/relay-incident-monitor.md still stated the old bars. Updated the threshold table, the 400-connection ceiling text, and the superseded 2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17 recalibration entry. Also pins the resumed-state case: a state file carrying a full streak now has a test proving it freezes on the next bad sample. Trusted evidence scripts remain untouched.
Orca Relay Operations
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local gcloud and gh credentials on the server; credentials and per-user Relay state never enter the browser. One cached gcloud auth print-access-token refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
Run locally
Prerequisites:
- Node 24 and pnpm 10
gcloudauthenticated foronorca-cloudandonorca-cloud-stagingghauthenticated with read access tostablyai/orca-cloud
From the repository root:
pnpm install
pnpm ops:relay
Open http://127.0.0.1:2455. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
Share through Tailscale
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
tailscale serve --bg http://127.0.0.1:2455
tailscale serve status
Share the HTTPS URL printed by tailscale serve status with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
tailscale serve reset
For a persistent host, run pnpm --filter @orca-cloud/relay-ops build and supervise pnpm --filter @orca-cloud/relay-ops start with the host's normal process manager. The process needs the same non-interactive gcloud and gh identities.
Optional staging controls
Controls are intentionally local-only and disabled unless explicitly enabled:
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
Even in this mode the service never changes GCP directly. It dispatches .github/workflows/power-relay-staging.yml, preserves the workflow's typed WAKE_STAGING / SLEEP_STAGING confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
Data and security boundaries
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
- Missing cell inventory renders as
Unknown, neverSleeping. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale. - Responses use
no-store, a restrictive CSP, frame denial, and no-referrer headers.
Verification
pnpm --filter @orca-cloud/relay-ops test
pnpm --filter @orca-cloud/relay-ops typecheck
pnpm --filter @orca-cloud/relay-ops build