Jinwoo Hong
4f839cc8c9
* perf(relay): cut the cell LB drain to 60s and widen the same-cap batch to ten cells Two independent sources of relay roll wall clock, neither of which protects a host: 1. `connection_draining_timeout_sec` on the per-cell backend services was 300s. The same-cap job drains every host off the cell to a restart-safe condition before Terraform runs, so the LB drain only ever covers a host still mid-handshake. Measured 2026-09-16 over ten same-cap cell jobs, it sat as ~5m55s of dead time between `Apply complete` and the old VM powering off, inside an 8.5-minute `wait-until --stable` step. Now 60s, and pinned in the topology `check` block beside the other fixed-one invariants. 2. The same-cap wave capped a batch at four cells, so a 22-cell roll needed six batches, six single-use monitor gates, and a human handoff per batch. The wave workflow now declares cell_1..cell_10 with the identical serial shape and chaining, and the validator accepts two to ten. The shared wave-index rule (`relay-monitor-evidence.mjs` and the relay-ops preflight CLI) widens from 0-3 to 0-9 so the later cells can present the same evidence; each job workflow keeps its own narrower range, so the capacity wave stays at four. Cells remain strictly serial, one at a time behind the rollout lease, each with its own live preflight. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap * fix(relay): align the Asia topology plan validator with the 60s cell drain `validate-relay-asia-topology-plan.mjs` rejected any Asia backend whose `connection_draining_timeout_sec` was not 300, and `cloud-deploy-relay-asia-topology.yml` targets `google_compute_backend_service.relay_gce_cell["<cell>"]` per cell. With the Terraform local at 60 that workflow would have failed its own plan review. The validator's two restated topology values are now named exports, and a new census test reads `relay-gce-cells.tf` and equates three statements of each: the `relay_gce_topology` local, the topology `check` assert that pins it, and the validator constant. Terraform cannot export a local to JS, so reading the source is the only way to stop them drifting; the test was confirmed to fail when the local alone is moved back to 300. Repo-wide grep finds no other pin of the drain value. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap
Orca Relay Operations
A private, aggregate dashboard for the Orca Relay control and data planes. It reads local gcloud and gh credentials on the server; credentials and per-user Relay state never enter the browser. One cached gcloud auth print-access-token refresh feeds concurrent read-only Google APIs so the collector does not stampede the local credential store.
Run locally
Prerequisites:
- Node 24 and pnpm 10
gcloudauthenticated foronorca-cloudandonorca-cloud-stagingghauthenticated with read access tostablyai/orca-cloud
From the repository root:
pnpm install
pnpm ops:relay
Open http://127.0.0.1:2455. The server binds only to loopback and refreshes aggregate data every minute. Production and staging are read-only by default.
The cost panel is a labeled planning estimate. There is currently no Cloud Billing export in either project, so the dashboard cannot claim exact billed spend. GCP Billing remains authoritative.
Share through Tailscale
Keep the dashboard bound to loopback and let Tailscale provide identity, TLS, and tailnet ACL enforcement:
tailscale serve --bg http://127.0.0.1:2455
tailscale serve status
Share the HTTPS URL printed by tailscale serve status with the team. Limit access to the intended operator group in the tailnet ACL. Do not use a public funnel. Stop sharing with:
tailscale serve reset
For a persistent host, run pnpm --filter @orca-cloud/relay-ops build and supervise pnpm --filter @orca-cloud/relay-ops start with the host's normal process manager. The process needs the same non-interactive gcloud and gh identities.
Optional staging controls
Controls are intentionally local-only and disabled unless explicitly enabled:
RELAY_OPS_ENABLE_STAGING_CONTROLS=1 pnpm ops:relay
Even in this mode the service never changes GCP directly. It dispatches .github/workflows/power-relay-staging.yml, preserves the workflow's typed WAKE_STAGING / SLEEP_STAGING confirmation, and always wakes only configured-admission cells. Requests require the loopback origin and a per-process CSRF token, so controls stay unavailable through the Tailscale view.
Data and security boundaries
- Browser payloads contain aggregate Monitoring points, resource health, immutable image digests, alert-policy metadata, and workflow metadata.
- Account IDs, host IDs, device IDs, pairing state, assignment rows, bearer tokens, service-account tokens, startup scripts, secret values, and individual Relay-admin state are excluded.
- Sleeping staging is inventory-only. Viewing it does not probe or cold-start Cloud Run services and cannot resize empty MIGs.
- Partial GCP or GitHub failures degrade the affected panel and produce a sanitized warning.
- Missing cell inventory renders as
Unknown, neverSleeping. After one successful read, transient credential or collector failures retain the last good snapshot and mark it stale. - Responses use
no-store, a restrictive CSP, frame denial, and no-referrer headers.
Verification
pnpm --filter @orca-cloud/relay-ops test
pnpm --filter @orca-cloud/relay-ops typecheck
pnpm --filter @orca-cloud/relay-ops build