The egress probe now needs no Vercel token. It uses dev workspaces with a dev-JWK user, runs its
commands in a workspace terminal through the shipped orca CLI's pairing code, and uses
postman-echo.com as the echo host for bogus keys. It covers the custom-base-URL swap,
chatgpt.com through the relay, api.github.com, #758 anonymous git and the App repo clone.
Premerge runs boot-twice-check.sh for image/** changes. The step has its own omarchy slot and
a 10 min limit, and fails open with a warning without Docker.
Boot-twice (image/boot-twice/): provisions the golden image in a container with the real
provision-golden.sh, then boots the same disk twice through the control plane's own boot steps
(entry-point probe, startOrcad with the workspace prelude, ensureEntryPoints). Each start is a new
kernel boot id. The second boot runs offline behind a DNS sink, and must install, fetch and redo
nothing (docker diff allowlist, logged lookups, sampled command lines). It must come up, carry
this tree's entry points and stamp, and match every manifest version pin. ~15 s cached, ~5 min
cold. It catches the pre-#781 OpenCode first run that repeated on every wake.
Workflow: cloud-vm-boot-twice.yml (PRs touching image/**, GHA layer cache).
Egress probe (scripts/egress-probe.ts): against dev, with bogus tokens and an echo sandbox.
Firewall layer: egressPolicy()'s real policy forwards placeholders for chatgpt.com,
api.anthropic.com, api.github.com and github.com with other auth headers untouched. Credential
hosts fail closed, and anonymous git works without an App token (#758). Swap layer: the dev egress
proxy swaps a bogus ANTHROPIC_API_KEY in for its placeholder on a custom base URL, and leaks no
Vercel hop headers. Workflow: cloud-egress-probe.yml (dispatch + nightly; skips without secrets).
Fix found by the harness: provision-golden.sh linked a release bundle's runtimes dir, and every
VM script's `find .../runtimes -type f` skips a symlinked start point, so --release goldens had
no node. Move the tree instead. No entry-point change (stamp unchanged).
- sandbox-control: an agent the person signs in inside each workspace gets mode 'inside' (keys
dropped as before) and the session env carries ORCA_SIGN_IN_INSIDE=<agents> (ORCA_*: never in
shells). The VM env script then leaves Claude's first-run onboarding unmarked while there is no
login in the workspace, so Claude opens on its own sign-in screens instead of 'Not logged in';
once ~/.claude/.credentials.json exists it is marked as usual. No golden rotation: the env
script reaches older snapshots through the boot-time entry-point refresh.
- Structured chats: a 'not signed in' verdict or start failure for such an agent reads
'{{agent}} needs you to sign in inside this workspace' with 'Open terminal to sign in', which
opens a terminal tab on the workspace running the agent's login (claude auth login, codex login
--device-auth), instead of the Orca-account Renew advice.
- Copy: the choice's detail adds 'Team API keys won't be used for this agent.' (strip, Settings
row, and the Settings button's tooltip).
MEMGUARD. A build or test that eats a cloud VM's memory now costs that process, not the workspace.
- orca-memguard (new VM entry point, root via sudo from orca-sandbox-start): ranks processes for the
kernel OOM killer (Orca runtime -900, terminal shells and agent CLIs -500, everything else 500),
stops the largest unprotected process when MemAvailable drops under 8% and records it, adds 2 GiB
swap files on demand (up to 1.5x RAM) and drops them after 10 quiet minutes, and remounts /dev/shm
at 50% of RAM (the provider mounts 64 MB). Reaches old snapshots through the entry-point refresh.
- orca-sandbox-drain releases on-demand swap before a stop so it doesn't ride the snapshot.
- Status read: the VM metrics probe reads the guard's state; status gains an optional memory
{pressure, guardKills, lastGuardVictim, ...} (metrics keeps its strict shape for older desktops).
- Desktop: guard kills count in the OOM tracker and show 'Orca stopped <process> to keep the
workspace responsive' with a Machine size action; the sidebar row gets a quiet memory mark under
pressure (amber when critical).
A per-person default Source 'in-workspace' for Claude and Codex. Orca stores and injects nothing
for that agent: no account placeholder, no API key, and so no egress route on the vendor's
hosts. The agent starts as it is and shows its own sign-in in the pane; its token stays in the
workspace's home directory and survives sleep.
- sandbox-control: migration 0062 widens agent_sources.source (table rebuilt, rows kept);
PUT /v1/agent-sources accepts 'in-workspace' as a default only (400 with workspaceId);
agentModesFor maps it to 'omit'; GET adds inWorkspace per agent while defaults/effective
keep their two values for older desktops.
- Desktop: coverage kind 'inWorkspace' (no hold, no per-host sign-in gate on Orca Cloud);
'Set up once' strip gets a quiet ⋯ choice 'Sign in inside this workspace instead' that
saves the choice and starts every launch held on it; Settings › Cloud agent sign-ins offers
it on a not-set-up row and shows 'Signs in inside each workspace' with a way back.
- Tests: Worker (env, open/limited policy, live switch, migration) and renderer (model, hold,
spawn gate, strip, settings, build row); fixtures for the strip menu and the settings row.
A create or new environment with nothing pinned used to start in pdx1 whatever the caller's location,
and every frame of a remote session paid that round trip. Now the sandbox-control Worker ranks the
served regions (the default and its fallbacks with an active golden) by distance from Cloudflare's
request geolocation and starts in the nearest.
- Explicit regions (request, API project) and an environment's build region stay authoritative.
- Environments record their region (migration 0061): builds, golden refreshes, staleness and the
security floor use it, so snapshots stay bootable and workspaces from them stay together. Legacy
environments (NULL) live in the default region as before.
- Automatic environments live where the create that made them ran.
- The capacity fallback tries the next nearest served region.
- The warm pool keeps VMs in every served region.
- VERCEL_NEAREST_REGION=off restores default-region-only.
- Desktop: the workspace card's Details shows the region (e.g. Portland (pdx1)).
Ask for Orca GitHub App access while the sandbox starts, not in a toast afterwards.
Server (sandbox-control):
- GET /v1/github/repo-access answers `push`: ok | not_installed | repo_not_selected |
not_connected, with `configureUrl` (the installation's settings page) for repo_not_selected.
A public repo that can't push now gets an install URL too.
- GitHub installation lookups (repo and new owner lookup) cached in D1 (migration 0061):
found 60 s, 404 15 s, dropped by installation / installation_repositories webhooks and the
setup return. The owner lookup never holds the answer (3 s cap, errors read as not_installed).
Desktop:
- "GitHub access" row after Start sandbox, checked when the create starts (parallel, so no
latency when access is fine). Missing: Install GitHub App / Add repository on GitHub /
Copy request for an owner, plus Skip for now ("You can pull; pushes need access later").
Polls (2 s for 3 min, then 10 s; on focus) and auto-advances on grant.
- The create holds before Open workspace only while the step asks and the sheet is on screen.
- Skip remembered per repo on this device; later creates show it skipped (Set up) without
holding; forgotten once the repo has access.
- A grant that lands after the sandbox started calls applyCloudGitHubAccessToSandbox, a stub
until gh-access's live refresh lands.
chatgpt.com answers Cloudflare Workers' egress with a 403 "Unable to load site" page, so Codex via
the Orca account could never get an answer through the egress proxy (#767 made that a printed
error). Cloud Run's egress is answered normally (probed, then deleted).
- cloud/apps/chatgpt-relay: a dependency-free Node relay. chatgpt.com only, the same Codex path
pinning as the Worker (shared sandbox-control/src/codex-paths.ts), SSE streamed chunk by chunk,
no WebSocket upgrades, hop and Cloud Run headers dropped, logs without headers, bodies or queries.
- Cloud Run IAM authenticates the Worker: one invoker service account (run.invoker on the relay
only); the Worker mints a Google ID token for the relay URL and sends it in
X-Serverless-Authorization, so the swapped ChatGPT Authorization passes through untouched.
- sandbox-control: CHATGPT_RELAY_URL + CHATGPT_RELAY_SERVICE_ACCOUNT route the egress proxy's
chatgpt.com fetches through the relay; unset, chatgpt.com goes direct as before. A relay
sign-in failure answers Codex a 503 JSON detail instead of reaching chatgpt.com.
- cloud/infra/chatgpt-relay: Terraform (staging backend applied; production is a runbook step).
OpenCode's first run in a VM installs its plugin SDK into ~/.config/opencode, creates its database and fetches its model list (first launch 5.2 s vs 2.7 s warm; in a pane on a fresh VM ~11 s, with the opencode process starting 0.7 s after the click, so the time is OpenCode's own). orca-sandbox-start now runs `opencode models` (does all of it, no credentials, exits) in the background at nice 19 right after orca-sandbox-env writes the config, only while ~/.config/opencode/node_modules is missing (once per disk; wakes skip it).
Neil: an Orca Cloud part inside Settings › Agent accounts was confusing. The Claude and ChatGPT
sign-ins for cloud workspaces (set up, paste a token, renew, remove) are now their own page in the
Orca Cloud group, right after Account & billing. Agent accounts keeps This computer and Hosts and
one row, "Orca Cloud sign-ins live in Orca Cloud settings", that opens the new page.
Every deep link moves with it: Renew in Settings, the sign-in sheet's footer, the host panel, and
old links to pane 'accounts' at 'orca-cloud' / 'accounts-orca-cloud' (resolveCloudAgentSignInsTarget).
Copy that sent people to "Settings → Agent accounts" for Orca Cloud now says "Settings → Cloud agent
sign-ins" (new i18n keys so stale translations don't point at the old place), including
sandbox-control's expired-Claude and RS-4 Codex messages. Search: the Orca Cloud keywords
(subscription, chatgpt, setup-token, renew) move to the new page. Behavior is unchanged.
chatgpt.com answers every request from Cloudflare Workers' egress IPs with a 403 "Unable to load
site" HTML page, whatever the token (verified from a throwaway Worker; the same request from
elsewhere gets a normal 401). Codex via the Orca account goes through the Worker egress proxy, so
it got that page on /responses and /wham/usage: "Working…" without an answer, and /status had no
limits.
The proxy now recognises a provider edge's own refusal (403, text/html, server: cloudflare) on
routes that define an answer for it, logs egress_upstream_blocked, and answers Codex in ChatGPT's
error shape so Codex prints a plain message. The block says nothing about the credential: no
status hook, so no refresh and no revoke. This doesn't make Codex work; routing chatgpt.com around
the Worker is a separate decision.
team-lead follow-up to #762: lookups/updates now fail at 20 s; a GET that hits the deadline is asked once more (a stalled write is never repeated: it may have landed); each stall logs provider_stall with the call's shape. orca-sandbox-control-dev turns on Workers observability so provider_stall, grant_failed and lifecycle_timing are retained.
A fresh VM pages the agent CLIs in lazily from its snapshot, so the first agent launch paid for it (OpenCode 11 s cold vs 3.75 s warm in the same VM, n71). orca-sandbox-start now reads the claude, codex and opencode packages (files over 256 KB) in the background at nice 19 / idle IO, detached, after orcad starts; never on the ready path. Best effort.
Two dev failures (n70/n71) were the Worker waiting on a stalled Vercel API connection with no
client-side deadline: a create spent ~2 min before its write-ahead insert ("Start sandbox" stuck),
and POST /v1/team/workspaces/:id/grants (opening a new workspace on this computer) hung its full
120 s (wallTime 119995 ms, no response) although its VM command is bounded at 45 s.
- Every provider request now has a deadline: 30 s for lookups and updates, 60 s to boot a VM,
5 min to snapshot, a command's own timeout + 20 s, and 3 h only for an explicit unbounded wait
on a build's exit. A stall becomes a provider error the callers already handle (grant_failed,
the create fallback), instead of an open-ended wait.
- issueGrant logs its phases (session, grant command, member git) when it takes over 10 s, and
logs grant_failed with its timings and error, so the next slow open names its phase.
A workspace on a public repo the Orca GitHub App isn't installed on gets no App token, so its git
sends no credential. When a member had connected GitHub, their placeholder rules made github.com a
credential host, and compileNetworkPolicy ended it with the firewall's 403: the doctor's
`git ls-remote origin HEAD` failed ("This workspace can't reach <repo> on GitHub right now"), and so
did every plain fetch and pull.
- With no App token, the member github.com route is allowUnclaimed. With a token the workspace git
route keeps github.com fail-closed as before. api.github.com is unchanged.
- A wake rewrites the policy when member identities apply: a resumed session keeps its last
policy, so a workspace created before this fix would otherwise keep the 403.
- cloud/infra/secrets-kms: own Terraform root and state (terraform/secrets-kms) for the Cloud KMS key
that wraps the secret-store KEK, the Worker's decrypt-only service account, KMS/Secret Manager
data-access audit logs and the KEK escrow secret. Applied to onorca-cloud-staging (dev Worker).
- SECRETS_KEK_PLAINTEXT_RETIRED_ID: in kms mode, also open rows under that id with the plaintext
SECRETS_KEK as a retired key, only after KMS handed over the active KEK. Neither dev nor prod has a
copy of its KEK outside Cloudflare, so the KEK can't be wrapped; this lets the rewrap sweep move
rows onto a new KMS-held KEK instead. Never seals, never stands in for KMS (tests).
- wrangler.dev.jsonc: SECRETS_KEK_SOURCE=kms, SECRETS_KMS_KEY, SECRETS_KEK_ID=k2.
Every Orca socket into orcad already gets a WS ping every 10-15 s (orcad's server heartbeat and the
desktop's client liveness). Share links were the gap: the in-VM share proxy piped upgraded sockets raw, so
a preview app's silent WebSocket was closed by Vercel after ~340 s (1006), well before the link's
deadline. The proxy now follows the app's frame boundaries and, after 30 s with no traffic either way,
sends the visitor an empty ping frame between two frames (never inside one). Only 101 upgrades to
websocket get pings; other upgrade protocols stay a raw pipe.
VM script: reaches live VMs through the boot-time entry-point sync (new stamp), and the next golden.
Vercel's command POST has no idempotency key, so a retry after a lost answer (dropped connection, gateway
5xx, a Worker that died mid-request) can start the command again (vercel/sandbox#363). Message delivery
did exactly that: a transient failure re-queued the message and the cron typed it into the agent again,
or launched a second agent.
- command-once.ts: an in-VM guard with a per-id ledger (mkdir claim, recorded exit + output). A repeat of
the same id waits for and replays the first run; a dead first run reports an unknown outcome; a failed
run is forgotten so a retry runs for real. runCommandOnce retries uncertain provider failures with the
same id (never a 4xx that proves the command did not start).
- deliverMessage uses msg-<id> for send/launch; an unknown outcome fails the message
(delivery_outcome_unknown) instead of re-sending.
- POST /messages and POST /sessions take Idempotency-Key (same semantics as workspace create); a session
create taking over a died attempt replays the first launch.
- The REST client keeps sending each command POST exactly once (test).
- snapshot-refs.ts: one check for what live things still boot from (goldens of any live status, pool
images, active environment builds, live workspaces' own snapshots, and a live workspace's boot image
until it has a snapshot of its own).
- Retired goldens beyond the rollback and deleted environments' builds are marked deleted only after the
provider delete succeeded (a failed delete used to be marked and leaked for good), and are deferred
while in use. A new hourly housekeeping step (sweepReleasedSnapshots) retries both.
- Team delete marks the team's environments deleted, so their build snapshots are freed (they leaked).
- Environment prune skips a build a live workspace still boots from.
- Orphan sweeps (per box and project-wide automatic) also respect pool and boot-image references.
- The golden smoke box is non-persistent: it no longer leaves a ~325 MB automatic snapshot behind.
"Plan & billing", "Upgrade to Orca Cloud", "Choose plan" and every "Compare plans" now open an
in-app plans dialog first: the team's plan and status (trial hours left, active with hours and
seats, payment failed, or no plan with the trial billing offers), then billing's plan cards (Team
with a monthly/annual toggle, Enterprise with contact sales). "Continue in browser" opens the
console's billing page through the existing sign-in ticket. Members see the same plan info with
"Ask a team admin to upgrade" (Plan & billing is back in their account menu, read-only). Read-only:
no checkout, usage history or admin settings return to the desktop.
Everything shown is billing's: names, Stripe prices, seat pricing, feature bullets and trial terms
from GET /v1/plans, now also each plan's tagline, feature-list lead and button label (new optional
catalog `marketing`, unversioned copy, plus `omitFeatures` so Enterprise stops advertising egress
controls). Main's lenient parse keeps a newer or older service safe; without the catalog the
entitlement's offers stand in, and with neither the dialog says plans didn't load and still offers
the console. While billing loads the title is neutral, so a paying team never sees an upsell flash.
Settings > Account's billing row opens the dialog ("View plans"); "Update payment method" still
goes straight to the console.
One Checkout sends checkout.session.completed, customer.subscription.created and invoice.paid,
often concurrently; each that re-read the row before another wrote it saw the plan start and
pushed billing.plan_started with its own 'at', so the event ids differed and the team's audit
log showed it twice. A plan starting or ending now carries a once key (subscription id and
period start), so every racer pushes the same event id and the control plane's INSERT OR IGNORE
keeps one.
POST /v1/billing/checkout and /portal accept returnTo: 'console'. The return URL is built from
this Worker's CONSOLE_URL (else the setup pages' auth origin, which forwards /console) and the
caller's own team, never from the request. Desktop-started flows keep the /return handoff page.