egress-docs' probe 2 re-run through #803 found transformRuleCount always 0: Vercel's redacted shapes
name header swaps injectionRules (session) next to forwardRules and responseRules. The audit now
reports each kind separately (count + header names), so a scrubbed policy is checkable: 0 injection
rules, response rules only.
Two more dev-only operator routes so the probe needs no personal Vercel login:
- POST …/resume-raw: resume the provider session from its stored record with no policy write of
ours (the row is left as it was; dev throwaways only).
- POST …/echo-probe?marker=orcaprobebogus…: one fixed request from inside the VM to httpbin.org
or postman-echo.com carrying a placeholder header (placeholders only, validated); the answer
is whether the echo contained the marker. No body comes back or is logged.
For egress-docs' probe 2, re-run on the Worker's own Vercel token (dev only; elsewhere these routes
don't exist; operator token):
- GET /v1/internal/sandboxes/:id/provider-policy-audit?marker=orcaprobebogus…: the raw record and
session read from Vercel, answered as redacted facts only (state, session status, whether a
network policy is present, transform rule count, header names, markerFound by substring test).
No value or body is returned or logged.
- POST /v1/internal/sandboxes/:id/provider-policy-scrub-test: A4's primitive, scrubRecordPolicy,
which PATCHes the persistent record with the token-free compile (every firewall value answered by
the firewall, no App token; proxy routes hold no values anyway).
compileSandboxPolicy gains { tokenFree }; the provider gains updateRecordNetworkPolicy and a
dev-audit raw read.
Follow-up to #801: the rest of the Workers' shared in-flight promises.
- Auth (sandbox-control desktop, staging and dev keys; billing) and the egress proxy's OIDC keys used
jose's createRemoteJWKSet, which shares one in-flight JWKS fetch across every caller: a request
cancelled mid-fetch hung every token check behind it, and jose refetches after ten minutes. That
fits the 'Start sandbox' creates stuck before any log 10–15 min after a deploy and cleared by a
redeploy. settledRemoteJWKSet keeps the key set as a value; a request that needs a fresh one
fetches it for itself (refetch every 10 min, an unknown kid at most every 30 s).
- applySandboxPolicy no longer chains writes per sandbox in the isolate (a chained promise from a
cancelled request would hang every later write for that sandbox); the D1 generation orders them.
- The KMS keyring cache keeps the loaded keyring and the last failure as values; no in-flight load
is stored at all.
- Guard (both Workers): a unit test fails on createRemoteJWKSet, or a module-level Map, variable
or object holding a Promise, unless the line says why it is safe ('xreq-ok:', e.g. a WebCrypto
import).
Egress proxy secret routes (custom-base-URL keys, OPENAI_BASE_URL) hung until the client gave up
(dev, proxy mode, 2026-11-15). The Worker was invoked (tail: `canceled`, wall 65 s, CPU 3 ms, no log
line) and was waiting, never reaching its upstream fetch. GitHub routes on the same sandbox answered.
An isolate serves many requests, and a promise one request starts is bound to that request's I/O. If
that request is cancelled while the promise waits on D1, KMS or fetch, the promise never settles, and
every later request awaiting the same cached promise hangs. Several module-level caches shared
in-flight promises: the KMS keyring (forever, per isolate), decrypted provider keys and account
credentials (per minute), the relay's ID token mint, the App's pending-request owners, create timing
and share-link lookups.
They now cache settled values only (settled-cache.ts and the same pattern inline). A request that
misses loads for itself; a cold isolate may run a few duplicate loads.
Workstream C of the egress migration. `scripts/egress-probe.ts --mode proxy|firewall|both` flips the
same two dev workspaces with the operator egress-mode route and runs every check in each mode. Each
request is a Node fetch (undici, like the agents) with a 30 s limit, so the transform body hang shows
up as a named failure. New coverage:
- a chunked stream, a 1 MB upload and a WebSocket upgrade through the swap;
- a placeholder sent to an unrelated host must arrive unswapped;
- the WebSocket host fails closed;
- the LFS batch API, git fetch, push and delete;
- the App-repo clone through the desktop's create.
Latency is reported per class (median/p90). The terminal wait is now 15 min, and a project whose
clone failed is registered as a folder, so the checks still run.
On dev at 2cc12610 (A2+A3), firewall mode passes 22/22. Proxy-mode findings went to gh-access.
Security review of A1/A2 (egress-docs):
- F1: the Copilot PAT never reaches the firewall. Eligibility is decided per secret
(firewallEligibleSecret): no credential the proxy narrows on any host (response filtering,
paths, per-host paths). Before, its Copilot API host entries had no paths and compiled to
transforms, putting a user's GitHub PAT in the policy store.
- F2: claims are ordered but landed writes weren't. After a write, if a newer generation was
claimed meanwhile, the write is redone from current state, so a stale compile (say from before a
share) is never the last to land.
- F3: a config-hash change fails closed only when it removes access or custody (firewall to proxy,
a firewall class removed, an account provider turned off, open to limited). Anything else retries
via policy_stale_at, so a deploy during a provider blip can't sleep every workspace. The stamp now
carries the deciding inputs; a pre-F3 bare-hash stamp is retried.
Egress firewall migration, workstream A3 (design §1). In EGRESS_SWAP_MODE=firewall the Codex account
token is swapped at the firewall on every Codex path, exact paths only (the hot calls plus the
catalog GETs); anything else with the placeholder gets the 403. Nothing forwards to the egress proxy,
so the ChatGPT relay is unused for these sandboxes.
Codex's own refresh of its placeholder login (POST auth.openai.com/oauth/token) is answered by the
firewall itself with the placeholder login again; the real token is refreshed centrally on schedule.
The #776 alias (CODEX_EGRESS_SWAP=firewall in proxy mode) is unchanged: hot paths at the firewall,
the rest through the proxy.
A real ChatGPT token is stored on dev; the firewall swap stays off (proxy + relay) until the
Host/SNI binding probe shows a transform can't be reached through another destination.
Egress firewall migration, workstream A2 (design §1, §3, §4, §7 of
.context/cloud/reports/egress-firewall-migration.md). Behind EGRESS_SWAP_MODE=firewall (per sandbox:
sandboxes.egress_swap_mode); proxy mode is unchanged.
- Firewall mode compiles the real values into the sandbox policy: model keys (every header form the
client sends, mapped to the same form of the key), the Claude sign-in, and the GitHub App token
(git Basic on github.com, Bearer/token on api.github.com). The 403 rule still ends every
credential host.
- Fail closed: a key no longer usable, or an account that no longer applies (signed out, revoked,
owner changed, shared, provider off), is answered by the firewall itself (provider-shaped
signed-out body, or the keys-withheld body), never forwarded with its placeholder.
- Stays on the egress proxy: Cursor's key exchange (W1: the reply is filtered), the Copilot PAT on
api.github.com (W2: a prefix scope can't be pinned at the firewall), member identities (W3).
- EGRESS_FIREWALL_CLASSES narrows the classes (model_key, claude, codex, github_app); a removed
class goes back to the proxy.
- policy_config_hash: each apply stamps a hash of the operator inputs; reconcile re-applies a
running sandbox whose hash differs (a kill switch reaches every live policy within one tick) or
whose last write failed (policy_stale_at).
- In firewall mode an env refresh always applies the policy (a key edited in place, a share).
- POST /v1/internal/sandboxes/:id/egress-mode {mode: proxy|firewall|null}: one sandbox's mode,
applied at once (leaving the firewall fails closed).
The desktop's create (clone-progress) runs the workspace clone detached. orca-sandbox-start then writes
the same extraheader into ~/.gitconfig, and http.<url>.extraheader is a list. So the clone's later
requests (the blobless clone's checkout fetch, and the retry clone) carried Authorization twice. The
firewall answered 400 and nothing reached the egress proxy. The clone failed with "could not fetch
... from promisor remote", and an App-installed private repo opened empty.
Reproduced on dev: `git -c extraheader=<A> ls-remote` with the global header set gives 400, and with
`-c extraheader=` first it gives rc 0. The prelude's git auth now clears the list before adding its
header. Worker-only (the prelude ships per create), so the VM scripts stamp is unchanged. Found by the
egress probe, after #790 fixed the proxy side.
Egress firewall migration, workstream A1 (design: .context/cloud/reports/egress-firewall-migration.md
§2, §7). No behaviour change in proxy mode.
- compileSandboxPolicy(deps, row): the whole policy from current state (git token, member
identities, firewall account tokens, keys). A pure read of credentials.
- applySandboxPolicy(deps, row, reason): the one live writer. Serialised per sandbox in the isolate
and ordered across isolates by sandboxes.policy_generation (compare-and-swap): an older compile
never lands over a newer one. 3 tries; a failed widening write marks policy_stale_at (reconcile
retries); a failed narrowing write (revoke, share, uninstall, kill switch, ...) sleeps the
sandbox so a real credential is never left live. Logs the provider code only.
- refreshGitCredentials, env refresh, egress hosts and the Codex refresh push all go through it;
rewriteLivePolicy stays as a deprecated alias.
- EGRESS_SWAP_MODE + sandboxes.egress_swap_mode with egressSwapModeFor(deps, row); firewall mode
implies the Codex firewall swap (CODEX_EGRESS_SWAP becomes an alias).
- Migration 0065: policy_generation, policy_stale_at, egress_swap_mode, policy_config_hash.
- The Codex refresh push no longer logs String(error).
A fresh workspace on an App-installed private repo opened empty. Its clone runs before the control
plane records the row running (the create's prelude, or the background clone right after ready). The
egress proxy refused every row that wasn't `running`, so github.com's forwarded git got
`remote: sandbox_not_running` (403). A wake's git before its running claim hit the same refusal.
Found by the dev egress probe (#786).
The proxy now serves `creating` and `resuming` rows as well as `running`. Only the sandbox that
Vercel's OIDC token names gets them, and only that row's routes. Suspending, suspended, archived,
destroying, destroyed and failed rows are still refused. Tests cover each state, the create-time
clone and the wake-time git before the claim, and a booting VM presenting another sandbox's
placeholder.
The egress probe now needs no Vercel token. It uses dev workspaces with a dev-JWK user, runs its
commands in a workspace terminal through the shipped orca CLI's pairing code, and uses
postman-echo.com as the echo host for bogus keys. It covers the custom-base-URL swap,
chatgpt.com through the relay, api.github.com, #758 anonymous git and the App repo clone.
Premerge runs boot-twice-check.sh for image/** changes. The step has its own omarchy slot and
a 10 min limit, and fails open with a warning without Docker.
Boot-twice (image/boot-twice/): provisions the golden image in a container with the real
provision-golden.sh, then boots the same disk twice through the control plane's own boot steps
(entry-point probe, startOrcad with the workspace prelude, ensureEntryPoints). Each start is a new
kernel boot id. The second boot runs offline behind a DNS sink, and must install, fetch and redo
nothing (docker diff allowlist, logged lookups, sampled command lines). It must come up, carry
this tree's entry points and stamp, and match every manifest version pin. ~15 s cached, ~5 min
cold. It catches the pre-#781 OpenCode first run that repeated on every wake.
Workflow: cloud-vm-boot-twice.yml (PRs touching image/**, GHA layer cache).
Egress probe (scripts/egress-probe.ts): against dev, with bogus tokens and an echo sandbox.
Firewall layer: egressPolicy()'s real policy forwards placeholders for chatgpt.com,
api.anthropic.com, api.github.com and github.com with other auth headers untouched. Credential
hosts fail closed, and anonymous git works without an App token (#758). Swap layer: the dev egress
proxy swaps a bogus ANTHROPIC_API_KEY in for its placeholder on a custom base URL, and leaks no
Vercel hop headers. Workflow: cloud-egress-probe.yml (dispatch + nightly; skips without secrets).
Fix found by the harness: provision-golden.sh linked a release bundle's runtimes dir, and every
VM script's `find .../runtimes -type f` skips a symlinked start point, so --release goldens had
no node. Move the tree instead. No entry-point change (stamp unchanged).
- sandbox-control: an agent the person signs in inside each workspace gets mode 'inside' (keys
dropped as before) and the session env carries ORCA_SIGN_IN_INSIDE=<agents> (ORCA_*: never in
shells). The VM env script then leaves Claude's first-run onboarding unmarked while there is no
login in the workspace, so Claude opens on its own sign-in screens instead of 'Not logged in';
once ~/.claude/.credentials.json exists it is marked as usual. No golden rotation: the env
script reaches older snapshots through the boot-time entry-point refresh.
- Structured chats: a 'not signed in' verdict or start failure for such an agent reads
'{{agent}} needs you to sign in inside this workspace' with 'Open terminal to sign in', which
opens a terminal tab on the workspace running the agent's login (claude auth login, codex login
--device-auth), instead of the Orca-account Renew advice.
- Copy: the choice's detail adds 'Team API keys won't be used for this agent.' (strip, Settings
row, and the Settings button's tooltip).
MEMGUARD. A build or test that eats a cloud VM's memory now costs that process, not the workspace.
- orca-memguard (new VM entry point, root via sudo from orca-sandbox-start): ranks processes for the
kernel OOM killer (Orca runtime -900, terminal shells and agent CLIs -500, everything else 500),
stops the largest unprotected process when MemAvailable drops under 8% and records it, adds 2 GiB
swap files on demand (up to 1.5x RAM) and drops them after 10 quiet minutes, and remounts /dev/shm
at 50% of RAM (the provider mounts 64 MB). Reaches old snapshots through the entry-point refresh.
- orca-sandbox-drain releases on-demand swap before a stop so it doesn't ride the snapshot.
- Status read: the VM metrics probe reads the guard's state; status gains an optional memory
{pressure, guardKills, lastGuardVictim, ...} (metrics keeps its strict shape for older desktops).
- Desktop: guard kills count in the OOM tracker and show 'Orca stopped <process> to keep the
workspace responsive' with a Machine size action; the sidebar row gets a quiet memory mark under
pressure (amber when critical).
A per-person default Source 'in-workspace' for Claude and Codex. Orca stores and injects nothing
for that agent: no account placeholder, no API key, and so no egress route on the vendor's
hosts. The agent starts as it is and shows its own sign-in in the pane; its token stays in the
workspace's home directory and survives sleep.
- sandbox-control: migration 0062 widens agent_sources.source (table rebuilt, rows kept);
PUT /v1/agent-sources accepts 'in-workspace' as a default only (400 with workspaceId);
agentModesFor maps it to 'omit'; GET adds inWorkspace per agent while defaults/effective
keep their two values for older desktops.
- Desktop: coverage kind 'inWorkspace' (no hold, no per-host sign-in gate on Orca Cloud);
'Set up once' strip gets a quiet ⋯ choice 'Sign in inside this workspace instead' that
saves the choice and starts every launch held on it; Settings › Cloud agent sign-ins offers
it on a not-set-up row and shows 'Signs in inside each workspace' with a way back.
- Tests: Worker (env, open/limited policy, live switch, migration) and renderer (model, hold,
spawn gate, strip, settings, build row); fixtures for the strip menu and the settings row.
A create or new environment with nothing pinned used to start in pdx1 whatever the caller's location,
and every frame of a remote session paid that round trip. Now the sandbox-control Worker ranks the
served regions (the default and its fallbacks with an active golden) by distance from Cloudflare's
request geolocation and starts in the nearest.
- Explicit regions (request, API project) and an environment's build region stay authoritative.
- Environments record their region (migration 0061): builds, golden refreshes, staleness and the
security floor use it, so snapshots stay bootable and workspaces from them stay together. Legacy
environments (NULL) live in the default region as before.
- Automatic environments live where the create that made them ran.
- The capacity fallback tries the next nearest served region.
- The warm pool keeps VMs in every served region.
- VERCEL_NEAREST_REGION=off restores default-region-only.
- Desktop: the workspace card's Details shows the region (e.g. Portland (pdx1)).
chatgpt.com answers the sandbox's own egress (probed on a throwaway sandbox: a bogus token gets
ChatGPT's JSON 401, and the firewall's header transform, path and method matchers all work), so the
Codex account token no longer needs the Worker + Cloud Run relay hop for Codex's hot calls.
- Policy compiler: claims scoped by path (exact/prefix, query ignored) and method; transforms can
set extra headers; path-only forwards (no placeholder header).
- With CODEX_EGRESS_SWAP=firewall (dev on), the owner's Codex token is swapped at the firewall for
POST /backend-api/codex/responses(/compact), GET /backend-api/codex/models and GET
/backend-api/wham/usage, with chatgpt-account-id. Exact paths only: a prefix rule was walked out of
live with an encoded dot segment that the upstream normalises. Every other placeholder call still
goes through the egress proxy, whose pinning rejects that; then the 403.
- Same eligibility as the proxy (owner's own account, not shared, provider on, token usable); none:
proxy route only, which answers the signed-out cases. Every policy write path carries the swap
(boot, create, git refresh, env refresh, egress hosts via rewriteLivePolicy); a wake rewrites it.
- Central refresh (already single-flight per person, compare-and-swap) now pushes a refreshed token
to the person's running sandboxes' policies, including refreshes the egress 401 path starts.
- Codex refreshing its placeholder login (POST auth.openai.com/oauth/token) is forwarded by path to
the proxy, which answers with the placeholder again (never a real token) and refreshes centrally.
- VM: the placeholder auth.json's last_refresh is 2099-01-01 (Codex never refreshes it itself);
the orca_chatgpt provider (supports_websockets = false) is also written to /etc/codex/config.toml
(best effort, passwordless sudo); [features] plugins = false while the Orca account applies,
unless the person's own config has a [features] table (CODEX-PS-CALLS: ~170 catalog calls per
session start).
- Off by default: CODEX_EGRESS_SWAP unset keeps every call on the proxy (and the relay).
Neil installed the Orca GitHub App on stablyai, and "Orca Cloud can't push to stablyai/orca yet"
with Check again never cleared: a running VM only got the App's repo token at boot (policy route +
the git placeholder header), so Check again re-ran a check that could never pass.
- sandbox-control: POST /v1/sandboxes/:id/git/access (owner) mints the repo token, swaps it into
the live egress policy, sets the VM git's placeholder header, and records the token expiry.
Answers applied, or not_installed / not_running / not_github. Idempotent.
- Desktop: Check again on the push and can't-reach lines, the poll after Install, and focus
rechecks apply the live refresh first, then re-run the check. If the App reaches the repo and
the check still fails after that, the line says "Restart the workspace to use the new access"
with Restart: never an endless Check again.
- No installation cache to go stale on our side: tokenForRepo caches minted tokens only.