stablyai/orca#42960 (Neil, dev) was authored by orca-cloud-dev[bot]. Neil had connected GitHub; member
identities only reached git: gh ran on the workspace placeholder (the App's installation token), and
the Worker's member route on api.github.com allowed only REST under the repo while gh pr create is
GraphQL.
- VM: a gh wrapper runs gh with the attested sender's own placeholder (orcad attests, as for git);
plain gh otherwise. Behind GH_AS_MEMBER=on (dev on).
- Worker, api.github.com with a member placeholder: the member's own token only for REST under the
workspace repo and for allowlisted authorship mutations whose every target resolves to it
(repositoryId equals its node id; other ids looked up with the installation token, cached per repo,
failures not cached). graphql-js parse; variable defaults, owner-level projectIds, duplicate
top-level keys and anything unexpected are not member requests. Everything else goes out on the
installation token as before; an out-of-repo pinned mutation is refused with a GraphQL error.
- Shared workspaces never approve or merge as a member (GraphQL and REST): those go out on the
installation token, so branch protection still applies.
- Identity rule: the only person who can use a workspace always acts; in a shared one the latest
sender acts until another member sends, for at most MEMBER_GIT_SHARED_MAX_MINUTES (default 480).
- A GitHub connect or disconnect mid-session reaches the running VM's identities now.
- Tokens never enter the VM.
Puts the token-free compile on a probe account's LIVE session, so the probe can see whether a
header-matched respond rule answers at all (separating 'respond not enforced' from 'resume restored
something else').
egress-docs' A4 run: after resume-raw the echo to the custom base-URL host answered 200 where the
token-free record should answer the placeholder 403. The Worker's token-free compile is right (the
host keeps its respond rules: checked in the harness), so the probe needs to see what Vercel holds
per host and what status came back:
- the audit adds domains: {host: {injection, forward, response}} (hostnames only);
- echo-probe returns the HTTP status again. The echoed body stays in a shell variable, never a file.
A Cloud Run cold start cost Codex's first chatgpt.com call ~10.8 s on dev (seen twice on
2026-11-15, each right after an idle period). min_instance_count 0 -> 1 on staging (applied).
egress-docs' review of #805: the echoed request goes straight into grep -c (a swapped value never
touches the VM's disk), and both routes refuse any row outside the throwaway probe accounts
(account_id dev:egress-…).
scrubStoppedRecord re-reads the row right before the PATCH and only scrubs one that is still
suspended or archived with no session; if the row resumed while the PATCH was in flight, the live
policy is re-applied at once (reason scrub_race). Failed rows are left out of the sweep on purpose:
failAndRelease destroys their provider sandbox.
egress-docs' probe 2 re-run through #803 found transformRuleCount always 0: Vercel's redacted shapes
name header swaps injectionRules (session) next to forwardRules and responseRules. The audit now
reports each kind separately (count + header names), so a scrubbed policy is checkable: 0 injection
rules, response rules only.
Two more dev-only operator routes so the probe needs no personal Vercel login:
- POST …/resume-raw: resume the provider session from its stored record with no policy write of
ours (the row is left as it was; dev throwaways only).
- POST …/echo-probe?marker=orcaprobebogus…: one fixed request from inside the VM to httpbin.org
or postman-echo.com carrying a placeholder header (placeholders only, validated); the answer
is whether the echo contained the marker. No body comes back or is logged.
A wake resumes from the scrubbed record, so until its policy write lands the firewall answers the
placeholders itself (the keys-withheld or signed-out errors agents print). The write now retries
(250 ms, 1 s, 3 s); if it never lands, the wake still completes (fail closed, never a hang), logs
policy_apply_failed {reason: boot} by code, and marks policy_stale_at for reconcile's next tick.
Archive and delete stop the VM but nothing it started elsewhere (a preview
database branch keeps billing). The control plane now runs scripts.archive
on the still-running VM: bash with set -e, as the user, with the session's
env and secrets, 2 min for all checkouts, output per checkout and in
~/.orca-sandbox/archive-hook/archive.log.
- POST/GET /v1/sandboxes/:id/archive-hook; migration 0062 remembers whether
the repo has a hook (learned at each sleep) so an asleep archive/delete can
offer a wake for it, and never wakes one without a hook.
- Desktop archive runs it before the Sleep; a failure shows the output with
Archive anyway / Keep it. Asleep: the Undo toast says it wakes first and
offers Skip hook.
- Asleep/unreachable delete runs it through the control plane; a failure
refuses with the usual Delete anyway waiver.
- Unattended archives (public API, team list) run it on a running VM and
archive even if it fails (logged).
- Docs: orca.yaml scripts.archive for Orca Cloud workspaces.
Probe 2 (egress-docs): a session's last network policy survives a stop or a timeout on the
persistent record and is re-applied on resume, so firewall-mode values would sit at rest in Vercel
while a workspace sleeps.
- After every voluntary stop (sleep, rollover, restart, archive) and when reconcile finds the
provider stopped a session (timeout), the record is PATCHed with the token-free compile (same
routes, every firewall value answered by the firewall, no App token). A failed PATCH sets
policy_scrub_pending.
- Reconcile scrubs stopped rows that are pending, or whose last policy carried a firewall class and
was written after their last scrub (timeouts, crashes, and a one-time sweep on deploy).
- A wake PATCHes the record with the full policy right before resuming (no gap); the post-resume
apply stays as the backstop.
- Boots stamp policy_config_hash (and a generation one below now), so create and wake writes are
covered like applies. Proxy-only policies hold no values: nothing to scrub, no record writes.
- Migration 0067 (0064 and 0066 are held by #783 and #792): policy_scrubbed_at,
policy_scrub_pending.
For egress-docs' probe 2, re-run on the Worker's own Vercel token (dev only; elsewhere these routes
don't exist; operator token):
- GET /v1/internal/sandboxes/:id/provider-policy-audit?marker=orcaprobebogus…: the raw record and
session read from Vercel, answered as redacted facts only (state, session status, whether a
network policy is present, transform rule count, header names, markerFound by substring test).
No value or body is returned or logged.
- POST /v1/internal/sandboxes/:id/provider-policy-scrub-test: A4's primitive, scrubRecordPolicy,
which PATCHes the persistent record with the token-free compile (every firewall value answered by
the firewall, no App token; proxy routes hold no values anyway).
compileSandboxPolicy gains { tokenFree }; the provider gains updateRecordNetworkPolicy and a
dev-audit raw read.
Follow-up to #801: the rest of the Workers' shared in-flight promises.
- Auth (sandbox-control desktop, staging and dev keys; billing) and the egress proxy's OIDC keys used
jose's createRemoteJWKSet, which shares one in-flight JWKS fetch across every caller: a request
cancelled mid-fetch hung every token check behind it, and jose refetches after ten minutes. That
fits the 'Start sandbox' creates stuck before any log 10–15 min after a deploy and cleared by a
redeploy. settledRemoteJWKSet keeps the key set as a value; a request that needs a fresh one
fetches it for itself (refetch every 10 min, an unknown kid at most every 30 s).
- applySandboxPolicy no longer chains writes per sandbox in the isolate (a chained promise from a
cancelled request would hang every later write for that sandbox); the D1 generation orders them.
- The KMS keyring cache keeps the loaded keyring and the last failure as values; no in-flight load
is stored at all.
- Guard (both Workers): a unit test fails on createRemoteJWKSet, or a module-level Map, variable
or object holding a Promise, unless the line says why it is safe ('xreq-ok:', e.g. a WebCrypto
import).
Egress proxy secret routes (custom-base-URL keys, OPENAI_BASE_URL) hung until the client gave up
(dev, proxy mode, 2026-11-15). The Worker was invoked (tail: `canceled`, wall 65 s, CPU 3 ms, no log
line) and was waiting, never reaching its upstream fetch. GitHub routes on the same sandbox answered.
An isolate serves many requests, and a promise one request starts is bound to that request's I/O. If
that request is cancelled while the promise waits on D1, KMS or fetch, the promise never settles, and
every later request awaiting the same cached promise hangs. Several module-level caches shared
in-flight promises: the KMS keyring (forever, per isolate), decrypted provider keys and account
credentials (per minute), the relay's ID token mint, the App's pending-request owners, create timing
and share-link lookups.
They now cache settled values only (settled-cache.ts and the same pattern inline). A request that
misses loads for itself; a cold isolate may run a few duplicate loads.
Workstream C of the egress migration. `scripts/egress-probe.ts --mode proxy|firewall|both` flips the
same two dev workspaces with the operator egress-mode route and runs every check in each mode. Each
request is a Node fetch (undici, like the agents) with a 30 s limit, so the transform body hang shows
up as a named failure. New coverage:
- a chunked stream, a 1 MB upload and a WebSocket upgrade through the swap;
- a placeholder sent to an unrelated host must arrive unswapped;
- the WebSocket host fails closed;
- the LFS batch API, git fetch, push and delete;
- the App-repo clone through the desktop's create.
Latency is reported per class (median/p90). The terminal wait is now 15 min, and a project whose
clone failed is registered as a folder, so the checks still run.
On dev at 2cc12610 (A2+A3), firewall mode passes 22/22. Proxy-mode findings went to gh-access.
Security review of A1/A2 (egress-docs):
- F1: the Copilot PAT never reaches the firewall. Eligibility is decided per secret
(firewallEligibleSecret): no credential the proxy narrows on any host (response filtering,
paths, per-host paths). Before, its Copilot API host entries had no paths and compiled to
transforms, putting a user's GitHub PAT in the policy store.
- F2: claims are ordered but landed writes weren't. After a write, if a newer generation was
claimed meanwhile, the write is redone from current state, so a stale compile (say from before a
share) is never the last to land.
- F3: a config-hash change fails closed only when it removes access or custody (firewall to proxy,
a firewall class removed, an account provider turned off, open to limited). Anything else retries
via policy_stale_at, so a deploy during a provider blip can't sleep every workspace. The stamp now
carries the deciding inputs; a pre-F3 bare-hash stamp is retried.
Egress firewall migration, workstream A3 (design §1). In EGRESS_SWAP_MODE=firewall the Codex account
token is swapped at the firewall on every Codex path, exact paths only (the hot calls plus the
catalog GETs); anything else with the placeholder gets the 403. Nothing forwards to the egress proxy,
so the ChatGPT relay is unused for these sandboxes.
Codex's own refresh of its placeholder login (POST auth.openai.com/oauth/token) is answered by the
firewall itself with the placeholder login again; the real token is refreshed centrally on schedule.
The #776 alias (CODEX_EGRESS_SWAP=firewall in proxy mode) is unchanged: hot paths at the firewall,
the rest through the proxy.
A real ChatGPT token is stored on dev; the firewall swap stays off (proxy + relay) until the
Host/SNI binding probe shows a transform can't be reached through another destination.
Egress firewall migration, workstream A2 (design §1, §3, §4, §7 of
.context/cloud/reports/egress-firewall-migration.md). Behind EGRESS_SWAP_MODE=firewall (per sandbox:
sandboxes.egress_swap_mode); proxy mode is unchanged.
- Firewall mode compiles the real values into the sandbox policy: model keys (every header form the
client sends, mapped to the same form of the key), the Claude sign-in, and the GitHub App token
(git Basic on github.com, Bearer/token on api.github.com). The 403 rule still ends every
credential host.
- Fail closed: a key no longer usable, or an account that no longer applies (signed out, revoked,
owner changed, shared, provider off), is answered by the firewall itself (provider-shaped
signed-out body, or the keys-withheld body), never forwarded with its placeholder.
- Stays on the egress proxy: Cursor's key exchange (W1: the reply is filtered), the Copilot PAT on
api.github.com (W2: a prefix scope can't be pinned at the firewall), member identities (W3).
- EGRESS_FIREWALL_CLASSES narrows the classes (model_key, claude, codex, github_app); a removed
class goes back to the proxy.
- policy_config_hash: each apply stamps a hash of the operator inputs; reconcile re-applies a
running sandbox whose hash differs (a kill switch reaches every live policy within one tick) or
whose last write failed (policy_stale_at).
- In firewall mode an env refresh always applies the policy (a key edited in place, a share).
- POST /v1/internal/sandboxes/:id/egress-mode {mode: proxy|firewall|null}: one sandbox's mode,
applied at once (leaving the firewall fails closed).
The desktop's create (clone-progress) runs the workspace clone detached. orca-sandbox-start then writes
the same extraheader into ~/.gitconfig, and http.<url>.extraheader is a list. So the clone's later
requests (the blobless clone's checkout fetch, and the retry clone) carried Authorization twice. The
firewall answered 400 and nothing reached the egress proxy. The clone failed with "could not fetch
... from promisor remote", and an App-installed private repo opened empty.
Reproduced on dev: `git -c extraheader=<A> ls-remote` with the global header set gives 400, and with
`-c extraheader=` first it gives rc 0. The prelude's git auth now clears the list before adding its
header. Worker-only (the prelude ships per create), so the VM scripts stamp is unchanged. Found by the
egress probe, after #790 fixed the proxy side.
Egress firewall migration, workstream A1 (design: .context/cloud/reports/egress-firewall-migration.md
§2, §7). No behaviour change in proxy mode.
- compileSandboxPolicy(deps, row): the whole policy from current state (git token, member
identities, firewall account tokens, keys). A pure read of credentials.
- applySandboxPolicy(deps, row, reason): the one live writer. Serialised per sandbox in the isolate
and ordered across isolates by sandboxes.policy_generation (compare-and-swap): an older compile
never lands over a newer one. 3 tries; a failed widening write marks policy_stale_at (reconcile
retries); a failed narrowing write (revoke, share, uninstall, kill switch, ...) sleeps the
sandbox so a real credential is never left live. Logs the provider code only.
- refreshGitCredentials, env refresh, egress hosts and the Codex refresh push all go through it;
rewriteLivePolicy stays as a deprecated alias.
- EGRESS_SWAP_MODE + sandboxes.egress_swap_mode with egressSwapModeFor(deps, row); firewall mode
implies the Codex firewall swap (CODEX_EGRESS_SWAP becomes an alias).
- Migration 0065: policy_generation, policy_stale_at, egress_swap_mode, policy_config_hash.
- The Codex refresh push no longer logs String(error).
A fresh workspace on an App-installed private repo opened empty. Its clone runs before the control
plane records the row running (the create's prelude, or the background clone right after ready). The
egress proxy refused every row that wasn't `running`, so github.com's forwarded git got
`remote: sandbox_not_running` (403). A wake's git before its running claim hit the same refusal.
Found by the dev egress probe (#786).
The proxy now serves `creating` and `resuming` rows as well as `running`. Only the sandbox that
Vercel's OIDC token names gets them, and only that row's routes. Suspending, suspended, archived,
destroying, destroyed and failed rows are still refused. Tests cover each state, the create-time
clone and the wake-time git before the claim, and a booting VM presenting another sandbox's
placeholder.
The egress probe now needs no Vercel token. It uses dev workspaces with a dev-JWK user, runs its
commands in a workspace terminal through the shipped orca CLI's pairing code, and uses
postman-echo.com as the echo host for bogus keys. It covers the custom-base-URL swap,
chatgpt.com through the relay, api.github.com, #758 anonymous git and the App repo clone.
Premerge runs boot-twice-check.sh for image/** changes. The step has its own omarchy slot and
a 10 min limit, and fails open with a warning without Docker.
Boot-twice (image/boot-twice/): provisions the golden image in a container with the real
provision-golden.sh, then boots the same disk twice through the control plane's own boot steps
(entry-point probe, startOrcad with the workspace prelude, ensureEntryPoints). Each start is a new
kernel boot id. The second boot runs offline behind a DNS sink, and must install, fetch and redo
nothing (docker diff allowlist, logged lookups, sampled command lines). It must come up, carry
this tree's entry points and stamp, and match every manifest version pin. ~15 s cached, ~5 min
cold. It catches the pre-#781 OpenCode first run that repeated on every wake.
Workflow: cloud-vm-boot-twice.yml (PRs touching image/**, GHA layer cache).
Egress probe (scripts/egress-probe.ts): against dev, with bogus tokens and an echo sandbox.
Firewall layer: egressPolicy()'s real policy forwards placeholders for chatgpt.com,
api.anthropic.com, api.github.com and github.com with other auth headers untouched. Credential
hosts fail closed, and anonymous git works without an App token (#758). Swap layer: the dev egress
proxy swaps a bogus ANTHROPIC_API_KEY in for its placeholder on a custom base URL, and leaks no
Vercel hop headers. Workflow: cloud-egress-probe.yml (dispatch + nightly; skips without secrets).
Fix found by the harness: provision-golden.sh linked a release bundle's runtimes dir, and every
VM script's `find .../runtimes -type f` skips a symlinked start point, so --release goldens had
no node. Move the tree instead. No entry-point change (stamp unchanged).