Files
orca/cloud/apps/sandbox-control/scripts/README.md
T
Neil 46a1c8d79f ci(cloud): egress probe in proxy and firewall modes, with streams, uploads, WebSocket, git and latency
Workstream C of the egress migration. `scripts/egress-probe.ts --mode proxy|firewall|both` flips the
same two dev workspaces with the operator egress-mode route and runs every check in each mode. Each
request is a Node fetch (undici, like the agents) with a 30 s limit, so the transform body hang shows
up as a named failure. New coverage:
- a chunked stream, a 1 MB upload and a WebSocket upgrade through the swap;
- a placeholder sent to an unrelated host must arrive unswapped;
- the WebSocket host fails closed;
- the LFS batch API, git fetch, push and delete;
- the App-repo clone through the desktop's create.
Latency is reported per class (median/p90). The terminal wait is now 15 min, and a project whose
clone failed is registered as a folder, so the checks still run.

On dev at 2cc12610 (A2+A3), firewall mode passes 22/22. Proxy-mode findings went to gh-access.
2026-11-15 13:54:30 -08:00

4.6 KiB

Egress release probe

egress-probe.ts checks sandbox egress against dev before a release. It goes through Orca's own paths: real dev workspaces created with a dev-JWK user token, with commands run in a workspace terminal over the shipped orca CLI's pairing code (the desktop's E2EE path to orcad). Every credential is bogus. httpbin.org plays the model provider, ws.postman-echo.com is the WebSocket host, and postman-echo.com is an unrelated host.

Setup for a throwaway identity, all of it removed afterwards:

  • the dev GitHub App's installation, bound for the test repo (operator route);
  • a stored GitHub identity, so member placeholders exist (the dev member-github route);
  • bogus personal secrets:
    • ANTHROPIC_API_KEY, with ANTHROPIC_BASE_URL set to the echo host;
    • OPENAI_API_KEY, with OPENAI_BASE_URL set to chatgpt.com/backend-api/codex;
    • COPILOT_PROVIDER_API_KEY, with COPILOT_PROVIDER_BASE_URL set to the WebSocket echo host.

The probe creates two workspaces through the desktop's create path (clone after ready):

  • public: a public repo the App isn't installed on. Member placeholder, no App token (#758).
  • app: the App-installed test repo. App token, github.com fail-closed.

Each mode is checked on the same two workspaces. --mode proxy|firewall|both flips them with the operator route POST /v1/internal/sandboxes/:id/egress-mode; without --mode, the environment's default applies. Every request is a Node fetch (undici, like the agents) with a 30 s limit, so the undici body hang shows up as a named failure.

Host or class What must happen
echo host (custom base URL, a model key) The provider sees the bogus key where the VM sent the placeholder. The other authorization header arrives exactly as sent, and no vercel-forwarded-* header or OIDC token arrives. A foreign key gets the firewall 403 orca-egress-denied. A chunked stream finishes. A 1 MB upload goes through: the public echo hosts refuse bigger bodies themselves.
chatgpt.com The placeholder reaches ChatGPT: any provider answer counts, but not the firewall 403 and not Cloudflare's edge page. A foreign bearer gets the firewall 403.
WebSocket host A WebSocket upgrade (101) through the swap. No credential gets the firewall 403.
an unrelated host A placeholder sent there arrives unswapped: the real value never leaves for the wrong host.
api.github.com No credential gets the firewall 403. With the workspace placeholder, GitHub answers.
github.com Git smart-HTTP and the LFS batch API work with the workspace header, as do git fetch, a push of a probe branch and its delete. Anonymous git ls-remote works on public (#758) and is refused on app. Both repos clone through the desktop's create.
VM env The keys hold placeholders. This is checked by sha256, so the bogus values never cross the terminal.

The report includes latency per class (median and p90 over 5 keep-alive samples where set): model_key, model_key_stream, upload_1mb, codex, websocket, github_api, git_http, git_lfs. It reports these per mode and per workspace.

What it can't cover with bogus credentials: real provider SSE (Anthropic /v1/messages, Codex /responses answer 401 before streaming). The chunked stream from the echo host exercises the same transform path instead.

The probe takes about 3 minutes per mode, deletes what it created (--keep leaves it), prints one JSON report and exits 1 on any failed expectation. It refuses any control plane other than dev.

Run it locally

You need the Orca app installed (for its orca CLI) and gcloud access to the dev secrets.

cd cloud
pnpm install --filter @orca-cloud/sandbox-control...
K=$(mktemp -d)   # 0700; delete it afterwards
gcloud --account <you> secrets versions access latest --project onorca-cloud-staging \
  --secret orca-sandbox-control-dev-test-jwk > "$K/jwk.json"
gcloud --account <you> secrets versions access latest --project onorca-cloud-staging \
  --secret orca-sandbox-control-dev-operator-token > "$K/operator"
ORCA_PROBE_DEV_JWK_FILE="$K/jwk.json" ORCA_PROBE_OPERATOR_TOKEN_FILE="$K/operator" \
  pnpm --filter @orca-cloud/sandbox-control exec tsx scripts/egress-probe.ts --mode both
rm -rf "$K"

ORCA_PROBE_CLI points the probe at another orca CLI. The default is the installed app's.

CI

.github/workflows/cloud-egress-probe.yml runs on manual dispatch. It installs the CLI from the latest release's Linux package. It needs the ORCA_DEV_TEST_JWK and ORCA_DEV_OPERATOR_TOKEN repository secrets, and warns and skips when they are missing. None are configured today.