Commit Graph
955 Commits
Author SHA1 Message Date
Neil dca419d3fd fix(cloud): CODEX-EGRESS-BLOCKED say so when chatgpt.com refuses the egress proxy
chatgpt.com answers every request from Cloudflare Workers' egress IPs with a 403 "Unable to load
site" HTML page, whatever the token (verified from a throwaway Worker; the same request from
elsewhere gets a normal 401). Codex via the Orca account goes through the Worker egress proxy, so
it got that page on /responses and /wham/usage: "Working…" without an answer, and /status had no
limits.

The proxy now recognises a provider edge's own refusal (403, text/html, server: cloudflare) on
routes that define an answer for it, logs egress_upstream_blocked, and answers Codex in ChatGPT's
error shape so Codex prints a plain message. The block says nothing about the credential: no
status hook, so no refresh and no revoke. This doesn't make Codex work; routing chatgpt.com around
the Worker is a separate decision.
2026-11-15 00:09:55 -08:00
Neil c24f97588e fix(sandbox-control): 20 s provider deadline, one retry for stalled reads; dev Worker keeps its logs
team-lead follow-up to #762: lookups/updates now fail at 20 s; a GET that hits the deadline is asked once more (a stalled write is never repeated: it may have landed); each stall logs provider_stall with the call's shape. orca-sandbox-control-dev turns on Workers observability so provider_stall, grant_failed and lifecycle_timing are retained.
2026-11-14 23:40:25 -08:00
Neil 6f1bbcfc55 Merge pull request #763 from nwparker/cloud/golden-agent-prefetch
perf(golden): prefetch the agent CLIs into the page cache at boot
2026-11-14 23:22:32 -08:00
Neil 6a02019931 chore(golden): regenerate the Worker-embedded VM scripts 2026-11-14 23:20:02 -08:00
Neil 845e495128 perf(golden): prefetch the agent CLIs into the page cache at boot
A fresh VM pages the agent CLIs in lazily from its snapshot, so the first agent launch paid for it (OpenCode 11 s cold vs 3.75 s warm in the same VM, n71). orca-sandbox-start now reads the claude, codex and opencode packages (files over 256 KB) in the background at nice 19 / idle IO, detached, after orcad starts; never on the ready path. Best effort.
2026-11-14 23:17:54 -08:00
Neil 027fc090d5 fix(sandbox-control): deadlines on every Vercel API call; slow grants log their phases
Two dev failures (n70/n71) were the Worker waiting on a stalled Vercel API connection with no
client-side deadline: a create spent ~2 min before its write-ahead insert ("Start sandbox" stuck),
and POST /v1/team/workspaces/:id/grants (opening a new workspace on this computer) hung its full
120 s (wallTime 119995 ms, no response) although its VM command is bounded at 45 s.

- Every provider request now has a deadline: 30 s for lookups and updates, 60 s to boot a VM,
  5 min to snapshot, a command's own timeout + 20 s, and 3 h only for an explicit unbounded wait
  on a build's exit. A stall becomes a provider error the callers already handle (grant_failed,
  the create fallback), instead of an open-ended wait.
- issueGrant logs its phases (session, grant command, member git) when it takes over 10 s, and
  logs grant_failed with its timings and error, so the next slow open names its phase.
2026-11-14 23:14:24 -08:00
Neil 6b05760eb7 fix(cloud): GH-ANON-403 member GitHub identities no longer 403 anonymous git to github.com
A workspace on a public repo the Orca GitHub App isn't installed on gets no App token, so its git
sends no credential. When a member had connected GitHub, their placeholder rules made github.com a
credential host, and compileNetworkPolicy ended it with the firewall's 403: the doctor's
`git ls-remote origin HEAD` failed ("This workspace can't reach <repo> on GitHub right now"), and so
did every plain fetch and pull.

- With no App token, the member github.com route is allowUnclaimed. With a token the workspace git
  route keeps github.com fail-closed as before. api.github.com is unchanged.
- A wake rewrites the policy when member identities apply: a resumed session keeps its last
  policy, so a workspace created before this fix would otherwise keep the 403.
2026-11-14 22:12:45 -08:00
Neil f52303e359 fix(infra/secrets-kms): ignore escrow version aliases/annotations set by gcloud (plan was clearing k2 alias) 2026-11-14 21:29:13 -08:00
Neil 786e3064e3 Merge pull request #753 from nwparker/cloud/secrets-kms-dev
SEC-3: KMS-wrapped secrets KEK on dev (Terraform root + plaintext-KEK migration opt-in)
2026-11-14 21:24:39 -08:00
Neil 2400ca2cd9 Merge pull request #750 from nwparker/cloud/snapshot-leaks
fix(sandbox-control): no snapshot leaks on any delete path; nothing deletes a snapshot still in use
2026-11-14 21:10:13 -08:00
Neil f637ad8cfb Merge pull request #752 from nwparker/cloud/share-ws-keepalive
fix(sandbox-control): share-link WebSockets get a keepalive ping under Vercel's ~340 s idle cut
2026-11-14 21:08:14 -08:00
Neil 797aaeb178 feat(sandbox-control): SEC-3 KMS turn-on for dev: Terraform root for the KEK wrap key, plaintext-KEK migration opt-in
- cloud/infra/secrets-kms: own Terraform root and state (terraform/secrets-kms) for the Cloud KMS key
  that wraps the secret-store KEK, the Worker's decrypt-only service account, KMS/Secret Manager
  data-access audit logs and the KEK escrow secret. Applied to onorca-cloud-staging (dev Worker).
- SECRETS_KEK_PLAINTEXT_RETIRED_ID: in kms mode, also open rows under that id with the plaintext
  SECRETS_KEK as a retired key, only after KMS handed over the active KEK. Neither dev nor prod has a
  copy of its KEK outside Cloudflare, so the KEK can't be wrapped; this lets the rewrap sweep move
  rows onto a new KMS-held KEK instead. Never seals, never stands in for KMS (tests).
- wrangler.dev.jsonc: SECRETS_KEK_SOURCE=kms, SECRETS_KMS_KEY, SECRETS_KEK_ID=k2.
2026-11-14 20:58:28 -08:00
Neil 0148e0c48c fix(sandbox-control): share-link WebSockets get a keepalive ping under Vercel's ~340 s idle cut
Every Orca socket into orcad already gets a WS ping every 10-15 s (orcad's server heartbeat and the
desktop's client liveness). Share links were the gap: the in-VM share proxy piped upgraded sockets raw, so
a preview app's silent WebSocket was closed by Vercel after ~340 s (1006), well before the link's
deadline. The proxy now follows the app's frame boundaries and, after 30 s with no traffic either way,
sends the visitor an empty ping frame between two frames (never inside one). Only 101 upgrades to
websocket get pings; other upgrade protocols stay a raw pipe.

VM script: reaches live VMs through the boot-time entry-point sync (new stamp), and the next golden.
2026-11-14 20:57:31 -08:00
Neil 44595111e2 fix(sandbox-control): a retried sandbox command never runs twice (agent messages, launches)
Vercel's command POST has no idempotency key, so a retry after a lost answer (dropped connection, gateway
5xx, a Worker that died mid-request) can start the command again (vercel/sandbox#363). Message delivery
did exactly that: a transient failure re-queued the message and the cron typed it into the agent again,
or launched a second agent.

- command-once.ts: an in-VM guard with a per-id ledger (mkdir claim, recorded exit + output). A repeat of
  the same id waits for and replays the first run; a dead first run reports an unknown outcome; a failed
  run is forgotten so a retry runs for real. runCommandOnce retries uncertain provider failures with the
  same id (never a 4xx that proves the command did not start).
- deliverMessage uses msg-<id> for send/launch; an unknown outcome fails the message
  (delivery_outcome_unknown) instead of re-sending.
- POST /messages and POST /sessions take Idempotency-Key (same semantics as workspace create); a session
  create taking over a died attempt replays the first launch.
- The REST client keeps sending each command POST exactly once (test).
2026-11-14 20:57:30 -08:00
Neil bde0bed3ce fix(sandbox-control): no snapshot leaks on any delete path, and nothing deletes a snapshot still in use
- snapshot-refs.ts: one check for what live things still boot from (goldens of any live status, pool
  images, active environment builds, live workspaces' own snapshots, and a live workspace's boot image
  until it has a snapshot of its own).
- Retired goldens beyond the rollback and deleted environments' builds are marked deleted only after the
  provider delete succeeded (a failed delete used to be marked and leaked for good), and are deferred
  while in use. A new hourly housekeeping step (sweepReleasedSnapshots) retries both.
- Team delete marks the team's environments deleted, so their build snapshots are freed (they leaked).
- Environment prune skips a build a live workspace still boots from.
- Orphan sweeps (per box and project-wide automatic) also respect pool and boot-image references.
- The golden smoke box is non-persistent: it no longer leaves a ~325 MB automatic snapshot behind.
2026-11-14 20:57:17 -08:00
Neil f2888317b4 feat(cloud): plans dialog before the web console; billing serves each plan's card copy
"Plan & billing", "Upgrade to Orca Cloud", "Choose plan" and every "Compare plans" now open an
in-app plans dialog first: the team's plan and status (trial hours left, active with hours and
seats, payment failed, or no plan with the trial billing offers), then billing's plan cards (Team
with a monthly/annual toggle, Enterprise with contact sales). "Continue in browser" opens the
console's billing page through the existing sign-in ticket. Members see the same plan info with
"Ask a team admin to upgrade" (Plan & billing is back in their account menu, read-only). Read-only:
no checkout, usage history or admin settings return to the desktop.

Everything shown is billing's: names, Stripe prices, seat pricing, feature bullets and trial terms
from GET /v1/plans, now also each plan's tagline, feature-list lead and button label (new optional
catalog `marketing`, unversioned copy, plus `omitFeatures` so Enterprise stops advertising egress
controls). Main's lenient parse keeps a newer or older service safe; without the catalog the
entitlement's offers stand in, and with neither the dialog says plans didn't load and still offers
the console. While billing loads the title is neutral, so a paying team never sees an upsell flash.

Settings > Account's billing row opens the dialog ("View plans"); "Update payment method" still
goes straight to the console.
2026-11-14 16:37:59 -08:00
Neil 8104f84b94 billing: record one plan start per Checkout
One Checkout sends checkout.session.completed, customer.subscription.created and invoice.paid,
often concurrently; each that re-read the row before another wrote it saw the plan start and
pushed billing.plan_started with its own 'at', so the event ids differed and the team's audit
log showed it twice. A plan starting or ending now carries a once key (subscription id and
period start), so every racer pushes the same event id and the control plane's INSERT OR IGNORE
keeps one.
2026-11-14 03:45:08 -08:00
Neil b9a2ff9896 feat(billing): Checkout and the Customer Portal started from the web console return to its billing page
POST /v1/billing/checkout and /portal accept returnTo: 'console'. The return URL is built from
this Worker's CONSOLE_URL (else the setup pages' auth origin, which forwards /console) and the
caller's own team, never from the request. Desktop-started flows keep the /return handoff page.
2026-11-14 02:18:51 -08:00
Neil f1573ece1c test(billing): use the real help@stably.ai inbox in supportUrl fixtures (orca.dev has no MX) (#740) 2026-11-14 02:01:12 -08:00
Neil 84a776923c docs(cloud): sandbox-control typecheck split pinned by path; README on why raw tsc shows imported diagnostics 2026-11-13 11:16:23 -08:00
Neil ec679a8f52 fix(cloud): sandbox-control typecheck to zero (SANDBOX-CONTROL-TYPECHECK) 2026-11-13 11:14:50 -08:00
Neil d20dc4e20a feat(cloud): TRANSFER-SLOW-N60 a completed owner hand-off logs how long each step took 2026-11-13 11:04:44 -08:00
Neil 46f081f336 Merge pull request #703 from nwparker/cloud/g9-claude-key-tail
fix(cloud): G9 never write a real ANTHROPIC_API_KEY tail into ~/.claude.json
2026-11-13 10:33:28 -08:00
Neil 648f5bb67f feat(cloud): OWNER-TRANSFER previous-owner notices expire after 90 days and go with a deleted team 2026-11-13 07:39:07 -08:00
Neil a58aad5522 Merge pull request #713 from nwparker/cloud/owner-transfer-notify
feat(cloud): OWNER-TRANSFER tell the previous owner; expose the owner-changed mark
2026-11-13 07:27:20 -08:00
Neil 881b82e401 Merge pull request #711 from nwparker/create-speed/snap404-skip-deleted
fix(cloud): the fresh-build 404 retry never runs for a create its owner deleted
2026-11-13 07:16:53 -08:00
Neil 04457ec384 fix(cloud): OWNER-TRANSFER an email answer with no mode is not a delivery (sso-scim review) 2026-11-13 07:15:58 -08:00
Neil 8026ca98af feat(cloud): OWNER-TRANSFER tell the previous owner (notice in Orca, email asked of auth, audit row); expose the owner-changed mark 2026-11-13 07:14:33 -08:00
Neil 950a8165c8 Merge pull request #691 from nwparker/cloud/ep10-cancel-on-delete
feat(billing): EP-10 team delete can cancel a plan that's already set to end (cancel-now)
2026-11-13 07:11:50 -08:00
Neil 7922504e4b fix(cloud): the fresh-build 404 retry never runs for a create its owner deleted 2026-11-13 06:59:42 -08:00
Neil e05dee2312 Merge remote-tracking branch 'private/cloud/integration' into cloud/env-build-churn 2026-11-13 06:50:47 -08:00
Neil 59ed67dd0c Merge pull request #689 from nwparker/cloud/ep9-billing-mint-org
fix(cloud): EP-9 billing tokens name the team this device shows (cross-device active-team leak)
2026-11-13 06:49:21 -08:00
Neil 5e6fcd30f3 Merge pull request #702 from nwparker/cloud/owner-transfer-fix
fix(cloud): OWNER-TRANSFER a workspace is only handed to a current team member, and a transferred workspace stays in shared mode for credentials
2026-11-13 06:42:12 -08:00
Neil 77db6ffc48 fix(billing): EP-10 cancel-now decides on Stripe's live state: a plan resumed after auth's delete check is refused (409 not_cancelling) (sso-scim) 2026-11-13 06:39:43 -08:00
Neil 3e228f227d feat(billing): EP-10 team delete can cancel a plan that is already set to end
GET /v1/internal/accounts/:accountId/subscription adds cancelling / endsAt / periodEnd (additive);
new POST .../subscription/cancel-now (membership-sync token) cancels the team's subscription
immediately (invoice_now: reported overage is billed; prorate false: no refund) but only one
already set to end (cancel_at_period_end or a portal cancel_at); a renewing plan gets 409
not_cancelling. Idempotent; a Stripe failure is a 502 and changes nothing, so auth must not
delete the team. Mirror re-read from Stripe, grants revoked.
2026-11-13 06:39:43 -08:00
Neil 15b94d9567 Merge remote-tracking branch 'private/cloud/integration' into cloud/ep9-billing-mint-org 2026-11-13 06:39:11 -08:00
Neil 1b59a54b9f Merge remote-tracking branch 'private/cloud/integration' into cloud/env-build-churn 2026-11-13 06:39:03 -08:00
Neil 5faae7d94b Merge pull request #686 from nwparker/cloud/ep9-cancel-at
fix(billing): EP-9 a portal cancel (Stripe cancel_at) shows as the team's own plan ending
2026-11-13 06:32:14 -08:00
Neil b671273718 Merge remote-tracking branch 'private/cloud/integration' into cloud/env-build-churn
# Conflicts:
#	cloud/apps/sandbox-control/src/environments.ts
2026-11-13 06:18:54 -08:00
Neil 477583f9f4 fix(cloud): EP-5 a cancelled create never goes on with a new row; cancel sweeps the workspace 2026-11-13 06:18:39 -08:00
Neil 771b19d532 Merge remote-tracking branch 'private/cloud/integration' into cloud/owner-transfer-fix
# Conflicts:
#	cloud/apps/sandbox-control/src/index.ts
2026-11-13 06:14:23 -08:00
Neil 0a9b8a6bde test(cloud): OWNER-TRANSFER a wake racing the hand-off refuses it and leaves the VM as woken 2026-11-13 06:13:47 -08:00
Neil 90e181af0f fix(cloud): OWNER-TRANSFER stop before the write; refuse during a boot or stop, and when the stop fails (sso-scim review) 2026-11-13 06:13:02 -08:00
Neil ff8f915c05 fix(cloud): G9 ~/.claude.json pre-approves only an Orca placeholder's tail; a real ANTHROPIC_API_KEY leaves nothing of itself on disk and an older tail is scrubbed 2026-11-13 06:10:57 -08:00
Neil e746a764e5 Merge pull request #699 from nwparker/cloud/e3-swap-fail-closed
fix(cloud): EGRESS-SWAP-FAIL-CLOSED withhold keys without the egress proxy in deployed envs; health 503 egress_proxy_missing
2026-11-13 06:09:26 -08:00
Neil 93f2066812 fix(cloud): OWNER-TRANSFER a workspace is only handed to a current team member, and a transferred workspace is in shared mode for credentials for good 2026-11-13 06:09:22 -08:00
Neil 20c83d1412 fix(cloud): EGRESS-SWAP-FAIL-CLOSED review: egress_swap_disabled log when an operator turns placeholders off, proxy refuses without its URL 2026-11-13 06:05:44 -08:00
Neil c9f68dfa63 fix(cloud): EGRESS-SWAP-FAIL-CLOSED merged spec: egress_swap_unavailable at error level (count only), health keySwap on/off, design-squad message, deployed-with-proxy test 2026-11-13 06:04:30 -08:00
Neil a5cf31a7a8 Merge pull request #694 from nwparker/create-speed/build-snapshot-404
fix(cloud): retry a fresh build snapshot's 404 once; name orcad start failures
2026-11-13 06:03:14 -08:00
Neil ee19ee2684 fix(cloud): EGRESS-SWAP-FAIL-CLOSED a deployed env without EGRESS_PROXY_URL withholds model keys and account sign-ins (firewall answers the placeholder), logs once per session, /v1/health 503 egress_proxy_missing 2026-11-13 06:03:03 -08:00