orca/cloud/apps/acceptance/findings.json

{
  "findings": [
    "<b>Wake delivery (2026-11-10, coordinator target p95 &lt; 6 s):</b> NOT MET. The golden with #141 (5adc8fb, n=12) gives total p50 6.8 s, p95 11.0 s, against 7.7 / 9.5 s before #141 (n=10). #141 cut orcad ready from p50 2.4 to 1.5 s (p95 3.4 to 2.2 s). Vercel resume to orcad spawn stays at p50 3.0 s, and its tail (p95 6.4 s; 2 of 12 above 5.5 s) now dominates p95. After ready: first terminal list 0.5 s (includes the E2EE handshake), terminal create 0.5 s, typed command to echo 1.3 s (the new login shell's startup). Biggest remaining levers: the Vercel resume tail (core/Vercel), login-shell startup in the image (about 1.3 s), and reusing the warm connection the desktop already holds. Script: <code>desktop/wake-split.ts</code>.",
    "<b>W1-1 (fixed, #118/#123):</b> builds used to advance only while a client streamed <code>/events</code>; GET polling sat in <code>running</code> for 25 min. With the Durable Object runner deployed, builds finish when the client only polls GET: build 1 of a fresh environment in 17 s, build 2 in 18 s. An earlier 250 s first build had started before #123 went live. The harness now polls by default; <code>ORCA_ACCEPTANCE_BUILD_EVENTS=1</code> follows the events stream.",
    "<b>D-1 (bug, core desktop, high): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> starting an Orca Cloud create before signing in, then signing in from the card, created a sandbox owned by the fallback dev user (<code>desktop-dogfood</code>), not the signed-in user. Keep-alive, Copy SSH Command and Delete then got <code>not_found</code>. Likely cause: <code>getHostedSandboxConfig()</code> caches the token for the app process and does not reset it on sign-in (<code>src/main/hosted-sandbox/hosted-sandbox-config.ts</code>).",
    "<b>D-2 (bug, core desktop, high): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> Delete Workspace hides the row and reports success when the control plane answers <code>not_found</code>, while the VM keeps running (a billing leak).",
    "<b>D-3 (bug, core desktop): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> after a cloud create, the checkout at <code>/vercel/sandbox/project</code> is added as a new project named \"project\" instead of being grouped under orca-sandbox-push-test.",
    "<b>W3-dl (pass):</b> download over ONE persistent paired connection (the desktop lane): 50 MiB in 3.5 s (14.2 MiB/s from iad1 to this Mac, sha256 matches), 50 MiB folder zip in 5.4 s and complete. That matches W3's desktop numbers (50 MiB p50 15.3 MB/s; 200 MiB about 19 MiB/s; 2,000-file zip 1.8 s). The earlier 1.5 MiB/s came from the harness opening a new connection per RPC (corrected).",
    "<b>Remote desktop (pass):</b> the harness verifies the route and the ticket gate (101 with a ticket, 403 with a bogus one). W3 re-verified end to end with no manual steps: ticket mint p50 282 ms, connected in 130 ms, ticket reuse and a wrong Origin both 403. #138 (polish) and #146 (0-RTT forward open, p50 127 to 64 ms) ship with the next golden.",
    "<b>D-4 (minor, desktop): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> the \"Sign in to use Orca Cloud\" card stays after sign-in; the Ports panel says \"Port scan unavailable on unknown\" while waking; \"Move to this computer\" has no confirmation and copies rather than moves (the VM keeps running); the agent picker is disabled for Orca Cloud.",
    "<b>W6-1 (resolved):</b> provider keys are now egress placeholders in orcad's env (swap on by default on dev), so <code>sec.precedence</code> reads <code>AWS_REGION</code>. W6 confirmed the swap resolves keys with the same precedence as injection (team &gt; repo &gt; personal for agent keys) and added a test at the egress proxy (cloud/w6-precedence-test). Real Claude Code and Codex PRs: nwparker/orca-sandbox-push-test#5, #6.",
    "<b>Fixed since the last report:</b> F1 resume on a running sandbox now returns in 0.5 s (was a 60 s 502); F3 ai-gateway.vercel.sh is allowlisted (real Claude Code task passes); F4 the image now has orca, rg, tmux, vim, git-lfs and Chrome; F5 disk is 64 GB.",
    "<b>Staging cleanup (ops):</b> two throwaway staging teams (<code>org_887caf34…</code>, <code>org_3b7b1fd1…</code>) are left over. Deleting a team needs the PropelAuth API key through gcloud, and gcloud needs an interactive re-login on this Mac. Auth has no team-delete route.",
    "<b>Desktop run paused:</b> the user is on this Mac (no visible windows), so the Electron run is paused. The last desktop results are from desktop-20261109-154725.",
    "<b>Note:</b> kill-switch drain mode is not exercised on the shared dev Worker (it would suspend other workstreams' sandboxes); refusal of creates and wakes is verified."
  ],
  "gaps": [
    "Real Anthropic-model Claude run is pending an AI Gateway credit top-up (no anthropic/* models on free credits); agent.real-task uses moonshotai/kimi-k2 through Claude Code.",
    "Payment-grace expiry (96 h) is not waited on: dunning verifies past_due, blocked starts, grace window, and pay leading to resume.",
    "Team create/invite need the Orca Cloud auth endpoints on staging (team-billing-ux is standing them up).",
    "The 15-min idle auto-sleep is not waited out in the routine run (config value checked); the mobile app UI is covered by W8's own runs (device-session API verified here)."
  ]
}