orca/cloud/apps/acceptance/findings.json

{
  "findings": [
    "<b>Wake delivery (target p95 &lt; 6 s) — MET in pdx1:</b> after core U17/U18 (default region pdx1, Standard 8 vCPU/16 GB, golden 63b6999), n=20, persistent connection. Wake to echo p50 5.04 s, p95 5.85 s, p99 5.94 s; 0/20 slow hosts. Pinned to iad1 at 8/16: p50 5.31, p95 10.59, p99 11.80 s, with 4/20 slow hosts. BEFORE (iad1 4/8): 5.28 / 8.99 / 9.40 s. Region, not size, removes the tail. From this Mac the first terminal list (0.18 vs 0.44 s) and echo (0.07 vs 0.15 s) are also faster in pdx1. Create p50: pdx1 2.03 s, iad1-pinned 4.26 s (pinned creates seem to skip the warm pool).",
    "<b>run.ports (cloud) passes on a #233 golden (iad1 d323e4a):</b> first workspace 3000/10/3000, a second worktree on the same VM 26600/10/26600, both unchanged after sleep/wake, both persisted in the VM registry. pdx1 still runs 2d99d9b (pre-#233) and gives a hash block until its golden rotates. Desktop/golden version skew is a launch risk to watch. run.nonconcurrent passes with #233 (only the same action stops, 471 ms).",
    "<b>Read while asleep (gap 5, #189/#218), verified by W9 live:</b> a teammate decrypted a sleeping team workspace (git incl. a new file, PR, sessions) with no wake: the Vercel session stayed stopped. 20 reads p50 89 ms, max 131 ms, vs about 5-6 s to wake. An outsider gets 404, and the plaintext marker is not in D1. Removal/private access control is covered by life.read-asleep-access. The phone view was checked in W9's own headless simulator run: “Latest from this workspace · As of just now” showed the marker branch and NOTES.md, and Vercel stayed stopped.",
    "<b>Read-asleep access, triggered by W9 itself:</b> the operator members/removed route revokes the member; their pre-removal token and a fresh token both get 401, and their wrapped keys go 1 to 0, while the other teammate still reads. Making the workspace private gives that teammate 404 and 0 keys. Phone protocol: device token plus mobile reader key reads a sleeping private workspace in 125 ms, and unpairing revokes access.",
    "<b>Independent audit corrections (2026-11-10, reports/conductor-parity-audit.html):</b> five rows are capped at partial because they passed on a weaker bar than the Conductor column: real-model agent run, subscription sign-in, default machine size, and the download and remote-desktop UIs. Four rows Conductor has that our bar left out are added as not landed: local core workflow, Run actions with per-workspace ports, read-while-asleep from any device, and per-turn code checkpoints. Our 'pass' counts measure the team's own bar. Every cloud result is on dev Workers, Stripe TEST mode and dev/staging tokens; nothing is shipped (mvp-tracker §5 launch items are all open).",
    "<b>Agents and machine size (2026-11-10):</b> image.agents: Claude Code, Codex, Cursor, OpenCode, Gemini and Copilot all answer --version on a fresh VM. agent.keys-swap: stand-in Gemini/Cursor keys exist on the VM only as placeholders; the provider sees the swapped key (Gemini 400 API_KEY_INVALID, Cursor 401); a foreign key or a disallowed path is refused (403); the stand-in values appear in 0 files and 0 process environments. sub.size cap lifted: Standard is now 8 vCPU / 16 GB (core U18, #194), and machineSize large maps to standard.",
    "<b>Audit caps lifted by real-UI runs (m4air, 855d8ed34b6, desktop-20261109-234651):</b> acc.download, file-tree Download: 20 MiB file in 2.1 s and a folder as zip in 0.6 s, with real progress and success toasts and matching sha256. Caveat: only the macOS Save panel answer was stubbed in main, because CDP cannot drive a native panel. acc.remote-desktop: Open remote desktop tab connected in 3.4 s; noVNC canvas streams non-blank pixels, and a red xterm launched in the VM appeared on the canvas. local.core-workflow passes (local worktree in 0.6 s, terminal edit, diff in Source Control, stage, commit).",
    "<b>Rows closed 2026-11-10:</b> real 15-min idle sleep: an idle VM with no client stopped by itself after 19.2 min. Agent sign-in persistence: Claude answers, onboarding is pre-seeded, and the auth homes are kept fresh, after a wake and after a rollover. Auto-resume after Restart on m4air: one <code>continue</code> 29.4 s after the click, turn finished, no double continue. Wake on message: a message to a sleeping workspace was delivered in 6.1 s. Scheduled routine: fired by itself and succeeded on a fresh VM, which was then destroyed. File search: ripgrep over the pairing found both needles in 5,002 files, p50 88 ms. Usage metering: control-plane outbox, then billing hours, then the Stripe meter. Overage cap blocks new starts at $5. Dunning end to end: grace expiry (past_due_since backdated 97 h) leads the control-plane cron to suspend the running VM in 123 s; paying lets the account start again and the VM resumes. Desktop keystroke n=50: p50/p95/p99 84/94/144 ms; the 407 ms outlier did not recur. Web client is post-MVP.",
    "<b>Control-plane billing gate (closed via core #166):</b> a <code>dev:billing-test-*</code> account goes through the real billing Worker. No plan → 403 no_subscription; active plan → 200; 4th concurrent VM on 1 seat → 403; included hours used up → 403; failed renewal → 403 payment_past_due; grace expired → the running VM was suspended by the cron in 71 s and resume → 403; paid → resume 200. Staging team checks pass again (teams are deleted by the shared cleanup script).",
    "<b>CLI create (closed via W5 #167):</b> <code>orca worktree create --on cloud --repo …</code> with an API key, headless: running in 3.7 s; <code>orca cloud workspace destroy</code> works; signed out gives cloud_not_signed_in. <b>Move, Sleep, then Delete:</b> no wake, destroyed, Vercel 404, no misleading toasts. Row removal went from 31 s to 1.5 s / 1.7 s after core #171, which skips the hooks check for asleep sandboxes.",
    "<b>orca.yaml setup in the VM (closed via core #168):</b> an API-created workspace on test-repo branch <code>w9-setup-script</code> ran scripts.setup inside the VM exactly once, including after sleep/wake. The environment-build skip path is unit-tested by core; the desktop path runs after the trust prompt.",
    "<b>W1-1 (fixed, #118/#123):</b> builds used to advance only while a client streamed <code>/events</code>; GET polling sat in <code>running</code> for 25 min. With the Durable Object runner deployed, builds finish when the client only polls GET: build 1 of a fresh environment in 17 s, build 2 in 18 s. An earlier 250 s first build had started before #123 went live. The harness now polls by default; <code>ORCA_ACCEPTANCE_BUILD_EVENTS=1</code> follows the events stream.",
    "<b>D-1 (bug, core desktop, high): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> starting an Orca Cloud create before signing in, then signing in from the card, created a sandbox owned by the fallback dev user (<code>desktop-dogfood</code>), not the signed-in user. Keep-alive, Copy SSH Command and Delete then got <code>not_found</code>. Likely cause: <code>getHostedSandboxConfig()</code> caches the token for the app process and does not reset it on sign-in (<code>src/main/hosted-sandbox/hosted-sandbox-config.ts</code>).",
    "<b>D-2 (bug, core desktop, high): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> Delete Workspace hides the row and reports success when the control plane answers <code>not_found</code>, while the VM keeps running (a billing leak).",
    "<b>D-3 (bug, core desktop): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> after a cloud create, the checkout at <code>/vercel/sandbox/project</code> is added as a new project named \"project\" instead of being grouped under orca-sandbox-push-test.",
    "<b>W3-dl (pass):</b> download over ONE persistent paired connection (the desktop lane): 50 MiB in 3.5 s (14.2 MiB/s from iad1 to this Mac, sha256 matches), 50 MiB folder zip in 5.4 s and complete. That matches W3's desktop numbers (50 MiB p50 15.3 MB/s; 200 MiB about 19 MiB/s; 2,000-file zip 1.8 s). The earlier 1.5 MiB/s came from the harness opening a new connection per RPC (corrected).",
    "<b>Remote desktop (pass):</b> the harness verifies the route and the ticket gate (101 with a ticket, 403 with a bogus one). W3 re-verified end to end with no manual steps: ticket mint p50 282 ms, connected in 130 ms, ticket reuse and a wrong Origin both 403. #138 (polish) and #146 (0-RTT forward open, p50 127 to 64 ms) ship with the next golden.",
    "<b>D-4 (minor, desktop): — fixed in #127 (bab8369a229); main-process paths verified headlessly by core against the dev control plane; desktop UI re-run pending (no-windows rule):</b> the \"Sign in to use Orca Cloud\" card stays after sign-in; the Ports panel says \"Port scan unavailable on unknown\" while waking; \"Move to this computer\" has no confirmation and copies rather than moves (the VM keeps running); the agent picker is disabled for Orca Cloud.",
    "<b>W6-1 (resolved):</b> provider keys are now egress placeholders in orcad's env (swap on by default on dev), so <code>sec.precedence</code> reads <code>AWS_REGION</code>. W6 confirmed the swap resolves keys with the same precedence as injection (team &gt; repo &gt; personal for agent keys) and added a test at the egress proxy (cloud/w6-precedence-test). Real Claude Code and Codex PRs: nwparker/orca-sandbox-push-test#5, #6.",
    "<b>Fixed since the last report:</b> F1 resume on a running sandbox now returns in 0.5 s (was a 60 s 502); F3 ai-gateway.vercel.sh is allowlisted (real Claude Code task passes); F4 the image now has orca, rg, tmux, vim, git-lfs and Chrome; F5 disk is 64 GB.",
    "<b>Note:</b> kill-switch drain mode is not exercised on the shared dev Worker (it would suspend other workstreams' sandboxes); refusal of creates and wakes is verified.",
    "<b>Stray sandbox resolved:</b> <code>desktop-dogfood</code>'s sbx_36ffc95c… was W2's w2predict sandbox, created by <code>~/w2-close/launch.sh</code> with ORCA_CLOUD_DEV_USER_ID=desktop-dogfood, and has been deleted."
  ],
  "gaps": [
    "Real-model agent run and subscription sign-in on a VM: to be proven in the user's own e2e session with their own credentials (no AI Gateway runs, per the user decision). Both rows stay capped at partial until then."
  ],
  "auditCaps": {
    "agent.real-task": "Proven only with moonshotai/kimi-k2 through the Claude Code CLI. Per the user decision (2026-11-10) there are no AI Gateway real-model runs; lifted by the user's own e2e session with their own credentials, and on the coordinator's e2e checklist. Image now also has Cursor, OpenCode, Gemini and Copilot (image.agents); W9 has shown Cursor and Gemini launch with their keys swapped at egress (agent.keys-swap), but no real-model run for those two.",
    "agent.signin": "Auth folders persist across sleep/wake and rollover, but subscription OAuth on a VM was never exercised. This cap is lifted by the user's own e2e session with their own credentials (after their review of #40228). It is on the coordinator's e2e checklist.",
    "agent.more-agents": "Owner-reported (gap-agents) real-task runs on kimi-k2 via the gateway. W9 is not re-running them (no gateway spend per user decision). Lifted by the user e2e session with their own credentials."
  }
}