mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Neil
6cd477a2f1
Test-only. No production code. ## The freeze repro was rotted in three ways, not one #16764 tracks four stale call sites. There were three separate problems: 1. **Stale call sites** — `execInTerminal` gained a `ptyId` and `splitActiveTerminalPane` gained a direction. (`startDockerSshRelayTarget`'s missing `testInfo` was the third; #18257 has since landed it on main.) 2. **It connected before session restore settled**, so the seeded tab never bound to a remote PTY and the terminal sat on "Connecting…" forever. 3. **It could never have passed, even once.** It waited for a one-shot `READY:` line through a 4000-char terminal window while its own 2 KB-every-8 ms flood buries that line within ~16 ms. Readiness is now keyed on the repeating `BG:` flood marker, which is strictly stronger — it proves the pane is streaming rather than merely started. It now runs end to end and prints a measurement instead of dying on a call site: ``` [freeze-repro R2] hiddenFloodMaxLagMs 2.1 bulkOpenMaxLagMs 41.5 interactionProbeMs 53.6 softFreeze false hardFreeze false ``` **It is still not CI-gateable, and the exclusion comment now says so.** The same spec on the same commit measured `bulkOpen 2575.6ms / interaction 3464.2ms` on a GitHub ubuntu runner against a 2500 ms soft budget — a ~60x spread on the number the budget reads, with the relay still streaming. That is the budget failing, not the product. The earlier draft of this comment claimed "repaired and passing", which was true only of the host it was measured on; gating this needs a host-relative oracle, not a bigger constant. ## New: a half-open link is judged, not wedged The fixture image has no `iptables` and the container has no `NET_ADMIN`, so `docker pause` is used instead — a harder case, because the container's TCP stack keeps ACKing: no FIN, no RST, and the socket looks perfectly healthy. Only an application-level probe can detect it. ``` [half-open] {"verdict":"reconnecting","verdictMs":25135,"budgetMs":90000} ``` Nothing in the suite covered the failure mode behind the "SSH hangs until I restart Orca" reports. ## New: resource accumulation measured on the remote host 6 terminals, then 5 reconnect cycles, counted on the container itself: ``` open: pts 1->6 (exactly 1/terminal), relay fds 25->30 (exactly 1/terminal) reconnect: pts flat at 6, relay procs flat at 1, node procs flat at 3 ``` `leakedMasterFdCount` is now **asserted**, not merely recorded. It counts PTY master fds held by non-relay processes: without `FD_CLOEXEC` a master is inherited by every later child, so terminal k adds k of them — the triangular signature measured as 15 across 5 terminals before the fix. #17914 patched the app and daemon and #17920 shipped the same patch to the relay host, and both are now on main, so the correct value is 0 and the probe holds it there: ``` baseline leakedMasterFdCount 0 6 terminals leakedMasterFdCount 0 (holders: only relay.js, n=6) reconnects leakedMasterFdCount 0 across all 5 cycles ``` Any growth here means the relay's node-pty rebuild did not take on that host, which is exactly what a remote-host probe exists to catch — and it is the half of #17914's claim that no unit test can reach. ## Routing Both new probes are claimed by `run-ssh-docker-e2e.mjs` (a Docker-gated spec no runner names self-skips everywhere and still reports green) **and** by the `ssh-terminal-source` route in `pr-e2e-source-routing.mjs`, so they run when the relay and SSH code they guard changes rather than only on a scheduled lane.
116 lines
6.2 KiB
JavaScript
116 lines
6.2 KiB
JavaScript
import { spawnSync } from 'node:child_process'
|
|
|
|
const rawExtraArgs = process.argv.slice(2)
|
|
const extraArgs = rawExtraArgs[0] === '--' ? rawExtraArgs.slice(1) : rawExtraArgs
|
|
const pnpm = process.platform === 'win32' ? 'pnpm.cmd' : 'pnpm'
|
|
const env = {
|
|
...process.env,
|
|
ORCA_E2E_SSH_DOCKER: '1',
|
|
ORCA_E2E_WEB_CLIENT: '1'
|
|
}
|
|
|
|
// Why: Node's CVE-2024-27980 hardening rejects .cmd spawns without shell on Windows.
|
|
const spawnOptions = {
|
|
stdio: 'inherit',
|
|
env,
|
|
shell: process.platform === 'win32'
|
|
}
|
|
|
|
const runtime = spawnSync(pnpm, ['run', 'ensure:electron-runtime'], spawnOptions)
|
|
|
|
if (runtime.status !== 0) {
|
|
process.exit(runtime.status ?? 1)
|
|
}
|
|
|
|
// Why one explicit list: these specs self-skip without ORCA_E2E_SSH_DOCKER and no sharded lane
|
|
// sets it, so a spec in no runner runs nowhere. The gate contract proves every flag-reading
|
|
// spec is claimed here, by the watcher-isolation or parking runner, or by a listed exclusion.
|
|
//
|
|
// Deliberately absent, and therefore still covered by no CI trigger:
|
|
// ssh-docker-relay-perf.spec.ts — wall-clock latency thresholds; flaky budgets here would
|
|
// cost the lane its credibility. NOTE: a runner script test:e2e:ssh-docker-perf exists in
|
|
// package.json but NO workflow invokes it, so this spec currently runs in no CI lane at
|
|
// all. Recorded as a real gap, not as coverage living somewhere else.
|
|
// ssh-codex-display-artifacts-repro.spec.ts — installs a real remote codex binary that CI
|
|
// runners do not have (observed as `spawn codex ENOENT`). Runs in no CI lane at all.
|
|
// ssh-docker-bulk-open-freeze-repro.spec.ts — un-rotted and now measurable, and marked
|
|
// `test.fixme` because its oracle cannot gate. Absent from this list AND skipped, so the
|
|
// two cannot drift: it is also reachable from the changed-specs lane whenever the spec
|
|
// itself is edited, and a wall-clock oracle that fails there is worth no more than one
|
|
// that fails here.
|
|
// The rot (#16764) is fixed: the stale call sites are repaired, it connects after session
|
|
// restore instead of before, and readiness keys on the repeating flood marker rather than
|
|
// a one-shot READY line the flood buries within ~16ms. It runs end to end and prints a
|
|
// measurement instead of dying on a call site.
|
|
// What it is NOT is portable. Three runs of the same measurement path:
|
|
// developer workstation: hiddenFlood 2.1ms bulkOpen 41.5ms interaction 53.6ms
|
|
// GitHub ubuntu runner A: hiddenFlood 1.5ms bulkOpen 2575.6ms interaction 3464.2ms
|
|
// GitHub ubuntu runner B: hiddenFlood 0.2ms bulkOpen 397.4ms interaction 3386.7ms
|
|
// bulkOpen swings 6.5x between two CI runs of the same code, so a fixed threshold on it is
|
|
// a coin flip; interaction sits stably ~64x over the workstation figure because it times a
|
|
// view remount, not the renderer freeze the issue reports, and only shares the budget
|
|
// constant because both are milliseconds. Every failure so far is the soft budget; hard
|
|
// has never tripped, and the relay was still streaming each time — the budget failed, not
|
|
// the product. Same rule as ssh-docker-relay-perf above. Gating needs a distribution
|
|
// first, then a host-relative oracle; a bigger constant, or a ratio picked from three
|
|
// samples, is the same arbitrary number in different clothes.
|
|
// COVERAGE GAP, recorded as such: 5 simultaneously flooding SSH panes exercise writer
|
|
// saturation, ACK/credit accounting and per-pane polling together, and nothing else covers
|
|
// that combination. Flip `test.fixme` back to `test` to run it. Tracked in
|
|
// stablyai/orca#16764.
|
|
//
|
|
// Why both projects: ssh-port-forward-lifecycle is @headful, which the headless project
|
|
// grep-inverts away.
|
|
//
|
|
// Known gaps in SSH e2e coverage, recorded here because nothing else names them:
|
|
// - The job that runs this is still called `ssh-docker-watcher-isolation`, though watcher
|
|
// isolation is now one spec of many. Renaming it changes the GitHub check name and can
|
|
// break required-check config, so the name understates the job on purpose.
|
|
// - E2E does not gate merges: `verify.needs` in pr.yml omits `e2e` while the suite is red on
|
|
// main. Nothing in this lane blocks a PR yet. pr.yml's Require-successful-checks comment
|
|
// has the exact wiring to flip it, and the gate contract asserts the current state.
|
|
// - Five specs and one unit test are gated on env vars no workflow sets, so they run nowhere
|
|
// and are not Docker-gated, which puts them outside this file's contract:
|
|
// local-ssh-browser-routing (ORCA_E2E_LOCAL_SSH_BROWSER)
|
|
// ssh-client-hosted-browser-drop-reconnect (ORCA_E2E_SSH_CLIENT_HOSTED_BROWSER)
|
|
// nested-runtime-ssh-lifecycle, nested-runtime-ssh-routing (ORCA_E2E_NESTED_RUNTIME_SSH)
|
|
// ssh-localhost (ORCA_E2E_SSH_LOCALHOST)
|
|
// ssh-browser-network-execution-route.docker.unit.test.ts (ORCA_RUN_DOCKER_SSH_BROWSER_E2E)
|
|
// Runner scripts for the first four sit unused in package.json; no workflow calls them.
|
|
const result = spawnSync(
|
|
pnpm,
|
|
[
|
|
'exec',
|
|
'playwright',
|
|
'test',
|
|
'tests/e2e/pty-input-write-queue-ssh.spec.ts',
|
|
'tests/e2e/ssh-ai-vault-session-history.spec.ts',
|
|
'tests/e2e/ssh-cold-activation-restore.spec.ts',
|
|
'tests/e2e/ssh-cold-hydration-gap-tab-seeding.spec.ts',
|
|
'tests/e2e/ssh-docker-half-open-link.spec.ts',
|
|
'tests/e2e/ssh-docker-quick-open-large-listing.spec.ts',
|
|
'tests/e2e/ssh-docker-reconnect-pane-restore.spec.ts',
|
|
'tests/e2e/ssh-docker-resource-accumulation.spec.ts',
|
|
'tests/e2e/ssh-docker-transport-drop-recovery.spec.ts',
|
|
'tests/e2e/ssh-external-image-preview.spec.ts',
|
|
'tests/e2e/ssh-lost-kill-tab-resurrection.spec.ts',
|
|
'tests/e2e/ssh-pi-compatible-agent-title.spec.ts',
|
|
'tests/e2e/ssh-port-forward-lifecycle.spec.ts',
|
|
'tests/e2e/ssh-reconnect-tab-destruction.spec.ts',
|
|
'tests/e2e/ssh-restart-tab-accumulation.spec.ts',
|
|
'tests/e2e/ssh-skill-installation.spec.ts',
|
|
'tests/e2e/ssh-terminal-window-wake-stale-grid-repro.spec.ts',
|
|
'--config',
|
|
'tests/playwright.config.ts',
|
|
'--project',
|
|
'electron-headless',
|
|
'--project',
|
|
'electron-headful',
|
|
'--workers=1',
|
|
...extraArgs
|
|
],
|
|
spawnOptions
|
|
)
|
|
|
|
process.exit(result.status ?? 1)
|