18 Commits

Include renames

Author	SHA1	Message	Date
Neil	7ea01279cd	feat(search): bundle ripgrep for local, WSL, and SSH search (#22396) ... * feat(search): bundle ripgrep for local, WSL, and SSH search Ship @vscode/ripgrep-universal's prebuilt rg for all six relay platforms in every desktop artifact. Local and WSL searches spawn the bundled binary and drop the git ls-files / git grep fallbacks; SSH deploys upload the remote's binary once per ripgrep version and the relay prefers it over PATH rg. * fix(search): address bundled ripgrep review findings - Key the SSH ripgrep cache on the binary's content hash; a package bump is the only update step - glibc verifier: read arch tokens below the slice root and accept static ELFs (arm64 release blocker) - Ship ripgrep/PCRE2/musl license notices; bundle rg with orcad - Packaged builds never spawn a bare rg; report fd pressure as transient - SSH: install rg before sweep/GC, size-validate installs, back off instead of disabling on launch failure - Scope Dependabot to @vscode/ripgrep-universal; revert unrelated lockfile churn * chore(search): drop bundled-ripgrep reference doc; assert full packaging layout parity * refactor(search): one entry point for spawning the bundled ripgrep Local Quick Open, Quick Open path search, the Explorer name filter, and runtime text search each repeated the same three steps: resolve the bundled command, spread in the WSL distro, spread in the WSL shell expression. Fold that into spawnBundledRipgrep so one place owns the rule that a bare 'rg' must never reach spawn, and simplify the resolver's command/packaged checks. Restore the AGENTS.md ripgrep rule dropped alongside its reference doc in 63f4dac, and note why the relay's availability probe may spawn a bare 'rg'. No behaviour change; verified by the existing suites plus a new test that pins the local, WSL-routed, and distro-routed-but-Windows-output cases. * refactor(search): drop the local install-ripgrep path; enforce the rg rule Bundling rg removed the local git/readdir fallback, so nothing can produce the "install ripgrep on the host running the Quick Open scan" guidance any more -- only a remote host an upload never reached still reaches the capped listing. Drop the host parameter, the renderer's local branch and its translation key, and the relay wrapper that existed only to pass 'remote'. Add a ratchet test for bare 'rg' spawns, since the AGENTS.md rule alone had nothing enforcing it. Its one allowlist entry is the relay's PATH probe, which asks about PATH by definition. Verified the guard catches a planted offender rather than passing vacuously. Also stop chaining the remote cleanup sweep behind the ripgrep upload: on a cold host that is a multi-MB transfer, and stale upload stages and superseded version dirs were left on the remote for its whole duration. The two touch different trees, so they now run concurrently. * test(ssh): pin that the cleanup sweep does not wait on the ripgrep upload * fix(search): derive rg spawn types instead of importing node:child_process A type-only import still counts against the child_process ratchet, whose pin and allowlist only ever shrink. Derive both types from wslAwareSpawn instead. * fix(search): surface an unreachable WSL workspace instead of an empty result Inside `bash -c`, a failed `cd` exits 1 -- the same code ripgrep uses for "no matches" -- so a WSL workspace whose directory had gone away reported an empty listing as a successful scan. main did not have this hole: checkRgAvailable ran the same `cd` wrapper first and settled on `code === 0`, diverting to the git fallback that this PR deletes. The WSL wrapper now takes an optional cwdFailureExitCode; rg passes 97, and all four close handlers reject with a clear error before the unavailable check can blame the install. Also from review: - Bound the fire-and-forget ripgrep upload with deploySignal. The controller aborts only on the deploy timeout, never on success, so this cancels a still-running upload when the deploy gives up. - Run the stale-stage sweep before the installed check rather than inside its else branch. Once rg was installed every later deploy took the PRESENT path, so a stage orphaned by a dropped connection was never collected again. - Note in orcad-remote-deploy.ts why wiring it up needs ripgrep work first: build-orcad.mjs copies only the build host's rg, and orcad reports isPackaged() === true, so a remote of another platform would find nothing. ssh-relay-deploy.test.ts sat at the max-lines cap, so any edit to it failed the gate. Split the four Windows named-pipe deploys into their own file (926 -> 737 + 333); both are now well clear of it. * fix(search): name the unreachable root in every handler, not three of four Round-two review caught that the missing-cwd branch in scanRipgrepPaths sat AFTER isRipgrepUnavailableExit, which classifies any code above 2 as a broken install -- so for exit 97 it was dead code and Quick Open still told the user to reinstall Orca. Reordered; all four handlers now check it first. Also from review: - A vanished workspace makes spawn fail with ENOENT, which read as a damaged install on every local path. Confirm the cwd with isRipgrepSpawnCwdUsable -- the guard the relay already applies -- before blaming the binary. The async continuation re-checks `resolved`, because finish() drops its argument once settled and the rejected promise would otherwise go unhandled. - bundledRipgrepCommand returned a bare 'rg' for an arch outside the bundled set, bypassing the guard that exists so Windows cannot resolve a bare name against the repo cwd. A packaged app now always names an absolute path. Drop ci-shards/unit-assignment.json, a 9,425-line CI artifact swept in from reproducing a shard locally, and gitignore the directory that produced it. The "rg genuinely cannot start" test pointed at a synthetic /repo, which the new guard correctly reports as unreachable; it now resolves to a real root so it still tests what its name says. * fix(search): let the error handler own the spawn-failure verdict A failed spawn emits 'error' and THEN 'close' with a negative code. The cwd check added in the error handler did not settle, so the close handler settled first -- synchronously, with the reinstall message -- and won the race every time. The branch was not merely flaky, it was unreachable in all four handlers: it is guarded by pid === undefined, which is exactly the case that always produces a following close(code < 0). Verified against a real spawn: 3/3 runs give error(ENOENT) -> close(-2). The error handler now detaches 'close' before the probe, so it owns the outcome. The probe also had no rejection handler, so a probe that rejected left the search unsettled forever -- a hang, not just a wrong message. It now falls back to the prior verdict rather than inventing one. Tests: filesystem-search-rg-timeout and orca-runtime-files-search already cover error-first and close-first, but against synthetic roots that the new guard correctly calls unreachable; they now resolve to a real root, keeping each test's stated intent. Added a Quick Open case for the vanished-workspace path and confirmed it fails with the old ordering. * test(search): cover exit code 97 in all four ripgrep close handlers Round-four review found the missing-cwd branch had zero handler coverage: no test anywhere emitted close(97), only -2/0/1/2/127. Ordering was correct, but guarded by source-line order alone -- and that exact ordering was wrong in three of four handlers two commits ago. Each suite now drives close(97) through its real handler and expects the unreachable-root message. Verified the tests earn their place: neutering the missing-cwd check fails exactly four tests, one per handler. Also drop a Reflect.get the anti-slop gate rejects, in favour of `in` narrowing. * docs(search): stop claiming the close handler always wins the race The previous commit asserted close "would beat this threadpool round-trip every time", from an n=3 sample that measured event ordering -- which was never in dispute -- rather than probe-vs-close. Two later measurements disagree with each other: 50/50 close-first here, 30/50 probe-first in review. Either way it is a race on a sub-millisecond margin, and the detach is what makes the verdict deterministic. Why this wording matters: "close wins every time" is an argument for deleting the detach as a guard against an impossible race. No test would catch that -- the suites emit error and close in the same synchronous tick. * chore(search): ship the jemalloc and libunwind notices the Linux rg needs The statically linked Linux builds carry jemalloc (BSD-2-Clause) and LLVM libunwind (Apache-2.0 WITH LLVM-exception) in addition to PCRE2 and musl, and both require their notice on binary redistribution. Confirmed with `strings`: their symbols are present in linux-x64 and linux-arm64 and absent from the darwin and win32 builds. Texts taken from the upstream canonical sources. extraResources already copies the whole licenses directory, so these ship without a packaging change. * fix(relay): stop spawning a bare rg, name unreachable roots, collect old builds Three gaps the reviews surfaced on the remote side, all pre-existing on main. Bare `rg` on Windows remotes. Both relay spawn sites pass the user's repo as cwd, and CreateProcessW searches the cwd before PATH -- the same hijack the desktop side already fixes. The relay now walks PATH itself and spawns an absolute rg.exe, skipping relative PATH entries because those resolve against the cwd. No rg on PATH yields null, which callers treat as "ripgrep unavailable" rather than handing spawn a bare name. POSIX keeps the bare name: execvp never consults the cwd, so there is nothing to resolve and nothing to gain. With the last probe converted, the bare-spawn ratchet allowlist is empty. Empty results for an unreachable root. settleLaunchFailure resolved an empty, successful-looking scan when the root was gone but PATH rg existed, and the git/readdir chain never engaged because it only triggers on RipgrepUnavailableError. Both relay paths now reject naming the root, matching local workspaces. Missing-rg keeps precedence over a missing root, because only that verdict engages the fallback chain -- two tests pinned that deliberately and it would have been wrong to flip it. Unbounded ~/.orca-remote/ripgrep/. Nothing collected this tree; the relay's version GC only matches `relay-*`, so every rg bump left another ~5 MB per host forever. The probe command now also drops sibling builds older than two weeks, sparing the current one and live upload stages, on POSIX and PowerShell alike. Two weeks because a client pinned to an older build may still be using it; the cost of collecting one early is that client re-uploading once. * fix(relay): probe the rg that failed, and close the drive-relative PATH hole Five review findings against the previous commit, all reproduced first. The launch-failure classifier probed PATH rg, but the spawn that failed was the bundled binary. On the normal remote setup -- no rg on PATH, which is why Orca uploads one -- the probe failed and a moved workspace was reported as a missing ripgrep, telling the user to install what Orca already ships. So the fix was inert on exactly the hosts the uploader exists for. It now takes a candidate list and asks the binary that actually failed first, then PATH. path.win32.isAbsolute accepts `\tools` and `/tools`: rooted, but carrying no drive, so they resolve against whatever drive the process is on. The probe would have validated one against the relay's drive while the spawn, running with the user's repo as cwd, resolved it against the repo's -- the same cwd-dependence this lookup removes, narrowed from directory to drive. A real drive letter or UNC root is now required. probeRipgrepVersion had lost the timeout's kill in the rewrite, leaking a live process and a ref'd handle per launch failure -- for a hang, which is the very case the bundled-rg back-off exists for. It also spawned without windowsHide, which would flash a console; fixing that made an allowlist entry stale, so the entry is gone and the pin ratchets down 63 -> 62. `windowsPathRipgrep ??= …` never memoised a miss, because null is nullish. The caching was inverted against cost: a hit stops at the first directory, a miss stats every one, and only the miss was repeated -- per spawn. The bare-spawn ratchet claimed "nothing in production spawns a bare rg", which is false on POSIX. It now also matches PATH_RIPGREP_COMMAND at a spawn site, and the comment states plainly what a textual guard cannot see: the POSIX bare name reaches spawn as a parameter, and is safe because execvp ignores the cwd. The drive-rooted predicate is tested directly rather than through the filesystem -- a temp dir on a POSIX CI host has no drive letter to exercise win32 semantics with, so the filesystem test could never have caught this. * test(mobile): repin the session closure past #22452's two shared modules Merging main brought the closure to 4220 against a pin of 4218. The two extra modules are `src/shared/agent-turn-outcome.ts` and `src/shared/main-agent-status.ts` from #22452, which the status projection this route already reaches import. That change was src/shared-only, so the mobile job never ran on it -- the same way the structured tool line slipped past, as the ledger above already records. Repinned here because this PR's file set is what next made the job run, not because this PR reaches either module. Verified: of the 28 source files this branch changes, none appear anywhere in the route's 4220-module closure. * fix(search): preserve remote binaries and complete runtime packaging * test(relay): pin the probe's env now that it inherits the relay's PATH 8d6759a threaded the relay env into probeRipgrepVersion -- correctly, since the probe decides whether a launch failure was the binary or the root and so has to resolve the same rg the failed spawn would have. It left the assertion that pins the probe's spawn arguments behind, which is what CI caught. Asserting buildRelayCommandEnv() rather than loosening the match to any object: under process.env the probe could resolve a different rg, or none, which is the regression the change exists to prevent. * feat(ssh): collect remote ripgrep builds by reference, not by age Nothing collected `~/.orca-remote/ripgrep/`: the version GC matches only `relay-*`, so every change to the shipped bytes left another ~5 MB on every SSH host, permanently. The age window this replaces was the wrong instrument -- a directory's mtime is when it was written, not when it was last used, so it cannot tell a superseded build from the one a live relay was launched against. Deleting the latter is not graceful degradation: without a PATH ripgrep remote text search rejects outright, and listing drops to the capped walk this PR exists to remove. So the question is reference. Each relay directory now records the build it runs against in `.ripgrep-ref`, written only once that binary is confirmed present, and the GC collects a build only when no installation names it. The discipline is ssh-relay-native-deps-cache-gc.ts': anything the pass cannot account for blocks the whole pass. A relay directory with no readable marker is an older Orca's, possibly running right now against a binary it never recorded, so the pass declines rather than guessing. Those directories are removed by the version GC in time, which is what makes their builds collectable -- hence running after it, not beside it. Deletion is the same tombstone, recheck under the rename, then remove, so a deploy that takes a reference mid-pass gets its tree restored. Windows has no pass yet, matching the native-deps cache's gate. One test note: the first version of the "unaccountable blocks the pass" test passed against a deliberately broken guard, because the tombstone recheck masked its absence. The test now puts a readable recheck behind an unreadable first scan, which is the only shape that fails when that guard is removed. Recording the reference lives inside ensureRemoteBundledRipgrep rather than at the call site: it is the same concern, and it keeps the deploy's ripgrep surface to one call for the tests that mock it to protect their exec queues. * feat(ssh): collect Windows remotes too, and ship the Rust crate notices Three items previously left documented-but-open. Windows remote accumulation. The cache GC was POSIX-gated, so the leak did not go away -- it moved to the platform with the larger binary (rg.exe is 5.43 MB on win32-x64, against 4.77 MB for linux-arm64). The PowerShell dialect now does the same reference scan: entries and references carry token prefixes, because PowerShell writes every uncaptured value to stdout and an untokenised listing would feed Remove-Item whatever a cmdlet happened to emit. Verified on a real Windows host rather than a mock: the listing emits its ENTRY/LIST_OK tokens, a relay directory carrying a marker yields REF <entry>, and a relay directory without one yields REFS_ERR -- the safety path, on the real interpreter. Rust crate notices. The crate set was read out of the shipped binary's symbols and the licence identifiers taken from crates.io rather than assumed. Where a crate offers the Unlicense, Orca elects it: a public-domain dedication carries no notice obligation, and that covers eight of them. The four that do not offer it get their MIT text reproduced. encoding_rs carries a BSD-3-Clause notice for its WHATWG-derived encoding data that is joined by AND, not OR, so electing MIT does not discharge it. Release-only validation, corrected rather than repeated. Linux AppImage/deb/rpm already runs in CI's package job on every PR, and Windows signing was already rehearsed on this branch. macOS notarization is the only item a release must still exercise, and the exposure is narrow: notarization requires signatures on Mach-O binaries, and of the six bundled builds only the two darwin ones are Mach-O -- `file` reports ELF for linux and PE32+ for win32 -- so signIgnore excludes only files the notary never asks about. orcad-artifacts.test.ts caught the new notice file missing from the standalone runtime's shipped list, which is exactly the gap that test exists to catch: a notice committed to the repo but never actually shipped. * fix(search): protect relay cache references and handle failed spawns * fix(ripgrep): close review gaps and repair deployment fixtures * test(mobile): refresh merged session module census * fix(ssh): preserve ripgrep caches with empty legacy references * test(mobile): assert bundle boundaries instead of global module count	2026-09-24 17:25:48 -07:00
Neil	d35e1dcba8	ci: route pane close and retirement changes to the close specs (#21572) ... Every terminal-pane route named what BINDS a pane — the pty transports, the ssh reconnect ledgers, the park watchers. None named what unbinds one. #21005 changed the pane close and retirement lifecycle across three transports, and replaying its fifteen paths through the selector returns [] with --reusable-workflow false: it merged with E2E skipped outright. #21001, the same seam a week earlier, ran E2E only because it happened to also touch an ssh-named file, so the close specs were never selected even then. Unbinding is the half that strands a PTY or leaves a retired leaf mounted as a blank pane, so the three specs that judge it now gate it: the parked-tab close retirement, the split-pane close layout consistency, and the paired client's view of a leaf the host retired. Scope held deliberately narrow. The route is not an SSH source route, so it does not start a Docker relay; close reaches SSH only through the shared provider the non-Docker specs already cover. runtime-rpc-client.ts is left out although #21005 touched it: it carries no close decision and churns about three times as often as these files, so routing on it would run this lane on unrelated runtime work. Replaying twenty merged PRs shows exactly one selection change, #21001.	2026-09-18 23:26:01 -07:00
Neil	066b4951b9	fix(terminal): keep a split's real direction when the leaf set moves (#21294) ... resolveTerminalLayoutRoot discarded any known tree that did not cover the published leaf set exactly and rebuilt the tab as a flat chain with a guessed 'horizontal' direction, restacking side-by-side panes. The guess is then published, mirrored to every paired client, and written back over the real tree, so the direction is gone from disk. Prune a known tree to the leaves that survive and graft only the leaves no tree places, which is now the sole place a direction is invented and is still reported through onSynthesize.	2026-09-17 21:21:01 -07:00
Neil	1e301ab1df	test: cover native Wayland Hangul in isolated CI (#19174) ... * test: exercise native Wayland Hangul in isolated CI session * test: wait for nested compositor socket before selecting IBus * test: align Wayland IBus discovery with GNOME environment filtering * test: assert Wayland launch and register native Hangul evidence	2026-09-06 19:02:54 -07:00
Neil	2e8fa3fe9b	test: exercise packaged browser compatibility in scheduled CI (#19157) ... * test: exercise packaged browser compatibility in scheduled CI * test: record final packaged workflow participation evidence * test: expose manual packaged revision and simplify executable check * test: reject missing package checksum assertion	2026-09-06 17:42:49 -07:00
Jinwoo Hong	0c33f58e8a	fix(ssh-relay): daemon owns the endpoint credential; a losing start never rotates it (#19052) ... <!-- orca-pr-loc --> <!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. --> | | Files | Added | Deleted | Net | | :--- | ---: | ---: | ---: | ---: | | Test | 19 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​962 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​136 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​826 | | Prod | 18 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​295 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​116 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​179 | <!-- /orca-pr-loc --> ## Symptom Live 2026-09-05 (Orca 1.4.198 client, Ubuntu host): both relay processes `kill -STOP`ped for 20 s, then `-CONT`. The client redeployed while the host was frozen. Its fresh daemon lost the socket bind (`Socket path already in use`) but had **already rewritten** `relay-<id>.sock.credential`. The surviving daemon kept its in-memory credential, so every later `--connect` got `Endpoint credential mismatch; closing socket`, then `Grace started … timeoutMs=0 … ptys=1, clients=0` every ~20 s, forever. Only a manual `kill -TERM` cleared it. Receipts: `review-archive/orchestration-v3-pr16904/smoke-receipts-t012b/E16,E17,E18,E24`. Three independent defects kept the wedge alive; each is fixed at its own seam. ## Fix **1. The relay daemon owns credential publication (race-free under two concurrent starters).** `relay-daemon.ts` binds the socket first, then publishes via the new `src/relay/relay-endpoint-credential-publication.ts`: adopt a valid pre-existing file (older clients still pre-write), else mint 32 random bytes and write temp+rename at 0600. A start that loses the bind exits inside `listen()` and never reaches the file. Why this option and not restore-on-loss or a client-side write: the only process that can *prove* ownership is the one whose `listen()` succeeded, and that proof is atomic with the bind. The client-side pre-write (`ssh-relay-endpoint-credential.ts`) and the launch-command `chmod 600`/`icacls` are removed on POSIX and Windows. The racing test also exposed that macOS reports a mid-bind collision as `EEXIST` rather than `EADDRINUSE`; `relay-socket-ownership.ts` now treats both as "held or stale". **2. The client distinguishes "no daemon" from "daemon present but not answering", and never rewrites.** A credential refusal is now typed on the wire: the daemon replies `orca-relay-handshake-credential-mismatch` (same frame type, no new opcode) and the bridge exits **43**; `waitForSentinel` maps it to `RelayCredentialMismatchError`, which the takeover treats as handshake-refusal evidence exactly like exit 42. A relay that holds the endpoint but **never refused** (the stalled-host shape: kernel backlog accepts the probe, handshake gets no answer) is now `RelayEndpointUnresponsiveError`, routed to the relay-lost backoff instead of the terminal Reset Relay path. Silence is not a decision (`docs/reference/ssh-execution-boundary.md`). **2b. Deploy honours the verdict.** The 40 s live run exposed that the `--connect` catch block in `deployAndLaunchRelay` predates the incumbent probe and swallowed both verdicts as "probe failed, launch fresh", so a fresh daemon was still launched over the live one (it lost the bind by luck, which is exactly the collision in the incident). Held and Unresponsive now propagate; the session backs off on Unresponsive and surfaces Reset Relay on Held. Red-first in `ssh-relay-deploy-incumbent-verdict.test.ts`. **3. The daemon cannot be wedged by a rotated file, because nothing can rotate it.** The credential lives in the content-hashed relay dir, and after (1) the only writer is the daemon that owns the socket, so the "file changed under a live daemon" state the incident depended on is no longer reachable in-product. The credential is therefore fixed for the daemon's lifetime, as a plain secret should be. A hand-edited file is refused with the typed reply until restored (tested). Startup adoption of a pre-written file applies an owner-only + same-uid rule (review finding): anything else is replaced by a fresh mint. An earlier revision of this PR also re-read the file on mismatch and adopted it; that was removed as unreachable machinery that turned the credential into a per-handshake file-ownership check. **3b. Fail closed between bind and publication.** A client that arrives after `listen()` resolves but before the credential is set is refused, not admitted as `unproved`. Nothing can be delivered in that window today; the guard makes the boundary structural instead of an event-loop ordering fact. Red-first in `relay-reconnect-listener-credential-gate.test.ts`. **Wire compat.** New optional handshake reply only; an old `--connect` hits `Unknown handshake type` and exits 1 pre-sentinel, which it already treated as a generic failure. New daemon adopts an old client's pre-written file; new client still passes `--credential-file` so an old daemon reads it as before. Absence of exit 43 is never used as evidence. **Also.** `terminal create` on a reconnecting SSH host now says what to do instead of a bare `No PTY provider for connection "<id>"` (prefix preserved; the renderer matches it). ## Tests (red first) - `src/relay/subprocess.test.ts`: two `--detached` starts race one socket + credential file → exactly one reaches the sentinel, loser exits 1 with `Socket path already in use`, file valid + 0600, a `--connect` reading it reaches `relay.status` and reports the winner's pid. Red before (both starters died: daemon required a pre-existing file), green 6/6 after. - `src/relay/relay-endpoint-credential-publication.test.ts`: mints after bind; adopts a pre-written 0600 file; replaces a pre-written 0644 file with a fresh mint; refuses a stale credential with exit 43 while still serving the real one, and keeps refusing a rewritten file until it is restored. - `src/relay/relay-reconnect-listener-credential-gate.test.ts`: a client in the bind-to-publish window is refused and never attached; after publication the right credential is accepted and a wrong one refused; a daemon launched without a credential file is not gated. Red without the guard. - `ssh-relay-deploy-incumbent-verdict.test.ts`: live-but-silent incumbent → `RelayEndpointUnresponsiveError`, refused → `RelayEndpointHeldError`, and in neither case is `--detached` launched; a failed `test -S` probe still launches fresh. Red 2/3 without the deploy change. - `ssh-relay-deploy-helpers.test.ts` (exit 43), `ssh-relay-endpoint-takeover.test.ts` (refused → Held even with no `lsof`; silent → Unresponsive, nothing unlinked or signalled), `ssh-relay-session-terminal-error.test.ts` (Unresponsive → `onRelayLost`, not terminal). Deploy/namespace/native-deps tests updated to assert the client writes **no** credential. ## Live proof New `tests/e2e/ssh-docker-relay-stall-credential.spec.ts` (claimed in `run-ssh-docker-e2e.mjs` and PR source routing), two cases: `kill -STOP` every relay pid in the container, send input during the freeze, hold **20 s** (the incident's duration, which races the mux liveness timeout) or **40 s** (past it for sure), `kill -CONT`; assert status back to `connected`, same pty, same daemon pid, same credential inode and content, relay.log did not shrink (a relaunch truncates it) and has zero `Endpoint credential mismatch` / `Socket path already in use` lines, in-stall input delivered at most once. Run output (local, fixture image `orca-e2e-ssh-relay:3a864c665ba2cefd`, `ORCA_E2E_SSH_DOCKER=1 SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 … --project electron-headless --workers=1`, head `c2c20fd994`; re-run identically on the final head after the credential-lifetime change, 2 passed (1.7m), same annotations, and the bind-to-publish refusal never fired): ``` ✓ keeps the same daemon and credential across a 20s relay freeze (38.3s) relay-processes-stopped: 2 relay-processes-continued: 2 bridge-pids-before-after: 480 -> 480 socket-clients-accepted-before-after: 1 -> 1 in-stall-input-delivered: 1 ✓ backs off and reattaches, never relaunching, across a 40s relay freeze (57.5s) relay-processes-stopped: 2 relay-processes-continued: 4 bridge-pids-before-after: 480 -> 1202 socket-clients-accepted-before-after: 1 -> 3 in-stall-input-delivered: 1 2 passed (1.6m) ``` Client log in the 40 s case shows the new path end to end: `Relay channel lost … reconnect attempt 1/6` → `Socket probe result: "ALIVE"` → `Socket reconnect failed … Relay failed to start within 10s` → `Relay endpoint incumbent: … verdict=live evidence=accepted-connection holders=unenumerable` → `Failed to re-establish relay … A relay still owns … but did not answer the handshake … Orca will retry` → `reconnect attempt 2/6` → `Reconnected to existing relay via socket`. The 20 s case never left the frozen bridge (same bridge pid, one accept), so it exercises the "silence is not death" side of the same race. The 20 s case passed 6/6 across the session; the 40 s case was red on the prior head (`Socket path already in use` + `Startup failed: listen EADDRINUSE` in relay.log from the swallowed verdict) and is green after 2b. Before the fix the same injection produced a fresh daemon that rewrote the credential and a survivor refusing every client. The `relay-processes-continued` count exceeds `stopped` in the 40 s case because the timed-out client's `--connect` bridge and the loser-side processes are parked behind the frozen listener when `CONT` runs; they exit on their own once it resumes. ## Gates `pnpm test src/relay src/main/ssh` 332 files / 3884 tests pass · `pnpm typecheck:tsc:node` clean · `check:code-quality:changed` 0 findings · `check:react-doctor:changed` 0 findings · `pr-e2e-gate-contract.test.mjs` 42 pass · no lint disables or max-lines bumps added. ## Noted, not fixed here - `terminal list` `orphaned:false` / `terminal close` `ptyKilled:true` for a pane whose relay is gone (`orca-runtime-stop-explicitly-closed-tab-ptys.ts`): different seam, `@ts-nocheck` characterization-covered file. - On a host with no `lsof`, a stalled relay still cannot be enumerated as the holder; it is now retried rather than declared held, but a relay frozen past the backoff budget still ends in the existing "reconnect manually" banner.	2026-09-06 14:39:25 -04:00
Neil	3be526c5e6	test: cover SSH reattach replay and enable deterministic Codex CI (#19106) ... * test: cover SSH replay replies and run deterministic Codex restore scenarios * test: register replay probe unit command in reliability gate	2026-09-06 11:25:58 -07:00
Neil	4d9e963ffd	test: enable localhost SSH terminal and hook journey in CI (#19097) ... * test: run localhost SSH terminal and hooks in CI * test: isolate localhost SSH session fixtures across repetitions * test: route remote agent hook source changes to localhost journey * test: record localhost SSH reliability evidence and remaining gaps * test: route the real SSH session hook authority	2026-09-06 10:05:26 -07:00
Neil	f5960cec00	test: enable Docker SSH browser network route coverage in CI (#19095) ... * test: enable Docker SSH browser network route journeys in CI * test: register Docker browser job in token permissions contract * test: declare SSH client dependency and narrow browser fixture routing	2026-09-06 09:29:56 -07:00
Neil	f952f1ac96	test: run real WSL terminal launch and paste in PR CI (#19072) ... * test: continuously exercise real WSL terminal launch and paste * test: establish live WSL reader before changing default shell * ci: pin WSL kernel installer and participation selectors * ci: route deleted WSL paths and record immutable run evidence * test: require exactly three WSL repetitions in lane contract	2026-09-06 05:24:54 -07:00
Neil	6031c19e9f	ci: reduce dependency, checkout, and test deadline overhead (#18968) ... * ci: reduce dependency, checkout, and test deadline overhead * ci: avoid generic E2E jobs for native-only IME changes	2026-09-05 18:20:12 -07:00
Neil	6cd477a2f1	test(e2e): un-rot the SSH freeze repro and probe two failure modes nothing covered (#17940) ... Test-only. No production code. ## The freeze repro was rotted in three ways, not one #16764 tracks four stale call sites. There were three separate problems: 1. **Stale call sites** — `execInTerminal` gained a `ptyId` and `splitActiveTerminalPane` gained a direction. (`startDockerSshRelayTarget`'s missing `testInfo` was the third; #18257 has since landed it on main.) 2. **It connected before session restore settled**, so the seeded tab never bound to a remote PTY and the terminal sat on "Connecting…" forever. 3. **It could never have passed, even once.** It waited for a one-shot `READY:` line through a 4000-char terminal window while its own 2 KB-every-8 ms flood buries that line within ~16 ms. Readiness is now keyed on the repeating `BG:` flood marker, which is strictly stronger — it proves the pane is streaming rather than merely started. It now runs end to end and prints a measurement instead of dying on a call site: ``` [freeze-repro R2] hiddenFloodMaxLagMs 2.1 bulkOpenMaxLagMs 41.5 interactionProbeMs 53.6 softFreeze false hardFreeze false ``` **It is still not CI-gateable, and the exclusion comment now says so.** The same spec on the same commit measured `bulkOpen 2575.6ms / interaction 3464.2ms` on a GitHub ubuntu runner against a 2500 ms soft budget — a ~60x spread on the number the budget reads, with the relay still streaming. That is the budget failing, not the product. The earlier draft of this comment claimed "repaired and passing", which was true only of the host it was measured on; gating this needs a host-relative oracle, not a bigger constant. ## New: a half-open link is judged, not wedged The fixture image has no `iptables` and the container has no `NET_ADMIN`, so `docker pause` is used instead — a harder case, because the container's TCP stack keeps ACKing: no FIN, no RST, and the socket looks perfectly healthy. Only an application-level probe can detect it. ``` [half-open] {"verdict":"reconnecting","verdictMs":25135,"budgetMs":90000} ``` Nothing in the suite covered the failure mode behind the "SSH hangs until I restart Orca" reports. ## New: resource accumulation measured on the remote host 6 terminals, then 5 reconnect cycles, counted on the container itself: ``` open: pts 1->6 (exactly 1/terminal), relay fds 25->30 (exactly 1/terminal) reconnect: pts flat at 6, relay procs flat at 1, node procs flat at 3 ``` `leakedMasterFdCount` is now **asserted**, not merely recorded. It counts PTY master fds held by non-relay processes: without `FD_CLOEXEC` a master is inherited by every later child, so terminal k adds k of them — the triangular signature measured as 15 across 5 terminals before the fix. #17914 patched the app and daemon and #17920 shipped the same patch to the relay host, and both are now on main, so the correct value is 0 and the probe holds it there: ``` baseline leakedMasterFdCount 0 6 terminals leakedMasterFdCount 0 (holders: only relay.js, n=6) reconnects leakedMasterFdCount 0 across all 5 cycles ``` Any growth here means the relay's node-pty rebuild did not take on that host, which is exactly what a remote-host probe exists to catch — and it is the half of #17914's claim that no unit test can reach. ## Routing Both new probes are claimed by `run-ssh-docker-e2e.mjs` (a Docker-gated spec no runner names self-skips everywhere and still reports green) **and** by the `ssh-terminal-source` route in `pr-e2e-source-routing.mjs`, so they run when the relay and SSH code they guard changes rather than only on a scheduled lane.	2026-09-02 15:58:42 -07:00
Neil	0dbe9d0504	test(ssh): dockerized relay fault injection with verdict assertions (#18017) ... * test(ssh): add a dockerized SSH fault-injection lane with four fault shapes The existing SSH reconnect specs all reconnect by calling ssh.disconnect() then ssh.connect() - a clean cycle the client knows is coming. Nothing covered the faults the reconnect machinery exists for. Four shapes, each documented with why it is not the others: killing sshd's per-connection forks (transport dies, relay survives), `docker pause` (silence with TCP still established), SIGKILLing every relay.js (the only fault where `exited` is the correct verdict), and a 48MB flood with nobody attached. The relay-kill case is the one that makes the rest meaningful: every other case asserts the session survived, which only means something if a genuinely dead session is distinguishable. It is the only case where replacing the pane is correct, so it pins the boundary in docs/reference/ssh-execution-boundary.md rather than just testing reconnection. The `docker pause` case pins the other side of that boundary: after 30s of silence from a healthy host the pane keeps its PTY and its scrollback, because loss of contact is never evidence of death. No network-blackhole fault: reconnecting the fixture does not restore its published port mapping, so that fault is not reversible on this container and would strand the worker it ran on. * test(ssh): fixme the flood case pending #18018 It fails in CI on its first real run: the pane keeps its PTY and repaints, but a command run after the flood produces no output within the poll budget. Same shape as #18018 and not caused by this spec. The three verdict assertions around it stay enforced.	2026-09-01 23:35:47 -07:00
NeilandBrennan Benson	fbe94ceff6	fix: close readiness gaps found by merged-change audit (#17159) ... * fix(ssh): fence stale kills and retired pane replay * fix(ssh): support cancellable interactive authentication * fix(ssh): await remote catalog before snapshot adoption * fix(pty): contain Windows ConPTY input failures * fix(power): avoid redundant macOS display blocking * perf(editor): narrow markdown override subscriptions * fix(quick-open): close directory handles after reads * refactor(linux): remove unused proc socket scanner * fix(usage): apply flat Sonnet 4.6 pricing * ci: prime Node next native test cache * docs(skills): resolve snapshot cleanup data path * fix(ssh): recover install locks after host reboot * test(ssh): recognize boot-aware install locks * test(ssh): prove previous-boot lock recovery live * test(wire): pin pre-metadata release coverage * fix(terminal): preserve remote tab ownership through recovery races * test(runtime): fence replaced terminal handles in agent guard * fix(ssh): preserve remote snapshot authority across polls * fix(pty): contain late ConPTY output EPIPE * test(pty): register Windows exit watcher before kill * fix: close SSH and tab readiness race gaps * fix(tabs): retain headless order and placeholder titles * fix(build): avoid parallel electron-vite config race * test(windows): avoid MSYS temp path rewriting * test(windows): avoid killing exited PTY * fix(pty): avoid late ConPTY input teardown race * fix(terminal): sync reconnect error ownership after commit * fix(runtime): use canonical worktree identity comparison * test(ssh): assert complete cold-hydration baseline * test(windows): invoke quoted retention fixture via PowerShell * test(windows): read ConPTY grid through mode con * fix(terminal): publish PTY replacements atomically * fix(terminal): infer stale identity on reattach * fix(terminal): fence stale pane PTY callbacks * fix(terminal): fence stale pane binds after rebind * fix(terminal): reject stale pane transport callbacks * fix(terminal): fence mirrored reattach spawn callbacks * fix(terminal): replace stale pane PTYs on remount * fix(ci): size the Windows launcher-compile test budget from measurement `native-smoke (windows-latest)` fails ~4.5% of runs on `preserves a multiline argument through the compiled remote launcher` with "Test timed out in 15000ms" — on unrelated PRs, for reasons that have nothing to do with them. Across 176 sampled attempts it is the only red that job produced, and it hit seven different PRs in two days: #16900, #16904, #16915, #16955 (twice), #16979, #17014, #17085. The test is six process creations: powershell.exe forks csc.exe, then the freshly compiled orca.exe forks node.exe, twice. Hosted Windows runners periodically slow process creation down, and this test amplifies that far harder than anything else in the job. Comparing the 80 attempts where it ran under 3s against the 12 where it ran over 12s, its own median goes 2198ms -> 15917ms (7.2x) while the same file's powershell-only test moves 556 -> 686ms (1.2x), the cmd.exe and Git Bash process tests in the neighbouring file move 1.4x, and the other 35 files put together move 1.5x. Measured across those 176 attempts: 1881ms to 35438ms, p50 4264ms, correlation +0.881 with the job's total Vitest duration. 8 of 176 (4.5%) exceeded the 15s cap; 2 of 176 (1.1%) also exceeded the shared 30s testTimeout, so deleting the override and inheriting the config is not enough on its own. 60s clears all 176 with 1.7x headroom on the worst. This is slow, not hung. Every body here is synchronous spawnSync, so Vitest cannot interrupt one — the timer fires only after the body returns and the reported duration is real elapsed time. That is why a failure reads `× ... 22464ms` under `Test timed out in 15000ms`. The work finished; the stopwatch was short. Seven reruns at one identical head measured 2053 / 4680 / 5551 / 8732 / 13506 / 14868 / 21937ms — the last of those would have been red on code that had not changed. The 15s came from #8897, which raised this test off Vitest's built-in 5s default because the job then ran bare `pnpm vitest run`. #8909 landed 3h27m later and pointed the job at config/vitest.config.ts, which is the real fix for that. The constant stayed behind and has been the binding budget ever since. * fix(terminal): fence stale remount reattach ownership * fix(terminal): reconcile mounted pane identity after replacement * fix(terminal): fence stale reattach fallback ownership * fix(terminal): fence deferred SSH reattach ownership * fix(terminal): fence stale split pane ownership callbacks * fix(terminal): keep stale spawns from consuming startup --------- Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>	2026-08-31 08:17:40 -07:00
Neil	7b467bd0a6	ci: gate PRs on a real input method, and prove the lane engaged one (#17365) ... * ci: gate PRs on a real input method, and prove the lane engaged one No job on the PR gate has ever run a real input method. pr.yml and e2e.yml are ubuntu-latest with CDP `Input.imeSetComposition`, which is a synthetic composition; the only job that drives ibus-hangul through xdotool is terminal-ime-e2e.yml, and it is schedule + dispatch only. A PR could turn the real-IME path red and merge green. Route IME source to that lane from pr.yml through the existing pr-e2e-source-routing mechanism, so it runs on IME-touching PRs and nothing else. The lane stays out of verify.needs — advisory, like `e2e` — because its reliability is known only from nightly main runs. Deliberately no continue-on-error: that reports green and hides the signal. The harness fails open in ways that all look like success: Playwright reports a skipped test as a pass, so an unset ORCA_E2E_NATIVE_IBUS_HANGUL, a renamed test, or a session with no engine all exit 0 having exercised nothing. The specs now append an engagement receipt only after observing real composition events, and the runner requires one per expected test before the lane may report success. Also drop the native spec from changed-e2e: it was already routed there by its own filename, where it self-skips for want of an ibus session and reported that skip as coverage. * ci: let the real-IME step report even when the synthetic step failed	2026-08-30 01:58:32 -07:00
Neil	971d987c4b	ci(e2e): trigger the Docker-SSH lane from SSH source and claim every gated spec (#16746) ... The Docker-SSH e2e lane only ran when a PR's changed specs happened to include `ssh-startup-exec-readiness.spec.ts` or `paired-startup-exec-readiness.spec.ts`. Editing SSH source itself did not trigger it, and pruning either spec from a route's list would have silently retired the whole lane. Meanwhile the sharded lanes set no `ORCA_E2E_SSH_DOCKER`, so every Docker-gated spec skipped itself while the shard still reported green -- the exact silent-skip shape `docs/reference/ssh-reconnect-source-recovery.md` blames for four regressions that reached users. Separately, the modules that actually own direct-SSH workspace and tab restore carry no "ssh" in their names, so the `ssh-terminal-source` route never reached them. Measured on the real script before this change: printf '%s\n' src/renderer/src/hooks/remote-workspace-session-merge.ts \ src/main/ipc/remote-workspace-snapshot-normalization.ts \ src/renderer/src/lib/worktree-initial-terminal-seeding.ts \ src/shared/remote-workspace-session-projection.ts \ | node config/scripts/pr-e2e-source-routing.mjs => [] Three changes, all pinned by the executable gate contract: - `hasSshSourceChange` derives an `ssh_source_changed` signal from the SSH routes themselves, plumbed pr.yml -> e2e.yml, so the lane triggers on source rather than on a spec name surviving in a list. One list, so the two cannot drift. - A sibling `ssh-workspace-session-restore` route names the restore seams (`remote-workspace-*`, `worktree-initial-terminal-seeding`, `worktree-default-terminal-tabs`, `initial-terminal`) and routes them to the two restore specs -- a sibling rather than more paths on `ssh-terminal-source` so a tab-tombstone edit does not run the whole SSH terminal list. - A new `test:e2e:ssh-docker` runner claims the remaining Docker-gated specs on the one VM that sets the flag, and the contract now fails by name when any Docker-gated spec is claimed by no runner. `ssh-docker-relay-perf` and `ssh-codex-display-artifacts-repro` are recorded exemptions (wall-clock budgets; needs a real remote codex binary) and the contract asserts each exemption still corresponds to a real gated spec, so a stale one cannot quietly excuse a gap. Lane timeout raised 35 -> 60 minutes for the added serial specs. The lane's first act was to surface four latent bugs in a spec that had been silently skipping. `ssh-docker-bulk-open-freeze-repro.spec.ts` is four call sites out of date against `tests/e2e/helpers/terminal.ts`: `startDockerSshRelayTarget()` is called with no argument though the helper dereferences `testInfo.workerIndex` (a 100% failure, not a flake), `execInTerminal` gained a `ptyId` parameter, and `splitActiveTerminalPane` gained a direction. It was invisible because it ran nowhere and `typecheck:e2e` is red on main with 240 pre-existing errors, so four more could not be seen. The `testInfo` bug is fixed here -- correct on its own, and it removes one real error from `typecheck:e2e` (240 -> 239). The other three are not, because they are not argument plumbing: repairing them requires choosing which ptyId to capture and which split direction to use, and both change what the repro measures. The spec is therefore added to the exemption list rather than repaired, for two independent reasons recorded in the runner: it is a perf oracle, not a correctness one (`SOFT_FREEZE_LAG_MS=2500` / `HARD_FREEZE_LAG_MS=5000` measured under a deliberate 5-pane flood on a 420s budget -- the same rule already applied to `ssh-docker-relay-perf.spec.ts`), and it is known-rotted. Repair is tracked in stablyai/orca#16764. Applying an existing written rule to a sibling that plainly meets it is consistency; inventing a new exemption to dodge a red would not be. Three hardening fixes to the contract itself: - Runner text is comment-stripped before the claimed-by-a-lane scan. A substring scan over raw text lets a spec merely *discussed* in a runner comment count as claimed -- the silent skip this assertion exists to catch, re-entering through the documentation. Not live today only because the existing comments write the spec names without their `tests/e2e/` prefix. - An exempt spec must not be invoked by any runner. `unreachableSpecs` short-circuits the unclaimed check, so a spec could be documented as exempt while a runner still ran it -- an exemption that reads as coverage removal but changes nothing, leaving the lane red for a reason the file says it excluded. This is not hypothetical: adding the bulk-open exemption without removing it from the runner's spec list produced exactly that state, and this assertion is what caught it. - The Docker-gate detector is now `/ORCA_E2E_SSH_DOCKER\s*[!=]==\s*['"]1['"]/` rather than one fixed string, so a double-quoted or `!==` spelling can no longer escape the contract. `ssh-restart-tab-accumulation.spec.ts` is a new three-cycle restart fence asserting tab-id set identity, not just the active pane's reclaimed ptyId as `ssh-cold-activation-restore.spec.ts:241` did. It passes today; it was validated by a negative control that injected one tab after cycle 1 and correctly failed.	2026-08-27 19:40:38 -07:00
Jinwoo HongandJinwoo-H	a9781a4118	STA-4150: client-hosted remote browser (consolidated) (#15448) ... Co-authored-by: Jinwoo-H <jinwoo@stably.ai>	2026-08-25 15:36:51 -07:00
Jinwoo Hong	c618ec7393	test(reliability): protect recent P0 regression invariants (#16163)	2026-08-24 09:38:46 -07:00