42 Commits

Include renames

Author	SHA1	Message	Date
3 people	93d8b1f042	fix(ssh): complete keyboard-interactive MFA prompt handling (#15588) ... Honor SSH keyboard-interactive prompt echo and empty responses, reuse login passwords without replaying rejected values, and stop cancelled or stale credential requests from continuing authentication or restoring the cache. Original implementation: Junho Kim (#8750). Port and follow-up work: Allen (#15588). Verified with 2,773 SSH/credential tests, full typecheck, changed-code quality, a production Electron build, real-socket MFA fixtures, and rendered UI checks. Fixes #8622 Co-authored-by: Junho Kim <arkimjh@illinois.edu> Co-authored-by: microdaery <microdaery@gapp.nthu.edu.tw>	2026-09-27 01:12:29 -07:00
NeilandBrennan Benson	fbe94ceff6	fix: close readiness gaps found by merged-change audit (#17159) ... * fix(ssh): fence stale kills and retired pane replay * fix(ssh): support cancellable interactive authentication * fix(ssh): await remote catalog before snapshot adoption * fix(pty): contain Windows ConPTY input failures * fix(power): avoid redundant macOS display blocking * perf(editor): narrow markdown override subscriptions * fix(quick-open): close directory handles after reads * refactor(linux): remove unused proc socket scanner * fix(usage): apply flat Sonnet 4.6 pricing * ci: prime Node next native test cache * docs(skills): resolve snapshot cleanup data path * fix(ssh): recover install locks after host reboot * test(ssh): recognize boot-aware install locks * test(ssh): prove previous-boot lock recovery live * test(wire): pin pre-metadata release coverage * fix(terminal): preserve remote tab ownership through recovery races * test(runtime): fence replaced terminal handles in agent guard * fix(ssh): preserve remote snapshot authority across polls * fix(pty): contain late ConPTY output EPIPE * test(pty): register Windows exit watcher before kill * fix: close SSH and tab readiness race gaps * fix(tabs): retain headless order and placeholder titles * fix(build): avoid parallel electron-vite config race * test(windows): avoid MSYS temp path rewriting * test(windows): avoid killing exited PTY * fix(pty): avoid late ConPTY input teardown race * fix(terminal): sync reconnect error ownership after commit * fix(runtime): use canonical worktree identity comparison * test(ssh): assert complete cold-hydration baseline * test(windows): invoke quoted retention fixture via PowerShell * test(windows): read ConPTY grid through mode con * fix(terminal): publish PTY replacements atomically * fix(terminal): infer stale identity on reattach * fix(terminal): fence stale pane PTY callbacks * fix(terminal): fence stale pane binds after rebind * fix(terminal): reject stale pane transport callbacks * fix(terminal): fence mirrored reattach spawn callbacks * fix(terminal): replace stale pane PTYs on remount * fix(ci): size the Windows launcher-compile test budget from measurement `native-smoke (windows-latest)` fails ~4.5% of runs on `preserves a multiline argument through the compiled remote launcher` with "Test timed out in 15000ms" — on unrelated PRs, for reasons that have nothing to do with them. Across 176 sampled attempts it is the only red that job produced, and it hit seven different PRs in two days: #16900, #16904, #16915, #16955 (twice), #16979, #17014, #17085. The test is six process creations: powershell.exe forks csc.exe, then the freshly compiled orca.exe forks node.exe, twice. Hosted Windows runners periodically slow process creation down, and this test amplifies that far harder than anything else in the job. Comparing the 80 attempts where it ran under 3s against the 12 where it ran over 12s, its own median goes 2198ms -> 15917ms (7.2x) while the same file's powershell-only test moves 556 -> 686ms (1.2x), the cmd.exe and Git Bash process tests in the neighbouring file move 1.4x, and the other 35 files put together move 1.5x. Measured across those 176 attempts: 1881ms to 35438ms, p50 4264ms, correlation +0.881 with the job's total Vitest duration. 8 of 176 (4.5%) exceeded the 15s cap; 2 of 176 (1.1%) also exceeded the shared 30s testTimeout, so deleting the override and inheriting the config is not enough on its own. 60s clears all 176 with 1.7x headroom on the worst. This is slow, not hung. Every body here is synchronous spawnSync, so Vitest cannot interrupt one — the timer fires only after the body returns and the reported duration is real elapsed time. That is why a failure reads `× ... 22464ms` under `Test timed out in 15000ms`. The work finished; the stopwatch was short. Seven reruns at one identical head measured 2053 / 4680 / 5551 / 8732 / 13506 / 14868 / 21937ms — the last of those would have been red on code that had not changed. The 15s came from #8897, which raised this test off Vitest's built-in 5s default because the job then ran bare `pnpm vitest run`. #8909 landed 3h27m later and pointed the job at config/vitest.config.ts, which is the real fix for that. The constant stayed behind and has been the binding budget ever since. * fix(terminal): fence stale remount reattach ownership * fix(terminal): reconcile mounted pane identity after replacement * fix(terminal): fence stale reattach fallback ownership * fix(terminal): fence deferred SSH reattach ownership * fix(terminal): fence stale split pane ownership callbacks * fix(terminal): keep stale spawns from consuming startup --------- Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>	2026-08-31 08:17:40 -07:00
Neil	24e662adc1	feat(ssh): verify host keys, and restore panes correctly across a reconnect (#14844) ... * docs(ssh): design for real host key verification (STA-4319) Today's ssh2 verifier records a fingerprint and returns true — every host key is accepted, with no known_hosts consult and no change detection anywhere in src/main/ssh/. Scope is per-connection, so exec, SFTP, port forwarding, the watcher and relay deploy all ride that one unverified handshake, and the ProxyJump path puts the final hop — the topology most likely to cross untrusted network — on ssh2 specifically. Decisions worth calling out: - Read the user's known_hosts as a trust source but NEVER write to it. That file is shared with every other SSH tool on the machine; appending means line endings, permissions, concurrent writers and a corruption blast radius well beyond us. Accepted keys go to our own per-target store. Reading theirs is also the entire migration story: most developers already have their hosts there. - Mismatch is scoped to the SAME key type. A host with only an RSA entry that presents ed25519 is unknown, not changed. ssh2 negotiates ed25519 first, so without this we would fire a change-of-key alarm at nearly every existing user on their first upgraded connect — training them to dismiss the one warning that is supposed to mean something. Flagged in review as the decision I am least sure of; a downgrade-vector argument against it is being tested. - Changed key hard-fails with no override button; recovery is a separate explicit action, offered only when OUR store is what disagreed, because forgetting our record cannot unblock a known_hosts conflict. - Background reconnects deny rather than prompt. A dialog the user cannot place in context only teaches click-through. Two traps are documented because either would make the fix silently do nothing: an async verifier returns a Promise, which ssh2 reads as truthy and accepts immediately; and the existing test mock invokes hostVerifier with one argument and ignores the return, so it would pass against a verifier that never decides. Design only — no behaviour change. The doc is added to the tracked-reference allowlist in .gitignore alongside the other docs/reference entries. * docs(ssh): revise the host key design after security and migration review Three things the reviews changed, kept visible rather than quietly edited out. THREAT MODEL WAS WRONG IN THREE PLACES. Jump hosts are not the worst case — they are already safe: shouldUseSystemSshTransport branches on exactly the inputs resolveEffectiveProxy does, and attemptConnect returns after the system probe, so ProxyJump goes through OpenSSH and is verified. Agent forwarding was overstated (gated on the user's ForwardAgent). Credential theft was understated: any auth error counts as agent fallback, so a MITM walks the user to the password AND private-key passphrase prompts, and cachedPassword replays without prompting. The relay claim was backwards — the attacker owns their own machine; the real impact is the return direction, where they become the host our workspace trusts. TYPE SCOPING IS A DOWNGRADE VECTOR WITHOUT ALGORITHM ORDERING. This was the decision I flagged as least certain and asked to have argued both ways. OpenSSH is safe only because order_hostkeyalgs() puts known types first and RFC 4253 gives the client's order priority. ssh2 negotiates ed25519 first regardless, so an attacker who cannot forge the RSA key on file just presents ed25519 and gets a friendly first-contact prompt instead of a hard failure. Keep scoping, but set algorithms.serverHostKey to lead with the types on file — and add a sixth outcome for 'unknown type, known host', which must never read as first contact. SHIP THE DEFENCE BEFORE THE DIALOG. Startup restore fires eager connects for all targets in parallel with a 15s timeout while a prompt would live 120s; ephemeral VM targets present a new key every launch; paired-web connects run on the host desktop, so the dialog opens on someone else's screen. Phase 1 is therefore no modal at all: consult known_hosts and our store, match connects, unknown persists with accept-new semantics, mismatch and revoked hard-fail. That is the whole MITM defence with none of the migration risk. Also folded in, verified live against OpenSSH 10.2p1: the without-port fallback (bracketed lookup first, then bare, where the second pass can only yield match or unknown — otherwise a bare line plus a non-default port produces a spurious prompt); hashed entries hash the candidate form; multiple files union; a cert-authority line does not match a plain key. IPv6 and bracket parsing moved INTO scope — that is a parser requirement, not a scope call, and getting it wrong produces the prompt-training harm the design exists to avoid. * feat(ssh): parse and match OpenSSH known_hosts The matcher half of STA-4319. No behaviour change yet — nothing calls this. Hand-rolled because no maintained JS implementation exists, and written against behaviour observed from OpenSSH 10.2p1 rather than inferred from the man page. Three of those behaviours a reasonable reading gets wrong: - A non-default port is TWO ordered lookups, not one candidate set: '[host]:port' first, then bare host ('checking without port identifier' in ssh -v). The fallback pass can only yield match or unknown — OpenSSH downgrades a wrong key there rather than reporting a change. Collapse them and anyone holding a bare line who connects off-port gets a spurious first-contact result; treat the fallback as authoritative and they get a false change-of-key alarm. - Revocation resolves in its own pass so the verdict cannot depend on line order. Verified both orderings. - A cert-authority line never matches a plain host key; it only validates certificates. A normal line alongside it still decides. Mismatch is scoped to the same key type, and a host known by a DIFFERENT type returns unknown-type-known-host rather than plain unknown — an attacker who cannot forge the key on file must not get a friendly first-contact result by presenting another type. That outcome is only half the defence; the other half (leading serverHostKey with known types) lands with the wiring. 47 tests from vectors executed against real sshd, including ssh-keygen -H hashed entries. Each of six mutations reddens it: collapsing the passes, letting the fallback report mismatch, dropping type scoping, resolving revocation in line order, honouring an unrecognised marker, and skipping the blob/type agreement check. * feat(ssh): decide what to do with a presented host key The policy half of STA-4319, kept separate from the ssh2 wiring so it is testable without a handshake and injected rather than importing its sources, so a test states its own trust state instead of writing files. Phase 1 ships no dialog — a test asserts the decision is never 'prompt'. Startup restore opens every previously-active target at once, ephemeral VM targets would ask every launch, and paired-web connects run on the host desktop where the dialog would appear on someone else's screen. Ordering that matters: revocation outranks everything including StrictHostKeyChecking=no, because a revoked key is a statement that this key is known-bad rather than merely unrecognised. known_hosts is named before our own store on a change, because its remedy (ssh-keygen -R) is the one that also unblocks ssh and git — pointing at a remedy that cannot work is worse than none. Two carve-outs with reasons: an ephemeral runtime target accepts WITHOUT recording, since a fresh VM presents a new key every launch and a stored record would accumulate per launch and eventually read as a spurious change; and when ssh -G ran on the HOME-divergent path that suppresses /etc/ssh/ssh_config, an unknown host is denied, because a site-wide policy may forbid it and being laxer than ssh is the one outcome that is never acceptable. Rejection text deliberately avoids 'authentication failed' and 'permission denied': the reconnect ladder classifies on those substrings, so a denial phrased that way is retried forever against a decision that will never change. Pinned by a test. * feat(ssh): build the host key verifier and the algorithm order that makes it safe Still not wired into the handshake — that lands next. This is the piece that turns a decision into an ssh2 callback, plus the half of the design that is easy to forget because it lives in a different config field. The verifier MUST be a plain function returning undefined. ssh2 does 'const ret = verifier(key, verify); if (ret !== undefined) verify(ret)', so an async function returns a Promise — neither undefined nor falsy — and ssh2 accepts the key immediately while ignoring whatever the callback later decides. Making this async would silently restore exactly the accept-everything behaviour the module exists to remove, so a test asserts the return value is undefined. orderServerHostKeyAlgorithms is what makes type-scoped matching safe rather than a downgrade. RFC 4253 gives the client's algorithm order priority, so leading with the types we already hold for a host denies a server the choice of presenting some other type to convert a hard failure into first contact. Without it, an attacker who cannot forge the key on file just offers a different algorithm. Revoked entries never contribute to that order. Also fails closed on two paths that would otherwise hang or over-trust: a key whose own length-prefixed header cannot be read is refused rather than reasoned about, and a throw from any dependency denies, because ssh2 may not catch an exception raised inside the verifier and the handshake would hang instead of failing. 18 tests. Includes the two negative cases that matter — first-contact keys are recorded, but keys we already know, rejected keys, ephemeral runtime targets and a lax StrictHostKeyChecking are not. * fix(ssh): promote every RSA signature algorithm for a known ssh-rsa key A known_hosts entry names the KEY type, which is not the negotiated ALGORITHM name. One ssh-rsa key is offered as rsa-sha2-512, rsa-sha2-256 or ssh-rsa depending on the signature algorithm, so matching the literal name only would leave a host we know by RSA ordered behind ed25519 — precisely the ordering this function exists to prevent, and precisely the population (RSA-era known_hosts entries) it was written for. Verified from ssh2's own negotiation while wiring this: kex.js iterates the CLIENT list and takes the first entry the server also offers, so client order does decide, as RFC 4253 says. ssh2's default order leads with ed25519 and places the RSA algorithms fifth through seventh. * fix(ssh): verify host keys instead of accepting every one (STA-4319) The actual fix. ssh-connection's verifier recorded a fingerprint and returned true, so every ssh2 connection accepted every host key — no known_hosts consult, no change detection. It now consults the user's known_hosts plus our own store and refuses a changed, revoked or unverifiable key. Phase 1 by design: no dialog. Unknown hosts are accepted and recorded (accept-new semantics), because startup restore opens every previously-active target at once, ephemeral VM targets present a new key each launch, and paired-web connects run on the host desktop where a prompt would appear on someone else's screen. The MITM defence lands now; the prompt is Phase 2. Also sets algorithms.serverHostKey to lead with the types already known for the host. Without it the type-scoped matching is a downgrade — an attacker who cannot forge the key on file just presents another type and turns a hard failure into first contact. Verified from ssh2's kex.js that the client list decides. Denial replaces ssh2's generic handshake error with the specific reason, because the reconnect ladder cannot distinguish a generic failure from a transient fault and would retry forever against a decision that will never change. An unreadable trust store degrades to known_hosts only rather than failing the connect: a changed key is still refused, and a host trusted only by us falls back to first contact and is re-recorded, reaching the same decision. The ssh2 mock now uses the callback form and aborts the handshake on denial. As written it called hostVerifier(key) with one argument and ignored the result, so it would have passed against a verifier that never decides — flagged in the design as a mock that had to change, not a test to quietly rewrite. Two new tests pin the wiring rather than the module: an unidentifiable blob is refused, and a well-formed key is accepted. Note for review: commit 2d2a0880ba unintentionally swept in two modules built concurrently (ssh-known-hosts-source, ssh-host-key-store) because I staged with 'git add -A'; its message describes only the verifier. Both are covered by their own tests, but the attribution in that commit is wrong. 1461 SSH tests pass. * fix(ssh): bind the host key store to the active profile at startup Without this the store reports nothing trusted on every launch. Safe — known_hosts still decides, and a host trusted only by us degrades to first contact and is re-recorded — but it silently discarded our own accept records, so the store the design calls for was not actually in use. Bound beside the profile Store, since it is a sidecar of the same data file. Also records why the paired-web carve-out the migration review asked for is NOT implemented in Phase 1, rather than leaving it looking forgotten. That carve-out exists to stop a web client waiting out the 120s prompt timeout — a hang only reachable if a prompt exists, and Phase 1 has none, which the decision function pins with a test asserting it never returns 'prompt'. An RPC connect therefore behaves exactly like a local one. Adding a fail-fast path now would introduce a failure mode for a hang that cannot occur; it becomes load-bearing when the dialog lands and is listed under Phase 2. Noted there for whoever builds Phase 2: runtime/rpc/methods/ssh.ts already swallows the specific error and rethrows a generic one, so the host-key reason will not reach a web user without a change there too. Full unit suite: 52,352 pass. The 8 failures are the known environment baseline (5 osc8, 2 IME) plus one browser-cookie suite-ordering flake that passes in isolation — none in src/main/ssh, and none related to this change. * fix(ssh): close two downgrades the implementation review found Both were in my wiring, not the design, and two reviewers found the first independently. 1. OUR STORE WAS TYPE-DOWNGRADABLE. The inline lookup filtered by key type first and could only answer match/mismatch/unknown, so a record of a DIFFERENT type for the same endpoint read as "unknown". A host learned on first contact — ed25519, since ssh2 proposes it first — and absent from known_hosts could then be impersonated by presenting RSA: both sources say unknown, so accept-and-remember, silently. That is exactly the downgrade D3 says the design cannot ship without, applied to the records we create ourselves. The store's own isTrusted already computed the right answer and had no production caller. Stored types now also feed the algorithm ordering, without which the guard is only half present. 2. WE KEYED ON THE ORCA LABEL, NOT THE DIALED HOST. "ssh -G" echoes its own argument back as its hostname field when no Host block matches, so for a manual target that field IS the Orca label — the one name D2 forbids keying on, and one ssh never wrote. We consulted no entries at all, so an impersonated host read as first contact. Now keys on the dialed host, which buildConnectConfig has already resolved through HostName, with HostKeyAlias still winning. That inverted an existing test rather than deleting it: "uses the resolved hostname, never the Orca label" encoded an assumption disproved against OpenSSH 10.2p1, so it is renamed and reversed with the reason recorded in the test. 3. NO READABLE SOURCE IS NOT FIRST CONTACT. Every known_hosts file failing to read was indistinguishable from "this host is unknown", so a changed key would be accepted the one time we could not check. The loader now reports how many files it could read, and zero readable sources with an empty store takes the strict path instead of recording trust. 4. A superseded attempt's rejection could replace the live attempt's error, and substituting a new Error drops ssh2's code, so a transient ECONNRESET would stop being classified as retryable. The rejection is now local to its attempt. 5. displayHost was the Orca label, so a mismatch could print "ssh-keygen -R <label>" — a remedy that removes nothing. Also adds the tests that would have caught 1 and 2, the stale-attempt denial, and IPv6 literals, which the design moved into scope and nothing covered. 1,469 SSH tests pass. * fix(ssh): stop offering credentials to a host we just refused A refused host key ended the handshake and then fell into the credential ladder, because ssh2 reports a denied key as a generic auth failure and the passphrase branch is eligible on message shape alone whenever an encrypted identity file is configured. So the sequence was: decide this host may not be who it claims to be, then ask the user for their passphrase and hand it to it. Failing that, prompt for a password. Failing that, retry over the system ssh binary, which for a disagreement with our own store rather than known_hosts would simply connect. That inverts the point of checking at all. A denied key is now final for the attempt: recognised by type before any fallback runs, and again inside the agent-fallback retry, which re-runs the handshake and so can be the attempt that denies. Two things had to change for that to hold. The rejection is now a HostKeyVerificationError rather than a rebuilt Error, so the connect path recognises it by type. Substring matching would have worked today and quietly stopped working the first time a reason string was reworded — and these strings are already worded around the auth-error classifier, so they are exactly the kind that get edited. And the verifier now reports the denials that skipped the policy: an unreadable key blob and an internal failure both denied without calling onDecision, so the connect path saw only ssh2's generic failure and walked the ladder. That was the actual path the new test hit first. The report carries no fingerprint, since there is no host key to identify, and the connection no longer overwrites the fingerprint it holds with an empty one — the relay keys install-lock isolation on that value. The reconnect ladder also refuses to retry it. Its classifier is otherwise substring-driven, so a reason containing "connection reset" would have been retried until the ladder gave up, burying the reason under nine attempts. Tests: no credential prompt after a refusal, an encrypted key configured so the passphrase branch is eligible; refused reports as 'error', not 'auth-failed', which would invite the user to re-enter credentials that are not the problem; no retry; both bypassing denials report; a throwing listener still denies rather than hanging the handshake. Also drops three test files a bisect resurrected from before the revert that deleted them. 172 tests pass across the three touched files; typecheck and lint clean. Pre-existing on origin/main and untouched here: 3 failures in ssh-connection-sftp-namespace.test.ts. * fix(ssh): stop treating an absent known_hosts as a source we failed to read The previous commit's "no readable source is not first contact" guard was right about the danger and wrong about how to detect it, and the version that shipped would have refused every connection a new profile ever makes. A file that does not exist and a file that refuses to open both arrive as a rejected readFile, and I counted them the same way. They are opposites. An absent known_hosts is the normal state — ssh creates it on its own first connect, and an Orca profile that has never connected has none — and it is real evidence that no host is known. A file that exists and will not open is evidence withheld: the entry that would have said "this key changed" may be sitting in it. The default list is the reason this was fatal rather than obscure. It always names known_hosts2, which essentially never exists, so on a machine with a normal known_hosts the count was 1-of-2 and everything worked; on a fresh profile it was 0-of-2 and every connect failed with "the system SSH configuration could not be read" — a message about a file the user does not have and an error they cannot act on. The suite passed only because the machine running it happens to have a known_hosts. Pointing HOME at an empty directory fails 62 connection tests, which is what the new wiring test does. So the count is now unreadableFileCount: files that exist and could not be read, which is the condition the guard was always trying to express. Any one of them takes the strict path; an absent or empty file takes none. The store clause is gone with it — a store hit already returns accept before this is consulted, so it never changed an outcome. Tests: absent, empty, permission-denied, directory, and all-parsed at the source; first contact with no known_hosts at all at the wiring level. 1,482 SSH tests pass. The 3 failures in ssh-connection-sftp-namespace.test.ts are pre-existing on origin/main and untouched here. * fix(ssh): let an ephemeral runtime outrank sources we could not read Three things, all about the same question: when we cannot see everything that decides a host key, what does that actually license us to refuse? 1. ON-DEMAND RUNTIMES WERE REFUSED FOR A POLICY THEY COULD NEVER SATISFY. A machine provisioned a minute ago cannot be in known_hosts, by construction — which is why it has a carve-out at all. But the carve-out sat BELOW the incomplete-sources check, so anyone whose HOME diverges from their passwd home (sandboxes, our own E2E isolation) took the `-F` path, and every on-demand runtime connection was refused, pointing at a config file the user cannot fix. Refusing there buys nothing. No policy, seen or unseen, is satisfiable by a host that did not exist yesterday; the trust comes from the provisioning channel. So the carve-out now outranks it. An EXPLICIT StrictHostKeyChecking=yes still wins over both — that one we can read, and the user asked for it. 2. THE FLAG WAS NAMED FOR ONE OF ITS TWO MEANINGS. `siteConfigSuppressed` started as "-F hid /etc/ssh/ssh_config" and had since grown "a known_hosts file exists and would not open" — which is not a site config, and reading the name in the decision function told you nothing about why an unreadable file landed there. It is `verificationSourcesIncomplete` now: we could not see something that decides this, so do not extend NEW trust. A host we already know still connects, because a match is decided before this is reached, and that is now pinned by name. 3. THE COPIED ssh2 ALGORITHM LIST HAD NO DRIFT ALARM. We reorder ssh2's default host-key proposal, which meant hand-copying a list ssh2 exports only from a deep internal path. ssh2 throws `Unsupported algorithm` on anything outside its supported list, so drift does not degrade — every target stops connecting, before a socket opens, with a message about an algorithm the user never chose. Worth knowing that the list is also built conditionally on ed25519 support. Kept as a literal rather than a deep import, since silently adopting a new proposal order is the wrong default — the order is what makes type-scoped matching safe, so a change deserves review. A test now compares it against ssh2's real constant and checks every entry is one ssh2 accepts. Moved next to the ordering function it feeds, and out from between the import statements. 1,488 SSH tests pass. The 3 in ssh-connection-sftp-namespace.test.ts are pre-existing on origin/main. * docs(ssh): record what review changed and the STA-4319 follow-ups The design survived implementation; every defect found afterwards was in the wiring. Worth recording the pattern, because it repeated five times: each one made us either blind or unusable, never subtly wrong — and four of the five broke legitimate hosts rather than admitting bad ones. Action items separate the things Phase 2 must decide (UpdateHostKeys, which is now the likeliest way a legitimate user meets a rejection; the web client never seeing the reason; RPC fail-fast) from the gaps Phase 1 knowingly accepts (WSL, CheckHostIP, ca-only hosts, the hand-copied ssh2 algorithm list). Also flags the rollout risk plainly: this is the first release in which Orca can refuse an SSH connection at all. * test(ssh): pin the ephemeral carve-out where it is actually observable The first version of this test asserted an on-demand runtime connects on first contact, which every target does — it would have passed with the carve-out deleted. Pointing HOME at a home whose known_hosts exists and will not open makes the two cases diverge: a normal target is refused there, an on-demand runtime is not. Removing the carve-out now fails three tests instead of none. Also covers the wiring for the unreadable-source refusal itself, which until now was only pinned at the decision level. * test(ssh): pin the parser against real ssh -G output, and accept-new against ask Two gaps the audit named. The ssh -G fixtures were all hand-written, which means they encode what I expect ssh to print. This one is verbatim OpenSSH_10.2p1 output for a Host block using HostName, HostKeyAlias, StrictHostKeyChecking accept-new, two UserKnownHostsFile paths and a non-default port. The format detail that matters: each file list arrives space-separated on ONE line, so reading it as a single path would consult nothing for anyone with more than one file configured. And accept-new was untested despite being a real OpenSSH value. It currently behaves identically to ask, which is exactly right while no dialog exists and is what lets the defence ship without a modal — so the equivalence is now pinned, and Phase 2 has to break it deliberately rather than discover it. Plus a guard that accept-new never falls into the strict branch. * test(ssh): cover the wire from an accepted key to a record on disk Nothing covered the store end to end. Every connection test runs with it unwired — a real path, since it degrades to known_hosts only — so an accepted first-contact key was never observed becoming a record, and the record was never observed being believed on the next connection. Without that wire the store is dead weight: an unknown host connects every time and is never learned. Its own file for two reasons. initSshHostKeyStoreFile binds module-level state for the rest of the process, and binding it makes the connect prelude do real disk I/O — which the shared suite cannot absorb, because its reconnect tests drive the clock with fake timers and an fs round trip does not complete inside an advanced tick. Adding these there turned 12 unrelated tests red. Covers: the record is written; it is read back as a match; the same key is not recorded twice; a DIFFERENT key for a host we recorded ourselves is refused (the store's entire security value, and a case known_hosts cannot catch since it has never heard of the host); and an on-demand runtime records nothing. Stubbing rememberHostKey to a no-op fails four of the five. * fix(settings): stop truncating the SSH connection error to one line The host key messages are written to be actionable — a mismatch ends in "Run: ssh-keygen -R <host>", which is the remedy that also unblocks ssh and git. The only place in the renderer that displays an SSH connection error clamped it to a single line with `truncate` and carried no title attribute, so the remedy was unreachable, not even on hover. The careful wording reached a CSS ellipsis. Wraps instead, with [overflow-wrap:anywhere] because a long hostname offers no break opportunity and would overflow the column on its own. The paragraph only renders on failure, so the extra height costs nothing in the normal case. CORRECTION to the four preceding commits: they each claimed 3 pre-existing failures in ssh-connection-sftp-namespace.test.ts. That was my error — I had been running `npx vitest` without `--config config/vitest.config.ts`, so the project's setupFiles and execArgv were absent. Under the real config those 3 pass, and have throughout. The whole SSH suite is green: 1,504 passed, 13 skipped. Still failing on this branch and unrelated to it (different subsystems, no file overlap): 5 in terminal-snapshot-osc8-roundtrip and 2 in browser-cookie-import. * fix(terminal): show why the SSH connection failed, not just that it did The reconnect overlay took only a status, so every failure rendered the same sentence: "The SSH connection to devbox failed. Connect again to continue this terminal session." A refused host key and a network timeout were indistinguishable there, and the host key message — the only place that names the remedy, down to `ssh-keygen -R <host>` — reached no terminal user at all. The state carried it the whole way; the overlay simply never asked for it. Adds it as a second line rather than replacing the sentence. The sentence says what to do, the detail says what happened, and keeping both means an errno failure does not lose its guidance to make room for "connect ETIMEDOUT". Wrapped, for the same reason as the settings card: the remedy is at the end. Suppressed for a removed target, which already explains itself and can never reconnect — a stale connection error underneath would contradict it. The new selector mirrors selectRuntimeAwareSshStatus branch for branch, including the unreachable-environment and un-hydrated-bucket nulls, so the pair cannot disagree about which source they read and a detail is never shown next to a status it did not come from. Known and NOT addressed here: "Connect again" is still the wrong advice for a decision that will never change. Telling those apart needs a typed reason on the wire rather than a string, which is a remote-wire-compatibility decision; it is recorded in the STA-4319 follow-ups. Pre-existing on this branch and untouched: 2 failures in terminal-ime-xterm-resumed-preedit-visibility. * docs(ssh): record where the rejection message actually lands Traced end to end, because a rejection the user cannot read is a half-shipped feature — and two of the surfaces were dropping it entirely, both now fixed. What is left is written down rather than guessed at: the status bar renders only 'Error', the terminal overlay's call to action still invites a retry that cannot succeed, toasts carry Electron's remote-method prefix, and the paired-web path replaces the text with 'SSH connection unavailable' on every route — which also affects a DESKTOP user viewing a host owned by a remote Orca server, not just web clients. * refactor(ssh): give the store one matcher instead of two The connect path had its own copy of the store comparison, because ssh2's verifier decides synchronously and cannot await the file, so records are preloaded. That copy is precisely where the type downgrade came from: it answered only match/mismatch/unknown, so a record of a different type for the same endpoint read as first contact and a host learned on first contact could be impersonated by presenting another key type. Fixing it left two implementations that have to agree forever, which is the same bug waiting to happen. matchTrustedHostKeys is now the single pure matcher; isTrusted is a load plus a call to it, and the connect path calls it directly on preloaded records. Same for the key types that feed the algorithm ordering, which the connect path was also filtering by hand. The two copies had in fact already drifted: the connect path lower-cased the query host where the write trims AND lower-cases. Not reachable today — the host is trimmed before it reaches there, which I confirmed by mutating the normalisation and watching the wiring test pass anyway. So this is not a bug fix, and the wiring-level test I first wrote for it proved nothing and is gone. The unit test that replaces it drives the matcher directly, where the input is mine to control, and it does fail when the normalisation diverges. Also adds an equivalence test across all six outcomes between the preloaded and awaited paths, so the two can never answer differently again. Restores the local name siteConfigSuppressed for the `-F` check; it was renamed along with the decision input, but at that site it really does mean only the one thing, and the union with the unreadable-file count happens one line later. 1,509 SSH tests pass. * test(ssh): check the matcher against a live OpenSSH client, not against my beliefs Every other test in the parser file states what I believe ssh does. These state what it did: an OpenSSH 10.2p1 client against a real sshd on 127.0.0.1:2222, with the client's own verdict recorded from its output, and ssh-keygen -H's own salt and hash pinned as a vector. Two assumptions the design leans on were worth more than an argument. THE FALLBACK PASS MAY NOT REPORT A CHANGE. We look up `[host]:port` first and retry the bare host, and only the first pass may answer `mismatch`. With StrictHostKeyChecking=accept-new, a bare line holding a DIFFERENT key, dialed on 2222, ssh connected and appended a new `[127.0.0.1]:2222` line — no IDENTIFICATION HAS CHANGED banner. It read that as first contact. Had we reported a change there we would refuse hosts ssh connects to happily, and the wrongness would have been invisible: refusing looks like the cautious choice. AND THE TYPE-SCOPING REJECTION IS NOT AN INVENTION. known_hosts holding an ssh-rsa key while the server offers ed25519 makes ssh print IDENTIFICATION HAS CHANGED and refuse. So unknown-type-known-host is neither stricter nor laxer than ssh — treating it as first contact, which is what a naive type-scoped lookup does, is the laxer mistake. That second result also fixes the message. ssh is blocked too, so `ssh-keygen -R <host>` is the remedy that unblocks both, and we were naming it only for a same-type mismatch — leaving this case with a diagnosis and no way out. Named now when known_hosts is the source that disagrees, and still not named when it is our own store, which ssh-keygen would not touch. 1,516 SSH tests pass. * docs(ssh): record the two assumptions a live client confirmed Both were load-bearing and neither was obvious: the bare-host fallback pass may not report a change, and unknown-type-known-host is what ssh itself does rather than something we invented. Getting the first backwards would have refused hosts ssh connects to happily, which is the failure mode that looks like caution. * style(ssh): satisfy the code-quality lints in the two new test files A string concatenation that should be a template literal, and an inline import() type annotation that should be a type-only namespace import — erased before vi.mock's hoisted factory runs, so the mock is unaffected. pnpm lint is clean. * fix(ssh): honour StrictHostKeyChecking, which had never once been read correctly `ssh -G` does not echo the value the user wrote. StrictHostKeyChecking is rendered through fmt_multistate_int, which prints the first entry of multistate_strict_hostkey, and that table lists true/false before yes/no: yes -> true | no -> false | off -> false | accept-new -> accept-new | ask -> ask Verified against OpenSSH 10.2p1 from both a config file and -o. Not 10.2-specific; the table ordering is old. The decision function tested only 'yes'/'always' and 'no'/'off' — spellings that cannot arrive. So `StrictHostKeyChecking yes` fell through to the default branch and we accepted AND PERSISTED a host the user's config explicitly says to refuse. That is the worst outcome this feature can produce, and it was the behaviour for every strict user from the first commit. `no`/`off` landed there too, breaking the documented "lax settings never persist" invariant. Every unit test passed throughout, because they fed the function 'yes' — the value a human writes, not the one that reaches the code. My ssh -G parity test did capture real output, but I happened to configure accept-new, one of only two values that round-trip unchanged. The new table is keyed on configured value -> what ssh -G actually prints, and asserts both reach the same verdict, so the question "is this the spelling that arrives?" cannot be assumed again. Found by a parity review against a live OpenSSH client. * fix(ssh): stop the fallback pass accepting a changed key, and refusing a new one One wrong loop scope, two opposite errors, both reproduced against a live OpenSSH 10.2p1 client and an ed25519-only sshd on 127.0.0.1:2223. ACCEPTING A CHANGED KEY. ssh runs the bare-host fallback only when the port-qualified lookup matched no plain entry of ANY key type. We ran it unless pass 0 produced a match or a SAME-TYPE mismatch. So with an off-port RSA entry plus a bare, correct ed25519 line — an ordinary shape, an old off-port entry beside one written by a port-22 connect — ssh printed IDENTIFICATION HAS CHANGED and refused, with no "checking without port identifier" in -v because the fallback never ran, while we reached the bare line and returned `match`. REFUSING A NEW ONE. sawKnownHostOtherType and sawCertAuthority were declared outside the pass loop, so an entry found only on the fallback pass could set them. A bare ssh-rsa entry, dialed on a non-default port against an ed25519-only server, made ssh add the host and connect — plain first contact — where we returned unknown-type-known-host and hard-failed. That is Gitea/Forgejo, dev containers, Gerrit, Vagrant: an off-port service on a host already in known_hosts. So the flags are per-pass now, and pass 0 decides as soon as it finds any plain entry for the host. Which of the two rejections it reports only picks the message; ssh calls both HOST_CHANGED. Also drops the type check from the match test: byte equality already implies the types agree, because the blob carries its own algorithm name and parsing rejects any line whose declared type disagrees with it. Reverting either half of the scope fix fails exactly the two new tests. 1,526 SSH tests pass. * fix(ssh): refuse the known_hosts lines ssh itself refuses to parse Three ways a line could be trusted by us and invisible to the user's own ssh — or, worse, raise a CHANGED alarm from an entry ssh drops. Buffer.from does not fail on bad base64, it SKIPS invalid characters, so `<key>!!!` and a blob with `@@` spliced into it both decoded to the correct key and matched. Verified live against OpenSSH 10.2p1 on 127.0.0.1:2224: the unmodified control reached authentication and all three malformed variants produced "No ED25519 host key is known". A re-encode-and-compare makes us agree. `<key>AAAA` is the interesting one, and the reason the first fix was not enough: 68 characters plus 4 is still legal base64, and the algorithm header still reads ssh-ed25519, so neither the base64 check nor the existing header check sees anything wrong. ssh parses the whole key structure. We decoded 54 bytes where an ed25519 key is 51 and reported `mismatch` — a man-in-the-middle warning caused by a typo in a file ssh silently ignores. So the blob is now walked as what it is: a run of length-prefixed fields that must consume it exactly. Algorithm-agnostic on purpose, so a key type we do not model is checked as well as one we do. It also rejects a length prefix that overruns the buffer, which readHostKeyType only checked for the first field. And ssh's extract_salt demands exactly one SHA1 digest — "expected salt len 20, got 16" — where we accepted any non-empty salt. A short salt is still a usable HMAC key for us, so a hand-crafted line could match for us and be a parse error for ssh. ssh-keygen -H always writes 20 bytes, so refusing loses no real entry. Worth recording that ssh-keygen -F cannot answer any of this: it matches host names and prints lines without ever decoding the key, so it reports "found" for all four blobs. The real client was the only instrument that worked. Found by a parity review; the base64 finding as reported was right about the behaviour and wrong about the mechanism for the padded case, which is what led to the structural check. 1,533 SSH tests pass. * fix(ssh): name a ssh-keygen -R target that actually removes the entry Verified against OpenSSH 10.2p1: with both `[h.example]:2222` and `h.example` on file, `ssh-keygen -R h.example` removes only the bare line and leaves the bracketed one — and there is no port flag, `-R host -p 2222` is "Too many arguments". An off-port target is keyed `[host]:port` in known_hosts, so the command we printed removed nothing: the user runs it, reconnects, and meets the identical failure with no indication of why. The message now names the bracketed form, quoted because the brackets are shell glob characters, whenever the port is not 22. Found by a parity review. * fix(ssh): read ssh2's host key algorithm list instead of copying it ssh2 builds DEFAULT_SERVER_HOST_KEY at load time and prepends ssh-ed25519 only when a RUNTIME PROBE succeeds — it signs and verifies with a fixed Ed25519 key. On a build where that probe fails, ssh-ed25519 is absent from ssh2's SUPPORTED list too, and generateAlgorithmList throws `Unsupported algorithm: ssh-ed25519` from inside client.connect. That throw matches no retry classifier and no transport-fallback classifier, so it is permanent — and because we only set `algorithms` for hosts we already know, it would fire on trusted hosts while new ones kept working. A copied list cannot be merely stale here; it can be wrong. So it is read from ssh2 now, which also removes the drift risk the previous commit could only report. ssh2 is external in the main bundle and the bundle is CJS, so the deep path resolves at runtime from packaged node_modules. The copy stays as a fallback in case a future ssh2 moves the file — losing the proposal order degrades the type-scoping guarantee, but refusing to connect at all is worse. The test that used to pin the copy against ssh2 now pins the fallback, which is the only part that can still drift. Found by an availability review. pnpm lint clean; 1,537 SSH tests pass. * fix(ssh): look a HostKeyAlias up the way ssh does — without the port HostKeyAlias suppresses the port entirely. Verified against OpenSSH 10.2p1 on port 2225 with HostKeyAlias=myalias: an entry keyed `myalias` authenticates, and one keyed `[myalias]:2225` gives "No ED25519 host key is known for myalias". We built [['[alias]:port'], ['alias']] and consulted a form ssh never writes. On its own that was a stale-entry false alarm. The previous commit made it worse: now that the first pass decides as soon as it finds any entry for the host, a leftover `[alias]:2225` line STOPS the bare lookup ssh actually performs — so the one population D2 cites HostKeyAlias for, bastions tunnelled through localhost:port, would get a hard failure on a host ssh connects to. So resolveKnownHostsLookupHost reports whether the name came from the alias, not just what it is, and that flag reaches both the matcher and the algorithm ordering. Returning the name alone is what made the bug invisible: the caller had no way to know it was holding something that must not be bracketed. Found by a parity review. 1,543 SSH tests pass. * fix(ssh): only claim the site config was suppressed when ssh -G actually ran sshGArgsForHost reports which arguments WOULD be used, not what happened. It returns the -F form whenever ~/.ssh/config exists and os.homedir() diverges from the passwd home, so a machine with no usable ssh at all — Windows without OpenSSH, a restricted sandbox, a timed-out probe — was judged by whether it happens to have a ~/.ssh/config, and rejected every unknown host permanently if it did. The same broken machine WITHOUT one stayed fully permissive, which is the tell: the flag is a claim about a config file we could not read, and when ssh never ran there is no such claim to make. Narrow but total where it lands: anything that sets HOME explicitly (wrapper scripts, sudo -E, devcontainers), macOS mobile and network accounts, and the E2E isolation this branch was written for. Found by an availability review. * fix(ssh): stop refusing hosts that ssh itself connects to Two product decisions, both taken deliberately after a review priced their blast radius, and both moving us from stricter-than-ssh to matching it. CERTIFICATE-AUTHORITY HOSTS NO LONGER FAIL. The point of an SSH CA is that the client holds ONE line — very often `@cert-authority *` — instead of per-host entries. That line matches every candidate, so for a Teleport / Vault-SSH / Smallstep / in-house-CA user EVERY target failed, not just CA-signed ones, including on-demand runtime VMs, and StrictHostKeyChecking=no did not help. The documented escape was an environment variable, which an Electron app launched from the Dock or Start Menu never sees. Meanwhile OpenSSH, verified live, treats a CA-covered host presenting a plain key as first contact and connects: ssh2 cannot validate certificates at all, so refusing bought nothing ssh was not already giving up. The residual risk is real and accepted — for a CA-protected host we take a plain key we cannot tie to the CA — and the ca-only outcome is carried through the decision so it stays visible in the log. AN UNREADABLE known_hosts NO LONGER REFUSES EVERYTHING. Any non-ENOENT read error on any configured file rejected every unknown host, with a message blaming the system SSH configuration, which was not what happened. The common trigger is not exotic: a Windows OneDrive Known Folder Move placeholder while offline fails with a cloud-file error, not ENOENT. It was also asymmetric with our own store, which degrades an unreadable file to "nothing trusted" and connects. We now connect as ssh does — it warns and treats the host as unknown — but record NOTHING, so a first contact we could not check never becomes durable trust. That second half is the reason the first is acceptable, so it is pinned end to end with the store actually bound. Which meant splitting verificationSourcesIncomplete back apart. It had been one flag for two claims that now diverge: "a site policy may exist that we cannot read" still refuses, "a file we could not open may contradict this" does not. Merging them was what made the second inherit a strictness only the first justified. Both still lose to evidence we DID read: a mismatch, a revoked key, or an explicit StrictHostKeyChecking still refuse in either state. pnpm lint clean; 1,549 SSH tests pass. * docs(ssh): correct the design where a live client disproved it D2, D3 and D4 each stated something about OpenSSH that turned out to be wrong when tested against a real client and sshd rather than read from the source. D3's premise is the notable one: OpenSSH is not type-scoped at all, so it does not avoid the RSA-era false alarm the way the doc claimed. It avoids the situation via order_hostkeyalgs and hard-fails when the situation arises anyway. The conclusion survives — the ordering is still what makes our scoping safe — but for a different reason than the one written down, and a reader would have drawn the wrong lesson. D2 gains the two rules that actually bite: the entry condition to the fallback pass, and HostKeyAlias suppressing the port. D4 records both reversals with their reasoning and the residual risk each one accepts, and the ssh -G spelling trap that made StrictHostKeyChecking dead on arrival. Corrections are kept visible rather than edited out, per the note at the top of the file. * fix(ssh): repaint the panes after a reconnect, not just reattach them Reported: disconnect an SSH host from the Remote Hosts popup, reconnect, and the terminals come back blank — but resizing a split or toggling the sidebar makes them render correctly. That last detail is the diagnosis. The panes were never broken: reattach restores each pane's buffer but not its painted frame. xterm repaints on a write or a resize, and a reconnect produces neither for a pane that was already correctly sized — so nothing paints until a relayout forces it, which is exactly what resizing or toggling the sidebar does. The renderer already has refitAndRefreshAllTerminalPanes for this shape ('after bulk desktop restore, background panes may have correct cols/rows but a stale xterm renderer until focus forces a repaint'). Its only callers were the mobile fit-reclaim paths; the SSH reconnect path never used it. Scheduled from finalizeHydratedTerminalPanes, on both a frame and a 100ms settled pass — the same pattern the desktop-restore path uses, because rAF alone lands while panes are still remounting. Mutation-proved: removing the schedule reddens the new test, which is the reported symptom. * fix(ssh): repaint background-tab panes revealed after a reconnect Completes 834a495038, which only fixed the ACTIVE tab. Reported: split panes of plain shells on another tab were still blank after reconnect until a divider drag or a sidebar toggle. The repaint did reach background managers — they stay mounted, only rendererVisible flips — but it could not land. A tab-hidden pane measures as a 0-size box, so canMeasurePaneForFit bails and the fit is a no-op, and refreshAllPanes marks rows dirty on a pane with no presented frame, which cannot repair a grid the reattach's direct terminal.resize left diverged. The reveal then takes the light resume path, which deliberately does not fit, and scheduleRevealRepaint only reattaches WebGL. So nothing ever fixed the geometry — and a divider drag or sidebar toggle is a real fit, which is why those appeared to work. Parks the repaint on a hidden manager and replays it on reveal, reusing the existing reveal-fit machinery rather than adding a mechanism. Flag-gated so the light path still does not fit in the ordinary case — 'does not fit on a light tab reveal' stays green. Splits are not special: the gap is per-manager, so it is identical for 1 or N panes. Splits just expose it, because users find the workaround (drag a divider) that a single full-tab pane rarely gets. A never-mounted tab is unaffected — it has no live manager and fits through the normal initial-fit lifecycle. Mutation-proved twice: removing the deferral, and reverting the reveal-side condition. Each reddens only the new tests. * test(terminal): pin that panes are PAINTED, not merely bound — and fix a broken commit Two problems, both mine. 1) 0103a80b48 swept in an untracked fixture and left the branch failing typecheck (unused Terminal import in painted-pane-fixture.ts). Its canvas stub also threw 'clearRect is not a function' on every refresh. Fixed here. 2) direct-ssh-reconnect-repaint.test.ts, which I wrote to guard the reconnect repaint, is VACUOUS: it re-implements finalizeHydratedTerminalPanes inside the test and mocks the registry, so deleting the real fix from useIpcEvents leaves it green. direct-ssh-reconnect-repaint-wiring.test.ts replaces that guarantee by capturing the real callback the hook hands the coordinator and running it against live panes — deleting the two scheduling lines now reddens it. The gap this closes: content survival was already well covered at the BYTE layer (snapshot roundtrip, hide/reveal stitching, cold-restore scrollback), but every pane test stubbed terminal as {cols, rows, refresh: vi.fn()}, so 'repainted' only ever meant 'a spy fired'. No test ran a real xterm through a real PaneManager. pane-content-survival.test.ts does, reading .xterm-rows — what the user actually sees — across reconnect, restart-shaped restore, tab reveal, window show, split and unsplit, for plain shells and alt-screen TUIs. The alt-screen distinction is now pinned explicitly: forcing a resize inside fitAllPanes reddens only the TUI test, because a plain shell reflows and survives while a TUI frame does not. That asymmetry is why the reported bug looked like a plain-shell problem. 11 tests, each mutation-proven to redden only its own. 760 pane-manager tests green; the 2 failures here are the known environmental IME baseline. Flagged, not fixed: the unsplit path reparents the DOM without the dispose/ reattach that splitManagedPane does explicitly because 'DOM reparenting can silently invalidate a WebGL context without firing contextlost', and follows it with a safeFit that no-ops when the box is unchanged. Same shape as the reconnect bug. happy-dom has no WebGL, so only a real-GPU E2E can confirm it. * fix(ssh): send the pane its screen back on reconnect A reconnect left every remote terminal blank. Measured on a live relay, not inferred: pty.attach returned no replay for every pane, taking the activation === 'existing' early return in the relay's attach. 'existing' means a source delivery is already open for this client, so it must already be receiving live output and cannot need its screen re-sent. That holds for a duplicate attach. It is false for a reconnect, for a reason neither side can see alone: the client keeps its id across the drop (detachClient refuses to detach the primary, and setWrite revives that same id) so the delivery outlives the dead transport, while the RENDERER has already thrown its terminal away. A reconnect bumps tab.generation, which is the pane's React key, so TerminalPane remounts and the old xterm is disposed with its buffer, and nothing on that path captures it first. Both halves are individually reasonable and together they guarantee a blank pane: the relay reports the client already has the screen, to a client holding a brand-new empty terminal, and nothing paints until new output happens to arrive. Resizing appeared to fix it only because a TUI redraws itself. So the client says which case it is. reattachSshPtySession is by definition painting into a new terminal, so it asks; nobody else does, and the early return keeps working for them. Optional on the wire, so an older relay ignores it and behaves exactly as it does today. Falling through rather than returning the replay inline is deliberate: the path below already drops the pending batched bytes that are also in the buffer, which is what stops the live delivery rendering them twice. Reproduced first as a test against the real dispatcher, source publication and PTY handler (the second attach for one client, which is what a reconnect is) and it fails on the exact symptom before the fix. A second test pins that a caller which does NOT ask still gets nothing, so this cannot become a double-render for the duplicate-attach case the early return exists for. Also updates four provider tests that assert the exact attach params. NOT yet verified in the running app; the log will show replay=true on reconnect. * chore(ssh): drop the temporary reconnect-replay diagnostic Served its purpose: it is what turned 'the panes look blank' into replay=false, replayLen=0 on every pane, and then into replay=true with real byte counts once the relay fix landed. The permanent log line keeps the boolean, which is the part worth having. * revert: drop the reconnect repaint commits; they cannot fix the blank panes Reverts 2fdab478c0, 34fc1424f0 and dc6f6bf685, which I cherry-picked onto this branch to test alongside the host key work. Their stated premise is 'reattach restores each pane's buffer but not its painted frame'. That is false for this flow: a reconnect bumps tab.generation, which is the pane's React key, so TerminalPane remounts and the old xterm is disposed WITH its buffer, and nothing on that path captures it first. Refitting and refreshing a terminal whose buffer is empty paints an empty pane. The blank screen was the relay declining to re-send the scrollback, fixed separately and verified on screen. Their tests pass without exercising the real case: the fixture blanks the painted rows and deliberately LEAVES THE BUFFER INTACT, which is the one situation that never occurs here, and the hidden-tab test replaces the pane manager with a stub that reports no panes. They may still address a separate symptom — a diverged grid after a resize on a hidden tab — but that is unproven, unrelated to this branch, and the originals are untouched on nwparker/sta-3077-fix-v3 where they came from. Carrying unproven renderer changes with a false premise in their message on a security-focused branch is not worth it. Reverting first and re-running the full two-step reconnect test is the point: the earlier verification passed with these present, so it did not establish that the relay fix stands alone. * fix(ssh): repaint a reconnected pane from the grid, not a byte tail A reconnect restored plain shells correctly but was reported to bring full-screen apps back as fragments of a frame — Claude Code showed a few rules and its cost line until a resize forced it to repaint. The two payloads are not interchangeable. Relay replay is a byte TAIL: it can begin mid-escape, and it misses the alt-screen enter, the clears and the absolute cursor positioning that built the frame, so replaying it into a fresh terminal paints whatever fragments survive. The model snapshot is a serialized GRID — which is what tmux repaints on attach, and the only payload that reliably restores a TUI. Orca already had the grid path and already preferred it; it was gated to PARKING. A reconnect needs it for the same underlying reason a park does: the pane paints into a terminal holding nothing, because a reconnect bumps tab.generation, which is the pane's React key, so TerminalPane remounts and the old xterm is disposed with its buffer. So the gate now admits both, and prepaintParkedSshSnapshot is prepaintSshModelSnapshot since parking is no longer the only caller. Deliberately NOT inheriting the parking kill switch: main keeps its headless model regardless of terminalSshViewParking, so a user who turns view parking off would otherwise be stranded on the tail. Every safety gate below eligibility is untouched, and pinned that way: null, renderer-sourced, sourceless, empty, and escape-tail-only snapshots all still degrade to relay replay, so widening WHY the model is trusted cannot widen WHAT is trusted and cannot regress to a blank pane. Reverting either half of the gate fails three of the new tests. HONESTY ABOUT WHAT THIS IS VERIFIED TO DO. I could not reproduce the corruption it targets. Two attempts against a live host, both on a build WITHOUT this change, both restored correctly: a freshly started Claude Code and Codex side by side, and an alt-screen `less` scrolled 4000 lines so its original full paint had aged out of the relay's 100KB tail. The reporter's case also involved pulling wifi — an abrupt drop rather than a clean disconnect — which is the one variable I cannot simulate here. So this is verified to be correct-by-construction and non-regressing: with it applied, the same scenarios still restore correctly (top live, less at its scrolled offset in alt-screen, both agent TUIs coherent). It is NOT verified to fix the reported symptom, because the symptom did not reproduce. Treat the symptom as open until someone confirms it on an abrupt drop. Also: top was a poor proxy for a TUI in my earlier verification precisely because it repaints every second and therefore self-heals within a tick. * fix(ssh): stop the reconnect prepaint firing after its mount is spent Regression I introduced with the snapshot-first reconnect paint. The payload path consumes mountFollowsTerminalPark — it clears the flag after the first reattach so a later in-place reconnect on the SAME mount cannot repaint. I replaced the prepaint's read of that mutable flag with a const snapshot of it, so my combined flag stayed true for the life of the mount. A snapshot could then be written on a later reattach, into a terminal that already had live content, and its own isCurrent() guard could no longer go false either. The visible symptom was a tab that came up blank with no prompt and stayed generically titled Terminal N — the title only stays generic when the shell never printed a prompt for Orca to read one from. Every such tab in my session had been through a remount; four tabs created cleanly with Cmd+T were all fine. So the flag is mutable again and is consumed alongside the one it was derived from. Both reasons a mount paints into an empty terminal — a park and a reconnect — are spent by the first reattach, which is what the original code meant. Worth stating plainly: my earlier claim that the snapshot change was non-regressing was tested only against reconnect scenarios. I never exercised creating a tab afterwards, which is exactly where this showed up. * test(ssh): cover what a pane SHOWS after a reconnect, and after a new tab The gap that let both regressions reach a user. Nothing asserted the rendered pane: the existing SSH coverage checks pty ids, statuses and spy calls, and every one of those was correct while the screen was blank. Covers one flow end to end against the dockerized relay: write a marker, reconnect, require the marker to still be on screen, then open a tab and require the new shell to answer. Three choices worth keeping: A MARKER, NOT A PROMPT. A prompt reappears on its own after a reconnect, so asserting one cannot tell restored scrollback from a fresh shell. The marker only exists if the pane kept what it had. ECHO, NOT EXISTENCE. The new tab must run a command and show its output. A pty id proves a session was created; it does not prove the pane is usable, which is the exact distinction the reported bug lived in. AND THE TAB TITLE. It stays 'Terminal N' only when the shell never printed a prompt for Orca to read one from, which is what the report showed and the cheapest signal available. Gated on ORCA_E2E_SSH_DOCKER=1 like the other relay specs. * chore(ssh): rename the snapshot prefetch off its park-only name The probe serves reconnect remounts as well now, so parkedSshSnapshotPrefetch described only half of what it holds. * revert: drop the snapshot-first reconnect paint; unproven and it regressed Reverts e6541fe9b8, its follow-up c497a26788, and the rename f680a0b1cb. The reasoning behind it still looks right — a byte tail cannot rebuild an alt-screen application, a grid snapshot can, and that is what tmux repaints on attach. What I could never do is show it fixing the reported symptom. Two attempts to reproduce the corruption on a build WITHOUT it both restored correctly: freshly started Claude Code and Codex, and an alt-screen `less` scrolled 4000 lines so its full paint had aged out of the relay's 100KB tail. Meanwhile it cost two real regressions. It fired on mounts that were not reconnects, leaving a new tab with no prompt and a placeholder title, which a user hit within minutes. The fix for that consumed the eligibility flag with the one it was derived from — and after it, a reconnected Claude Code came back as fragments of a frame, the exact symptom the change was meant to remove. So the consume-once semantics that stop stale paints and the repaint a reconnect needs are in direct tension, and I do not yet understand the ordering well enough to satisfy both. Shipping an unproven change that has already broken two things twice is worse than shipping the blank-pane fix alone, which IS reproduced, A/B'd and visually verified. The TUI corruption goes back to open — but now with something it never had before: a reproduction. It shows up on a reconnect against a Claude Code that has been running a while, not one just started, which is why my earlier checks kept passing. The e2e coverage stays. It asserts what the relay fix guarantees — a marker surviving a reconnect, and a tab opened afterwards reaching a shell that answers — and neither of those depends on this change. * docs(ssh): name the root cause the reconnect replay fix does not address requireReplay fixes the blank pane at the symptom. The cause is that a PTY source delivery is the only per-client relay state that outlives its client detaching: fs-handler, git-handler and relay-filesystem-watch-registry all subscribe to dispatcher.onClientDetached and release theirs, and relay-pty-source-publication never does. The primary client keeps its id across a transport replacement, so its delivery survives a dead transport and activate() answers 'existing' to a client that cannot receive anything. Retiring the delivery on detach is the real fix. Not doing it here is a choice, not an oversight — it is the flow-control and credit path, and I could not verify it before handing this over. Recorded in the test that guards the symptom, which is where someone changing this will actually look. * docs(ssh): record the three root-cause routes that do not work I went after the cause and failed three times. Each attempt looks correct until it runs, so the dead ends are worth more written down than the time they cost: RETIRING THE DELIVERY ON onClientDetached — the obvious fix, and the one I argued for, since fs-handler, git-handler and the watch registry all release their per-client state exactly there. It breaks checkpoint recovery: 10 tests across relay-pty-source-recovery-interleavings and restore-retry. A delivery outliving its client is DELIBERATE; that is what lets a reconnecting client resume from a checkpoint instead of re-receiving everything. This class omits the subscription on purpose, and that omission is not the bug. RETIRING WITHOUT session.cancelDelivery() — the credit ledger keeps one upstream owner per pty, so dropping the record without releasing it leaves the slot taken and the next open throws 'PTY source delivery already has an upstream owner'. I saw that live as an error toast over a blank pane. COMPARING clientGeneration — the delivery identity carries one, but it is client-supplied through pty.openClient and RequestContext has none to compare against, so the relay cannot tell the generations apart on its own. Which points where I would start next, unverified: the SSH client presents the SAME clientGeneration across a reconnect, so the relay cannot distinguish the new connection and reuses its delivery. reattachSshPtySession never sends sourceRecovery at all — the recovery protocol exists and the SSH reattach path simply does not participate in it. That is likely the real fix, and it is on the client, not in the relay. The symptom fix stays because it is verified and the tree is green; the cause stays open with a map instead of a guess. * fix(ssh): repaint a reconnected full-screen app from the grid A reconnected TUI came back as fragments of a frame — Claude Code showed a few rules and its cost line until a resize made it repaint itself. Relay replay is a byte TAIL. It can begin mid-escape and it misses the alt-screen enter, the clears and the absolute positioning that built the frame, so replaying it into the fresh xterm a remount just created paints whatever fragments survive. Main already keeps the thing that does restore a frame: a real @xterm/headless grid, alt-screen aware, fed unconditionally for SSH. Local terminals already repaint from it; SSH was the only path that did not. So this routes an SSH reconnect into the painter that already exists, at the one expression that chooses model over tail. No new call site, no second lifecycle, and every existing gate still applies — a null, renderer-sourced or empty snapshot still degrades to the tail, so it cannot paint blank. ONLY ON THE ALTERNATE SCREEN, and that is the whole design. The reconnect replay reaches the renderer without passing through main's model — forwardReattachReplay and the inline attach replay both bypass onPtyData — so at that moment the model is stale by exactly the outage. For a full-screen app that trade is right: a tail cannot rebuild a frame it no longer contains, a grid can, and the SIGWINCH the restore already sends makes the app redraw the delta. For a scrolling shell it would be wrong: the tail holds output the model never saw, and preferring the grid would drop it for good. A park has no such hole, so it keeps using the model either way. Derived from the PENDING retry, not directSshRetryAttempt. That also matches the live binding, which is written at the same tab generation once a reconnect succeeds and then outlives it — so it stays truthy for every later remount of that generation. Reading it directly is what made my first attempt fire on mounts that were not reconnects. Consumed alongside mountFollowsTerminalPark for the same reason. Verified live against the reported app: Claude Code restores identical to its pre-disconnect frame, top restores coherent and live, and a plain shell still shows output written before the disconnect. 5,470 tests pass across the touched suites. Not the whole story, and the remaining half is already written down: the model's gap exists because the SSH reattach asks for a tail instead of participating in the checkpointed resume the relay already implements. Close that and this paint is not merely coherent but exactly correct, for shells too. * test(ssh): cover a full-screen frame across a reconnect, not just scrollback The case a byte tail cannot serve, and the one that reached a user twice. A tail can begin mid-escape and misses the alt-screen enter and absolute positioning that built the frame, so replaying it paints fragments — which is what a reconnected Claude Code showed. Uses top: present on any Linux image, and it repaints on a fixed interval, so a whole header after the reconnect is unambiguous rather than a timing artifact. Asserts the header AND the column row, because a tail that lost the frame start still shows rows. The spec now covers all three payloads one reconnect has to get right: a shell's scrollback, a full-screen app's frame, and a tab opened afterwards reaching a shell that answers. * ci(e2e): actually run the Docker-SSH specs in the changed-specs lane "I am surprised this was not caught" has a mechanical answer: these tests do not run. A spec that reads ORCA_E2E_SSH_DOCKER test.skip()s itself when it is unset, and exactly one place in CI set it — gated on tests/e2e/ephemeral-vm-provisioned- root.spec.ts being among the changed files. So editing any SSH spec ran it as a skip and reported green. Eighteen specs reference that variable, including both reconnect regressions I have been chasing. Now it is also enabled when any changed spec references the variable, which is the same grep -l idiom the @headful check two lines below already uses. The original clause stays: that spec needs Docker without naming the variable, so replacing it rather than adding to it would have traded one silent skip for another. Simulated against the real files — the reconnect spec, the original trigger, a multi-spec change, a non-Docker SSH spec, and a deleted path — enabling in the first three, staying off in the last two, and not failing the step on a path that no longer exists. * fix(ssh): let the replay veto a stale alternate-screen belief Adversarial review of the previous commit found a case where it is worse than the bug it fixes, and it is the exact inverse of what that commit reasoned about. The model reports alternateScreen from bytes it consumed, and it never consumes the outage. So if a full-screen app EXITS during the disconnect — an agent finishes, a command ends, the process dies — the model still says alternate. The gate then painted a frozen frame of an application that no longer exists and, via the else-if chain, discarded the replay carrying the shell's real output. Frozen and wrong beats fragments, which were at least current bytes. The replay is the only witness to the outage, so it now gets a veto: its last 47/1047/1049 transition, if any, outranks the model's belief. Leaving reset means the frame is gone and the tail wins; re-entering means the model is right after all. Same review found the width-mismatch guard drops the alt frame and leaves a cleared screen for the app to repaint — free for a park with no tail to lose, but here it meant discarding a usable one for a blank pane, so that degrades too. Both vetoes are skipped when there is no replay, where they would only trade a stale frame for an empty one. Extracted as sshReconnectPaintsFromModel rather than more inline ternary, because every interesting case is a disagreement between a stale belief and a replay — awkward to stage end-to-end, trivial to state as a table. 14 unit tests, including the two that fail against the previous commit. The e2e comment is corrected in the same spirit: top redraws itself, so it never discriminated the paint source and should not have claimed to. Also from the review: the kitty flag stack was left stale on this path, since the app's pushes during the outage exist only in the replay we discard — scanned now, after the snapshot so the outage layers on the pre-outage baseline. And the consume-once comment asserted an invariant that does not exist; followsDirectSshReconnect is a const captured per connect, bounded by connectStarted and the gates rather than by the read. Corrected rather than restructured. Known and NOT fixed, because it predates this work and is a behavior change of its own: the model probe is gated on the terminalSshViewParking kill switch, so turning off view parking also silently disables this repaint. Defaults on. * docs(ssh): make the parking kill switch's reach over the reconnect repaint deliberate Review flagged that terminalSshViewParking silently disables the full-screen reconnect repaint, since both go through the same model probe, and that nothing said so. Keeping the coupling and documenting it rather than threading a reason through. The switch is the kill for painting an SSH pane from main's model at all, and a reconnect does exactly that; off should restore the relay-tail behavior that predates the machinery, which is what an escape hatch is for. That matters more than usual here: this repaint is new and review already found one case where it was worse than the bug, so a way to turn it off in the field is worth its cost — a user who disables parking also loses the reconnect repaint. The alternative is worse than it looks anyway: the probe memo is keyed on ptyId and shared with the park path, so a per-call reason would be reused by whichever path created it first. * docs(ssh): record why the obvious reconnect follow-up does not work I proposed making the pane-retry path request source recovery the way reattachKnownPtys does, and argued it was probably client-side routing. Tracing it says the wiring is indeed trivial and the checkpoint state does survive a drop — and that the change would still be wrong three ways, one of them harmful. The relay short-circuits to 'existing' on a same-clientId attach BEFORE it looks at the recovery argument, and the reconnecting client has already rotated the delivery onto its id. A failed reattachKnownPtys then deletes the checkpoint on purpose, so a later pane retry presents checkpointUnavailable, which becomes restoreRequired and then SSH_SESSION_EXPIRED_ERROR — trading a blank pane with a tail for a killed session. And the payloads answer different questions anyway: recovery replays the post-checkpoint delta to keep main's model whole, while the tail is a screen snapshot for a fresh empty xterm. Even a successful recovery would put almost nothing in a remounted pane. Also corrects the argument I had been leaning on hardest. "Old relays ignore requireReplay, so those users still get blank panes" is false for the SSH relay: the client deploys its own relay into a version-scoped directory and rejects any grant whose serverBuildId differs, because client and relay ship in one build. Mixed versions cannot occur on this channel. The independent-update rule still governs remote runtime hosts, just not this one — so there is no stranded population, and the urgency that framing created was imaginary. What replaces it is a sharper question. Source recovery is gated on outputFlowControl and on the client presenting a NEW clientId. We have empirical evidence it does not: the blank-pane bug existed because the relay concluded this client already held the stream, and the shipped fix works by bypassing that exact early return. If the id is reused, reattachKnownPtys' recovery hits the same short-circuit — meaning checkpointed recovery may never have run for SSH reconnects, and the tail is not a fallback but the only path. Whether that is so turns on daemon versus stdio-primary relay mode, which I did not verify and which decides whether the work is "extend recovery" or "recovery has never run here." * docs(ssh): the root cause — checkpointed recovery never runs on a reconnect Chasing why the pane-retry path could not request source recovery turned up the real answer: nothing can. Recovery is dead on every SSH reconnect, and the byte tail is not a fallback but the only path that has ever run. Five links, each read rather than inferred. setWrite reuses primaryClient including its id, so a reconnected client presents the SAME clientId. activate() tests exactly that at line 99 and returns 'existing' at 108, which makes the rotateDelivery branch at 118-142 reachable only when the ids differ — never here. So no sourceRecovery comes back, so finishSourceRecovery fails its !pendingRecovery guard and abandons, cancelling the delivery and deleting the checkpoint. The pane retry then opens fresh and takes the tail. This also explains the blank panes exactly. The relay concluded that this client already held the stream because, by its own identity rule, it does. The fix that implies is smaller than anything proposed so far and avoids what sank the three earlier attempts: bump a transport generation on the client record in setWrite and compare it alongside clientId, so a reconnect rotates the delivery instead of matching as 'existing'. Deliveries still outlive their clients and nothing retires on onClientDetached — the rotation happens on re-attach, which is what the recovery design already intends. RequestContext, setWrite and the publication are all relay-internal, and client and relay ship in one build, so there is no wire change and no compatibility exposure. Left explicitly unverified: whether rotateDelivery's identity preconditions hold at that moment, whether outputFlowControl is granted on the reconnected session, and what a rotation gives the RENDERER — which still remounts an empty xterm and needs a screen, not a post-checkpoint delta. Recovery keeps main's model whole; it does not by itself repaint a fresh terminal, so the tail may still be wanted for the pane even once the model stops going stale. * ci(e2e): run the Docker-SSH specs when SSH SOURCE changes, not just specs The earlier fix only helped when a spec file itself changed. Edit pty-connection, pty-handler or ssh-relay-session and touch no test — which is what every one of these regressions actually looked like — and the lane still did not run. pr.yml now maps SSH source paths onto five Docker-backed specs. Five rather than all fifteen because the rest are covered by unit tests that prove the same source without paying for a container; that is a deliberate narrowing and this comment is where it is admitted rather than left implicit. Test files are excluded from triggering, since they prove themselves. Simulated against the real paths this PR touches: pty-connection.ts, pty-handler.ts, ssh-relay-session.ts and ssh-pty-session-reattach.ts all now pull the SSH specs in, while pty-connection.test.ts, ssh-known-hosts.test.ts, SshTargetCard.tsx and README.md correctly do not. The gate contract test covers the mapping: 11 pass. e2e.yml pays for it — 30 to 45 minutes, because the lane can now build a container image and run SSH specs serially on top of whatever changed — and installs openssh-client, which the fixture shells out to and which the lane did not need back when it never received these specs. * test(ssh): make the reconnect spec actually run — it now fails on a real bug It had never executed once. The CI condition that enables Docker-SSH was gated on an unrelated spec, so this skipped and reported green — and running it for the first time found two bugs in the spec itself, both of which a typechecked tests/ would have caught instantly. startDockerSshRelayTarget returns a DockerSshRelayTarget, which has no targetId; the id comes from connectDockerSshRelayTarget's return value, which the spec discarded. So every reconnect call passed undefined and the relay answered 'SSH target "undefined" not found'. And openNewTerminalTabInActiveWorkspace takes the group to open into; called with no argument the new tab lands nowhere. The third problem was the fixture rather than the spec. The image ships Debian's /etc/bash.bashrc with the xterm title block commented out and an all-comments /root/.bashrc, so its shell never emits OSC 0 — which is what Orca derives a tab title from. The title assertion could not have passed for any shell, healthy or not, so it was proving nothing. enableDockerSshRelayTargetShellTitle opts a spec into the title-setting PS1 a real user's shell already has. IT STILL FAILS, and that is the point: it fails on a PRODUCT bug it was written to catch. An SSH reconnect destroys the terminal state behind a tab whose local creation has not yet reached the host. remote-workspace-session-merge.ts:86-89 spreads the host's tab list over the local one for that worktree, so a local tab missing from the host snapshot has no surviving branch; the upload that would have put it there is DROPPED rather than deferred inside the 1s suppression window after a snapshot apply. The tab bar still renders the tab, correctly titled, but the terminal slice holds one tab and no pane manager exists for the second — so the user clicks a tab that never paints, with no error and no recovery, while the process keeps running on the host. Pre-existing: none of remote-workspace-target-sync.ts, remote-workspace-session-merge.ts, use-app-session-persistence.ts or remote-workspace-snapshot-apply.ts is touched by this branch, and nothing in the merge range touches them either. Not worked around here. Waiting for the upload would hide it, and a user opening a tab right after a reconnect has no such signal to wait on. An earlier version of this message claimed the spec passes. It does not; I had seen five green runs out of six and generalised from them. Sustained runs are about three in eleven before the merge and zero in four after. * chore(e2e): add a typecheck entry point for tests/, unenforced for now tests/ has never been typechecked. That is how a spec could read target.targetId off a type with no such field, and call a function without its required argument, while the suite reported green — the spec was skipping, so nothing ever disagreed with it. Pointing tsc at tests/ finds both immediately. It also finds ~198 errors across ~94 files, which is a cleanup project rather than a change to make here, so this ships as pnpm typecheck:e2e and is deliberately NOT added to the typecheck chain or to CI. An unenforced script is worth less than a gate, but it is worth more than nothing: it is runnable, it is discoverable, and the header says plainly what it is so nobody mistakes it for coverage we have. runtime-types.ts is the one fix included, because it was actively misleading: every PaneManagerLike method was optional, so every call site was a possibly-undefined invocation that TypeScript could not help with. They are real methods on a real instance. Also widens AppStore to the StoreApi that window.__store actually is. * test(ssh): separate the reconnect paint guard from the tab-destruction bug The paint guard was failing about two runs in three, and after the merge every run, for a reason that has nothing to do with painting. It staged its full-screen check in a tab it had opened AFTER a reconnect — which is exactly the tab an unrelated session-sync bug destroys on the NEXT reconnect. Two independent failures were riding on one assertion, and the one that fired was not the one the spec is for. Running top in the ORIGINAL tab fixes it. That tab predates every reconnect, so it is in the host snapshot and survives. No assertion changed, none were weakened, and the new-tab case simply moves after the full-screen case rather than before it — it still opens its tab after a reconnect, which is the regression it exists to cover. Five consecutive runs pass at ~13s, against three in eleven before. The bug itself is not swept up. ssh-reconnect-tab-destruction.spec.ts records it as a fixme with the mechanism written down: session-merge spreads the host tab list over the local one, so a local tab missing from the host snapshot has no surviving branch, and the upload that would have put it there is dropped rather than deferred inside the 1s window after a snapshot apply. It is worse than a vanishing tab — the tab bar keeps rendering it, correctly titled, while the terminal slice has dropped it and no pane manager exists, so the user clicks a selected tab that never paints, with no error and no recovery, while the process runs on untouched. fixme rather than a workaround because waiting for the upload would hide it, and a user opening a tab right after a reconnect has no such signal to wait on. It is pre-existing: none of the four files in that path is touched by this branch, and nothing in the merge range touches them either. Also lifts openTerminalTab into a shared helper, since both specs need it and the group argument it must pass is the kind of thing worth stating once. * fix(ssh): stop a reconnect deleting local state the host has not seen Reported from a 60-second manual test: reconnect an SSH workspace and the app drops to the home screen, a second tab running pnpm install is gone entirely, and one launched agent is listed twice. Three symptoms, one cause. The snapshot is applied as the whole truth for the reconnecting target. The tab merge iterates only the host's worktrees, then the result is spread over a gap where every local tab for that worktree has just been dropped — so a tab created locally whose upload has not landed has no branch that keeps it. Not a race: it cannot survive. Same for the pointers, where a snapshot that names no active worktree nulls activeWorktreeId and activeWorkspaceKey, which is the home screen while the user's terminals are still running. So the host is now authoritative for what it knows and not for what it has never been told. A local tab absent from the snapshot is kept, the worktree union is used so a snapshot with no entry for it at all cannot erase it, and a null active worktree only defers to local state when that workspace demonstrably still exists in the merged result. Two guards this change had to earn rather than assume. A null activeTabId is NOT missing information — it is a deliberate deselect that arms the duplicate-tab repair, and my first attempt defeated it and broke that test; it is honoured verbatim now. And preserving by tab id alone reintroduces the duplicate agent, because the host can carry the same session under a new tab id, so the preserve also checks the remote session id — the identity that survives a tab-id change. Testing, which is the part that failed here before. Eight tests fail on the unfixed code and pass on this one, at two levels: the merge decision table, and the real apply path driven through a store. The end-to-end version of the same scenario is deliberately NOT the guard and now says so in its header — measured against unfixed code it only reproduces about one run in three, because the destruction needs the tab created inside the debounced upload's suppression window and nothing external can force that. Its earlier green run is exactly why this shipped. * fix(ssh): let agent session history recover once the relay is ready Reported against the adhoc build: a workspace whose editor was loading remote files perfectly still showed "SSH relay is not ready" and "0 shown · 0 recent" in the Agent Session History panel, permanently. That string is what the relay throws before it is ready, which is ordinary at startup and again for the window a reconnect leaves the session not-ready. The panel had three refresh triggers — mount, window refocus, and a newly seen agent session id — and none of them fire when the relay simply becomes ready. So a transient startup error became a stuck panel next to a workspace that plainly worked, which is why the report described it as broken while everything else was fine. The file explorer already recovers from exactly this, off exactly this signal, with the rationale written down at use-file-explorer-tree-load-effects.ts: it loads before SSH providers are registered, so it retries when sshConnectedGeneration bumps. This panel simply never did. Same idiom, same gate — only retries when there was a prior error, so a local workspace or one that already listed fine does not rescan every time some unrelated host connects. Two tests in the existing suite. The retry one fails on the unfixed code with "the panel never retried after SSH became ready"; the second pins the gate, since a retry that fires on every connection bump would turn one bug into a rescan storm. * fix(worktrees): name the create route when a raw filesystem error escapes A worktree create over SSH failed with a bare "ENOENT: no such file or directory, lstat '/home/neil/projects/orca-test1234'". That message names nothing. An lstat is Node's LOCAL filesystem, so hitting one against a path that lives on an SSH host means creation ran a local implementation for a remote repo — but the user cannot know that, and neither could I without re-deriving the routing by hand and then failing to reproduce it. worktrees:create picks between three implementations, and the order matters: isFolderRepo is consulted BEFORE connectionId, so a folder-kind repo on an SSH host never reaches the remote path at all. Which route ran, and what the repo looked like when it was chosen, is the entire diagnosis — and it is knowable exactly at the throw site, where the decision was just made. So it is stated there now: route, repo kind, connection id, path, and the original message. Deliberately additive and deliberately narrow. Only ENOENT/EACCES/EPERM are rewritten; a git failure, a relay-not-ready, or a validation error already says what went wrong and burying it under a worse message would be a regression. The original error is kept as `cause`, so anything matching on `code` or reading the stack is unaffected. This does NOT fix the reported failure — I could not reproduce it. On current code I created a worktree at that exact path, at a second path, with a leftover directory already present remotely (correctly suffixed -2), and with the SSH target disconnected (clean actionable error, no ENOENT). What it does is make the next occurrence identify itself in one screenshot instead of costing another investigation. * fix(ssh): recognise a missing path reported by the relay Creating a worktree over SSH failed with a raw "ENOENT: no such file or directory, lstat '/home/neil/projects/orca-test1234'". The path was the one about to be created, so its absence was correct. The caller asks exactly that question — remotePathExists returns false on ENOENT — and could not get an answer, so it rethrew at the user instead. The trace log settles where the error comes from, and it is not where I spent a long time looking. The stack starts at SshChannelMultiplexer.handleResponse: the lstat ran on the SSH HOST, and the failure travelled back as JSON-RPC. An lstat in an ENOENT message is normally Node's local filesystem, which sent me hunting for a local fs call on a remote path; there is none. handleResponse rebuilds the error as `new Error(msg.error.message)` and then sets `code` from `msg.error.code` — the TRANSPORT's numeric JSON-RPC code. Node's 'ENOENT' string code does not survive that, and isENOENT tested only for the string, so a remote missing path could never be recognised as missing. Every caller of that predicate asks the same question, so this was wrong for all of them, not just worktree create. The message is now consulted as well, matched on Node's full canonical phrase so a branch name or log line that merely contains the word cannot make an existing path look absent — that would silently skip a collision check rather than report one. Fixed on the client because it holds for every relay version, including ones already deployed; teaching the relay to send the original code would only help hosts redeployed afterwards. The two other copies of this predicate, in filesystem-rename-collision and git-discard-path-safety, are deliberately left alone: both run against a local filesystem — one inside the relay, one on the desktop — where the string code is intact and broadening would only add false positives. Seven tests, three of which fail on the unfixed code: the relay-rebuilt error, the same error through the IPC wrapper the renderer sees, and one carrying no code at all. * revert: drop the worktree-create error-context wrapper Written to make an unexplained ENOENT self-identifying when creating a worktree over SSH. The cause is now known and fixed — the error came back from the RELAY and isENOENT could not recognise it, because the multiplexer rebuilds a remote error with the transport's numeric code — so the wrapper is scaffolding for a solved problem. Worse, its central claim is false. It reported 'the remote (SSH) path failed on a local filesystem call', and the trace log shows the lstat ran on the SSH host, not locally. Keeping a message that asserts the wrong thing about the one failure it was built for is worse than not having it. 183 lines and a rewritten error at the IPC boundary, removed. * docs(ssh): drop a comment claim about older relays that is not true The requireReplay comment said the field is optional on the wire so an older relay ignores it. It cannot happen: the client deploys its own relay into a version-scoped directory and validateGrant rejects any grant whose serverBuildId differs, so client and relay are the same build by construction. The field IS optional, which is why the relay reads it as !== true — that part stands on its own and needs no story about versions. A comment asserting a compatibility property the code does not have is worse than no comment, because the next person plans around it. * fix(ssh): act on the host key review — three must-fixes and two hazards M1. A stale record of ours outranked known_hosts, so the remedy we print did not work. `ssh-keygen -R host` then reconnect leaves known_hosts holding the NEW key while our store still holds the old one, and the store was consulted first — the one state that cure produces was the one state we refused. Permanently, since nothing in the app clears the store. known_hosts now decides a match first, which concedes nothing: it is the artefact ssh itself obeys, so an attacker who can rewrite it has already won. Both directions of the precedence are pinned now; the rotation case fails without this change. That leaves one rejection known_hosts cannot cure — a host trusted only on first contact that later rotates its key. "Remove the saved key" named nothing a user could find, so it now names the store file. M2. A superseded attempt could put a passphrase prompt in front of a host we had just refused. The verifier deliberately does not record a rejection for an attempt nobody is waiting on, so nothing identified it and ssh2's generic handshake error walked the credential ladder. Guarded on the generation, which catches it whatever the error turned out to be. Deliberately NOT by rejecting with a cancellation: an existing test pins that connect() still reports the raw late-startup error, and that behaviour did not need to change to fix this. M3. Every unknown host was refused whenever HOME diverges from the passwd home — devcontainers, `su`, Nix shells, some corporate launchers — because `-F` makes ssh ignore /etc/ssh/ssh_config and being blind to a site policy was treated as reason to refuse. Being blind is only a reason to refuse if we cannot go and look, so it now asks ssh for the system config on its own and takes the stricter of the two. Only a probe that fails leaves the strict rule standing. Costs one `ssh -G` on the rare path that already needed -F. N1. A rejected key's fingerprint was still adopted, and the relay scopes install locks by it — locks keyed to a host we refused to talk to. N2. The fallback algorithm list re-introduced the throw its own comment describes. ssh2 prepends ssh-ed25519 only when a runtime probe succeeds, so on a build where that probe fails, proposing it makes generateAlgorithmList throw inside client.connect — and only for hosts we already know. Not reading ssh2's list is a reason to leave its defaults alone, not to guess: it returns null now and the caller skips reordering. 1555 tests pass in src/main/ssh. * docs(ssh): state the merge's real trade instead of claiming it has none The preserve comment said a genuinely closed tab is never in the local list, because closing removes it. True for a close on THIS client; false for one closed on another client sharing the host, where the tab is still local, still absent from the snapshot, and now kept. That is a deliberate trade, not an oversight — absence cannot distinguish 'never uploaded' from 'closed elsewhere', and the outcomes are not symmetric: keeping a tab a moment too long is recoverable by closing it, deleting a live one with a process in it is not. But the comment asserted the case could not arise, which is the kind of claim that gets planned around. Now stated, and pinned by a test so the next person can see it was chosen rather than missed. * fix(ssh): stop a newer host key store being silently downgraded The store writes a version and never read it back. A file from a future Orca would have had every record dropped by validateRecord — the shape would not match — and then been REWRITTEN as version 1, so a rollback silently discarded whatever that version knew. Trust records are user-owned state; losing them costs a first-contact prompt per host and, worse, re-establishes trust from nothing. v1 is the only place this can be made safe, because v2 cannot retrofit a v1 that already clobbers it. A newer file is now left alone: nothing is trusted from it, and trustHostKey declines to write rather than downgrade. The check sits inside the snapshot queue so it cannot be separated from the write by another writer, and declining is not an error the caller fails on — the key still verified, and the next connect re-derives the same decision from known_hosts. The test writes a version-99 file and asserts it is byte-identical afterwards; it fails on the unfixed code with the file rewritten as version 1. * perf(ssh): skip the reconnect snapshot probe the replay has already ruled out Every SSH reconnect paid up to the 750ms model-snapshot timeout, including the ones where the answer was discarded. The gate needs the snapshot's alternate-screen flag, so the probe looked unavoidable — but one of its two vetoes does not: if the replay shows the app LEFT the alternate screen, no snapshot can be used whatever it says. Asking that first costs a regex over the replay and removes the probe entirely for that case. It also shrinks the window that matters most: the await sits inside the structural replay coordinator with live PTY bytes deferred, and the payload can be superseded while it runs. Behaviour is unchanged — sshReconnectPaintsFromModel returns false for a null snapshot exactly as it did for a fetched one it then vetoed, and its tests still pin both vetoes. * test(ssh): cover the site host key policy probe It shipped untested. Three cases, and the third is the one that matters: a system config naming no policy answers 'ask', not null, because parseSshGOutput fills the OpenSSH default — and that distinction is exactly what the caller keys on. Null means 'we could not look', which is the only state that keeps refusing unknown hosts; a successful read that sets nothing clears the blindness without relaxing anything, since strictestHostKeyChecking leaves the user's value alone against 'ask'. I expected null there and was wrong about my own code; the test now records the behaviour rather than my assumption. Also pins that the probe passes the null device and terminates its args with -- so a host starting with '-' stays a host. * fix(ssh): three release blockers from the readiness review P1-1 was my own fix from the previous round, and it was wrong. I claimed `ssh -G -F /dev/null` reads the system config while excluding the user's. It does not: -F excludes /etc/ssh/ssh_config too, which sshGArgsForHost's own comment says and I quoted before contradicting. Confirmed live against OpenSSH 10.2p1 — plain `ssh -G` reports the sendenv lines from /etc/ssh/ssh_config, `ssh -F /dev/null -G` reports none. So the probe returned built-in defaults on every machine, the fail-closed guard never engaged, and a site-wide StrictHostKeyChecking yes was silently ignored while we accepted AND durably recorded a key the user's own ssh refuses. That is worse than the lockout it was meant to fix. There is no ssh-only way to ask this, so the file is read directly — and the question asked is deliberately weaker than "what is the policy". Anything ambiguous (unreadable, an Include that will not resolve, the directive present at all) answers yes and the caller stays fail-closed. Only a site config that demonstrably says nothing about host keys clears it, which is the common case that was being punished. Includes are followed, since macOS and most distros ship `Include /etc/ssh/ssh_config.d/*` and missing that would read as "no policy" on nearly every machine that has one. strictestHostKeyChecking goes with it: there is no separately-read site value left to merge. P1-2. `ssh -G` prints UserKnownHostsFile unquoted and space-separated even when the config quoted it — verified the same way. One path containing a space is therefore indistinguishable from two, and splitting shreds C:\Users\John Doe\.ssh\known_hosts into fragments that resolve to nothing. Every fragment misses with ENOENT, which reads as "absent" rather than "unreadable", so the user appears to know no hosts and a CHANGED key is accepted as first contact. The filesystem is the only thing that can disambiguate, so it decides: if no fragment exists but the rejoined path does, it was one path. A list where any fragment exists is a genuine multi-file config and is left alone. P1-3. oxlint is a PR gate and this diff failed it on two lines. Both fixed — including by splitting the replay on ESC rather than matching it, which is equivalent since every private-mode sequence begins right after one, and respects no-control-regex instead of suppressing it. That gate failure is on me twice over: I reported LINT clean repeatedly while filtering oxlint's output with a grep that could never match its `path:line:col: error` format. Verification is by exit code now. 5183 tests pass; each fix has a test that fails without it. * fix(ssh): the readiness review's P2s P2-1. activeRepoId and activeWorktreeId could describe different workspaces. The repo followed the host while the worktree came from local state, and it split in exactly the case the preservation exists for — "the host named no worktree" is precisely when it can still name a repo. All three active-* fields now derive from whichever worktree won, rather than each picking a source. The nested ternaries that hid it are gone. P2-2. The trust-source reads sit AHEAD of client.connect, and readyTimeout only covers the handshake — nothing wrapped attemptConnect. A home directory on a stalled NFS or SMB mount made readFile hang forever, leaving the connection wedged in `connecting` with no ladder entry and no recovery. Bounded at 5s, reusing the existing withTimeout helper. The fallback is the one an unreadable file already produces — evidence withheld, connect as ssh does but record nothing — not the far worse "no hosts known" that would let a changed key through as first contact. That helper absorbs rejections into its fallback, so the store's catch had to move INSIDE the timeout; wrapping the other way silently swallowed the warning that is the only signal the store is unwired rather than merely slow. P2-4. doSsh2Connect runs up to five times per attempt as the credential ladder advances, and each run re-read every known_hosts file, re-read the store, and re-scanned the system config. Nothing writes those while a handshake is in flight, so they are read once per attempt — which matters more now that each read can cost up to 5s. Keyed by connect generation rather than cleared, so a superseded attempt can never hand its sources to the live one. P2-3. forgetHostKey was exported, tested and referenced by nothing. The store-mismatch rejection now names the store file, so the case it was meant to cure has a cure without it; an exported API nothing can reach is unverified in production. Removed until D5 ships its UI, and the doc says so. P2-5 needed no change: the site-policy branch it called dead is reachable again now that the probe reads the real config. The design doc drifted from the code in the two places this review checks, and both are corrected: revocation now propagates for the ordinary rotation because a known_hosts match is decided first, and the -F blindness is resolved by reading the file rather than by refusing. * fix(ssh): rejoin a spaced known_hosts path even beside an ordinary one The whole-list check only fired when NOTHING in the reported list existed, so a config naming both a spaced path and an ordinary one kept the spaced one in fragments — the ordinary path existing was enough to leave it alone. The file the user actually verified their hosts in then never got read, which is the same failure the rejoin exists to prevent, just harder to notice. Longest run first now: the longest sequence of tokens that resolves to a real file is taken as one path and the scan continues after it, falling back to the single token when no run resolves. A genuinely absent path is still reported as-is rather than invented. The mixed case fails against the previous version. * test(ssh): pin the site config scanner's edge cases This control decides whether an unknown host is refused when we cannot see the site policy, and my first attempt at it was a security regression, so the cases that decide 'policy present' deserve to be written down rather than assumed. Seven, and each could have gone the wrong way. A commented-out directive must NOT read as a policy or the lockout returns for every distro shipping the line commented. A directive inside a Host or Match block MUST read as one, because no attempt is made to evaluate whether the block applies — guessing wrong in the permissive direction is the failure that matters. The equals form counts; StrictHostKeyCheckingExtended does not. A nested Include is followed, since a policy one level down is still a policy. An Include cycle terminates and answers false, which is knowledge rather than doubt: both files were read in full and neither mentions it. All seven passed as written, so this pins behaviour rather than fixing it. * test(ssh): assert tab survival, record the reattach gap rather than flake on it Running the two SSH e2e specs — which neither review executed — showed the tab-destruction spec failing on liveness three times out of three. The screenshot disproved the obvious reading: the marker was on screen, echoed by a live shell. getTerminalContent resolves the store's active tab id and returns '' when paneManagers has no entry under it, which is indistinguishable from 'the shell said nothing', and across a reconnect those two disagree. Scanning every mounted pane instead fixed the read, and then measured the real thing: three runs in four. The tab survives every time; the reattach behind it does not. So the merge fix is real and incomplete — the store keeps the tab, the tab bar renders it, and the pane sometimes never rebinds, which is the frozen-tab shape the original report described, one layer down from the deletion that used to cause it. Asserting that would put a one-in-four flake into the lane built to catch this class, and a lane nobody trusts is how the original silent-skip failure happened. So the spec asserts survival, which is deterministic at five runs in five, and the liveness gap is written down in docs/reference/ssh-reconnect-source-recovery.md with the first place to look. * fix(ssh): stop an unreadable host key store from wiping every pinned key Second readiness pass, checking each of the first pass's ten fixes rather than taking them on trust. Nine held. This is the one that did not, plus three fail-open shapes in the site-config scanner that a live OpenSSH disproved. P1 — the store. loadTrustedHostKeys returns [] for ANY read failure, and trustHostKey then wrote [...that empty list, newRecord]: one transient EMFILE followed by one first-contact accept replaced the file with a single record. Every other host re-TOFUs, and one whose key genuinely changed in between is accepted as first contact rather than refused — the exact outcome pinning exists to prevent. It also contradicted the doctrine this PR applies to known_hosts two files away, where a file that exists and refuses to open is evidence withheld. Fixed by classifying one read instead of guessing twice: readStore returns ok/absent/withheld, the read path flattens withheld to 'nothing trusted' so it still fails closed, and the write path declines. That subsumes the separate newer-version probe, so trustHostKey now reads the file once inside the queue rather than twice. The 'Trusted host key' log moved inside the branch that actually writes — it was already claiming success on the newer-version path. P2 — the site-config scanner documents 'doubt wins on every path' and had three where it did not, each the same shape: a path resolved WRONG still resolves to something, and a nonexistent Include reads as 'nothing there', which is indistinguishable from 'no policy'. Verified against OpenSSH 10.2p1: relative Includes resolve against a fixed dir, not the including file's, so a directive two deep was missed; ? and [...] are globs it honours; ~ and %-tokens expand before use. All three now answer doubt. P2 — credential prompts are gated on the attempt generation in one place rather than per rung. A superseded attempt is denied without a recorded decision, so isHostKeyVerificationError reads false and the ladder ran on to prompt for a passphrase nobody was waiting on. P2 — resolveKnownHostsFiles is async. Its rejoin existsSync-scans the very paths the 5s bound protects, and sat outside it as an eagerly-evaluated argument, so a stalled NFS/SMB mount blocked the whole main process. Tests fail against the pre-fix code: 2 for the store wipe, 3 for the scanner. Also: the 4 IME failures I previously reported as pre-existing main breakage were a stale node_modules — the xterm patch from #14758 was not applied here (402,643 bytes installed vs 403,181 expected). pnpm install applies it and all 4 pass. The OSC8 and SFTP failures were the same cause. * fix(ssh): make the merge non-duplicating, and close the last scanner hole Third readiness pass. Three P2s, all fixed. The merge one is the one I most wanted a verdict on, and it is real: hostUnknown filtered against ids the HOST knows and never against ids this same merge had already emitted, so a tab id local state holds under two worktrees was re-added under both. Two panes then share one terminalLayoutsByTabId entry and one remoteSessionIdsByTabId entry — one remote PTY — plus an activeTabId that never converges, which is the self-retriggering repair loop active-tab-owner-worktree .ts exists to mitigate (React #185). This PR does not create that state. It used to DESTROY it, by deleting every local tab under a replaced worktree, and keeping live panes cost that accidental cure. So the guarantee is made explicit rather than incidental: the merge now never emits one tab id twice, whatever it is handed. The active worktree is walked first so the surviving copy is the one the user is looking at, which is the owner resolveActiveTabOwnerWorktreeId already prefers — merge and repair now agree instead of each picking differently. Scanner: an Include path that is quoted AND contains a space was split before it was unquoted, so both halves missed and two absent paths read as 'no site policy'. OpenSSH honours that form -- 10.2p1 applies an Include of a quoted spaced path -- and it is likelier on Windows. Quote-aware splitting rather than 'any quote is doubt', because answering doubt for an ordinary quoted Include with no space would reinstate the lockout this scanner exists to avoid. An unclosed quote is doubt. Unquoted spaces still split, which is also what OpenSSH does. The reconnect paint gate took the replay and re-scanned it, having already been scanned by the caller that decides whether to fetch a snapshot at all — two full splits of up to 100KB per pane per reconnect. It now takes the transition. hasReplay is passed separately because it cannot be inferred: a replay with no mode change and no replay at all both give null. Tests fail against the pre-fix code for the merge and all four scanner shapes. Correcting my own evidence claim from last round: of the two store tests, only the wipe one fails pre-fix. The other guards the asymmetry the fix creates and passes either way — worth keeping, but I should not have counted it. * fix(ssh): honour every Include quoting form OpenSSH does Fourth readiness pass. Two findings; one fixed, one deliberately not, with the evidence for refusing it. The tokenizer modelled double quotes only. A live 10.2p1 honours single quotes and backslash-escaped spaces too, and both fell into the same silent fail-open the double-quote case was raised for: fragments that resolve to nothing, and 'nothing there' is indistinguishable from 'no site policy'. The escape is limited to a backslash before whitespace, NOT a general one. A general escape would be catastrophic on the platform this most needs to be right for: the Windows site config lives at C:\ProgramData\ssh\ssh_config, so it would eat every separator in an Include beneath it and resolve to nothing -- reintroducing the fail-open it was meant to close. The test for that is discriminating rather than incidental: it gives the file a literal backslash in its name, so a swallowed separator resolves elsewhere and fails, where a plain 'expect false' could not tell the two apart. Verified it catches the naive version, and that the other two catch the old tokenizer. NOT fixed: the non-duplication guarantee still stops at the worktrees the merge rewrites. A worktree that is neither replaced nor named by the host is never walked, so a duplicate straddling that boundary survives. Extending the guarantee to the assembly point was implemented and REVERTED. Any rule there has to pick a survivor, and the ones available are wrong during a worktree-id change -- which is the very thing that produces these duplicates. Preferring the active worktree keeps the OLD id's copy at the moment a rename lands, because the active worktree has not moved yet; the new worktree was left with no tabs and its groups were never created. remote-workspace-snapshot-duplicate-tab-repair.test.ts caught it, which is the only reason I know the stronger version was wrong rather than merely bolder. A surviving duplicate is mitigated by active-tab-owner-worktree.ts; deleting the tabs of the worktree the user is about to land in is not. The comment now claims only what holds, and says why it is not stronger. Also records the exit from the isENOENT message-matching trade in remote-wire-compatibility.md, where someone touching the relay error path will be standing. * fix(ssh): expand Windows OpenSSH's __PROGRAMDATA__ token in known_hosts paths Captured real 'ssh -G' output from a Windows host rather than reasoning about it, which is the one thing that could not be inferred from the POSIX format. Two things came back that the code did not handle correctly, and one of them is the security failure mode this work exists to prevent. Native Windows OpenSSH prints the system paths with its own token UNEXPANDED: globalknownhostsfile __PROGRAMDATA__\ssh/ssh_known_hosts __PROGRAMDATA__\ssh/ssh_known_hosts2 userknownhostsfile C:\Users\neil/.ssh/known_hosts C:\Users\neil/.ssh/known_hosts2 Passed through as a literal path, __PROGRAMDATA__\ssh/ssh_known_hosts misses with ENOENT -- and an absent file is deliberately treated as 'no host is known there' rather than 'evidence withheld', because that is the normal state. So a site-managed known_hosts on Windows was silently invisible: every host in it read as first contact, and one whose key an admin had rotated produced a TOFU accept where it should have produced a mismatch. Now expanded from process.env.ProgramData, and left literal when that is unset rather than guessed -- a wrong path reads as absent, which is the very failure being fixed. The second finding is reassurance rather than a bug: separators are MIXED within one path (C:\Users\neil/.ssh/...), which Node's fs accepts on Windows, and a spaced home prints unquoted exactly as it does on POSIX. So the space-rejoin design is confirmed against the real format rather than assumed -- its motivating example, C:\Users\John Doe, splits the way the rejoin expects. The captured output is pinned as a literal fixture. Parsing and the rejoin are pure string work, so this covers the input shape honestly off Windows; it does not pretend to cover the platform's path arithmetic. The expansion test fails without the fix. Also confirms C:\ProgramData\ssh is the right site-config directory -- it exists on the host, empty -- so the scanner is looking in the right place. * fix(ssh): branch Include backslash handling on platform, both halves measured Fifth readiness pass found that the previous narrowing traded one fail-open for another. Both rules are right, on different platforms: POSIX 10.2p1: Include conf\.d/x.conf resolves as conf.d/x.conf four backslashes needed to survive as one -- argv_split and glob() each consume a level Windows: Include C:\Users\...\x.conf resolves, separators intact So a backslash before an ordinary character ESCAPES on POSIX and SEPARATES on Windows, and either rule applied everywhere fails open on the other platform. Preserving on POSIX means looking for a path with a literal backslash, missing, and reading 'no site policy'. Answering doubt on Windows means every absolute Include is doubt, which is the lockout the scanner exists to avoid. Now branched. POSIX answers doubt rather than emulating two rounds of glob escaping for a question this coarse -- a backslash in a POSIX system config path is vanishingly rare, so fail-closed costs nothing there. The review offered the Windows half as a reasoned assumption and flagged it as such. It is now measured on a real Windows host instead: backslash separators resolve, AND an escaped space still escapes amid them (C:\Users\neil\sshprobe\sp\ ace\x.conf -> port 2802), which is exactly the rule implemented. Two other worries were checked and came back unfounded -- a backslash-space inside EITHER quote is consumed by ssh, and a single quote inside double quotes is an ordinary character, which the single quote-state variable already reproduced. The tokenizer takes the platform as a parameter, so both sides are pinned from one host. Every expectation in the new oracle came from running a real ssh and reading what it resolved to, not from reading source or shell convention -- the tokenizer's whole job is to agree with ssh about which file it would read. Also narrows an overclaiming comment: the dedupe set is consulted only by the host-unknown filter, so a duplicate in the HOST's own snapshot still propagates. Pre-existing and unchanged; the comment now says what the code actually does. * fix(ssh): only expand __PROGRAMDATA__ when it is a whole path segment Found by probing the expansion I had just written, rather than by reading it: a bare startsWith also matches a path that merely BEGINS with those characters, so __PROGRAMDATA__evil/known_hosts was rewritten to C:\ProgramData\evil\known_hosts -- a directory the user never named. Same prefix-collision class I checked the site-config scanner for and then did not check here. Low reachability, since the token only appears because Windows OpenSSH emitted it, and it emits it as a whole segment. Fixed because the expansion is one review pass old and sits in the security path: a rewritten known_hosts path resolves somewhere unintended, and a path that resolves to nothing reads as 'no host is known', which is the fail-open this whole line of work has been closing. Now requires the token to be the entire path or be followed by a separator -- both separators, since the path is Windows-shaped but may be parsed anywhere. The test fails without the check. * test(ssh): split the pty provider spawn tests into their own file CI's static analysis went red on the merge of main: ssh-pty-provider.test.ts reached 803 counted lines against a maximum of 800. Both sides contributed -- main grew the file and this branch added 7 lines to it -- so neither shows the violation alone, which is why local lint stayed green until main was merged in. AGENTS.md forbids disabling max-lines or bumping a per-file limit, and that rule is right here: the file was doing two jobs. Spawn owns the startup contract -- ingress version, env scrubbing, execution ownership, and the reconnect races -- and is 630 of the 922 lines. It reads as its own unit rather than as an overflow file, so it moves to ssh-pty-provider-spawn.test.ts and the shared relay stub moves beside it under a name that says what it is. Same tests, same count: 713 provider tests pass, and the line total is unchanged across the two files. * fix(ssh): remove the dead lint suppressions, and reach Terminal 1 in the restore spec Two CI failures, both surfaced by this branch rather than caused by it. Static analysis: the two no-require-imports suppressions on the ssh2 constants require() are now unused -- main's config no longer reports that rule there -- and the changed-code audit treats a dead directive as an error. Removed; the audit CI runs passes locally on the merge. E2E: ssh-cold-activation-restore failed on clicking Terminal 1. This PR is what routes that spec into the changed-e2e lane at all -- before, the Docker-SSH specs only ran when someone edited a spec file, which is the gap this branch set out to close -- so its first run in CI was here, and the failure is pre-existing rather than new. The trace shows the cause: six restored tabs overflow the strip at CI's window size and the restore pins it to the END, so Terminal 1 sits outside the scroll viewport. Playwright's own scroll-into-view loses that race against the sticky-to-end effect and times out on an element it can see but never reaches. The spec's intent is to activate the first tab and prove it remounted, not to exercise strip scrolling, so it now scrolls the strip to the start first. Not papering over a product bug: the strip is a native overflow container with working arrow controls, so a user can reach the tab -- it is Playwright that cannot drive a moving target. Six specs pass locally in CI's exact order and worker count. * test(ssh): press the restored first tab directly instead of waiting for it to hold still The previous attempt swapped a click for scrollIntoViewIfNeeded and hit the same 30s timeout, which identifies the real cause: not that Terminal 1 is out of view, but that it never holds STILL. Both APIs wait for the element to stop moving, and the strip keeps re-laying-out while the relay reconnects behind it -- so both time out on an element they can see and never settle on. Driving the pointer directly needs no element to be stable, only to be somewhere at the moment it is pressed, and the attempt is retried against the store rather than believed. Activation is deferred to pointerup and suppressed past a drag threshold, so it has to be a real down/up pair at one position -- a synthetic click event would not select the tab at all. Passes twice locally. The previous version also passed locally, so the honest statement is that the local runs prove the interaction still works, not that they reproduce CI's instability -- CI is the oracle for that.	2026-08-17 16:40:01 -07:00
Neil	9367169888	refactor(tests): split every oversized test file off the max-lines suppression list (#14728) ... * refactor(tests): split oversized test files off the max-lines suppression list Every `*.test.ts`/`*.spec.ts` that carried an `eslint/oxlint-disable max-lines` directive is now split into focused, behavior-scoped suites that fit the 800-line test budget, with shared setup extracted into co-located `*-test-harness.ts` / `*-test-fixtures.ts` modules (300-line budget). 83 files became ~930; the largest output is 797 effective lines. `orca-runtime.test.ts` is intentionally untouched. Test bodies were moved by scripted line-range slicing rather than retyped, so assertions are byte-identical. The only permitted body edits were mechanical rebinding where a shared value moved into a harness (e.g. `tmpHome` -> `homes.tmpHome`). Registries that enumerate test files were updated in lockstep: - config/max-lines-baseline.txt: pruned 341 -> 258 entries (all 83 removed). - config/reliability-gates.jsonc: 33 gates repointed at the split files, with assertionRefs split per file where a gate's coverage now spans several. - .github/workflows/pr.yml: the real-zsh lane now lists the 4 split files that actually exercise zsh, so they keep running in the dedicated shell lane. Also renamed agent-hooks `server-test-fixtures.ts` to `server.test-fixtures.ts` so the global-fetch call-site audit keeps skipping it, and added `.js` extensions to the CLI suites' dynamic harness imports (node16 resolution) to unbreak `build:cli`. Verification: full suite 52,449 passing vs 52,448 at baseline with zero assertions lost; `pnpm lint`, `pnpm typecheck`, and `pnpm build:cli` all exit 0; the terminal-pane e2e spec runs 31/31 headless. * refactor(tests): split hook-idle arbitration suite that oxfmt pushed over budget The pre-commit oxfmt pass reflowed pty-connection-hook-idle-arbitration.test.ts to 811 effective lines, 11 over the test budget. Split the hook-completion side effect and replacement-agent veto cases into their own suite; both files now sit well under the cap and the 15 tests are unchanged. * test: port upstream test changes into the split files after rebase Rebasing onto main surfaced 27 tests that main had added to files this branch deleted, plus edits to tests that had already moved. Taking the deletion side of those modify/delete conflicts would have dropped that coverage silently, so each upstream change is ported into the split file that now owns the behavior — for example main's six orchestration mailbox tests land across orchestration-runs, -send, and -check. Also repoints `orchestration.notification-mailbox-consistency`, a gate main added after this branch's gate remap, at those same three split files, and re-prunes the max-lines baseline against main's (257 entries). Verified: all 27 upstream test titles present; full suite 52,761 passing with the only diff vs baseline being 12 tests main itself removed and 3 that moved from skipped to passing; lint and typecheck exit 0. * fix(test): flush pending continuations before tearing down terminal test globals CI shard 5/16 failed on both Node 24 and 26 with `ReferenceError: window is not defined` from pty-connection.ts, surfacing through pty-connection-daemon-snapshot-replay.test.ts. The reattach/settle chains `await` a real promise and then touch `window.api`. Under fake timers those continuations cannot run, so they only become schedulable once restoreTerminalTestGlobals() switches back to real timers — which previously happened immediately before `delete globalThis.window`, so a late continuation threw and failed the whole file. Flush async ticks in that window instead. This is latent in the source rather than new: the pre-split 25k-line file kept running other tests after these, which gave the chains time to settle before teardown. Splitting the file moved teardown directly behind them. * fix(test): keep an inert window after terminal test teardown instead of deleting it The async-tick flush was not enough: the reattach/settle chain can resolve after teardown regardless of how long we drain, so CI shard 5/16 still failed with `ReferenceError: window is not defined` from pty-connection.ts. A real renderer never loses `window`, so deleting it was the artificial part. Swap in an inert proxy whose properties resolve to callables and whose calls resolve to undefined, making a late `window.api.pty.*` call a harmless no-op. The next test replaces it wholesale via installTerminalTestGlobals(), and no test asserts that `window` is absent.	2026-08-15 00:54:20 -07:00
Jinjing	637c7e94c9	Add SSH config host picker to add-host dialog (#12334) ... * feat(ssh): add SSH config host picker for add-host form Users can now click 'Fill from ~/.ssh/config…' to browse available SSH config hosts in a picker, select one, and have the form automatically prefill with resolved connection details (hostname, port, username, auth). Previously, an 'import' button provided bulk sync on this form—confusing and unhelpful when everything was already synced. That action is now available as a secondary 'Add all' option in the picker. * fix(ssh): import filter preservation and label fallback - Reuse search loader on import completion to preserve active filter inside generation guard - Fall back to hostname when manual host has no label, not empty string - Make alias duplicate detection case-insensitive to match config picker behavior - Validate host availability when restoring project group selection - Add aria-selected attribute to picker options for accessibility * fix(ssh): harden config picker import, alias folding, and host targeting Review findings on the ~/.ssh/config picker + bulk add: - Guard config-host resolution with a generation counter so a late resolve cannot overwrite a later pick or a form the user backed out of; freeze the other rows while a pick resolves. - Stop "Add all N" from re-adopting deleted hosts — it now imports without reAdopt, matching the new-host count it advertises. Settings → Import keeps the explicit re-adopt path. - Fold SSH aliases through a shared normalizeSshConfigAlias for import ownership, delete tombstones, reclaim, picker search, and the save-time duplicate check, which now occupies configHost *and* label like the picker. - Persist GSSAPIAuthentication only when a parsed Host entry asks for it, not when `ssh -G` merely echoes the /etc/ssh system default. - Fail closed with unavailable/setup-not-found when an explicit projectHostSetupId names a non-actionable host instead of silently creating the workspace on a sibling host. - Cache the parsed config for the picker session (refresh on open/retry) so filter keystrokes no longer reparse and Include-expand the file, keep the filter usable during loads, add a Retry on load errors, explain an empty Identity file after a config fill, and drop the always-false aria-selected. * refactor(ssh): centralize host result limit and extract folder group val Move SSH_CONFIG_HOST_RESULT_LIMIT to shared types so the renderer's limit message cannot drift from the host's query limit. Extract findActionableFolderProjectGroup to avoid repeating the folder-host-availability check across the composer hook. * fix(ssh): pass -F to ssh -G when HOME differs from passwd home In E2E tests and sandboxes, isolated HOME can differ from the system passwd home. OpenSSH resolves the default config via getpwuid (passwd), while Node's loadUserSshConfig uses os.homedir() (HOME-aware). Pass -F to explicitly specify the config path when they diverge, so ssh -G and the picker resolve the same file. * fix(ssh): verify config host exists before resolving with ssh -G When a user edits ~/.ssh/config and removes a host, the import picker should not fall back to ssh -G's echoed response (which treats any alias as valid). Check the reloaded config file before resolving. - Force reload config on each resolve to catch user edits post-open - Reject aliases not in the current config before calling ssh -G - Add test for deleted alias edge case - Fix workspace-target fallback to honor explicit host selection * fix(ssh): let tombstoned aliases be re-picked in the config picker Allow users to reclaim a deleted SSH host by re-picking it from ~/.ssh/config. Tombstoned aliases now appear in the picker with a "Removed from Orca" badge and remain pickable, but don't count toward "Add all" operations — ensuring passive import never resurrects a deleted alias while still giving the user a recovery path.	2026-08-03 17:32:13 -07:00
Jinjing	a000839465	Add first prompt to agent session history rows (#12085) ... * Add first user prompt to AI Vault session history rows Re-parse transcripts on demand to extract and display the untruncated first user prompt for copy/reuse. List scans omit the body (payload/perf); UI loads it when session details expand. Grok sessions extract the typed ask from <user_query> envelope, skipping injected <user_info> bootstrap rows. Supports Claude, Codex, Grok, and OpenCode agents. * fix(ai-vault): split SessionTime out to pass max-lines lint AiVaultSessionDetails exceeded the 400-line oxlint limit after adding first-prompt UI; move SessionTime into its own module. * fix(ai-vault): handle corrupt transcripts and fix OpenCode prompt captur Corrupt transcripts now resolve null instead of rejecting the IPC call, matching behavior for other unavailable cases. OpenCode SQLite parsing now correctly captures all text parts from the earliest user message only, fixing truncation of large prompts and padding of small ones. Add stale-response guard in the UI to prevent late results from overwriting the current session when tabs switch. Consolidate text slicing via `sliceAtCodeUnitLimit` to avoid surrogate-pair splits across all callers. * test(ai-vault): add first-user-prompt UTF-16 safety tests Ensure truncation at safety limits doesn't split UTF-16 surrogate pairs, preventing corruption of astral characters in captured prompts. * fix(ai-vault): key first-prompt-card by session.id Remounting the card on session switches prevents late responses from a previous load from writing stale data into the component's refs. Also improves conversation-turn key stability. * fix(ai-vault): preserve first prompt after preview truncation * refactor(ai-vault): improve first user prompt capture robustness and per - Add 15s timeout to full-prompt load to prevent indefinite loading states - Extract seedFullFirstUserPrompt helper for reuse across parsers - Prevent AI-generated summaries from becoming the copyable first prompt - Fix truncation detection in OpenCode SQLite by probing for N+1 rows - Optimize text bounding to apply safety limit before toLowerCase - Gate synthetic OpenCode path detection on agent type, not just # presence - Add test coverage for remote execution host handling * Fix FirstPromptCard loading state stranded by stale promise reuse Clears loadPromiseRef during cleanup to prevent the dedupe handle from causing StrictMode remounts to await stale in-flight requests. Stops loading when session becomes non-loadable mid-request. Adds tests for StrictMode double-invoke resolution and main-process timeout scenarios. * refactor(ai-vault): split session parsers into modular files Split secondary-parsers into individual files per agent type (copilot, cursor, hermes, opencode) for improved modularity. Add test coverage for first-user-prompt envelope handling: unwrap user_query tags and reject bare user_info dumps. * fix(ci): clear max-lines and flaky portal readiness check Collapse an accidental multi-line regex wrap in ssh-connection-utils that pushed counted lines to 301. Harden the latched-readiness test's ready transition so CI load can re-observe attach after MutationObserver gaps. * fix(ssh): extract proxy command helpers to pass max-lines Move resolveEffectiveProxy/spawnProxyCommand out of ssh-connection-utils so oxfmt line wrapping cannot push that file over the 300-line lint cap. * capture first user prompt by ordering OpenCode messages by creation time - Add `readOpenCodeMessagesInOrder` to rebuild transcript by timestamp, handling corrupt/partial files gracefully instead of discarding sessions - Extract SSH proxy command tests to dedicated file; add backpressure handling and stderr draining to prevent proxy process stalls - On Windows, reject unsafe characters in ProxyCommand values instead of pretending to escape them; properly format cmd.exe invocation with verbatim arguments - Expand ProxyJump chains into -J plus final hop, mirroring OpenSSH behavior - Decouple portal readiness reapply budget from flip-count budget via explicit constant	2026-08-02 18:45:55 -07:00
Jinjing	ce5b639e03	fix(P1-B): recover SSH targets and remote file watchers after a network drop (#12032) ... * fix(P1-B): recover system-SSH targets after a network drop Two defects stopped a remote workspace auto-recovering after a blip. runReconnectAttempt classified failures with isTransientError, which only matches ETIMEDOUT/ECONNREFUSED/ECONNRESET by errno code or literal substring. The system-SSH transport — the only transport FIDO2 and ProxyUseFdpass targets can use — reports network failures as OpenSSH prose ("System SSH connection timed out"), so the ladder published a permanent 'error' on the first timeout and the target never came back without a manual reconnect. isTransientReconnectError adds a network-shaped prose table on top of isTransientError and is used only on the reconnect path: connect() keeps the narrow classifier so an unreachable host still fails fast instead of burning five 30s attempts and five security-key touch prompts. Auth and passphrase failures stay permanent on both paths. runReconnectAttempt also had no generation fence, so a superseded attempt published its cancellation as a permanent error over the winner's live connection — reachable when a system-transport proc.onExit schedules a reconnect while an attempt is still in flight. Cancellation now carries a stable error name, and both connect() and runReconnectAttempt claim their connectGeneration and stay silent when a newer attempt owns the state. * fix(P1-B): retry a dropped watcher overflow marker on real capacity emitWatcherOverflowToClient published the {kind:'overflow'} resync marker with controlOverflow:'reject'. A full control queue rejects at admission with no settlement callback, so the marker was silently discarded and the remote File Explorer stayed stale until some later watcher event happened to produce another one — for a quiet tree, possibly never. The emitter now retains a rejected marker per (client, root) and republishes it when the sink actually frees up. The existing onLegacyPtyCapacity signal cannot drive that: it is gated on producer retention, so it stays silent exactly under the dual-queue pressure that caused the rejection. RelayDispatcher.onClientCapacity is an ungated per-client capacity signal that fires on every writer settlement and drain. It lives on the dispatcher rather than the writer so a retained marker survives setWrite() replacing the primary sink, and setWrite notifies capacity once afterwards so the marker does not wait on traffic that may never arrive. Retention is bounded to one marker per (client, root), released on settlement and purged on client detach. * fix(P1-B): address all review findings on SSH network recovery Fix four issues from code review: 1. **Bug — admitted overflow markers lost on setWrite**: Retain markers when settlement fails `ok: false`, not just on admission rejection. Prevents desynced filesystem trees after SSH sink replacement. 2. **SSH error classification expanded**: Add missing OpenSSH patterns (`ssh_exchange_identification`, `connection closed by remote`) and new `isDefiniteSystemSshHostFailure()` classifier. 3. **ControlMaster retry optimization**: Skip second probe when first failure is already definite host-level (network timeout, refused, unreachable). Saves ~30s per reconnect ladder step. 4. **Overflow flush under dual-queue pressure**: Gate pending marker retries on control-lane headroom instead of re-attempting on every capacity notification. Reduces thrash proportional to producer traffic. Add regression tests for marker republish on sink replacement and validate auth error detection against live OpenSSH credential rejection messages. * rm random doc * fix(P1-B): skip credential-failure retries and recover watcher markers o - Auth and passphrase errors fail immediately without retry attempts - Bare "System SSH probe failed (exit 255)" is transient only for reconnect - Watcher markers survive client invalidation when switching SSH connections - Add network error patterns: "lost connection", "remote end closed"	2026-08-01 22:21:37 -07:00
Jinjing	de75003df9	fix(P1-C): gate FIDO2 system-SSH transport on an OpenSSH binary (#12029) ... * fix(ssh): gate FIDO2 system-transport on an OpenSSH binary `ssh -G` echoes OpenSSH's built-in default identity list for every host, so `usesDefaultPaths` was almost never true and the security-key gate returned `!usesDefaultPaths || findSystemSsh() !== null` — forcing system transport without checking that an `ssh` binary exists. `spawnSystemSsh()` then throws `No system ssh binary found`, hard-failing connections that worked on ssh2. The same flag also stopped the default scan at the first existing normal private key, so a host that only accepts a FIDO2 key never reached system OpenSSH when `~/.ssh/id_rsa` happened to exist. Both decisions are independent of where an identity path came from: always require `findSystemSsh() !== null` before forcing system transport, and scan every candidate identity instead of stopping on the first normal key. `shouldUseSystemSshTransport()` is untouched, so ProxyCommand / ProxyJump / ProxyUseFdpass keep their intentional system transport. * test(ssh): isolate connection tests from the developer's own FIDO2 keys Transport selection now scans every default identity instead of stopping at the first normal key, so a `~/.ssh/id_ed25519_sk` on the machine running the suite would decide which transport the default-target tests take. Mock `findSystemSsh` to null by default and opt the two security-key tests in.	2026-08-01 18:45:37 -07:00
JinjingandOrca	a07427e970	fix(ssh, relay): keep remote sessions alive through reconnects and backpressure (#11999) ... * fix(ssh,relay): stop remote connections from being killed by backoff and frame caps Three independent connection killers found in the SSH/remote freeze audit. FINDING A - the reconnect ladder never escalated for post-handshake drops. scheduleReconnect() used the single published state.reconnectAttempt for both the delay index and the give-up test, and runReconnectAttempt() zeroed it before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host. New SshReconnectLadder splits the delay index (advanced by every retry) from the failure streak (advanced only by a failed handshake), so flaps back off while give-up semantics stay byte-identical to shipped. FINDING B - notify() closed the client whenever a frame exceeded the producer frame capacity, conflating a permanently un-sendable frame with transient backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the watcher flood killed the link and re-killed on every reattach+replay. notify() now drops and logs once per generation; fs.changed is chunked to each sink's capacity with a control-lane overflow marker as the resync fallback; agent-hook envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit. FINDING B2 - sendResponse routed >1MB responses to a lane whose admission ignores the frame cap and closed the client on rejection, so a large fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the request fails instead of the connection. Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame cannot be dropped by the producer-lane check. Co-authored-by: Orca <help@stably.ai> * fix(relay): stop the overflow marker from re-killing the link it protects Round-1 review fixes on the P0 freeze work. The control-lane overflow marker could reinstate the exact failure this P0 removes: dispatcher-client-writer closes the client when control-lane admission fails, and admitControl is the only lane that returns an error, so one marker per failing batch accumulated to the 256-frame/1MB bound and dropped the link. Markers are now deduped to one outstanding per (client, root), cleared on settle. Chunking also defeated the renderer's per-payload directory dedupe -- events are now stable-grouped by parent directory so one directory lands in one chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast path paid three JSON encodes; both are fixed by publishing first and sizing from a measured bytes-per-event estimate. Agent-hook shedding now surrenders the blocking interactive prompt LAST rather than first, so a degraded envelope cannot strand a pane at state=waiting with no answerable question card. The dropped-notification log now distinguishes over-capacity from producer queue backpressure and no longer lets the first dropped method silence every other producer for the life of the connection. * fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another Round-2 review fixes. The round-0 change from close-on-rejection to silent drop removed the only redelivery path for agent.hook envelopes: they are fire-and-forget and the per-pane cache only replays on handler install, so a saturated link stranded a pane on a stale Working spinner until reconnect. Closing used to guarantee delivery by forcing that replay. Envelopes now publish per client and pend for bounded latest-wins redelivery when the producer queue rejects them. Shed fields are now named on the wire. The subagent roster is not cosmetic -- the renderer replaces rather than merges it, and hibernation gates on its length -- so an unmarked shed could sleep a live pane. fs.streamEnd rode the control lane because it must not be dropped, but that lane kills rather than drops. The stream's concurrency slot is now held until the terminal frame settles rather than until the fd closes, capping queued terminal frames well under the control budget; overflow costs one refused read instead of the connection. The watcher chunk walk now stops while producer retention sits past its reserve and degrades to a resync, so a 5000-event flood cannot fill the queue that interactive PTY traffic shares and stall every remote terminal. The reconnect ladder caps its flap-path delay so delay plus handshake timeout cannot cross the relay grace floor and let the remote daemon kill live PTYs. Also: the suppression key no longer embeds a NUL byte, which had made the file binary to git and grep; producerEnvelopeBudget no longer reports infinite capacity for a departed client; the drop logger no longer encodes a frame it will not log; and an over-capacity response substitution no longer settles as if the result had been delivered. * fix(relay,ssh): restore relay-shed status fields and scope backpressure per client Round 3 + 4 review fixes. Watcher chunking is now gated on the *client's* retention reserve rather than the dispatcher-wide one, so one stalled peer no longer forces a healthy client into a full file-tree resync. The relay-lost redeploy ladder no longer burns its 6-attempt budget while the SSH transport itself is down: it holds at the 15s step with a non-terminal status and rearms, so a laptop that slept past the ladder comes back instead of landing on a terminal "give up" banner. The shedFields wire marker had no consumer, so an agent-hook envelope whose subagent roster was dropped to fit the frame read as "roster cleared" on the Orca side: live child rows blanked and a done pane became hibernation-eligible while its teammates were still running. ingestRemote now restores shed fields from the cached payload (interactivePrompt deliberately excluded — a stale answerable question card is worse than none). Also: stream terminal-frame slots are counted per client, since the control queue they protect is per client; the chunking fast path no longer logs a drop for a batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity. Test debt from the review: pending-pane eviction, per-client stream isolation, and the reconnect budget are now asserted rather than assumed; four fragile exact-byte pins dropped in favour of the tier comparisons that carry the requirement. * fix(relay,ssh): restore relay-shed status fields and scope backpressure - Oversized relay responses now fail their request instead of closing the connection, preventing one frame from killing every pane on the host - Restore subagent state for correct hibernation; don't resurrect stale prose across turns - Account for relay re-establishment and PTY reattach time in SSH flap delay caps - Only log drops of final unsendable envelopes, not temporary rejections during measurement probes - Fix watcher overflow marker release race when notification admission rejects without settlement; use precise byte counting for event batching * Restore relay-shed fields with digest validation and scoped backpressure Validate that shed subagent rosters match their wire digest and turn identity before restoration, preventing stale roster resurrection. Compact interactive prompts for waiting states instead of dropping them. Demote control-queue overflow to non-fatal rejection so clients can retry on capacity recovery, keeping the link alive during transient backpressure. * fix(relay): correct ResponseOverCapacity error code ResponseOverCapacity should use -33008 to stay in the -33xxx range for relay protocol errors, not -32010. * fix(relay): close client when pty.replay overflows control queue Replay is never retried, so it uses the control lane where overflow is fatal — the writer closes the client and reconnect reloads history rather than stranding a short buffer. * fix(relay): prevent infinite redeploy on flapping SSH transports Charge reconnect attempts when connection restores mid-backoff, preventing infinite loop on transports that flap between states. Refactor control overflow handling to use entry property instead of WeakSet marker for clarity. --------- Co-authored-by: Orca <help@stably.ai>	2026-08-01 14:27:16 -07:00
Neil	33c14bc716	fix(ssh): fall back to OpenSSH for FIDO2 keys (#11913) ... Closes #11645	2026-08-01 03:27:42 -07:00
Henry SuandOrcaWin	5fe3aaf2b7	fix(ssh): preserve first config directive value (#11297) ... * fix(ssh): preserve first config directive value * test(ssh): cover false-first config booleans * fix(ssh): trust fresh OpenSSH config authority * fix(ssh): preserve ordered config identities --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>	2026-07-30 14:23:30 -07:00
JinjingandOrcaWin	a40183389b	feat: bound direct SSH reconnect fan-out and recovery (#11003) ... * docs: design for direct SSH reconnect fan-out Capture the implementation-ready plan for host-qualified, epoch-fenced SSH reconnect recovery after two rounds of multi-model LLM counsel review. * docs: reconcile SSH reconnect fan-out design * docs: close reconnect design consistency gaps * feat: implement bounded direct SSH reconnect recovery * fix: bound direct SSH retry settlement * fix: harden direct SSH reconnect authority * fix: preserve split SSH retry ownership * fix: preserve SSH split continuation authority * docs: record final SSH reconnect validation * fix: preserve SSH authority through retained and detached state * fix: retain SSH authority across delayed split mounts * fix: close SSH authority recovery gaps * fix: fence stale SSH transport replacement * fix: serialize SSH target teardown * fix: settle SSH teardown failures before reconnect * fix: retire failed SSH reset sessions * test: reconcile current main E2E contracts * fix: close direct SSH reconnect review gaps * fix: fence stale SSH reconnect side effects * fix: close final SSH reconnect lifecycle gaps * test: stabilize current-main reliability gates * test: prove plugin navigation containment * test: make plugin navigation oracle authoritative * test: make plugin navigation oracle deterministic * ci: allow sharded e2e suite to finish * test: wait for runtime pane publication * test: classify pane readiness by error code * test: select close persistence terminal by tab identity * docs: mark reconnect implementation validated --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>	2026-07-28 12:33:17 -07:00
Neil	badf91101b	fix(quality): enforce performance-safe lint baseline (#11074) ... * fix(quality): clear safe existing lint findings * fix(quality): keep lint cleanup allocation-free * fix(quality): enforce performance-safe baseline * test(terminal): drain deferred confirmation cleanup	2026-07-27 20:54:02 -07:00
3 people	6d39e49480	fix(ssh): accept GitHub restricted-shell SSH probes (#6988) (#7659) ... * fix(ssh): accept GitHub restricted-shell SSH probes (#6988) * fix: match first stderr line for GitHub restricted-shell probe (bug-bash takeover) Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> Co-authored-by: Orca <help@stably.ai>	2026-07-24 00:24:53 -07:00
NeilandOrca	aab112933e	Revert "fix(memory): bound OOM-prone accumulators (#10179)" (#10255) ... Co-authored-by: Orca <help@stably.ai>	2026-07-23 18:35:31 -07:00
Neil	8f40ddf328	fix(memory): bound OOM-prone accumulators (#10179)	2026-07-23 06:22:56 -07:00
3 people	4f81dfc128	perf(ssh): cut warm high-latency connects from 88.7s to 7.7s (#9015) ... Move managed agent-hook filesystem work behind one relay RPC so high-latency SSH connects pay one WAN round trip instead of hundreds. Keep installers serial, lock shared account config across relay processes, and fence cancelled connection generations from replacement state. Co-authored-by: nasagong <zinho2000@gachon.ac.kr> Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>	2026-07-20 12:18:57 -07:00
gatsby74andJinjing	8ed8f0d109	feat(ssh): download folders from remote explorer (#7793) ... * feat(ssh): download folders from remote explorer * fix(ssh): harden remote folder downloads * Add missing getRepo stub to worktree cwd test mock Restoring headless mobile tabs looks up the repo for the active worktree id; the mock lacked getRepo, so the test only passed incidentally. Add it explicitly and return undefined since wt-1 is a worktree id, not a registered repo. * Enable SSH folder downloads, gated for system-SSH connections - Folder downloads require SFTP, unavailable on system SSH (which offers only raw file operations). Add supportsFolderDownload flag to gate the feature in the UI layer. - Reject symlinks at directory-entry level, preventing tree escapes and eliminating unnecessary stat calls. - Check abort signal before opening dialog for better responsiveness when renderer closes. - Log cleanup errors without re-throwing to preserve underlying transfer failures. * Gate SSH folder downloads to SFTP-capable connections Enforce fail-closed gating and add Windows path traversal validation to ensure downloads are only available when explicitly supported and safe. * Gate SSH folder downloads to SFTP-capable connections Enforce fail-closed gating and add Windows path traversal validation to ensure downloads are only available when explicitly supported and safe. * fix(ssh): keep provider types under max-lines after main merge Move FolderDownloadOptions next to the SFTP download implementation and narrow IFilesystemProvider.downloadFolder options to AbortSignal only so types.ts stays within the 300-line oxlint budget when merged with main. --------- Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>	2026-07-18 23:01:29 -07:00
5 people	fa85536f3a	fix(ssh): repair unbuilt relay native deps (#8686) ... Co-authored-by: Orca <help@stably.ai> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com>	2026-07-16 21:32:13 -07:00
Haohan Lin	190ed7d4ed	fix(ssh): make POSIX relay wrapper survive csh/tcsh login shells (#8709) ... Reconstruct remote POSIX commands with bounded printf arguments so non-POSIX SSH login shells can forward them without requiring remote base64. Preserve relay and system-SSH stdin, and centralize login-shell flag selection for csh/tcsh compatibility.\n\nValidated against real csh and tcsh OpenSSH targets with built-in and system SSH, including cold relay deployment, stdin upload, PTY I/O, file mutation, and reconnect.	2026-07-15 02:38:40 -07:00
6 people	0302ae86b8	feat(ssh): support Kerberos/GSSAPI hosts via the system OpenSSH transport (#7507) ... * feat(ssh): support Kerberos/GSSAPI hosts via the system OpenSSH transport ssh2 has no gssapi-with-mic support, and adding it would mean forking its protocol layer plus packaging the kerberos native module for three platforms. Instead, route GSSAPI hosts through the existing system-OpenSSH transport, which delegates Kerberos (tickets, SSPI on Windows) to the platform ssh binary. Two tiers, because RHEL-family distros enable GSSAPIAuthentication globally in /etc/ssh/ssh_config and ssh -G therefore reports it for every host: - Targets whose ~/.ssh/config Host block explicitly sets GSSAPIAuthentication yes (imported as target.gssapiAuthentication) try system ssh first, falling through to ssh2 so key auth and credential prompts still work when no ticket is available. - When ssh2 exhausts key/agent auth and the ssh -G-resolved config enables GSSAPI, retry over system ssh before prompting for credentials, so Kerberos-only hosts on distro-default configs connect without a password prompt. Hosts where keys work never leave the ssh2 path. Manual targets flagged for GSSAPI pass -o GSSAPIAuthentication=yes explicitly since they bypass ssh_config. Both tiers work headless (no credential callbacks required). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ssh): harden GSSAPI transport selection (review fixes for PR #7507) Review fixes on top of the Kerberos/GSSAPI feature branch (s546126/kerberos-ssh): - HIGH: reset useSystemSshTransport on the ssh2 fall-through. doSystemSshProbe sets the flag before spawnSystemSshCommand, which throws synchronously when no system ssh binary is on PATH (outside the probe try/catch). The proactive fall-through previously reset only 2 of 3 transport fields, so exec/sftp kept routing through the failed transport - breaking GSSAPI on Windows-with-Git-ssh and headless Linux. - MEDIUM: throw a cancellation error (not the stale ssh2 authError) when a disconnect supersedes the reactive probe mid-flight, and guard connect()'s catch on disposed, so a deliberate disconnect is not overwritten with auth-failed. - MEDIUM: skip the encrypted-key passphrase prompt when the GSSAPI fallback applies, so a Kerberos ticket is tried before prompting; the general prompt still fires if the probe fails. Adds 3 mutation-verified regression tests and hardens two existing tests to assert the probe actually ran. Not connected to any PR remote. Co-authored-by: Orca <help@stably.ai> * fix(ssh): isolate GSSAPI system transport Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: s546126 <268420947+s546126@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai>	2026-07-15 01:58:46 -07:00
3 people	c2aa9c2ead	feat(ssh): support file transfers over system ssh (#7804) ... * feat(ssh): support file transfers over system ssh * fix(ssh): harden system file transfers * test(runtime): stub getRepo in headless mobile tab cwd test The mobile-session selector validator (getValidatedExplicitWorktreeIdSelector, from main) calls this.store?.getRepo to reject repo ids passed as worktree ids. The store stub only implemented getWorkspaceSession, so the guarded call threw 'getRepo is not a function' once main merged into this branch. Add a getRepo that returns null (wt-1 is a worktree, not a repo). Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com> Co-authored-by: Orca <help@stably.ai>	2026-07-12 01:29:55 -07:00
4 people	f215a48064	fix: pr-bug-scan validated finding from #6952 (#7180) ... * fix: address pr-bug-scan validated finding from #6952 throwNodeNotFound() now re-raises AbortError when the shared signal is aborted, so a signal-cancelled node probe no longer launders into 'Node.js not found'; sequential fallback runs. * fix(ssh): make session-limited (MaxSessions=1) relay deploys actually succeed Review of #7180 verified the parent fallback end-to-end against a real MaxSessions=1 sshd and found the connect still failed. Four gaps, in order of discovery: - isSshSessionLimitError missed stock OpenSSH, which refuses session channels over MaxSessions with SSH2_OPEN_CONNECT_FAILED (2) and 'open failed' — reason 4 never matched, so the fallback never triggered. - execCommand settled aborted commands before the channel finished closing, so the sequential fallback reissued execs while sshd still counted the old session. - SshConnection.waitForSshCallback rejected aborts mid-channel-open immediately, leaking a confirmed-late channel that held the only session slot; it now settles after the late channel closes (bounded) and drains its streams so ssh2 emits 'close'. - Session channel opens now retry transient session-limit refusals (sshd frees the slot only after processing our close-ack, which the next open can beat by microseconds), and the remote orca CLI shim install is non-fatal like the managed-hook install — after the relay bridge occupies the sole session slot, raw-connection extras must degrade instead of failing the connection. Verified live against Docker sshd (OpenSSH 9.2, MaxSessions=1): fresh deploy (upload + native deps + launch), reconnect cycles, and a PTY round-trip all succeed; unrestricted-sshd regression run also passes. Co-authored-by: Orca <help@stably.ai> * Handle ssh execution aborts immediately during retry backoff or hangs - Cancel the session-limit retry delay immediately if the operation is aborted during backoff. - Limit the wait time to a 5-second grace period when aborted during a channel open that is hung and never invokes its callback, rather than waiting for the full connection timeout. --------- Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai> Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com> Co-authored-by: Orca <help@stably.ai>	2026-07-04 01:26:55 -07:00
Brandon Barker	e6ca3087c5	perf(ssh): resolve node path concurrently with remote install state (#6952) ... Run independent SSH relay bootstrap probes concurrently when the connection can safely support overlapping execs. Preserve the old sequential path for system SSH without reusable ControlMaster and for remotes that reject concurrent session channels.	2026-07-02 13:55:36 -07:00
4 people	4dbc9f3817	feat(ssh): add ControlMaster multiplexing for system SSH transport (#6922) ... * feat(ssh): add ControlMaster multiplexing for system SSH transport System SSH transport spawns a new OpenSSH process per exec command (platform detect, relay install check, node resolution, relay launch, socket probe). Each process pays the full SSH handshake cost — ~9s on Uber devpods — making a typical relay connect take 54s+ and reliably exceeding the 15s startup reconnect budget. Add SSH ControlMaster multiplexing via a per-target socket in $TMPDIR/orca-ssh-ctl/<hash>.sock. The first command establishes the master; subsequent commands reuse it at ~100ms per exec instead of ~9s. ControlPersist=300 keeps the master alive after commands exit so rapid reconnects (e.g. on tab focus) also benefit. Windows is excluded since OpenSSH's ControlMaster support there is limited. * fix(ssh): address ControlMaster key collision and directory permission risks - Use target.id in the socket key so distinct SSH targets can never collide even when configHost/port/user happen to match - Switch from SHA1 to SHA256 and extend hash slice from 12 to 16 chars - Stat the control-socket directory after mkdirSync to reject pre-existing dirs that are symlinks, foreign-owned, or have group/other write bits (mkdirSync mode is ignored on pre-existing dirs) - Update two tests that used exact spawn-arg arrays; replace with ordering assertions (forward flags before --) that stay correct regardless of which extra ControlMaster options are injected * fix(ssh): bind ControlPath identity to route and reject symlinked ctl dir Fold proxyCommand/jumpHost/identity fields into the ControlPath hash so a target whose route is edited no longer reuses a still-alive master built on the old route. Switch the control-socket dir check from statSync to lstatSync so a planted symlink fails the directory validation outright. * test(ssh): drop tautological argv re-assertion in spawn checks The toHaveBeenCalledWith re-passed the args array extracted from the same mock call, making that argument position always pass. argv content is already verified by the index-ordering assertions above; use expect.any(Array) so the spawn check only claims what it actually verifies (binary path, stdio). * fix(ssh): harden system ssh connection reuse Co-authored-by: Orca <help@stably.ai> * test(ssh): isolate control socket runtime dir Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Test <test@example.com> Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai>	2026-07-01 21:29:48 -07:00
NeilandOrca	46646d7ff1	chore(lint): upgrade oxlint to 1.71 + enable 7 new rules (autofixed backlog) (#6841) ... * chore(lint): upgrade oxlint to 1.71 and enable 7 new rules Upgrade oxlint 1.67.0 -> 1.71.0 (1.72 was blocked by the repo's 3-day minimum-release-age supply-chain guard; nothing here needs it). The bump is a no-op on the existing config. Enable 3 error rules (backlog autofixed to zero in this commit) and 4 warn rules (surface signal without gating CI): error (autofixed, behavior-preserving): - unicorn/prefer-node-protocol (~1531 sites: bare builtin -> node:) - typescript/no-import-type-side-effects (~36: all-inline-type -> import type) - unicorn/no-array-reverse (19: copy-then-reverse -> toReversed) warn (real signal, current fires are test-only/correct): - unicorn/no-array-fill-with-reference-type (aliasing footgun guard) - typescript/no-unsafe-function-type (bans bare Function type) - unicorn/prefer-array-flat-map (map().flat() -> flatMap()) - unicorn/prefer-regexp-test (.match() in bool ctx -> .test()) mobile/.oxlintrc.json extends root, so it inherits all 7; the autofix ran from root and covered mobile/ too. Verification (all green): oxlint 0 errors (root+mobile+aux configs), oxfmt clean, typecheck (node+cli+web), vitest 22795 passed / 0 failed, builds (electron-vite + web + cli) succeed. node: rewrites confirmed to skip embedded SSH/CLI string payloads (AST-only); all toReversed sites verified to operate on fresh copies or write-once locals. * chore(lint): bump mobile oxlint to 1.71 so inherited rules parse mobile/ is a standalone pnpm project pinning its own oxlint@1.67, which lacks unicorn/no-array-fill-with-reference-type (needs >=1.70). Since mobile/.oxlintrc.json extends the root config, mobile CI's 'cd mobile && oxlint' failed to parse the new rule. Bump mobile to match root (1.71). Verified in mobile/: oxlint 0 errors, oxfmt --check clean, tsc --noEmit pass, vitest 978 passed / 0 failed. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai>	2026-06-29 22:38:29 -07:00
Jinwoo HongandOrca	c455c576ae	Fallback to system SSH on reachability errors (#5303) ... Co-authored-by: Orca <help@stably.ai>	2026-06-12 20:06:20 -07:00
3 people	98d02bca47	fix: support windows ssh hosts (#5004) ... * feat: add windows ssh relay base support * feat: support windows ssh relay runtime services * fix: default windows ssh pty cwd to user profile * fix: support windows hosts over system ssh * fix: preserve degraded windows relay native deps * fix: gate windows shell args by relay platform * fix: preserve windows relay fallback pipes * test: align windows native deps relay fixture * fix: build valid windows install lock command * fix: address windows SSH relay review findings Resolve correctness, efficiency, and reuse issues found reviewing the Windows SSH native-host support: - GC liveness on Windows now probes the actual named pipe (via node net.connect against markers + deterministic candidates) instead of substring-matching Win32_Process command lines, which could remove a live relay dir. Reports ALIVE conservatively only when there is no liveness signal at all (no markers and no seed pipes). - Resolve the remote node path once per deploy and thread it through install/repair/launch instead of re-resolving 3-7x. - Replace the 200ms node -e poll loop with a single long-lived remote wait process during Windows relay startup. - Skip the no-op executable command on Windows in uploadRelay. - Make the Windows fallback pipe name deterministic and recoverable (drop the global counter), with an extra reconnect attempt. - Normalize the prepended node bin dir to backslashes on Windows PATH. - Batch the system-SSH Windows directory upload into a single streamed JSON package instead of one ssh process per file. - Extract relay endpoint/marker helpers into ssh-relay-endpoints.ts and consolidate the PowerShell EncodedCommand encoding into the shared powershell-command-encoding module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Support cancellation and timeouts in Windows port scanning - Propagate the request AbortSignal and a 5-second timeout to both PowerShell and netstat child processes during Windows port scanning. - Avoid spawning the netstat fallback process if the port scan has already been aborted. - Wrap the .NET OSArchitecture check in a try/catch block during SSH Windows platform detection to robustly fall back to environment variables if needed. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>	2026-06-09 01:17:34 -07:00
Jinwoo HongandOrca	9c2cec3d6c	Use system ssh for proxy-based targets (#4911) ... Co-authored-by: Orca <help@stably.ai>	2026-06-08 14:43:25 -07:00
Neil	c7c2330723	fix: close late ssh channel callbacks (#4116)	2026-05-31 04:48:28 -07:00
3 people	ccfc1a9b97	fix: time out ssh channel opens (#3742) ... * fix: time out ssh channel opens * test: cover ssh channel-open timeout edge cases Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com> Co-authored-by: Orca <help@stably.ai>	2026-05-30 11:43:40 -07:00
Neil	8044be7cc7	fix: clean up system ssh probe listeners (#3824)	2026-05-30 11:32:49 -07:00
Jinjing	41f0bc6394	test: preserve ssh mock event listeners (#3015) ... * test: preserve ssh mock event listeners * test: reuse runtime file command helper	2026-05-28 18:30:35 -07:00
Jinjing	186327af08	fix: guard ssh startup destroy errors (#3013)	2026-05-28 16:13:09 -07:00
Neil	9d396e115c	fix: remove SSH startup listeners after connect (#2882)	2026-05-26 20:24:36 -07:00
Neil	eb579a0e54	Fix SSH sessions after host sleep	2026-05-26 14:53:16 -07:00
Jinjing	ffec74c7ba	Fix SSH agent and identity file auth ordering (#2681) ... * fix: address review findings * fix: address CI failures	2026-05-23 01:56:15 -07:00
Jinwoo HongandOrca	9c67557a0c	Support system SSH transport for fdpass proxies (#2279) ... Co-authored-by: Orca <help@stably.ai>	2026-05-18 16:03:34 -07:00
Neil	fdc38bfc33	Fix SSH relay connect when default shell is fish (#2239)	2026-05-18 01:31:45 -07:00
Jinwoo HongandOrca	9a39b1345a	fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660) (#1679) ... * fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660) ssh2 leaves Nagle's algorithm on by default. For single-byte keystrokes through a remote PTY, Nagle interacts with the kernel's delayed-ACK timer and adds up to ~40 ms per keystroke — visible as the typing lag reported in #1660. OpenSSH's `ssh` client sets TCP_NODELAY whenever a PTY is allocated; this change mirrors that on the ssh2 client right after the `ready` event in doSsh2Connect, covering both initial connect and auto-reconnect. Proxy-command / proxy-jump connections (where ssh2's underlying socket is a custom Duplex over a child-process pipe) are a no-op by design, gated by the public Client.setNoDelay()'s own type guard. A discriminating log line records which path each connect took. Tests cover initial connect and a full reconnect cycle to guard against the regression class "Nagle is re-enabled because someone refactored only the initial connect path." Co-authored-by: Orca <help@stably.ai> * fix(ssh): bound relay-lost reconnect with exponential backoff When the relay exec channel keeps dying (e.g. a remote-side bug closes every fresh --connect channel right after handshake, or a stale bridge keeps being replaced), the unguarded _onRelayLost handler reconnects as fast as the network allows — spawning relay deploy attempts in a tight loop until the user force-quits. Each iteration spawns a fresh ssh2 exec channel, hammers sshd's MaxSessions counter, and floods the renderer with state churn. Add per-target exponential backoff (500ms → 15s, capped at 6 attempts) so the loop terminates instead of running forever. After the cap the session goes to 'error' state with a 'Relay channel kept dropping. Please reconnect.' message — visible in the renderer instead of an invisible failure where typing in remote terminals just stops working. Successful 'ready' resets the attempt counter only if the session stabilized for >= 5s; faster flaps preserve the counter so a flaky remote backs off rather than retrying indefinitely on every brief ready→lost cycle. Backoff state is cleared on explicit disconnect, on session replacement during reconnect, and on connect failures, so a real reconnect attempt after backoff exhaustion always starts from zero. Co-authored-by: Orca <help@stably.ai> * fix(ssh): detect stale relay daemons via running-version marker The on-disk relay version check compares local .version against the remote .version file in the relay dir. A daemon launched by an earlier deploy keeps running its in-memory copy of the OLD relay code, so when the client later rewrites relay.js + .version on disk and bridges in via --connect, the new bridge process drives a stale daemon. Protocol or behavior changes between the two versions then tear down the channel in a tight reconnect loop (observed against PR #1672 on a daemon predating that change). The daemon now writes its running version into a .running-version sidecar at startup, anchored to the relay-script directory rather than process.cwd() so test spawns cannot pollute the repo root. Before attaching to an existing socket, the client probes that marker and, on mismatch with the locally-deployed .version, kills the stale daemon (TERM only, never KILL) and falls through to a fresh launch. Conservative defaults: when either marker is unreadable, attach so older builds keep their live PTYs. Co-authored-by: Orca <help@stably.ai> * Revert "fix(ssh): detect stale relay daemons via running-version marker" This reverts commit e58acf07c0. * fix(ssh): isolate relay versions via per-version install dirs and wire handshake The relay's previous single-dir layout (~/.orca-remote/relay-v0.1.0/) let the deploy step rewrite relay.js in place while a daemon was still loaded in memory at the previous version. New clients then drove that stale daemon, surfacing as a reconnect loop (issue #1660 follow-up) and the field failure observed against an 8-day-old daemon on openclaw. Switch to a VS Code-style versioned layout where each (RELAY_VERSION + content-hash) bundle installs into its own directory and is never mutated after install. A v2 client's --connect socket path is rooted in relay-${v2-hash}/ and structurally cannot reach a v1 daemon's socket. Defense-in-depth: the daemon now reads exactly one Handshake-typed frame on each newly-accepted Unix socket before attaching the JSON-RPC dispatcher (mirrors VS Code's remoteExtensionHostAgentServer.ts:340). Mismatch closes the socket; the bridge exits with code 42; client maps that to a typed RelayVersionMismatchError and skips the relay-lost backoff loop instead of retrying through 6 attempts. Other deploy hardening: - atomic mkdir-based install lock with stale-lock recovery serialises concurrent first-installs of the same version - .install-complete sentinel distinguishes a finished install from a crashed-mid-install partial that should be retried - gcOldRelayVersions removes unreferenced sibling dirs (allowlist regex, skips locked or incomplete dirs, skips dirs with a live socket) - readLocalFullVersion fails fast on a missing/empty local .version rather than silently falling back to a path where a daemon from a different code generation may already be running Includes a cross-version isolation test that fails any future refactor which collapses the per-version layout back to a shared dir. Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden relay versioning per review feedback Address must-fix and should-fix findings from the parallel triple review of 26d1666e: - Surface RelayVersionMismatchError to ssh.ts on initial establish() (not just reconnect), so the user sees the typed terminal error instead of silent retry on first connect (#13). - Give the sentinel timeout a 500ms grace window for the close handler to deliver exit-42, so a slow remote does not misclassify a wire-handshake mismatch as a generic timeout (#D11). - Drain the handshake decoder's residue at the handshake -> dispatcher transition on both daemon and --connect sides; pipelined frames that were coalesced with the handshake are now forwarded into the dispatcher / stdout instead of silently dropped (#A1, #A2). - Reset the install-lock acquire timer after a stale-lock recovery so a single post-recovery race does not immediately exhaust the budget (#E14). - Treat a stale install-lock as recoverable in the GC pass when .install-complete is present (covers an interrupted finalize where the rm-lock failed) (#E15). - GC legacy relay-v\d+\.\d+\.\d+ install dirs whose daemons have died, now that .install-complete is no longer required for them (#12). - Resolve symlinks in readLaunchVersion() so a daemon launched via a symlinked entry script still reads .version next to the real file (#G21). - Flush stderr before exit-42 in --connect handshake mismatch path so the diagnostic line reaches the client before the process tears down (#C8). Tests: - Round-trip handshake over a real Socket pair: matching version, mismatch exit-42, leftover bytes preserved on both sides when frames are coalesced with the handshake. - waitForSentinel exit-42 -> RelayVersionMismatchError, exit-1 -> generic. - SshRelaySession terminal-error callback fires on both establish() and reconnect() when deployAndLaunchRelay throws RelayVersionMismatchError. - acquireInstallLock concurrent BUSY -> OK polling, stale-lock recovery with reset timeout window, and fresh-lock timeout failure path. - gcOldRelayVersions stale-lock-with-complete branch, legacy-dead path, legacy-alive path; existing locked-test asserts fresh-lock now keeps. - Cross-version isolation test now asserts a blanket invariant that every v1-referencing command from a v2 deploy is a read-only liveness probe. Lint and typecheck clean across all 3 tsconfigs; 426 SSH/relay tests pass. Co-authored-by: Orca <help@stably.ai> * fix(ssh): bypass npm init for content-hashed relay dirs and harden install probe The versioned-install dirs land at `relay-${version}+${hash}/` (e.g. `relay-0.1.0+07994a7870e1`). npm 11 / Node 26 reject the `+` in derived package names and `npm init -y` exits 1 — silently, since both stderr and the failure landed inside the `2>/dev/null && ...` chain. The catch swallowed the throw, `.install-complete` was written anyway, and every reconnect surfaced 'node-pty is not available' at first pty.spawn. Sidestep `npm init` entirely: SFTP-write a hardcoded minimal package.json (`name: orca-relay`, `type: commonjs`) and run `npm install node-pty` directly. `type: commonjs` pins the module system against future Node default flips or remote-side .npmrc overrides. Also harden the install path against the same class of silent failure: - npm install errors now propagate (no more `.install-complete` on hard fail; future reconnects retry instead of stranding the user) - Replace the weak `test -d node-pty` post-install probe with `node -e 'require("node-pty")'` so built-but-unloadable installs (missing prebuild, wrong arch, broken native binding) surface clearly - Add a session-level error handler on the SFTP write so a torn-down session rejects the promise instead of hanging until enclosing timeout Separate fix: add `for-each-ref` to the relay's git subcommand allowlist. Client code (`src/main/git/repo.ts` ref-search and worktree-listing) calls `git for-each-ref` over SSH; the relay was rejecting it. The `--shell`/`--python`/`--perl`/`--tcl` format flags only control output quoting (no eval) and the relay invokes git via execFileAsync (no shell), so the read-only allowlist treatment matches `rev-parse`, `log`, etc. Co-authored-by: Orca <help@stably.ai> * comment(ssh-relay): TODO link to #1693 for VS Code-style pre-bundled node-pty Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden node-pty install probe and tighten review-fix tests Round-3 review fixes on top of 963f56d7. deploy.ts: - Replace endsWith('OK') with includes('ORCA-NPTY-PROBE-OK'). Node can emit deprecation/experimental warnings to stderr after our stdout 'OK' write, and 2>&1 would push them past 'OK' producing false NPTY-MISSING warnings. A unique sentinel survives any trailing stderr noise. - Switch sftpPkg/ws .on -> .once for error/close. A late session 'error' after the promise had already settled would otherwise become an unhandled EventEmitter error and crash main. - Trim per-block comments to 1-2 lines per AGENTS.md (was 7-9). Tests: - Pin the BEFORE-ordering contract: SftpWriteCapture now records the count of execCommand calls observed at the moment ws.end() ran for each path, and the test asserts that count <= the index of npm install. Catches a future Promise.all-style refactor that would still pass final-state checks. - Strengthen the SSH-channel-failure test: assert the rejection actually came from the probe call (not an earlier exec) by finding the probe invocation in mock.calls. Also assert NPTY-INSTALL-FAIL is NOT logged (channel failure must not be conflated with install failure) and that abandonInstall was called so the lock is released. - Fix misleading clearAllMocks comment: it claims to wipe mockReturnValue, but actually clearAllMocks only resets .mock.calls. Re-priming was defense-in-depth, not a correctness requirement. Validator: - Add for-each-ref negative cases (--git-dir, --output, --work-tree) to the global-denied-flags it.each. The first round of for-each-ref enablement trusted that the post-subcommand GLOBAL_DENIED_FLAGS check applied; this pins it so a future allowlist refactor that bypasses the global check fails loudly. Co-authored-by: Orca <help@stably.ai> * fix(ssh): split node-pty probe into test-d guard + load-test Round-4 review fixes for the install probe in installNativeDeps: (1) test -d guard runs before the load-test. If the install dir vanished between npm install and probe (concurrent rm, fs unmount, permission flip), the deploy now throws and the next reconnect retries fresh — previously the cd failure flowed into '|| echo MISSING' and we'd write .install-complete, stranding the user in degraded mode. (2) Load-test discards stderr (2>/dev/null) so customized .bashrc output (NVM init, conda greetings, etc.) can't pollute the sentinel match. The shell-level '|| echo MISSING' is preserved so SSH-channel rejections still propagate as exec errors, distinct from require failures which exit the node process nonzero. (3) PROBE_OK is passed via process.argv[1] so the JS literal stays trivial regardless of future sentinel characters. Test changes: - New 'dir-gone' probe mode in makeExecResponses - New test pinning that vanished-dir throws (not silent MISSING) - SSH-channel test now asserts probeCallIdx > npmInstallIdx - cross-version-isolation feeds an extra '' for the test -d slot Co-authored-by: Orca <help@stably.ai> * fix(ssh): simplify node-pty probe and harden test ordering pins Round-5 review fixes for installNativeDeps: Production: - Drop redundant test -d guard. `cd ${dir} && (...)` short-circuits on cd-failure (dir-vanished) and propagates as exec reject already; the separate guard added a round trip without preventing anything. - Capture probe stderr to a per-deploy file rather than 2>&1 or 2>/dev/null. .bashrc noise can't pollute the sentinel match, but the require() error message is preserved in the [NPTY-MISSING] log breadcrumb so bug reports point at the real cause (e.g. GLIBC version mismatch). - Mirror the install command's PATH (export PATH=${binDir}:$PATH) so any future require-time child_process call resolves the same node binary used during install. - Add platform tuple to [NPTY-MISSING] and [NPTY-INSTALL-FAIL] logs for triageable bug reports without asking users to dig out their arch. - Trim probe comment per AGENTS.md (why-only, no mechanism narration). Tests: - Pin full installNativeDeps ordering: npm install < chmod prebuilds < probe. Catches refactors that probe before install or move chmod after. - Pressure-test .includes(PROBE_OK) survives bashrc/MOTD noise prefixed to probe stdout (corporate banner / NVM init / conda greeting case). - Pressure-test MISSING detection survives Node deprecation warnings prepended to the MISSING token. - Pin platform tuple appears in [NPTY-MISSING] log. - Pin finalizeInstall called exactly once + abandonInstall not called on happy paths; reverse on failure paths. - Strengthen dir-gone test: assert probeIdx > npmInstallIdx so a refactor that swaps order doesn't silently let the test pass on its own injected error string. - New probeStdoutOverride option in makeExecResponses for shell-noise injection tests. Cross-version-isolation: dropped obsolete test -d slot, added rm-stderr cleanup slot to match the new probe shape. eslint-disable max-lines on both files with rationale (pattern used widely in this repo for cohesive single-responsibility modules). 441/441 tests pass; lint clean; typecheck clean. Probe shape verified end-to-end on real remote. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai>	2026-05-11 14:28:59 -07:00
Jinwoo Hong	42e04268fb	fix: harden SSH connection reliability and add passphrase prompt (#636)	2026-04-14 18:29:50 -07:00
Jinwoo Hong	cc66e120eb	feat: Add SSH remote support (beta) (#590)	2026-04-13 19:23:09 -07:00