Commit Graph
42 Commits
Author SHA1 Message Date
Neil 6dcfcdface fix(ssh): stop treating an absent known_hosts as a source we failed to read
The previous commit's "no readable source is not first contact" guard was right
about the danger and wrong about how to detect it, and the version that shipped
would have refused every connection a new profile ever makes.

A file that does not exist and a file that refuses to open both arrive as a
rejected readFile, and I counted them the same way. They are opposites. An
absent known_hosts is the normal state — ssh creates it on its own first connect,
and an Orca profile that has never connected has none — and it is real evidence
that no host is known. A file that exists and will not open is evidence withheld:
the entry that would have said "this key changed" may be sitting in it.

The default list is the reason this was fatal rather than obscure. It always
names known_hosts2, which essentially never exists, so on a machine with a normal
known_hosts the count was 1-of-2 and everything worked; on a fresh profile it was
0-of-2 and every connect failed with "the system SSH configuration could not be
read" — a message about a file the user does not have and an error they cannot
act on. The suite passed only because the machine running it happens to have a
known_hosts. Pointing HOME at an empty directory fails 62 connection tests, which
is what the new wiring test does.

So the count is now unreadableFileCount: files that exist and could not be read,
which is the condition the guard was always trying to express. Any one of them
takes the strict path; an absent or empty file takes none. The store clause is
gone with it — a store hit already returns accept before this is consulted, so it
never changed an outcome.

Tests: absent, empty, permission-denied, directory, and all-parsed at the source;
first contact with no known_hosts at all at the wiring level.

1,482 SSH tests pass. The 3 failures in ssh-connection-sftp-namespace.test.ts are
pre-existing on origin/main and untouched here.
2026-08-14 03:32:48 -07:00
Neil 5839ac2d84 fix(ssh): stop offering credentials to a host we just refused
A refused host key ended the handshake and then fell into the credential
ladder, because ssh2 reports a denied key as a generic auth failure and the
passphrase branch is eligible on message shape alone whenever an encrypted
identity file is configured. So the sequence was: decide this host may not be
who it claims to be, then ask the user for their passphrase and hand it to it.
Failing that, prompt for a password. Failing that, retry over the system ssh
binary, which for a disagreement with our own store rather than known_hosts
would simply connect.

That inverts the point of checking at all. A denied key is now final for the
attempt: recognised by type before any fallback runs, and again inside the
agent-fallback retry, which re-runs the handshake and so can be the attempt that
denies.

Two things had to change for that to hold.

The rejection is now a HostKeyVerificationError rather than a rebuilt Error, so
the connect path recognises it by type. Substring matching would have worked
today and quietly stopped working the first time a reason string was reworded —
and these strings are already worded around the auth-error classifier, so they
are exactly the kind that get edited.

And the verifier now reports the denials that skipped the policy: an
unreadable key blob and an internal failure both denied without calling
onDecision, so the connect path saw only ssh2's generic failure and walked the
ladder. That was the actual path the new test hit first. The report carries no
fingerprint, since there is no host key to identify, and the connection no
longer overwrites the fingerprint it holds with an empty one — the relay keys
install-lock isolation on that value.

The reconnect ladder also refuses to retry it. Its classifier is otherwise
substring-driven, so a reason containing "connection reset" would have been
retried until the ladder gave up, burying the reason under nine attempts.

Tests: no credential prompt after a refusal, an encrypted key configured so the
passphrase branch is eligible; refused reports as 'error', not 'auth-failed',
which would invite the user to re-enter credentials that are not the problem;
no retry; both bypassing denials report; a throwing listener still denies rather
than hanging the handshake.

Also drops three test files a bisect resurrected from before the revert that
deleted them.

172 tests pass across the three touched files; typecheck and lint clean.
Pre-existing on origin/main and untouched here: 3 failures in
ssh-connection-sftp-namespace.test.ts.
2026-08-14 03:29:23 -07:00
Neil a1331d2602 fix(ssh): verify host keys instead of accepting every one (STA-4319)
The actual fix. ssh-connection's verifier recorded a fingerprint and returned
true, so every ssh2 connection accepted every host key — no known_hosts consult,
no change detection. It now consults the user's known_hosts plus our own store
and refuses a changed, revoked or unverifiable key.

Phase 1 by design: no dialog. Unknown hosts are accepted and recorded
(accept-new semantics), because startup restore opens every previously-active
target at once, ephemeral VM targets present a new key each launch, and
paired-web connects run on the host desktop where a prompt would appear on
someone else's screen. The MITM defence lands now; the prompt is Phase 2.

Also sets algorithms.serverHostKey to lead with the types already known for the
host. Without it the type-scoped matching is a downgrade — an attacker who cannot
forge the key on file just presents another type and turns a hard failure into
first contact. Verified from ssh2's kex.js that the client list decides.

Denial replaces ssh2's generic handshake error with the specific reason, because
the reconnect ladder cannot distinguish a generic failure from a transient fault
and would retry forever against a decision that will never change.

An unreadable trust store degrades to known_hosts only rather than failing the
connect: a changed key is still refused, and a host trusted only by us falls back
to first contact and is re-recorded, reaching the same decision.

The ssh2 mock now uses the callback form and aborts the handshake on denial. As
written it called hostVerifier(key) with one argument and ignored the result, so
it would have passed against a verifier that never decides — flagged in the
design as a mock that had to change, not a test to quietly rewrite. Two new tests
pin the wiring rather than the module: an unidentifiable blob is refused, and a
well-formed key is accepted.

Note for review: commit 2d2a0880ba unintentionally swept in two modules built
concurrently (ssh-known-hosts-source, ssh-host-key-store) because I staged with
'git add -A'; its message describes only the verifier. Both are covered by their
own tests, but the attribution in that commit is wrong.

1461 SSH tests pass.
2026-08-14 03:03:45 -07:00
Neil 2d2a0880ba feat(ssh): build the host key verifier and the algorithm order that makes it safe
Still not wired into the handshake — that lands next. This is the piece that
turns a decision into an ssh2 callback, plus the half of the design that is easy
to forget because it lives in a different config field.

The verifier MUST be a plain function returning undefined. ssh2 does
'const ret = verifier(key, verify); if (ret !== undefined) verify(ret)', so an
async function returns a Promise — neither undefined nor falsy — and ssh2 accepts
the key immediately while ignoring whatever the callback later decides. Making
this async would silently restore exactly the accept-everything behaviour the
module exists to remove, so a test asserts the return value is undefined.

orderServerHostKeyAlgorithms is what makes type-scoped matching safe rather than
a downgrade. RFC 4253 gives the client's algorithm order priority, so leading
with the types we already hold for a host denies a server the choice of
presenting some other type to convert a hard failure into first contact. Without
it, an attacker who cannot forge the key on file just offers a different
algorithm. Revoked entries never contribute to that order.

Also fails closed on two paths that would otherwise hang or over-trust: a key
whose own length-prefixed header cannot be read is refused rather than reasoned
about, and a throw from any dependency denies, because ssh2 may not catch an
exception raised inside the verifier and the handshake would hang instead of
failing.

18 tests. Includes the two negative cases that matter — first-contact keys are
recorded, but keys we already know, rejected keys, ephemeral runtime targets and
a lax StrictHostKeyChecking are not.
2026-08-14 02:50:49 -07:00
Jinjing 637c7e94c9 Add SSH config host picker to add-host dialog (#12334)
* feat(ssh): add SSH config host picker for add-host form

Users can now click 'Fill from ~/.ssh/config…' to browse available SSH
config hosts in a picker, select one, and have the form automatically
prefill with resolved connection details (hostname, port, username, auth).

Previously, an 'import' button provided bulk sync on this form—confusing
and unhelpful when everything was already synced. That action is now
available as a secondary 'Add all' option in the picker.

* fix(ssh): import filter preservation and label fallback

- Reuse search loader on import completion to preserve active filter inside generation guard
- Fall back to hostname when manual host has no label, not empty string
- Make alias duplicate detection case-insensitive to match config picker behavior
- Validate host availability when restoring project group selection
- Add aria-selected attribute to picker options for accessibility

* fix(ssh): harden config picker import, alias folding, and host targeting

Review findings on the ~/.ssh/config picker + bulk add:

- Guard config-host resolution with a generation counter so a late resolve
  cannot overwrite a later pick or a form the user backed out of; freeze the
  other rows while a pick resolves.
- Stop "Add all N" from re-adopting deleted hosts — it now imports without
  reAdopt, matching the new-host count it advertises. Settings → Import keeps
  the explicit re-adopt path.
- Fold SSH aliases through a shared normalizeSshConfigAlias for import
  ownership, delete tombstones, reclaim, picker search, and the save-time
  duplicate check, which now occupies configHost *and* label like the picker.
- Persist GSSAPIAuthentication only when a parsed Host entry asks for it, not
  when `ssh -G` merely echoes the /etc/ssh system default.
- Fail closed with unavailable/setup-not-found when an explicit
  projectHostSetupId names a non-actionable host instead of silently creating
  the workspace on a sibling host.
- Cache the parsed config for the picker session (refresh on open/retry) so
  filter keystrokes no longer reparse and Include-expand the file, keep the
  filter usable during loads, add a Retry on load errors, explain an empty
  Identity file after a config fill, and drop the always-false aria-selected.

* refactor(ssh): centralize host result limit and extract folder group val

Move SSH_CONFIG_HOST_RESULT_LIMIT to shared types so the renderer's limit message
cannot drift from the host's query limit. Extract findActionableFolderProjectGroup
to avoid repeating the folder-host-availability check across the composer hook.

* fix(ssh): pass -F to ssh -G when HOME differs from passwd home

In E2E tests and sandboxes, isolated HOME can differ from the system
passwd home. OpenSSH resolves the default config via getpwuid (passwd),
while Node's loadUserSshConfig uses os.homedir() (HOME-aware). Pass -F
to explicitly specify the config path when they diverge, so ssh -G and
the picker resolve the same file.

* fix(ssh): verify config host exists before resolving with ssh -G

When a user edits ~/.ssh/config and removes a host, the import picker
should not fall back to ssh -G's echoed response (which treats any alias
as valid). Check the reloaded config file before resolving.

- Force reload config on each resolve to catch user edits post-open
- Reject aliases not in the current config before calling ssh -G
- Add test for deleted alias edge case
- Fix workspace-target fallback to honor explicit host selection

* fix(ssh): let tombstoned aliases be re-picked in the config picker

Allow users to reclaim a deleted SSH host by re-picking it from ~/.ssh/config. Tombstoned aliases now appear in the picker with a "Removed from Orca" badge and remain pickable, but don't count toward "Add all" operations — ensuring passive import never resurrects a deleted alias while still giving the user a recovery path.
2026-08-03 17:32:13 -07:00
Jinjing a000839465 Add first prompt to agent session history rows (#12085)
* Add first user prompt to AI Vault session history rows

Re-parse transcripts on demand to extract and display the untruncated first
user prompt for copy/reuse. List scans omit the body (payload/perf); UI loads
it when session details expand. Grok sessions extract the typed ask from
<user_query> envelope, skipping injected <user_info> bootstrap rows. Supports
Claude, Codex, Grok, and OpenCode agents.

* fix(ai-vault): split SessionTime out to pass max-lines lint

AiVaultSessionDetails exceeded the 400-line oxlint limit after adding
first-prompt UI; move SessionTime into its own module.

* fix(ai-vault): handle corrupt transcripts and fix OpenCode prompt captur

Corrupt transcripts now resolve null instead of rejecting the IPC call, matching behavior for other unavailable cases. OpenCode SQLite parsing now correctly captures all text parts from the earliest user message only, fixing truncation of large prompts and padding of small ones. Add stale-response guard in the UI to prevent late results from overwriting the current session when tabs switch. Consolidate text slicing via `sliceAtCodeUnitLimit` to avoid surrogate-pair splits across all callers.

* test(ai-vault): add first-user-prompt UTF-16 safety tests

Ensure truncation at safety limits doesn't split UTF-16 surrogate pairs,
preventing corruption of astral characters in captured prompts.

* fix(ai-vault): key first-prompt-card by session.id

Remounting the card on session switches prevents late responses from
a previous load from writing stale data into the component's refs.
Also improves conversation-turn key stability.

* fix(ai-vault): preserve first prompt after preview truncation

* refactor(ai-vault): improve first user prompt capture robustness and per

- Add 15s timeout to full-prompt load to prevent indefinite loading states
- Extract seedFullFirstUserPrompt helper for reuse across parsers
- Prevent AI-generated summaries from becoming the copyable first prompt
- Fix truncation detection in OpenCode SQLite by probing for N+1 rows
- Optimize text bounding to apply safety limit before toLowerCase
- Gate synthetic OpenCode path detection on agent type, not just # presence
- Add test coverage for remote execution host handling

* Fix FirstPromptCard loading state stranded by stale promise reuse

Clears loadPromiseRef during cleanup to prevent the dedupe handle from
causing StrictMode remounts to await stale in-flight requests. Stops loading
when session becomes non-loadable mid-request. Adds tests for StrictMode
double-invoke resolution and main-process timeout scenarios.

* refactor(ai-vault): split session parsers into modular files

Split secondary-parsers into individual files per agent type (copilot,
cursor, hermes, opencode) for improved modularity. Add test coverage
for first-user-prompt envelope handling: unwrap user_query tags and
reject bare user_info dumps.

* fix(ci): clear max-lines and flaky portal readiness check

Collapse an accidental multi-line regex wrap in ssh-connection-utils that
pushed counted lines to 301. Harden the latched-readiness test's ready
transition so CI load can re-observe attach after MutationObserver gaps.

* fix(ssh): extract proxy command helpers to pass max-lines

Move resolveEffectiveProxy/spawnProxyCommand out of ssh-connection-utils
so oxfmt line wrapping cannot push that file over the 300-line lint cap.

* capture first user prompt by ordering OpenCode messages by creation time

- Add `readOpenCodeMessagesInOrder` to rebuild transcript by timestamp, handling
  corrupt/partial files gracefully instead of discarding sessions
- Extract SSH proxy command tests to dedicated file; add backpressure handling
  and stderr draining to prevent proxy process stalls
- On Windows, reject unsafe characters in ProxyCommand values instead of
  pretending to escape them; properly format cmd.exe invocation with verbatim
  arguments
- Expand ProxyJump chains into -J plus final hop, mirroring OpenSSH behavior
- Decouple portal readiness reapply budget from flip-count budget via explicit
  constant
2026-08-02 18:45:55 -07:00
Jinjing ce5b639e03 fix(P1-B): recover SSH targets and remote file watchers after a network drop (#12032)
* fix(P1-B): recover system-SSH targets after a network drop

Two defects stopped a remote workspace auto-recovering after a blip.

runReconnectAttempt classified failures with isTransientError, which only
matches ETIMEDOUT/ECONNREFUSED/ECONNRESET by errno code or literal
substring. The system-SSH transport — the only transport FIDO2 and
ProxyUseFdpass targets can use — reports network failures as OpenSSH
prose ("System SSH connection timed out"), so the ladder published a
permanent 'error' on the first timeout and the target never came back
without a manual reconnect. isTransientReconnectError adds a
network-shaped prose table on top of isTransientError and is used only on
the reconnect path: connect() keeps the narrow classifier so an
unreachable host still fails fast instead of burning five 30s attempts
and five security-key touch prompts. Auth and passphrase failures stay
permanent on both paths.

runReconnectAttempt also had no generation fence, so a superseded attempt
published its cancellation as a permanent error over the winner's live
connection — reachable when a system-transport proc.onExit schedules a
reconnect while an attempt is still in flight. Cancellation now carries a
stable error name, and both connect() and runReconnectAttempt claim their
connectGeneration and stay silent when a newer attempt owns the state.

* fix(P1-B): retry a dropped watcher overflow marker on real capacity

emitWatcherOverflowToClient published the {kind:'overflow'} resync marker
with controlOverflow:'reject'. A full control queue rejects at admission
with no settlement callback, so the marker was silently discarded and the
remote File Explorer stayed stale until some later watcher event happened
to produce another one — for a quiet tree, possibly never.

The emitter now retains a rejected marker per (client, root) and
republishes it when the sink actually frees up. The existing
onLegacyPtyCapacity signal cannot drive that: it is gated on producer
retention, so it stays silent exactly under the dual-queue pressure that
caused the rejection. RelayDispatcher.onClientCapacity is an ungated
per-client capacity signal that fires on every writer settlement and
drain. It lives on the dispatcher rather than the writer so a retained
marker survives setWrite() replacing the primary sink, and setWrite
notifies capacity once afterwards so the marker does not wait on traffic
that may never arrive.

Retention is bounded to one marker per (client, root), released on
settlement and purged on client detach.

* fix(P1-B): address all review findings on SSH network recovery

Fix four issues from code review:

1. **Bug — admitted overflow markers lost on setWrite**: Retain markers when
   settlement fails `ok: false`, not just on admission rejection. Prevents
   desynced filesystem trees after SSH sink replacement.

2. **SSH error classification expanded**: Add missing OpenSSH patterns
   (`ssh_exchange_identification`, `connection closed by remote`) and new
   `isDefiniteSystemSshHostFailure()` classifier.

3. **ControlMaster retry optimization**: Skip second probe when first failure is
   already definite host-level (network timeout, refused, unreachable). Saves
   ~30s per reconnect ladder step.

4. **Overflow flush under dual-queue pressure**: Gate pending marker retries on
   control-lane headroom instead of re-attempting on every capacity notification.
   Reduces thrash proportional to producer traffic.

Add regression tests for marker republish on sink replacement and validate auth
error detection against live OpenSSH credential rejection messages.

* rm random doc

* fix(P1-B): skip credential-failure retries and recover watcher markers o

- Auth and passphrase errors fail immediately without retry attempts
- Bare "System SSH probe failed (exit 255)" is transient only for reconnect
- Watcher markers survive client invalidation when switching SSH connections
- Add network error patterns: "lost connection", "remote end closed"
2026-08-01 22:21:37 -07:00
Jinjing de75003df9 fix(P1-C): gate FIDO2 system-SSH transport on an OpenSSH binary (#12029)
* fix(ssh): gate FIDO2 system-transport on an OpenSSH binary

`ssh -G` echoes OpenSSH's built-in default identity list for every host, so
`usesDefaultPaths` was almost never true and the security-key gate returned
`!usesDefaultPaths || findSystemSsh() !== null` — forcing system transport
without checking that an `ssh` binary exists. `spawnSystemSsh()` then throws
`No system ssh binary found`, hard-failing connections that worked on ssh2.

The same flag also stopped the default scan at the first existing normal
private key, so a host that only accepts a FIDO2 key never reached system
OpenSSH when `~/.ssh/id_rsa` happened to exist.

Both decisions are independent of where an identity path came from: always
require `findSystemSsh() !== null` before forcing system transport, and scan
every candidate identity instead of stopping on the first normal key.
`shouldUseSystemSshTransport()` is untouched, so ProxyCommand / ProxyJump /
ProxyUseFdpass keep their intentional system transport.

* test(ssh): isolate connection tests from the developer's own FIDO2 keys

Transport selection now scans every default identity instead of stopping at
the first normal key, so a `~/.ssh/id_ed25519_sk` on the machine running the
suite would decide which transport the default-target tests take. Mock
`findSystemSsh` to null by default and opt the two security-key tests in.
2026-08-01 18:45:37 -07:00
JinjingandOrca a07427e970 fix(ssh, relay): keep remote sessions alive through reconnects and backpressure (#11999)
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps

Three independent connection killers found in the SSH/remote freeze audit.

FINDING A - the reconnect ladder never escalated for post-handshake drops.
scheduleReconnect() used the single published state.reconnectAttempt for both
the delay index and the give-up test, and runReconnectAttempt() zeroed it
before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every
post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay
redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host.
New SshReconnectLadder splits the delay index (advanced by every retry) from
the failure streak (advanced only by a failed handshake), so flaps back off
while give-up semantics stay byte-identical to shipped.

FINDING B - notify() closed the client whenever a frame exceeded the producer
frame capacity, conflating a permanently un-sendable frame with transient
backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the
watcher flood killed the link and re-killed on every reattach+replay. notify()
now drops and logs once per generation; fs.changed is chunked to each sink's
capacity with a control-lane overflow marker as the resync fallback; agent-hook
envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit.

FINDING B2 - sendResponse routed >1MB responses to a lane whose admission
ignores the frame cap and closed the client on rejection, so a large
fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the
request fails instead of the connection.

Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame
cannot be dropped by the producer-lane check.

Co-authored-by: Orca <help@stably.ai>

* fix(relay): stop the overflow marker from re-killing the link it protects

Round-1 review fixes on the P0 freeze work.

The control-lane overflow marker could reinstate the exact failure this P0
removes: dispatcher-client-writer closes the client when control-lane
admission fails, and admitControl is the only lane that returns an error, so
one marker per failing batch accumulated to the 256-frame/1MB bound and
dropped the link. Markers are now deduped to one outstanding per
(client, root), cleared on settle.

Chunking also defeated the renderer's per-payload directory dedupe -- events
are now stable-grouped by parent directory so one directory lands in one
chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast
path paid three JSON encodes; both are fixed by publishing first and sizing
from a measured bytes-per-event estimate.

Agent-hook shedding now surrenders the blocking interactive prompt LAST
rather than first, so a degraded envelope cannot strand a pane at
state=waiting with no answerable question card.

The dropped-notification log now distinguishes over-capacity from producer
queue backpressure and no longer lets the first dropped method silence every
other producer for the life of the connection.

* fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another

Round-2 review fixes.

The round-0 change from close-on-rejection to silent drop removed the only
redelivery path for agent.hook envelopes: they are fire-and-forget and the
per-pane cache only replays on handler install, so a saturated link stranded
a pane on a stale Working spinner until reconnect. Closing used to guarantee
delivery by forcing that replay. Envelopes now publish per client and pend
for bounded latest-wins redelivery when the producer queue rejects them.

Shed fields are now named on the wire. The subagent roster is not cosmetic --
the renderer replaces rather than merges it, and hibernation gates on its
length -- so an unmarked shed could sleep a live pane.

fs.streamEnd rode the control lane because it must not be dropped, but that
lane kills rather than drops. The stream's concurrency slot is now held until
the terminal frame settles rather than until the fd closes, capping queued
terminal frames well under the control budget; overflow costs one refused
read instead of the connection.

The watcher chunk walk now stops while producer retention sits past its
reserve and degrades to a resync, so a 5000-event flood cannot fill the queue
that interactive PTY traffic shares and stall every remote terminal.

The reconnect ladder caps its flap-path delay so delay plus handshake timeout
cannot cross the relay grace floor and let the remote daemon kill live PTYs.

Also: the suppression key no longer embeds a NUL byte, which had made the
file binary to git and grep; producerEnvelopeBudget no longer reports
infinite capacity for a departed client; the drop logger no longer encodes a
frame it will not log; and an over-capacity response substitution no longer
settles as if the result had been delivered.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure per client

Round 3 + 4 review fixes.

Watcher chunking is now gated on the *client's* retention reserve rather than
the dispatcher-wide one, so one stalled peer no longer forces a healthy client
into a full file-tree resync. The relay-lost redeploy ladder no longer burns its
6-attempt budget while the SSH transport itself is down: it holds at the 15s step
with a non-terminal status and rearms, so a laptop that slept past the ladder
comes back instead of landing on a terminal "give up" banner.

The shedFields wire marker had no consumer, so an agent-hook envelope whose
subagent roster was dropped to fit the frame read as "roster cleared" on the Orca
side: live child rows blanked and a done pane became hibernation-eligible while
its teammates were still running. ingestRemote now restores shed fields from the
cached payload (interactivePrompt deliberately excluded — a stale answerable
question card is worse than none).

Also: stream terminal-frame slots are counted per client, since the control queue
they protect is per client; the chunking fast path no longer logs a drop for a
batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity.

Test debt from the review: pending-pane eviction, per-client stream isolation, and
the reconnect budget are now asserted rather than assumed; four fragile exact-byte
pins dropped in favour of the tier comparisons that carry the requirement.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure

- Oversized relay responses now fail their request instead of closing the connection,
  preventing one frame from killing every pane on the host
- Restore subagent state for correct hibernation; don't resurrect stale prose
  across turns
- Account for relay re-establishment and PTY reattach time in SSH flap delay caps
- Only log drops of final unsendable envelopes, not temporary rejections during
  measurement probes
- Fix watcher overflow marker release race when notification admission rejects
  without settlement; use precise byte counting for event batching

* Restore relay-shed fields with digest validation and scoped backpressure

Validate that shed subagent rosters match their wire digest and turn identity before
restoration, preventing stale roster resurrection. Compact interactive prompts for waiting
states instead of dropping them. Demote control-queue overflow to non-fatal rejection so
clients can retry on capacity recovery, keeping the link alive during transient backpressure.

* fix(relay): correct ResponseOverCapacity error code

ResponseOverCapacity should use -33008 to stay in the -33xxx range
for relay protocol errors, not -32010.

* fix(relay): close client when pty.replay overflows control queue

Replay is never retried, so it uses the control lane where overflow
is fatal — the writer closes the client and reconnect reloads history
rather than stranding a short buffer.

* fix(relay): prevent infinite redeploy on flapping SSH transports

Charge reconnect attempts when connection restores mid-backoff, preventing
infinite loop on transports that flap between states. Refactor control overflow
handling to use entry property instead of WeakSet marker for clarity.

---------

Co-authored-by: Orca <help@stably.ai>
2026-08-01 14:27:16 -07:00
Neil 33c14bc716 fix(ssh): fall back to OpenSSH for FIDO2 keys (#11913)
Closes #11645
2026-08-01 03:27:42 -07:00
Henry SuandOrcaWin 5fe3aaf2b7 fix(ssh): preserve first config directive value (#11297)
* fix(ssh): preserve first config directive value

* test(ssh): cover false-first config booleans

* fix(ssh): trust fresh OpenSSH config authority

* fix(ssh): preserve ordered config identities

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-30 14:23:30 -07:00
JinjingandOrcaWin a40183389b feat: bound direct SSH reconnect fan-out and recovery (#11003)
* docs: design for direct SSH reconnect fan-out

Capture the implementation-ready plan for host-qualified, epoch-fenced
SSH reconnect recovery after two rounds of multi-model LLM counsel review.

* docs: reconcile SSH reconnect fan-out design

* docs: close reconnect design consistency gaps

* feat: implement bounded direct SSH reconnect recovery

* fix: bound direct SSH retry settlement

* fix: harden direct SSH reconnect authority

* fix: preserve split SSH retry ownership

* fix: preserve SSH split continuation authority

* docs: record final SSH reconnect validation

* fix: preserve SSH authority through retained and detached state

* fix: retain SSH authority across delayed split mounts

* fix: close SSH authority recovery gaps

* fix: fence stale SSH transport replacement

* fix: serialize SSH target teardown

* fix: settle SSH teardown failures before reconnect

* fix: retire failed SSH reset sessions

* test: reconcile current main E2E contracts

* fix: close direct SSH reconnect review gaps

* fix: fence stale SSH reconnect side effects

* fix: close final SSH reconnect lifecycle gaps

* test: stabilize current-main reliability gates

* test: prove plugin navigation containment

* test: make plugin navigation oracle authoritative

* test: make plugin navigation oracle deterministic

* ci: allow sharded e2e suite to finish

* test: wait for runtime pane publication

* test: classify pane readiness by error code

* test: select close persistence terminal by tab identity

* docs: mark reconnect implementation validated

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-28 12:33:17 -07:00
Neil badf91101b fix(quality): enforce performance-safe lint baseline (#11074)
* fix(quality): clear safe existing lint findings

* fix(quality): keep lint cleanup allocation-free

* fix(quality): enforce performance-safe baseline

* test(terminal): drain deferred confirmation cleanup
2026-07-27 20:54:02 -07:00
6d39e49480 fix(ssh): accept GitHub restricted-shell SSH probes (#6988) (#7659)
* fix(ssh): accept GitHub restricted-shell SSH probes (#6988)

* fix: match first stderr line for GitHub restricted-shell probe (bug-bash takeover)

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
Co-authored-by: Orca <help@stably.ai>
2026-07-24 00:24:53 -07:00
NeilandOrca aab112933e Revert "fix(memory): bound OOM-prone accumulators (#10179)" (#10255)
Co-authored-by: Orca <help@stably.ai>
2026-07-23 18:35:31 -07:00
Neil 8f40ddf328 fix(memory): bound OOM-prone accumulators (#10179) 2026-07-23 06:22:56 -07:00
4f81dfc128 perf(ssh): cut warm high-latency connects from 88.7s to 7.7s (#9015)
Move managed agent-hook filesystem work behind one relay RPC so high-latency SSH connects pay one WAN round trip instead of hundreds. Keep installers serial, lock shared account config across relay processes, and fence cancelled connection generations from replacement state.

Co-authored-by: nasagong <zinho2000@gachon.ac.kr>
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-20 12:18:57 -07:00
gatsby74andJinjing 8ed8f0d109 feat(ssh): download folders from remote explorer (#7793)
* feat(ssh): download folders from remote explorer

* fix(ssh): harden remote folder downloads

* Add missing getRepo stub to worktree cwd test mock

Restoring headless mobile tabs looks up the repo for the active
worktree id; the mock lacked getRepo, so the test only passed
incidentally. Add it explicitly and return undefined since wt-1
is a worktree id, not a registered repo.

* Enable SSH folder downloads, gated for system-SSH connections

- Folder downloads require SFTP, unavailable on system SSH (which
  offers only raw file operations). Add supportsFolderDownload flag
  to gate the feature in the UI layer.
- Reject symlinks at directory-entry level, preventing tree escapes
  and eliminating unnecessary stat calls.
- Check abort signal before opening dialog for better responsiveness
  when renderer closes.
- Log cleanup errors without re-throwing to preserve underlying
  transfer failures.

* Gate SSH folder downloads to SFTP-capable connections

Enforce fail-closed gating and add Windows path traversal validation to
ensure downloads are only available when explicitly supported and safe.

* Gate SSH folder downloads to SFTP-capable connections

Enforce fail-closed gating and add Windows path traversal validation to
ensure downloads are only available when explicitly supported and safe.

* fix(ssh): keep provider types under max-lines after main merge

Move FolderDownloadOptions next to the SFTP download implementation and
narrow IFilesystemProvider.downloadFolder options to AbortSignal only so
types.ts stays within the 300-line oxlint budget when merged with main.

---------

Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
2026-07-18 23:01:29 -07:00
fa85536f3a fix(ssh): repair unbuilt relay native deps (#8686)
Co-authored-by: Orca <help@stably.ai>
Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com>
Co-authored-by: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com>
2026-07-16 21:32:13 -07:00
Haohan Lin 190ed7d4ed fix(ssh): make POSIX relay wrapper survive csh/tcsh login shells (#8709)
Reconstruct remote POSIX commands with bounded printf arguments so non-POSIX SSH login shells can forward them without requiring remote base64. Preserve relay and system-SSH stdin, and centralize login-shell flag selection for csh/tcsh compatibility.\n\nValidated against real csh and tcsh OpenSSH targets with built-in and system SSH, including cold relay deployment, stdin upload, PTY I/O, file mutation, and reconnect.
2026-07-15 02:38:40 -07:00
0302ae86b8 feat(ssh): support Kerberos/GSSAPI hosts via the system OpenSSH transport (#7507)
* feat(ssh): support Kerberos/GSSAPI hosts via the system OpenSSH transport

ssh2 has no gssapi-with-mic support, and adding it would mean forking its
protocol layer plus packaging the kerberos native module for three
platforms. Instead, route GSSAPI hosts through the existing system-OpenSSH
transport, which delegates Kerberos (tickets, SSPI on Windows) to the
platform ssh binary.

Two tiers, because RHEL-family distros enable GSSAPIAuthentication
globally in /etc/ssh/ssh_config and ssh -G therefore reports it for every
host:

- Targets whose ~/.ssh/config Host block explicitly sets
  GSSAPIAuthentication yes (imported as target.gssapiAuthentication) try
  system ssh first, falling through to ssh2 so key auth and credential
  prompts still work when no ticket is available.
- When ssh2 exhausts key/agent auth and the ssh -G-resolved config
  enables GSSAPI, retry over system ssh before prompting for credentials,
  so Kerberos-only hosts on distro-default configs connect without a
  password prompt. Hosts where keys work never leave the ssh2 path.

Manual targets flagged for GSSAPI pass -o GSSAPIAuthentication=yes
explicitly since they bypass ssh_config. Both tiers work headless (no
credential callbacks required).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ssh): harden GSSAPI transport selection (review fixes for PR #7507)

Review fixes on top of the Kerberos/GSSAPI feature branch (s546126/kerberos-ssh):

- HIGH: reset useSystemSshTransport on the ssh2 fall-through. doSystemSshProbe
  sets the flag before spawnSystemSshCommand, which throws synchronously when no
  system ssh binary is on PATH (outside the probe try/catch). The proactive
  fall-through previously reset only 2 of 3 transport fields, so exec/sftp kept
  routing through the failed transport - breaking GSSAPI on Windows-with-Git-ssh
  and headless Linux.
- MEDIUM: throw a cancellation error (not the stale ssh2 authError) when a
  disconnect supersedes the reactive probe mid-flight, and guard connect()'s
  catch on disposed, so a deliberate disconnect is not overwritten with
  auth-failed.
- MEDIUM: skip the encrypted-key passphrase prompt when the GSSAPI fallback
  applies, so a Kerberos ticket is tried before prompting; the general prompt
  still fires if the probe fails.

Adds 3 mutation-verified regression tests and hardens two existing tests to
assert the probe actually ran. Not connected to any PR remote.

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): isolate GSSAPI system transport

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: s546126 <268420947+s546126@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com>
Co-authored-by: Orca <help@stably.ai>
2026-07-15 01:58:46 -07:00
c2aa9c2ead feat(ssh): support file transfers over system ssh (#7804)
* feat(ssh): support file transfers over system ssh

* fix(ssh): harden system file transfers

* test(runtime): stub getRepo in headless mobile tab cwd test

The mobile-session selector validator (getValidatedExplicitWorktreeIdSelector,
from main) calls this.store?.getRepo to reject repo ids passed as worktree ids.
The store stub only implemented getWorkspaceSession, so the guarded call threw
'getRepo is not a function' once main merged into this branch. Add a getRepo
that returns null (wt-1 is a worktree, not a repo).

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Co-authored-by: Orca <help@stably.ai>
2026-07-12 01:29:55 -07:00
f215a48064 fix: pr-bug-scan validated finding from #6952 (#7180)
* fix: address pr-bug-scan validated finding from #6952

throwNodeNotFound() now re-raises AbortError when the shared signal is aborted, so a signal-cancelled node probe no longer launders into 'Node.js not found'; sequential fallback runs.

* fix(ssh): make session-limited (MaxSessions=1) relay deploys actually succeed

Review of #7180 verified the parent fallback end-to-end against a real
MaxSessions=1 sshd and found the connect still failed. Four gaps, in order
of discovery:

- isSshSessionLimitError missed stock OpenSSH, which refuses session
  channels over MaxSessions with SSH2_OPEN_CONNECT_FAILED (2) and 'open
  failed' — reason 4 never matched, so the fallback never triggered.
- execCommand settled aborted commands before the channel finished
  closing, so the sequential fallback reissued execs while sshd still
  counted the old session.
- SshConnection.waitForSshCallback rejected aborts mid-channel-open
  immediately, leaking a confirmed-late channel that held the only
  session slot; it now settles after the late channel closes (bounded)
  and drains its streams so ssh2 emits 'close'.
- Session channel opens now retry transient session-limit refusals
  (sshd frees the slot only after processing our close-ack, which the
  next open can beat by microseconds), and the remote orca CLI shim
  install is non-fatal like the managed-hook install — after the relay
  bridge occupies the sole session slot, raw-connection extras must
  degrade instead of failing the connection.

Verified live against Docker sshd (OpenSSH 9.2, MaxSessions=1): fresh
deploy (upload + native deps + launch), reconnect cycles, and a PTY
round-trip all succeed; unrestricted-sshd regression run also passes.

Co-authored-by: Orca <help@stably.ai>

* Handle ssh execution aborts immediately during retry backoff or hangs

- Cancel the session-limit retry delay immediately if the operation is
  aborted during backoff.
- Limit the wait time to a 5-second grace period when aborted during a
  channel open that is hung and never invokes its callback, rather than
  waiting for the full connection timeout.

---------

Co-authored-by: orca-bug-scan-bot <orca-bug-scan-bot@stably.ai>
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Co-authored-by: Orca <help@stably.ai>
2026-07-04 01:26:55 -07:00
Brandon Barker e6ca3087c5 perf(ssh): resolve node path concurrently with remote install state (#6952)
Run independent SSH relay bootstrap probes concurrently when the connection can safely support overlapping execs. Preserve the old sequential path for system SSH without reusable ControlMaster and for remotes that reject concurrent session channels.
2026-07-02 13:55:36 -07:00
4dbc9f3817 feat(ssh): add ControlMaster multiplexing for system SSH transport (#6922)
* feat(ssh): add ControlMaster multiplexing for system SSH transport

System SSH transport spawns a new OpenSSH process per exec command
(platform detect, relay install check, node resolution, relay launch,
socket probe). Each process pays the full SSH handshake cost — ~9s on
Uber devpods — making a typical relay connect take 54s+ and reliably
exceeding the 15s startup reconnect budget.

Add SSH ControlMaster multiplexing via a per-target socket in
$TMPDIR/orca-ssh-ctl/<hash>.sock. The first command establishes the
master; subsequent commands reuse it at ~100ms per exec instead of ~9s.
ControlPersist=300 keeps the master alive after commands exit so rapid
reconnects (e.g. on tab focus) also benefit. Windows is excluded since
OpenSSH's ControlMaster support there is limited.

* fix(ssh): address ControlMaster key collision and directory permission risks

- Use target.id in the socket key so distinct SSH targets can never
  collide even when configHost/port/user happen to match
- Switch from SHA1 to SHA256 and extend hash slice from 12 to 16 chars
- Stat the control-socket directory after mkdirSync to reject pre-existing
  dirs that are symlinks, foreign-owned, or have group/other write bits
  (mkdirSync mode is ignored on pre-existing dirs)
- Update two tests that used exact spawn-arg arrays; replace with
  ordering assertions (forward flags before --) that stay correct
  regardless of which extra ControlMaster options are injected

* fix(ssh): bind ControlPath identity to route and reject symlinked ctl dir

Fold proxyCommand/jumpHost/identity fields into the ControlPath hash so a
target whose route is edited no longer reuses a still-alive master built on
the old route. Switch the control-socket dir check from statSync to lstatSync
so a planted symlink fails the directory validation outright.

* test(ssh): drop tautological argv re-assertion in spawn checks

The toHaveBeenCalledWith re-passed the args array extracted from the same
mock call, making that argument position always pass. argv content is
already verified by the index-ordering assertions above; use expect.any(Array)
so the spawn check only claims what it actually verifies (binary path, stdio).

* fix(ssh): harden system ssh connection reuse

Co-authored-by: Orca <help@stably.ai>

* test(ssh): isolate control socket runtime dir

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Test <test@example.com>
Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com>
Co-authored-by: Orca <help@stably.ai>
2026-07-01 21:29:48 -07:00
NeilandOrca 46646d7ff1 chore(lint): upgrade oxlint to 1.71 + enable 7 new rules (autofixed backlog) (#6841)
* chore(lint): upgrade oxlint to 1.71 and enable 7 new rules

Upgrade oxlint 1.67.0 -> 1.71.0 (1.72 was blocked by the repo's 3-day
minimum-release-age supply-chain guard; nothing here needs it). The
bump is a no-op on the existing config.

Enable 3 error rules (backlog autofixed to zero in this commit) and
4 warn rules (surface signal without gating CI):

error (autofixed, behavior-preserving):
- unicorn/prefer-node-protocol        (~1531 sites: bare builtin -> node:)
- typescript/no-import-type-side-effects (~36: all-inline-type -> import type)
- unicorn/no-array-reverse            (19: copy-then-reverse -> toReversed)

warn (real signal, current fires are test-only/correct):
- unicorn/no-array-fill-with-reference-type  (aliasing footgun guard)
- typescript/no-unsafe-function-type         (bans bare Function type)
- unicorn/prefer-array-flat-map              (map().flat() -> flatMap())
- unicorn/prefer-regexp-test                 (.match() in bool ctx -> .test())

mobile/.oxlintrc.json extends root, so it inherits all 7; the autofix
ran from root and covered mobile/ too.

Verification (all green): oxlint 0 errors (root+mobile+aux configs),
oxfmt clean, typecheck (node+cli+web), vitest 22795 passed / 0 failed,
builds (electron-vite + web + cli) succeed. node: rewrites confirmed to
skip embedded SSH/CLI string payloads (AST-only); all toReversed sites
verified to operate on fresh copies or write-once locals.

* chore(lint): bump mobile oxlint to 1.71 so inherited rules parse

mobile/ is a standalone pnpm project pinning its own oxlint@1.67, which
lacks unicorn/no-array-fill-with-reference-type (needs >=1.70). Since
mobile/.oxlintrc.json extends the root config, mobile CI's 'cd mobile &&
oxlint' failed to parse the new rule. Bump mobile to match root (1.71).

Verified in mobile/: oxlint 0 errors, oxfmt --check clean, tsc --noEmit
pass, vitest 978 passed / 0 failed.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-06-29 22:38:29 -07:00
Jinwoo HongandOrca c455c576ae Fallback to system SSH on reachability errors (#5303)
Co-authored-by: Orca <help@stably.ai>
2026-06-12 20:06:20 -07:00
98d02bca47 fix: support windows ssh hosts (#5004)
* feat: add windows ssh relay base support

* feat: support windows ssh relay runtime services

* fix: default windows ssh pty cwd to user profile

* fix: support windows hosts over system ssh

* fix: preserve degraded windows relay native deps

* fix: gate windows shell args by relay platform

* fix: preserve windows relay fallback pipes

* test: align windows native deps relay fixture

* fix: build valid windows install lock command

* fix: address windows SSH relay review findings

Resolve correctness, efficiency, and reuse issues found reviewing the
Windows SSH native-host support:

- GC liveness on Windows now probes the actual named pipe (via node
  net.connect against markers + deterministic candidates) instead of
  substring-matching Win32_Process command lines, which could remove a
  live relay dir. Reports ALIVE conservatively only when there is no
  liveness signal at all (no markers and no seed pipes).
- Resolve the remote node path once per deploy and thread it through
  install/repair/launch instead of re-resolving 3-7x.
- Replace the 200ms node -e poll loop with a single long-lived remote
  wait process during Windows relay startup.
- Skip the no-op executable command on Windows in uploadRelay.
- Make the Windows fallback pipe name deterministic and recoverable
  (drop the global counter), with an extra reconnect attempt.
- Normalize the prepended node bin dir to backslashes on Windows PATH.
- Batch the system-SSH Windows directory upload into a single streamed
  JSON package instead of one ssh process per file.
- Extract relay endpoint/marker helpers into ssh-relay-endpoints.ts and
  consolidate the PowerShell EncodedCommand encoding into the shared
  powershell-command-encoding module.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Support cancellation and timeouts in Windows port scanning

- Propagate the request AbortSignal and a 5-second timeout to both
  PowerShell and netstat child processes during Windows port scanning.
- Avoid spawning the netstat fallback process if the port scan has
  already been aborted.
- Wrap the .NET OSArchitecture check in a try/catch block during SSH
  Windows platform detection to robustly fall back to environment
  variables if needed.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
2026-06-09 01:17:34 -07:00
Jinwoo HongandOrca 9c2cec3d6c Use system ssh for proxy-based targets (#4911)
Co-authored-by: Orca <help@stably.ai>
2026-06-08 14:43:25 -07:00
Neil c7c2330723 fix: close late ssh channel callbacks (#4116) 2026-05-31 04:48:28 -07:00
ccfc1a9b97 fix: time out ssh channel opens (#3742)
* fix: time out ssh channel opens

* test: cover ssh channel-open timeout edge cases

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Jinwoo-H <jinwoo0825@gmail.com>
Co-authored-by: Orca <help@stably.ai>
2026-05-30 11:43:40 -07:00
Neil 8044be7cc7 fix: clean up system ssh probe listeners (#3824) 2026-05-30 11:32:49 -07:00
Jinjing 41f0bc6394 test: preserve ssh mock event listeners (#3015)
* test: preserve ssh mock event listeners

* test: reuse runtime file command helper
2026-05-28 18:30:35 -07:00
Jinjing 186327af08 fix: guard ssh startup destroy errors (#3013) 2026-05-28 16:13:09 -07:00
Neil 9d396e115c fix: remove SSH startup listeners after connect (#2882) 2026-05-26 20:24:36 -07:00
Neil eb579a0e54 Fix SSH sessions after host sleep 2026-05-26 14:53:16 -07:00
Jinjing ffec74c7ba Fix SSH agent and identity file auth ordering (#2681)
* fix: address review findings

* fix: address CI failures
2026-05-23 01:56:15 -07:00
Jinwoo HongandOrca 9c67557a0c Support system SSH transport for fdpass proxies (#2279)
Co-authored-by: Orca <help@stably.ai>
2026-05-18 16:03:34 -07:00
Neil fdc38bfc33 Fix SSH relay connect when default shell is fish (#2239) 2026-05-18 01:31:45 -07:00
Jinwoo HongandOrca 9a39b1345a fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660) (#1679)
* fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660)

ssh2 leaves Nagle's algorithm on by default. For single-byte keystrokes
through a remote PTY, Nagle interacts with the kernel's delayed-ACK timer
and adds up to ~40 ms per keystroke — visible as the typing lag reported
in #1660. OpenSSH's `ssh` client sets TCP_NODELAY whenever a PTY is
allocated; this change mirrors that on the ssh2 client right after the
`ready` event in doSsh2Connect, covering both initial connect and
auto-reconnect.

Proxy-command / proxy-jump connections (where ssh2's underlying socket
is a custom Duplex over a child-process pipe) are a no-op by design,
gated by the public Client.setNoDelay()'s own type guard. A discriminating
log line records which path each connect took.

Tests cover initial connect and a full reconnect cycle to guard against
the regression class "Nagle is re-enabled because someone refactored
only the initial connect path."

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): bound relay-lost reconnect with exponential backoff

When the relay exec channel keeps dying (e.g. a remote-side bug closes
every fresh --connect channel right after handshake, or a stale bridge
keeps being replaced), the unguarded _onRelayLost handler reconnects as
fast as the network allows — spawning relay deploy attempts in a tight
loop until the user force-quits. Each iteration spawns a fresh ssh2 exec
channel, hammers sshd's MaxSessions counter, and floods the renderer
with state churn.

Add per-target exponential backoff (500ms → 15s, capped at 6 attempts)
so the loop terminates instead of running forever. After the cap the
session goes to 'error' state with a 'Relay channel kept dropping.
Please reconnect.' message — visible in the renderer instead of an
invisible failure where typing in remote terminals just stops working.

Successful 'ready' resets the attempt counter only if the session
stabilized for >= 5s; faster flaps preserve the counter so a flaky
remote backs off rather than retrying indefinitely on every brief
ready→lost cycle.

Backoff state is cleared on explicit disconnect, on session replacement
during reconnect, and on connect failures, so a real reconnect attempt
after backoff exhaustion always starts from zero.

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): detect stale relay daemons via running-version marker

The on-disk relay version check compares local .version against the
remote .version file in the relay dir. A daemon launched by an earlier
deploy keeps running its in-memory copy of the OLD relay code, so when
the client later rewrites relay.js + .version on disk and bridges in
via --connect, the new bridge process drives a stale daemon. Protocol
or behavior changes between the two versions then tear down the
channel in a tight reconnect loop (observed against PR #1672 on a
daemon predating that change).

The daemon now writes its running version into a .running-version
sidecar at startup, anchored to the relay-script directory rather than
process.cwd() so test spawns cannot pollute the repo root. Before
attaching to an existing socket, the client probes that marker and,
on mismatch with the locally-deployed .version, kills the stale
daemon (TERM only, never KILL) and falls through to a fresh launch.
Conservative defaults: when either marker is unreadable, attach so
older builds keep their live PTYs.

Co-authored-by: Orca <help@stably.ai>

* Revert "fix(ssh): detect stale relay daemons via running-version marker"

This reverts commit e58acf07c0.

* fix(ssh): isolate relay versions via per-version install dirs and wire handshake

The relay's previous single-dir layout (~/.orca-remote/relay-v0.1.0/) let
the deploy step rewrite relay.js in place while a daemon was still loaded
in memory at the previous version. New clients then drove that stale
daemon, surfacing as a reconnect loop (issue #1660 follow-up) and the
field failure observed against an 8-day-old daemon on openclaw.

Switch to a VS Code-style versioned layout where each (RELAY_VERSION +
content-hash) bundle installs into its own directory and is never
mutated after install. A v2 client's --connect socket path is rooted in
relay-${v2-hash}/ and structurally cannot reach a v1 daemon's socket.

Defense-in-depth: the daemon now reads exactly one Handshake-typed frame
on each newly-accepted Unix socket before attaching the JSON-RPC
dispatcher (mirrors VS Code's remoteExtensionHostAgentServer.ts:340).
Mismatch closes the socket; the bridge exits with code 42; client maps
that to a typed RelayVersionMismatchError and skips the relay-lost
backoff loop instead of retrying through 6 attempts.

Other deploy hardening:
- atomic mkdir-based install lock with stale-lock recovery serialises
  concurrent first-installs of the same version
- .install-complete sentinel distinguishes a finished install from a
  crashed-mid-install partial that should be retried
- gcOldRelayVersions removes unreferenced sibling dirs (allowlist regex,
  skips locked or incomplete dirs, skips dirs with a live socket)
- readLocalFullVersion fails fast on a missing/empty local .version
  rather than silently falling back to a path where a daemon from a
  different code generation may already be running

Includes a cross-version isolation test that fails any future refactor
which collapses the per-version layout back to a shared dir.

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): harden relay versioning per review feedback

Address must-fix and should-fix findings from the parallel triple review of
26d1666e:

- Surface RelayVersionMismatchError to ssh.ts on initial establish() (not
  just reconnect), so the user sees the typed terminal error instead of
  silent retry on first connect (#13).
- Give the sentinel timeout a 500ms grace window for the close handler to
  deliver exit-42, so a slow remote does not misclassify a wire-handshake
  mismatch as a generic timeout (#D11).
- Drain the handshake decoder's residue at the handshake -> dispatcher
  transition on both daemon and --connect sides; pipelined frames that
  were coalesced with the handshake are now forwarded into the dispatcher
  / stdout instead of silently dropped (#A1, #A2).
- Reset the install-lock acquire timer after a stale-lock recovery so a
  single post-recovery race does not immediately exhaust the budget (#E14).
- Treat a stale install-lock as recoverable in the GC pass when
  .install-complete is present (covers an interrupted finalize where the
  rm-lock failed) (#E15).
- GC legacy relay-v\d+\.\d+\.\d+ install dirs whose daemons have died,
  now that .install-complete is no longer required for them (#12).
- Resolve symlinks in readLaunchVersion() so a daemon launched via a
  symlinked entry script still reads .version next to the real file (#G21).
- Flush stderr before exit-42 in --connect handshake mismatch path so the
  diagnostic line reaches the client before the process tears down (#C8).

Tests:
- Round-trip handshake over a real Socket pair: matching version, mismatch
  exit-42, leftover bytes preserved on both sides when frames are
  coalesced with the handshake.
- waitForSentinel exit-42 -> RelayVersionMismatchError, exit-1 -> generic.
- SshRelaySession terminal-error callback fires on both establish() and
  reconnect() when deployAndLaunchRelay throws RelayVersionMismatchError.
- acquireInstallLock concurrent BUSY -> OK polling, stale-lock recovery
  with reset timeout window, and fresh-lock timeout failure path.
- gcOldRelayVersions stale-lock-with-complete branch, legacy-dead path,
  legacy-alive path; existing locked-test asserts fresh-lock now keeps.
- Cross-version isolation test now asserts a blanket invariant that every
  v1-referencing command from a v2 deploy is a read-only liveness probe.

Lint and typecheck clean across all 3 tsconfigs; 426 SSH/relay tests pass.

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): bypass npm init for content-hashed relay dirs and harden install probe

The versioned-install dirs land at `relay-${version}+${hash}/` (e.g.
`relay-0.1.0+07994a7870e1`). npm 11 / Node 26 reject the `+` in derived
package names and `npm init -y` exits 1 — silently, since both stderr
and the failure landed inside the `2>/dev/null && ...` chain. The catch
swallowed the throw, `.install-complete` was written anyway, and every
reconnect surfaced 'node-pty is not available' at first pty.spawn.

Sidestep `npm init` entirely: SFTP-write a hardcoded minimal
package.json (`name: orca-relay`, `type: commonjs`) and run
`npm install node-pty` directly. `type: commonjs` pins the module
system against future Node default flips or remote-side .npmrc overrides.

Also harden the install path against the same class of silent failure:
- npm install errors now propagate (no more `.install-complete` on hard
  fail; future reconnects retry instead of stranding the user)
- Replace the weak `test -d node-pty` post-install probe with
  `node -e 'require("node-pty")'` so built-but-unloadable installs
  (missing prebuild, wrong arch, broken native binding) surface clearly
- Add a session-level error handler on the SFTP write so a torn-down
  session rejects the promise instead of hanging until enclosing timeout

Separate fix: add `for-each-ref` to the relay's git subcommand allowlist.
Client code (`src/main/git/repo.ts` ref-search and worktree-listing)
calls `git for-each-ref` over SSH; the relay was rejecting it. The
`--shell`/`--python`/`--perl`/`--tcl` format flags only control output
quoting (no eval) and the relay invokes git via execFileAsync (no shell),
so the read-only allowlist treatment matches `rev-parse`, `log`, etc.

Co-authored-by: Orca <help@stably.ai>

* comment(ssh-relay): TODO link to #1693 for VS Code-style pre-bundled node-pty

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): harden node-pty install probe and tighten review-fix tests

Round-3 review fixes on top of 963f56d7.

deploy.ts:
- Replace endsWith('OK') with includes('ORCA-NPTY-PROBE-OK'). Node can emit
  deprecation/experimental warnings to stderr after our stdout 'OK' write,
  and 2>&1 would push them past 'OK' producing false NPTY-MISSING warnings.
  A unique sentinel survives any trailing stderr noise.
- Switch sftpPkg/ws .on -> .once for error/close. A late session 'error'
  after the promise had already settled would otherwise become an unhandled
  EventEmitter error and crash main.
- Trim per-block comments to 1-2 lines per AGENTS.md (was 7-9).

Tests:
- Pin the BEFORE-ordering contract: SftpWriteCapture now records the count
  of execCommand calls observed at the moment ws.end() ran for each path,
  and the test asserts that count <= the index of npm install. Catches a
  future Promise.all-style refactor that would still pass final-state checks.
- Strengthen the SSH-channel-failure test: assert the rejection actually
  came from the probe call (not an earlier exec) by finding the probe
  invocation in mock.calls. Also assert NPTY-INSTALL-FAIL is NOT logged
  (channel failure must not be conflated with install failure) and that
  abandonInstall was called so the lock is released.
- Fix misleading clearAllMocks comment: it claims to wipe mockReturnValue,
  but actually clearAllMocks only resets .mock.calls. Re-priming was
  defense-in-depth, not a correctness requirement.

Validator:
- Add for-each-ref negative cases (--git-dir, --output, --work-tree) to the
  global-denied-flags it.each. The first round of for-each-ref enablement
  trusted that the post-subcommand GLOBAL_DENIED_FLAGS check applied; this
  pins it so a future allowlist refactor that bypasses the global check
  fails loudly.

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): split node-pty probe into test-d guard + load-test

Round-4 review fixes for the install probe in installNativeDeps:

(1) test -d guard runs before the load-test. If the install dir vanished
    between npm install and probe (concurrent rm, fs unmount, permission
    flip), the deploy now throws and the next reconnect retries fresh —
    previously the cd failure flowed into '|| echo MISSING' and we'd
    write .install-complete, stranding the user in degraded mode.

(2) Load-test discards stderr (2>/dev/null) so customized .bashrc
    output (NVM init, conda greetings, etc.) can't pollute the sentinel
    match. The shell-level '|| echo MISSING' is preserved so SSH-channel
    rejections still propagate as exec errors, distinct from require
    failures which exit the node process nonzero.

(3) PROBE_OK is passed via process.argv[1] so the JS literal stays
    trivial regardless of future sentinel characters.

Test changes:
- New 'dir-gone' probe mode in makeExecResponses
- New test pinning that vanished-dir throws (not silent MISSING)
- SSH-channel test now asserts probeCallIdx > npmInstallIdx
- cross-version-isolation feeds an extra '' for the test -d slot

Co-authored-by: Orca <help@stably.ai>

* fix(ssh): simplify node-pty probe and harden test ordering pins

Round-5 review fixes for installNativeDeps:

Production:
- Drop redundant test -d guard. `cd ${dir} && (...)` short-circuits on
  cd-failure (dir-vanished) and propagates as exec reject already; the
  separate guard added a round trip without preventing anything.
- Capture probe stderr to a per-deploy file rather than 2>&1 or 2>/dev/null.
  .bashrc noise can't pollute the sentinel match, but the require() error
  message is preserved in the [NPTY-MISSING] log breadcrumb so bug reports
  point at the real cause (e.g. GLIBC version mismatch).
- Mirror the install command's PATH (export PATH=${binDir}:$PATH) so any
  future require-time child_process call resolves the same node binary
  used during install.
- Add platform tuple to [NPTY-MISSING] and [NPTY-INSTALL-FAIL] logs for
  triageable bug reports without asking users to dig out their arch.
- Trim probe comment per AGENTS.md (why-only, no mechanism narration).

Tests:
- Pin full installNativeDeps ordering: npm install < chmod prebuilds <
  probe. Catches refactors that probe before install or move chmod after.
- Pressure-test .includes(PROBE_OK) survives bashrc/MOTD noise prefixed
  to probe stdout (corporate banner / NVM init / conda greeting case).
- Pressure-test MISSING detection survives Node deprecation warnings
  prepended to the MISSING token.
- Pin platform tuple appears in [NPTY-MISSING] log.
- Pin finalizeInstall called exactly once + abandonInstall not called
  on happy paths; reverse on failure paths.
- Strengthen dir-gone test: assert probeIdx > npmInstallIdx so a refactor
  that swaps order doesn't silently let the test pass on its own injected
  error string.
- New probeStdoutOverride option in makeExecResponses for shell-noise
  injection tests.

Cross-version-isolation: dropped obsolete test -d slot, added rm-stderr
cleanup slot to match the new probe shape.

eslint-disable max-lines on both files with rationale (pattern used widely
in this repo for cohesive single-responsibility modules).

441/441 tests pass; lint clean; typecheck clean. Probe shape verified
end-to-end on real remote.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-05-11 14:28:59 -07:00
Jinwoo Hong 42e04268fb fix: harden SSH connection reliability and add passphrase prompt (#636) 2026-04-14 18:29:50 -07:00
Jinwoo Hong cc66e120eb feat: Add SSH remote support (beta) (#590) 2026-04-13 19:23:09 -07:00