mirror of
https://github.com/stablyai/orca.git
synced 2026-09-29 08:03:20 +00:00
* fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660) ssh2 leaves Nagle's algorithm on by default. For single-byte keystrokes through a remote PTY, Nagle interacts with the kernel's delayed-ACK timer and adds up to ~40 ms per keystroke — visible as the typing lag reported in #1660. OpenSSH's `ssh` client sets TCP_NODELAY whenever a PTY is allocated; this change mirrors that on the ssh2 client right after the `ready` event in doSsh2Connect, covering both initial connect and auto-reconnect. Proxy-command / proxy-jump connections (where ssh2's underlying socket is a custom Duplex over a child-process pipe) are a no-op by design, gated by the public Client.setNoDelay()'s own type guard. A discriminating log line records which path each connect took. Tests cover initial connect and a full reconnect cycle to guard against the regression class "Nagle is re-enabled because someone refactored only the initial connect path." Co-authored-by: Orca <help@stably.ai> * fix(ssh): bound relay-lost reconnect with exponential backoff When the relay exec channel keeps dying (e.g. a remote-side bug closes every fresh --connect channel right after handshake, or a stale bridge keeps being replaced), the unguarded _onRelayLost handler reconnects as fast as the network allows — spawning relay deploy attempts in a tight loop until the user force-quits. Each iteration spawns a fresh ssh2 exec channel, hammers sshd's MaxSessions counter, and floods the renderer with state churn. Add per-target exponential backoff (500ms → 15s, capped at 6 attempts) so the loop terminates instead of running forever. After the cap the session goes to 'error' state with a 'Relay channel kept dropping. Please reconnect.' message — visible in the renderer instead of an invisible failure where typing in remote terminals just stops working. Successful 'ready' resets the attempt counter only if the session stabilized for >= 5s; faster flaps preserve the counter so a flaky remote backs off rather than retrying indefinitely on every brief ready→lost cycle. Backoff state is cleared on explicit disconnect, on session replacement during reconnect, and on connect failures, so a real reconnect attempt after backoff exhaustion always starts from zero. Co-authored-by: Orca <help@stably.ai> * fix(ssh): detect stale relay daemons via running-version marker The on-disk relay version check compares local .version against the remote .version file in the relay dir. A daemon launched by an earlier deploy keeps running its in-memory copy of the OLD relay code, so when the client later rewrites relay.js + .version on disk and bridges in via --connect, the new bridge process drives a stale daemon. Protocol or behavior changes between the two versions then tear down the channel in a tight reconnect loop (observed against PR #1672 on a daemon predating that change). The daemon now writes its running version into a .running-version sidecar at startup, anchored to the relay-script directory rather than process.cwd() so test spawns cannot pollute the repo root. Before attaching to an existing socket, the client probes that marker and, on mismatch with the locally-deployed .version, kills the stale daemon (TERM only, never KILL) and falls through to a fresh launch. Conservative defaults: when either marker is unreadable, attach so older builds keep their live PTYs. Co-authored-by: Orca <help@stably.ai> * Revert "fix(ssh): detect stale relay daemons via running-version marker" This reverts commite58acf07c0. * fix(ssh): isolate relay versions via per-version install dirs and wire handshake The relay's previous single-dir layout (~/.orca-remote/relay-v0.1.0/) let the deploy step rewrite relay.js in place while a daemon was still loaded in memory at the previous version. New clients then drove that stale daemon, surfacing as a reconnect loop (issue #1660 follow-up) and the field failure observed against an 8-day-old daemon on openclaw. Switch to a VS Code-style versioned layout where each (RELAY_VERSION + content-hash) bundle installs into its own directory and is never mutated after install. A v2 client's --connect socket path is rooted in relay-${v2-hash}/ and structurally cannot reach a v1 daemon's socket. Defense-in-depth: the daemon now reads exactly one Handshake-typed frame on each newly-accepted Unix socket before attaching the JSON-RPC dispatcher (mirrors VS Code's remoteExtensionHostAgentServer.ts:340). Mismatch closes the socket; the bridge exits with code 42; client maps that to a typed RelayVersionMismatchError and skips the relay-lost backoff loop instead of retrying through 6 attempts. Other deploy hardening: - atomic mkdir-based install lock with stale-lock recovery serialises concurrent first-installs of the same version - .install-complete sentinel distinguishes a finished install from a crashed-mid-install partial that should be retried - gcOldRelayVersions removes unreferenced sibling dirs (allowlist regex, skips locked or incomplete dirs, skips dirs with a live socket) - readLocalFullVersion fails fast on a missing/empty local .version rather than silently falling back to a path where a daemon from a different code generation may already be running Includes a cross-version isolation test that fails any future refactor which collapses the per-version layout back to a shared dir. Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden relay versioning per review feedback Address must-fix and should-fix findings from the parallel triple review of26d1666e: - Surface RelayVersionMismatchError to ssh.ts on initial establish() (not just reconnect), so the user sees the typed terminal error instead of silent retry on first connect (#13). - Give the sentinel timeout a 500ms grace window for the close handler to deliver exit-42, so a slow remote does not misclassify a wire-handshake mismatch as a generic timeout (#D11). - Drain the handshake decoder's residue at the handshake -> dispatcher transition on both daemon and --connect sides; pipelined frames that were coalesced with the handshake are now forwarded into the dispatcher / stdout instead of silently dropped (#A1, #A2). - Reset the install-lock acquire timer after a stale-lock recovery so a single post-recovery race does not immediately exhaust the budget (#E14). - Treat a stale install-lock as recoverable in the GC pass when .install-complete is present (covers an interrupted finalize where the rm-lock failed) (#E15). - GC legacy relay-v\d+\.\d+\.\d+ install dirs whose daemons have died, now that .install-complete is no longer required for them (#12). - Resolve symlinks in readLaunchVersion() so a daemon launched via a symlinked entry script still reads .version next to the real file (#G21). - Flush stderr before exit-42 in --connect handshake mismatch path so the diagnostic line reaches the client before the process tears down (#C8). Tests: - Round-trip handshake over a real Socket pair: matching version, mismatch exit-42, leftover bytes preserved on both sides when frames are coalesced with the handshake. - waitForSentinel exit-42 -> RelayVersionMismatchError, exit-1 -> generic. - SshRelaySession terminal-error callback fires on both establish() and reconnect() when deployAndLaunchRelay throws RelayVersionMismatchError. - acquireInstallLock concurrent BUSY -> OK polling, stale-lock recovery with reset timeout window, and fresh-lock timeout failure path. - gcOldRelayVersions stale-lock-with-complete branch, legacy-dead path, legacy-alive path; existing locked-test asserts fresh-lock now keeps. - Cross-version isolation test now asserts a blanket invariant that every v1-referencing command from a v2 deploy is a read-only liveness probe. Lint and typecheck clean across all 3 tsconfigs; 426 SSH/relay tests pass. Co-authored-by: Orca <help@stably.ai> * fix(ssh): bypass npm init for content-hashed relay dirs and harden install probe The versioned-install dirs land at `relay-${version}+${hash}/` (e.g. `relay-0.1.0+07994a7870e1`). npm 11 / Node 26 reject the `+` in derived package names and `npm init -y` exits 1 — silently, since both stderr and the failure landed inside the `2>/dev/null && ...` chain. The catch swallowed the throw, `.install-complete` was written anyway, and every reconnect surfaced 'node-pty is not available' at first pty.spawn. Sidestep `npm init` entirely: SFTP-write a hardcoded minimal package.json (`name: orca-relay`, `type: commonjs`) and run `npm install node-pty` directly. `type: commonjs` pins the module system against future Node default flips or remote-side .npmrc overrides. Also harden the install path against the same class of silent failure: - npm install errors now propagate (no more `.install-complete` on hard fail; future reconnects retry instead of stranding the user) - Replace the weak `test -d node-pty` post-install probe with `node -e 'require("node-pty")'` so built-but-unloadable installs (missing prebuild, wrong arch, broken native binding) surface clearly - Add a session-level error handler on the SFTP write so a torn-down session rejects the promise instead of hanging until enclosing timeout Separate fix: add `for-each-ref` to the relay's git subcommand allowlist. Client code (`src/main/git/repo.ts` ref-search and worktree-listing) calls `git for-each-ref` over SSH; the relay was rejecting it. The `--shell`/`--python`/`--perl`/`--tcl` format flags only control output quoting (no eval) and the relay invokes git via execFileAsync (no shell), so the read-only allowlist treatment matches `rev-parse`, `log`, etc. Co-authored-by: Orca <help@stably.ai> * comment(ssh-relay): TODO link to #1693 for VS Code-style pre-bundled node-pty Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden node-pty install probe and tighten review-fix tests Round-3 review fixes on top of963f56d7. deploy.ts: - Replace endsWith('OK') with includes('ORCA-NPTY-PROBE-OK'). Node can emit deprecation/experimental warnings to stderr after our stdout 'OK' write, and 2>&1 would push them past 'OK' producing false NPTY-MISSING warnings. A unique sentinel survives any trailing stderr noise. - Switch sftpPkg/ws .on -> .once for error/close. A late session 'error' after the promise had already settled would otherwise become an unhandled EventEmitter error and crash main. - Trim per-block comments to 1-2 lines per AGENTS.md (was 7-9). Tests: - Pin the BEFORE-ordering contract: SftpWriteCapture now records the count of execCommand calls observed at the moment ws.end() ran for each path, and the test asserts that count <= the index of npm install. Catches a future Promise.all-style refactor that would still pass final-state checks. - Strengthen the SSH-channel-failure test: assert the rejection actually came from the probe call (not an earlier exec) by finding the probe invocation in mock.calls. Also assert NPTY-INSTALL-FAIL is NOT logged (channel failure must not be conflated with install failure) and that abandonInstall was called so the lock is released. - Fix misleading clearAllMocks comment: it claims to wipe mockReturnValue, but actually clearAllMocks only resets .mock.calls. Re-priming was defense-in-depth, not a correctness requirement. Validator: - Add for-each-ref negative cases (--git-dir, --output, --work-tree) to the global-denied-flags it.each. The first round of for-each-ref enablement trusted that the post-subcommand GLOBAL_DENIED_FLAGS check applied; this pins it so a future allowlist refactor that bypasses the global check fails loudly. Co-authored-by: Orca <help@stably.ai> * fix(ssh): split node-pty probe into test-d guard + load-test Round-4 review fixes for the install probe in installNativeDeps: (1) test -d guard runs before the load-test. If the install dir vanished between npm install and probe (concurrent rm, fs unmount, permission flip), the deploy now throws and the next reconnect retries fresh — previously the cd failure flowed into '|| echo MISSING' and we'd write .install-complete, stranding the user in degraded mode. (2) Load-test discards stderr (2>/dev/null) so customized .bashrc output (NVM init, conda greetings, etc.) can't pollute the sentinel match. The shell-level '|| echo MISSING' is preserved so SSH-channel rejections still propagate as exec errors, distinct from require failures which exit the node process nonzero. (3) PROBE_OK is passed via process.argv[1] so the JS literal stays trivial regardless of future sentinel characters. Test changes: - New 'dir-gone' probe mode in makeExecResponses - New test pinning that vanished-dir throws (not silent MISSING) - SSH-channel test now asserts probeCallIdx > npmInstallIdx - cross-version-isolation feeds an extra '' for the test -d slot Co-authored-by: Orca <help@stably.ai> * fix(ssh): simplify node-pty probe and harden test ordering pins Round-5 review fixes for installNativeDeps: Production: - Drop redundant test -d guard. `cd ${dir} && (...)` short-circuits on cd-failure (dir-vanished) and propagates as exec reject already; the separate guard added a round trip without preventing anything. - Capture probe stderr to a per-deploy file rather than 2>&1 or 2>/dev/null. .bashrc noise can't pollute the sentinel match, but the require() error message is preserved in the [NPTY-MISSING] log breadcrumb so bug reports point at the real cause (e.g. GLIBC version mismatch). - Mirror the install command's PATH (export PATH=${binDir}:$PATH) so any future require-time child_process call resolves the same node binary used during install. - Add platform tuple to [NPTY-MISSING] and [NPTY-INSTALL-FAIL] logs for triageable bug reports without asking users to dig out their arch. - Trim probe comment per AGENTS.md (why-only, no mechanism narration). Tests: - Pin full installNativeDeps ordering: npm install < chmod prebuilds < probe. Catches refactors that probe before install or move chmod after. - Pressure-test .includes(PROBE_OK) survives bashrc/MOTD noise prefixed to probe stdout (corporate banner / NVM init / conda greeting case). - Pressure-test MISSING detection survives Node deprecation warnings prepended to the MISSING token. - Pin platform tuple appears in [NPTY-MISSING] log. - Pin finalizeInstall called exactly once + abandonInstall not called on happy paths; reverse on failure paths. - Strengthen dir-gone test: assert probeIdx > npmInstallIdx so a refactor that swaps order doesn't silently let the test pass on its own injected error string. - New probeStdoutOverride option in makeExecResponses for shell-noise injection tests. Cross-version-isolation: dropped obsolete test -d slot, added rm-stderr cleanup slot to match the new probe shape. eslint-disable max-lines on both files with rationale (pattern used widely in this repo for cohesive single-responsibility modules). 441/441 tests pass; lint clean; typecheck clean. Probe shape verified end-to-end on real remote. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai>
260 lines
9.2 KiB
TypeScript
260 lines
9.2 KiB
TypeScript
// Wire-level handshake helpers for the Orca relay.
|
|
//
|
|
// Why this lives in its own module: oxlint enforces a 300-line limit (with
|
|
// blanks/comments stripped) on .ts files, and relay.ts already runs near that
|
|
// limit. Splitting the version-handshake plumbing into a sibling module keeps
|
|
// relay.ts focused on the daemon-lifecycle wiring and makes the handshake
|
|
// independently unit-testable.
|
|
|
|
import { dirname, join } from 'path'
|
|
import { existsSync, readFileSync, realpathSync } from 'fs'
|
|
import type { Socket } from 'net'
|
|
import {
|
|
RELAY_VERSION,
|
|
MessageType,
|
|
FrameDecoder,
|
|
encodeHandshakeFrame,
|
|
parseHandshakeMessage,
|
|
type DecodedFrame
|
|
} from './protocol'
|
|
|
|
// Why: a unique exit code reserved for the wire-level version-mismatch terminal
|
|
// condition. The client (waitForSentinel + ssh.ts) maps this exit code to a
|
|
// non-retryable RelayVersionMismatchError so _onRelayLost skips the backoff
|
|
// loop. Any other non-zero exit is treated as a transient transport error.
|
|
export const EXIT_CODE_VERSION_MISMATCH = 42
|
|
|
|
// Why: the deploy step writes a content-hashed version marker (e.g.
|
|
// "0.1.0+0a5fe134d020") into ${remoteDir}/.version next to relay.js. Read it
|
|
// from the directory the running script lives in (NOT process.cwd()) so test
|
|
// spawns from arbitrary working dirs still report a coherent version. We
|
|
// resolve symlinks via realpathSync so a daemon launched indirectly (e.g.
|
|
// `node /tmp/symlink-to-relay.js`) still finds .version next to the real
|
|
// script. Falls back to bare RELAY_VERSION only if the file truly cannot be
|
|
// read; the wire handshake then refuses a fresh content-hashed client and
|
|
// the user gets a clean typed error rather than a silent stale-daemon loop.
|
|
export function readLaunchVersion(): string {
|
|
try {
|
|
const entry = process.argv[1]
|
|
let dir: string
|
|
if (entry) {
|
|
let resolved = entry
|
|
try {
|
|
resolved = realpathSync(entry)
|
|
} catch {
|
|
/* fall back to the unresolved path */
|
|
}
|
|
dir = dirname(resolved)
|
|
} else {
|
|
dir = process.cwd()
|
|
}
|
|
const versionFile = join(dir, '.version')
|
|
if (existsSync(versionFile)) {
|
|
const v = readFileSync(versionFile, 'utf-8').trim()
|
|
if (v) {
|
|
return v
|
|
}
|
|
}
|
|
} catch {
|
|
/* fall through */
|
|
}
|
|
return RELAY_VERSION
|
|
}
|
|
|
|
// ── Daemon side ─────────────────────────────────────────────────────
|
|
|
|
export type DaemonHandshakeCallbacks = {
|
|
// Why: leftover is any bytes the FrameDecoder buffered AFTER the handshake
|
|
// frame (e.g. the bridge wrote handshake + a JSON-RPC frame in the same
|
|
// TCP send). The caller MUST feed leftover into the dispatcher before
|
|
// attaching the new 'data' listener, otherwise those bytes are silently
|
|
// lost.
|
|
onAccepted: (sock: Socket, leftover: Buffer) => void
|
|
launchVersion: string
|
|
}
|
|
|
|
// Why: pre-dispatcher version handshake. The daemon reads exactly one
|
|
// Handshake-typed frame off this freshly-accepted socket BEFORE the JSON-RPC
|
|
// dispatcher pipe is attached. Mismatch means the connecting bridge was
|
|
// launched against a different relay.js version than the daemon was; we close
|
|
// the socket so the bridge exits 42 and the client surfaces a typed error
|
|
// instead of looping over the dispatcher.
|
|
export function setupDaemonHandshake(sock: Socket, cb: DaemonHandshakeCallbacks): void {
|
|
let handshakeResolved = false
|
|
const decoder: FrameDecoder = new FrameDecoder(
|
|
(frame: DecodedFrame) => {
|
|
if (handshakeResolved) {
|
|
return
|
|
}
|
|
const accepted = handleDaemonHandshakeFrame(sock, frame, cb.launchVersion)
|
|
if (accepted) {
|
|
handshakeResolved = true
|
|
const leftover = decoder.drain()
|
|
detachHandshakeListener(sock)
|
|
cb.onAccepted(sock, leftover)
|
|
}
|
|
},
|
|
(err) => {
|
|
process.stderr.write(`[relay] Handshake decode error: ${err.message}\n`)
|
|
sock.destroy()
|
|
}
|
|
)
|
|
|
|
const onHandshakeData = (chunk: Buffer): void => {
|
|
decoder.feed(chunk)
|
|
}
|
|
sock.on('data', onHandshakeData)
|
|
;(sock as Socket & { __orcaOnHandshake?: typeof onHandshakeData }).__orcaOnHandshake =
|
|
onHandshakeData
|
|
}
|
|
|
|
export function detachHandshakeListener(sock: Socket): void {
|
|
const tagged = sock as Socket & { __orcaOnHandshake?: (chunk: Buffer) => void }
|
|
if (tagged.__orcaOnHandshake) {
|
|
sock.removeListener('data', tagged.__orcaOnHandshake)
|
|
delete tagged.__orcaOnHandshake
|
|
}
|
|
}
|
|
|
|
function handleDaemonHandshakeFrame(
|
|
sock: Socket,
|
|
frame: DecodedFrame,
|
|
launchVersion: string
|
|
): boolean {
|
|
if (frame.type !== MessageType.Handshake) {
|
|
process.stderr.write(
|
|
`[relay] Protocol violation pre-handshake: type=${frame.type}; closing socket\n`
|
|
)
|
|
sock.destroy()
|
|
return false
|
|
}
|
|
let msg: ReturnType<typeof parseHandshakeMessage>
|
|
try {
|
|
msg = parseHandshakeMessage(frame.payload)
|
|
} catch (err) {
|
|
process.stderr.write(
|
|
`[relay] Could not parse handshake: ${(err as Error).message}; closing socket\n`
|
|
)
|
|
sock.destroy()
|
|
return false
|
|
}
|
|
if (msg.type !== 'orca-relay-handshake') {
|
|
process.stderr.write(
|
|
`[relay] Unexpected handshake type from client: ${msg.type}; closing socket\n`
|
|
)
|
|
sock.destroy()
|
|
return false
|
|
}
|
|
if (msg.version !== launchVersion) {
|
|
process.stderr.write(
|
|
`[relay] Handshake mismatch: own=${launchVersion}, client=${msg.version}; closing socket\n`
|
|
)
|
|
try {
|
|
sock.write(
|
|
encodeHandshakeFrame({
|
|
type: 'orca-relay-handshake-mismatch',
|
|
expected: launchVersion,
|
|
got: msg.version
|
|
})
|
|
)
|
|
} catch {
|
|
/* best-effort — close+exit-42 still wins */
|
|
}
|
|
sock.end()
|
|
return false
|
|
}
|
|
process.stderr.write(`[relay] Handshake OK from version=${msg.version}\n`)
|
|
sock.write(encodeHandshakeFrame({ type: 'orca-relay-handshake-ok', version: launchVersion }))
|
|
return true
|
|
}
|
|
|
|
// ── --connect side ──────────────────────────────────────────────────
|
|
|
|
export type ConnectHandshakeCallbacks = {
|
|
// Why: leftover is any bytes the FrameDecoder buffered AFTER the
|
|
// handshake-ok frame. The caller MUST forward leftover to process.stdout
|
|
// (the SSH stdout pipe) before attaching the raw bridge, otherwise daemon
|
|
// bytes coalesced into the same TCP send as handshake-ok are silently
|
|
// dropped.
|
|
onAccepted: (leftover: Buffer) => void
|
|
}
|
|
|
|
// Why: the wire-level version handshake from the bridge side. Before we attach
|
|
// the bidirectional pipe (and before we write RELAY_SENTINEL to stdout to
|
|
// unblock the client), we send a Handshake-typed frame carrying our version
|
|
// and wait for the daemon's Handshake response. This is defense-in-depth on
|
|
// top of the versioned-install-dir layout: a corrupt/missing .version, hash
|
|
// collision, or legacy-fallback path would otherwise let a v2 bridge drive a
|
|
// v1 daemon. VS Code's remoteExtensionHostAgentServer.ts:340 does the same
|
|
// check.
|
|
export function runConnectHandshake(
|
|
sock: Socket,
|
|
myVersion: string,
|
|
cb: ConnectHandshakeCallbacks
|
|
): void {
|
|
let handshakeDone = false
|
|
|
|
const decoder: FrameDecoder = new FrameDecoder(
|
|
(frame: DecodedFrame) => {
|
|
if (handshakeDone) {
|
|
return
|
|
}
|
|
if (frame.type !== MessageType.Handshake) {
|
|
process.stderr.write(
|
|
`[relay-connect] Protocol violation: expected Handshake frame, got type=${frame.type}\n`
|
|
)
|
|
sock.destroy()
|
|
process.exit(1)
|
|
}
|
|
let msg: ReturnType<typeof parseHandshakeMessage>
|
|
try {
|
|
msg = parseHandshakeMessage(frame.payload)
|
|
} catch (err) {
|
|
process.stderr.write(
|
|
`[relay-connect] Could not parse handshake reply: ${(err as Error).message}\n`
|
|
)
|
|
sock.destroy()
|
|
process.exit(1)
|
|
}
|
|
if (msg.type === 'orca-relay-handshake-ok') {
|
|
process.stderr.write(`[relay-connect] Handshake OK at version=${msg.version}\n`)
|
|
handshakeDone = true
|
|
const leftover = decoder.drain()
|
|
sock.removeAllListeners('data')
|
|
cb.onAccepted(leftover)
|
|
return
|
|
}
|
|
if (msg.type === 'orca-relay-handshake-mismatch') {
|
|
// Why: explicit stderr flush + exit so the diagnostic line is
|
|
// delivered to the client BEFORE the process exits. Without this,
|
|
// process.stderr writes can be buffered/async on pipe transports
|
|
// and parseHandshakeMismatchStderr loses the version detail.
|
|
process.stderr.write(
|
|
`[relay-connect] Handshake mismatch: expected=${msg.expected}, daemon=${msg.got}; exiting ${EXIT_CODE_VERSION_MISMATCH}\n`,
|
|
() => {
|
|
sock.destroy()
|
|
process.exit(EXIT_CODE_VERSION_MISMATCH)
|
|
}
|
|
)
|
|
return
|
|
}
|
|
process.stderr.write(`[relay-connect] Unexpected handshake type: ${msg.type}\n`)
|
|
sock.destroy()
|
|
process.exit(1)
|
|
},
|
|
(err) => {
|
|
process.stderr.write(`[relay-connect] Handshake decode error: ${err.message}\n`)
|
|
sock.destroy()
|
|
process.exit(1)
|
|
}
|
|
)
|
|
|
|
sock.on('data', (chunk: Buffer) => {
|
|
if (!handshakeDone) {
|
|
decoder.feed(chunk)
|
|
}
|
|
})
|
|
|
|
sock.write(encodeHandshakeFrame({ type: 'orca-relay-handshake', version: myVersion }))
|
|
}
|