mirror of
https://github.com/stablyai/orca.git
synced 2026-10-27 08:02:51 +00:00
* fix(ssh): enable TCP_NODELAY on ssh2 client to eliminate per-keystroke typing lag (#1660) ssh2 leaves Nagle's algorithm on by default. For single-byte keystrokes through a remote PTY, Nagle interacts with the kernel's delayed-ACK timer and adds up to ~40 ms per keystroke — visible as the typing lag reported in #1660. OpenSSH's `ssh` client sets TCP_NODELAY whenever a PTY is allocated; this change mirrors that on the ssh2 client right after the `ready` event in doSsh2Connect, covering both initial connect and auto-reconnect. Proxy-command / proxy-jump connections (where ssh2's underlying socket is a custom Duplex over a child-process pipe) are a no-op by design, gated by the public Client.setNoDelay()'s own type guard. A discriminating log line records which path each connect took. Tests cover initial connect and a full reconnect cycle to guard against the regression class "Nagle is re-enabled because someone refactored only the initial connect path." Co-authored-by: Orca <help@stably.ai> * fix(ssh): bound relay-lost reconnect with exponential backoff When the relay exec channel keeps dying (e.g. a remote-side bug closes every fresh --connect channel right after handshake, or a stale bridge keeps being replaced), the unguarded _onRelayLost handler reconnects as fast as the network allows — spawning relay deploy attempts in a tight loop until the user force-quits. Each iteration spawns a fresh ssh2 exec channel, hammers sshd's MaxSessions counter, and floods the renderer with state churn. Add per-target exponential backoff (500ms → 15s, capped at 6 attempts) so the loop terminates instead of running forever. After the cap the session goes to 'error' state with a 'Relay channel kept dropping. Please reconnect.' message — visible in the renderer instead of an invisible failure where typing in remote terminals just stops working. Successful 'ready' resets the attempt counter only if the session stabilized for >= 5s; faster flaps preserve the counter so a flaky remote backs off rather than retrying indefinitely on every brief ready→lost cycle. Backoff state is cleared on explicit disconnect, on session replacement during reconnect, and on connect failures, so a real reconnect attempt after backoff exhaustion always starts from zero. Co-authored-by: Orca <help@stably.ai> * fix(ssh): detect stale relay daemons via running-version marker The on-disk relay version check compares local .version against the remote .version file in the relay dir. A daemon launched by an earlier deploy keeps running its in-memory copy of the OLD relay code, so when the client later rewrites relay.js + .version on disk and bridges in via --connect, the new bridge process drives a stale daemon. Protocol or behavior changes between the two versions then tear down the channel in a tight reconnect loop (observed against PR #1672 on a daemon predating that change). The daemon now writes its running version into a .running-version sidecar at startup, anchored to the relay-script directory rather than process.cwd() so test spawns cannot pollute the repo root. Before attaching to an existing socket, the client probes that marker and, on mismatch with the locally-deployed .version, kills the stale daemon (TERM only, never KILL) and falls through to a fresh launch. Conservative defaults: when either marker is unreadable, attach so older builds keep their live PTYs. Co-authored-by: Orca <help@stably.ai> * Revert "fix(ssh): detect stale relay daemons via running-version marker" This reverts commite58acf07c0. * fix(ssh): isolate relay versions via per-version install dirs and wire handshake The relay's previous single-dir layout (~/.orca-remote/relay-v0.1.0/) let the deploy step rewrite relay.js in place while a daemon was still loaded in memory at the previous version. New clients then drove that stale daemon, surfacing as a reconnect loop (issue #1660 follow-up) and the field failure observed against an 8-day-old daemon on openclaw. Switch to a VS Code-style versioned layout where each (RELAY_VERSION + content-hash) bundle installs into its own directory and is never mutated after install. A v2 client's --connect socket path is rooted in relay-${v2-hash}/ and structurally cannot reach a v1 daemon's socket. Defense-in-depth: the daemon now reads exactly one Handshake-typed frame on each newly-accepted Unix socket before attaching the JSON-RPC dispatcher (mirrors VS Code's remoteExtensionHostAgentServer.ts:340). Mismatch closes the socket; the bridge exits with code 42; client maps that to a typed RelayVersionMismatchError and skips the relay-lost backoff loop instead of retrying through 6 attempts. Other deploy hardening: - atomic mkdir-based install lock with stale-lock recovery serialises concurrent first-installs of the same version - .install-complete sentinel distinguishes a finished install from a crashed-mid-install partial that should be retried - gcOldRelayVersions removes unreferenced sibling dirs (allowlist regex, skips locked or incomplete dirs, skips dirs with a live socket) - readLocalFullVersion fails fast on a missing/empty local .version rather than silently falling back to a path where a daemon from a different code generation may already be running Includes a cross-version isolation test that fails any future refactor which collapses the per-version layout back to a shared dir. Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden relay versioning per review feedback Address must-fix and should-fix findings from the parallel triple review of26d1666e: - Surface RelayVersionMismatchError to ssh.ts on initial establish() (not just reconnect), so the user sees the typed terminal error instead of silent retry on first connect (#13). - Give the sentinel timeout a 500ms grace window for the close handler to deliver exit-42, so a slow remote does not misclassify a wire-handshake mismatch as a generic timeout (#D11). - Drain the handshake decoder's residue at the handshake -> dispatcher transition on both daemon and --connect sides; pipelined frames that were coalesced with the handshake are now forwarded into the dispatcher / stdout instead of silently dropped (#A1, #A2). - Reset the install-lock acquire timer after a stale-lock recovery so a single post-recovery race does not immediately exhaust the budget (#E14). - Treat a stale install-lock as recoverable in the GC pass when .install-complete is present (covers an interrupted finalize where the rm-lock failed) (#E15). - GC legacy relay-v\d+\.\d+\.\d+ install dirs whose daemons have died, now that .install-complete is no longer required for them (#12). - Resolve symlinks in readLaunchVersion() so a daemon launched via a symlinked entry script still reads .version next to the real file (#G21). - Flush stderr before exit-42 in --connect handshake mismatch path so the diagnostic line reaches the client before the process tears down (#C8). Tests: - Round-trip handshake over a real Socket pair: matching version, mismatch exit-42, leftover bytes preserved on both sides when frames are coalesced with the handshake. - waitForSentinel exit-42 -> RelayVersionMismatchError, exit-1 -> generic. - SshRelaySession terminal-error callback fires on both establish() and reconnect() when deployAndLaunchRelay throws RelayVersionMismatchError. - acquireInstallLock concurrent BUSY -> OK polling, stale-lock recovery with reset timeout window, and fresh-lock timeout failure path. - gcOldRelayVersions stale-lock-with-complete branch, legacy-dead path, legacy-alive path; existing locked-test asserts fresh-lock now keeps. - Cross-version isolation test now asserts a blanket invariant that every v1-referencing command from a v2 deploy is a read-only liveness probe. Lint and typecheck clean across all 3 tsconfigs; 426 SSH/relay tests pass. Co-authored-by: Orca <help@stably.ai> * fix(ssh): bypass npm init for content-hashed relay dirs and harden install probe The versioned-install dirs land at `relay-${version}+${hash}/` (e.g. `relay-0.1.0+07994a7870e1`). npm 11 / Node 26 reject the `+` in derived package names and `npm init -y` exits 1 — silently, since both stderr and the failure landed inside the `2>/dev/null && ...` chain. The catch swallowed the throw, `.install-complete` was written anyway, and every reconnect surfaced 'node-pty is not available' at first pty.spawn. Sidestep `npm init` entirely: SFTP-write a hardcoded minimal package.json (`name: orca-relay`, `type: commonjs`) and run `npm install node-pty` directly. `type: commonjs` pins the module system against future Node default flips or remote-side .npmrc overrides. Also harden the install path against the same class of silent failure: - npm install errors now propagate (no more `.install-complete` on hard fail; future reconnects retry instead of stranding the user) - Replace the weak `test -d node-pty` post-install probe with `node -e 'require("node-pty")'` so built-but-unloadable installs (missing prebuild, wrong arch, broken native binding) surface clearly - Add a session-level error handler on the SFTP write so a torn-down session rejects the promise instead of hanging until enclosing timeout Separate fix: add `for-each-ref` to the relay's git subcommand allowlist. Client code (`src/main/git/repo.ts` ref-search and worktree-listing) calls `git for-each-ref` over SSH; the relay was rejecting it. The `--shell`/`--python`/`--perl`/`--tcl` format flags only control output quoting (no eval) and the relay invokes git via execFileAsync (no shell), so the read-only allowlist treatment matches `rev-parse`, `log`, etc. Co-authored-by: Orca <help@stably.ai> * comment(ssh-relay): TODO link to #1693 for VS Code-style pre-bundled node-pty Co-authored-by: Orca <help@stably.ai> * fix(ssh): harden node-pty install probe and tighten review-fix tests Round-3 review fixes on top of963f56d7. deploy.ts: - Replace endsWith('OK') with includes('ORCA-NPTY-PROBE-OK'). Node can emit deprecation/experimental warnings to stderr after our stdout 'OK' write, and 2>&1 would push them past 'OK' producing false NPTY-MISSING warnings. A unique sentinel survives any trailing stderr noise. - Switch sftpPkg/ws .on -> .once for error/close. A late session 'error' after the promise had already settled would otherwise become an unhandled EventEmitter error and crash main. - Trim per-block comments to 1-2 lines per AGENTS.md (was 7-9). Tests: - Pin the BEFORE-ordering contract: SftpWriteCapture now records the count of execCommand calls observed at the moment ws.end() ran for each path, and the test asserts that count <= the index of npm install. Catches a future Promise.all-style refactor that would still pass final-state checks. - Strengthen the SSH-channel-failure test: assert the rejection actually came from the probe call (not an earlier exec) by finding the probe invocation in mock.calls. Also assert NPTY-INSTALL-FAIL is NOT logged (channel failure must not be conflated with install failure) and that abandonInstall was called so the lock is released. - Fix misleading clearAllMocks comment: it claims to wipe mockReturnValue, but actually clearAllMocks only resets .mock.calls. Re-priming was defense-in-depth, not a correctness requirement. Validator: - Add for-each-ref negative cases (--git-dir, --output, --work-tree) to the global-denied-flags it.each. The first round of for-each-ref enablement trusted that the post-subcommand GLOBAL_DENIED_FLAGS check applied; this pins it so a future allowlist refactor that bypasses the global check fails loudly. Co-authored-by: Orca <help@stably.ai> * fix(ssh): split node-pty probe into test-d guard + load-test Round-4 review fixes for the install probe in installNativeDeps: (1) test -d guard runs before the load-test. If the install dir vanished between npm install and probe (concurrent rm, fs unmount, permission flip), the deploy now throws and the next reconnect retries fresh — previously the cd failure flowed into '|| echo MISSING' and we'd write .install-complete, stranding the user in degraded mode. (2) Load-test discards stderr (2>/dev/null) so customized .bashrc output (NVM init, conda greetings, etc.) can't pollute the sentinel match. The shell-level '|| echo MISSING' is preserved so SSH-channel rejections still propagate as exec errors, distinct from require failures which exit the node process nonzero. (3) PROBE_OK is passed via process.argv[1] so the JS literal stays trivial regardless of future sentinel characters. Test changes: - New 'dir-gone' probe mode in makeExecResponses - New test pinning that vanished-dir throws (not silent MISSING) - SSH-channel test now asserts probeCallIdx > npmInstallIdx - cross-version-isolation feeds an extra '' for the test -d slot Co-authored-by: Orca <help@stably.ai> * fix(ssh): simplify node-pty probe and harden test ordering pins Round-5 review fixes for installNativeDeps: Production: - Drop redundant test -d guard. `cd ${dir} && (...)` short-circuits on cd-failure (dir-vanished) and propagates as exec reject already; the separate guard added a round trip without preventing anything. - Capture probe stderr to a per-deploy file rather than 2>&1 or 2>/dev/null. .bashrc noise can't pollute the sentinel match, but the require() error message is preserved in the [NPTY-MISSING] log breadcrumb so bug reports point at the real cause (e.g. GLIBC version mismatch). - Mirror the install command's PATH (export PATH=${binDir}:$PATH) so any future require-time child_process call resolves the same node binary used during install. - Add platform tuple to [NPTY-MISSING] and [NPTY-INSTALL-FAIL] logs for triageable bug reports without asking users to dig out their arch. - Trim probe comment per AGENTS.md (why-only, no mechanism narration). Tests: - Pin full installNativeDeps ordering: npm install < chmod prebuilds < probe. Catches refactors that probe before install or move chmod after. - Pressure-test .includes(PROBE_OK) survives bashrc/MOTD noise prefixed to probe stdout (corporate banner / NVM init / conda greeting case). - Pressure-test MISSING detection survives Node deprecation warnings prepended to the MISSING token. - Pin platform tuple appears in [NPTY-MISSING] log. - Pin finalizeInstall called exactly once + abandonInstall not called on happy paths; reverse on failure paths. - Strengthen dir-gone test: assert probeIdx > npmInstallIdx so a refactor that swaps order doesn't silently let the test pass on its own injected error string. - New probeStdoutOverride option in makeExecResponses for shell-noise injection tests. Cross-version-isolation: dropped obsolete test -d slot, added rm-stderr cleanup slot to match the new probe shape. eslint-disable max-lines on both files with rationale (pattern used widely in this repo for cohesive single-responsibility modules). 441/441 tests pass; lint clean; typecheck clean. Probe shape verified end-to-end on real remote. Co-authored-by: Orca <help@stably.ai> --------- Co-authored-by: Orca <help@stably.ai>
167 lines
5.9 KiB
TypeScript
167 lines
5.9 KiB
TypeScript
// Cross-version isolation guard.
|
|
//
|
|
// Why: this test is the executable form of the "Pattern Note" in
|
|
// docs/ssh-relay-versioned-install-dirs.md — it asserts that a v2 deploy
|
|
// targeting a remote where a v1 daemon is already running NEVER touches
|
|
// v1's install dir or socket. Without this test a future refactor that
|
|
// collapses to a shared dir passes every other unit test and re-introduces
|
|
// the original "stale daemon serves new client" bug.
|
|
|
|
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
|
|
|
vi.mock('electron', () => ({
|
|
app: { getAppPath: () => '/mock/app' }
|
|
}))
|
|
|
|
vi.mock('fs', () => ({
|
|
existsSync: vi.fn().mockReturnValue(true),
|
|
readFileSync: vi.fn().mockReturnValue('0.1.0+v2hash')
|
|
}))
|
|
|
|
vi.mock('./relay-protocol', () => ({
|
|
RELAY_VERSION: '0.1.0',
|
|
RELAY_REMOTE_DIR: '.orca-remote',
|
|
parseUnameToRelayPlatform: vi.fn().mockReturnValue('linux-x64'),
|
|
RELAY_SENTINEL: 'ORCA-RELAY v0.1.0 READY\n',
|
|
RELAY_SENTINEL_TIMEOUT_MS: 10_000
|
|
}))
|
|
|
|
vi.mock('./ssh-relay-deploy-helpers', () => ({
|
|
uploadDirectory: vi.fn().mockResolvedValue(undefined),
|
|
waitForSentinel: vi.fn().mockResolvedValue({
|
|
write: vi.fn(),
|
|
onData: vi.fn(),
|
|
onClose: vi.fn()
|
|
}),
|
|
execCommand: vi.fn(),
|
|
resolveRemoteNodePath: vi.fn().mockResolvedValue('/usr/bin/node')
|
|
}))
|
|
|
|
vi.mock('./ssh-connection-utils', () => ({
|
|
shellEscape: (s: string) => `'${s}'`
|
|
}))
|
|
|
|
import { deployAndLaunchRelay } from './ssh-relay-deploy'
|
|
import { execCommand } from './ssh-relay-deploy-helpers'
|
|
import type { SshConnection } from './ssh-connection'
|
|
|
|
function makeMockConnection(): SshConnection {
|
|
return {
|
|
exec: vi.fn().mockResolvedValue({
|
|
on: vi.fn(),
|
|
stderr: { on: vi.fn() },
|
|
stdin: {},
|
|
stdout: { on: vi.fn() },
|
|
close: vi.fn()
|
|
}),
|
|
sftp: vi.fn().mockResolvedValue({
|
|
mkdir: vi.fn((_p: string, cb: (err: Error | null) => void) => cb(null)),
|
|
on: vi.fn(),
|
|
once: vi.fn(),
|
|
createWriteStream: vi.fn().mockReturnValue({
|
|
on: vi.fn((_event: string, cb: () => void) => {
|
|
if (_event === 'close') {
|
|
setTimeout(cb, 0)
|
|
}
|
|
}),
|
|
once: vi.fn((_event: string, cb: () => void) => {
|
|
if (_event === 'close') {
|
|
setTimeout(cb, 0)
|
|
}
|
|
}),
|
|
end: vi.fn()
|
|
}),
|
|
end: vi.fn()
|
|
})
|
|
} as unknown as SshConnection
|
|
}
|
|
|
|
describe('cross-version isolation', () => {
|
|
beforeEach(() => {
|
|
vi.clearAllMocks()
|
|
})
|
|
|
|
it('a v2 deploy never references the v1 install dir or v1 socket path', async () => {
|
|
const conn = makeMockConnection()
|
|
const mockExec = vi.mocked(execCommand)
|
|
|
|
// Simulated remote where:
|
|
// v1 dir = ~/.orca-remote/relay-0.1.0+v1hash/ (live daemon, listening)
|
|
// v2 dir = ~/.orca-remote/relay-0.1.0+v2hash/ (does not yet exist)
|
|
// The v2 client has fullVersion='0.1.0+v2hash' (from the fs mock above).
|
|
//
|
|
// We feed enough exec results to walk through the deploy: platform,
|
|
// $HOME, isRelayAlreadyInstalled probe, lock acquire, upload (no exec),
|
|
// npm install, finalize, socket probe, socket poll, then GC scan.
|
|
const responses: string[] = [
|
|
'Linux x86_64', // uname -sm
|
|
'/home/u', // echo $HOME
|
|
'MISSING', // isRelayAlreadyInstalled (v2 dir doesn't exist)
|
|
'', // mkdir -p remoteRelayDir (v2)
|
|
'OK', // mkdir lock OK
|
|
'MISSING', // re-probe after lock → still missing → proceed with install
|
|
'', // mkdir remoteDir (uploadRelay)
|
|
'', // chmod +x node
|
|
'', // npm install
|
|
'', // chmod prebuilds
|
|
'ORCA-NPTY-PROBE-OK\n', // node -e require() load-test (post-install verify)
|
|
'', // rm -f probe-stderr (best-effort cleanup after probe resolved)
|
|
'', // touch .install-complete (finalizeInstall)
|
|
'', // rm -rf .install-lock
|
|
'DEAD', // launch socket probe
|
|
'READY', // socket poll
|
|
// GC scan begins here
|
|
'relay-0.1.0+v1hash\nrelay-0.1.0+v2hash\n', // ls listing
|
|
'OPEN', // v1 lock probe (siblings only — current dir is v2)
|
|
'COMPLETE', // v1 .install-complete probe
|
|
'ALIVE' // v1 socket probe → live → SKIP (don't GC v1)
|
|
]
|
|
for (const r of responses) {
|
|
mockExec.mockResolvedValueOnce(r)
|
|
}
|
|
|
|
await deployAndLaunchRelay(conn)
|
|
|
|
const allCmds = [
|
|
...mockExec.mock.calls.map(([, c]) => c),
|
|
...vi.mocked(conn.exec).mock.calls.map(([c]) => c as string)
|
|
]
|
|
|
|
// (a) the v2 deploy creates dirs/files under relay-0.1.0+v2hash
|
|
expect(allCmds.some((c) => c.includes('relay-0.1.0+v2hash'))).toBe(true)
|
|
|
|
// (b) the v2 launch and connect socket paths are rooted in v2 dir, never v1
|
|
const launchAndConnectCmds = vi
|
|
.mocked(conn.exec)
|
|
.mock.calls.map(([c]) => c as string)
|
|
.filter((c) => c.includes('--sock-path'))
|
|
expect(launchAndConnectCmds.length).toBeGreaterThan(0)
|
|
for (const cmd of launchAndConnectCmds) {
|
|
expect(cmd).toContain('relay-0.1.0+v2hash')
|
|
expect(cmd).not.toContain('relay-0.1.0+v1hash')
|
|
}
|
|
|
|
// (c) GC observes v1 has a live socket and never issues an rm -rf for it
|
|
const v1RemoveCmds = allCmds.filter(
|
|
(c) => c.includes('rm -rf') && c.includes('relay-0.1.0+v1hash')
|
|
)
|
|
expect(v1RemoveCmds).toHaveLength(0)
|
|
|
|
// (d) blanket isolation: every command that mentions v1hash MUST be a
|
|
// GC liveness probe (`ls`, `test -d`, `test -f`, or `for f in .../*.sock`)
|
|
// — never a write, mkdir, chmod, touch, rm, node launch, or socket poll.
|
|
// This prevents a future refactor that accidentally writes to the v1 dir
|
|
// (e.g. shared install-complete, upload over symlink) from passing.
|
|
const v1Refs = allCmds.filter((c) => c.includes('relay-0.1.0+v1hash'))
|
|
for (const cmd of v1Refs) {
|
|
const isReadOnlyProbe =
|
|
/^\s*ls\b/.test(cmd) ||
|
|
/\btest -d\b/.test(cmd) ||
|
|
/\btest -f\b/.test(cmd) ||
|
|
/\btest -S\b/.test(cmd) ||
|
|
/\bfor f in .*\.sock\b/.test(cmd)
|
|
expect(isReadOnlyProbe, `unexpected v1 reference: ${cmd}`).toBe(true)
|
|
}
|
|
})
|
|
})
|