Files
orca/src/main/ssh/ssh-connection-utils.ts
T
NeilandBrennan Benson fbe94ceff6 fix: close readiness gaps found by merged-change audit (#17159)
* fix(ssh): fence stale kills and retired pane replay

* fix(ssh): support cancellable interactive authentication

* fix(ssh): await remote catalog before snapshot adoption

* fix(pty): contain Windows ConPTY input failures

* fix(power): avoid redundant macOS display blocking

* perf(editor): narrow markdown override subscriptions

* fix(quick-open): close directory handles after reads

* refactor(linux): remove unused proc socket scanner

* fix(usage): apply flat Sonnet 4.6 pricing

* ci: prime Node next native test cache

* docs(skills): resolve snapshot cleanup data path

* fix(ssh): recover install locks after host reboot

* test(ssh): recognize boot-aware install locks

* test(ssh): prove previous-boot lock recovery live

* test(wire): pin pre-metadata release coverage

* fix(terminal): preserve remote tab ownership through recovery races

* test(runtime): fence replaced terminal handles in agent guard

* fix(ssh): preserve remote snapshot authority across polls

* fix(pty): contain late ConPTY output EPIPE

* test(pty): register Windows exit watcher before kill

* fix: close SSH and tab readiness race gaps

* fix(tabs): retain headless order and placeholder titles

* fix(build): avoid parallel electron-vite config race

* test(windows): avoid MSYS temp path rewriting

* test(windows): avoid killing exited PTY

* fix(pty): avoid late ConPTY input teardown race

* fix(terminal): sync reconnect error ownership after commit

* fix(runtime): use canonical worktree identity comparison

* test(ssh): assert complete cold-hydration baseline

* test(windows): invoke quoted retention fixture via PowerShell

* test(windows): read ConPTY grid through mode con

* fix(terminal): publish PTY replacements atomically

* fix(terminal): infer stale identity on reattach

* fix(terminal): fence stale pane PTY callbacks

* fix(terminal): fence stale pane binds after rebind

* fix(terminal): reject stale pane transport callbacks

* fix(terminal): fence mirrored reattach spawn callbacks

* fix(terminal): replace stale pane PTYs on remount

* fix(ci): size the Windows launcher-compile test budget from measurement

`native-smoke (windows-latest)` fails ~4.5% of runs on
`preserves a multiline argument through the compiled remote launcher`
with "Test timed out in 15000ms" — on unrelated PRs, for reasons that
have nothing to do with them. Across 176 sampled attempts it is the only
red that job produced, and it hit seven different PRs in two days:
#16900, #16904, #16915, #16955 (twice), #16979, #17014, #17085.

The test is six process creations: powershell.exe forks csc.exe, then
the freshly compiled orca.exe forks node.exe, twice. Hosted Windows
runners periodically slow process creation down, and this test amplifies
that far harder than anything else in the job. Comparing the 80 attempts
where it ran under 3s against the 12 where it ran over 12s, its own
median goes 2198ms -> 15917ms (7.2x) while the same file's
powershell-only test moves 556 -> 686ms (1.2x), the cmd.exe and Git Bash
process tests in the neighbouring file move 1.4x, and the other 35 files
put together move 1.5x.

Measured across those 176 attempts: 1881ms to 35438ms, p50 4264ms,
correlation +0.881 with the job's total Vitest duration. 8 of 176 (4.5%)
exceeded the 15s cap; 2 of 176 (1.1%) also exceeded the shared 30s
testTimeout, so deleting the override and inheriting the config is not
enough on its own. 60s clears all 176 with 1.7x headroom on the worst.

This is slow, not hung. Every body here is synchronous spawnSync, so
Vitest cannot interrupt one — the timer fires only after the body
returns and the reported duration is real elapsed time. That is why a
failure reads `× ... 22464ms` under `Test timed out in 15000ms`. The
work finished; the stopwatch was short. Seven reruns at one identical
head measured 2053 / 4680 / 5551 / 8732 / 13506 / 14868 / 21937ms — the
last of those would have been red on code that had not changed.

The 15s came from #8897, which raised this test off Vitest's built-in 5s
default because the job then ran bare `pnpm vitest run`. #8909 landed
3h27m later and pointed the job at config/vitest.config.ts, which is the
real fix for that. The constant stayed behind and has been the binding
budget ever since.

* fix(terminal): fence stale remount reattach ownership

* fix(terminal): reconcile mounted pane identity after replacement

* fix(terminal): fence stale reattach fallback ownership

* fix(terminal): fence deferred SSH reattach ownership

* fix(terminal): fence stale split pane ownership callbacks

* fix(terminal): keep stale spawns from consuming startup

---------

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
2026-08-31 08:17:40 -07:00

262 lines
8.8 KiB
TypeScript

import type { ConnectConfig } from 'ssh2'
import type { SshTarget, SshConnectionState } from '../../shared/ssh-types'
import type { SshResolvedConfig } from './ssh-config-parser'
import {
findEncryptedPrivateKeyPath,
resolveAgentConfigValue,
resolveAgentSocket,
resolvePrivateKeys,
resolveUnencryptedExplicitPrivateKeys
} from './ssh-auth-resolution'
import { configurePrivateKeyAuthentication } from './ssh-private-key-authentication'
import { isOpenSshConfigBackedTarget } from './system-ssh-args'
export { findDefaultKeyFile, resolveAgentSocket } from './ssh-auth-resolution'
export type SshCredentialKind = 'passphrase' | 'password' | 'keyboard-interactive'
export type SshConnectionCallbacks = {
onStateChange: (targetId: string, state: SshConnectionState) => void
onCredentialRequest?: (
targetId: string,
kind: SshCredentialKind,
detail: string,
signal?: AbortSignal
) => Promise<string | null>
}
export function isPassphraseError(err: Error): boolean {
const msg = err.message.toLowerCase()
return msg.includes('passphrase') || msg.includes('encrypted key') || msg.includes('bad decrypt')
}
export const INITIAL_RETRY_ATTEMPTS = 5
export const INITIAL_RETRY_DELAY_MS = 2000
export const RECONNECT_BACKOFF_MS = [1000, 2000, 5000, 5000, 10000, 10000, 10000, 30000, 30000]
export const CONNECT_TIMEOUT_MS = 30_000
export const SSH_CREDENTIAL_TIMEOUT_MS = 120_000
const TRANSIENT_ERROR_CODES = new Set([
'ETIMEDOUT',
'ECONNREFUSED',
'ECONNRESET',
'EHOSTUNREACH',
'ENETUNREACH',
'EAI_AGAIN'
])
function sshErrorLevel(err: Error): unknown {
return 'level' in err ? err.level : undefined
}
export function isAuthError(err: Error): boolean {
const msg = err.message.toLowerCase()
return (
msg.includes('all configured authentication methods failed') ||
msg.includes('authentication failed') ||
msg.includes('too many authentication failures') ||
/permission denied(?:, please try again\.?| \([^)]*(?:publickey|password|keyboard-interactive|gssapi|hostbased)[^)]*\))/.test(
msg
) ||
sshErrorLevel(err) === 'client-authentication'
)
}
export function isAgentFallbackError(err: Error): boolean {
return isAuthError(err) || sshErrorLevel(err) === 'agent'
}
export function isTransientError(err: Error): boolean {
if (
sshErrorLevel(err) === 'client-timeout' ||
err.message === 'Timed out while waiting for SSH authentication'
) {
return true
}
const code = 'code' in err && typeof err.code === 'string' ? err.code : undefined
if (code && TRANSIENT_ERROR_CODES.has(code)) {
return true
}
if (err.message.includes('ETIMEDOUT')) {
return true
}
if (err.message.includes('ECONNREFUSED')) {
return true
}
if (err.message.includes('ECONNRESET')) {
return true
}
return false
}
const SYSTEM_SSH_FALLBACK_ERROR_CODES = new Set(['EHOSTUNREACH', 'ENETUNREACH'])
export function isSystemSshFallbackError(err: Error): boolean {
const code = (err as NodeJS.ErrnoException).code
if (code && SYSTEM_SSH_FALLBACK_ERROR_CODES.has(code)) {
return true
}
return err.message.includes('EHOSTUNREACH') || err.message.includes('ENETUNREACH')
}
// Why: ssh2 has no gssapi-with-mic support. When the effective OpenSSH config
// enables GSSAPIAuthentication (often a distro-wide /etc/ssh default), a
// Kerberos ticket can still authenticate through the system ssh binary after
// key/agent auth fails — but only auth-shaped failures qualify, so network
// errors keep their existing retry semantics.
export function isGssapiSystemSshFallbackCandidate(
err: Error,
target: Pick<SshTarget, 'gssapiAuthentication'>,
resolved: Pick<SshResolvedConfig, 'gssapiAuthentication'> | null
): boolean {
// Why: targets with an explicit per-host flag already tried system ssh
// proactively during this attempt; probing again cannot succeed.
if (target.gssapiAuthentication === true) {
return false
}
return (isAuthError(err) || isPassphraseError(err)) && resolved?.gssapiAuthentication === true
}
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms))
}
export function shellEscape(s: string): string {
return `'${s.replace(/'/g, "'\\''")}'`
}
const REMOTE_COMMAND_CHUNK_MAX_BYTES = 1_024
const REMOTE_COMMAND_PRINTF_ESCAPED_BYTES = new Set([0x21, 0x27, 0x5c])
function encodeRemoteCommandForPrintf(command: string): string[] {
const chunks: string[] = []
let chunk = ''
let chunkBytes = 0
for (const character of command) {
const codePoint = character.codePointAt(0)!
const isSafePrintableAscii =
codePoint >= 0x20 && codePoint <= 0x7e && !REMOTE_COMMAND_PRINTF_ESCAPED_BYTES.has(codePoint)
const encodedCharacter =
codePoint > 0x7f || isSafePrintableAscii
? character
: `\\0${codePoint.toString(8).padStart(3, '0')}`
const encodedBytes = codePoint > 0x7f ? Buffer.byteLength(character) : encodedCharacter.length
if (chunkBytes + encodedBytes > REMOTE_COMMAND_CHUNK_MAX_BYTES) {
chunks.push(chunk)
chunk = ''
chunkBytes = 0
}
chunk += encodedCharacter
chunkBytes += encodedBytes
}
chunks.push(chunk)
return chunks
}
/** Wrap a POSIX snippet into one line that non-POSIX SSH login shells can forward. */
export function wrapRemoteCommandForPosixShell(command: string): string {
// Why: csh/tcsh split multiline SSH exec strings before /bin/sh sees them.
// POSIX printf rebuilds bounded argument chunks without consuming relay stdin.
const encodedChunks = encodeRemoteCommandForPrintf(command)
const decodeAndRun =
'decoded=$(printf %b "$@" && printf _) || exit $?; ' +
'decoded=${decoded%_}; exec /bin/sh -c "$decoded"'
const chunkArguments = encodedChunks.map(shellEscape).join(' ')
return `exec /bin/sh -c ${shellEscape(decodeAndRun)} orca-command ${chunkArguments}`
}
export type SshExecOptions = {
wrapCommand?: boolean
signal?: AbortSignal
}
export function createSshOperationAbortError(): Error & { name: string } {
const error = new Error('SSH operation was cancelled') as Error & { name: string }
error.name = 'AbortError'
return error
}
type BuildConnectConfigOptions = {
includeAgent?: boolean
includePrivateKey?: boolean
}
// Why: ssh2 tries privateKey before agent, but parses encrypted privateKey
// values before any agent auth can run. Keep unencrypted explicit keys first
// while deferring encrypted keys until the post-agent passphrase path.
export function buildConnectConfig(
target: SshTarget,
resolved: SshResolvedConfig | null,
options: BuildConnectConfigOptions = {}
): ConnectConfig {
const effectiveHost = resolveEffectiveHost(target, resolved)
const effectivePort = resolveEffectivePort(target, resolved)
const effectiveUser =
isOpenSshConfigBackedTarget(target) && resolved
? (resolved.user ?? target.username)
: target.username || resolved?.user || ''
const config: Record<string, unknown> = {
host: effectiveHost,
port: effectivePort,
username: effectiveUser,
readyTimeout: CONNECT_TIMEOUT_MS,
keepaliveInterval: 15_000,
tryKeyboard: true
}
const shouldIncludeAgent = options.includeAgent ?? true
const agentSocket = shouldIncludeAgent ? resolveAgentSocket(target, resolved) : undefined
const agent = agentSocket ? resolveAgentConfigValue(agentSocket, target, resolved) : undefined
if (agent) {
config.agent = agent
}
if (agent && resolved?.forwardAgent) {
config.agentForward = true
}
const keys =
(options.includePrivateKey ?? !agent)
? resolvePrivateKeys(target, resolved)
: resolveUnencryptedExplicitPrivateKeys(target, resolved)
configurePrivateKeyAuthentication(
config as ConnectConfig,
keys,
findEncryptedPrivateKeyPath(keys)
)
return config as ConnectConfig
}
function resolveEffectiveHost(target: SshTarget, resolved: SshResolvedConfig | null): string {
if (isOpenSshConfigBackedTarget(target) && resolved?.hostname) {
return resolved.hostname
}
if (shouldUseResolvedEndpoint(target, resolved)) {
return resolved!.hostname
}
return target.host || resolved?.hostname || target.label
}
function resolveEffectivePort(target: SshTarget, resolved: SshResolvedConfig | null): number {
if (isOpenSshConfigBackedTarget(target) && resolved) {
return resolved.port || target.port || 22
}
// Why: imported config aliases store 22 as the schema default even when an
// included/wildcard OpenSSH rule later resolves a different effective Port.
if (target.configHost && target.port === 22 && resolved?.port) {
return resolved.port
}
return target.port || resolved?.port || 22
}
function shouldUseResolvedEndpoint(target: SshTarget, resolved: SshResolvedConfig | null): boolean {
if (!target.configHost || !resolved?.hostname) {
return false
}
const host = target.host.trim()
return host === '' || host === target.configHost || host === target.label
}