Files
orca/src/main/ssh/ssh-relay-session.ts
T
JinjingandOrca a07427e970 fix(ssh, relay): keep remote sessions alive through reconnects and backpressure (#11999)
* fix(ssh,relay): stop remote connections from being killed by backoff and frame caps

Three independent connection killers found in the SSH/remote freeze audit.

FINDING A - the reconnect ladder never escalated for post-handshake drops.
scheduleReconnect() used the single published state.reconnectAttempt for both
the delay index and the give-up test, and runReconnectAttempt() zeroed it
before connecting (ssh.ts gates the relay redeploy on 0-at-connected). Every
post-handshake drop therefore re-entered at 1000ms forever, ~3600 relay
redeploys/hour, and 'reconnection-failed' was unreachable for a flapping host.
New SshReconnectLadder splits the delay index (advanced by every retry) from
the failure streak (advanced only by a failed handshake), so flaps back off
while give-up semantics stay byte-identical to shipped.

FINDING B - notify() closed the client whenever a frame exceeded the producer
frame capacity, conflating a permanently un-sendable frame with transient
backpressure. A 5000-event fs.changed is 425KB against a 49KB cap, so the
watcher flood killed the link and re-killed on every reattach+replay. notify()
now drops and logs once per generation; fs.changed is chunked to each sink's
capacity with a control-lane overflow marker as the resync fallback; agent-hook
envelopes shed lastAssistantMessage/interactivePrompt/subagents to fit.

FINDING B2 - sendResponse routed >1MB responses to a lane whose admission
ignores the frame cap and closed the client on rejection, so a large
fs.listFiles dropped the SSH host. It now substitutes a JSON-RPC error so the
request fails instead of the connection.

Also moves fs.streamEnd/fs.streamError to the control lane so a terminal frame
cannot be dropped by the producer-lane check.

Co-authored-by: Orca <help@stably.ai>

* fix(relay): stop the overflow marker from re-killing the link it protects

Round-1 review fixes on the P0 freeze work.

The control-lane overflow marker could reinstate the exact failure this P0
removes: dispatcher-client-writer closes the client when control-lane
admission fails, and admitControl is the only lane that returns an error, so
one marker per failing batch accumulated to the 256-frame/1MB bound and
dropped the link. Markers are now deduped to one outstanding per
(client, root), cleared on settle.

Chunking also defeated the renderer's per-payload directory dedupe -- events
are now stable-grouped by parent directory so one directory lands in one
chunk -- and the halving walk overshot the byte minimum ~1.7x while the fast
path paid three JSON encodes; both are fixed by publishing first and sizing
from a measured bytes-per-event estimate.

Agent-hook shedding now surrenders the blocking interactive prompt LAST
rather than first, so a degraded envelope cannot strand a pane at
state=waiting with no answerable question card.

The dropped-notification log now distinguishes over-capacity from producer
queue backpressure and no longer lets the first dropped method silence every
other producer for the life of the connection.

* fix(relay,ssh): keep status delivery and terminal frames from trading one freeze for another

Round-2 review fixes.

The round-0 change from close-on-rejection to silent drop removed the only
redelivery path for agent.hook envelopes: they are fire-and-forget and the
per-pane cache only replays on handler install, so a saturated link stranded
a pane on a stale Working spinner until reconnect. Closing used to guarantee
delivery by forcing that replay. Envelopes now publish per client and pend
for bounded latest-wins redelivery when the producer queue rejects them.

Shed fields are now named on the wire. The subagent roster is not cosmetic --
the renderer replaces rather than merges it, and hibernation gates on its
length -- so an unmarked shed could sleep a live pane.

fs.streamEnd rode the control lane because it must not be dropped, but that
lane kills rather than drops. The stream's concurrency slot is now held until
the terminal frame settles rather than until the fd closes, capping queued
terminal frames well under the control budget; overflow costs one refused
read instead of the connection.

The watcher chunk walk now stops while producer retention sits past its
reserve and degrades to a resync, so a 5000-event flood cannot fill the queue
that interactive PTY traffic shares and stall every remote terminal.

The reconnect ladder caps its flap-path delay so delay plus handshake timeout
cannot cross the relay grace floor and let the remote daemon kill live PTYs.

Also: the suppression key no longer embeds a NUL byte, which had made the
file binary to git and grep; producerEnvelopeBudget no longer reports
infinite capacity for a departed client; the drop logger no longer encodes a
frame it will not log; and an over-capacity response substitution no longer
settles as if the result had been delivered.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure per client

Round 3 + 4 review fixes.

Watcher chunking is now gated on the *client's* retention reserve rather than
the dispatcher-wide one, so one stalled peer no longer forces a healthy client
into a full file-tree resync. The relay-lost redeploy ladder no longer burns its
6-attempt budget while the SSH transport itself is down: it holds at the 15s step
with a non-terminal status and rearms, so a laptop that slept past the ladder
comes back instead of landing on a terminal "give up" banner.

The shedFields wire marker had no consumer, so an agent-hook envelope whose
subagent roster was dropped to fit the frame read as "roster cleared" on the Orca
side: live child rows blanked and a done pane became hibernation-eligible while
its teammates were still running. ingestRemote now restores shed fields from the
cached payload (interactivePrompt deliberately excluded — a stale answerable
question card is worse than none).

Also: stream terminal-frame slots are counted per client, since the control queue
they protect is per client; the chunking fast path no longer logs a drop for a
batch it goes on to deliver in full; -32010 is now RelayErrorCode.ResponseOverCapacity.

Test debt from the review: pending-pane eviction, per-client stream isolation, and
the reconnect budget are now asserted rather than assumed; four fragile exact-byte
pins dropped in favour of the tier comparisons that carry the requirement.

* fix(relay,ssh): restore relay-shed status fields and scope backpressure

- Oversized relay responses now fail their request instead of closing the connection,
  preventing one frame from killing every pane on the host
- Restore subagent state for correct hibernation; don't resurrect stale prose
  across turns
- Account for relay re-establishment and PTY reattach time in SSH flap delay caps
- Only log drops of final unsendable envelopes, not temporary rejections during
  measurement probes
- Fix watcher overflow marker release race when notification admission rejects
  without settlement; use precise byte counting for event batching

* Restore relay-shed fields with digest validation and scoped backpressure

Validate that shed subagent rosters match their wire digest and turn identity before
restoration, preventing stale roster resurrection. Compact interactive prompts for waiting
states instead of dropping them. Demote control-queue overflow to non-fatal rejection so
clients can retry on capacity recovery, keeping the link alive during transient backpressure.

* fix(relay): correct ResponseOverCapacity error code

ResponseOverCapacity should use -33008 to stay in the -33xxx range
for relay protocol errors, not -32010.

* fix(relay): close client when pty.replay overflows control queue

Replay is never retried, so it uses the control lane where overflow
is fatal — the writer closes the client and reconnect reloads history
rather than stranding a short buffer.

* fix(relay): prevent infinite redeploy on flapping SSH transports

Charge reconnect attempts when connection restores mid-backoff, preventing
infinite loop on transports that flap between states. Refactor control overflow
handling to use entry property instead of WeakSet marker for clarity.

---------

Co-authored-by: Orca <help@stably.ai>
2026-08-01 14:27:16 -07:00

2438 lines
88 KiB
TypeScript

/* oxlint-disable max-lines */
// Why: single authority for all relay lifecycle state per SSH target (previously scattered across module Maps/Sets with duplicated paths).
import { randomUUID } from 'node:crypto'
import type { BrowserWindow } from 'electron'
import { deployAndLaunchRelay } from './ssh-relay-deploy'
import { execCommand } from './ssh-relay-deploy-helpers'
import { isRelayVersionMismatchError } from './ssh-relay-version-mismatch-error'
import type { RelayVersionMismatchError } from './ssh-relay-version-mismatch-error'
import { SshChannelMultiplexer } from './ssh-channel-multiplexer'
import { SshPtyProvider } from '../providers/ssh-pty-provider'
import type { SshPtyAttachResult } from '../providers/ssh-pty-session-reattach'
import type { SshPtyDataCallback, SshPtyExitCallback } from '../providers/ssh-pty-provider-contract'
import type { SshPtyRecoveryActivationLease } from '../providers/ssh-pty-notification-routing'
import { isSshPtyIdentityMismatchError, isSshPtyNotFoundError } from '../providers/ssh-pty-errors'
import { toAppSshPtyId, toRelaySshPtyId } from '../providers/ssh-pty-id'
import { SshFilesystemProvider } from '../providers/ssh-filesystem-provider'
import { SshGitProvider } from '../providers/ssh-git-provider'
import { agentHookServer } from '../agent-hooks/server'
import { isAgentStatusHooksEnabled } from '../agent-hooks/managed-agent-hook-controls'
import {
buildManagedHookDetectionCommands,
detectedManagedHookAgents
} from '../agent-hooks/managed-hook-detection-commands'
import {
AGENT_HOOK_INSTALL_MANAGED_HOOKS_METHOD,
AGENT_HOOK_INSTALL_PLUGINS_METHOD,
AGENT_HOOK_NOTIFICATION_METHOD,
AGENT_HOOK_REQUEST_REPLAY_METHOD,
isRemoteAgentHooksEnabled
} from '../../shared/agent-hook-relay'
import { _internals as openCodeInternals } from '../opencode/hook-service'
import { getPiAgentStatusExtensionSource } from '../pi/agent-status-extension-source'
import {
registerSshPtyProvider,
unregisterSshPtyProvider,
getSshPtyProvider,
getPtyIdsForConnection,
clearPtyOwnershipForConnection,
clearProviderPtyState,
deletePtyOwnership,
setPtyOwnership,
restorePtyIncarnation,
isCurrentPtyExit
} from '../ipc/pty'
import {
acceptSshPtyOutputData,
acceptSshPtyOutputExit,
allocateSshPtyProviderGeneration,
applySshPtySourceCancellationProof,
applySshPtySourceRecoveryCancellationProof,
beginSshPtyOutputGenerationMigration,
closeSshPtyOutputGeneration,
getSshPtyAcceptedSourceCheckpoints,
installSshPtySourceAckPublisher,
installSshPtySourceCancellationPublisher
} from '../ipc/ssh-pty-output-intake-registry'
import {
registerSshFilesystemProvider,
unregisterSshFilesystemProvider,
getSshFilesystemProvider
} from '../providers/ssh-filesystem-dispatch'
import { registerSshGitProvider, unregisterSshGitProvider } from '../providers/ssh-git-dispatch'
import { notifyRemoteWorkspaceHandlers } from '../ipc/remote-workspace-events'
import { PortScanner } from './ssh-port-scanner'
import { isMainWindowVisible, onMainWindowBecameVisible } from '../window/main-window-visibility'
import type { SshPortForwardManager } from './ssh-port-forward'
import type { SshConnection } from './ssh-connection'
import { joinRemotePath, isWindowsRemoteHost, type RemoteHostPlatform } from './ssh-remote-platform'
import { makeRemoteDirectoryCommand } from './ssh-remote-commands'
import { createRemoteCliInstallPlan } from './ssh-remote-cli-launcher'
import {
DEFAULT_SSH_RELAY_GRACE_PERIOD_SECONDS,
type DetectedPort,
MAX_SSH_RELAY_GRACE_PERIOD_SECONDS,
MIN_SSH_RELAY_GRACE_PERIOD_SECONDS,
SSH_RELAY_CONFIGURE_GRACE_TIME_METHOD
} from '../../shared/ssh-types'
import type { Store } from '../persistence'
import type { OrcaRuntimeService } from '../runtime/orca-runtime'
import { DEFAULT_PTY_SOURCE_WINDOW_SU } from '../../shared/pty-source-credit-contract'
import { PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR } from '../../shared/pty-consumer-session'
import { runRemoteOrcaCli } from './ssh-remote-orca-cli'
import {
acknowledgeRemoteOrcaCliPostOutput,
parseRemoteOrcaCliPostOutput
} from './ssh-remote-orchestration-post-output'
import { toSshExecutionHostId, type ExecutionHostId } from '../../shared/execution-host'
import { isTerminalLeafId, makePaneKey } from '../../shared/stable-pane-id'
import { isValidTerminalTabId } from '../../shared/terminal-tab-id'
import {
openSshPtyConsumerSession,
type SshPtyConsumerOwnerState,
type SshPtyConsumerSessionState
} from './ssh-pty-consumer-session'
import type {
PtySourceRecoveryComplete,
PtySourceRecoveryPending,
PtySourceRecoveryRequest
} from '../../shared/pty-source-recovery-contract'
import { SshPtyRecoveryRetentionBudget } from './ssh-pty-recovery-retention-budget'
import { SshPtyRetiredSourceDeliveries } from './ssh-pty-retired-source-deliveries'
import {
claimSshPtyConsumerRecovery,
detachSshPtyConsumerRecovery,
forgetSshPtyConsumerRecovery,
getSshPtyConsumerRecovery,
rememberSshPtyConsumerRecovery,
removeSshPtyConsumerOwnerRecovery
} from './ssh-pty-consumer-recovery'
export type RelaySessionState = 'idle' | 'deploying' | 'ready' | 'reconnecting' | 'disposed'
type SshPtyExitPayload = Parameters<SshPtyExitCallback>[0]
type SshPtyDataPayload = Parameters<SshPtyDataCallback>[0]
type SshPtyLease = ReturnType<Store['getSshRemotePtyLeases']>[number]
const SSH_PTY_REATTACH_MAX_CONCURRENCY = 8
const SSH_PTY_REATTACH_ATTEMPT_TIMEOUT_MS = 10_000
const SSH_PTY_REATTACH_RETRY_MIN_DELAY_MS = 50
const SSH_PTY_REATTACH_RETRY_JITTER_MS = 200
const SSH_SOURCE_RECOVERY_CANCELLATION_FAILED = 'ssh_source_recovery_cancellation_failed'
type PendingPtyReattach = {
mux: SshChannelMultiplexer
providerGeneration: number
retentionKey: string
exits: SshPtyExitPayload[]
queuedData: SshPtyDataPayload[]
recoveryData: SshPtyDataPayload[]
liveData: SshPtyDataPayload[]
recovery?: PtySourceRecoveryPending
recoveryComplete?: PtySourceRecoveryComplete
nextRecoverySourceSu?: number
highestRecoverySourceEndSu?: number
replacementDeliveryToken?: string
restoreRequired?: string
recoveryWaiters: Set<() => void>
livePassthrough: boolean
activated: boolean
}
type RemoteCliBridgeEnv = {
remoteHome: string
binDir: string
relayDir: string
nodePath: string
sockPath: string
credentialFile?: string
hostPlatform: RemoteHostPlatform
pathDelimiter?: ':' | ';'
}
type ExpectedPtyIdentity = { paneKey?: string; tabId?: string }
function expectedIdentityForLease(lease: {
tabId?: string
leafId?: string
}): ExpectedPtyIdentity | null {
if (typeof lease.tabId !== 'string' || lease.tabId.length === 0) {
return null
}
const paneKey =
isValidTerminalTabId(lease.tabId) &&
typeof lease.leafId === 'string' &&
isTerminalLeafId(lease.leafId)
? makePaneKey(lease.tabId, lease.leafId)
: undefined
return {
...(paneKey ? { paneKey } : {}),
tabId: lease.tabId
}
}
function parseRecoveryComplete(params: Record<string, unknown>): PtySourceRecoveryComplete | null {
if (
typeof params.id !== 'string' ||
typeof params.deliveryToken !== 'string' ||
params.deliveryToken.length === 0 ||
typeof params.ptyIncarnation !== 'string' ||
params.ptyIncarnation.length === 0 ||
!positiveSafeInteger(params.clientGeneration) ||
!positiveSafeInteger(params.ownerGeneration) ||
!nonNegativeSafeInteger(params.checkpointSourceEndSu) ||
!nonNegativeSafeInteger(params.recoveryEndSu) ||
Number(params.recoveryEndSu) < Number(params.checkpointSourceEndSu)
) {
return null
}
return Object.freeze({
id: params.id,
deliveryToken: params.deliveryToken,
ptyIncarnation: params.ptyIncarnation,
clientGeneration: Number(params.clientGeneration),
ownerGeneration: Number(params.ownerGeneration),
checkpointSourceEndSu: Number(params.checkpointSourceEndSu),
recoveryEndSu: Number(params.recoveryEndSu)
})
}
function positiveSafeInteger(value: unknown): boolean {
return Number.isSafeInteger(value) && Number(value) > 0
}
function nonNegativeSafeInteger(value: unknown): boolean {
return Number.isSafeInteger(value) && Number(value) >= 0
}
function sourceRecoveryCancellationError(cause: unknown): Error {
return Object.assign(new Error(SSH_SOURCE_RECOVERY_CANCELLATION_FAILED), {
code: SSH_SOURCE_RECOVERY_CANCELLATION_FAILED,
cause
})
}
function isSourceRecoveryCancellationError(error: unknown): boolean {
return (error as { code?: unknown })?.code === SSH_SOURCE_RECOVERY_CANCELLATION_FAILED
}
export type SshRelayAiVaultHostInfo = {
targetId: string
executionHostId: ExecutionHostId
remoteHome: string
hostPlatform: RemoteHostPlatform
}
function normalizeRelayGracePeriodSeconds(graceTimeSeconds: number | undefined): number {
const raw = graceTimeSeconds ?? DEFAULT_SSH_RELAY_GRACE_PERIOD_SECONDS
const requested = Number.isFinite(raw) ? Math.floor(raw) : DEFAULT_SSH_RELAY_GRACE_PERIOD_SECONDS
return requested === 0
? 0
: Math.max(
MIN_SSH_RELAY_GRACE_PERIOD_SECONDS,
Math.min(MAX_SSH_RELAY_GRACE_PERIOD_SECONDS, requested)
)
}
export class SshRelaySession {
private _state: RelaySessionState = 'idle'
private mux: SshChannelMultiplexer | null = null
private abortController: AbortController | null = null
private muxDisposeCleanup: (() => void) | null = null
// Why: hold the notification-handler disposer so teardownProviders can release it on reconnect/shutdown (symmetric with muxDisposeCleanup).
private muxNotificationCleanup: (() => void) | null = null
// Why: onStateChange never fires when the relay channel closes but SSH stays up; this callback lets ssh.ts drive relay-level reconnect.
private _onRelayLost: ((targetId: string) => void) | null = null
// Why: version mismatch is terminal, so it needs a separate callback from _onRelayLost (which expects a recoverable transport drop).
private _onTerminalRelayError:
| ((targetId: string, err: RelayVersionMismatchError) => void)
| null = null
private _onReady: ((targetId: string) => void) | null = null
private portScanner: PortScanner | null = null
private currentConnection: SshConnection | null = null
private hostPlatform: RemoteHostPlatform | null = null
private remoteCliBridgeEnv: RemoteCliBridgeEnv | null = null
private pendingPtyReattaches = new Map<string, PendingPtyReattach>()
private readonly ptyRecoveryRetention = new SshPtyRecoveryRetentionBudget()
private activePtyProviderGeneration: number | null = null
private sourceAckPublisherCleanup: (() => void) | null = null
private sourceCancellationPublisherCleanup: (() => void) | null = null
private ptyRecoveryNotificationCleanups: (() => void)[] = []
private readonly sourceIdentityByRelayPtyId = new Map<
string,
Readonly<{
deliveryToken: string
clientGeneration: number
ownerGeneration: number
ptyIncarnation: string
nextSourceSu?: number
}>
>()
private readonly retiredSourceDeliveries = new SshPtyRetiredSourceDeliveries()
private readonly ptyConsumerClientInstanceId: string
private ptyConsumerSessionState: SshPtyConsumerSessionState | null = null
private activeCompatibilityAttachmentIds = new Set<string>()
constructor(
readonly targetId: string,
private getMainWindow: () => BrowserWindow | null,
private store: Store,
private portForwardManager: SshPortForwardManager,
private runtime?: OrcaRuntimeService,
private onDetectedPortsChanged?: (
targetId: string,
ports: DetectedPort[],
platform: string
) => void
) {
this.ptyConsumerClientInstanceId = claimSshPtyConsumerRecovery(targetId, store).clientInstanceId
}
refreshEnvironment(
getMainWindow: () => BrowserWindow | null,
store: Store,
portForwardManager: SshPortForwardManager,
runtime?: OrcaRuntimeService,
onDetectedPortsChanged?: (targetId: string, ports: DetectedPort[], platform: string) => void
): void {
this.getMainWindow = getMainWindow
this.store = store
this.portForwardManager = portForwardManager
this.runtime = runtime
this.onDetectedPortsChanged = onDetectedPortsChanged
}
setOnRelayLost(cb: (targetId: string) => void): void {
this._onRelayLost = cb
}
setOnTerminalRelayError(cb: (targetId: string, err: RelayVersionMismatchError) => void): void {
this._onTerminalRelayError = cb
}
setOnReady(cb: (targetId: string) => void): void {
this._onReady = cb
}
getState(): RelaySessionState {
return this._state
}
// Why: dispose() can mutate _state across await points, so defeat TS's control-flow narrowing that would otherwise reject the 'disposed' check.
private isDisposed(): boolean {
return (this._state as RelaySessionState) === 'disposed'
}
private requireReadyConnection(): SshConnection {
if (!this.currentConnection) {
throw new Error('SSH connection is not active')
}
return this.currentConnection
}
getMux(): SshChannelMultiplexer | null {
return this.mux
}
getHostPlatform(): RemoteHostPlatform | null {
return this.remoteCliBridgeEnv?.hostPlatform ?? this.hostPlatform
}
getAiVaultHostInfo(): SshRelayAiVaultHostInfo | null {
const env = this.remoteCliBridgeEnv
if (!env) {
return null
}
return {
targetId: this.targetId,
executionHostId: toSshExecutionHostId(this.targetId),
remoteHome: env.remoteHome,
hostPlatform: env.hostPlatform
}
}
getPortScanner(): PortScanner | null {
return this.portScanner
}
prepareForHostSleep(): void {
const mux = this.mux
if (!mux || mux.isDisposed() || this.isDisposed()) {
return
}
mux.notify(SSH_RELAY_CONFIGURE_GRACE_TIME_METHOD, { graceTimeSeconds: 0 })
}
// Why: single entry point for relay setup (initial connect + app-restart reconnect) so no path forgets a registration step.
async establish(conn: SshConnection, graceTimeSeconds?: number): Promise<void> {
if (this._state !== 'idle') {
throw new Error(`Cannot establish relay session in state: ${this._state}`)
}
this._state = 'deploying'
this.currentConnection = conn
try {
const {
transport,
serverBuildId,
remoteHome,
remoteRelayDir,
nodePath,
sockPath,
credentialFile,
hostPlatform
} = await deployAndLaunchRelay(conn, undefined, graceTimeSeconds, this.targetId)
this.hostPlatform = hostPlatform ?? null
this.remoteCliBridgeEnv =
remoteHome && remoteRelayDir && nodePath && sockPath && hostPlatform
? {
remoteHome,
binDir: joinRemotePath(hostPlatform, remoteHome, '.orca-relay', 'bin'),
relayDir: remoteRelayDir,
nodePath,
sockPath,
...(credentialFile ? { credentialFile } : {}),
hostPlatform,
pathDelimiter: hostPlatform.pathDelimiter
}
: null
// Why: dispose() can fire during the await above; if it did, creating a mux/providers now would leak with no owner to dispose them.
if (this.isDisposed()) {
const orphanMux = new SshChannelMultiplexer(transport)
orphanMux.dispose()
throw new Error('Session disposed during establish')
}
const mux = new SshChannelMultiplexer(transport)
this.mux = mux
const ownsAttempt = (): boolean => this.mux === mux && !this.isDisposed()
this.ptyConsumerSessionState = await this.openPtyConsumerSession(mux, serverBuildId)
this.rememberPtyConsumerRecovery(serverBuildId)
await mux.request('session.resolveHome', { path: '~' })
if (!ownsAttempt()) {
if (!mux.isDisposed()) {
mux.dispose()
}
throw new Error('Session disposed during establish')
}
const connectionIncarnation = randomUUID()
const registered = await this.registerProviders(mux, ownsAttempt, connectionIncarnation)
if (!registered) {
if (!mux.isDisposed()) {
mux.dispose()
}
throw new Error('Session disposed during establish')
}
// Why: registerProviders swallows mux errors, so an isDisposed check catches a transport that closed mid-registration before we reach 'ready'.
if (mux.isDisposed()) {
throw new Error('Relay connection lost during provider registration')
}
if (this.isDisposed()) {
this.teardownProviders('connection_lost')
throw new Error('Session disposed during establish')
}
// Why: explicit disconnect keeps PTY ownership, so a later manual connect must reattach those remote PTYs.
await this.reattachKnownPtys(mux, ownsAttempt)
if (!ownsAttempt()) {
throw new Error('Session disposed during establish')
}
this.configureRelayGraceTime(mux, graceTimeSeconds)
this.watchMuxForRelayLoss(mux)
this._state = 'ready'
this.startPortScanning()
this._onReady?.(this.targetId)
} catch (err) {
// Why: registerProviders can throw with a live mux and partial registration — tear everything down so a retry starts clean.
if (!this.isDisposed()) {
this.teardownProviders(
'connection_lost',
isSourceRecoveryCancellationError(err)
? SSH_SOURCE_RECOVERY_CANCELLATION_FAILED
: 'connection_lost'
)
this._state = 'idle'
}
// Why: a version mismatch on first connect is terminal (deployed binary vs. a still-running legacy daemon); notify the callback but still rethrow.
if (isRelayVersionMismatchError(err)) {
console.warn(
`[ssh-relay-session] Terminal relay version mismatch on initial connect for ${this.targetId}: ${err.message}`
)
this._onTerminalRelayError?.(this.targetId, err)
}
throw err
}
}
// Why: network-blip reconnect; AbortController-guarded so overlapping attempts from fast flaps cancel the stale one.
async reconnect(conn: SshConnection, graceTimeSeconds?: number): Promise<void> {
// Why: reconnect only from 'ready'/'reconnecting' — from 'deploying' it would tear down a mux establish() is still using; 'idle' has no session yet.
if (this._state !== 'ready' && this._state !== 'reconnecting') {
return
}
// Cancel any in-flight reconnect
this.abortController?.abort()
const abortController = new AbortController()
this.abortController = abortController
this._state = 'reconnecting'
this.currentConnection = conn
// Why: stop scanning before teardownProviders so the poll timer can't fire against a disposed multiplexer.
this.stopPortScanning()
await this.portForwardManager.removeAllForwards(this.targetId)
this.broadcastEmptyLists()
this.teardownProviders('connection_lost')
try {
const {
transport,
serverBuildId,
remoteHome,
remoteRelayDir,
nodePath,
sockPath,
credentialFile,
hostPlatform
} = await deployAndLaunchRelay(conn, undefined, graceTimeSeconds, this.targetId)
this.hostPlatform = hostPlatform ?? null
this.remoteCliBridgeEnv =
remoteHome && remoteRelayDir && nodePath && sockPath && hostPlatform
? {
remoteHome,
binDir: joinRemotePath(hostPlatform, remoteHome, '.orca-relay', 'bin'),
relayDir: remoteRelayDir,
nodePath,
sockPath,
...(credentialFile ? { credentialFile } : {}),
hostPlatform,
pathDelimiter: hostPlatform.pathDelimiter
}
: null
if (abortController.signal.aborted || this.isDisposed()) {
// Why: relay is already running remotely — a throwaway mux we immediately dispose sends a clean shutdown so it doesn't linger until grace expires.
const orphanMux = new SshChannelMultiplexer(transport)
orphanMux.dispose()
return
}
const mux = new SshChannelMultiplexer(transport)
this.mux = mux
const ownsAttempt = (): boolean =>
this.abortController === abortController &&
!abortController.signal.aborted &&
!this.isDisposed()
this.ptyConsumerSessionState = await this.openPtyConsumerSession(mux, serverBuildId)
this.rememberPtyConsumerRecovery(serverBuildId)
if (!ownsAttempt()) {
if (!mux.isDisposed()) {
mux.dispose()
}
return
}
await mux.request('session.resolveHome', { path: '~' })
if (!ownsAttempt()) {
if (!mux.isDisposed()) {
mux.dispose()
}
return
}
const connectionIncarnation = randomUUID()
const registered = await this.registerProviders(mux, ownsAttempt, connectionIncarnation)
if (!registered) {
if (!mux.isDisposed()) {
mux.dispose()
}
return
}
if (mux.isDisposed()) {
throw new Error('Relay connection lost during provider registration')
}
// Why: dispose() during registration/attach already cleaned up, but this.mux was reassigned above — clean up the new mux so it doesn't leak.
if (!ownsAttempt()) {
if (this.mux === mux) {
this.teardownProviders('shutdown')
} else if (!mux.isDisposed()) {
mux.dispose()
}
return
}
await this.reattachKnownPtys(mux, ownsAttempt)
if (!ownsAttempt()) {
return
}
this.configureRelayGraceTime(mux, graceTimeSeconds)
this.watchMuxForRelayLoss(mux)
this._state = 'ready'
this.startPortScanning()
this._onReady?.(this.targetId)
} catch (err) {
// Why: tear down a partially-registered mux so its keepalive/timeout timers don't keep running on a half-initialized session.
if (this.abortController === abortController && !this.isDisposed()) {
this.teardownProviders(
'connection_lost',
isSourceRecoveryCancellationError(err)
? SSH_SOURCE_RECOVERY_CANCELLATION_FAILED
: 'connection_lost'
)
}
// Why: version-mismatch is terminal — fire the typed callback and drop out of 'reconnecting' since backoff retry can't reconcile it.
if (isRelayVersionMismatchError(err)) {
console.warn(
`[ssh-relay-session] Terminal relay version mismatch for ${this.targetId}: ${err.message}`
)
if (this.abortController === abortController && !this.isDisposed()) {
this._state = 'idle'
}
this._onTerminalRelayError?.(this.targetId, err)
return
}
// Why: stay in 'reconnecting' (not 'ready') since the provider stack is torn down; the SSH manager will fire another onStateChange to retry.
console.warn(
`[ssh-relay-session] Failed to re-establish relay for ${this.targetId}: ${err instanceof Error ? err.message : String(err)}`
)
if (this.abortController === abortController && !this.isDisposed()) {
// Why: treat non-not-found attach failures as relay loss so ssh.ts's bounded backoff retries instead of stranding the session in 'reconnecting'.
this._onRelayLost?.(this.targetId)
}
} finally {
if (this.abortController === abortController) {
this.abortController = null
}
}
}
dispose(): void {
if (this._state === 'disposed') {
return
}
this.abortController?.abort()
this.stopPortScanning()
// Why: fire-and-forget — nothing rebinds after dispose, so no need to await port release.
void this.portForwardManager.removeAllForwards(this.targetId)
this.broadcastEmptyLists()
this.teardownProviders('shutdown')
this.store.markSshRemotePtyLeases(this.targetId, 'terminated')
this.currentConnection = null
this._state = 'disposed'
forgetSshPtyConsumerRecovery(this.targetId, this.ptyConsumerClientInstanceId, this.store)
}
detach(): void {
if (this._state === 'disposed') {
return
}
this.abortController?.abort()
this.stopPortScanning()
this.broadcastEmptyLists()
// Why: window disconnect is non-destructive — unregister local providers but keep PTY ownership so reattach works (relay owns the grace timer).
this.teardownProviders('connection_lost')
this.store.markSshRemotePtyLeases(this.targetId, 'detached')
this.currentConnection = null
this._state = 'disposed'
detachSshPtyConsumerRecovery(this.targetId, this.ptyConsumerClientInstanceId)
}
// ── Private ───────────────────────────────────────────────────────
// Why: onStateChange only fires on SSH-level reconnects, so watch for relay-channel loss while SSH stays up and fire onRelayLost.
private watchMuxForRelayLoss(mux: SshChannelMultiplexer): void {
this.muxDisposeCleanup?.()
this.muxDisposeCleanup = mux.onDispose((reason) => {
if (reason === 'connection_lost' && this.mux === mux && !this.isDisposed()) {
console.warn(
`[ssh-relay-session] Relay channel lost for ${this.targetId}, triggering reconnect`
)
this._onRelayLost?.(this.targetId)
}
})
}
// Why: shared by establish() and reconnect() so both use the exact same registration sequence.
private async registerProviders(
mux: SshChannelMultiplexer,
shouldContinue: (() => boolean) | undefined,
connectionIncarnation: string
): Promise<boolean> {
await this.registerRelayRoots(mux)
if (shouldContinue && !shouldContinue()) {
return false
}
await this.installPluginsOnRelay(mux)
if (shouldContinue && !shouldContinue()) {
return false
}
try {
await this.installRemoteOrcaCliLauncher()
} catch (error) {
// Why: on MaxSessions=1 remotes the relay holds the only slot, so this raw-connection install can fail — don't fail the whole connection.
console.warn(
`[ssh-relay-session] remote orca CLI launcher install failed for ${this.targetId}: ${
error instanceof Error ? error.message : String(error)
}`
)
}
if (shouldContinue && !shouldContinue()) {
return false
}
this.wireUpRemoteOrcaCli(mux, connectionIncarnation)
const providerGeneration = allocateSshPtyProviderGeneration()
const ptyProvider = new SshPtyProvider(
this.targetId,
mux,
this.remoteCliBridgeEnv ?? undefined,
providerGeneration
)
const consumerOwnerState = this.activePtyConsumerOwner()
if (consumerOwnerState) {
ptyProvider.setPtyDeliveryPauseAdapter?.(({ id, providerGeneration: generation, paused }) => {
if (
generation !== providerGeneration ||
this.activePtyProviderGeneration !== providerGeneration ||
this.mux !== mux
) {
return
}
const sourceIdentity = this.sourceIdentityByRelayPtyId.get(id)
if (consumerOwnerState.outputFlowControl && !sourceIdentity) {
return
}
mux.notify('pty.setDeliveryPaused', {
id,
paused,
clientGeneration: consumerOwnerState.clientGeneration,
ownerGeneration: consumerOwnerState.ownerGeneration,
...(sourceIdentity ? { deliveryToken: sourceIdentity.deliveryToken } : {})
})
})
}
this.sourceAckPublisherCleanup?.()
this.sourceAckPublisherCleanup = null
this.sourceCancellationPublisherCleanup?.()
this.sourceCancellationPublisherCleanup = null
if (consumerOwnerState?.outputFlowControl) {
this.sourceAckPublisherCleanup = installSshPtySourceAckPublisher(
providerGeneration,
(batch, onSettled) =>
mux.notifyWithSettlement(
'pty.ackData',
batch as unknown as Record<string, unknown>,
onSettled
)
)
this.sourceCancellationPublisherCleanup = installSshPtySourceCancellationPublisher(
providerGeneration,
async (request) => {
const result = (await mux.request('pty.cancelDelivery', {
...request,
id: toRelaySshPtyId(this.targetId, request.id)
})) as Record<string, unknown>
if (
result.canceled !== true ||
!Number.isSafeInteger(result.sentEndSu) ||
!Number.isSafeInteger(result.creditedEndSu)
) {
throw new Error('ssh_source_cancellation_proof_invalid')
}
return {
sentEndSu: result.sentEndSu as number,
creditedEndSu: result.creditedEndSu as number
}
}
)
}
this.activePtyProviderGeneration = providerGeneration
registerSshPtyProvider(this.targetId, ptyProvider)
this.installPtyRecoveryNotifications(mux)
const connection = this.requireReadyConnection()
const createSftp =
connection.usesSystemSshTransport?.() === true
? undefined
: (options?: { signal?: AbortSignal }) => this.requireReadyConnection().sftp(options)
// Why: getHostPlatform() falls back to this.hostPlatform when bridge env is incomplete, so path rules still match the host.
const hostPlatform = this.getHostPlatform() ?? undefined
const fsProvider = new SshFilesystemProvider(
this.targetId,
mux,
createSftp,
{
downloadFile: (sourcePath, destinationPath) =>
this.requireReadyConnection().downloadFile(sourcePath, destinationPath, {
hostPlatform
}),
openFileUploadSession: () =>
this.requireReadyConnection().openFileUploadSession({
hostPlatform
}),
writeBuffer: (remotePath, contents, options) =>
this.requireReadyConnection().writeBuffer(remotePath, contents, {
hostPlatform,
append: options.append,
exclusive: options.exclusive
})
},
hostPlatform
)
registerSshFilesystemProvider(this.targetId, fsProvider)
const gitProvider = new SshGitProvider(
this.targetId,
mux,
this.remoteCliBridgeEnv?.hostPlatform ?? null
)
registerSshGitProvider(this.targetId, gitProvider)
this.wireUpPtyEvents(ptyProvider, mux, providerGeneration)
this.wireUpAgentHookEvents(mux)
this.wireUpRemoteWorkspaceEvents(mux)
void this.installManagedHooksOnRemote(mux, shouldContinue)
return true
}
private activePtyConsumerOwner(): SshPtyConsumerOwnerState | null {
const state = this.ptyConsumerSessionState
return state && state.mode !== 'legacy-fallback' ? state : null
}
private recoverablePtyConsumerOwner(
serverBuildId: string | undefined
): SshPtyConsumerOwnerState | null {
const active = this.activePtyConsumerOwner()
if (active) {
return active
}
const recovery = getSshPtyConsumerRecovery(this.targetId)
return serverBuildId && recovery?.serverBuildId === serverBuildId
? (recovery?.owner ?? null)
: null
}
private async openPtyConsumerSession(
mux: SshChannelMultiplexer,
serverBuildId: string | undefined
): Promise<SshPtyConsumerSessionState> {
const previousOwner = this.recoverablePtyConsumerOwner(serverBuildId)
const options = {
clientInstanceId: this.ptyConsumerClientInstanceId,
expectedServerBuildId: serverBuildId,
allowSameBuildLegacyFallback: true,
outputFlowControl: { requestedWindowSu: DEFAULT_PTY_SOURCE_WINDOW_SU }
}
try {
return await openSshPtyConsumerSession(mux, {
...options,
...(previousOwner
? {
resume: {
ownerGeneration: previousOwner.ownerGeneration,
ownerLease: previousOwner.ownerLease
}
}
: {})
})
} catch (error) {
if (
!previousOwner ||
(error as { code?: unknown }).code !== PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR
) {
throw error
}
const recovery = getSshPtyConsumerRecovery(this.targetId)
if (recovery) {
delete recovery.owner
recovery.checkpointsByAppPtyId.clear()
for (const [ptyId, migration] of recovery.modelMigrationsByAppPtyId) {
recovery.modelMigrationsByAppPtyId.set(
ptyId,
migration.then(() =>
Object.freeze({
status: 'checkpoint-unavailable' as const,
reason: 'completion-failed' as const
})
)
)
}
}
removeSshPtyConsumerOwnerRecovery(this.targetId, this.ptyConsumerClientInstanceId, this.store)
this.ptyConsumerSessionState = null
return openSshPtyConsumerSession(mux, options)
}
}
private rememberPtyConsumerRecovery(serverBuildId: string | undefined): void {
const owner = this.activePtyConsumerOwner()
if (!owner || !serverBuildId) {
return
}
rememberSshPtyConsumerRecovery({
targetId: this.targetId,
clientInstanceId: this.ptyConsumerClientInstanceId,
serverBuildId,
owner,
store: this.store
})
}
private configureRelayGraceTime(
mux: SshChannelMultiplexer,
graceTimeSeconds: number | undefined
): void {
mux.notify(SSH_RELAY_CONFIGURE_GRACE_TIME_METHOD, {
graceTimeSeconds: normalizeRelayGracePeriodSeconds(graceTimeSeconds)
})
}
private async installManagedHooksOnRemote(
mux: SshChannelMultiplexer,
shouldContinue?: () => boolean
): Promise<void> {
if (
!isRemoteAgentHooksEnabled() ||
!this.areAgentStatusHooksEnabled() ||
(shouldContinue && !shouldContinue())
) {
return
}
if (
this.remoteCliBridgeEnv?.hostPlatform &&
isWindowsRemoteHost(this.remoteCliBridgeEnv.hostPlatform)
) {
// Why: managed hook installers emit POSIX-only scripts/paths; Windows remotes rely on relay-injected env + plugin overlays instead.
return
}
try {
const store = this.store as { getSettings?: Store['getSettings'] }
const detected = (await mux.request('preflight.detectAgents', {
commands: buildManagedHookDetectionCommands(store.getSettings?.() ?? null, 'linux')
})) as { agents?: unknown }
const agents = detectedManagedHookAgents(detected?.agents)
if (agents.length === 0 || (shouldContinue && !shouldContinue())) {
return
}
const hostKeyFingerprint = this.requireReadyConnection().getHostKeyFingerprint?.()
const params = {
...(hostKeyFingerprint ? { hostKeyFingerprint } : {}),
agents
}
const result = (await mux.request(AGENT_HOOK_INSTALL_MANAGED_HOOKS_METHOD, params)) as {
errors?: unknown
}
if (typeof result.errors === 'number' && result.errors > 0) {
console.warn(
`[ssh-relay-session] ${result.errors} remote managed hook installers failed for ${this.targetId}`
)
}
} catch (error) {
// Why: teardown routinely cancels this best-effort request; only warn for
// installer failures that survive the connection lifecycle.
const code = (error as { code?: unknown })?.code
if (
code === -32601 ||
code === 'CONNECTION_LOST' ||
code === 'DISPOSED' ||
mux.isDisposed()
) {
return
}
console.warn(
`[ssh-relay-session] relay managed hook install failed for ${this.targetId}: ${
error instanceof Error ? error.message : String(error)
}`
)
}
}
private async installRemoteOrcaCliLauncher(): Promise<void> {
if (!this.remoteCliBridgeEnv) {
return
}
const { binDir, hostPlatform } = this.remoteCliBridgeEnv
const plan = createRemoteCliInstallPlan(this.remoteCliBridgeEnv)
const conn = this.requireReadyConnection()
await execCommand(conn, makeRemoteDirectoryCommand(hostPlatform, binDir), {
wrapCommand: !isWindowsRemoteHost(hostPlatform)
})
if (typeof conn.writeFile === 'function') {
for (const file of plan.files) {
await conn.writeFile(file.path, file.contents, { hostPlatform })
}
} else {
const sftp = await conn.sftp()
try {
for (const file of plan.files) {
await new Promise<void>((resolve, reject) => {
const ws = sftp.createWriteStream(file.path)
sftp.once('error', reject)
ws.once('close', resolve)
ws.once('error', reject)
ws.end(file.contents)
})
}
} finally {
sftp.end()
}
}
for (const command of plan.postWriteCommands) {
await execCommand(conn, command, { wrapCommand: !isWindowsRemoteHost(hostPlatform) })
}
}
private wireUpRemoteOrcaCli(mux: SshChannelMultiplexer, connectionIncarnation: string): void {
mux.onRequest('orca.cli', async (params) => {
if (!this.runtime) {
throw new Error('Orca runtime is unavailable')
}
const argv = Array.isArray(params.argv)
? params.argv.filter((item): item is string => typeof item === 'string')
: []
const cwd = typeof params.cwd === 'string' && params.cwd.length > 0 ? params.cwd : '/'
const rawEnv = params.env
const env =
rawEnv && typeof rawEnv === 'object' && !Array.isArray(rawEnv)
? Object.fromEntries(
Object.entries(rawEnv).filter(
(entry): entry is [string, string] =>
typeof entry[0] === 'string' && typeof entry[1] === 'string'
)
)
: {}
const stdin = typeof params.stdin === 'string' ? params.stdin : undefined
const runtimeAuthority = this.runtime.registerOrchestrationCompatibilitySshAttachment(
this.targetId,
connectionIncarnation
)
this.activeCompatibilityAttachmentIds.add(runtimeAuthority.attachmentId)
try {
return await runRemoteOrcaCli(this.runtime, {
argv,
cwd,
env,
...(stdin !== undefined ? { stdin } : {}),
runtimeAuthority
})
} finally {
this.activeCompatibilityAttachmentIds.delete(runtimeAuthority.attachmentId)
this.runtime.releaseOrchestrationCompatibilitySshAttachment(runtimeAuthority.attachmentId)
}
})
mux.onRequest('orca.cli.postOutput', async (params) => {
if (!this.runtime) {
throw new Error('Orca runtime is unavailable')
}
const rawEnv = params.env
const env =
rawEnv && typeof rawEnv === 'object' && !Array.isArray(rawEnv)
? Object.fromEntries(
Object.entries(rawEnv).filter(
(entry): entry is [string, string] =>
typeof entry[0] === 'string' && typeof entry[1] === 'string'
)
)
: {}
const runtimeAuthority = this.runtime.registerOrchestrationCompatibilitySshAttachment(
this.targetId,
connectionIncarnation
)
this.activeCompatibilityAttachmentIds.add(runtimeAuthority.attachmentId)
try {
await acknowledgeRemoteOrcaCliPostOutput(this.runtime, {
postOutput: parseRemoteOrcaCliPostOutput(params.postOutput),
env,
runtimeAuthority
})
return { acknowledged: true }
} finally {
this.activeCompatibilityAttachmentIds.delete(runtimeAuthority.attachmentId)
this.runtime.releaseOrchestrationCompatibilitySshAttachment(runtimeAuthority.attachmentId)
}
})
}
// Why: ship plugin/extension source from Orca so agent-event changes don't force a relay redeploy (agent-status-over-ssh.md §4/§8). Best-effort.
private async installPluginsOnRelay(mux: SshChannelMultiplexer): Promise<void> {
if (!isRemoteAgentHooksEnabled() || !this.areAgentStatusHooksEnabled()) {
return
}
try {
await mux.request(AGENT_HOOK_INSTALL_PLUGINS_METHOD, {
opencodePluginSource: openCodeInternals.getOpenCodePluginSource(),
piExtensionSource: getPiAgentStatusExtensionSource('pi'),
ompExtensionSource: getPiAgentStatusExtensionSource('omp')
})
} catch (err) {
// Why: -32601 = older relay without the handler; CONNECTION_LOST/DISPOSED = routine mid-flight teardown — swallow both.
const code = (err as { code?: unknown })?.code
if (code === -32601 || code === 'CONNECTION_LOST' || code === 'DISPOSED') {
return
}
if (mux.isDisposed()) {
return
}
console.warn(
`[ssh-relay-session] agent_hook.installPlugins failed for ${this.targetId}: ${
err instanceof Error ? err.message : String(err)
}`
)
}
}
private areAgentStatusHooksEnabled(): boolean {
const store = this.store as { getSettings?: Store['getSettings'] }
return isAgentStatusHooksEnabled(store.getSettings?.())
}
private wireUpRemoteWorkspaceEvents(mux: SshChannelMultiplexer): void {
mux.onNotification((method, params) => {
notifyRemoteWorkspaceHandlers(this.targetId, method, params)
})
}
// Why: relay sends connectionId:null, so stamp this.targetId here so the renderer can drop events from torn-down connections.
private wireUpAgentHookEvents(mux: SshChannelMultiplexer): void {
if (!isRemoteAgentHooksEnabled()) {
return
}
// Why: capture the disposer so teardownProviders can release this handler and re-wiring can't double-register it.
this.muxNotificationCleanup?.()
this.muxNotificationCleanup = mux.onNotification((method, params) => {
if (method !== AGENT_HOOK_NOTIFICATION_METHOD) {
return
}
const envelope = params as {
paneKey?: unknown
launchToken?: unknown
tabId?: unknown
worktreeId?: unknown
env?: unknown
version?: unknown
hasExplicitPrompt?: unknown
promptInteractionKey?: unknown
hookEventName?: unknown
toolUseId?: unknown
toolAgentId?: unknown
toolAgentType?: unknown
isReplay?: unknown
providerSession?: unknown
providerSessionOnly?: unknown
shedFields?: unknown
payload?: unknown
}
if (typeof envelope.paneKey !== 'string') {
return
}
// Why: forward env/version verbatim so cross-build warn-once diagnostics fire on remote events too (agent-status-over-ssh.md §3).
agentHookServer.ingestRemote(
{
paneKey: envelope.paneKey,
launchToken: typeof envelope.launchToken === 'string' ? envelope.launchToken : undefined,
tabId: typeof envelope.tabId === 'string' ? envelope.tabId : undefined,
worktreeId: typeof envelope.worktreeId === 'string' ? envelope.worktreeId : undefined,
env: typeof envelope.env === 'string' ? envelope.env : undefined,
version: typeof envelope.version === 'string' ? envelope.version : undefined,
hasExplicitPrompt: envelope.hasExplicitPrompt === true ? true : undefined,
promptInteractionKey:
typeof envelope.promptInteractionKey === 'string'
? envelope.promptInteractionKey
: undefined,
hookEventName:
typeof envelope.hookEventName === 'string' ? envelope.hookEventName : undefined,
toolUseId: typeof envelope.toolUseId === 'string' ? envelope.toolUseId : undefined,
toolAgentId: typeof envelope.toolAgentId === 'string' ? envelope.toolAgentId : undefined,
toolAgentType:
typeof envelope.toolAgentType === 'string' ? envelope.toolAgentType : undefined,
isReplay: envelope.isReplay === true ? true : undefined,
providerSession: envelope.providerSession,
providerSessionOnly: envelope.providerSessionOnly === true ? true : undefined,
// Why: names the fields the relay dropped to fit the frame; ingestRemote restores them.
shedFields: envelope.shedFields,
payload: envelope.payload
},
this.targetId
)
})
// Why: request replay of cached paneKeys only after the handler is wired, so replayed events can't arrive before we subscribe. Best-effort.
void mux.request(AGENT_HOOK_REQUEST_REPLAY_METHOD).catch((err) => {
const code = (err as { code?: unknown })?.code
if (code === -32601 || code === 'CONNECTION_LOST' || code === 'DISPOSED') {
return
}
if (mux.isDisposed()) {
return
}
// Why: suppress the warn when a normal teardown rejects the in-flight request, so reconnect cycles aren't noisy.
if (mux.isDisposed()) {
return
}
console.warn(
`[ssh-relay-session] agent_hook.requestReplay failed for ${this.targetId}: ${
err instanceof Error ? err.message : String(err)
}`
)
})
}
private teardownProviders(
reason: 'shutdown' | 'connection_lost',
outputGenerationReason: string = reason
): void {
this.muxDisposeCleanup?.()
this.muxDisposeCleanup = null
this.muxNotificationCleanup?.()
this.muxNotificationCleanup = null
for (const cleanup of this.ptyRecoveryNotificationCleanups) {
cleanup()
}
this.ptyRecoveryNotificationCleanups = []
if (this.activePtyProviderGeneration !== null) {
const providerGeneration = this.activePtyProviderGeneration
if (reason === 'connection_lost' && this.activePtyConsumerOwner()?.outputFlowControl) {
this.beginPtyModelMigration(providerGeneration, outputGenerationReason)
} else {
closeSshPtyOutputGeneration(providerGeneration, outputGenerationReason)
}
this.activePtyProviderGeneration = null
}
this.sourceAckPublisherCleanup?.()
this.sourceAckPublisherCleanup = null
this.sourceCancellationPublisherCleanup?.()
this.sourceCancellationPublisherCleanup = null
if (this.mux && !this.mux.isDisposed()) {
this.mux.dispose(reason)
}
this.mux = null
for (const attachmentId of this.activeCompatibilityAttachmentIds) {
this.runtime?.releaseOrchestrationCompatibilitySshAttachment(attachmentId)
}
this.activeCompatibilityAttachmentIds.clear()
if (reason === 'shutdown') {
clearPtyOwnershipForConnection(this.targetId)
} else {
// Why: handlers detached above, so no late event can re-stamp status between this clear and reconnect replay.
agentHookServer.clearStatusEntriesForConnection(this.targetId)
}
const ptyProvider = getSshPtyProvider(this.targetId)
if (ptyProvider && 'dispose' in ptyProvider) {
;(ptyProvider as { dispose: () => void }).dispose()
}
const fsProvider = getSshFilesystemProvider(this.targetId)
if (fsProvider && 'dispose' in fsProvider) {
;(fsProvider as { dispose: () => void }).dispose()
}
unregisterSshPtyProvider(this.targetId)
unregisterSshFilesystemProvider(this.targetId)
unregisterSshGitProvider(this.targetId)
this.sourceIdentityByRelayPtyId.clear()
this.retiredSourceDeliveries.clear()
for (const pending of this.pendingPtyReattaches.values()) {
for (const resolve of pending.recoveryWaiters) {
resolve()
}
}
this.pendingPtyReattaches.clear()
this.ptyRecoveryRetention.clear()
}
// Why: back-compat for old relays that gate FS ops on registered roots; removable post-cutover (docs/relay-fs-allowlist-removal.md).
private async registerRelayRoots(mux: SshChannelMultiplexer): Promise<void> {
const remoteRepos = this.store.getRepos().filter((r) => r.connectionId === this.targetId)
for (const repo of remoteRepos) {
mux.notify('session.registerRoot', { rootPath: repo.path })
}
// Why: git.listWorktrees requires the repo root to be registered first.
await Promise.all(
remoteRepos.map(async (repo) => {
try {
const worktrees = (await mux.request('git.listWorktrees', {
repoPath: repo.path
})) as { path: string }[]
for (const wt of worktrees) {
if (wt.path !== repo.path) {
mux.notify('session.registerRoot', { rootPath: wt.path })
}
}
} catch {
// git worktree list may fail for folder-mode repos — not fatal
}
})
)
}
// Why: shared by establish()/reconnect() so both paths reset renderer lists the same way.
private broadcastEmptyLists(): void {
const win = this.getMainWindow()
if (!win || win.isDestroyed()) {
return
}
win.webContents.send('ssh:port-forwards-changed', {
targetId: this.targetId,
forwards: []
})
win.webContents.send('ssh:detected-ports-changed', {
targetId: this.targetId,
ports: []
})
}
private startPortScanning(): void {
if (!this.mux || this.isDisposed()) {
return
}
// Why: each scan walks /proc/*/fd remotely, so skip ticks while the window is hidden and rescan when it returns.
const scanner = new PortScanner({
isWindowVisible: () => isMainWindowVisible(this.getMainWindow()),
onWindowBecameVisible: onMainWindowBecameVisible
})
this.portScanner = scanner
// Why: guard against a late ports.detect callback from a pre-reconnect scanner publishing stale results into the new session.
scanner.startScanning(this.targetId, this.mux, (targetId, ports, platform) => {
if (this.portScanner !== scanner) {
return
}
this.onDetectedPortsChanged?.(targetId, ports, platform)
})
}
private stopPortScanning(): void {
if (this.portScanner) {
this.portScanner.stopScanning(this.targetId)
this.portScanner = null
}
}
private wireUpPtyEvents(
ptyProvider: SshPtyProvider,
mux: SshChannelMultiplexer,
providerGeneration: number
): void {
ptyProvider.onData((payload) => {
if (
this.mux !== mux ||
this.activePtyProviderGeneration !== providerGeneration ||
payload.providerGeneration !== providerGeneration
) {
return
}
const pending = this.pendingPtyReattaches.get(payload.id)
if (pending && this.activePtyConsumerOwner()?.outputFlowControl) {
if (pending.livePassthrough) {
void this.acceptPtyData(payload).catch(() => {})
return
}
this.quarantineReattachData(pending, payload)
return
}
void this.acceptPtyData(payload).catch(() => {})
})
ptyProvider.onReplay((payload) => {
if (this.mux !== mux || this.activePtyProviderGeneration !== providerGeneration) {
return
}
const win = this.getMainWindow()
if (win && !win.isDestroyed()) {
win.webContents.send('pty:replay', payload)
}
})
ptyProvider.onExit((payload) => {
if (
this.mux !== mux ||
this.activePtyProviderGeneration !== providerGeneration ||
payload.providerGeneration !== providerGeneration
) {
return
}
const pendingReattach = this.pendingPtyReattaches.get(payload.id)
if (pendingReattach && !pendingReattach.activated) {
// Why: attach response and exit can share one transport batch, before incarnation restoration runs.
pendingReattach.exits.push(payload)
this.wakeRecovery(pendingReattach)
return
}
if (!isCurrentPtyExit(payload)) {
return
}
void this.acceptPtyExit(payload).catch(() => {})
})
}
private acceptPtyData(payload: SshPtyDataPayload): Promise<unknown> {
const consumerOwner = this.activePtyConsumerOwner()
const offeredSource = payload.source
if (
offeredSource &&
this.retiredSourceDeliveries.has(payload.providerGeneration, offeredSource)
) {
return Promise.resolve()
}
if (
consumerOwner?.outputFlowControl &&
(!offeredSource ||
payload.sourceMalformed ||
offeredSource.clientGeneration !== consumerOwner.clientGeneration ||
offeredSource.ownerGeneration !== consumerOwner.ownerGeneration)
) {
closeSshPtyOutputGeneration(
payload.providerGeneration,
'ssh_source_frame_malformed_or_missing'
)
this.mux?.dispose('connection_lost')
return Promise.reject(new Error('ssh_source_frame_malformed_or_missing'))
}
const source = consumerOwner?.outputFlowControl ? offeredSource : undefined
if (source) {
const current = this.sourceIdentityByRelayPtyId.get(source.relayPtyId)
if (
source.sourceEndSu <= source.sourceStartSu ||
(current &&
(current.deliveryToken !== source.deliveryToken ||
current.clientGeneration !== source.clientGeneration ||
current.ownerGeneration !== source.ownerGeneration ||
current.ptyIncarnation !== payload.ptyIncarnation ||
(current.nextSourceSu !== undefined && current.nextSourceSu !== source.sourceStartSu)))
) {
closeSshPtyOutputGeneration(
payload.providerGeneration,
'ssh_source_frame_stale_or_non_contiguous'
)
this.mux?.dispose('connection_lost')
return Promise.reject(new Error('ssh_source_frame_stale_or_non_contiguous'))
}
this.sourceIdentityByRelayPtyId.set(source.relayPtyId, {
deliveryToken: source.deliveryToken,
clientGeneration: source.clientGeneration,
ownerGeneration: source.ownerGeneration,
ptyIncarnation: payload.ptyIncarnation,
nextSourceSu: source.sourceEndSu
})
}
const rawLength = payload.sequenceChars ?? payload.data.length
return acceptSshPtyOutputData({
id: payload.id,
data: payload.data,
providerGeneration: payload.providerGeneration,
ptyIncarnation: payload.ptyIncarnation,
rawLength,
transformed: payload.transformed === true,
...(typeof payload.seq === 'number' ? { sequence: payload.seq } : {}),
...(source ? { source } : {})
})
}
private quarantineReattachData(pending: PendingPtyReattach, payload: SshPtyDataPayload): void {
this.observePrivateRecoveryFrame(pending, payload)
if (pending.restoreRequired) {
return
}
const sourceSu = payload.source
? payload.source.sourceEndSu - payload.source.sourceStartSu
: (payload.sequenceChars ?? payload.data.length)
if (!this.ptyRecoveryRetention.tryRetain(pending.retentionKey, payload.data, sourceSu)) {
pending.restoreRequired = 'recoveryQuarantineCapacityExceeded'
this.wakeRecovery(pending)
return
}
this.routeQuarantinedReattachData(pending, payload)
}
private routeQuarantinedReattachData(
pending: PendingPtyReattach,
payload: SshPtyDataPayload
): void {
this.observePrivateRecoveryFrame(pending, payload)
if (!pending.recovery) {
pending.queuedData.push(payload)
return
}
if (
pending.recoveryComplete &&
pending.nextRecoverySourceSu === pending.recovery.recoveryEndSu
) {
pending.liveData.push(payload)
return
}
this.admitRecoveryData(pending, payload)
}
private observePrivateRecoveryFrame(
pending: PendingPtyReattach,
payload: SshPtyDataPayload
): void {
const recovery = pending.recovery
if (
recovery &&
payload.source?.deliveryToken === recovery.deliveryToken &&
payload.source.clientGeneration === recovery.clientGeneration &&
payload.source.ownerGeneration === recovery.ownerGeneration &&
payload.ptyIncarnation === recovery.ptyIncarnation
) {
pending.highestRecoverySourceEndSu = Math.max(
pending.highestRecoverySourceEndSu ?? recovery.checkpointSourceEndSu,
payload.source.sourceEndSu
)
}
}
private admitRecoveryData(pending: PendingPtyReattach, payload: SshPtyDataPayload): void {
if (pending.restoreRequired) {
return
}
const recovery = pending.recovery
const nextSourceSu = pending.nextRecoverySourceSu
if (
!recovery ||
!payload.source ||
nextSourceSu === undefined ||
payload.source.deliveryToken !== recovery.deliveryToken ||
payload.source.clientGeneration !== recovery.clientGeneration ||
payload.source.ownerGeneration !== recovery.ownerGeneration ||
payload.source.sourceStartSu !== nextSourceSu ||
payload.source.sourceEndSu <= payload.source.sourceStartSu ||
payload.source.sourceEndSu > recovery.recoveryEndSu ||
payload.ptyIncarnation !== recovery.ptyIncarnation
) {
pending.restoreRequired = 'recoveryFrameIdentityMismatch'
this.wakeRecovery(pending)
return
}
pending.nextRecoverySourceSu = payload.source.sourceEndSu
pending.recoveryData.push(payload)
}
private installPtyRecoveryNotifications(mux: SshChannelMultiplexer): void {
for (const cleanup of this.ptyRecoveryNotificationCleanups) {
cleanup()
}
this.ptyRecoveryNotificationCleanups = [
mux.onNotificationByMethod('pty.recoveryComplete', (params) => {
if (this.mux !== mux) {
return
}
const id = typeof params.id === 'string' ? toAppSshPtyId(this.targetId, params.id) : ''
const pending = this.pendingPtyReattaches.get(id)
if (!pending || pending.mux !== mux) {
return
}
const complete = parseRecoveryComplete(params)
if (!complete) {
pending.restoreRequired = 'invalidRecoveryComplete'
} else {
pending.recoveryComplete = complete
}
this.wakeRecovery(pending)
}),
mux.onNotificationByMethod('pty.restoreRequired', (params) => {
if (this.mux !== mux) {
return
}
const id = typeof params.id === 'string' ? toAppSshPtyId(this.targetId, params.id) : ''
const pending = this.pendingPtyReattaches.get(id)
if (!pending || pending.mux !== mux) {
return
}
pending.restoreRequired =
typeof params.reason === 'string' ? params.reason : 'relayRestoreRequired'
this.wakeRecovery(pending)
}),
mux.onNotificationByMethod('pty.deliveryCanceled', (params) => {
if (this.mux !== mux) {
return
}
const id = typeof params.id === 'string' ? params.id : ''
const identity = this.sourceIdentityByRelayPtyId.get(id)
if (
!identity ||
params.deliveryToken !== identity.deliveryToken ||
params.clientGeneration !== identity.clientGeneration ||
params.ownerGeneration !== identity.ownerGeneration ||
params.ptyIncarnation !== identity.ptyIncarnation
) {
return
}
const replacementDeliveryToken =
typeof params.replacementDeliveryToken === 'string' ? params.replacementDeliveryToken : ''
const pending = this.pendingPtyReattaches.get(toAppSshPtyId(this.targetId, id))
if (pending?.mux === mux) {
if (
replacementDeliveryToken.length === 0 ||
replacementDeliveryToken === identity.deliveryToken
) {
pending.restoreRequired =
typeof params.reason === 'string'
? `relayDeliveryCanceled:${params.reason}`
: 'relayDeliveryCanceled'
this.wakeRecovery(pending)
return
}
if (
pending.replacementDeliveryToken &&
pending.replacementDeliveryToken !== replacementDeliveryToken
) {
pending.restoreRequired = 'recoveryReplacementTokenMismatch'
this.wakeRecovery(pending)
return
}
pending.replacementDeliveryToken = replacementDeliveryToken
return
}
const generation = this.activePtyProviderGeneration
if (
generation !== null &&
Number.isSafeInteger(params.sentEndSu) &&
Number.isSafeInteger(params.creditedEndSu)
) {
try {
applySshPtySourceCancellationProof(
{
id: toAppSshPtyId(this.targetId, id),
code: -1,
providerGeneration: generation,
ptyIncarnation: identity.ptyIncarnation
},
{
sentEndSu: params.sentEndSu as number,
creditedEndSu: params.creditedEndSu as number
}
)
this.retiredSourceDeliveries.retire(generation, {
relayPtyId: id,
...identity
})
this.sourceIdentityByRelayPtyId.delete(id)
} catch {
/* Invalid proof retains the active token identity. */
}
}
})
]
}
private wakeRecovery(pending: PendingPtyReattach): void {
for (const resolve of pending.recoveryWaiters) {
resolve()
}
pending.recoveryWaiters.clear()
}
private async acceptPtyExit(payload: SshPtyExitPayload): Promise<void> {
await acceptSshPtyOutputExit({
id: payload.id,
code: payload.code,
providerGeneration: payload.providerGeneration,
ptyIncarnation: payload.ptyIncarnation
})
if (isCurrentPtyExit(payload)) {
this.retireExitedPty(payload, true)
}
}
private retireExitedPty(payload: SshPtyExitPayload, deliveryHandled = false): void {
const relayPtyId = toRelaySshPtyId(this.targetId, payload.id)
this.retiredSourceDeliveries.activate(relayPtyId)
clearProviderPtyState(payload.id)
deletePtyOwnership(payload.id)
getSshPtyConsumerRecovery(this.targetId)?.checkpointsByAppPtyId.delete(payload.id)
getSshPtyConsumerRecovery(this.targetId)?.checkpointsByAppPtyId.delete(
toRelaySshPtyId(this.targetId, payload.id)
)
this.store.markSshRemotePtyLease(this.targetId, relayPtyId, 'terminated')
if (deliveryHandled) {
return
}
this.runtime?.onPtyExit(payload.id, payload.code, payload.incarnationId)
const win = this.getMainWindow()
if (win && !win.isDestroyed()) {
win.webContents.send('pty:exit', payload)
}
}
private forwardReattachReplay(appPtyId: string, data: string): void {
if (!data) {
return
}
const win = this.getMainWindow()
if (win && !win.isDestroyed()) {
win.webContents.send('pty:replay', { id: appPtyId, data })
}
}
private async reattachKnownPtys(
mux: SshChannelMultiplexer,
shouldContinue: () => boolean
): Promise<void> {
const activeLeases = this.store
.getSshRemotePtyLeases(this.targetId)
.filter((lease) => lease.state !== 'terminated' && lease.state !== 'expired')
const activeLeaseByPtyId = new Map(activeLeases.map((lease) => [lease.ptyId, lease]))
const leasedPtyIds = activeLeases.map((lease) => lease.ptyId)
// Why: pass pane identity so the relay can reject cross-generation id collisions; tabId falls back for pre-leafId leases.
const expectedIdentityByPtyId = new Map(
activeLeases
.map((lease): [string, ExpectedPtyIdentity] | null => {
const expected = expectedIdentityForLease(lease)
return expected ? [lease.ptyId, expected] : null
})
.filter((entry): entry is [string, ExpectedPtyIdentity] => entry !== null)
)
// Why: after app restart ptyOwnership is empty, but durable SSH leases still describe grace-window survivors.
const ptyIds = Array.from(
new Set([
...getPtyIdsForConnection(this.targetId).map((ptyId) =>
toRelaySshPtyId(this.targetId, ptyId)
),
...leasedPtyIds
])
)
const ptyProvider = getSshPtyProvider(this.targetId) as SshPtyProvider | undefined
const providerGeneration = this.activePtyProviderGeneration
if (!ptyProvider || providerGeneration === null || this.mux !== mux) {
return
}
let nextPtyIndex = 0
const worker = async (): Promise<void> => {
while (shouldContinue()) {
const ptyId = ptyIds[nextPtyIndex++]
if (ptyId === undefined) {
return
}
try {
await this.reattachKnownPty({
ptyProvider,
ptyId,
activeLeaseByPtyId,
expectedIdentityByPtyId,
mux,
providerGeneration,
shouldContinue
})
} catch (error) {
if (isSourceRecoveryCancellationError(error)) {
throw error
}
console.warn(
`[ssh-relay-session] PTY ${ptyId} reattach processing failed for ${this.targetId}: ${
error instanceof Error ? error.message : String(error)
}`
)
}
}
}
await Promise.all(
Array.from({ length: Math.min(SSH_PTY_REATTACH_MAX_CONCURRENCY, ptyIds.length) }, worker)
)
}
private async reattachKnownPty(args: {
ptyProvider: SshPtyProvider
ptyId: string
activeLeaseByPtyId: Map<string, SshPtyLease>
expectedIdentityByPtyId: Map<string, ExpectedPtyIdentity>
mux: SshChannelMultiplexer
providerGeneration: number
shouldContinue: () => boolean
}): Promise<void> {
const {
ptyProvider,
ptyId,
activeLeaseByPtyId,
expectedIdentityByPtyId,
mux,
providerGeneration,
shouldContinue
} = args
const appPtyId = toAppSshPtyId(this.targetId, ptyId)
const pendingReattach: PendingPtyReattach = {
mux,
providerGeneration,
retentionKey: `${providerGeneration}\0${appPtyId}\0${randomUUID()}`,
exits: [],
queuedData: [],
recoveryData: [],
liveData: [],
recoveryWaiters: new Set(),
livePassthrough: false,
activated: false
}
this.pendingPtyReattaches.set(appPtyId, pendingReattach)
let sourceActivationLease: SshPtyAttachResult['sourceActivationLease']
let recoveryActivationLease: SshPtyRecoveryActivationLease | undefined
try {
const recoveryRequest = await this.sourceRecoveryRequest(appPtyId)
const attachResult = await this.attachPtyWithRetry(
ptyProvider,
ptyId,
expectedIdentityByPtyId.get(ptyId),
recoveryRequest,
shouldContinue
)
sourceActivationLease = attachResult.sourceActivationLease
if (!shouldContinue()) {
return
}
const exitDuringAttach = pendingReattach.exits.find(
(exit) =>
!exit.incarnationId ||
!attachResult.incarnationId ||
exit.incarnationId === attachResult.incarnationId
)
if (exitDuringAttach && !recoveryRequest) {
if (attachResult.incarnationId) {
restorePtyIncarnation(appPtyId, attachResult.incarnationId)
this.runtime?.acceptPtyIncarnationForExit(appPtyId, attachResult.incarnationId)
}
await this.acceptPtyExit(exitDuringAttach)
return
}
if (recoveryRequest) {
const recovered = await this.finishSourceRecovery(
ptyId,
appPtyId,
attachResult,
recoveryRequest,
pendingReattach,
shouldContinue,
() => {
const lease = sourceActivationLease
if (!lease) {
return
}
recoveryActivationLease = lease.transferToRecovery((payload) =>
this.quarantineReattachData(pendingReattach, payload)
)
sourceActivationLease = undefined
}
)
if (!recovered) {
const recoveryExit = this.findExactPendingExit(
pendingReattach,
attachResult.incarnationId
)
if (
recoveryExit &&
shouldContinue() &&
this.ownsPtyRecoveryAttempt(appPtyId, pendingReattach)
) {
if (recoveryActivationLease) {
recoveryActivationLease.retire()
recoveryActivationLease = undefined
} else if (sourceActivationLease) {
const canceled = await sourceActivationLease.rollback()
sourceActivationLease = undefined
if (!canceled) {
throw sourceRecoveryCancellationError(
new Error('ssh_source_activation_cancellation_unproven')
)
}
}
this.preparePtyIncarnationForExit(appPtyId, attachResult.incarnationId)
await this.acceptPtyExit(recoveryExit)
}
return
}
const recoveryExit = this.findExactPendingExit(pendingReattach, attachResult.incarnationId)
if (recoveryExit) {
this.preparePtyIncarnationForExit(appPtyId, attachResult.incarnationId)
pendingReattach.activated = true
recoveryActivationLease?.commit()
recoveryActivationLease = undefined
await this.acceptPtyExit(recoveryExit)
return
}
}
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pendingReattach)) {
return
}
setPtyOwnership(appPtyId, this.targetId)
if (attachResult.incarnationId) {
restorePtyIncarnation(appPtyId, attachResult.incarnationId)
this.restoreReattachedPtyRuntime(
appPtyId,
attachResult.incarnationId,
activeLeaseByPtyId.get(ptyId)
)
}
this.store.markSshRemotePtyLease(this.targetId, ptyId, 'attached')
pendingReattach.activated = true
recoveryActivationLease?.commit()
recoveryActivationLease = undefined
const exitAfterActivation = pendingReattach.exits.find(
(exit) =>
!exit.incarnationId ||
!attachResult.incarnationId ||
exit.incarnationId === attachResult.incarnationId
)
if (exitAfterActivation) {
await this.acceptPtyExit(exitAfterActivation)
return
}
if (!recoveryRequest) {
this.forwardReattachReplay(appPtyId, attachResult.replay ?? '')
}
sourceActivationLease?.commit()
sourceActivationLease = undefined
} catch (error) {
if (isSourceRecoveryCancellationError(error)) {
throw error
}
if (!shouldContinue()) {
return
}
this.handlePtyReattachFailure(ptyId, appPtyId, pendingReattach, error)
} finally {
recoveryActivationLease?.retire()
sourceActivationLease?.rollback()
if (this.pendingPtyReattaches.get(appPtyId) === pendingReattach) {
this.pendingPtyReattaches.delete(appPtyId)
}
this.ptyRecoveryRetention.release(pendingReattach.retentionKey)
}
}
private findExactPendingExit(
pending: PendingPtyReattach,
ptyIncarnation: string | undefined
): SshPtyExitPayload | undefined {
if (!ptyIncarnation) {
return undefined
}
return pending.exits.find(
(exit) =>
exit.providerGeneration === pending.providerGeneration &&
exit.ptyIncarnation === ptyIncarnation
)
}
private preparePtyIncarnationForExit(appPtyId: string, ptyIncarnation: string | undefined): void {
if (!ptyIncarnation) {
return
}
restorePtyIncarnation(appPtyId, ptyIncarnation)
this.runtime?.acceptPtyIncarnationForExit(appPtyId, ptyIncarnation)
}
private restoreReattachedPtyRuntime(
appPtyId: string,
incarnationId: string,
lease: SshPtyLease | undefined
): void {
if (lease?.worktreeId && lease.tabId && lease.leafId) {
this.runtime?.registerPty(appPtyId, lease.worktreeId, this.targetId, {
tabId: lease.tabId,
leafId: lease.leafId,
incarnationId
})
try {
this.store.persistPtyBinding({
worktreeId: lease.worktreeId,
tabId: lease.tabId,
leafId: lease.leafId,
ptyId: appPtyId,
incarnationId
})
} catch (error) {
console.error('[ssh-relay-session] Failed to persist reconnect incarnation:', error)
}
return
}
this.runtime?.onPtySpawned(appPtyId, incarnationId, { awaitsRegistration: false })
}
private async attachPtyWithRetry(
ptyProvider: SshPtyProvider,
ptyId: string,
expectedIdentity: ExpectedPtyIdentity | undefined,
recoveryRequest: PtySourceRecoveryRequest | undefined,
shouldContinue: () => boolean
): Promise<SshPtyAttachResult> {
let lastError: unknown
for (let attempt = 0; attempt < 2; attempt++) {
if (!shouldContinue()) {
throw lastError ?? new Error('PTY reattach attempt is no longer current')
}
try {
return await this.attachPtyWithDeadline(
ptyProvider,
ptyId,
expectedIdentity,
recoveryRequest
)
} catch (error) {
lastError = error
if (!shouldContinue() || isSshPtyNotFoundError(error) || attempt === 1) {
throw error
}
await this.waitForPtyReattachRetry()
}
}
throw lastError
}
private async attachPtyWithDeadline(
ptyProvider: SshPtyProvider,
ptyId: string,
expectedIdentity: ExpectedPtyIdentity | undefined,
recoveryRequest: PtySourceRecoveryRequest | undefined
): Promise<SshPtyAttachResult> {
let timer: ReturnType<typeof setTimeout> | undefined
let timedOut = false
const timeout = new Promise<never>((_, reject) => {
timer = setTimeout(() => {
timedOut = true
reject(
new Error(`PTY reattach attempt timed out after ${SSH_PTY_REATTACH_ATTEMPT_TIMEOUT_MS}ms`)
)
}, SSH_PTY_REATTACH_ATTEMPT_TIMEOUT_MS)
timer.unref?.()
})
try {
const attach = expectedIdentity
? recoveryRequest
? ptyProvider.attachForReconnect(ptyId, expectedIdentity, recoveryRequest)
: ptyProvider.attachForReconnect(ptyId, expectedIdentity)
: recoveryRequest
? ptyProvider.attachForReconnect(ptyId, undefined, recoveryRequest)
: ptyProvider.attachForReconnect(ptyId)
const guardedAttach = attach.then((result) => {
if (timedOut) {
result.sourceActivationLease?.rollback()
}
return result
})
return (await Promise.race([guardedAttach, timeout])) ?? {}
} finally {
if (timer) {
clearTimeout(timer)
}
}
}
private async waitForPtyReattachRetry(): Promise<void> {
const delayMs =
SSH_PTY_REATTACH_RETRY_MIN_DELAY_MS +
Math.floor(Math.random() * (SSH_PTY_REATTACH_RETRY_JITTER_MS + 1))
await new Promise<void>((resolve) => {
const timer = setTimeout(resolve, delayMs)
timer.unref?.()
})
}
private handlePtyReattachFailure(
ptyId: string,
appPtyId: string,
pending: PendingPtyReattach,
error: unknown
): void {
if (!isSshPtyNotFoundError(error)) {
pending.restoreRequired = 'reattachAttemptsExhausted'
this.wakeRecovery(pending)
console.warn(
`[ssh-relay-session] Leaving PTY ${ptyId} detached for ${this.targetId} after bounded reattach attempts failed: ${
error instanceof Error ? error.message : String(error)
}`
)
return
}
if (isSshPtyIdentityMismatchError(error)) {
console.warn(
`[ssh-relay-session] Ignoring stale PTY ${ptyId} for ${this.targetId} after relay identity mismatch: ${
error instanceof Error ? error.message : String(error)
}`
)
return
}
console.warn(
`[ssh-relay-session] Dropping stale PTY ${ptyId} for ${this.targetId} after relay reattach failed: ${
error instanceof Error ? error.message : String(error)
}`
)
clearProviderPtyState(appPtyId)
deletePtyOwnership(appPtyId)
this.store.markSshRemotePtyLease(this.targetId, ptyId, 'expired')
const win = this.getMainWindow()
if (win && !win.isDestroyed()) {
win.webContents.send('pty:exit', { id: appPtyId, code: -1 })
}
}
private async sourceRecoveryRequest(
appPtyId: string
): Promise<PtySourceRecoveryRequest | undefined> {
if (!this.activePtyConsumerOwner()?.outputFlowControl) {
return undefined
}
const recovery = getSshPtyConsumerRecovery(this.targetId)
const migration = recovery?.modelMigrationsByAppPtyId.get(appPtyId)
if (migration) {
const outcome = await migration
if (recovery?.modelMigrationsByAppPtyId.get(appPtyId) === migration) {
recovery.modelMigrationsByAppPtyId.delete(appPtyId)
}
if (outcome.status !== 'settled') {
return Object.freeze({ status: 'checkpointUnavailable' })
}
}
const checkpoints = recovery?.checkpointsByAppPtyId
const relayPtyId = toRelaySshPtyId(this.targetId, appPtyId)
// Why: every checkpoint writer records app-id keys now, so the relay-id
// lookup (and its paired delete below) is a legacy guard only.
const checkpoint = checkpoints?.get(appPtyId) ?? checkpoints?.get(relayPtyId)
if (!checkpoint) {
return Object.freeze({ status: 'checkpointUnavailable' })
}
return Object.freeze({
status: 'checkpoint',
clientGeneration: checkpoint.clientGeneration,
ownerGeneration: checkpoint.ownerGeneration,
ptyIncarnation: checkpoint.ptyIncarnation,
deliveryToken: checkpoint.deliveryToken,
acceptedSourceEndSu: checkpoint.acceptedSourceEndSu
})
}
private beginPtyModelMigration(providerGeneration: number, closeReason: string): void {
const recovery = getSshPtyConsumerRecovery(this.targetId)
if (!recovery) {
closeSshPtyOutputGeneration(providerGeneration, closeReason)
return
}
for (const checkpoint of getSshPtyAcceptedSourceCheckpoints(providerGeneration)) {
recovery.checkpointsByAppPtyId.set(checkpoint.id, checkpoint)
}
const migration = beginSshPtyOutputGenerationMigration(providerGeneration)
for (const [ptyId, result] of migration.byPty) {
const previous = recovery.modelMigrationsByAppPtyId.get(ptyId)
const fence = previous ? previous.then(() => result) : result
recovery.modelMigrationsByAppPtyId.set(ptyId, fence)
void fence.then((outcome) => {
const current = getSshPtyConsumerRecovery(this.targetId)
if (current?.modelMigrationsByAppPtyId.get(ptyId) !== fence) {
return
}
if (outcome.status === 'settled') {
current.checkpointsByAppPtyId.set(ptyId, outcome.checkpoint)
} else {
current.checkpointsByAppPtyId.delete(ptyId)
current.checkpointsByAppPtyId.delete(toRelaySshPtyId(this.targetId, ptyId))
}
current.modelMigrationsByAppPtyId.delete(ptyId)
})
}
void migration.completion.then(() => {
closeSshPtyOutputGeneration(providerGeneration, closeReason)
})
}
private async finishSourceRecovery(
relayPtyId: string,
appPtyId: string,
attachResult: SshPtyAttachResult,
request: PtySourceRecoveryRequest,
pending: PendingPtyReattach,
shouldContinue: () => boolean,
activateRecoveryQuarantine: () => void
): Promise<boolean> {
const recovery = attachResult.sourceRecovery
const pendingRecovery = recovery?.status === 'pending' ? recovery : undefined
const owner = this.activePtyConsumerOwner()
if (
!owner?.outputFlowControl ||
!pendingRecovery ||
request.status !== 'checkpoint' ||
pendingRecovery.clientGeneration !== owner.clientGeneration ||
pendingRecovery.ownerGeneration !== owner.ownerGeneration ||
pendingRecovery.ptyIncarnation !== attachResult.incarnationId ||
pendingRecovery.ptyIncarnation !== request.ptyIncarnation ||
pendingRecovery.checkpointSourceEndSu !== request.acceptedSourceEndSu ||
(pending.replacementDeliveryToken !== undefined &&
pending.replacementDeliveryToken !== pendingRecovery.deliveryToken)
) {
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return false
}
await this.abandonPtySourceRecovery(relayPtyId, appPtyId, pending)
return false
}
const acceptedRecovery = pendingRecovery
pending.recovery = acceptedRecovery
pending.nextRecoverySourceSu = acceptedRecovery.checkpointSourceEndSu
this.retiredSourceDeliveries.activate(relayPtyId)
this.sourceIdentityByRelayPtyId.set(relayPtyId, {
deliveryToken: acceptedRecovery.deliveryToken,
clientGeneration: acceptedRecovery.clientGeneration,
ownerGeneration: acceptedRecovery.ownerGeneration,
ptyIncarnation: acceptedRecovery.ptyIncarnation,
nextSourceSu: acceptedRecovery.checkpointSourceEndSu
})
activateRecoveryQuarantine()
for (const payload of pending.queuedData.splice(0)) {
this.routeQuarantinedReattachData(pending, payload)
}
await this.waitForRecoveryFence(pending, shouldContinue)
const exactExit = this.findExactPendingExit(pending, acceptedRecovery.ptyIncarnation)
const complete = pending.recoveryComplete ?? (exactExit ? acceptedRecovery : undefined)
if (
!shouldContinue() ||
pending.restoreRequired ||
!complete ||
complete.deliveryToken !== acceptedRecovery.deliveryToken ||
complete.clientGeneration !== acceptedRecovery.clientGeneration ||
complete.ownerGeneration !== acceptedRecovery.ownerGeneration ||
complete.ptyIncarnation !== acceptedRecovery.ptyIncarnation ||
complete.checkpointSourceEndSu !== acceptedRecovery.checkpointSourceEndSu ||
complete.recoveryEndSu !== acceptedRecovery.recoveryEndSu ||
pending.nextRecoverySourceSu !== acceptedRecovery.recoveryEndSu
) {
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return false
}
await this.abandonPtySourceRecovery(relayPtyId, appPtyId, pending)
return false
}
let nextLiveSourceSu = acceptedRecovery.recoveryEndSu
for (const payload of pending.liveData) {
if (
!payload.source ||
payload.source.deliveryToken !== acceptedRecovery.deliveryToken ||
payload.source.clientGeneration !== acceptedRecovery.clientGeneration ||
payload.source.ownerGeneration !== acceptedRecovery.ownerGeneration ||
payload.source.sourceStartSu !== nextLiveSourceSu ||
payload.source.sourceEndSu <= payload.source.sourceStartSu ||
payload.ptyIncarnation !== acceptedRecovery.ptyIncarnation
) {
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return false
}
await this.abandonPtySourceRecovery(relayPtyId, appPtyId, pending)
return false
}
nextLiveSourceSu = payload.source.sourceEndSu
}
try {
for (const payload of pending.recoveryData) {
await this.acceptPtyData(payload)
}
for (const payload of pending.liveData) {
await this.acceptPtyData(payload)
}
pending.livePassthrough = true
} catch {
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return false
}
await this.abandonPtySourceRecovery(relayPtyId, appPtyId, pending)
return false
}
if (!shouldContinue() || !this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return false
}
const acceptedSourceEndSu = pending.liveData.reduce(
(endSu, payload) => Math.max(endSu, payload.source?.sourceEndSu ?? endSu),
acceptedRecovery.recoveryEndSu
)
// Why: checkpoints are app-id keyed; a relay-id entry here would be shadowed
// by a staler app-id entry on the next sourceRecoveryRequest lookup.
getSshPtyConsumerRecovery(this.targetId)?.checkpointsByAppPtyId.set(
appPtyId,
Object.freeze({
id: appPtyId,
providerGeneration: this.activePtyProviderGeneration!,
clientGeneration: acceptedRecovery.clientGeneration,
ownerGeneration: acceptedRecovery.ownerGeneration,
ptyIncarnation: acceptedRecovery.ptyIncarnation,
deliveryToken: acceptedRecovery.deliveryToken,
acceptedSourceEndSu
})
)
return true
}
private async waitForRecoveryFence(
pending: PendingPtyReattach,
shouldContinue: () => boolean
): Promise<void> {
const deadline = Date.now() + SSH_PTY_REATTACH_ATTEMPT_TIMEOUT_MS
while (
shouldContinue() &&
!pending.recoveryComplete &&
!pending.restoreRequired &&
!this.findExactPendingExit(pending, pending.recovery?.ptyIncarnation) &&
Date.now() < deadline
) {
await new Promise<void>((resolve) => {
const timer = setTimeout(
() => {
pending.recoveryWaiters.delete(settle)
resolve()
},
Math.max(1, deadline - Date.now())
)
timer.unref?.()
const settle = (): void => {
clearTimeout(timer)
resolve()
}
pending.recoveryWaiters.add(settle)
})
}
if (
!pending.recoveryComplete &&
!pending.restoreRequired &&
!this.findExactPendingExit(pending, pending.recovery?.ptyIncarnation)
) {
pending.restoreRequired = 'recoveryFenceTimeout'
}
}
private async abandonPtySourceRecovery(
relayPtyId: string,
appPtyId: string,
pending: PendingPtyReattach
): Promise<void> {
if (!this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return
}
const recovery = pending.recovery
const { mux, providerGeneration } = pending
if (recovery && !mux.isDisposed()) {
const cancellationRequest = {
id: relayPtyId,
clientGeneration: recovery.clientGeneration,
ownerGeneration: recovery.ownerGeneration,
deliveryToken: recovery.deliveryToken
}
this.retiredSourceDeliveries.retire(providerGeneration, {
relayPtyId,
deliveryToken: recovery.deliveryToken,
clientGeneration: recovery.clientGeneration,
ownerGeneration: recovery.ownerGeneration
})
try {
const result = (await mux.request('pty.cancelDelivery', cancellationRequest)) as Record<
string,
unknown
>
const highestPrivateSourceEndSu = pending.liveData.reduce(
(endSu, payload) => Math.max(endSu, payload.source?.sourceEndSu ?? endSu),
Math.max(
pending.nextRecoverySourceSu ?? recovery.checkpointSourceEndSu,
pending.highestRecoverySourceEndSu ?? recovery.checkpointSourceEndSu
)
)
if (
result.canceled !== true ||
!Number.isSafeInteger(result.sentEndSu) ||
(result.sentEndSu as number) < highestPrivateSourceEndSu ||
!Number.isSafeInteger(result.creditedEndSu) ||
result.creditedEndSu !== recovery.checkpointSourceEndSu ||
(result.creditedEndSu as number) > (result.sentEndSu as number)
) {
throw new Error('ssh_source_cancellation_proof_invalid')
}
if (!this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return
}
const identity = this.sourceIdentityByRelayPtyId.get(relayPtyId)
if (identity && !this.sameSourceDelivery(identity, recovery)) {
return
}
if (identity) {
const applied = applySshPtySourceRecoveryCancellationProof(
{
id: appPtyId,
code: -1,
providerGeneration,
ptyIncarnation: recovery.ptyIncarnation
},
{
sentEndSu: result.sentEndSu as number,
creditedEndSu: result.creditedEndSu as number
}
)
if (!applied) {
throw new Error('ssh_source_cancellation_proof_rejected')
}
}
} catch (error) {
if (!this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return
}
console.warn(
`[ssh-relay-session] Failed to cancel replacement delivery for ${relayPtyId}: ${
error instanceof Error ? error.message : String(error)
}`
)
throw sourceRecoveryCancellationError(error)
}
}
if (!this.ownsPtyRecoveryAttempt(appPtyId, pending)) {
return
}
const identity = this.sourceIdentityByRelayPtyId.get(relayPtyId)
if (!identity || !recovery || this.sameSourceDelivery(identity, recovery)) {
this.sourceIdentityByRelayPtyId.delete(relayPtyId)
}
getSshPtyConsumerRecovery(this.targetId)?.checkpointsByAppPtyId.delete(appPtyId)
getSshPtyConsumerRecovery(this.targetId)?.checkpointsByAppPtyId.delete(relayPtyId)
this.store.markSshRemotePtyLease(this.targetId, relayPtyId, 'detached')
}
private ownsPtyRecoveryAttempt(appPtyId: string, pending: PendingPtyReattach): boolean {
return (
this.pendingPtyReattaches.get(appPtyId) === pending &&
this.mux === pending.mux &&
this.activePtyProviderGeneration === pending.providerGeneration &&
!pending.mux.isDisposed()
)
}
private sameSourceDelivery(
left: Readonly<{
deliveryToken: string
clientGeneration: number
ownerGeneration: number
ptyIncarnation: string
}>,
right: Readonly<{
deliveryToken: string
clientGeneration: number
ownerGeneration: number
ptyIncarnation: string
}>
): boolean {
return (
left.deliveryToken === right.deliveryToken &&
left.clientGeneration === right.clientGeneration &&
left.ownerGeneration === right.ownerGeneration &&
left.ptyIncarnation === right.ptyIncarnation
)
}
}