mirror of
https://github.com/stablyai/orca.git
synced 2026-10-22 08:03:37 +00:00
Jinjing
ce5b639e03
* fix(P1-B): recover system-SSH targets after a network drop
Two defects stopped a remote workspace auto-recovering after a blip.
runReconnectAttempt classified failures with isTransientError, which only
matches ETIMEDOUT/ECONNREFUSED/ECONNRESET by errno code or literal
substring. The system-SSH transport — the only transport FIDO2 and
ProxyUseFdpass targets can use — reports network failures as OpenSSH
prose ("System SSH connection timed out"), so the ladder published a
permanent 'error' on the first timeout and the target never came back
without a manual reconnect. isTransientReconnectError adds a
network-shaped prose table on top of isTransientError and is used only on
the reconnect path: connect() keeps the narrow classifier so an
unreachable host still fails fast instead of burning five 30s attempts
and five security-key touch prompts. Auth and passphrase failures stay
permanent on both paths.
runReconnectAttempt also had no generation fence, so a superseded attempt
published its cancellation as a permanent error over the winner's live
connection — reachable when a system-transport proc.onExit schedules a
reconnect while an attempt is still in flight. Cancellation now carries a
stable error name, and both connect() and runReconnectAttempt claim their
connectGeneration and stay silent when a newer attempt owns the state.
* fix(P1-B): retry a dropped watcher overflow marker on real capacity
emitWatcherOverflowToClient published the {kind:'overflow'} resync marker
with controlOverflow:'reject'. A full control queue rejects at admission
with no settlement callback, so the marker was silently discarded and the
remote File Explorer stayed stale until some later watcher event happened
to produce another one — for a quiet tree, possibly never.
The emitter now retains a rejected marker per (client, root) and
republishes it when the sink actually frees up. The existing
onLegacyPtyCapacity signal cannot drive that: it is gated on producer
retention, so it stays silent exactly under the dual-queue pressure that
caused the rejection. RelayDispatcher.onClientCapacity is an ungated
per-client capacity signal that fires on every writer settlement and
drain. It lives on the dispatcher rather than the writer so a retained
marker survives setWrite() replacing the primary sink, and setWrite
notifies capacity once afterwards so the marker does not wait on traffic
that may never arrive.
Retention is bounded to one marker per (client, root), released on
settlement and purged on client detach.
* fix(P1-B): address all review findings on SSH network recovery
Fix four issues from code review:
1. **Bug — admitted overflow markers lost on setWrite**: Retain markers when
settlement fails `ok: false`, not just on admission rejection. Prevents
desynced filesystem trees after SSH sink replacement.
2. **SSH error classification expanded**: Add missing OpenSSH patterns
(`ssh_exchange_identification`, `connection closed by remote`) and new
`isDefiniteSystemSshHostFailure()` classifier.
3. **ControlMaster retry optimization**: Skip second probe when first failure is
already definite host-level (network timeout, refused, unreachable). Saves
~30s per reconnect ladder step.
4. **Overflow flush under dual-queue pressure**: Gate pending marker retries on
control-lane headroom instead of re-attempting on every capacity notification.
Reduces thrash proportional to producer traffic.
Add regression tests for marker republish on sink replacement and validate auth
error detection against live OpenSSH credential rejection messages.
* rm random doc
* fix(P1-B): skip credential-failure retries and recover watcher markers o
- Auth and passphrase errors fail immediately without retry attempts
- Bare "System SSH probe failed (exit 255)" is transient only for reconnect
- Watcher markers survive client invalidation when switching SSH connections
- Add network error patterns: "lost connection", "remote end closed"
84 lines
3.0 KiB
TypeScript
84 lines
3.0 KiB
TypeScript
import { isAuthError, isPassphraseError, isTransientError } from './ssh-connection-utils'
|
|
|
|
// Why: the system-SSH transport reports network failures as OpenSSH prose, not errno codes, so its
|
|
// probe timeouts and connect failures never match isTransientError's code table.
|
|
const NETWORK_LIKE_ERROR_FRAGMENTS = [
|
|
'system ssh connection timed out',
|
|
'timed out while waiting for handshake',
|
|
'connection timed out',
|
|
'operation timed out',
|
|
'connection refused',
|
|
'connection reset',
|
|
'no route to host',
|
|
'network is unreachable',
|
|
'network is down',
|
|
'host is down',
|
|
'broken pipe',
|
|
'temporary failure in name resolution',
|
|
'name or service not known',
|
|
'nodename nor servname',
|
|
'could not resolve hostname',
|
|
'kex_exchange_identification',
|
|
// Pre-7.x OpenSSH wording for the same banner-exchange failure.
|
|
'ssh_exchange_identification',
|
|
'lost connection',
|
|
'remote end closed',
|
|
// Deliberately not the bare 'connection closed by': OpenSSH prints "Connection closed by <ip> port 22"
|
|
// for server-side rejections (MaxStartups, DenyUsers) too, and those must stay permanent.
|
|
'connection closed by remote'
|
|
]
|
|
|
|
const DEFINITE_HOST_FAILURE_FRAGMENTS = [
|
|
'no route to host',
|
|
'network is unreachable',
|
|
'network is down',
|
|
'host is down',
|
|
'temporary failure in name resolution',
|
|
'name or service not known',
|
|
'nodename nor servname',
|
|
'could not resolve hostname'
|
|
]
|
|
|
|
const DEFINITE_HOST_FAILURE_CODES = new Set(['EHOSTUNREACH', 'ENETUNREACH', 'EAI_AGAIN'])
|
|
|
|
const OPENSSH_HOST_CONNECT_FAILURE =
|
|
/\bconnect to host .+? port \d+: (?:connection (?:timed out|refused|reset(?: by peer)?)|operation timed out)\b/i
|
|
|
|
const EMPTY_SYSTEM_SSH_PROBE_EXIT_255 = /^system ssh probe failed \(exit 255\)\.$/
|
|
|
|
/**
|
|
* Why a second classifier: connect() spends isTransientError on INITIAL_RETRY_ATTEMPTS, so widening
|
|
* it would turn one 30s system-SSH timeout into ~160s of silence and five security-key touch
|
|
* prompts. The reconnect ladder has its own backoff and give-up bound, so it can afford to treat a
|
|
* network-shaped failure as recoverable. Auth stays permanent on both paths.
|
|
*/
|
|
export function isTransientReconnectError(err: Error): boolean {
|
|
if (isAuthError(err) || isPassphraseError(err)) {
|
|
return false
|
|
}
|
|
if (isTransientError(err)) {
|
|
return true
|
|
}
|
|
const message = err.message.toLowerCase()
|
|
return (
|
|
EMPTY_SYSTEM_SSH_PROBE_EXIT_255.test(message) ||
|
|
NETWORK_LIKE_ERROR_FRAGMENTS.some((fragment) => message.includes(fragment))
|
|
)
|
|
}
|
|
|
|
/** Definite host failures where a ControlMaster-free retry cannot help. */
|
|
export function isDefiniteSystemSshHostFailure(err: unknown): boolean {
|
|
if (!(err instanceof Error)) {
|
|
return false
|
|
}
|
|
const code = (err as NodeJS.ErrnoException).code
|
|
if (code && DEFINITE_HOST_FAILURE_CODES.has(code)) {
|
|
return true
|
|
}
|
|
const message = err.message.toLowerCase()
|
|
return (
|
|
DEFINITE_HOST_FAILURE_FRAGMENTS.some((fragment) => message.includes(fragment)) ||
|
|
OPENSSH_HOST_CONNECT_FAILURE.test(err.message)
|
|
)
|
|
}
|