mirror of
https://github.com/stablyai/orca.git
synced 2026-10-24 00:04:01 +00:00
Jinjing
ce5b639e03
* fix(P1-B): recover system-SSH targets after a network drop
Two defects stopped a remote workspace auto-recovering after a blip.
runReconnectAttempt classified failures with isTransientError, which only
matches ETIMEDOUT/ECONNREFUSED/ECONNRESET by errno code or literal
substring. The system-SSH transport — the only transport FIDO2 and
ProxyUseFdpass targets can use — reports network failures as OpenSSH
prose ("System SSH connection timed out"), so the ladder published a
permanent 'error' on the first timeout and the target never came back
without a manual reconnect. isTransientReconnectError adds a
network-shaped prose table on top of isTransientError and is used only on
the reconnect path: connect() keeps the narrow classifier so an
unreachable host still fails fast instead of burning five 30s attempts
and five security-key touch prompts. Auth and passphrase failures stay
permanent on both paths.
runReconnectAttempt also had no generation fence, so a superseded attempt
published its cancellation as a permanent error over the winner's live
connection — reachable when a system-transport proc.onExit schedules a
reconnect while an attempt is still in flight. Cancellation now carries a
stable error name, and both connect() and runReconnectAttempt claim their
connectGeneration and stay silent when a newer attempt owns the state.
* fix(P1-B): retry a dropped watcher overflow marker on real capacity
emitWatcherOverflowToClient published the {kind:'overflow'} resync marker
with controlOverflow:'reject'. A full control queue rejects at admission
with no settlement callback, so the marker was silently discarded and the
remote File Explorer stayed stale until some later watcher event happened
to produce another one — for a quiet tree, possibly never.
The emitter now retains a rejected marker per (client, root) and
republishes it when the sink actually frees up. The existing
onLegacyPtyCapacity signal cannot drive that: it is gated on producer
retention, so it stays silent exactly under the dual-queue pressure that
caused the rejection. RelayDispatcher.onClientCapacity is an ungated
per-client capacity signal that fires on every writer settlement and
drain. It lives on the dispatcher rather than the writer so a retained
marker survives setWrite() replacing the primary sink, and setWrite
notifies capacity once afterwards so the marker does not wait on traffic
that may never arrive.
Retention is bounded to one marker per (client, root), released on
settlement and purged on client detach.
* fix(P1-B): address all review findings on SSH network recovery
Fix four issues from code review:
1. **Bug — admitted overflow markers lost on setWrite**: Retain markers when
settlement fails `ok: false`, not just on admission rejection. Prevents
desynced filesystem trees after SSH sink replacement.
2. **SSH error classification expanded**: Add missing OpenSSH patterns
(`ssh_exchange_identification`, `connection closed by remote`) and new
`isDefiniteSystemSshHostFailure()` classifier.
3. **ControlMaster retry optimization**: Skip second probe when first failure is
already definite host-level (network timeout, refused, unreachable). Saves
~30s per reconnect ladder step.
4. **Overflow flush under dual-queue pressure**: Gate pending marker retries on
control-lane headroom instead of re-attempting on every capacity notification.
Reduces thrash proportional to producer traffic.
Add regression tests for marker republish on sink replacement and validate auth
error detection against live OpenSSH credential rejection messages.
* rm random doc
* fix(P1-B): skip credential-failure retries and recover watcher markers o
- Auth and passphrase errors fail immediately without retry attempts
- Bare "System SSH probe failed (exit 255)" is transient only for reconnect
- Watcher markers survive client invalidation when switching SSH connections
- Add network error patterns: "lost connection", "remote end closed"
95 lines
4.1 KiB
TypeScript
95 lines
4.1 KiB
TypeScript
import { describe, it, expect } from 'vitest'
|
|
import { isTransientError } from './ssh-connection-utils'
|
|
import {
|
|
isDefiniteSystemSshHostFailure,
|
|
isTransientReconnectError
|
|
} from './ssh-reconnect-error-classification'
|
|
|
|
describe('isTransientReconnectError', () => {
|
|
it('treats the system SSH connect timeout as recoverable', () => {
|
|
const err = new Error('System SSH connection timed out')
|
|
// Guards the split: widening isTransientError would spend 5 connect() attempts on this.
|
|
expect(isTransientError(err)).toBe(false)
|
|
expect(isTransientReconnectError(err)).toBe(true)
|
|
})
|
|
|
|
it.each([
|
|
'System SSH probe failed (exit 255).',
|
|
'System SSH probe failed (exit 255). stderr: ssh: connect to host box port 22: Connection refused',
|
|
'System SSH probe failed (exit 255). stderr: ssh: connect to host box port 22: No route to host',
|
|
'System SSH probe failed (exit 255). stderr: ssh: connect to host box port 22: Network is unreachable',
|
|
'System SSH probe failed (exit 255). stderr: kex_exchange_identification: read: Connection reset by peer',
|
|
'System SSH probe failed (exit 255). stderr: ssh: Could not resolve hostname box: Name or service not known',
|
|
'System SSH probe failed (exit 255). stderr: ssh_exchange_identification: read: Connection reset by peer',
|
|
'System SSH probe failed (exit 255). stderr: ssh_exchange_identification: Connection closed by remote host',
|
|
'System SSH probe failed (exit 255). stderr: Connection closed by remote host',
|
|
'System SSH probe failed (exit 255). stderr: Lost connection',
|
|
'System SSH probe failed (exit 255). stderr: Remote end closed connection'
|
|
])('treats OpenSSH network prose as recoverable: %s', (message) => {
|
|
expect(isTransientReconnectError(new Error(message))).toBe(true)
|
|
})
|
|
|
|
it('keeps the bare probe failure narrow to the reconnect classifier', () => {
|
|
const err = new Error('System SSH probe failed (exit 255).')
|
|
expect(isTransientError(err)).toBe(false)
|
|
expect(isTransientReconnectError(err)).toBe(true)
|
|
})
|
|
|
|
// A server-side rejection prints the same verb without "remote"; retrying it forever would hide a
|
|
// real misconfiguration behind the ladder.
|
|
it('keeps a bare "Connection closed by <host> port <n>" permanent', () => {
|
|
expect(
|
|
isTransientReconnectError(
|
|
new Error(
|
|
'System SSH probe failed (exit 255). stderr: Connection closed by 10.0.0.4 port 22'
|
|
)
|
|
)
|
|
).toBe(false)
|
|
})
|
|
|
|
it('keeps credential failures permanent', () => {
|
|
expect(
|
|
isTransientReconnectError(new Error('All configured authentication methods failed'))
|
|
).toBe(false)
|
|
expect(
|
|
isTransientReconnectError(
|
|
new Error('System SSH probe failed (exit 255). stderr: Permission denied (publickey).')
|
|
)
|
|
).toBe(false)
|
|
expect(
|
|
isTransientReconnectError(
|
|
new Error('Encrypted private OpenSSH key detected, but no passphrase given')
|
|
)
|
|
).toBe(false)
|
|
})
|
|
|
|
it.each([
|
|
'ssh: connect to host box port 22: Connection refused',
|
|
'ssh: connect to host box port 22: Connection reset by peer',
|
|
'ssh: connect to host box port 22: Connection timed out',
|
|
'ssh: connect to host box port 22: No route to host',
|
|
'ssh: Could not resolve hostname box: Name or service not known'
|
|
])('recognizes definite host failures for the ControlMaster retry gate: %s', (message) => {
|
|
expect(isDefiniteSystemSshHostFailure(new Error(message))).toBe(true)
|
|
})
|
|
|
|
it.each([
|
|
'System SSH connection timed out',
|
|
'Connection refused',
|
|
'Connection reset by peer',
|
|
'mux client failed: master is unresponsive'
|
|
])('keeps potentially mux-shaped failures eligible for direct retry: %s', (message) => {
|
|
expect(isDefiniteSystemSshHostFailure(new Error(message))).toBe(false)
|
|
})
|
|
|
|
it('still covers the errno codes isTransientError already matched', () => {
|
|
const err = new Error('refused') as NodeJS.ErrnoException
|
|
err.code = 'ECONNREFUSED'
|
|
expect(isTransientReconnectError(err)).toBe(true)
|
|
})
|
|
|
|
it('keeps unrelated failures permanent', () => {
|
|
expect(isTransientReconnectError(new Error('something went wrong'))).toBe(false)
|
|
})
|
|
})
|