Commit Graph
4 Commits
Author SHA1 Message Date
Jinjing 9fb4dbe8eb fix(ssh): handle rejected PTY deliveries with targeted recovery (#12746)
Add targeted recovery for rejected PTY source frames instead of
terminating the relay channel. Classify rejection reasons (malformed,
generation mismatch, range invalid) and attempt recovery based on the
rejection type. Implement admission control at publication time to ensure
frames aren't delivered after ownership changes. Bound recovery attempts
and retry with backoff to prevent exhaustion. Diagnose and log rejection
reasons to aid debugging.
2026-08-05 10:30:04 -07:00
Jinwoo Hong eea0bb64db fix(ssh): make PTY owner admission explicit and non-destructive (#12673)
An owner-capable `pty.openClient` had two failure modes that presented as something else.

If the relay still held an owner record but the request carried no matching resume proof, admission fell through to a SUBSCRIBER grant — a success-shaped response the client cannot use, which it then rejected as "did not grant an authenticated PTY session owner". And if the relay had forgotten the record the client named, admission threw a stale-recovery error, which the client answered by deleting its own recovery row — `clientInstanceId` included — and reopening. Two round trips, and the identity that lets it resume that target at all went with the deletion.

Now every owner grant carries a required `resumed` flag, a forgotten record mints a fresh claim in one round trip, a held claim returns one of three coded refusals, duplicate opens on one connection are rejected even when identical, and an attached-holder refusal becomes a typed error routed through the terminal-relay-error callback instead of feeding redeploy backoff a link that is working fine.

Independent review caught two regressions in the first attempt, both now fixed and both with tests that fail without them:

**A backpressure teardown could take a live owner's session.** The safety argument was that a record only becomes `disconnected` from an observed peer close — but two of the six paths there are capacity paths, where the relay destroys the client's socket itself because its lane queue filled. That is the signature of a client that is ALIVE but not draining fast enough. Demonstrated: the real owner is torn down for backpressure, a rival is granted ownership 270ms into a nominal 30s grace, and the owner's later reconnect with a valid resume proof is refused permanently, backoff cleared, no retry. Closes now carry a cause (`peer-closed` | `local`, defaulting to `local`, which only ever widens a grace), and the floor applies only to closes the transport actually observed on the peer's side. Capacity teardowns, decode faults and sink failures keep the default.

**A client's own zombie connection blocked it permanently.** Only `SshRelaySession` ever requests owner, and every endpoint-credential client shares one principal — so in a normal single-app deployment an `active` incumbent refusing you is almost always your own half-open connection the relay never saw close. That was refused as terminal, where main recovered on bounded backoff once keepalive noticed. The refusal already held both client identities; a match is now a distinct transient refusal that falls through to relay-lost backoff, restoring that recovery. A genuinely different client is still blocked.

Also: each retry deadline now starts when its own phase begins, instead of both being computed at entry where a slow first phase could leave the second with zero attempts.

Fixes STA-3365.
2026-08-05 02:56:07 -07:00
Jinjing 00867f06e2 fix(ssh): handle owner displacement and graceful shutdown (#12367)
* fix(ssh): handle owner displacement and graceful shutdown

SSH connections can reconnect with valid session proof after network loss or
device sleep. When the incumbent owner is still half-open, allow the
reconnecting client to displace it outright rather than wait for socket closure
— a window that may never close. Retain displaced deliveries for the new owner
to rotate. During app shutdown, drain SSH sessions without terminating recovery
operations, and retry pending owner grants in case a replacement commits mid-drain.

* fix(ssh): handle owner displacement and graceful shutdown

Make QuitTeardownStartGate a shared singleton so SSH connects use the
same shutdown fence as the main quit path. Track test-connection probes
to ensure they complete before final teardown. Guard owner displacement
to prevent stale owners from clearing recovery state claimed by newer
owners.

* fix(ssh): fix flaky test sync and add error code safety check

Test was using tick-based Promise.resolve() loops which don't guarantee
the async operation has started. Replace with signal-based synchronization
that waits for the actual lease flush. Also add nullish-coalescing to
error code check to prevent crashes if error is null or undefined.

* fix(ssh): fence reset transport opens during shutdown

* fix(ssh): keep recovery leases stable across reconnects

* fix(ssh): close transports owned by cancelled connect attempts

When a connect is cancelled after its transport has opened, that cancelled
attempt still owns the transport and must close it — otherwise it leaks. Add
disconnectConnection() to close by identity (not by target ID) so a cancelled
attempt closes only the transport it minted, without tearing down its
replacement's live transport. Track priorConnection to detect whether this
attempt opened a new transport or reused an existing one, and close only on
abandonment if this attempt owns the session.

* fix(ssh): fence old owner proofs and close superseded transports

When an owner reconnects with a new proof while an old one is still live,
the old proof is now fenced with SUPERSEDED_ERROR instead of retrying
indefinitely. The relay also closes stale transports to signal that their
recovery generation has been overtaken by a newer one.

This ensures overlapping reconnect scenarios complete with the newest proof
rather than getting blocked by stale recovery attempts.

* fix(relay): re-pin stdin/stdout fds after closing to prevent recycling

When the relay closes stdin/stdout to signal EOF to the SSH peer, the OS
can recycle those fds (0 and 1) for new sockets or files. If Node still
treats process.stdin/stdout as those numbers, subsequent operations
corrupt socket clients and trigger shutdown errors. Re-pin the fds by
opening /dev/null to keep them occupied and prevent recycling.
2026-08-04 01:37:25 -07:00
JinjingandOrcaWin 5f7807497e feat(ssh): bound relay PTY output end to end (#11005)
* docs: design SSH relay PTY backpressure

* fix(ssh): bound relay frame decoding

* fix(relay): bound PTY output publication

* fix(ssh): bound PTY model admission

* fix(ssh): settle closed model admissions

* feat(ssh): negotiate bounded PTY consumer sessions

* fix(ssh): fence exit on renderer settlement

* feat(ssh): track PTY source credit end to end

* fix(ssh): recover bounded PTY output across reconnect

* feat(ssh): complete relay PTY output backpressure

* fix(ssh): close final PTY source credit races

* docs(ssh): record final backpressure validation

* feat(ssh): complete relay PTY source-credit lifecycle

* test(ssh): complete provider notification fixture

* fix(ssh): preserve terminal source credit across rotation

* fix(ssh): fail closed on recovery cancellation

* fix(ssh): prioritize mux control writes after drain

* fix(ssh): retire canceled relay restore deliveries

* fix(ssh): order exit cancellation cleanup

* fix(ssh): gate provisional source activation

* test(ssh): register mux drain-priority coverage

* fix(ssh): type stale owner recovery mismatches

* fix(ssh): close projection replacement races

* fix(relay): contain streaming edge failures

* fix(ssh): secure relay endpoint credentials

* docs(ssh): reconcile final backpressure lifecycle

* fix(ssh): bound main IPC output lifecycle

* fix(ssh): close recovery ownership gaps

* docs(ssh): record exact artifact validation

* fix(ssh): reject reclaimed snapshot replacements

* fix(ssh): fence model admission across reconnect

* fix(ssh): contain migration failure per PTY

* docs(ssh): record final exact-head validation

* test(ssh): align deploy fixtures with credential publication

* feat(ssh): add per-target bounded output setting

* fix(ssh): close source recovery review gaps

* fix(ssh): latch source credit environment override

* feat(ssh): make PTY source credit the default

* docs(ssh): record always-on relay validation

* docs(ssh): bind validation to current main

* test(ssh): grant source credit in IPC fixture

* test(ssh): grant source credit in fake relay

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-29 17:03:15 -07:00