feat(ssh): complete relay PTY output backpressure

This commit is contained in:
OrcaWin
2026-07-28 20:13:36 -07:00
parent 022856af3b
commit 1ca217eb56
63 changed files with 4421 additions and 1504 deletions
File diff suppressed because it is too large Load Diff
@@ -2,14 +2,16 @@
Date: 2026-07-27
Status: architecture reviewed; incremental implementation plan ready
Status: architecture reviewed; SSH V1 implementation complete behind an
experimental startup gate; final branch/CI validation pending
Baseline: `origin/main` at `0edc95fa35e4b6e403e116d4e224ad35329b0159`
after rebasing this design on 2026-07-27
Historical unbounded baseline: `badf91101babf96fa09cb79a8294f7e23b9f081c`
(the implementation branch parent). The final validation record is updated
after rebasing this PR onto current `origin/main`.
## Scope
This plan bounds SSH relay PTY output from the native PTY through the relay
This implemented design bounds SSH relay PTY output from the native PTY through the relay
dispatcher, SSH channel or relay socket, Electron main process, and renderer
parser. It covers:
@@ -25,6 +27,50 @@ work does not change PTY input semantics, terminal-model interpretation, or
file/Git payload semantics. It does change replay transport, SSH producer
pause wiring, and bulk frame admission where they share the dispatcher sink.
## Implemented architecture and evidence boundary
The architecture review approved one shared semantic
`PtyConsumerSession` state machine with an SSH-specific transport adapter.
It did not approve a universal wire protocol. In this PR:
- `PtyConsumerSession` owns authenticated client/owner generations, lease
recovery, capability intersection, and grant publication commit/rollback.
- `SshPtyConsumerSessionAdapter` binds those semantics to the existing
authenticated relay connection and the first framed `pty.openClient`
request. The relay dispatcher remains the only sink-publication authority.
- The relay source ledger owns immutable source spans, outstanding source
credit, token rotation, sealed exit, and cancellation proof. The dispatcher
writer owns byte admission, priorities, `write(false)`, callbacks, and drain.
- `SshRelaySession` owns grant/reconnect state, bounded recovery quarantine,
exact recovery fencing, and cumulative ACK publication through the main SSH
mux.
- The main SSH intake owns one atomic admission across the model, desktop
projection, and required remote consumers. Model completion and consumer
terminality make ACKs eligible; only the mux write callback publishes them.
- Desktop and remote replacement transactions carry immutable span identity.
Reserve happens before publication, commit requires the exact
generation/sequence fence, and failure or stale publication rolls back
without credit.
The implementation is intentionally scoped:
| Surface | State in this PR | Evidence |
| ------------------------------------------- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Same-build direct SSH/deployed relay, V1 on | implemented | deterministic relay/main contracts plus macOS-hosted Docker OpenSSH |
| Same-build SSH, V1 off | implemented bounded legacy mode | deterministic rollout and writer contracts |
| Reconnect/owner recovery | implemented | exact contiguous recovery, stale-owner one-shot retry, eight-wide reattach tests, Docker reconnect |
| Headed/headless remote consumers | source-range reserve/commit/rollback implemented | deterministic runtime/main seam only; no live paired-runtime claim |
| WSL stdio and Windows named pipe/ConPTY | shared transport paths preserved | deterministic/common-contract evidence only; no physical run |
| Local provider and local daemon | unchanged | outside SSH V1 negotiation; existing behavior only |
| Folder workspace | no `.git` dependency added | code-path review only; no dedicated live fixture |
| Prior-version/mixed-version peers | fail-closed or version-scoped legacy behavior preserved | deterministic rollout/deploy contracts; no live old binary |
| Ubuntu 20.04/glibc 2.31 | no native dependency added | cross-target relay build only; no physical packaging run |
Docker SSH proves the Linux SSH provider and deployed-relay topology. It does
not prove headed or headless paired-runtime behavior, WSL, Windows ConPTY or
named pipes, local daemon/provider behavior, folder workspaces, prior-version
processes, mixed-version clients, or the Ubuntu 20.04 packaging floor.
## Verified baseline and migration boundary
The SSH path is not currently protected by the main pending-data bound:
@@ -213,6 +259,20 @@ ship independently. A relay-only `pty.getCapabilities` followed by
`pty.negotiateClient` is rejected because it would create a second readiness
authority beside the existing authenticated handshakes.
The 2026-07-28 architecture/terminal review closed its three blocking
findings as follows:
| Finding | Implemented resolution |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Exited delivery removed while uncredited spans remain | Native exit seals the token; logical state survives until cumulative ACK, exact transfer, token-scoped cancellation proof, or generation close. |
| Terminal obligations depended circularly on ACK publication | Obligation terminality advances `obligationsTerminalEndSu`; the coalescer independently queues `ackQueuedEndSu`; only mux write settlement advances `ackPublishedEndSu`. |
| Desktop projection admission lacked immutable range identity and rollback | Model, source span, projection range, and scanner snapshot reserve atomically; pre-commit failure rolls back, while post-commit replacement transfers with proof. |
Later adversarial review also required exact recovery continuity, stale-attempt
isolation, startup-latched rollout semantics, and idempotent remote detach
after token reclamation. Those are incorporated in the activation, lifecycle,
cleanup, tests, and rollout sections below.
The common state machine accepts only:
- an authenticated principal and owner-eligibility decision from the adapter;
@@ -259,20 +319,17 @@ the semantic fields, and calls the fence.
The reviewed decision preserves the existing connection machinery:
| Path | Authentication and identity | Shared-session binding |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Local in-process provider | trusted main-process construction | instantiate semantics directly; no RPC or capability probe |
| Local daemon | token-authenticated `HelloMessage`, stable `clientId`, paired control/stream sockets, and `daemonIdentity` | control hello stores an optional offer; stream hello response publishes the grant after pairing |
| SSH relay socket/named pipe | private endpoint credential validated by `--connect` before its sentinel | first framed `pty.openClient` carries the optional offer; its response is the provider-readiness fence |
| Direct SSH stdio and WSL child stdio | launch-bound ephemeral nonce consumed by the relay adapter | first framed `pty.openClient` proves the nonce and carries the offer; its response is the readiness fence |
| Remote-runtime server | paired-device/E2EE identity and connection ID | adapt `terminal.multiplex` subscriptions; never issue SSH owner leases |
| Path | Authentication and identity | Binding in this PR |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| Local in-process provider | trusted main-process construction | unchanged; it does not negotiate SSH V1 |
| Local daemon | token-authenticated `HelloMessage`, stable `clientId`, paired control/stream sockets, and `daemonIdentity` | unchanged; an exact-version daemon remains outside the SSH wire contract |
| SSH relay socket/named pipe | existing authenticated relay endpoint and dispatcher client identity | first framed `pty.openClient`; response settlement is the readiness fence |
| Primary relay stdio and WSL child stdio | no authenticated dispatcher principal in this PR | bounded legacy transport only; an unproved client cannot become session owner |
| Remote-runtime server | paired-device/E2EE identity and connection ID | source-range reserve/commit/rollback only; never receives an SSH owner lease |
Current direct stdio and WSL have a one-way sentinel, not a bidirectional
pre-dispatch handshake. The reviewed SSH binding therefore chooses one
mandatory post-sentinel `pty.openClient` request for every SSH adapter; there
is no remaining readiness-vs-request choice. After extracting the exact
sentinel and residue, main sends this as the first framed request and prohibits
spawn, attach, or ordinary dispatch until its response validates. The relay
The implemented owner path is the authenticated detached relay endpoint used
by the SSH provider. After extracting the exact sentinel and residue, main
sends `pty.openClient` before spawn, attach, or ordinary dispatch. The relay
adapter queues later decoded frames behind the session response's write
callback, so an eager next request cannot race activation. Main marks the
provider ready only after receiving the valid response.
@@ -281,29 +338,24 @@ Detached `--connect` first reads the version-scoped endpoint credential and
authenticates its Unix-socket/named-pipe bridge before emitting its sentinel.
The adapter converts that proof into a principal and
`allowSessionOwner = true`; `pty.openClient` carries no secret to the generic
RPC handler. Direct ssh2, system SSH, and WSL launches receive a safely quoted
one-use nonce from their existing platform-specific command builder. The
pre-dispatch adapter consumes and strips the nonce from `pty.openClient`; a
manual relay without valid proof is subscriber-only. The nonce expires on the
first accepted open or process exit.
RPC handler. The primary relay stdin/stdout dispatcher identity remains
explicitly unproved and therefore subscriber-only; there is no launch nonce in
this implementation. WSL child stdio shares the bounded decoder/writer
transport changes but does not negotiate an SSH V1 owner in this PR. A manual
relay without the version-scoped endpoint credential is also subscriber-only.
The review compared sentinel extension and the single request against one
readiness authority, no first-spawn race, authenticated owner replacement,
response publication fencing, reconnect idempotency, exact residue transfer,
and deterministic testability. The single request wins because it works with
the current one-way sentinel on ssh2, system SSH, and WSL. A separate
capability probe, two-step negotiation, and a generic cross-transport wire API
are not approved.
and deterministic testability. The single request wins for the authenticated
deployed-relay bridge. A separate capability probe, two-step negotiation, and
a generic cross-transport wire API are not approved.
The daemon adapter preserves its current sequential connection order. The
control hello is authenticated and answered immediately with daemon identity;
an optional offer is only pending state keyed by that authenticated
`clientId`. The later stream hello proves the socket pair, applies the common
state transition once, and may return the grant in its response. The client
does not mark the daemon connected until that response. An implementation
changes the daemon protocol version as usual, so an exact-version prior daemon
continues independently rather than partially interpreting the optional
fields.
The local daemon adapter is deliberately not changed in this PR. Its existing
sequential control/stream hello, authentication, exact-version fencing, and
prior-version isolation remain authoritative. Reusing the semantic state
machine there is a later design decision and is not required to ship the SSH
bound.
The optional V1 offer is present only when the main rollout gate is on. The
server intersects the offer with its supported versions and clamps
@@ -338,10 +390,10 @@ transport client and installed generation. `pty.attach` must receive request
context just as `pty.spawn` does. Spawn failure, identity mismatch, stale
context, and response cancellation create no active V1 subscription. A legacy
spawn/attach returns the current identity/replay shape and creates only the
bounded transport subscription above—no token or source coordinate. During
kill-switch rotation, an old V1 token may be closing while its replacement
legacy transport subscription is opening, but no legacy token exists and no
live token changes mode.
bounded transport subscription above—no token or source coordinate. The
startup-latched mode never rotates a live V1 token into legacy service; normal
shutdown closes the old token before a restarted gate-off main opens a bounded
legacy connection.
Session ownership is granted by the authenticated session hello, not
constructor-assigned:
@@ -378,16 +430,16 @@ other than the one installed by its synchronous response hook. The constructor
stdout client is never implicitly a session owner. Production launches
detached on POSIX and Windows, invalidates stdout, and connects the desktop
bridge through `attachClient` over the versioned Unix socket or named pipe.
Direct stdio receives owner authority only from its launch-bound session hello.
Unproved direct stdio remains subscriber-only.
Capability support and rollout enablement are separate. Fresh POSIX and
Windows launches receive a safely quoted
`--pty-output-flow-control=v1|off` argument controlling advertisement. A new
main may decline V1 from an already-running detached relay even if it still
advertises it. Disabling the main gate rotates current tokens with the bounded
cancel/reattach procedure below; it never silently changes a live token's
semantics. Sink drain, writer ordering, decoder bounds, and header-ACK
hardening are correctness fixes and are not disabled by this gate.
advertises it. Changing the gate requires a main-process restart and never
changes a live token's semantics. Sink drain, writer ordering, decoder bounds,
and header-ACK hardening are correctness fixes and are not disabled by this
gate.
Production deployment does not form arbitrary mixed-build main/relay pairs.
`computeRemoteRelayDir` content-hash-scopes the install directory and its
@@ -401,7 +453,7 @@ diagnostic `unsupported-version-skew`, not a rollout cohort.
The reachable upgrade skew is an orphaned prior-version daemon in its old
version directory with live PTYs while the new main connects to a new endpoint.
It retains its old behavior until its own grace/cleanup completes and is not
reachable by the new main's kill switch. Upgrade diagnostics enumerate these
reachable by the new main's startup gate. Upgrade diagnostics enumerate these
versioned orphan processes; V1 neither adopts their PTYs nor claims to bound
their memory.
@@ -669,6 +721,25 @@ direct/manual compatibility clients; it is targeted, producer-admitted rather
than control-queued, never broadcast, and returns `restoreRequired` instead of
writing a body that cannot fit current non-reserved capacity.
Main quarantines the entire candidate recovery transaction until it can prove
the fence. The first source-bearing recovery range must start at the accepted
checkpoint, every later range must be exactly contiguous, and the final end
must equal `recoveryEndSu`. Empty recovery is valid only when the checkpoint
already equals `recoveryEndSu`; main retains that end as the required start of
the first later live frame even after activation. The first live frame and
every successor must also be exactly contiguous. Gap, overlap, incomplete
suffix, missing body, bad empty recovery, or bad live handoff reaches neither
the model nor desktop projection.
A failed or stale recovery attempt cancels only its own replacement token.
It never sends cancellation on a newer mux, clears a newer checkpoint, shuts
down the physical PTY/process, deletes PTY ownership, or expires the owner
lease. A valid token-scoped cancellation proof is applied locally when it
matches; otherwise the lease remains detached and retryable. A fresh relay
that no longer retains the cached owner returns typed error `-32041`; main
clears only the cached owner/checkpoints and retries `pty.openClient` exactly
once without `resume`.
`pty.serialize` follows the same rule: its response is a bounded stream marker,
snapshot chunks use the producer-owned bulk lane, and a completion fence ends
the stream. No replay-, gap-, or serialize-bearing response embeds its body in
@@ -1231,6 +1302,17 @@ replacement owner and atomically transfer every remaining mapping; otherwise
cancel that consumer and its bounded stream. Old stream generations cannot
settle new mappings.
The main registry stores the immutable source identity with each admitted
remote mapping, not only its `spanId`. A replacement reservation moves the
named consumer obligations to `transferring`; only an accepted headed or
headless `SnapshotEnd` publication at or beyond the reservation's required
sequence commits them. Stale generation, failed publication, disconnect, or
explicit rollback restores the original live-stream obligation. If
token-scoped cancellation proof has already reclaimed the authoritative span,
late ACK, detach, rollback, and replacement commit prune the cached mapping
and become idempotent no-ops; they never dereference reclaimed state or create
credit.
The stall policy is deliberately split. Desktop-only parse failure reaches its
bounded projection cap, transfers to model restore, and lets upstream credit
continue. A stalled model receipt or attached lossless remote has no automatic
@@ -1256,18 +1338,22 @@ explicit/connection-close proof. The transaction creates the replacement
owner before marking source `transferring`; only exact recovery-complete
coverage terminally transfers it.
| Event | Mandatory transition |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| model accepts; desktop/mobile parses | settle that consumer |
| hidden thinning, empty transform, pending-cap salvage | preserve ordered terminal facts/scanner state, then transfer desktop to model and emit restore marker |
| renderer reload/destroy/send failure/heal | transfer all desktop obligations to model before clearing queue/accounting |
| pane closes while provider/token remains live | transfer recoverable views, then request token cancel if no required consumer remains |
| PTY exit | relay seals token; main retains prior receipts/projections/remotes; runtime/renderer exit only after terminality or bounded cancellation proof |
| provider replacement/reconnect | close old client generation; transfer only exact ranges proven by replacement recovery, otherwise cancel and restore |
| relay `pty.deliveryCanceled` | without replacement, cancel matching remainder and restore; with replacement, enter `transferring` pending coverage |
| same-client token supersession | create replacement first, transfer exact covered remainder after recovery, cancel any uncovered range |
| explicit live-token reset/kill switch | `pty.cancelDelivery` response proves relay cancellation before local discard; failure closes the provider transport |
| relay/client dispose | relay cancels token/cursors; main cancels only after close-generation proof |
| Event | Mandatory transition |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| model accepts; desktop/mobile parses | settle that consumer |
| hidden thinning, empty transform, pending-cap salvage | preserve ordered terminal facts/scanner state, then transfer desktop to model and emit restore marker |
| renderer reload/destroy/send failure/heal | transfer all desktop obligations to model before clearing queue/accounting |
| pane closes while provider/token remains live | transfer recoverable views, then request token cancel if no required consumer remains |
| PTY exit | relay seals token; main retains prior receipts/projections/remotes; runtime/renderer exit only after terminality or bounded cancellation proof |
| provider replacement/reconnect | close old client generation; transfer only exact ranges proven by replacement recovery, otherwise cancel and restore |
| relay `pty.deliveryCanceled` | without replacement, cancel matching remainder and restore; with replacement, enter `transferring` pending coverage |
| same-client token supersession | create replacement first, transfer exact covered remainder after recovery, cancel any uncovered range |
| empty or non-empty recovery completion | require checkpoint-to-`recoveryEndSu` continuity, retain that live-start anchor after activation, then admit only an exactly contiguous live frame |
| stale/overlapping recovery attempt | cancel only that attempt's token; never mutate the replacement mux, checkpoint, PTY ownership, physical process, or owner lease |
| remote snapshot replacement | reserve immutable span IDs/ranges, commit only after current-generation `SnapshotEnd` coverage, otherwise roll back |
| late remote detach after token cancellation | prune already-reclaimed mappings and detach idempotently without dereference or new credit |
| explicit live-token reset | `pty.cancelDelivery` response proves relay cancellation before local discard; failure closes the provider transport |
| relay/client dispose | relay cancels token/cursors; main cancels only after close-generation proof |
There is no “abandon live token” transition. A main-side path that cannot prove
settlement, transfer, or relay cancellation must keep the ledger or close the
@@ -1330,7 +1416,10 @@ main token span:
main receive token:
unseen --beforeResolve--> receiving-activation
receiving-activation --validated response/no recovery--> active
receiving-activation --source-ranged recovery + complete fence--> active
receiving-activation --quarantine exact checkpoint..recoveryEnd + complete fence-->
active(expectedLiveStart = recoveryEnd)
active --first/next live starts at expectedLiveStart--> active(advance expectedLiveStart)
receiving-activation|active --gap/overlap/bad fence--> canceling-only-this-token
active --pty.exit--> exit-sealed
exit-sealed --prior model/projection/remote terminal--> exit-ack-queued
exit-ack-queued --runtime + renderer exit cleanup--> exited-awaiting-ack-publication
@@ -1339,6 +1428,12 @@ main receive token:
canceling-exited-token --token cancellation proof--> closed(restore-required)
canceling-exited-token --proof cannot publish--> closing-provider
receiving-activation|active --overflow/cancel/close--> closed
remote projection replacement:
live --reserve immutable spans + required SnapshotEnd seq--> transferring
transferring --current generation SnapshotEnd covers required seq--> transferred
transferring --stale/failure/disconnect--> live(rollback)
live|transferring --authoritative token already canceled--> detached(prune cached spans)
```
```ts
@@ -1424,6 +1519,25 @@ function onAckWriteSettled(token: DeliveryToken, endSu: number, result: SinkWrit
reclaimPublishedPrefix(token)
maybeCloseSealedDelivery(token)
}
function finishRecovery(candidate: RecoveryCandidate, fence: RecoveryComplete): void {
const expectedStart = candidate.checkpointSourceEndSu
const recoveredEnd = validateExactContiguousQuarantine(candidate.frames, expectedStart)
if (recoveredEnd !== fence.recoveryEndSu) {
cancelReplacementTokenOnly(candidate.identity, 'recovery-coverage-mismatch')
return
}
commitQuarantinedRecovery(candidate)
installLiveStartAnchor(candidate.identity, fence.recoveryEndSu)
}
function replaceRemoteConsumer(stream: RemoteStream, requiredSeq: number): void {
const reservation = remoteRanges.reserveImmutableSpans(stream, requiredSeq)
publishSnapshotEnd(stream).then(
(publication) => remoteRanges.commitIfCurrentAndCovered(reservation, publication),
(error) => remoteRanges.rollback(reservation, error)
)
}
```
Token/span validation, charged admission, and every membership obligation occur
@@ -1573,6 +1687,16 @@ source checkpoint is authoritative. Exit before reattach may still synthesize
`code: -1` after `notFound`; retaining remote exit tombstones is a separate
behavior change and is not required for the memory bound.
Recovery receipt is transactional. Main retains quarantined recovery bodies,
the candidate mux, token, checkpoint, and `recoveryEndSu` until the exact
completion fence validates. It commits no partial body to the model. On
failure it requests cancellation only through the candidate mux and accepts
only matching proof; a stale attempt may not touch current mux state. After a
successful empty recovery it keeps `recoveryEndSu` as a live continuity anchor
until the first live frame arrives. Failure marks the existing PTY lease
detached and retryable rather than deleting ownership or terminating the
physical process.
For an operation-ID spawn replay, physical result lookup completes first; the
current request then creates and fences a new subscription as specified above.
Disconnect after physical commit/before response leaves the PTY retained but
@@ -1727,12 +1851,12 @@ without parsing logs.
## Incremental implementation map
Do not implement this design as one architectural rewrite. Each slice below is
independently reviewable, keeps legacy framing valid, and must land with its
own deterministic oracle. Later slices may revise file placement, but not the
semantic boundaries or invariants.
This PR implemented the design as the independently testable layers below,
without replacing local/daemon protocols or requiring every deployment
topology to enable V1. The slices remain useful review and rollback
boundaries even though they ship in one PR.
### Slice 1: hostile-header and accounting hardening
### Slice 1: hostile-header and accounting hardening — implemented
- In `src/main/ssh/ssh-channel-multiplexer.ts`, clamp header ACKs and delete
only present timestamp keys; cap retained timestamps.
@@ -1740,7 +1864,7 @@ semantic boundaries or invariants.
and liveness rebasing.
- No relay, daemon, runtime, or provider protocol change.
### Slice 2: bounded decoder turns
### Slice 2: bounded decoder turns — implemented
- In both relay protocol decoders, bound frames and time per turn, preserve a
synchronous first handshake frame, pause reads during continuation, and
@@ -1750,7 +1874,7 @@ semantic boundaries or invariants.
- No PTY credit or session negotiation; prior-version peers keep the same
frame format.
### Slice 3: one drain-aware legacy writer
### Slice 3: one drain-aware legacy writer — implemented
- Add a concretely named dispatcher writer owning ordinary, bulk, control,
sentinel, residue, and `--connect` bridge bytes.
@@ -1780,7 +1904,7 @@ semantic boundaries or invariants.
and detached relay socket/named-pipe clients without changing the wire. The
separate local terminal daemon is not exercised or changed.
### Slice 4: bounded main SSH intake
### Slice 4: bounded main SSH intake — implemented
- Make `SshRelaySession.wireUpPtyEvents` hand each event exactly once to a
main-only intake under `src/main/ipc/`; SSH no longer bypasses the existing
@@ -1801,7 +1925,7 @@ semantic boundaries or invariants.
- Extend the existing Docker ACK-stall test only for behavior this slice
implements: direct SSH/deployed relay/desktop intake and active typing.
### Slice 5a: SSH session semantics
### Slice 5a: SSH session semantics — implemented for SSH
- Add the transport-neutral `PtyConsumerSession` state machine and only the SSH
readiness adapter decision above.
@@ -1811,7 +1935,7 @@ semantic boundaries or invariants.
- Local in-process, daemon hello, and remote-runtime adapters reuse semantics
only when their own changes need it; they are not prerequisites for SSH V1.
### Slice 5b: direct SSH source credit
### Slice 5b: direct SSH source credit — implemented
- Add the relay immutable span ledger/scheduler and tokenized cumulative ACKs
for one direct SSH desktop/model consumer path.
@@ -1820,7 +1944,7 @@ semantic boundaries or invariants.
implementation.
- Gate the feature and retain exact same-build legacy behavior.
### Slice 5c: exit, reconnect, and replay
### Slice 5c: exit, reconnect, and replay — implemented
- Add sealed-unsettled exit, cancellation proofs, owner reconnect grace,
token replacement, and exact transfer.
@@ -1830,7 +1954,7 @@ semantic boundaries or invariants.
- Do not enable reconnect V1 until late-ACK, timeout, supersession, and
generation-close oracles pass.
### Slice 5d: required remote consumers
### Slice 5d: required remote consumers — deterministic seam implemented
- Map remote `terminal.multiplex` encoded-byte ACK frames to accepted source
ranges without changing its transport authentication or granting it an SSH
@@ -1843,17 +1967,17 @@ semantic boundaries or invariants.
- Extend the extracted `terminal-output-frame-chunks.ts` seam with immutable
composite mapping input while preserving `694363805`'s code-unit scanner,
allocation profile, sequence rounding, and equivalence benchmark.
- Add headed paired-server first, then headless `orca serve`; neither is
inferred from Docker SSH.
- Live headed paired-server and headless `orca serve` validation remains
required before claiming those topologies; neither is inferred from Docker
SSH.
- Keep best-effort remote streams outside upstream obligations.
### Slice 5e: remaining adapters and rollout
### Slice 5e: remaining adapters and rollout — deliberately deferred
- Adapt local in-process and daemon hello only if shared session semantics
materially reduce their own lifecycle duplication; neither must adopt SSH
framing or source credit for the SSH release.
- Add WSL, Windows ConPTY/named-pipe, local daemon/provider, folder workspace,
mixed-version, and prior-version-orphan evidence separately.
- Local in-process and daemon hello remain unchanged; neither adopts SSH
framing or source credit in this PR.
- WSL, Windows ConPTY/named-pipe, local daemon/provider, folder workspace,
mixed-version, and prior-version-orphan live evidence remains separate.
- Promote reliability-gate provider coverage only for topologies with recorded
executable evidence.
@@ -1863,7 +1987,28 @@ native dependencies, or a per-file lint exception.
## Tests
### Unit and property tests
The checklist below is the normative matrix, not a claim that every physical
topology ran. The executable implementation evidence in this PR is:
- shared session negotiation, grant publication, stale-owner recovery, source
ledger, ACK queue/publication separation, and cancellation proof contracts;
- relay writer admission/drain/liveness, zero-display source publication,
operation-ID retry activation, fixed-size filesystem compatibility,
capacity-aware Git/PTY slicing, decoder caps, and token rotation;
- main model admission, desktop identity/rollback, exit barriers, remote
reserve/commit/rollback, token cancellation, exact recovery continuity,
eight-wide reconnect, retention budgets, rollout latching, and deploy policy;
- the four-test Docker OpenSSH/deployed-relay suite for source-window plateau,
concurrent typing, fixed-size filesystem/Git churn, and owner reconnect.
The deterministic recovery seam specifically covers an empty recovery followed
by a gapped live frame, overlapping reconnect attempts, late frames after
cancel, typed stale-owner retry, and the negative assertions that no physical
PTY shutdown, provider-state clear, ownership delete, or lease expiry occurs.
The remote-consumer seam covers token cancellation followed by late detach and
replacement commit, proving reclaimed span IDs are pruned idempotently.
### Normative unit and property matrix
- accept monotonic cumulative ACKs and reject duplicates, regressions,
over-credit, invalid numbers, wrong clients, wrong PTYs, and stale tokens;
@@ -2025,24 +2170,23 @@ one last chunk to prove the transient overshoot bound.
### Integration and E2E
The existing `tests/e2e/ssh-docker-relay-perf.spec.ts` is baseline evidence,
not V1 evidence. Its ACK-stall case holds desktop renderer ACKs, builds
main-to-renderer pressure, and checks active typing responsiveness through
direct SSH and the deployed relay. It does not saturate relay stdout/socket
writable queues, pause the remote PTY, exercise source-credit ACKs, prove model
admission bounds, or create a paired Orca runtime.
`tests/e2e/ssh-docker-relay-perf.spec.ts` now contains V1 evidence for the
implemented Linux SSH/deployed-relay path. With
`ORCA_SSH_PTY_SOURCE_CREDIT_V1=1`, it:
Add Docker assertions only after the corresponding runtime slice exists:
1. stalls desktop ACK and observes an exact 256 Ki-source-unit negotiated
plateau while a second SSH PTY remains responsive;
2. preserves fixed `STREAM_CHUNK_SIZE` filesystem semantics and completes Git
churn without corruption while active typing stays within budget;
3. reconnects the negotiated owner lease and proves the existing SSH
workspace terminal remains usable;
4. retains the direct typing latency control.
1. Slice 3 saturates actual direct-SSH/deployed-relay stdio or socket output,
proves `write(false)` stops later ordinary frames, writable length
plateaus, control progresses, and interactive echo stays bounded.
2. Slice 4 stalls main model admission and desktop ACKs independently, proves
charged intake plateaus, and preserves the current active-typing oracle.
3. Slices 5b/5c exhaust the negotiated source window, exercise sealed-exit late
ACK and cancellation, reconnects a new owner generation, rejects stale
ACKs, and verifies exact model/renderer output.
4. A separate concurrent bulk case proves the writer reserve and completion.
Writer `write(false)`/drain, callback failure, liveness bypass, decoder
scheduling, sealed exit, stale ACK, exact recovery, and remote replacement are
proved at deterministic unit/service seams. The Docker suite does not force
the main-to-relay sink to remain saturated beyond the health deadline, does
not prove paired-runtime behavior, and must not be cited for those claims.
Docker SSH proves the Linux SSH provider/relay path only. It must never be
reported as headed paired-server, headless `orca serve`, remote-runtime
@@ -2068,21 +2212,22 @@ adopt their PTYs, or claim the new main bounds their memory. Keep any new
native artifact compatible with Ubuntu 20.04/glibc 2.31; this design itself
adds none.
The current
`terminal-performance.output-backpressure-budget` reliability gate is
experimental and partial. It lists local, daemon, SSH, and remote-runtime
providers, but `coveredProviders` is empty and its executable evidence is
local macOS main-to-renderer/runtime behavior. Slice 3 adds relay
write/drain evidence, Slice 4 adds SSH intake/model admission, Slices 5b/5c add
source-credit lifecycle, and Slices 5d/5e add topology-specific runs before any
provider is marked covered. This documentation PR does not change or promote
that gate and does not claim unimplemented runtime behavior was exercised.
The
`terminal-performance.output-backpressure-budget` reliability gate remains
`experimental` and `partial`. Its `coveredProviders: ["ssh"]` means the
deterministic direct-SSH contracts plus the macOS-hosted Docker
OpenSSH/deployed-relay run described above. It does not cover every SSH host
platform and does not mark local, daemon, remote-runtime, headed, headless,
WSL, Windows, folder-workspace, prior-version, mixed-version, or Ubuntu 20.04
packaging topologies as executed.
## Rollout
Land V1 behind a main runtime gate and a relay advertisement launch policy.
Tests and development enable both. Main never negotiates merely because a
long-lived relay advertises; its gate is the final authority. Fresh POSIX and
V1 is behind a main-process startup gate and a persisted per-endpoint relay
launch policy. `ORCA_SSH_PTY_SOURCE_CREDIT_V1` is read once when the main
process starts; it is not a live kill switch. Tests and development enable the
gate before startup. Main never negotiates merely because a long-lived relay
advertises; its startup-latched gate is the final authority. Fresh POSIX and
Windows launch commands carry the explicit advertisement argument rather than
assuming a local environment variable reaches the remote process.
@@ -2092,15 +2237,16 @@ independently. Same-build gate-off sessions use the bounded legacy writer but
do not wait for V1 credit. A prior-version orphan remains governed by its own
binary and cleanup.
Rollout stages:
Implementation and rollout stages:
1. land Slices 1-4 independently under legacy framing, with each narrow
reliability oracle green before the next slice;
2. add Slices 5a-5c behind the main gate and relay advertisement, initially in
unit/service contracts and direct Docker SSH only;
3. add Slices 5d/5e for headed paired, headless, WSL, Windows, local-daemon,
local-provider, and folder-workspace evidence before marking those
providers/topologies covered;
1. Slices 1-4 are implemented under legacy-compatible framing with narrow
reliability oracles.
2. Slices 5a-5c are implemented behind the startup gate and relay launch
policy with unit/service contracts and direct Docker SSH evidence.
3. Slice 5d's remote source-range transaction is implemented at deterministic
main/runtime seams. Live headed paired, headless, WSL, Windows,
local-daemon, local-provider, and folder-workspace evidence remains future
work before those topologies are marked covered.
4. run an internal same-build canary comparing gate-off and V1 sink,
model-admission, latency, sealed-exit, recovery, and reconnect metrics while
separately counting orphan prior-version daemons;
@@ -2109,27 +2255,25 @@ Rollout stages:
fallback and versioned-orphan cleanup stay healthy through the supported
upgrade window.
Kill-switch behavior is explicit:
Rollback behavior is explicit:
1. Disable the main gate to stop all new V1 negotiations against the
same-build detached relay.
2. For each live token, stop new desktop membership changes, flush eligible
cumulative ACKs, and retain
`obligationsTerminalEndSu`/`ackQueuedEndSu`/`ackPublishedEndSu` until their
callbacks settle. Send `pty.cancelDelivery` for every remaining active or
sealed-unsettled token and wait up to 10 seconds for cancellation proof.
3. Reattach the same PTYs in bounded waves without requesting V1. Failure to
prove cancellation closes/reconnects the client so relay generation cleanup
is the proof; local ledgers and desktop range identities are not discarded
earlier.
4. Changing the relay launch policy affects fresh processes. Replacing a
long-lived remote relay is required only to remove its implementation or
advertisement, not to make a gated main decline the feature.
1. Change the environment/launch setting, then restart the main process. The
existing process keeps its original mode until shutdown.
2. Shutdown performs the normal token-scoped cancellation/exit cleanup; it
does not rotate live V1 tokens into legacy mode in place.
3. On the next connection, gate-off omits the V1 offer and accepts same-build
bounded legacy service. Gate-on requires a V1-launched endpoint.
4. If a persisted detached endpoint was launched gate-off, an off-to-on
connection fails with reset/reconnect guidance rather than silently
downgrading. Resetting that endpoint and reconnecting launches it with the
new policy.
5. A gate-off client remains compatible with a same-build V1-capable relay
because omission of the offer selects bounded legacy behavior.
Rollback never mutates a live token into legacy mode. The remote process may
remain alive throughout a successful cancel/reattach rotation. An orphaned
prior-version daemon is outside this sequence and is only observed until its
own version-scoped cleanup.
The remote process may remain alive only when its persisted launch policy is
compatible with the new startup mode. An orphaned prior-version daemon is
outside this sequence and is only observed until its own version-scoped
cleanup.
Release criteria:
@@ -2165,7 +2309,8 @@ Release criteria:
truncation.
- Content-hashed endpoints make mixed production binaries unreachable. The
real upgrade skew is an orphaned prior-version daemon outside the new main's
kill switch; V1 reports but cannot adopt, cancel, or retroactively bound it.
startup-gated endpoint; V1 reports but cannot adopt, cancel, or retroactively
bound it.
- Preserving the real exit code for a PTY that exits before any new client
reattaches would require attach-visible tombstone retention. V1 documents and
tests the existing synthetic `-1` fallback instead of coupling that separate
+7 -2
View File
@@ -159,7 +159,11 @@ import {
} from './pty-hidden-delivery-gate'
import { PtyPendingDataDrainQueue, type PendingPtyData } from './pty-pending-data-drain-queue'
import { SshPtyOutputIntake } from './ssh-pty-output-intake'
import { installSshPtyOutputIntake, publishSshPtySourceAck } from './ssh-pty-output-intake-registry'
import {
cancelSshPtySourceDelivery,
installSshPtyOutputIntake,
publishSshPtySourceAck
} from './ssh-pty-output-intake-registry'
import type { LegacySshProjectionSemantics } from './ssh-pty-legacy-projection'
import {
clearNativeWindowsConptyPty,
@@ -3169,7 +3173,8 @@ export function registerPtyHandlers(
}
}
},
publishSourceAck: publishSshPtySourceAck
publishSourceAck: publishSshPtySourceAck,
cancelSourceDelivery: cancelSshPtySourceDelivery
})
runtime?.setRemoteTerminalSourceRangeConsumerHooks?.(
sshOutputIntake.getRemoteSourceRangeConsumerHooks()
@@ -0,0 +1,145 @@
import type { SshPtyModelAdmissionKey } from './ssh-pty-model-admission-contract'
import {
admissionKeyId,
pressureHasAdmissionKey,
takePausedGeneration,
type AdmissionEntry,
type PtyUsage
} from './ssh-pty-model-admission-entry'
import type { SshPtyModelAdmissionLimits } from './ssh-pty-model-admission-limits'
type PressureOptions = {
limits: SshPtyModelAdmissionLimits
pauseProvider: (key: SshPtyModelAdmissionKey) => boolean
resumeProvider: (key: SshPtyModelAdmissionKey) => void
}
type PromotionOptions = {
usageByPty: ReadonlyMap<string, PtyUsage>
disposed: boolean
canReserve: (entry: AdmissionEntry) => boolean
reserve: (entry: AdmissionEntry) => void
isBelowGlobalLowWatermark: () => boolean
}
export class SshPtyModelAdmissionPressure {
private readonly entries: AdmissionEntry[] = []
private readonly pausedKeys = new Map<string, SshPtyModelAdmissionKey>()
private retainedBytes = 0
constructor(private readonly options: PressureOptions) {}
get values(): readonly AdmissionEntry[] {
return this.entries
}
get frameCount(): number {
return this.entries.length
}
get bytes(): number {
return this.retainedBytes
}
get pausedPtyCount(): number {
return this.pausedKeys.size
}
has(key: SshPtyModelAdmissionKey): boolean {
return pressureHasAdmissionKey(this.entries, key)
}
admit(entry: AdmissionEntry): boolean {
const id = admissionKeyId(entry.key)
const paused = this.pausedKeys.has(id) || this.options.pauseProvider(entry.key)
if (paused) {
this.pausedKeys.set(id, entry.key)
}
if (
!paused ||
this.entries.length >= this.options.limits.pressureMaxFrames ||
this.retainedBytes + entry.charge.bytes > this.options.limits.pressureMaxBytes
) {
return false
}
entry.state = 'pressure'
this.entries.push(entry)
this.retainedBytes += entry.charge.bytes
return true
}
cancelGeneration(
providerGeneration: number,
cancelPressureAndReserved: (entries: AdmissionEntry[]) => number
): void {
this.retainedBytes -= cancelPressureAndReserved(this.entries)
for (const key of takePausedGeneration(this.pausedKeys, providerGeneration)) {
this.resume(key)
}
}
cancelPty(key: SshPtyModelAdmissionKey, error: Error, cancelReserved: () => void): void {
const id = admissionKeyId(key)
const canceled: AdmissionEntry[] = []
for (let index = this.entries.length - 1; index >= 0; index--) {
if (admissionKeyId(this.entries[index]!.key) === id) {
canceled.push(this.entries.splice(index, 1)[0]!)
}
}
this.rejectPressureEntries(canceled, error)
cancelReserved()
const paused = this.pausedKeys.get(id)
if (paused) {
this.pausedKeys.delete(id)
this.resume(paused)
}
}
promoteAndResume(options: PromotionOptions): void {
if (options.disposed) {
return
}
for (let index = 0; index < this.entries.length; ) {
const entry = this.entries[index]!
const hasEarlierEntryForPty = this.entries
.slice(0, index)
.some((earlier) => admissionKeyId(earlier.key) === admissionKeyId(entry.key))
if (hasEarlierEntryForPty || !options.canReserve(entry)) {
index++
continue
}
this.entries.splice(index, 1)
this.retainedBytes -= entry.charge.bytes
options.reserve(entry)
}
if (!options.isBelowGlobalLowWatermark()) {
return
}
for (const [id, key] of this.pausedKeys) {
const usage = options.usageByPty.get(id)
if (
this.has(key) ||
(usage?.sourceUnits ?? 0) > this.options.limits.perPtyLowSourceUnits ||
(usage?.bytes ?? 0) > this.options.limits.perPtyLowBytes
) {
continue
}
this.pausedKeys.delete(id)
this.resume(key)
}
}
private rejectPressureEntries(entries: readonly AdmissionEntry[], error: Error): void {
for (const entry of entries) {
this.retainedBytes -= entry.charge.bytes
entry.state = 'settled'
entry.reject(error)
}
}
private resume(key: SshPtyModelAdmissionKey): void {
try {
this.options.resumeProvider(key)
} catch {}
}
}
+78 -100
View File
@@ -9,10 +9,8 @@ import {
admissionKeyId,
canReserveAdmission,
cancelAdmissionGeneration,
pressureHasAdmissionKey,
retainedBytes,
resolveAdmissionIdleWaiters,
takePausedGeneration,
type AdmissionCharge,
type AdmissionEntry,
type PtyUsage
@@ -21,6 +19,7 @@ import {
resolveSshPtyModelAdmissionLimits,
type SshPtyModelAdmissionLimits
} from './ssh-pty-model-admission-limits'
import { SshPtyModelAdmissionPressure } from './ssh-pty-model-admission-pressure'
export type {
SshPtyModelAdmissionKey,
@@ -30,23 +29,22 @@ export type {
export class SshPtyModelAdmission {
private readonly limits: SshPtyModelAdmissionLimits
private readonly pauseProvider: (key: SshPtyModelAdmissionKey) => boolean
private readonly resumeProvider: (key: SshPtyModelAdmissionKey) => void
private readonly closeProvider: (providerGeneration: number, reason: string) => void
private readonly usageByPty = new Map<string, PtyUsage>()
private readonly pressure: AdmissionEntry[] = []
private readonly pausedKeys = new Map<string, SshPtyModelAdmissionKey>()
private readonly pressure: SshPtyModelAdmissionPressure
private readonly idleWaiters = new Map<string, Set<() => void>>()
private readonly closingGenerations = new Set<number>()
private globalSourceUnits = 0
private globalBytes = 0
private pressureBytes = 0
private disposed = false
constructor(options: SshPtyModelAdmissionOptions = {}) {
this.limits = resolveSshPtyModelAdmissionLimits(options)
this.pauseProvider = options.pauseProvider ?? (() => false)
this.resumeProvider = options.resumeProvider ?? (() => {})
this.pressure = new SshPtyModelAdmissionPressure({
limits: this.limits,
pauseProvider: options.pauseProvider ?? (() => false),
resumeProvider: options.resumeProvider ?? (() => {})
})
this.closeProvider = options.closeProvider ?? (() => {})
}
@@ -72,45 +70,65 @@ export class SshPtyModelAdmission {
reject,
state: 'queued'
}
if (this.canReserve(key, charge) && !pressureHasAdmissionKey(this.pressure, key)) {
if (this.canReserve(key, charge) && !this.pressure.has(key)) {
this.reserveAndQueue(entry)
return
}
this.admitPressure(entry)
if (!this.pressure.admit(entry)) {
entry.reject(admissionError('ssh_model_admission_pressure_exhausted'))
this.closeProvider(entry.key.providerGeneration, 'model-admission-pressure')
}
})
}
closeGeneration(providerGeneration: number, reason = 'provider-generation-closed'): void {
this.closingGenerations.add(providerGeneration)
const error = admissionError(reason)
this.pressureBytes -= cancelAdmissionGeneration({
pressure: this.pressure,
usageByPty: this.usageByPty,
idleWaiters: this.idleWaiters,
providerGeneration,
error,
release: (key, charge) => this.release(key, charge)
})
for (const key of takePausedGeneration(this.pausedKeys, providerGeneration)) {
try {
this.resumeProvider(key)
} catch {}
}
this.pressure.cancelGeneration(providerGeneration, (pressure) =>
cancelAdmissionGeneration({
pressure,
usageByPty: this.usageByPty,
idleWaiters: this.idleWaiters,
providerGeneration,
error,
release: (key, charge) => this.release(key, charge)
})
)
const generationPrefix = `${providerGeneration}\0`
for (const id of this.idleWaiters.keys()) {
if (id.startsWith(generationPrefix)) {
resolveAdmissionIdleWaiters(this.usageByPty, this.pressure, this.idleWaiters, id)
resolveAdmissionIdleWaiters(this.usageByPty, this.pressure.values, this.idleWaiters, id)
}
}
}
cancelPty(key: SshPtyModelAdmissionKey, reason: string): void {
const id = admissionKeyId(key)
const error = admissionError(reason)
this.pressure.cancelPty(key, error, () => {
const usage = this.usageByPty.get(id)
if (usage) {
const canceled = [...usage.queued, ...(usage.running ? [usage.running] : [])]
usage.queued = []
usage.running = null
for (const entry of canceled) {
if (entry.state === 'settled') {
continue
}
entry.state = 'settled'
this.release(entry.key, entry.charge)
entry.reject(error)
}
this.usageByPty.delete(id)
}
})
resolveAdmissionIdleWaiters(this.usageByPty, this.pressure.values, this.idleWaiters, id)
}
whenIdle(key: SshPtyModelAdmissionKey): Promise<void> {
const id = admissionKeyId(key)
const usage = this.usageByPty.get(id)
const hasPressure = this.pressure.some(
(entry) =>
entry.key.providerGeneration === key.providerGeneration && entry.key.ptyId === key.ptyId
)
const hasPressure = this.pressure.has(key)
if ((!usage || (!usage.running && usage.queued.length === 0)) && !hasPressure) {
return Promise.resolve()
}
@@ -135,7 +153,7 @@ export class SshPtyModelAdmission {
generations.add(entry.key.providerGeneration)
}
}
for (const entry of this.pressure) {
for (const entry of this.pressure.values) {
generations.add(entry.key.providerGeneration)
}
for (const generation of generations) {
@@ -147,9 +165,9 @@ export class SshPtyModelAdmission {
return {
sourceUnits: this.globalSourceUnits,
bytes: this.globalBytes,
pressureFrames: this.pressure.length,
pressureBytes: this.pressureBytes,
pausedPtys: this.pausedKeys.size
pressureFrames: this.pressure.frameCount,
pressureBytes: this.pressure.bytes,
pausedPtys: this.pressure.pausedPtyCount
}
}
@@ -181,33 +199,13 @@ export class SshPtyModelAdmission {
this.startNext(id, usage)
}
private admitPressure(entry: AdmissionEntry): void {
const id = admissionKeyId(entry.key)
const paused = this.pausedKeys.has(id) || this.pauseProvider(entry.key)
if (paused) {
this.pausedKeys.set(id, entry.key)
}
if (
!paused ||
this.pressure.length >= this.limits.pressureMaxFrames ||
this.pressureBytes + entry.charge.bytes > this.limits.pressureMaxBytes
) {
entry.reject(admissionError('ssh_model_admission_pressure_exhausted'))
this.closeProvider(entry.key.providerGeneration, 'model-admission-pressure')
return
}
entry.state = 'pressure'
this.pressure.push(entry)
this.pressureBytes += entry.charge.bytes
}
private startNext(id: string, usage: PtyUsage): void {
if (usage.running) {
return
}
const entry = usage.queued.shift()
if (!entry) {
this.maybeResumeAndPromote()
this.promotePressureAndResumeProvider()
return
}
usage.running = entry
@@ -233,13 +231,24 @@ export class SshPtyModelAdmission {
})
},
(error) => {
if (this.finishEntry(id, usage, entry)) {
entry.reject(error instanceof Error ? error : new Error(String(error)))
}
this.failEntry(id, usage, entry, error instanceof Error ? error : new Error(String(error)))
}
)
}
private failEntry(id: string, usage: PtyUsage, entry: AdmissionEntry, error: Error): void {
if (entry.state !== 'running' || usage.running !== entry) {
return
}
this.closingGenerations.add(entry.key.providerGeneration)
usage.running = null
entry.state = 'settled'
this.release(entry.key, entry.charge)
entry.reject(error)
this.closeGeneration(entry.key.providerGeneration, 'ssh_model_admission_completion_failed')
this.cleanupUsage(id, usage)
}
private finishEntry(id: string, usage: PtyUsage, entry: AdmissionEntry): boolean {
if (entry.state !== 'running' || usage.running !== entry) {
return false
@@ -256,7 +265,7 @@ export class SshPtyModelAdmission {
if (!usage.running && usage.queued.length === 0 && usage.sourceUnits === 0) {
this.usageByPty.delete(id)
}
resolveAdmissionIdleWaiters(this.usageByPty, this.pressure, this.idleWaiters, id)
resolveAdmissionIdleWaiters(this.usageByPty, this.pressure.values, this.idleWaiters, id)
}
private release(key: SshPtyModelAdmissionKey, charge: AdmissionCharge): void {
@@ -267,49 +276,18 @@ export class SshPtyModelAdmission {
}
this.globalSourceUnits = Math.max(0, this.globalSourceUnits - charge.sourceUnits)
this.globalBytes = Math.max(0, this.globalBytes - charge.bytes)
this.maybeResumeAndPromote()
this.promotePressureAndResumeProvider()
}
private maybeResumeAndPromote(): void {
if (this.disposed) {
return
}
for (let index = 0; index < this.pressure.length; ) {
const entry = this.pressure[index]!
const hasEarlierEntryForPty = this.pressure
.slice(0, index)
.some((earlier) => admissionKeyId(earlier.key) === admissionKeyId(entry.key))
if (hasEarlierEntryForPty || !this.canReserve(entry.key, entry.charge)) {
index++
continue
}
this.pressure.splice(index, 1)
this.pressureBytes -= entry.charge.bytes
this.reserveAndQueue(entry)
}
if (
this.globalSourceUnits > this.limits.globalLowSourceUnits ||
this.globalBytes > this.limits.globalLowBytes
) {
return
}
for (const [id, key] of this.pausedKeys) {
const usage = this.usageByPty.get(id)
const stillPressured = this.pressure.some(
(entry) =>
entry.key.providerGeneration === key.providerGeneration && entry.key.ptyId === key.ptyId
)
if (
stillPressured ||
(usage?.sourceUnits ?? 0) > this.limits.perPtyLowSourceUnits ||
(usage?.bytes ?? 0) > this.limits.perPtyLowBytes
) {
continue
}
this.pausedKeys.delete(id)
try {
this.resumeProvider(key)
} catch {}
}
private promotePressureAndResumeProvider(): void {
this.pressure.promoteAndResume({
usageByPty: this.usageByPty,
disposed: this.disposed,
canReserve: (entry) => this.canReserve(entry.key, entry.charge),
reserve: (entry) => this.reserveAndQueue(entry),
isBelowGlobalLowWatermark: () =>
this.globalSourceUnits <= this.limits.globalLowSourceUnits &&
this.globalBytes <= this.limits.globalLowBytes
})
}
}
@@ -0,0 +1,161 @@
import type { SshPtyLegacyProjectionLedger } from './ssh-pty-legacy-projection'
import type { SshPtyModelAdmission } from './ssh-pty-model-admission'
import type {
SshPtyOutputExitEvent,
SshPtyOutputIntakeDependencies
} from './ssh-pty-output-intake-contract'
import { outputIntakeError, type SshPtyExitBarrier } from './ssh-pty-output-intake-validation'
import type { SshPtyOutputSourceObligations } from './ssh-pty-output-source-obligations'
type SshPtyOutputExitDeadlineDependencies = Readonly<{
admission: SshPtyModelAdmission
projections: SshPtyLegacyProjectionLedger
sourceObligations: SshPtyOutputSourceObligations
intake: SshPtyOutputIntakeDependencies
barrierMs?: number
cancellationProofMs?: number
}>
export class SshPtyOutputExitDeadline {
private readonly barriersByGeneration = new Map<number, Set<SshPtyExitBarrier>>()
private readonly barrierMs: number
private readonly cancellationProofMs: number
constructor(private readonly dependencies: SshPtyOutputExitDeadlineDependencies) {
this.barrierMs = dependencies.barrierMs ?? 30_000
this.cancellationProofMs = dependencies.cancellationProofMs ?? 10_000
}
wait(event: SshPtyOutputExitEvent, promise: Promise<void>): Promise<void> {
return new Promise((resolve, reject) => {
let timeoutStarted = false
let settled = false
const settle = (result: { ok: true } | { ok: false; error: Error }): void => {
if (settled) {
return
}
settled = true
clearTimeout(barrier.timer)
this.remove(event.providerGeneration, barrier)
if (result.ok) {
resolve()
} else {
reject(result.error)
}
}
const barrier: SshPtyExitBarrier = {
timer: setTimeout(() => {
timeoutStarted = true
void this.cancelTimedOutExit(event).then(
() => settle({ ok: true }),
(error) => {
this.dependencies.intake.closeProvider?.(
event.providerGeneration,
'ssh-exit-cancellation-proof-failed'
)
settle({
ok: false,
error: error instanceof Error ? error : outputIntakeError(String(error))
})
}
)
}, this.barrierMs),
reject: (error) => settle({ ok: false, error })
}
barrier.timer.unref?.()
let barriers = this.barriersByGeneration.get(event.providerGeneration)
if (!barriers) {
barriers = new Set()
this.barriersByGeneration.set(event.providerGeneration, barriers)
}
barriers.add(barrier)
void promise.then(
() => {
if (!timeoutStarted) {
settle({ ok: true })
}
},
(error) => {
if (!timeoutStarted) {
settle({ ok: false, error })
}
}
)
})
}
closeGeneration(providerGeneration: number, error: Error): void {
const barriers = this.barriersByGeneration.get(providerGeneration)
if (!barriers) {
return
}
for (const barrier of barriers) {
clearTimeout(barrier.timer)
barrier.reject(error)
}
this.barriersByGeneration.delete(providerGeneration)
}
get activeBarriers(): number {
return Array.from(this.barriersByGeneration.values()).reduce(
(total, barriers) => total + barriers.size,
0
)
}
private async cancelTimedOutExit(event: SshPtyOutputExitEvent): Promise<void> {
const cancel = this.dependencies.intake.cancelSourceDelivery
if (!cancel) {
throw outputIntakeError('ssh_source_cancellation_publisher_unavailable')
}
this.dependencies.admission.cancelPty(
{ ptyId: event.id, providerGeneration: event.providerGeneration },
'ssh_exit_delivery_canceled'
)
this.dependencies.sourceObligations.sealPty(event)
const cancellation = this.dependencies.sourceObligations.cancelPty(event, (request) =>
cancel(event.providerGeneration, request)
)
const canceled = await this.withCancellationProofDeadline(cancellation)
if (!canceled) {
throw outputIntakeError('ssh_source_cancellation_identity_unavailable')
}
this.dependencies.projections.transferPty(event.id, 'ssh-exit-delivery-canceled')
this.dependencies.intake.prepareExit(event)
this.dependencies.intake.finalizeExit(event)
this.dependencies.projections.closePty(
event.id,
event.providerGeneration,
event.ptyIncarnation,
'ssh-exit-delivery-canceled'
)
}
private withCancellationProofDeadline<T>(promise: Promise<T>): Promise<T> {
return new Promise((resolve, reject) => {
const timer = setTimeout(
() => reject(outputIntakeError('ssh_source_cancellation_proof_timeout')),
this.cancellationProofMs
)
timer.unref?.()
void promise.then(
(value) => {
clearTimeout(timer)
resolve(value)
},
(error) => {
clearTimeout(timer)
reject(error)
}
)
})
}
private remove(providerGeneration: number, barrier: SshPtyExitBarrier): void {
const barriers = this.barriersByGeneration.get(providerGeneration)
barriers?.delete(barrier)
if (barriers?.size === 0) {
this.barriersByGeneration.delete(providerGeneration)
}
}
}
+14 -1
View File
@@ -11,14 +11,26 @@ export async function settleSshPtyOutputExit(args: {
projections: SshPtyLegacyProjectionLedger
dependencies: SshPtyOutputIntakeDependencies
validateGeneration: () => void
afterAdmissionIdle?: () => void
waitForSourceTerminal?: () => Promise<void>
beforeFinalize?: () => void
}): Promise<void> {
const { event, admission, projections, dependencies, validateGeneration, beforeFinalize } = args
const {
event,
admission,
projections,
dependencies,
validateGeneration,
afterAdmissionIdle,
waitForSourceTerminal,
beforeFinalize
} = args
await admission.whenIdle({
ptyId: event.id,
providerGeneration: event.providerGeneration
})
validateGeneration()
afterAdmissionIdle?.()
try {
dependencies.prepareExit(event)
} catch (error) {
@@ -38,6 +50,7 @@ export async function settleSshPtyOutputExit(args: {
'pty-exit-unpublished'
)
await projections.whenPtyTerminal(event.id, event.providerGeneration, event.ptyIncarnation)
await waitForSourceTerminal?.()
validateGeneration()
try {
beforeFinalize?.()
@@ -5,6 +5,18 @@ import type {
} from './ssh-pty-model-admission-contract'
import type { PtySourceCreditAckBatch } from '../../shared/pty-source-credit-contract'
export type SshPtySourceCancellationRequest = Readonly<{
id: string
clientGeneration: number
ownerGeneration: number
deliveryToken: string
}>
export type SshPtySourceCancellationProof = Readonly<{
sentEndSu: number
creditedEndSu: number
}>
export type SshPtyOutputDataEvent = Readonly<{
id: string
data: string
@@ -52,8 +64,13 @@ export type SshPtyOutputIntakeDependencies = {
batch: PtySourceCreditAckBatch,
onSettled: (result: { ok: true } | { ok: false; error: Error }) => void
) => void
cancelSourceDelivery?: (
providerGeneration: number,
request: SshPtySourceCancellationRequest
) => Promise<SshPtySourceCancellationProof>
}
export type SshPtyOutputIntakeOptions = SshPtyModelAdmissionOptions & {
exitBarrierMs?: number
exitCancellationProofMs?: number
}
+40 -1
View File
@@ -2,7 +2,9 @@ import type { SshPtyOutputIntake } from './ssh-pty-output-intake'
import type {
SshPtyOutputDataEvent,
SshPtyOutputExitEvent,
SshPtyOutputReceipt
SshPtyOutputReceipt,
SshPtySourceCancellationProof,
SshPtySourceCancellationRequest
} from './ssh-pty-output-intake-contract'
import type { SshPtyAcceptedSourceCheckpoint } from './ssh-pty-output-source-obligations'
import type { PtySourceCreditAckBatch } from '../../shared/pty-source-credit-contract'
@@ -10,12 +12,17 @@ import type { PtySourceCreditAckBatch } from '../../shared/pty-source-credit-con
let installedIntake: SshPtyOutputIntake | null = null
let nextProviderGeneration = 1
const sourceAckPublishers = new Map<number, SshPtySourceAckPublisher>()
const sourceCancellationPublishers = new Map<number, SshPtySourceCancellationPublisher>()
type SshPtySourceAckPublisher = (
batch: PtySourceCreditAckBatch,
onSettled: (result: { ok: true } | { ok: false; error: Error }) => void
) => void
type SshPtySourceCancellationPublisher = (
request: SshPtySourceCancellationRequest
) => Promise<SshPtySourceCancellationProof>
export function allocateSshPtyProviderGeneration(): number {
return nextProviderGeneration++
}
@@ -82,6 +89,38 @@ export function publishSshPtySourceAck(
publisher(batch, onSettled)
}
export function installSshPtySourceCancellationPublisher(
providerGeneration: number,
cancel: SshPtySourceCancellationPublisher
): () => void {
if (sourceCancellationPublishers.has(providerGeneration)) {
throw new Error('ssh_source_cancellation_publisher_duplicate_generation')
}
sourceCancellationPublishers.set(providerGeneration, cancel)
return () => {
if (sourceCancellationPublishers.get(providerGeneration) === cancel) {
sourceCancellationPublishers.delete(providerGeneration)
}
}
}
export function cancelSshPtySourceDelivery(
providerGeneration: number,
request: SshPtySourceCancellationRequest
): Promise<SshPtySourceCancellationProof> {
const publisher = sourceCancellationPublishers.get(providerGeneration)
return publisher
? publisher(request)
: Promise.reject(new Error('ssh_source_cancellation_publisher_unavailable'))
}
export function applySshPtySourceCancellationProof(
event: SshPtyOutputExitEvent,
proof: SshPtySourceCancellationProof
): boolean {
return installedIntake?.applySourceCancellationProof(event, proof) ?? false
}
function outputIntakeUnavailableError(): Error {
return Object.assign(new Error('ssh_output_intake_unavailable'), {
code: 'ssh_output_intake_unavailable'
+164 -3
View File
@@ -376,6 +376,65 @@ describe('SshPtyOutputIntake', () => {
expect(harness.order).toEqual(['model:aaaa', 'project:aaaa', 'exit'])
})
it('admits queued pre-exit source spans before sealing the token', async () => {
const harness = createHarness({}, { exitBarrierMs: 1000 })
const first = harness.intake.acceptData(
event({
source: {
spanId: 'span-1',
clientGeneration: 2,
ownerGeneration: 3,
deliveryToken: 'token-1',
sourceStartSu: 0,
sourceEndSu: 4
}
})
)
const second = harness.intake.acceptData(
event({
data: 'bbbb',
source: {
spanId: 'span-2',
clientGeneration: 2,
ownerGeneration: 3,
deliveryToken: 'token-1',
sourceStartSu: 4,
sourceEndSu: 8
}
})
)
const exit = harness.intake.acceptExit({
id: 'pty-1',
code: 0,
providerGeneration: 1,
ptyIncarnation: 'incarnation-1'
})
harness.completions[0]!.resolve()
await first
harness.completions[1]!.resolve()
await Promise.all([second, exit])
expect(harness.order).toEqual([
'model:aaaa',
'project:aaaa',
'model:bbbb',
'project:bbbb',
'exit'
])
})
it('does not pump queued model work after a running completion fails', async () => {
const harness = createHarness()
const first = harness.intake.acceptData(event({ data: 'first' }))
const queued = harness.intake.acceptData(event({ data: 'queued' }))
harness.completions[0]!.reject(new Error('emulator failed'))
await expect(first).rejects.toThrow('emulator failed')
await expect(queued).rejects.toThrow('ssh_model_admission_completion_failed')
expect(harness.order).toEqual(['model:first', 'project:first'])
expect(harness.completions).toHaveLength(1)
})
it('retains exited delivery until published renderer projections settle', async () => {
const harness = createHarness({}, { exitBarrierMs: 1000 })
const dataReceipt = harness.intake.acceptData(event())
@@ -408,8 +467,57 @@ describe('SshPtyOutputIntake', () => {
expect(harness.intake.getDebugSnapshot().projection.records).toBe(0)
})
it('retains exit until a required remote source consumer settles', async () => {
const harness = createHarness({}, { exitBarrierMs: 1000 })
const remote = harness.intake.getRemoteSourceRangeConsumerHooks()
const stream = {
ptyId: 'pty-1',
consumerId: 'remote-1',
streamGeneration: 'stream-1'
}
expect(remote.attach(stream)).toBe(true)
const dataReceipt = harness.intake.acceptData(
event({
source: {
spanId: 'span-1',
clientGeneration: 2,
ownerGeneration: 3,
deliveryToken: 'token-1',
sourceStartSu: 0,
sourceEndSu: 4
}
})
)
harness.completions[0]!.resolve()
const receipt = await dataReceipt
let exited = false
const exitReceipt = harness.intake
.acceptExit({
id: 'pty-1',
code: 0,
providerGeneration: 1,
ptyIncarnation: 'incarnation-1'
})
.then(() => {
exited = true
})
await Promise.resolve()
expect(exited).toBe(false)
remote.settle(stream, [receipt.projection.desktopSpan!])
await exitReceipt
expect(harness.order.at(-1)).toBe('exit')
expect(harness.dependencies.closeProvider).not.toHaveBeenCalled()
})
it('keeps timed-out exit projections until generation-close proof', async () => {
const harness = createHarness({}, { exitBarrierMs: 1 })
const harness = createHarness(
{
cancelSourceDelivery: () => Promise.reject(new Error('cancel transport failed'))
},
{ exitBarrierMs: 1, exitCancellationProofMs: 10 }
)
const dataReceipt = harness.intake.acceptData(event())
harness.completions[0]!.resolve()
const receipt = await dataReceipt
@@ -426,14 +534,67 @@ describe('SshPtyOutputIntake', () => {
providerGeneration: 1,
ptyIncarnation: 'incarnation-1'
})
).rejects.toThrow('ssh_exit_barrier_timeout')
expect(harness.dependencies.closeProvider).toHaveBeenCalledWith(1, 'ssh-exit-barrier-timeout')
).rejects.toThrow('ssh_source_cancellation_identity_unavailable')
expect(harness.dependencies.closeProvider).toHaveBeenCalledWith(
1,
'ssh-exit-cancellation-proof-failed'
)
expect(harness.intake.getDebugSnapshot().projection.records).toBe(1)
harness.intake.closeGeneration(1, 'provider-closed')
expect(harness.intake.getDebugSnapshot().projection.records).toBe(0)
})
it('cancels only the timed-out source delivery and keeps the provider usable', async () => {
const cancelSourceDelivery = vi.fn(async () => ({ sentEndSu: 4, creditedEndSu: 0 }))
const harness = createHarness(
{ cancelSourceDelivery },
{ exitBarrierMs: 1, exitCancellationProofMs: 100 }
)
const remote = harness.intake.getRemoteSourceRangeConsumerHooks()
const stream = {
ptyId: 'pty-1',
consumerId: 'remote-1',
streamGeneration: 'stream-1'
}
remote.attach(stream)
const dataReceipt = harness.intake.acceptData(
event({
source: {
spanId: 'span-1',
clientGeneration: 2,
ownerGeneration: 3,
deliveryToken: 'token-1',
sourceStartSu: 0,
sourceEndSu: 4
}
})
)
harness.completions[0]!.resolve()
await dataReceipt
await harness.intake.acceptExit({
id: 'pty-1',
code: 0,
providerGeneration: 1,
ptyIncarnation: 'incarnation-1'
})
expect(cancelSourceDelivery).toHaveBeenCalledWith(1, {
id: 'pty-1',
clientGeneration: 2,
ownerGeneration: 3,
deliveryToken: 'token-1'
})
expect(harness.dependencies.closeProvider).not.toHaveBeenCalled()
expect(harness.order.at(-1)).toBe('exit')
const sibling = harness.intake.acceptData(
event({ id: 'pty-2', ptyIncarnation: 'incarnation-2' })
)
harness.completions[1]!.resolve()
await expect(sibling).resolves.toMatchObject({ ptyId: 'pty-2' })
})
it('rejects late same-generation data after ordered exit cleanup', async () => {
const harness = createHarness({}, { exitBarrierMs: 1000 })
const first = harness.intake.acceptData(event())
+26 -59
View File
@@ -3,6 +3,7 @@ import {
type LegacySshProjectionSemantics
} from './ssh-pty-legacy-projection'
import { SshPtyModelAdmission } from './ssh-pty-model-admission'
import { SshPtyOutputExitDeadline } from './ssh-pty-output-exit-deadline'
import { settleSshPtyOutputExit } from './ssh-pty-output-exit'
import { SshPtyOutputGenerationGuard } from './ssh-pty-output-generation-guard'
import {
@@ -16,7 +17,7 @@ import type {
SshPtyOutputIntakeOptions,
SshPtyOutputReceipt
} from './ssh-pty-output-intake-contract'
import { outputIntakeError, type SshPtyExitBarrier } from './ssh-pty-output-intake-validation'
import { outputIntakeError } from './ssh-pty-output-intake-validation'
export type {
SshPtyOutputDataEvent,
@@ -31,8 +32,7 @@ export class SshPtyOutputIntake {
private readonly sourceObligations: SshPtyOutputSourceObligations
private readonly generationGuard = new SshPtyOutputGenerationGuard(() => this.disposed)
private readonly admission: SshPtyModelAdmission
private readonly exitBarrierMs: number
private readonly exitBarriersByGeneration = new Map<number, Set<SshPtyExitBarrier>>()
private readonly exitDeadline: SshPtyOutputExitDeadline
private disposed = false
constructor(
@@ -44,7 +44,6 @@ export class SshPtyOutputIntake {
onSettled: (span) => this.sourceObligations.settleDesktop(span, 'renderer-parse'),
onTransferred: (span, reason) => this.sourceObligations.transferDesktop(span, reason)
})
this.exitBarrierMs = options.exitBarrierMs ?? 30_000
this.admission = new SshPtyModelAdmission({
...options,
pauseProvider: (key) =>
@@ -54,6 +53,14 @@ export class SshPtyOutputIntake {
closeProvider: (providerGeneration, reason) =>
this.dependencies.closeProvider?.(providerGeneration, reason)
})
this.exitDeadline = new SshPtyOutputExitDeadline({
admission: this.admission,
projections: this.projections,
sourceObligations: this.sourceObligations,
intake: this.dependencies,
barrierMs: options.exitBarrierMs,
cancellationProofMs: options.exitCancellationProofMs
})
}
acceptData(event: SshPtyOutputDataEvent): Promise<SshPtyOutputReceipt> {
@@ -141,7 +148,9 @@ export class SshPtyOutputIntake {
'model-admission-failed'
)
}
this.dependencies.closeProvider?.(event.providerGeneration, 'model-admission-failed')
if ((error as { code?: unknown }).code !== 'ssh_exit_delivery_canceled') {
this.dependencies.closeProvider?.(event.providerGeneration, 'model-admission-failed')
}
throw error
}
)
@@ -149,8 +158,7 @@ export class SshPtyOutputIntake {
async acceptExit(event: SshPtyOutputExitEvent): Promise<void> {
this.generationGuard.sealExit(event)
this.sourceObligations.sealPty(event)
await this.withExitDeadline(event.providerGeneration, this.finishExit(event))
await this.exitDeadline.wait(event, this.finishExit(event))
}
private async finishExit(event: SshPtyOutputExitEvent): Promise<void> {
@@ -160,6 +168,8 @@ export class SshPtyOutputIntake {
projections: this.projections,
dependencies: this.dependencies,
validateGeneration: () => this.generationGuard.validate(event),
afterAdmissionIdle: () => this.sourceObligations.sealPty(event),
waitForSourceTerminal: () => this.sourceObligations.whenPtyTerminal(event),
beforeFinalize: () => this.sourceObligations.markExitPublished(event)
})
}
@@ -196,14 +206,7 @@ export class SshPtyOutputIntake {
this.dependencies.onGenerationClosed?.(providerGeneration, reason)
this.projections.closeGeneration(providerGeneration, reason)
this.sourceObligations.closeGeneration(providerGeneration, reason)
const barriers = this.exitBarriersByGeneration.get(providerGeneration)
if (barriers) {
for (const barrier of barriers) {
clearTimeout(barrier.timer)
barrier.reject(outputIntakeError(reason))
}
this.exitBarriersByGeneration.delete(providerGeneration)
}
this.exitDeadline.closeGeneration(providerGeneration, outputIntakeError(reason))
}
dispose(): void {
@@ -226,55 +229,19 @@ export class SshPtyOutputIntake {
return this.sourceObligations.acceptedCheckpoints(providerGeneration)
}
applySourceCancellationProof(
event: SshPtyOutputExitEvent,
proof: Readonly<{ sentEndSu: number; creditedEndSu: number }>
): boolean {
return this.sourceObligations.applyCancellationProof(event, proof)
}
getDebugSnapshot() {
return {
model: this.admission.getDebugSnapshot(),
projection: this.projections.getDebugSnapshot(),
source: this.sourceObligations.getDebugSnapshot(),
exitBarriers: Array.from(this.exitBarriersByGeneration.values()).reduce(
(total, barriers) => total + barriers.size,
0
)
}
}
private withExitDeadline(providerGeneration: number, promise: Promise<void>): Promise<void> {
return new Promise((resolve, reject) => {
const barrier: SshPtyExitBarrier = {
timer: setTimeout(() => {
this.removeExitBarrier(providerGeneration, barrier)
this.dependencies.closeProvider?.(providerGeneration, 'ssh-exit-barrier-timeout')
reject(outputIntakeError('ssh_exit_barrier_timeout'))
}, this.exitBarrierMs),
reject
}
barrier.timer.unref?.()
let barriers = this.exitBarriersByGeneration.get(providerGeneration)
if (!barriers) {
barriers = new Set()
this.exitBarriersByGeneration.set(providerGeneration, barriers)
}
barriers.add(barrier)
void promise.then(
() => {
clearTimeout(barrier.timer)
this.removeExitBarrier(providerGeneration, barrier)
resolve()
},
(error) => {
clearTimeout(barrier.timer)
this.removeExitBarrier(providerGeneration, barrier)
reject(error)
}
)
})
}
private removeExitBarrier(providerGeneration: number, barrier: SshPtyExitBarrier): void {
const barriers = this.exitBarriersByGeneration.get(providerGeneration)
barriers?.delete(barrier)
if (barriers?.size === 0) {
this.exitBarriersByGeneration.delete(providerGeneration)
exitBarriers: this.exitDeadline.activeBarriers
}
}
}
@@ -9,7 +9,9 @@ import type { DesktopProjectionSpan } from './ssh-pty-legacy-projection'
import type {
SshPtyOutputDataEvent,
SshPtyOutputExitEvent,
SshPtyOutputIntakeDependencies
SshPtyOutputIntakeDependencies,
SshPtySourceCancellationProof,
SshPtySourceCancellationRequest
} from './ssh-pty-output-intake-contract'
import { SshPtyRemoteSourceRangeConsumers } from './ssh-pty-remote-source-range-consumers'
import type { SshPtySourceAdmissionReservation } from './ssh-pty-source-obligation-contract'
@@ -133,8 +135,39 @@ export class SshPtyOutputSourceObligations {
}
}
whenPtyTerminal(event: SshPtyOutputExitEvent): Promise<void> {
const identity = this.identityByPty.get(this.ptyKey(event))
return identity ? this.coordinator.whenTerminal(identity) : Promise.resolve()
}
async cancelPty(
event: SshPtyOutputExitEvent,
cancel: (request: SshPtySourceCancellationRequest) => Promise<SshPtySourceCancellationProof>
): Promise<boolean> {
const identity = this.identityByPty.get(this.ptyKey(event))
if (!identity) {
return false
}
const request = this.coordinator.beginExitTimeout(identity)
const proof = await cancel(request)
this.coordinator.applyCancellationProof(identity, proof)
return true
}
applyCancellationProof(
event: SshPtyOutputExitEvent,
proof: SshPtySourceCancellationProof
): boolean {
const identity = this.identityByPty.get(this.ptyKey(event))
if (!identity) {
return false
}
this.coordinator.applyCancellationProof(identity, proof)
return true
}
closeGeneration(providerGeneration: number, reason: string): void {
this.remoteConsumers.closeGeneration(providerGeneration)
this.remoteConsumers.closeGeneration(providerGeneration, reason)
this.coordinator.closeGeneration(providerGeneration, reason)
const prefix = `${providerGeneration}\0`
for (const key of this.openedTokens) {
@@ -83,7 +83,40 @@ describe('SshPtyRemoteSourceRangeConsumers', () => {
expect(progress).toHaveBeenCalledTimes(1)
})
it('atomically transfers remaining mappings on explicit detach', () => {
it.each(['headless', 'renderer'] as const)(
'commits remaining mappings only after a %s snapshot publication',
(source) => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
const replacement = consumers.hooks.reserveReplacement(stream, 4, 'initial-snapshot')
expect(replacement).not.toBeNull()
expect(ledger.obligation('span-1', 'remote:consumer-1')).toMatchObject({
state: 'transferring',
to: 'remote:snapshot:consumer-1'
})
expect(consumers.hooks.commitReplacement(replacement!, { source, seq: 3 })).toBe(false)
expect(ledger.obligation('span-1', 'remote:consumer-1').state).toBe('transferring')
expect(consumers.hooks.commitReplacement(replacement!, { source, seq: 4 })).toBe(true)
expect(ledger.obligation('span-1', 'remote:consumer-1')).toMatchObject({
state: 'transferred',
to: 'remote:snapshot:consumer-1'
})
expect(consumers.requiredConsumers('pty-1')).toEqual(['remote:consumer-1'])
}
)
it('rolls a failed replacement publication back to the live stream obligation', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
@@ -96,13 +129,12 @@ describe('SshPtyRemoteSourceRangeConsumers', () => {
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
consumers.hooks.transfer(stream, [range('span-1')], 'stream-detached')
const replacement = consumers.hooks.reserveReplacement(stream, 4, 'initial-snapshot')
expect(consumers.hooks.rollbackReplacement(replacement!, 'snapshot-write-failed')).toBe(true)
expect(ledger.obligation('span-1', 'remote:consumer-1')).toMatchObject({
state: 'transferred',
to: 'remote:snapshot:consumer-1'
state: 'open'
})
expect(consumers.requiredConsumers('pty-1')).toEqual([])
})
it('settles a split source span only after its complete ordered range is acknowledged', () => {
@@ -139,7 +171,7 @@ describe('SshPtyRemoteSourceRangeConsumers', () => {
expect(ledger.obligation('span-1', 'remote:consumer-1').state).toBe('settled')
})
it('transfers an admitted span even when no encoded mapping was sent', () => {
it('cancels an admitted span on detach without minting a snapshot owner', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
@@ -152,10 +184,127 @@ describe('SshPtyRemoteSourceRangeConsumers', () => {
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
consumers.hooks.transfer(stream, [], 'stream-detached')
consumers.hooks.cancel(stream, [], 'stream-detached')
expect(ledger.obligation('span-1', 'remote:consumer-1')).toMatchObject({
state: 'transferred'
state: 'canceled',
reason: 'stream-detached'
})
expect(consumers.requiredConsumers('pty-1')).toEqual([])
})
it('rolls back a pending replacement before disconnect cancellation', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
const replacement = consumers.hooks.reserveReplacement(stream, 4, 'initial-snapshot')
consumers.hooks.cancel(stream, [], 'connection-closed')
expect(consumers.hooks.commitReplacement(replacement!, { source: 'headless', seq: 4 })).toBe(
false
)
expect(ledger.obligation('span-1', 'remote:consumer-1')).toMatchObject({
state: 'canceled',
reason: 'connection-closed'
})
})
it('rejects stale stream generations without changing the current obligation', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
expect(() =>
consumers.hooks.reserveReplacement(
{ ...stream, streamGeneration: 'stale' },
4,
'initial-snapshot'
)
).toThrow('stale')
expect(ledger.obligation('span-1', 'remote:consumer-1').state).toBe('open')
})
it('rolls back replacement admission before provider-generation close', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
const replacement = consumers.hooks.reserveReplacement(stream, 4, 'initial-snapshot')
consumers.closeGeneration(identity.providerGeneration, 'provider-replaced')
expect(ledger.obligation('span-1', 'remote:consumer-1').state).toBe('open')
expect(consumers.hooks.commitReplacement(replacement!, { source: 'headless', seq: 4 })).toBe(
false
)
})
it('detaches cleanly after cancellation proof reclaims tracked spans', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
ledger.seal(identity)
ledger.beginExitTimeout(identity)
ledger.applyCancellationProof(identity, { sentEndSu: 4, creditedEndSu: 0 })
expect(() => consumers.hooks.settle(stream, [range('span-1')])).not.toThrow()
expect(() => consumers.hooks.cancel(stream, [], 'stream-detached')).not.toThrow()
expect(consumers.requiredConsumers('pty-1')).toEqual([])
})
it('rejects a replacement commit after cancellation proof reclaims its spans', () => {
const ledger = createCoordinator()
const consumers = new SshPtyRemoteSourceRangeConsumers(ledger)
const stream = { ptyId: 'pty-1', consumerId: 'consumer-1', streamGeneration: 'stream-1' }
ledger.open(identity)
consumers.hooks.attach(stream)
const reservation = ledger.reserve(identity, span('span-1'), [
'model',
...consumers.requiredConsumers('pty-1')
])
ledger.commit(reservation)
consumers.trackSpan('pty-1', 'span-1', reservation.requiredConsumers)
const replacement = consumers.hooks.reserveReplacement(stream, 4, 'initial-snapshot')
ledger.seal(identity)
ledger.beginExitTimeout(identity)
ledger.applyCancellationProof(identity, { sentEndSu: 4, creditedEndSu: 0 })
expect(consumers.hooks.commitReplacement(replacement!, { source: 'headless', seq: 4 })).toBe(
false
)
expect(() => consumers.hooks.cancel(stream, [], 'stream-detached')).not.toThrow()
expect(consumers.requiredConsumers('pty-1')).toEqual([])
})
})
@@ -6,8 +6,10 @@ import type {
RemoteTerminalSourceRangeConsumerHooks,
RemoteTerminalSourceRangeStreamIdentity
} from '../runtime/remote-terminal-source-range-consumer'
import type { PtySourceSpan } from '../../shared/pty-source-credit-contract'
import type { SshPtySourceConsumerId } from './ssh-pty-source-obligation-contract'
import type { SshPtySourceObligationCoordinator } from './ssh-pty-source-obligation-coordinator'
import { SshPtyRemoteSourceRangeReplacements } from './ssh-pty-remote-source-range-replacement'
function remoteConsumerId(
identity: RemoteTerminalSourceRangeStreamIdentity
@@ -15,34 +17,35 @@ function remoteConsumerId(
return `remote:${identity.consumerId}`
}
function replacementConsumerId(
identity: RemoteTerminalSourceRangeStreamIdentity
): SshPtySourceConsumerId {
return `remote:snapshot:${identity.consumerId}`
}
function uniqueSpanIds(ranges: readonly TerminalOutputSourceRange[]): string[] {
return Array.from(new Set(ranges.map((range) => range.spanId)))
}
type RemoteConsumerState = {
streamGeneration: string
spanGenerations: Map<string, number>
spans: Map<string, PtySourceSpan>
ackedEndBySpan: Map<string, number>
}
export class SshPtyRemoteSourceRangeConsumers {
private readonly consumersByPty = new Map<string, Map<string, RemoteConsumerState>>()
private readonly replacements: SshPtyRemoteSourceRangeReplacements
constructor(
private readonly coordinator: SshPtySourceObligationCoordinator,
private readonly onProgress: (range: TerminalOutputSourceRange) => void = () => {}
) {}
) {
this.replacements = new SshPtyRemoteSourceRangeReplacements(coordinator)
}
readonly hooks: RemoteTerminalSourceRangeConsumerHooks = {
attach: (identity) => this.attach(identity),
settle: (identity, ranges) => this.settle(identity, ranges),
transfer: (identity, ranges, reason) => this.transfer(identity, ranges, reason),
reserveReplacement: (identity, requiredSeq, reason) =>
this.reserveReplacement(identity, requiredSeq, reason),
commitReplacement: (reservation, publication) =>
this.commitReplacement(reservation, publication),
rollbackReplacement: (reservation, reason) => this.rollbackReplacement(reservation, reason),
cancel: (identity, ranges, reason) => this.cancel(identity, ranges, reason)
}
@@ -61,17 +64,18 @@ export class SshPtyRemoteSourceRangeConsumers {
): void {
for (const [consumerId, state] of this.consumersByPty.get(ptyId) ?? []) {
if (requiredConsumers.includes(`remote:${consumerId}`)) {
state.spanGenerations.set(spanId, this.coordinator.spanIdentity(spanId).providerGeneration)
state.spans.set(spanId, this.coordinator.spanIdentity(spanId))
}
}
}
closeGeneration(providerGeneration: number): void {
closeGeneration(providerGeneration: number, reason: string): void {
this.replacements.closeGeneration(providerGeneration, reason)
for (const consumers of this.consumersByPty.values()) {
for (const state of consumers.values()) {
for (const [spanId, generation] of state.spanGenerations) {
if (generation === providerGeneration) {
state.spanGenerations.delete(spanId)
for (const [spanId, source] of state.spans) {
if (source.providerGeneration === providerGeneration) {
state.spans.delete(spanId)
state.ackedEndBySpan.delete(spanId)
}
}
@@ -90,7 +94,7 @@ export class SshPtyRemoteSourceRangeConsumers {
identity.consumerId,
current ?? {
streamGeneration: identity.streamGeneration,
spanGenerations: new Map(),
spans: new Map(),
ackedEndBySpan: new Map()
}
)
@@ -110,10 +114,15 @@ export class SshPtyRemoteSourceRangeConsumers {
const nextEnds = new Map(state.ackedEndBySpan)
const completed = new Set<string>()
for (const range of ranges) {
if (!state.spanGenerations.has(range.spanId)) {
const source = state.spans.get(range.spanId)
if (!source) {
continue
}
if (!this.coordinator.hasRetainedSpan(range.spanId)) {
state.spans.delete(range.spanId)
nextEnds.delete(range.spanId)
continue
}
const source = this.coordinator.spanIdentity(range.spanId)
const currentEnd = nextEnds.get(range.spanId) ?? source.sourceStartSu
if (
!sameTerminalOutputSourceIdentity(source, range) ||
@@ -129,14 +138,19 @@ export class SshPtyRemoteSourceRangeConsumers {
}
state.ackedEndBySpan = nextEnds
for (const spanId of completed) {
const source = this.coordinator.spanIdentity(spanId)
const source = state.spans.get(spanId)
if (!source || !this.coordinator.hasRetainedSpan(spanId)) {
state.spans.delete(spanId)
state.ackedEndBySpan.delete(spanId)
continue
}
this.coordinator.settle({
identity: source,
spanId,
consumer,
reason: 'remote-frame-ack'
})
state.spanGenerations.delete(spanId)
state.spans.delete(spanId)
state.ackedEndBySpan.delete(spanId)
}
for (const range of ranges) {
@@ -144,40 +158,48 @@ export class SshPtyRemoteSourceRangeConsumers {
}
}
private transfer(
private reserveReplacement(
identity: RemoteTerminalSourceRangeStreamIdentity,
ranges: readonly TerminalOutputSourceRange[],
requiredSeq: number,
reason: string
): void {
): ReturnType<SshPtyRemoteSourceRangeReplacements['reserve']> {
if (!this.isCurrent(identity)) {
throw new Error('ssh_remote_source_range_stale_generation')
}
const consumer = remoteConsumerId(identity)
const replacement = replacementConsumerId(identity)
const state = this.requireState(identity)
const spanIds = Array.from(
new Set([
...state.spanGenerations.keys(),
...uniqueSpanIds(ranges).filter((spanId) => state.spanGenerations.has(spanId))
])
)
for (const spanId of spanIds) {
if (this.coordinator.obligation(spanId, consumer).state !== 'open') {
throw new Error('ssh_remote_source_range_transfer_invalid')
for (const spanId of state.spans.keys()) {
if (!this.coordinator.hasRetainedSpan(spanId)) {
state.spans.delete(spanId)
state.ackedEndBySpan.delete(spanId)
}
}
for (const spanId of spanIds) {
const source = this.coordinator.spanIdentity(spanId)
this.coordinator.beginTransfer({ identity: source, spanId, consumer, reason }, replacement)
}
for (const spanId of spanIds) {
const source = this.coordinator.spanIdentity(spanId)
this.coordinator.commitTransfer({ identity: source, spanId, consumer })
}
this.detachIdentity(identity)
for (const range of ranges) {
this.onProgress(range)
}
const spanIds = Array.from(state.spans.keys())
return this.replacements.reserve(identity, spanIds, requiredSeq, reason)
}
private commitReplacement(
reservation: Parameters<SshPtyRemoteSourceRangeReplacements['commit']>[0],
publication: Parameters<SshPtyRemoteSourceRangeReplacements['commit']>[1]
): boolean {
return this.replacements.commit(
reservation,
publication,
this.isCurrent(reservation.identity),
(spanIds) => {
const state = this.requireState(reservation.identity)
for (const spanId of spanIds) {
state.spans.delete(spanId)
state.ackedEndBySpan.delete(spanId)
}
}
)
}
private rollbackReplacement(
reservation: Parameters<SshPtyRemoteSourceRangeReplacements['rollback']>[0],
reason: string
): boolean {
return this.replacements.rollback(reservation, reason)
}
private cancel(
@@ -188,14 +210,18 @@ export class SshPtyRemoteSourceRangeConsumers {
if (!this.isCurrent(identity)) {
return
}
this.replacements.rollbackIdentity(identity, `${reason}-replacement-aborted`)
const consumer = remoteConsumerId(identity)
const state = this.requireState(identity)
const spanIds = new Set([
...state.spanGenerations.keys(),
...uniqueSpanIds(ranges).filter((spanId) => state.spanGenerations.has(spanId))
...state.spans.keys(),
...uniqueSpanIds(ranges).filter((spanId) => state.spans.has(spanId))
])
for (const spanId of spanIds) {
const source = this.coordinator.spanIdentity(spanId)
const source = state.spans.get(spanId)
if (!source || !this.coordinator.hasRetainedSpan(spanId)) {
continue
}
const transition = { identity: source, spanId, consumer, reason }
if (this.coordinator.beginTransfer(transition, consumer)) {
this.coordinator.cancelTransfer(transition)
@@ -0,0 +1,155 @@
import type {
RemoteTerminalSourceRangeReplacementPublication,
RemoteTerminalSourceRangeReplacementReservation,
RemoteTerminalSourceRangeStreamIdentity
} from '../runtime/remote-terminal-source-range-consumer'
import type { PtySourceSpan } from '../../shared/pty-source-credit-contract'
import type { SshPtySourceConsumerId } from './ssh-pty-source-obligation-contract'
import type { SshPtySourceObligationCoordinator } from './ssh-pty-source-obligation-coordinator'
type ReplacementReservationRecord = {
reservation: RemoteTerminalSourceRangeReplacementReservation
spans: readonly PtySourceSpan[]
consumer: SshPtySourceConsumerId
replacement: SshPtySourceConsumerId
reason: string
}
function remoteConsumerId(
identity: RemoteTerminalSourceRangeStreamIdentity
): SshPtySourceConsumerId {
return `remote:${identity.consumerId}`
}
export class SshPtyRemoteSourceRangeReplacements {
private readonly reservations = new Map<string, ReplacementReservationRecord>()
private nextReservationId = 1
constructor(private readonly coordinator: SshPtySourceObligationCoordinator) {}
reserve(
identity: RemoteTerminalSourceRangeStreamIdentity,
spanIds: readonly string[],
requiredSeq: number,
reason: string
): RemoteTerminalSourceRangeReplacementReservation | null {
if (spanIds.length === 0) {
return null
}
if (!Number.isSafeInteger(requiredSeq) || requiredSeq < 0) {
throw new Error('ssh_remote_source_range_replacement_sequence_invalid')
}
const consumer = remoteConsumerId(identity)
const replacement = `remote:snapshot:${identity.consumerId}` as const
const spans = spanIds.map((spanId) => this.coordinator.spanIdentity(spanId))
for (const { spanId } of spans) {
if (this.coordinator.obligation(spanId, consumer).state !== 'open') {
throw new Error('ssh_remote_source_range_transfer_invalid')
}
}
for (const source of spans) {
const { spanId } = source
this.coordinator.beginTransfer({ identity: source, spanId, consumer, reason }, replacement)
}
const reservation = Object.freeze({
reservationId: `remote-source-replacement:${this.nextReservationId++}`,
identity: Object.freeze({ ...identity }),
requiredSeq
})
this.reservations.set(reservation.reservationId, {
reservation,
spans: Object.freeze(spans),
consumer,
replacement,
reason
})
return reservation
}
commit(
reservation: RemoteTerminalSourceRangeReplacementReservation,
publication: RemoteTerminalSourceRangeReplacementPublication,
isCurrent: boolean,
onCommitted: (spanIds: readonly string[]) => void
): boolean {
const record = this.reservations.get(reservation.reservationId)
if (
!record ||
record.reservation !== reservation ||
!isCurrent ||
!Number.isSafeInteger(publication.seq) ||
publication.seq < reservation.requiredSeq ||
(publication.source !== 'headless' && publication.source !== 'renderer')
) {
return false
}
if (
record.spans.some(({ spanId }) => {
if (!this.coordinator.hasRetainedSpan(spanId)) {
return true
}
const obligation = this.coordinator.obligation(spanId, record.consumer)
return obligation.state !== 'transferring' || obligation.to !== record.replacement
})
) {
this.reservations.delete(reservation.reservationId)
return false
}
for (const source of record.spans) {
const { spanId } = source
if (
!this.coordinator.commitTransfer({ identity: source, spanId, consumer: record.consumer })
) {
throw new Error('ssh_remote_source_range_replacement_commit_invalid')
}
}
this.reservations.delete(reservation.reservationId)
onCommitted(record.spans.map(({ spanId }) => spanId))
return true
}
rollback(reservation: RemoteTerminalSourceRangeReplacementReservation, reason: string): boolean {
const record = this.reservations.get(reservation.reservationId)
if (!record || record.reservation !== reservation) {
return false
}
this.reservations.delete(reservation.reservationId)
let rolledBack = true
for (const source of record.spans) {
const { spanId } = source
if (!this.coordinator.hasRetainedSpan(spanId)) {
continue
}
rolledBack =
this.coordinator.rollbackTransfer({
identity: source,
spanId,
consumer: record.consumer,
reason
}) && rolledBack
}
return rolledBack
}
rollbackIdentity(identity: RemoteTerminalSourceRangeStreamIdentity, reason: string): void {
for (const record of Array.from(this.reservations.values())) {
if (
record.reservation.identity.ptyId === identity.ptyId &&
record.reservation.identity.consumerId === identity.consumerId &&
record.reservation.identity.streamGeneration === identity.streamGeneration
) {
this.rollback(record.reservation, reason)
}
}
}
closeGeneration(providerGeneration: number, reason: string): void {
for (const record of Array.from(this.reservations.values())) {
if (
record.spans.some(({ providerGeneration: generation }) => generation === providerGeneration)
) {
this.rollback(record.reservation, `${reason}-replacement-aborted`)
}
}
}
}
@@ -1,4 +1,5 @@
import {
ptySourceDeliveryKey,
samePtySourceDelivery,
type PtySourceDeliveryIdentity,
type PtySourceSpan
@@ -22,9 +23,20 @@ export type SshPtySourceObligationTransition = Readonly<{
reason: string
}>
type TerminalWaiter = {
resolve: () => void
reject: (error: Error) => void
}
type TerminalWaiterGroup = {
identity: PtySourceDeliveryIdentity
waiters: Set<TerminalWaiter>
}
export class SshPtySourceObligationCoordinator {
private readonly ledger: SshPtySourceObligationLedger
private readonly acknowledgements: SshPtySourceAckCoalescer
private readonly terminalWaiters = new Map<string, TerminalWaiterGroup>()
private disposed = false
constructor(options: SshPtySourceAckCoalescerOptions) {
@@ -56,7 +68,11 @@ export class SshPtySourceObligationCoordinator {
}
rollbackCommitted(reservation: SshPtySourceAdmissionReservation): boolean {
return this.ledger.rollbackCommitted(reservation)
const rolledBack = this.ledger.rollbackCommitted(reservation)
if (rolledBack) {
this.maybeResolveTerminal(reservation.span)
}
return rolledBack
}
settle(transition: SshPtySourceObligationTransition): boolean {
@@ -89,7 +105,15 @@ export class SshPtySourceObligationCoordinator {
return changed
}
rollbackTransfer(transition: SshPtySourceObligationTransition): boolean {
this.requireSpanIdentity(transition)
return this.ledger.rollbackTransfer(transition.spanId, transition.consumer)
}
seal(identity: PtySourceDeliveryIdentity): void {
if (this.ledger.snapshot(identity).state === 'sealed-unsettled') {
return
}
this.ledger.seal(identity)
}
@@ -98,6 +122,21 @@ export class SshPtySourceObligationCoordinator {
this.ledger.markExitPublished(identity)
}
whenTerminal(identity: PtySourceDeliveryIdentity): Promise<void> {
if (this.isTerminal(identity)) {
return Promise.resolve()
}
const key = ptySourceDeliveryKey(identity)
let group = this.terminalWaiters.get(key)
if (!group) {
group = { identity: Object.freeze({ ...identity }), waiters: new Set() }
this.terminalWaiters.set(key, group)
}
return new Promise((resolve, reject) => {
group!.waiters.add({ resolve, reject })
})
}
beginExitTimeout(identity: PtySourceDeliveryIdentity) {
return this.ledger.beginExitTimeout(identity)
}
@@ -106,10 +145,26 @@ export class SshPtySourceObligationCoordinator {
identity: PtySourceDeliveryIdentity,
proof: Readonly<{ sentEndSu: number; creditedEndSu: number }>
): void {
const snapshot = this.ledger.snapshot(identity)
if (
snapshot.state === 'closed' &&
snapshot.receivedEndSu === proof.sentEndSu &&
snapshot.ackPublishedEndSu === proof.creditedEndSu
) {
return
}
this.ledger.applyCancellationProof(identity, proof)
this.rejectWaiters(
(group) => samePtySourceDelivery(group.identity, identity),
new Error('ssh_source_delivery_canceled')
)
}
closeGeneration(providerGeneration: number, reason: string): number {
this.rejectWaiters(
(group) => group.identity.providerGeneration === providerGeneration,
new Error(reason)
)
const closed = this.ledger.closeGeneration(providerGeneration, reason)
this.acknowledgements.cancelGeneration(providerGeneration, reason)
return closed
@@ -131,6 +186,10 @@ export class SshPtySourceObligationCoordinator {
return this.ledger.spanIdentity(spanId)
}
hasRetainedSpan(spanId: string): boolean {
return this.ledger.hasRetainedSpan(spanId)
}
flushAcknowledgements(): void {
this.acknowledgements.flush()
}
@@ -140,6 +199,7 @@ export class SshPtySourceObligationCoordinator {
return
}
this.disposed = true
this.rejectWaiters(() => true, new Error(reason ?? 'SSH PTY source obligations disposed'))
this.ledger.closeAll(reason ?? 'SSH PTY source obligation coordinator disposed')
this.acknowledgements.dispose(reason)
}
@@ -149,6 +209,41 @@ export class SshPtySourceObligationCoordinator {
if (publication) {
this.acknowledgements.enqueue(publication)
}
this.maybeResolveTerminal(identity)
}
private isTerminal(identity: PtySourceDeliveryIdentity): boolean {
const snapshot = this.ledger.snapshot(identity)
return (
snapshot.obligationsTerminalEndSu === snapshot.receivedEndSu &&
snapshot.ackQueuedEndSu === snapshot.receivedEndSu
)
}
private maybeResolveTerminal(identity: PtySourceDeliveryIdentity): void {
if (!this.isTerminal(identity)) {
return
}
const group = this.terminalWaiters.get(ptySourceDeliveryKey(identity))
if (!group) {
return
}
this.terminalWaiters.delete(ptySourceDeliveryKey(identity))
for (const waiter of group.waiters) {
waiter.resolve()
}
}
private rejectWaiters(predicate: (group: TerminalWaiterGroup) => boolean, error: Error): void {
for (const [key, group] of this.terminalWaiters) {
if (!predicate(group)) {
continue
}
this.terminalWaiters.delete(key)
for (const waiter of group.waiters) {
waiter.reject(error)
}
}
}
private requireSpanIdentity(
@@ -37,6 +37,7 @@ import {
cancelSourceObligationTransfer,
commitSourceObligationTransfer,
modelAcceptedSourceEnd,
rollbackSourceObligationTransfer,
transitionOpenSourceObligation
} from './ssh-pty-source-obligation-transitions'
@@ -160,6 +161,10 @@ export class SshPtySourceObligationLedger {
return cancelSourceObligationTransfer(this.spanOwners, spanId, consumer, reason)
}
rollbackTransfer(spanId: string, consumer: SshPtySourceConsumerId): boolean {
return rollbackSourceObligationTransfer(this.spanOwners, spanId, consumer)
}
queueAck(identity: PtySourceDeliveryIdentity): SshPtySourceAckPublication | null {
const token = this.requireToken(identity)
if (token.obligationsTerminalEndSu <= token.ackQueuedEndSu) {
@@ -263,6 +268,10 @@ export class SshPtySourceObligationLedger {
return requireSourceSpan(this.spanOwners, spanId).span.span
}
hasRetainedSpan(spanId: string): boolean {
return this.spanOwners.has(spanId)
}
private transitionOpen(
spanId: string,
consumer: SshPtySourceConsumerId,
@@ -56,6 +56,19 @@ export function cancelSourceObligationTransfer(
return true
}
export function rollbackSourceObligationTransfer(
spanOwners: ReadonlyMap<string, TokenRecord>,
spanId: string,
consumer: SshPtySourceConsumerId
): boolean {
const { span } = requireSourceSpan(spanOwners, spanId)
if (span.obligations.get(consumer)?.state !== 'transferring') {
return false
}
span.obligations.set(consumer, Object.freeze({ state: 'open' }))
return true
}
export function modelAcceptedSourceEnd(token: TokenRecord): number {
let acceptedEndSu = token.ackPublishedEndSu
for (const record of token.spans) {
+18 -10
View File
@@ -684,10 +684,14 @@ describe('SshPtyProvider', () => {
replay: 'restored output',
incarnationId: 'incarnation-reconnect'
})
expect(mux.request).toHaveBeenCalledWith('pty.attach', {
id: 'pty-1',
suppressReplayNotification: true
})
expect(mux.request).toHaveBeenCalledWith(
'pty.attach',
{
id: 'pty-1',
suppressReplayNotification: true
},
{ timeoutMs: 10_000 }
)
})
it('keeps missing incarnation compatible with an old relay', async () => {
@@ -712,12 +716,16 @@ describe('SshPtyProvider', () => {
tabId: 'tab-a'
})
expect(mux.request).toHaveBeenCalledWith('pty.attach', {
id: 'pty-1',
suppressReplayNotification: true,
expectedPaneKey: 'tab-a:leaf-a',
expectedTabId: 'tab-a'
})
expect(mux.request).toHaveBeenCalledWith(
'pty.attach',
{
id: 'pty-1',
suppressReplayNotification: true,
expectedPaneKey: 'tab-a:leaf-a',
expectedTabId: 'tab-a'
},
{ timeoutMs: 10_000 }
)
})
it('write sends pty.data notification', () => {
+8 -7
View File
@@ -198,14 +198,15 @@ export class SshPtyProvider implements IPtyProvider {
// be filtered before they reach the renderer. The expected identity lets the
// relay reject a cross-generation id collision instead of reattaching this
// lease to a different pane's freshly spawned PTY.
const params = {
id: this.toRelayPtyId(id),
suppressReplayNotification: true,
...(sourceRecovery ? { sourceRecovery } : {}),
...(expected?.paneKey ? { expectedPaneKey: expected.paneKey } : {}),
...(expected?.tabId ? { expectedTabId: expected.tabId } : {})
}
const result = parseSshPtyAttachResult(
await this.mux.request('pty.attach', {
id: this.toRelayPtyId(id),
suppressReplayNotification: true,
...(sourceRecovery ? { sourceRecovery } : {}),
...(expected?.paneKey ? { expectedPaneKey: expected.paneKey } : {}),
...(expected?.tabId ? { expectedTabId: expected.tabId } : {})
})
await this.mux.request('pty.attach', params, { timeoutMs: 10_000 })
)
this.outputState.rememberPtyIncarnation(this.toRelayPtyId(id), result.incarnationId)
return result
+31 -4
View File
@@ -18,6 +18,8 @@ import type { TerminalOscColorQueryReplyColors } from '../../shared/terminal-osc
import type { TerminalOutputSourceRange } from '../../shared/terminal-output-source-range'
import type {
RemoteTerminalSourceRangeConsumerHooks,
RemoteTerminalSourceRangeReplacementPublication,
RemoteTerminalSourceRangeReplacementReservation,
RemoteTerminalSourceRangeStreamIdentity
} from './remote-terminal-source-range-consumer'
import {
@@ -8887,12 +8889,37 @@ export class OrcaRuntimeService {
this.remoteTerminalSourceRangeConsumerHooks?.settle(identity, ranges)
}
transferRemoteTerminalSourceRanges(
reserveRemoteTerminalSourceRangeReplacement(
identity: RemoteTerminalSourceRangeStreamIdentity,
ranges: readonly TerminalOutputSourceRange[],
requiredSeq: number,
reason: string
): void {
this.remoteTerminalSourceRangeConsumerHooks?.transfer(identity, ranges, reason)
): RemoteTerminalSourceRangeReplacementReservation | null {
return (
this.remoteTerminalSourceRangeConsumerHooks?.reserveReplacement(
identity,
requiredSeq,
reason
) ?? null
)
}
commitRemoteTerminalSourceRangeReplacement(
reservation: RemoteTerminalSourceRangeReplacementReservation,
publication: RemoteTerminalSourceRangeReplacementPublication
): boolean {
return (
this.remoteTerminalSourceRangeConsumerHooks?.commitReplacement(reservation, publication) ??
false
)
}
rollbackRemoteTerminalSourceRangeReplacement(
reservation: RemoteTerminalSourceRangeReplacementReservation,
reason: string
): boolean {
return (
this.remoteTerminalSourceRangeConsumerHooks?.rollbackReplacement(reservation, reason) ?? false
)
}
cancelRemoteTerminalSourceRanges(
@@ -6,17 +6,36 @@ export type RemoteTerminalSourceRangeStreamIdentity = Readonly<{
streamGeneration: string
}>
export type RemoteTerminalSourceRangeReplacementReservation = Readonly<{
reservationId: string
identity: RemoteTerminalSourceRangeStreamIdentity
requiredSeq: number
}>
export type RemoteTerminalSourceRangeReplacementPublication = Readonly<{
source: 'headless' | 'renderer'
seq: number
}>
export type RemoteTerminalSourceRangeConsumerHooks = {
attach: (identity: RemoteTerminalSourceRangeStreamIdentity) => boolean
settle: (
identity: RemoteTerminalSourceRangeStreamIdentity,
ranges: readonly TerminalOutputSourceRange[]
) => void
transfer: (
reserveReplacement: (
identity: RemoteTerminalSourceRangeStreamIdentity,
ranges: readonly TerminalOutputSourceRange[],
requiredSeq: number,
reason: string
) => void
) => RemoteTerminalSourceRangeReplacementReservation | null
commitReplacement: (
reservation: RemoteTerminalSourceRangeReplacementReservation,
publication: RemoteTerminalSourceRangeReplacementPublication
) => boolean
rollbackReplacement: (
reservation: RemoteTerminalSourceRangeReplacementReservation,
reason: string
) => boolean
cancel: (
identity: RemoteTerminalSourceRangeStreamIdentity,
ranges: readonly TerminalOutputSourceRange[],
+87 -48
View File
@@ -69,6 +69,7 @@ import {
sameTerminalOutputSourceIdentity,
type TerminalOutputSourceRange
} from '../../../../shared/terminal-output-source-range'
import type { RemoteTerminalSourceRangeReplacementReservation } from '../../remote-terminal-source-range-consumer'
const REQUESTED_SNAPSHOT_BYTE_BUDGET = 2 * 1024 * 1024
const TERMINAL_OUTPUT_FLUSH_MS = 5
@@ -124,6 +125,7 @@ type TerminalMultiplexStream = {
streamGeneration: string
sourceRangeLedger: TerminalSourceRangeLedger | null
sourceRangeConsumerAttached: boolean
sourceRangeReplacement: RemoteTerminalSourceRangeReplacementReservation | null
ackInFlightBytes: number
ackWindowBytes: number
supportsDesktopViewportClaims: boolean
@@ -599,36 +601,45 @@ async function serializeBudgetedRequestedSnapshot(
}
function sendSnapshotFrames(
sendFrame: (opcode: TerminalStreamOpcode, payload?: Uint8Array<ArrayBufferLike>) => void,
sendFrame: (
opcode: TerminalStreamOpcode,
payload?: Uint8Array<ArrayBufferLike>
) => boolean | void,
options: SnapshotFrameOptions
): { bytes: number; chunks: number } {
sendFrame(
TerminalStreamOpcode.SnapshotStart,
encodeTerminalStreamJson({
kind: options.kind,
cols: options.cols,
rows: options.rows,
requestId: options.requestId,
displayMode: options.displayMode,
reason: options.reason,
seq: options.seq,
cwd: options.cwd,
source: options.source,
oscLinks: options.oscLinks,
pendingEscapeTailAnsi: options.pendingEscapeTailAnsi,
truncated: options.truncated === true,
truncatedByByteBudget: options.truncatedByByteBudget === true
})
)
): { bytes: number; chunks: number; published: boolean } {
if (
sendFrame(
TerminalStreamOpcode.SnapshotStart,
encodeTerminalStreamJson({
kind: options.kind,
cols: options.cols,
rows: options.rows,
requestId: options.requestId,
displayMode: options.displayMode,
reason: options.reason,
seq: options.seq,
cwd: options.cwd,
source: options.source,
oscLinks: options.oscLinks,
pendingEscapeTailAnsi: options.pendingEscapeTailAnsi,
truncated: options.truncated === true,
truncatedByByteBudget: options.truncatedByByteBudget === true
})
) === false
) {
return { bytes: 0, chunks: 0, published: false }
}
let chunks = 0
let bytes = 0
for (const chunk of iterateTerminalStreamTextPayloads(options.data)) {
if (sendFrame(TerminalStreamOpcode.SnapshotChunk, chunk) === false) {
return { bytes, chunks, published: false }
}
chunks++
bytes += chunk.byteLength
sendFrame(TerminalStreamOpcode.SnapshotChunk, chunk)
}
sendFrame(TerminalStreamOpcode.SnapshotEnd)
return { bytes, chunks }
const published = sendFrame(TerminalStreamOpcode.SnapshotEnd) !== false
return { bytes, chunks, published }
}
async function serializeBudgetedMobileSnapshot(
@@ -1863,16 +1874,9 @@ export const TERMINAL_METHODS: RpcAnyMethod[] = [
const transfer = ledger.beginTransfer()
const ranges = transfer.frames.flatMap((frame) => frame.sourceRanges)
try {
runtime.transferRemoteTerminalSourceRanges(identity, ranges, reason)
runtime.cancelRemoteTerminalSourceRanges(identity, ranges, reason)
} finally {
transfer.commit()
} catch {
transfer.rollback()
const cancellation = ledger.beginTransfer()
try {
runtime.cancelRemoteTerminalSourceRanges(identity, ranges, `${reason}-transfer-failed`)
} finally {
cancellation.commit()
}
}
}
const detachStream = (
@@ -2319,6 +2323,7 @@ export const TERMINAL_METHODS: RpcAnyMethod[] = [
streamGeneration,
sourceRangeLedger: sourceRangeConsumerAttached ? sourceRangeLedger : null,
sourceRangeConsumerAttached,
sourceRangeReplacement: null,
ackInFlightBytes: 0,
ackWindowBytes: TERMINAL_MULTIPLEX_ACK_STREAM_INITIAL_WINDOW_BYTES,
supportsDesktopViewportClaims: request.capabilities?.desktopViewportClaims === 1,
@@ -2410,6 +2415,18 @@ export const TERMINAL_METHODS: RpcAnyMethod[] = [
return
}
const replacementIdentity = {
ptyId,
consumerId: stream.remoteDesktopSubscriptionKey,
streamGeneration: stream.streamGeneration
}
stream.sourceRangeReplacement = stream.ackOutputSourceRanges
? runtime.reserveRemoteTerminalSourceRangeReplacement(
replacementIdentity,
runtime.getPtyOutputSequence(ptyId),
'initial-snapshot'
)
: null
let read = await runtime.readTerminal(request.terminal)
let serialized = await serializeBudgetedMobileSnapshot(runtime, ptyId, isMobile)
if (closed || streams.get(request.streamId) !== stream) {
@@ -2455,22 +2472,44 @@ export const TERMINAL_METHODS: RpcAnyMethod[] = [
initialOutputOverflowed ||
(serialized ? read.truncated : isTerminalReadPayloadIncomplete(read))
})
sendSnapshotFrames((opcode, payload) => sendFrame(request.streamId, opcode, payload), {
kind: 'scrollback',
cols: serialized?.cols ?? size?.cols ?? 80,
rows: serialized?.rows ?? size?.rows ?? 24,
displayMode,
seq: snapshotFrameSeq,
cwd: serialized?.cwd,
truncated:
initialOutputOverflowed ||
(serialized ? read.truncated : isTerminalReadPayloadIncomplete(read)),
truncatedByByteBudget: serialized?.truncatedByByteBudget,
source: serialized?.source,
oscLinks: serialized?.oscLinks,
pendingEscapeTailAnsi: serialized?.pendingEscapeTailAnsi,
data: serialized?.data ?? (read.tail.length > 0 ? `${read.tail.join('\r\n')}\r\n` : '')
})
const snapshotPublication = sendSnapshotFrames(
(opcode, payload) => sendFrame(request.streamId, opcode, payload),
{
kind: 'scrollback',
cols: serialized?.cols ?? size?.cols ?? 80,
rows: serialized?.rows ?? size?.rows ?? 24,
displayMode,
seq: snapshotFrameSeq,
cwd: serialized?.cwd,
truncated:
initialOutputOverflowed ||
(serialized ? read.truncated : isTerminalReadPayloadIncomplete(read)),
truncatedByByteBudget: serialized?.truncatedByByteBudget,
source: serialized?.source,
oscLinks: serialized?.oscLinks,
pendingEscapeTailAnsi: serialized?.pendingEscapeTailAnsi,
data:
serialized?.data ?? (read.tail.length > 0 ? `${read.tail.join('\r\n')}\r\n` : '')
}
)
const replacement = stream.sourceRangeReplacement
stream.sourceRangeReplacement = null
if (replacement) {
const committed =
snapshotPublication.published &&
serialized?.source !== undefined &&
typeof serialized.seq === 'number' &&
runtime.commitRemoteTerminalSourceRangeReplacement(replacement, {
source: serialized.source,
seq: serialized.seq
})
if (!committed) {
runtime.rollbackRemoteTerminalSourceRangeReplacement(
replacement,
'initial-snapshot-unpublished'
)
}
}
// Why: baseline for resize re-stream gating; the client already rewrapped to these cols via the initial snapshot replay.
stream.lastResizeCols = serialized?.cols ?? size?.cols
stream.buffering = false
+135 -4
View File
@@ -33,6 +33,7 @@ function stubRuntime(overrides: Partial<OrcaRuntimeService> = {}): OrcaRuntimeSe
isPtyResizeDrivenRemotely: vi.fn().mockReturnValue(false),
getRemoteDesktopFitHold: vi.fn().mockReturnValue({ mode: 'desktop-fit', cols: 120, rows: 40 }),
isRemoteDesktopViewerOwner: vi.fn().mockReturnValue(false),
getPtyOutputSequence: vi.fn().mockReturnValue(0),
...overrides
} as OrcaRuntimeService
}
@@ -138,15 +139,145 @@ function sendDesktopMultiplexSubscribe(
}
describe('terminal multiplex RPC', () => {
it.each(['headless', 'renderer'] as const)(
'commits a source-range replacement only after the %s snapshot publishes',
async (source) => {
const reservation = {
reservationId: 'replacement-1',
identity: {
ptyId: 'pty-1',
consumerId: 'multiplex:conn-desktop-first-paint:7',
streamGeneration: 'generation'
},
requiredSeq: 0
}
const reserve = vi.fn(() => reservation)
const commit = vi.fn(() => true)
const rollback = vi.fn(() => true)
const harness = startDesktopMultiplexSubscribe({
attachRemoteTerminalSourceRangeConsumer: vi.fn(() => true),
reserveRemoteTerminalSourceRangeReplacement: reserve,
commitRemoteTerminalSourceRangeReplacement: commit,
rollbackRemoteTerminalSourceRangeReplacement: rollback,
cancelRemoteTerminalSourceRanges: vi.fn(),
serializeTerminalBuffer: vi
.fn()
.mockResolvedValue({ data: 'snapshot', cols: 120, rows: 40, source, seq: 4 })
})
await vi.waitFor(() => expect(harness.handlers.has(0)).toBe(true))
harness.handlers.get(0)?.(
decodeTerminalStreamFrame(
encodeTerminalStreamFrame({
opcode: TerminalStreamOpcode.Subscribe,
streamId: 0,
seq: 1,
payload: encodeTerminalStreamJson({
streamId: 7,
terminal: 'terminal-1',
client: { id: 'desktop-1', type: 'desktop' },
capabilities: { ackOutput: 1, ackOutputSourceRanges: 1 }
})
})
)!
)
await vi.waitFor(() => expect(commit).toHaveBeenCalledOnce())
expect(reserve).toHaveBeenCalledWith(expect.any(Object), 0, 'initial-snapshot')
expect(commit).toHaveBeenCalledWith(reservation, { source, seq: 4 })
expect(rollback).not.toHaveBeenCalled()
const snapshotEndIndex = harness.binaryFrames.findIndex(
(bytes) => decodeTerminalStreamFrame(bytes)?.opcode === TerminalStreamOpcode.SnapshotEnd
)
expect(snapshotEndIndex).toBeGreaterThanOrEqual(0)
harness.cleanups.get('terminal-multiplex:conn-desktop-first-paint')?.()
await harness.dispatchPromise
}
)
it('rolls back replacement admission when snapshot publication is refused', async () => {
const reservation = {
reservationId: 'replacement-1',
identity: {
ptyId: 'pty-1',
consumerId: 'multiplex:conn-desktop-first-paint:7',
streamGeneration: 'generation'
},
requiredSeq: 0
}
const commit = vi.fn(() => true)
const rollback = vi.fn(() => true)
const harness = startDesktopMultiplexSubscribe(
{
attachRemoteTerminalSourceRangeConsumer: vi.fn(() => true),
reserveRemoteTerminalSourceRangeReplacement: vi.fn(() => reservation),
commitRemoteTerminalSourceRangeReplacement: commit,
rollbackRemoteTerminalSourceRangeReplacement: rollback,
cancelRemoteTerminalSourceRanges: vi.fn(),
serializeTerminalBuffer: vi.fn().mockResolvedValue({
data: 'snapshot',
cols: 120,
rows: 40,
source: 'headless',
seq: 4
})
},
undefined,
(bytes) => decodeTerminalStreamFrame(bytes)?.opcode !== TerminalStreamOpcode.SnapshotEnd
)
await vi.waitFor(() => expect(harness.handlers.has(0)).toBe(true))
harness.handlers.get(0)?.(
decodeTerminalStreamFrame(
encodeTerminalStreamFrame({
opcode: TerminalStreamOpcode.Subscribe,
streamId: 0,
seq: 1,
payload: encodeTerminalStreamJson({
streamId: 7,
terminal: 'terminal-1',
client: { id: 'desktop-1', type: 'desktop' },
capabilities: { ackOutput: 1, ackOutputSourceRanges: 1 }
})
})
)!
)
await harness.dispatchPromise
expect(commit).not.toHaveBeenCalled()
expect(rollback).toHaveBeenCalledWith(reservation, 'initial-snapshot-unpublished')
})
it('keeps legacy multiplex clients outside source replacement admission', async () => {
const attach = vi.fn(() => true)
const reserve = vi.fn()
const harness = startDesktopMultiplexSubscribe({
attachRemoteTerminalSourceRangeConsumer: attach,
reserveRemoteTerminalSourceRangeReplacement: reserve
})
await vi.waitFor(() => expect(harness.handlers.has(0)).toBe(true))
sendDesktopMultiplexSubscribe(harness.handlers)
await vi.waitFor(() =>
expect(
harness.messages.some((message) => JSON.parse(message).result?.type === 'subscribed')
).toBe(true)
)
expect(attach).not.toHaveBeenCalled()
expect(reserve).not.toHaveBeenCalled()
harness.cleanups.get('terminal-multiplex:conn-desktop-first-paint')?.()
await harness.dispatchPromise
})
it('settles negotiated source ranges only at accepted encoded boundaries', async () => {
let dataListener: ((data: string, meta?: RuntimeTerminalDataMeta) => void) | null = null
const settle = vi.fn()
const transfer = vi.fn()
const cancel = vi.fn()
const harness = startDesktopMultiplexSubscribe({
attachRemoteTerminalSourceRangeConsumer: vi.fn(() => true),
settleRemoteTerminalSourceRanges: settle,
transferRemoteTerminalSourceRanges: transfer,
cancelRemoteTerminalSourceRanges: vi.fn(),
reserveRemoteTerminalSourceRangeReplacement: vi.fn(() => null),
commitRemoteTerminalSourceRangeReplacement: vi.fn(() => false),
rollbackRemoteTerminalSourceRangeReplacement: vi.fn(() => false),
cancelRemoteTerminalSourceRanges: cancel,
subscribeToTerminalData: vi.fn((_ptyId, listener) => {
dataListener = listener
return vi.fn()
@@ -263,7 +394,7 @@ describe('terminal multiplex RPC', () => {
})
)!
)
expect(transfer).toHaveBeenCalledWith(expect.any(Object), [], 'stream-detached')
expect(cancel).toHaveBeenCalledWith(expect.any(Object), [], 'stream-detached')
})
it.each(['refuses', 'throws'] as const)(
+1 -1
View File
@@ -6,7 +6,7 @@ export const MAX_MESSAGE_SIZE = 16 * 1024 * 1024
export const FRAME_DECODER_MAX_FRAMES_PER_TURN = 64
export const FRAME_DECODER_MAX_BYTES_PER_TURN = MAX_MESSAGE_SIZE + HEADER_LENGTH
export const FRAME_DECODER_MAX_TURN_MS = 4,
FRAME_DECODER_MAX_RETAINED_BYTES = 2 * (MAX_MESSAGE_SIZE + HEADER_LENGTH)
FRAME_DECODER_MAX_RETAINED_BYTES = MAX_MESSAGE_SIZE + HEADER_LENGTH + 1024 * 1024
export class FrameDecoder {
private chunks: Buffer[] = []
@@ -41,6 +41,27 @@ function frame(id: number, payload = `${id}`): Buffer {
}
describe('FrameDecoder bounded turns', () => {
it('retains at most one maximum frame plus one MiB of partial input', () => {
expect(FRAME_DECODER_MAX_RETAINED_BYTES).toBe(MAX_MESSAGE_SIZE + HEADER_LENGTH + 1024 * 1024)
const maximumFrame = encodeFrame(MessageType.Regular, 1, 0, Buffer.alloc(MAX_MESSAGE_SIZE))
const acceptedError = vi.fn()
const accepted = new FrameDecoder(vi.fn(), acceptedError)
accepted.feed(Buffer.concat([maximumFrame, Buffer.alloc(1024 * 1024)]))
expect(acceptedError).not.toHaveBeenCalled()
expect(accepted.drain()).toHaveLength(1024 * 1024)
const excessError = vi.fn()
const excess = new FrameDecoder(vi.fn(), excessError)
excess.feed(Buffer.concat([maximumFrame, Buffer.alloc(1024 * 1024 + 1)]))
expect(excessError).toHaveBeenCalledWith(
expect.objectContaining({ message: expect.stringContaining('retained-input') })
)
expect(excess.drain()).toHaveLength(0)
})
it('fails closed when one delivered chunk exceeds retained-input capacity', () => {
const onError = vi.fn()
const decoder = new FrameDecoder(vi.fn(), onError)
@@ -71,6 +71,7 @@ type MuxInternals = {
disposeHandlers: unknown[]
lastReceivedAt: number
unackedTimestamps: Map<number, number>
writerSaturated: boolean
}
function getMuxInternals(instance: SshChannelMultiplexer): MuxInternals {
@@ -330,6 +331,43 @@ describe('SshChannelMultiplexer', () => {
vi.advanceTimersByTime(25_000)
expect(mux.isDisposed()).toBe(true)
})
it('suppresses false death while locally saturated and rebases both clocks on drain', () => {
mux.dispose()
let drain = (): void => {}
const written: Buffer[] = []
const saturatedTransport: MultiplexerTransport = {
write: (data) => {
written.push(data)
return false
},
supportsWriteSettlement: true,
onDrain: (callback) => {
drain = callback
},
onData: vi.fn(),
onClose: vi.fn()
}
mux = new SshChannelMultiplexer(saturatedTransport)
vi.advanceTimersByTime(5_000)
expect(getMuxInternals(mux).writerSaturated).toBe(true)
vi.advanceTimersByTime(25_000)
expect(mux.isDisposed()).toBe(false)
expect(written).toHaveLength(1)
drain()
const resumedAt = Date.now()
const internals = getMuxInternals(mux)
expect(internals.writerSaturated).toBe(false)
expect(internals.lastReceivedAt).toBe(resumedAt)
expect(new Set(internals.unackedTimestamps.values())).toEqual(new Set([resumedAt]))
vi.advanceTimersByTime(20_000)
expect(mux.isDisposed()).toBe(false)
vi.advanceTimersByTime(5_000)
expect(mux.isDisposed()).toBe(true)
})
})
describe('wake guard (timer pause across system sleep, #7773)', () => {
+18 -5
View File
@@ -61,6 +61,7 @@ export class SshChannelMultiplexer {
private connectionHealthTimer: ReturnType<typeof setInterval> | null = null
private disposed = false
private decoderReadPaused = false
private writerSaturated = false
// Track the oldest unacked outgoing message timestamp
private unackedTimestamps = new Map<number, number>()
@@ -71,8 +72,10 @@ export class SshChannelMultiplexer {
constructor(transport: MultiplexerTransport) {
this.transport = transport
this.writer = new SshMultiplexerTransportWriter(transport, (error) =>
this.handleProtocolError(error)
this.writer = new SshMultiplexerTransportWriter(
transport,
(error) => this.handleProtocolError(error),
(saturated) => this.handleWriterSaturationChange(saturated)
)
this.decoder = new FrameDecoder(
@@ -369,10 +372,13 @@ export class SshChannelMultiplexer {
if (this.disposed) {
return
}
const seq = this.nextOutgoingSeq++
const seq = this.nextOutgoingSeq
const frame = encodeKeepAliveFrame(seq, this.highestReceivedSeq)
if (!this.writer.enqueue(frame, 'liveness')) {
return
}
this.nextOutgoingSeq++
this.trackOutgoingTimestamp(seq, true)
this.writer.enqueue(frame, 'control')
}
private handleFrame(frame: DecodedFrame): void {
@@ -528,7 +534,7 @@ export class SshChannelMultiplexer {
this.sendKeepAlive()
if (this.disposed || resumedAfterWake || this.decoderReadPaused) {
if (this.disposed || resumedAfterWake || this.decoderReadPaused || this.writerSaturated) {
return
}
@@ -590,6 +596,13 @@ export class SshChannelMultiplexer {
}
}
private handleWriterSaturationChange(saturated: boolean): void {
this.writerSaturated = saturated
if (!saturated && !this.disposed) {
this.rebaseHealthClocks(Date.now())
}
}
private rebaseHealthClocks(now: number): void {
this.lastReceivedAt = now
for (const seq of this.unackedTimestamps.keys()) {
@@ -70,6 +70,44 @@ describe('SshMultiplexerTransportWriter', () => {
expect(failed).not.toHaveBeenCalled()
})
it('allows one coalesced liveness bypass per saturated write settlement', () => {
const harness = transportHarness([false, true, true, true])
const writer = new SshMultiplexerTransportWriter(harness.transport, vi.fn())
writer.enqueue(Buffer.from('ordinary-1'), 'ordinary')
writer.enqueue(Buffer.from('ordinary-2'), 'ordinary')
expect(writer.enqueue(Buffer.from('liveness-1'), 'liveness')).toBe(true)
expect(writer.enqueue(Buffer.from('liveness-coalesced'), 'liveness')).toBe(false)
expect(harness.writes.map(String)).toEqual(['ordinary-1', 'liveness-1'])
harness.callbacks[1]({ ok: true })
expect(writer.enqueue(Buffer.from('liveness-2'), 'liveness')).toBe(true)
expect(harness.writes.map(String)).toEqual(['ordinary-1', 'liveness-1', 'liveness-2'])
harness.drain()
expect(harness.writes.map(String)).toEqual([
'ordinary-1',
'liveness-1',
'liveness-2',
'ordinary-2'
])
writer.dispose()
})
it('reports saturated epochs and their drain boundary exactly once', () => {
const harness = transportHarness([false])
const saturation = vi.fn()
const writer = new SshMultiplexerTransportWriter(harness.transport, vi.fn(), saturation)
writer.enqueue(Buffer.from('ordinary'), 'ordinary')
writer.enqueue(Buffer.from('queued'), 'ordinary')
expect(saturation.mock.calls).toEqual([[true]])
harness.drain()
expect(saturation.mock.calls).toEqual([[true], [false]])
writer.dispose()
})
it('keeps a full ordinary lane from consuming the control reserve', () => {
const harness = transportHarness([false, true, true])
const writer = new SshMultiplexerTransportWriter(harness.transport, vi.fn())
@@ -13,7 +13,7 @@ export type MultiplexerTransport = {
close?: () => void
}
export type MultiplexerWriterLane = 'ordinary' | 'control'
export type MultiplexerWriterLane = 'ordinary' | 'control' | 'liveness'
type WriterEntry = {
data: Buffer
@@ -53,11 +53,13 @@ export class SshMultiplexerTransportWriter {
private drainObservedDuringWrite = false
private pumping = false
private closed = false
private livenessOutstanding = false
private removeDrainListener: (() => void) | null = null
constructor(
private readonly transport: MultiplexerTransport,
private readonly onFailure: (error: Error) => void
private readonly onFailure: (error: Error) => void,
private readonly onSaturationChange: (saturated: boolean) => void = () => {}
) {
if (transport.onDrain) {
const remove = transport.onDrain(() => this.handleDrain())
@@ -75,6 +77,9 @@ export class SshMultiplexerTransportWriter {
settle({ ok: false, error: new Error('Multiplexer writer is closed') })
return false
}
if (lane === 'liveness' && this.livenessOutstanding) {
return false
}
const admissionError = this.admissionError(data.length, lane)
if (admissionError) {
settle({ ok: false, error: admissionError })
@@ -83,8 +88,12 @@ export class SshMultiplexerTransportWriter {
}
const entry = { data, lane, onSettled: settle, settled: false }
this.retain(entry)
this.queue.push(entry)
this.pump()
if (lane === 'liveness' && this.saturated) {
this.writeEntry(entry)
} else {
this.queue.push(entry)
this.pump()
}
return true
}
@@ -156,7 +165,7 @@ export class SshMultiplexerTransportWriter {
if (!this.transport.onDrain) {
throw new Error('Multiplexer transport returned write(false) without drain support')
}
this.saturated = !this.drainObservedDuringWrite
this.setSaturated(!this.drainObservedDuringWrite)
if (this.transport.supportsWriteSettlement !== true && this.saturated) {
this.settleOnDrain.add(entry)
} else if (this.transport.supportsWriteSettlement !== true) {
@@ -201,7 +210,7 @@ export class SshMultiplexerTransportWriter {
if (!this.saturated) {
return
}
this.saturated = false
this.setSaturated(false)
for (const entry of Array.from(this.settleOnDrain)) {
this.release(entry, { ok: true })
}
@@ -217,6 +226,9 @@ export class SshMultiplexerTransportWriter {
this.controlBytes += entry.data.length
this.controlFrames++
}
if (entry.lane === 'liveness') {
this.livenessOutstanding = true
}
}
private release(entry: WriterEntry, result: MultiplexerWriteSettlement): void {
@@ -233,9 +245,20 @@ export class SshMultiplexerTransportWriter {
this.controlBytes -= entry.data.length
this.controlFrames--
}
if (entry.lane === 'liveness') {
this.livenessOutstanding = false
}
entry.onSettled(result)
}
private setSaturated(saturated: boolean): void {
if (this.saturated === saturated) {
return
}
this.saturated = saturated
this.onSaturationChange(saturated)
}
private fail(error: Error): void {
if (this.closed) {
return
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest'
import { SshPtyRecoveryRetentionBudget } from './ssh-pty-recovery-retention-budget'
describe('SshPtyRecoveryRetentionBudget', () => {
it('bounds fragmented recovery by per-PTY and session source, charged bytes, and frames', () => {
const budget = new SshPtyRecoveryRetentionBudget({
perPtySourceSu: 4,
perPtyBytes: 300,
perPtyFrames: 2,
sessionSourceSu: 6,
sessionBytes: 500,
sessionFrames: 3
})
expect(budget.tryRetain('pty-1', 'aa', 2)).toBe(true)
expect(budget.tryRetain('pty-1', 'bb', 2)).toBe(true)
expect(budget.tryRetain('pty-1', 'c', 0)).toBe(false)
expect(budget.tryRetain('pty-2', 'cc', 2)).toBe(true)
expect(budget.snapshot()).toEqual({ sourceSu: 6, bytes: 396, frames: 3, ptys: 2 })
expect(budget.tryRetain('pty-2', 'd', 1)).toBe(false)
budget.release('pty-1')
expect(budget.snapshot()).toEqual({ sourceSu: 2, bytes: 132, frames: 1, ptys: 1 })
expect(budget.tryRetain('pty-2', 'dd', 2)).toBe(true)
})
it('charges UTF-16 storage plus one record before aggregate admission', () => {
const budget = new SshPtyRecoveryRetentionBudget({
perPtySourceSu: 100,
perPtyBytes: 1_000,
perPtyFrames: 10,
sessionSourceSu: 100,
sessionBytes: 500,
sessionFrames: 10
})
expect(budget.tryRetain('pty-1', '\u0000'.repeat(64), 1)).toBe(true)
expect(budget.snapshot().bytes).toBe(256)
expect(budget.tryRetain('pty-2', '\u0000'.repeat(64), 1)).toBe(false)
expect(budget.snapshot()).toEqual({ sourceSu: 1, bytes: 256, frames: 1, ptys: 1 })
})
})
@@ -0,0 +1,95 @@
import { DEFAULT_PTY_SOURCE_WINDOW_SU } from '../../shared/pty-source-credit-contract'
import { chargedPtyRetainedStringBytes } from '../../shared/pty-retained-string-memory'
export const SSH_PTY_RECOVERY_PER_PTY_MAX_SOURCE_SU = DEFAULT_PTY_SOURCE_WINDOW_SU
export const SSH_PTY_RECOVERY_PER_PTY_MAX_BYTES = 2 * 1024 * 1024
export const SSH_PTY_RECOVERY_PER_PTY_MAX_FRAMES = 1_024
export const SSH_PTY_RECOVERY_SESSION_MAX_SOURCE_SU = 50 * DEFAULT_PTY_SOURCE_WINDOW_SU
export const SSH_PTY_RECOVERY_SESSION_MAX_BYTES = 64 * 1024 * 1024
export const SSH_PTY_RECOVERY_SESSION_MAX_FRAMES = 64 * 1_024
export type SshPtyRecoveryRetentionLimits = Readonly<{
perPtySourceSu: number
perPtyBytes: number
perPtyFrames: number
sessionSourceSu: number
sessionBytes: number
sessionFrames: number
}>
type RetainedPty = {
sourceSu: number
bytes: number
frames: number
}
const DEFAULT_LIMITS: SshPtyRecoveryRetentionLimits = Object.freeze({
perPtySourceSu: SSH_PTY_RECOVERY_PER_PTY_MAX_SOURCE_SU,
perPtyBytes: SSH_PTY_RECOVERY_PER_PTY_MAX_BYTES,
perPtyFrames: SSH_PTY_RECOVERY_PER_PTY_MAX_FRAMES,
sessionSourceSu: SSH_PTY_RECOVERY_SESSION_MAX_SOURCE_SU,
sessionBytes: SSH_PTY_RECOVERY_SESSION_MAX_BYTES,
sessionFrames: SSH_PTY_RECOVERY_SESSION_MAX_FRAMES
})
export class SshPtyRecoveryRetentionBudget {
private readonly retainedByPty = new Map<string, RetainedPty>()
private sourceSu = 0
private bytes = 0
private frames = 0
constructor(private readonly limits: SshPtyRecoveryRetentionLimits = DEFAULT_LIMITS) {}
tryRetain(ptyId: string, data: string, sourceSu: number): boolean {
if (!Number.isSafeInteger(sourceSu) || sourceSu < 0) {
return false
}
const retained = this.retainedByPty.get(ptyId) ?? { sourceSu: 0, bytes: 0, frames: 0 }
const chargedBytes = chargedPtyRetainedStringBytes(data)
if (
retained.sourceSu + sourceSu > this.limits.perPtySourceSu ||
retained.bytes + chargedBytes > this.limits.perPtyBytes ||
retained.frames + 1 > this.limits.perPtyFrames ||
this.sourceSu + sourceSu > this.limits.sessionSourceSu ||
this.bytes + chargedBytes > this.limits.sessionBytes ||
this.frames + 1 > this.limits.sessionFrames
) {
return false
}
retained.sourceSu += sourceSu
retained.bytes += chargedBytes
retained.frames++
this.retainedByPty.set(ptyId, retained)
this.sourceSu += sourceSu
this.bytes += chargedBytes
this.frames++
return true
}
release(ptyId: string): void {
const retained = this.retainedByPty.get(ptyId)
if (!retained) {
return
}
this.retainedByPty.delete(ptyId)
this.sourceSu -= retained.sourceSu
this.bytes -= retained.bytes
this.frames -= retained.frames
}
clear(): void {
this.retainedByPty.clear()
this.sourceSu = 0
this.bytes = 0
this.frames = 0
}
snapshot(): Readonly<{ sourceSu: number; bytes: number; frames: number; ptys: number }> {
return Object.freeze({
sourceSu: this.sourceSu,
bytes: this.bytes,
frames: this.frames,
ptys: this.retainedByPty.size
})
}
}
@@ -1,6 +1,8 @@
import { describe, expect, it } from 'vitest'
import {
assertSshPtySourceCreditRelayLaunchPolicy,
isSshPtySourceCreditV1Enabled,
SshPtySourceCreditRestartRequiredError,
SSH_PTY_SOURCE_CREDIT_V1_ENV
} from './ssh-pty-source-credit-rollout'
@@ -11,4 +13,39 @@ describe('SSH PTY source-credit rollout', () => {
expect(isSshPtySourceCreditV1Enabled({ [SSH_PTY_SOURCE_CREDIT_V1_ENV]: 'true' })).toBe(false)
expect(isSshPtySourceCreditV1Enabled({ [SSH_PTY_SOURCE_CREDIT_V1_ENV]: '1' })).toBe(true)
})
it('surfaces a deliberate restart action for a live off-to-on transition', () => {
expect(new SshPtySourceCreditRestartRequiredError()).toMatchObject({
name: 'SshPtySourceCreditRestartRequiredError',
code: 'ssh_pty_source_credit_restart_required',
message: expect.stringContaining('Reset the relay')
})
})
it('permits stable gate-off legacy and stable gate-on V1 startup selections', () => {
expect(() => assertSshPtySourceCreditRelayLaunchPolicy(false, 'off')).not.toThrow()
expect(() => assertSshPtySourceCreditRelayLaunchPolicy(false, 'v1')).not.toThrow()
expect(() => assertSshPtySourceCreditRelayLaunchPolicy(true, 'v1')).not.toThrow()
})
it.each(['off', '', 'unknown'])('fails closed when gate-on reaches policy %j', (policy) => {
expect(() => assertSshPtySourceCreditRelayLaunchPolicy(true, policy)).toThrow(
SshPtySourceCreditRestartRequiredError
)
})
it('keeps the process-start selection stable across later environment mutation', () => {
const startupSelection = isSshPtySourceCreditV1Enabled()
const previous = process.env[SSH_PTY_SOURCE_CREDIT_V1_ENV]
process.env[SSH_PTY_SOURCE_CREDIT_V1_ENV] = startupSelection ? '0' : '1'
try {
expect(isSshPtySourceCreditV1Enabled()).toBe(startupSelection)
} finally {
if (previous === undefined) {
delete process.env[SSH_PTY_SOURCE_CREDIT_V1_ENV]
} else {
process.env[SSH_PTY_SOURCE_CREDIT_V1_ENV] = previous
}
}
})
})
+27 -1
View File
@@ -1,5 +1,31 @@
export const SSH_PTY_SOURCE_CREDIT_V1_ENV = 'ORCA_SSH_PTY_SOURCE_CREDIT_V1'
export function isSshPtySourceCreditV1Enabled(env: NodeJS.ProcessEnv = process.env): boolean {
function readSshPtySourceCreditV1Selection(env: NodeJS.ProcessEnv): boolean {
return env[SSH_PTY_SOURCE_CREDIT_V1_ENV] === '1'
}
const SSH_PTY_SOURCE_CREDIT_V1_STARTUP_SELECTION = readSshPtySourceCreditV1Selection(process.env)
export function isSshPtySourceCreditV1Enabled(env?: NodeJS.ProcessEnv): boolean {
return env ? readSshPtySourceCreditV1Selection(env) : SSH_PTY_SOURCE_CREDIT_V1_STARTUP_SELECTION
}
export class SshPtySourceCreditRestartRequiredError extends Error {
readonly code = 'ssh_pty_source_credit_restart_required'
constructor() {
super(
'SSH PTY source credit was enabled after this detached relay started. Reset the relay, then reconnect to launch it with source-credit support.'
)
this.name = 'SshPtySourceCreditRestartRequiredError'
}
}
export function assertSshPtySourceCreditRelayLaunchPolicy(
enabled: boolean,
relayPolicy: string
): void {
if (enabled && relayPolicy !== 'v1') {
throw new SshPtySourceCreditRestartRequiredError()
}
}
+4
View File
@@ -484,6 +484,7 @@ describe('deployAndLaunchRelay', () => {
expect(launchCommand).toContain(`--grace-time ${DEFAULT_SSH_RELAY_GRACE_PERIOD_SECONDS}`)
expect(launchCommand).not.toContain('--pty-source-credit-v1')
expect(launchCommand).toContain("printf '%s\\n' 'off'")
})
it('allows an unlimited SSH disconnect grace window', async () => {
@@ -504,6 +505,7 @@ describe('deployAndLaunchRelay', () => {
expect(launchCommand).toContain('--grace-time 0')
expect(launchCommand).toContain('--pty-source-credit-v1')
expect(launchCommand).toContain("printf '%s\\n' 'v1'")
})
it('clamps configured SSH disconnect grace to the seven-day maximum', async () => {
@@ -770,6 +772,8 @@ describe('deployAndLaunchRelay', () => {
)
expect(launchScript).toContain('--endpoint-dir')
expect(launchScript).toContain('--pty-source-credit-v1')
expect(launchScript).toContain('.pty-source-credit-policy')
expect(launchScript).toContain("-Value 'v1'")
expect(launchScript).not.toContain('\\\\.\\pipe\\agent-hooks')
const waitScript = decodedScripts.find((script) => script.includes('deadline=Date.now()')) ?? ''
expect(waitScript).toContain('setTimeout(attempt,intervalMs)')
+78 -6
View File
@@ -54,6 +54,7 @@ import {
isWindowsRemoteHost,
joinRemotePath,
normalizeRemoteHome,
remoteBasename,
validateRemoteHome,
type RemoteHostPlatform
} from './ssh-remote-platform'
@@ -73,6 +74,10 @@ import {
MAX_SSH_RELAY_GRACE_PERIOD_SECONDS,
MIN_SSH_RELAY_GRACE_PERIOD_SECONDS
} from '../../shared/ssh-types'
import {
assertSshPtySourceCreditRelayLaunchPolicy,
SshPtySourceCreditRestartRequiredError
} from './ssh-pty-source-credit-rollout'
export type RelayDeployResult = {
transport: MultiplexerTransport
@@ -108,6 +113,37 @@ function execHostCommand(
})
}
function sourceCreditPolicyPath(
hostPlatform: RemoteHostPlatform,
remoteDir: string,
endpoint: string
): string {
return joinRemotePath(
hostPlatform,
remoteDir,
`${remoteBasename(endpoint, hostPlatform)}.pty-source-credit-policy`
)
}
async function assertExistingRelaySourceCreditPolicy(
conn: SshConnection,
hostPlatform: RemoteHostPlatform,
policyPath: string,
enabled: boolean,
signal?: AbortSignal
): Promise<void> {
if (!enabled) {
return
}
const command = isWindowsRemoteHost(hostPlatform)
? powerShellCommand(
`if (Test-Path -LiteralPath ${powerShellLiteral(policyPath)} -PathType Leaf) { Get-Content -LiteralPath ${powerShellLiteral(policyPath)} -Raw -ErrorAction SilentlyContinue }`
)
: `test -f ${shellEscape(policyPath)} && cat ${shellEscape(policyPath)} || true`
const policy = (await execHostCommand(conn, hostPlatform, command, { signal })).trim()
assertSshPtySourceCreditRelayLaunchPolicy(enabled, policy)
}
/**
* Deploy the relay to the remote host and launch it, returning the transport (relay's stdin/stdout) for multiplexer use.
*/
@@ -1191,18 +1227,21 @@ async function launchRelay(
const sockFile = relayEndpointForHost(hostPlatform, remoteDir, sockName)
const endpointDir = relayHookEndpointDirForHost(hostPlatform, remoteDir, sockFile)
const credentialFile = joinRemotePath(hostPlatform, remoteDir, `${sockName}.credential`)
const sourceCreditPolicyFile = sourceCreditPolicyPath(hostPlatform, remoteDir, sockFile)
if (isWindowsRemoteHost(hostPlatform)) {
const activePipeMarkerPath = windowsActivePipeMarkerPath(hostPlatform, remoteDir, sockName)
const activeEndpoint = (await readWindowsActiveRelayEndpoint(
const discoveredActiveEndpoint = await readWindowsActiveRelayEndpoint(
conn,
hostPlatform,
remoteDir,
activePipeMarkerPath,
signal
)) ?? {
)
const activeEndpoint = discoveredActiveEndpoint ?? {
sockPath: sockFile,
endpointDir
endpointDir,
sourceCreditPolicyFile
}
const fallbackEndpoint = buildWindowsRelayFallbackEndpoint(hostPlatform, remoteDir, sockName)
const launched = await launchWindowsRelay(
@@ -1213,6 +1252,7 @@ async function launchRelay(
nodePath,
sockPath: activeEndpoint.sockPath,
endpointDir: activeEndpoint.endpointDir,
sourceCreditPolicyFile: activeEndpoint.sourceCreditPolicyFile,
graceTime,
activePipeMarkerPath,
reconnectFallback: fallbackEndpoint,
@@ -1233,6 +1273,13 @@ async function launchRelay(
)
console.warn(`[ssh-relay] Socket probe result: "${probeOutput.trim()}"`)
if (probeOutput.trim() === 'ALIVE') {
await assertExistingRelaySourceCreditPolicy(
conn,
hostPlatform,
sourceCreditPolicyFile,
enablePtySourceCreditV1,
signal
)
console.log('[ssh-relay] Existing relay socket found, attempting reconnect...')
try {
const channel = await conn.exec(
@@ -1260,6 +1307,9 @@ async function launchRelay(
}
}
} catch (err) {
if (err instanceof SshPtySourceCreditRestartRequiredError) {
throw err
}
if (isUnconfirmedSshCommandTermination(err)) {
throw err
}
@@ -1273,7 +1323,8 @@ async function launchRelay(
await writeRelayEndpointCredential(conn, hostPlatform, credentialFile, signal)
// Why: --log-file lets the relay rotate relay.log in-process; the shell redirect stays to capture pre-JS boot/crash output.
const sourceCreditFlag = enablePtySourceCreditV1 ? ' --pty-source-credit-v1' : ''
const launchCmd = `cd ${escapedDir} && chmod 600 ${shellEscape(credentialFile)} && nohup ${escapedNode} relay.js --detached${sourceCreditFlag} --grace-time ${graceTime} --sock-path ${shellEscape(sockFile)} --credential-file ${shellEscape(credentialFile)} --log-file ${shellEscape(logFile)} > ${shellEscape(logFile)} 2>&1 </dev/null &`
const sourceCreditPolicy = enablePtySourceCreditV1 ? 'v1' : 'off'
const launchCmd = `cd ${escapedDir} && printf '%s\\n' ${shellEscape(sourceCreditPolicy)} > ${shellEscape(sourceCreditPolicyFile)} && chmod 600 ${shellEscape(credentialFile)} ${shellEscape(sourceCreditPolicyFile)} && nohup ${escapedNode} relay.js --detached${sourceCreditFlag} --grace-time ${graceTime} --sock-path ${shellEscape(sockFile)} --credential-file ${shellEscape(credentialFile)} --log-file ${shellEscape(logFile)} > ${shellEscape(logFile)} 2>&1 </dev/null &`
const launchChannel = await conn.exec(launchCmd, { signal })
launchChannel.on('data', () => {})
launchChannel.on('error', () => {})
@@ -1371,7 +1422,8 @@ function buildWindowsRelayFallbackEndpoint(
const sockPath = relayEndpointForHost(hostPlatform, remoteDir, fallbackSockName)
return {
sockPath,
endpointDir: relayHookEndpointDirForHost(hostPlatform, remoteDir, sockPath)
endpointDir: relayHookEndpointDirForHost(hostPlatform, remoteDir, sockPath),
sourceCreditPolicyFile: sourceCreditPolicyPath(hostPlatform, remoteDir, sockPath)
}
}
@@ -1399,7 +1451,8 @@ async function readWindowsActiveRelayEndpoint(
}
return {
sockPath,
endpointDir: relayHookEndpointDirForHost(hostPlatform, remoteDir, sockPath)
endpointDir: relayHookEndpointDirForHost(hostPlatform, remoteDir, sockPath),
sourceCreditPolicyFile: sourceCreditPolicyPath(hostPlatform, remoteDir, sockPath)
}
}
@@ -1429,6 +1482,7 @@ async function rememberWindowsActiveRelayEndpoint(
type WindowsRelayEndpoint = {
sockPath: string
endpointDir: string
sourceCreditPolicyFile: string
}
type WindowsRelayLaunchOptions = {
@@ -1450,6 +1504,13 @@ async function launchWindowsRelay(
): Promise<{ transport: MultiplexerTransport; nodePath: string; sockPath: string }> {
let launchOpts = opts
if ((await probeWindowsRelayPipe(conn, hostPlatform, opts, signal)) === 'READY') {
await assertExistingRelaySourceCreditPolicy(
conn,
hostPlatform,
opts.sourceCreditPolicyFile,
opts.enablePtySourceCreditV1,
signal
)
try {
const transport = await connectWindowsRelay(conn, hostPlatform, opts, signal)
await rememberWindowsActiveRelayEndpoint(
@@ -1482,6 +1543,13 @@ async function launchWindowsRelay(
launchOpts !== opts &&
(await probeWindowsRelayPipe(conn, hostPlatform, launchOpts, signal)) === 'READY'
) {
await assertExistingRelaySourceCreditPolicy(
conn,
hostPlatform,
launchOpts.sourceCreditPolicyFile,
launchOpts.enablePtySourceCreditV1,
signal
)
try {
const transport = await connectWindowsRelay(conn, hostPlatform, launchOpts, signal)
await rememberWindowsActiveRelayEndpoint(
@@ -1521,6 +1589,7 @@ async function launchWindowsRelay(
logFile,
errFile,
launchOpts.credentialFile,
launchOpts.sourceCreditPolicyFile,
launchOpts.enablePtySourceCreditV1
),
{ signal }
@@ -1614,6 +1683,7 @@ function windowsRelayLaunchCommand(
logFile: string,
errFile: string,
credentialFile: string,
sourceCreditPolicyFile: string,
enablePtySourceCreditV1 = false
): string {
const relayScript = joinRemotePath(hostPlatform, remoteDir, 'relay.js')
@@ -1644,7 +1714,9 @@ function windowsRelayLaunchCommand(
nodePath,
remoteDir,
[
`Set-Content -LiteralPath ${powerShellLiteral(sourceCreditPolicyFile)} -Value ${powerShellLiteral(enablePtySourceCreditV1 ? 'v1' : 'off')} -NoNewline`,
`& icacls.exe ${powerShellLiteral(credentialFile)} /inheritance:r /grant:r "$($env:USERNAME):(R,W)" | Out-Null`,
`& icacls.exe ${powerShellLiteral(sourceCreditPolicyFile)} /inheritance:r /grant:r "$($env:USERNAME):(R,W)" | Out-Null`,
`$result = Invoke-CimMethod -ClassName Win32_Process -MethodName Create -Arguments @{ CommandLine = ${powerShellLiteral(wmiCommandLine)}; CurrentDirectory = ${powerShellLiteral(remoteDir)} }`,
`if ($result.ReturnValue -ne 0) { throw "Win32_Process.Create failed with $($result.ReturnValue)" }`
].join('; ')
@@ -1,4 +1,5 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR } from '../../shared/pty-consumer-session'
import { SshRelaySession } from './ssh-relay-session'
import { createMockDeps, mockDeploySuccess } from './ssh-relay-session-test-fixtures'
@@ -34,7 +35,9 @@ vi.mock('../ipc/ssh-pty-output-intake-registry', () => ({
allocateSshPtyProviderGeneration: vi.fn(() => 23),
closeSshPtyOutputGeneration: vi.fn(),
getSshPtyAcceptedSourceCheckpoints: vi.fn(() => []),
installSshPtySourceAckPublisher: vi.fn(() => () => {})
installSshPtySourceAckPublisher: vi.fn(() => () => {}),
installSshPtySourceCancellationPublisher: vi.fn(() => () => {}),
applySshPtySourceCancellationProof: vi.fn()
}))
vi.mock('./ssh-channel-multiplexer', () => ({
@@ -105,9 +108,17 @@ vi.mock('../providers/ssh-git-dispatch', () => ({
unregisterSshGitProvider: vi.fn()
}))
const { getSshPtyProvider, getPtyIdsForConnection, registerSshPtyProvider } =
await import('../ipc/pty')
const {
clearProviderPtyState,
clearPtyOwnershipForConnection,
deletePtyOwnership,
getSshPtyProvider,
getPtyIdsForConnection,
registerSshPtyProvider,
setPtyOwnership
} = await import('../ipc/pty')
const { closeSshPtyOutputGeneration } = await import('../ipc/ssh-pty-output-intake-registry')
const { applySshPtySourceCancellationProof } = await import('../ipc/ssh-pty-output-intake-registry')
const { getSshPtyAcceptedSourceCheckpoints } = await import('../ipc/ssh-pty-output-intake-registry')
const { installSshPtySourceAckPublisher } = await import('../ipc/ssh-pty-output-intake-registry')
const { deployAndLaunchRelay } = await import('./ssh-relay-deploy')
@@ -117,6 +128,8 @@ describe('SshRelaySession data delivery', () => {
vi.clearAllMocks()
ptyDataHandlerRef.current = undefined
attachForReconnectMock.mockResolvedValue({})
vi.mocked(getPtyIdsForConnection).mockReturnValue([])
vi.mocked(getSshPtyAcceptedSourceCheckpoints).mockReturnValue([])
openConsumerSessionMock.mockImplementation(async (_mux, options) => ({
mode: 'negotiated',
clientInstanceId: options.clientInstanceId,
@@ -133,6 +146,122 @@ describe('SshRelaySession data delivery', () => {
mockDeploySuccess()
})
async function runRecoverySequence(args: {
targetId: string
recoveryEndSu: number
recoveryFrame?: readonly [startSu: number, endSu: number]
liveFrame?: readonly [startSu: number, endSu: number]
}) {
muxRequestMock.mockImplementation(async (method) =>
method === 'pty.cancelDelivery'
? { canceled: true, sentEndSu: args.recoveryEndSu, creditedEndSu: 4 }
: []
)
let generation = 0
openConsumerSessionMock.mockImplementation(async (_mux, options) => ({
mode: 'negotiated',
clientInstanceId: options.clientInstanceId,
clientGeneration: ++generation,
ownerGeneration: generation,
ownerLease: `owner-lease-${generation}`,
outputFlowControl: { version: 1, windowSu: 256 * 1024 }
}))
vi.mocked(getSshPtyAcceptedSourceCheckpoints).mockReturnValue([
{
id: `ssh:${args.targetId}@@pty-1`,
providerGeneration: 23,
clientGeneration: 1,
ownerGeneration: 1,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'old-token',
acceptedSourceEndSu: 4
}
])
const deps = createMockDeps()
const session = new SshRelaySession(
args.targetId,
deps.getMainWindow,
deps.mockStore,
deps.mockPortForward,
undefined,
undefined,
() => true
)
await session.establish(deps.mockConn)
vi.mocked(getPtyIdsForConnection).mockReturnValue([`ssh:${args.targetId}@@pty-1`])
vi.mocked(getSshPtyProvider).mockImplementation(
() => vi.mocked(registerSshPtyProvider).mock.calls.at(-1)?.[1]
)
attachForReconnectMock.mockImplementation(async () => {
queueMicrotask(() => {
if (args.recoveryFrame) {
const [sourceStartSu, sourceEndSu] = args.recoveryFrame
ptyDataHandlerRef.current?.({
id: `ssh:${args.targetId}@@pty-1`,
data: 'recovery',
providerGeneration: 23,
ptyIncarnation: 'incarnation-1',
sequenceChars: sourceEndSu - sourceStartSu,
source: {
relayPtyId: 'pty-1',
spanId: `new-token:${sourceStartSu}:${sourceEndSu}`,
clientGeneration: 2,
ownerGeneration: 2,
deliveryToken: 'new-token',
sourceStartSu,
sourceEndSu
}
})
}
const complete = onNotificationByMethodMock.mock.calls.findLast(
([method]) => method === 'pty.recoveryComplete'
)?.[1] as ((params: Record<string, unknown>) => void) | undefined
complete?.({
id: 'pty-1',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: args.recoveryEndSu
})
if (args.liveFrame) {
const [sourceStartSu, sourceEndSu] = args.liveFrame
ptyDataHandlerRef.current?.({
id: `ssh:${args.targetId}@@pty-1`,
data: 'live',
providerGeneration: 23,
ptyIncarnation: 'incarnation-1',
sequenceChars: sourceEndSu - sourceStartSu,
source: {
relayPtyId: 'pty-1',
spanId: `new-token:${sourceStartSu}:${sourceEndSu}`,
clientGeneration: 2,
ownerGeneration: 2,
deliveryToken: 'new-token',
sourceStartSu,
sourceEndSu
}
})
}
})
return {
incarnationId: 'incarnation-1',
sourceRecovery: {
status: 'pending',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: args.recoveryEndSu
}
}
})
await session.reconnect(deps.mockConn)
return { ...deps, session }
}
it('transfers negotiated owner recovery exactly once across an explicit detach', async () => {
const { mockConn, mockStore, mockPortForward, getMainWindow } = createMockDeps()
vi.mocked(deployAndLaunchRelay).mockResolvedValue({
@@ -163,6 +292,86 @@ describe('SshRelaySession data delivery', () => {
fresh.dispose()
})
it('clears stale owner recovery and retries a fresh same-build relay once without resume', async () => {
const targetId = 'fresh-relay-retry'
const { mockConn, mockStore, mockPortForward, getMainWindow } = createMockDeps()
vi.mocked(deployAndLaunchRelay).mockResolvedValue({
transport: { write: vi.fn(), onData: vi.fn(), onClose: vi.fn() },
platform: 'linux-x64',
serverBuildId: 'test-relay-build'
})
vi.mocked(getSshPtyAcceptedSourceCheckpoints).mockReturnValue([
{
id: `ssh:${targetId}@@pty-1`,
providerGeneration: 23,
clientGeneration: 1,
ownerGeneration: 1,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'old-token',
acceptedSourceEndSu: 4
}
])
const first = new SshRelaySession(
targetId,
getMainWindow,
mockStore,
mockPortForward,
undefined,
undefined,
() => true
)
await first.establish(mockConn)
first.detach()
openConsumerSessionMock
.mockRejectedValueOnce(
Object.assign(new Error('Owner recovery lease is stale'), {
code: PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR
})
)
.mockImplementationOnce(async (_mux, options) => ({
mode: 'negotiated',
clientInstanceId: options.clientInstanceId,
clientGeneration: 1,
ownerGeneration: 1,
ownerLease: 'fresh-owner-lease',
outputFlowControl: { version: 1, windowSu: 256 * 1024 }
}))
vi.mocked(getPtyIdsForConnection).mockReturnValue([`ssh:${targetId}@@pty-1`])
vi.mocked(getSshPtyProvider).mockImplementation(
() => vi.mocked(registerSshPtyProvider).mock.calls.at(-1)?.[1]
)
attachForReconnectMock.mockResolvedValue({
incarnationId: 'incarnation-1',
sourceRecovery: { status: 'restoreRequired', reason: 'checkpointUnavailable' }
})
const second = new SshRelaySession(
targetId,
getMainWindow,
mockStore,
mockPortForward,
undefined,
undefined,
() => true
)
const openCallCountBeforeRetry = openConsumerSessionMock.mock.calls.length
await second.establish(mockConn)
const retryCalls = openConsumerSessionMock.mock.calls
.slice(openCallCountBeforeRetry)
.map(([, options]) => options)
expect(retryCalls).toHaveLength(2)
expect(retryCalls[0]).toHaveProperty('resume')
expect(retryCalls[1]).not.toHaveProperty('resume')
expect(attachForReconnectMock).toHaveBeenCalledWith(
'pty-1',
undefined,
Object.freeze({ status: 'checkpointUnavailable' })
)
second.dispose()
})
it('delivers empty transformed relay spans with raw sequence metadata', async () => {
const { mockConn, mockStore, mockPortForward, getMainWindow, mockWindow } = createMockDeps()
const runtime = { onPtyData: vi.fn(() => 17), onPtyExit: vi.fn() }
@@ -474,17 +683,33 @@ describe('SshRelaySession data delivery', () => {
queueMicrotask(() => {
ptyDataHandlerRef.current?.({
id: 'ssh:target-1@@pty-1',
data: 'reco',
data: 're',
providerGeneration: 23,
ptyIncarnation: 'incarnation-1',
sequenceChars: 4,
sequenceChars: 2,
source: {
relayPtyId: 'pty-1',
spanId: 'new-token:4:8',
spanId: 'new-token:4:6',
clientGeneration: 2,
ownerGeneration: 2,
deliveryToken: 'new-token',
sourceStartSu: 4,
sourceEndSu: 6
}
})
ptyDataHandlerRef.current?.({
id: 'ssh:target-1@@pty-1',
data: 'co',
providerGeneration: 23,
ptyIncarnation: 'incarnation-1',
sequenceChars: 2,
source: {
relayPtyId: 'pty-1',
spanId: 'new-token:6:8',
clientGeneration: 2,
ownerGeneration: 2,
deliveryToken: 'new-token',
sourceStartSu: 6,
sourceEndSu: 8
}
})
@@ -543,7 +768,8 @@ describe('SshRelaySession data delivery', () => {
})
)
expect(acceptOutputDataMock.mock.calls.map(([payload]) => payload.data)).toEqual([
'reco',
're',
'co',
'live'
])
expect(transferDisposedMux).toBe(false)
@@ -562,4 +788,59 @@ describe('SshRelaySession data delivery', () => {
})
expect(closeSshPtyOutputGeneration).toHaveBeenCalledTimes(closeCount)
})
it.each([
['gap', [5, 8] as const],
['overlap', [3, 8] as const],
['incomplete suffix', [4, 6] as const],
['missing body', undefined]
])('rejects %s recovery without destroying the physical PTY or lease', async (label, frame) => {
const targetId = `invalid-recovery-${label.replace(' ', '-')}`
const { mockStore, mockWindow } = await runRecoverySequence({
targetId,
recoveryEndSu: 8,
...(frame ? { recoveryFrame: frame } : {})
})
expect(acceptOutputDataMock).not.toHaveBeenCalled()
expect(muxRequestMock).toHaveBeenCalledWith('pty.cancelDelivery', {
id: 'pty-1',
clientGeneration: 2,
ownerGeneration: 2,
deliveryToken: 'new-token'
})
expect(applySshPtySourceCancellationProof).toHaveBeenCalledWith(
{
id: `ssh:${targetId}@@pty-1`,
code: -1,
providerGeneration: 23,
ptyIncarnation: 'incarnation-1'
},
{ sentEndSu: 8, creditedEndSu: 4 }
)
expect(mockStore.markSshRemotePtyLease).toHaveBeenCalledWith(targetId, 'pty-1', 'detached')
expect(mockStore.markSshRemotePtyLease).not.toHaveBeenCalledWith(targetId, 'pty-1', 'expired')
expect(mockStore.markSshRemotePtyLeases).not.toHaveBeenCalled()
expect(clearProviderPtyState).not.toHaveBeenCalled()
expect(clearPtyOwnershipForConnection).not.toHaveBeenCalled()
expect(deletePtyOwnership).not.toHaveBeenCalled()
expect(setPtyOwnership).not.toHaveBeenCalled()
expect(muxDisposeMock).not.toHaveBeenCalledWith('shutdown')
expect(mockWindow.webContents.send).not.toHaveBeenCalledWith('pty:exit', expect.anything())
})
it('accepts empty recovery only when the checkpoint equals the recovery end', async () => {
const { mockStore } = await runRecoverySequence({
targetId: 'empty-recovery',
recoveryEndSu: 4,
liveFrame: [4, 8]
})
expect(acceptOutputDataMock.mock.calls.map(([payload]) => payload.data)).toEqual(['live'])
expect(mockStore.markSshRemotePtyLease).toHaveBeenCalledWith(
'empty-recovery',
'pty-1',
'attached'
)
})
})
@@ -138,6 +138,72 @@ describe('SshRelaySession reconnect incarnation ordering', () => {
vi.mocked(getPtyIdsForConnection).mockReturnValue([])
})
it('bounds fifty reattaches to eight workers without slow or failed sibling head-of-line delay', async () => {
const { mockConn, mockStore, mockPortForward, getMainWindow } = createMockDeps()
const session = new SshRelaySession('target-1', getMainWindow, mockStore, mockPortForward)
await session.establish(mockConn)
vi.clearAllMocks()
mockDeploySuccess()
const ptyIds = Array.from({ length: 50 }, (_, index) => `pty-${index}`)
let active = 0
let peakActive = 0
const attempts = new Map<string, number>()
const mockAttach = vi.fn((id: string) => {
attempts.set(id, (attempts.get(id) ?? 0) + 1)
if (id === 'pty-0') {
return new Promise<void>(() => {})
}
if (id === 'pty-1') {
return Promise.reject(new Error('isolated attach failure'))
}
active++
peakActive = Math.max(peakActive, active)
return new Promise<void>((resolve) => {
setTimeout(() => {
active--
resolve()
}, 100)
})
})
vi.mocked(getSshPtyProvider).mockReturnValue({
attachForReconnect: mockAttach,
dispose: vi.fn()
} as unknown as ReturnType<typeof getSshPtyProvider>)
vi.mocked(getPtyIdsForConnection).mockReturnValue(ptyIds)
const random = vi.spyOn(Math, 'random').mockReturnValue(0)
vi.useFakeTimers()
try {
const reconnect = session.reconnect(mockConn)
await vi.advanceTimersByTimeAsync(750)
expect(
vi
.mocked(mockStore.markSshRemotePtyLease)
.mock.calls.filter(([, , state]) => state === 'attached')
.map(([, id]) => id)
).toHaveLength(48)
expect(peakActive).toBeLessThanOrEqual(8)
await vi.advanceTimersByTimeAsync(20_000)
await reconnect
expect(mockAttach).toHaveBeenCalledTimes(52)
expect(attempts.get('pty-0')).toBe(2)
expect(attempts.get('pty-1')).toBe(2)
expect(session.getState()).toBe('ready')
expect(mockStore.markSshRemotePtyLease).not.toHaveBeenCalledWith(
'target-1',
expect.any(String),
'expired'
)
} finally {
vi.useRealTimers()
random.mockRestore()
}
})
it('restores and persists exact incarnation proof from reconnect attach', async () => {
const { mockConn, mockStore, mockPortForward, getMainWindow } = createMockDeps()
const incarnationId = 'incarnation-reconnect'
@@ -0,0 +1,361 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { SshRelaySession } from './ssh-relay-session'
import { createMockDeps, mockDeploySuccess } from './ssh-relay-session-test-fixtures'
const {
acceptOutputDataMock,
muxRequestMock,
onNotificationByMethodMock,
openConsumerSessionMock,
muxDisposeMock,
attachForReconnectMock,
ptyDataHandlerRef
} = vi.hoisted(() => ({
acceptOutputDataMock: vi.fn().mockResolvedValue(undefined),
muxRequestMock: vi.fn(),
onNotificationByMethodMock: vi.fn(),
openConsumerSessionMock: vi.fn(),
muxDisposeMock: vi.fn(),
attachForReconnectMock: vi.fn().mockResolvedValue({}),
ptyDataHandlerRef: { current: undefined as undefined | ((payload: unknown) => void) }
}))
vi.mock('./ssh-relay-deploy', () => ({ deployAndLaunchRelay: vi.fn() }))
vi.mock('./ssh-pty-consumer-session', () => ({
openSshPtyConsumerSession: openConsumerSessionMock
}))
vi.mock('../ipc/ssh-pty-output-intake-registry', () => ({
acceptSshPtyOutputData: acceptOutputDataMock,
acceptSshPtyOutputExit: vi.fn().mockResolvedValue(undefined),
allocateSshPtyProviderGeneration: vi.fn(() => 23),
closeSshPtyOutputGeneration: vi.fn(),
getSshPtyAcceptedSourceCheckpoints: vi.fn(() => []),
installSshPtySourceAckPublisher: vi.fn(() => () => {}),
installSshPtySourceCancellationPublisher: vi.fn(() => () => {}),
applySshPtySourceCancellationProof: vi.fn()
}))
vi.mock('./ssh-channel-multiplexer', () => ({
SshChannelMultiplexer: class MockSshChannelMultiplexer {
notify = vi.fn()
notifyWithSettlement = vi.fn()
request = muxRequestMock
onNotification = vi.fn().mockReturnValue(() => {})
onNotificationByMethod = onNotificationByMethodMock.mockImplementation(() => () => {})
onRequest = vi.fn().mockReturnValue(() => {})
onDispose = vi.fn().mockReturnValue(() => {})
dispose = muxDisposeMock
isDisposed = vi.fn().mockReturnValue(false)
}
}))
vi.mock('../providers/ssh-pty-provider', () => ({
isSshPtyNotFoundError: vi.fn().mockReturnValue(false),
isSshPtyIdentityMismatchError: vi.fn().mockReturnValue(false),
SshPtyProvider: class MockSshPtyProvider {
onData = vi.fn().mockImplementation((handler) => {
ptyDataHandlerRef.current = handler
return () => {}
})
onReplay = vi.fn().mockReturnValue(() => {})
onExit = vi.fn().mockReturnValue(() => {})
attachForReconnect = attachForReconnectMock
setPtyDeliveryPauseAdapter = vi.fn()
dispose = vi.fn()
}
}))
vi.mock('../providers/ssh-filesystem-provider', () => ({
SshFilesystemProvider: class MockSshFilesystemProvider {
dispose = vi.fn()
}
}))
vi.mock('../providers/ssh-git-provider', () => ({
SshGitProvider: class MockSshGitProvider {}
}))
vi.mock('../ipc/pty', () => ({
registerSshPtyProvider: vi.fn(),
unregisterSshPtyProvider: vi.fn(),
getSshPtyProvider: vi.fn().mockReturnValue({ dispose: vi.fn() }),
getPtyIdsForConnection: vi.fn().mockReturnValue([]),
clearPtyOwnershipForConnection: vi.fn(),
clearProviderPtyState: vi.fn(),
deletePtyOwnership: vi.fn(),
restorePtyIncarnation: vi.fn(),
setPtyOwnership: vi.fn()
}))
vi.mock('../providers/ssh-filesystem-dispatch', () => ({
registerSshFilesystemProvider: vi.fn(),
unregisterSshFilesystemProvider: vi.fn(),
getSshFilesystemProvider: vi.fn().mockReturnValue({ dispose: vi.fn() })
}))
vi.mock('../providers/ssh-git-dispatch', () => ({
registerSshGitProvider: vi.fn(),
unregisterSshGitProvider: vi.fn()
}))
const {
clearProviderPtyState,
clearPtyOwnershipForConnection,
deletePtyOwnership,
getSshPtyProvider,
getPtyIdsForConnection,
registerSshPtyProvider,
setPtyOwnership
} = await import('../ipc/pty')
const { closeSshPtyOutputGeneration, getSshPtyAcceptedSourceCheckpoints } =
await import('../ipc/ssh-pty-output-intake-registry')
describe('SshRelaySession recovery race fencing', () => {
beforeEach(() => {
vi.clearAllMocks()
ptyDataHandlerRef.current = undefined
attachForReconnectMock.mockResolvedValue({})
vi.mocked(getPtyIdsForConnection).mockReturnValue([])
vi.mocked(getSshPtyAcceptedSourceCheckpoints).mockReturnValue([])
muxRequestMock.mockResolvedValue([])
mockDeploySuccess()
})
function emitSourceFrame(args: {
targetId: string
token: string
clientGeneration: number
ownerGeneration: number
sourceStartSu: number
sourceEndSu: number
}): void {
ptyDataHandlerRef.current?.({
id: `ssh:${args.targetId}@@pty-1`,
data: 'late',
providerGeneration: 23,
ptyIncarnation: 'incarnation-1',
sequenceChars: args.sourceEndSu - args.sourceStartSu,
source: {
relayPtyId: 'pty-1',
spanId: `${args.token}:${args.sourceStartSu}:${args.sourceEndSu}`,
clientGeneration: args.clientGeneration,
ownerGeneration: args.ownerGeneration,
deliveryToken: args.token,
sourceStartSu: args.sourceStartSu,
sourceEndSu: args.sourceEndSu
}
})
}
async function prepareRecovery(targetId: string): Promise<{
session: SshRelaySession
deps: ReturnType<typeof createMockDeps>
}> {
let generation = 0
openConsumerSessionMock.mockImplementation(async (_mux, options) => ({
mode: 'negotiated',
clientInstanceId: options.clientInstanceId,
clientGeneration: ++generation,
ownerGeneration: generation,
ownerLease: `owner-lease-${generation}`,
outputFlowControl: { version: 1, windowSu: 256 * 1024 }
}))
vi.mocked(getSshPtyAcceptedSourceCheckpoints).mockReturnValue([
{
id: `ssh:${targetId}@@pty-1`,
providerGeneration: 23,
clientGeneration: 1,
ownerGeneration: 1,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'old-token',
acceptedSourceEndSu: 4
}
])
const deps = createMockDeps()
const session = new SshRelaySession(
targetId,
deps.getMainWindow,
deps.mockStore,
deps.mockPortForward,
undefined,
undefined,
() => true
)
await session.establish(deps.mockConn)
vi.mocked(getPtyIdsForConnection).mockReturnValue([`ssh:${targetId}@@pty-1`])
vi.mocked(getSshPtyProvider).mockImplementation(
() => vi.mocked(registerSshPtyProvider).mock.calls.at(-1)?.[1]
)
return { session, deps }
}
it('retains the empty recovery end as the first post-activation source anchor', async () => {
const targetId = 'empty-recovery-gap'
const { session, deps } = await prepareRecovery(targetId)
attachForReconnectMock.mockImplementation(async () => {
queueMicrotask(() => {
const complete = onNotificationByMethodMock.mock.calls.findLast(
([method]) => method === 'pty.recoveryComplete'
)?.[1] as ((params: Record<string, unknown>) => void) | undefined
complete?.({
id: 'pty-1',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 4
})
})
return {
incarnationId: 'incarnation-1',
sourceRecovery: {
status: 'pending',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 4
}
}
})
await session.reconnect(deps.mockConn)
const closeCount = vi.mocked(closeSshPtyOutputGeneration).mock.calls.length
emitSourceFrame({
targetId,
token: 'new-token',
clientGeneration: 2,
ownerGeneration: 2,
sourceStartSu: 5,
sourceEndSu: 8
})
expect(acceptOutputDataMock).not.toHaveBeenCalled()
expect(closeSshPtyOutputGeneration).toHaveBeenCalledTimes(closeCount + 1)
expect(closeSshPtyOutputGeneration).toHaveBeenLastCalledWith(
23,
'ssh_source_frame_stale_or_non_contiguous'
)
})
it('drops late frames from a token after its cancellation proof is validated', async () => {
const targetId = 'late-after-cancel'
muxRequestMock.mockImplementation(async (method) =>
method === 'pty.cancelDelivery' ? { canceled: true, sentEndSu: 8, creditedEndSu: 4 } : []
)
const { session, deps } = await prepareRecovery(targetId)
attachForReconnectMock.mockImplementation(async () => {
queueMicrotask(() => {
emitSourceFrame({
targetId,
token: 'new-token',
clientGeneration: 2,
ownerGeneration: 2,
sourceStartSu: 5,
sourceEndSu: 8
})
const complete = onNotificationByMethodMock.mock.calls.findLast(
([method]) => method === 'pty.recoveryComplete'
)?.[1] as ((params: Record<string, unknown>) => void) | undefined
complete?.({
id: 'pty-1',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 8
})
})
return {
incarnationId: 'incarnation-1',
sourceRecovery: {
status: 'pending',
clientGeneration: 2,
ownerGeneration: 2,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'new-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 8
}
}
})
await session.reconnect(deps.mockConn)
const closeCount = vi.mocked(closeSshPtyOutputGeneration).mock.calls.length
emitSourceFrame({
targetId,
token: 'new-token',
clientGeneration: 2,
ownerGeneration: 2,
sourceStartSu: 8,
sourceEndSu: 12
})
expect(acceptOutputDataMock).not.toHaveBeenCalled()
expect(closeSshPtyOutputGeneration).toHaveBeenCalledTimes(closeCount)
expect(deps.mockStore.markSshRemotePtyLease).toHaveBeenCalledWith(targetId, 'pty-1', 'detached')
expect(clearProviderPtyState).not.toHaveBeenCalled()
expect(clearPtyOwnershipForConnection).not.toHaveBeenCalled()
expect(deletePtyOwnership).not.toHaveBeenCalled()
expect(deps.mockWindow.webContents.send).not.toHaveBeenCalledWith('pty:exit', expect.anything())
expect(muxDisposeMock).not.toHaveBeenCalledWith('shutdown')
})
it('keeps a stale overlapping recovery from canceling or mutating its replacement', async () => {
const targetId = 'overlapping-recovery'
const { session, deps } = await prepareRecovery(targetId)
attachForReconnectMock.mockImplementation(async () => {
const ownerGeneration = openConsumerSessionMock.mock.calls.length
if (ownerGeneration === 3) {
queueMicrotask(() => {
const complete = onNotificationByMethodMock.mock.calls.findLast(
([method]) => method === 'pty.recoveryComplete'
)?.[1] as ((params: Record<string, unknown>) => void) | undefined
complete?.({
id: 'pty-1',
clientGeneration: 3,
ownerGeneration: 3,
ptyIncarnation: 'incarnation-1',
deliveryToken: 'replacement-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 4
})
})
}
return {
incarnationId: 'incarnation-1',
sourceRecovery: {
status: 'pending',
clientGeneration: ownerGeneration,
ownerGeneration,
ptyIncarnation: 'incarnation-1',
deliveryToken: ownerGeneration === 2 ? 'stale-token' : 'replacement-token',
checkpointSourceEndSu: 4,
recoveryEndSu: 4
}
}
})
const staleReconnect = session.reconnect(deps.mockConn)
await vi.waitFor(() => expect(attachForReconnectMock).toHaveBeenCalledTimes(1))
await Promise.resolve()
const replacementReconnect = session.reconnect(deps.mockConn)
await Promise.all([staleReconnect, replacementReconnect])
const recoveryRequests = attachForReconnectMock.mock.calls.map((call) => call[2])
expect(recoveryRequests).toHaveLength(2)
expect(recoveryRequests[1]).toMatchObject({
status: 'checkpoint',
deliveryToken: 'old-token',
acceptedSourceEndSu: 4
})
expect(muxRequestMock.mock.calls.filter(([method]) => method === 'pty.cancelDelivery')).toEqual(
[]
)
expect(deps.mockStore.markSshRemotePtyLease).toHaveBeenCalledTimes(1)
expect(deps.mockStore.markSshRemotePtyLease).toHaveBeenCalledWith(targetId, 'pty-1', 'attached')
expect(setPtyOwnership).toHaveBeenCalledTimes(1)
expect(clearProviderPtyState).not.toHaveBeenCalled()
expect(clearPtyOwnershipForConnection).not.toHaveBeenCalled()
expect(deletePtyOwnership).not.toHaveBeenCalled()
expect(deps.mockWindow.webContents.send).not.toHaveBeenCalledWith('pty:exit', expect.anything())
expect(muxDisposeMock).not.toHaveBeenCalledWith('shutdown')
expect(session.getState()).toBe('ready')
})
})
@@ -30,6 +30,7 @@ vi.mock('./ssh-channel-multiplexer', () => {
notify = vi.fn()
request = vi.fn().mockResolvedValue([])
onNotification = vi.fn().mockReturnValue(() => {})
onNotificationByMethod = vi.fn().mockReturnValue(() => {})
onRequest = vi.fn().mockReturnValue(() => {})
onDispose = vi.fn().mockReturnValue(() => {})
dispose = vi.fn()
+10 -4
View File
@@ -725,7 +725,7 @@ describe('SshRelaySession', () => {
})
})
it('routes transient reattach failures through relay-lost retry handling', async () => {
it('retries transient reattach failure without tearing down provider registration', async () => {
const { mockConn, mockStore, mockPortForward, getMainWindow } = createMockDeps()
const session = new SshRelaySession('target-1', getMainWindow, mockStore, mockPortForward)
const onRelayLost = vi.fn()
@@ -741,11 +741,17 @@ describe('SshRelaySession', () => {
dispose: vi.fn()
} as unknown as ReturnType<typeof getSshPtyProvider>)
vi.mocked(getPtyIdsForConnection).mockReturnValue(['pty-live'])
const random = vi.spyOn(Math, 'random').mockReturnValue(0)
await session.reconnect(mockConn)
try {
await session.reconnect(mockConn)
} finally {
random.mockRestore()
}
expect(mockAttach).toHaveBeenCalledWith('pty-live')
expect(onRelayLost).toHaveBeenCalledWith('target-1')
expect(mockAttach).toHaveBeenCalledTimes(2)
expect(onRelayLost).not.toHaveBeenCalled()
expect(session.getState()).toBe('ready')
expect(mockStore.markSshRemotePtyLease).not.toHaveBeenCalledWith(
'target-1',
'pty-live',
File diff suppressed because it is too large Load Diff
+18 -7
View File
@@ -300,24 +300,35 @@ describe('DispatcherClientWriter', () => {
expect(sink.accepted[1].data).toBe('cancel')
})
it('admits one bounded producer frame larger than an empty native sink window', () => {
it('enforces the encoded HWM-minus-reserve limit for splittable producers', () => {
const sink = new FakeSink()
sink.highWaterMark = 4096
sink.saturateNext = true
const writer = createWriter(sink, vi.fn(), 8192)
expect(writer.producerFrameCapacity).toBe(3072)
expect(writer.enqueue('bulk', () => Buffer.alloc(3073), 3073)).toBe(false)
expect(writer.enqueue('bulk', () => Buffer.alloc(3072), 3072)).toBe(true)
})
it('admits only one indivisible fixed filesystem frame before drain', () => {
const sink = new FakeSink()
sink.highWaterMark = 4096
sink.saturateNext = true
const writer = createWriter(sink, vi.fn(), 16_384)
const oversized = 'x'.repeat(5000)
expect(writer.producerFrameCapacity).toBe(3072)
const first = enqueue(writer, 'bulk', oversized)
enqueue(writer, 'ordinary', 'after-drain')
expect(writer.enqueue('bulk', () => Buffer.alloc(8193), 8193)).toBe(false)
expect(writer.enqueue('fixed-bulk', () => Buffer.from(oversized), 5000)).toBe(true)
expect(writer.enqueue('fixed-bulk', () => Buffer.from(oversized), 5000)).toBe(false)
expect(writer.enqueue('control', () => Buffer.from('cancel'), 6)).toBe(true)
expect(sink.accepted.map((write) => write.data)).toEqual([oversized])
sink.writableLength = 0
sink.accepted[0].settle({ ok: true })
expect(first).toHaveBeenCalledWith({ ok: true })
expect(sink.accepted).toHaveLength(1)
expect(writer.enqueue('fixed-bulk', () => Buffer.from(oversized), 5000)).toBe(false)
sink.drain()
expect(sink.accepted.map((write) => write.data)).toEqual([oversized, 'after-drain'])
expect(sink.accepted.map((write) => write.data)).toEqual([oversized, 'cancel'])
})
it('gives independent writers progress when one client stays saturated', () => {
+3 -3
View File
@@ -59,8 +59,8 @@ export class DispatcherClientWriter {
return this.sink.producerFrameCapacity
}
get producerQueueCapacity(): number {
return this.admission.producerQueueCapacity
get fixedFrameCapacity(): number {
return this.sink.frameCapacity('fixed-bulk', this.saturated)
}
canEnqueueProducer(bytes: number): boolean {
@@ -96,7 +96,7 @@ export class DispatcherClientWriter {
onSettled: onceDispatcherWriterSettlement(onSettled),
settled: false
}
const admission = this.admission.admit(entry, this.producerFrameCapacity)
const admission = this.admission.admit(entry, this.sink.frameCapacity(lane, this.saturated))
if (!admission.accepted) {
if (admission.error) {
entry.onSettled({ ok: false, error: admission.error })
+6 -6
View File
@@ -4,6 +4,7 @@ export type DispatcherWriterLane =
| 'legacy-response'
| 'interactive'
| 'ordinary'
| 'fixed-bulk'
| 'bulk'
export type DispatcherWriterSettlement = { ok: true } | { ok: false; error: Error }
@@ -53,6 +54,7 @@ export class DispatcherWriterAdmission {
'legacy-response': [],
interactive: [],
ordinary: [],
'fixed-bulk': [],
bulk: []
}
private controlBytes = 0
@@ -66,18 +68,13 @@ export class DispatcherWriterAdmission {
return this.producerBytes
}
get producerQueueCapacity(): number {
return this.producerQueueMaxBytes
}
get queuedEntries(): number {
return Object.values(this.queues).reduce((total, queue) => total + queue.length, 0)
}
canAdmitProducer(bytes: number, producerFrameCapacity: number): boolean {
return (
bytes <= Math.max(producerFrameCapacity, this.producerQueueMaxBytes) &&
this.producerBytes + bytes <= this.producerQueueMaxBytes
bytes <= producerFrameCapacity && this.producerBytes + bytes <= this.producerQueueMaxBytes
)
}
@@ -96,6 +93,9 @@ export class DispatcherWriterAdmission {
this.queues[entry.lane].push(entry)
return { accepted: true }
}
if (entry.lane === 'fixed-bulk' && this.producerBytes > 0) {
return { accepted: false }
}
if (!this.canAdmitProducer(entry.estimatedBytes, producerFrameCapacity)) {
return { accepted: false }
}
@@ -28,13 +28,13 @@ export class DispatcherWriterLaneScheduler {
if (legacyResponse && canWriteControl(legacyResponse)) {
return admission.shift('legacy-response')
}
const bulk = admission.peek('bulk')
const bulk = admission.peek('fixed-bulk') ?? admission.peek('bulk')
if (
bulk &&
this.producerWritesSinceBulk >= PRODUCER_WRITES_BEFORE_BULK &&
canWriteProducer(bulk)
) {
return admission.shift('bulk')
return admission.shift(bulk.lane)
}
const ordinary = admission.peek('ordinary')
if (
@@ -44,7 +44,7 @@ export class DispatcherWriterLaneScheduler {
) {
return admission.shift('ordinary')
}
for (const lane of ['interactive', 'ordinary', 'bulk'] as const) {
for (const lane of ['interactive', 'ordinary', 'fixed-bulk', 'bulk'] as const) {
const candidate = admission.peek(lane)
if (candidate && canWriteProducer(candidate)) {
return admission.shift(lane)
@@ -54,7 +54,7 @@ export class DispatcherWriterLaneScheduler {
}
recordWrite(lane: DispatcherWriterLane): void {
if (lane === 'bulk') {
if (lane === 'bulk' || lane === 'fixed-bulk') {
this.producerWritesSinceBulk = 0
} else if (lane === 'interactive' || lane === 'ordinary') {
this.producerWritesSinceBulk = Math.min(
+8
View File
@@ -1,5 +1,6 @@
import {
relayWriterControlReserve,
type DispatcherWriterLane,
type DispatcherWriterSettlement
} from './dispatcher-writer-admission'
@@ -47,6 +48,13 @@ export class DispatcherWriterSink {
return Math.max(0, this.highWaterMark - relayWriterControlReserve(this.highWaterMark))
}
frameCapacity(lane: DispatcherWriterLane, producerBlocked = false): number {
if (lane === 'fixed-bulk') {
return !producerBlocked && this.writableLength === 0 ? Number.MAX_SAFE_INTEGER : 0
}
return this.producerFrameCapacity
}
registerDrain(callback: () => void): { registered: boolean; remove: () => void } {
if (!this.options.waitWriteDrain) {
return { registered: false, remove: () => {} }
+13 -6
View File
@@ -863,11 +863,14 @@ export class RelayDispatcher {
private publishToClient(
client: RelayClient,
msg: JsonRpcNotification,
lane: 'interactive' | 'ordinary' | 'bulk',
lane: 'interactive' | 'ordinary' | 'fixed-bulk' | 'bulk',
onSettled: (result: SinkWriteSettlement) => void = () => {}
): boolean {
const bytes = this.estimateFrameBytes(msg)
if (!client.writer.canEnqueueProducer(bytes)) {
const fixedBlocked =
lane === 'fixed-bulk' &&
(client.writer.retainedProducerBytes > 0 || bytes > client.writer.fixedFrameCapacity)
if (fixedBlocked || (lane !== 'fixed-bulk' && !client.writer.canEnqueueProducer(bytes))) {
return false
}
const leases = this.publicationLedger.tryReserve([{ clientKey: this.clientKey(client), bytes }])
@@ -879,9 +882,13 @@ export class RelayDispatcher {
private publishBulkWhenAvailable(client: RelayClient, msg: JsonRpcNotification): Promise<void> {
const bytes = this.estimateFrameBytes(msg)
if (bytes > client.writer.producerQueueCapacity) {
const lane = msg.method === 'fs.streamChunk' ? 'fixed-bulk' : 'bulk'
if (bytes > DEFAULT_PRODUCER_QUEUE_MAX_BYTES) {
return Promise.reject(new Error('Relay bulk frame exceeds sink producer capacity'))
}
if (lane === 'bulk' && bytes > client.writer.producerFrameCapacity) {
return Promise.reject(new Error('Relay bulk frame exceeds sink frame capacity'))
}
return new Promise<void>((resolve, reject) => {
let removeCapacityListener: (() => void) | null = null
const finish = (): void => {
@@ -895,7 +902,7 @@ export class RelayDispatcher {
return
}
if (
this.publishToClient(client, msg, 'bulk', (result) => {
this.publishToClient(client, msg, lane, (result) => {
finish()
if (result.ok || this.disposed || client.closed) {
resolve()
@@ -917,14 +924,14 @@ export class RelayDispatcher {
private enqueueLeasedFrame(
client: RelayClient,
msg: JsonRpcNotification,
lane: 'interactive' | 'ordinary' | 'bulk',
lane: 'interactive' | 'ordinary' | 'fixed-bulk' | 'bulk',
lease: LegacyPublicationLease,
onSettled: (result: SinkWriteSettlement) => void = () => {}
): boolean {
const accepted = this.enqueueFrame(client, msg, lane, (result) => {
lease.release()
this.notifyLegacyCapacityIfLow()
onSettled(result)
this.notifyLegacyCapacityIfLow()
})
if (!accepted) {
lease.release()
@@ -28,6 +28,7 @@ import {
import { readFileViaStream } from '../main/ssh/ssh-filesystem-stream-reader'
import { RelayDispatcher } from './dispatcher'
import type { SinkWriteSettlement } from './dispatcher'
import { RelayContext } from './context'
import { FsHandler } from './fs-handler'
import { STREAM_CHUNK_SIZE } from './protocol'
@@ -95,7 +96,10 @@ function createHarness(opts: { congested: boolean }): Harness {
onClose: () => {}
}
const outQueue: Buffer[] = []
const outQueue: {
data: Buffer
settle: (result: SinkWriteSettlement) => void
}[] = []
let queuedBytes = 0
const drainWaiters = new Set<() => void>()
const fireDrainIfIdle = (): void => {
@@ -109,8 +113,8 @@ function createHarness(opts: { congested: boolean }): Harness {
}
const dispatcher = new RelayDispatcher(
(data: Buffer) => {
outQueue.push(data)
(data: Buffer, settle) => {
outQueue.push({ data, settle })
queuedBytes += data.length
if (!opts.congested) {
return true
@@ -118,6 +122,9 @@ function createHarness(opts: { congested: boolean }): Harness {
return queuedBytes < SINK_HIGH_WATER_MARK
},
{
supportsWriteCallback: true,
writableLength: () => queuedBytes,
writableHighWaterMark: () => SINK_HIGH_WATER_MARK,
waitWriteDrain: (cb: () => void) => {
drainWaiters.add(cb)
fireDrainIfIdle()
@@ -128,11 +135,12 @@ function createHarness(opts: { congested: boolean }): Harness {
const deliverAll = (): void => {
while (outQueue.length > 0) {
const buf = outQueue.shift()!
queuedBytes -= buf.length
const { data, settle } = outQueue.shift()!
queuedBytes -= data.length
for (const cb of clientDataCallbacks) {
cb(buf)
cb(data)
}
settle({ ok: true })
}
fireDrainIfIdle()
}
@@ -207,7 +215,7 @@ describe('fs.readFileStream vs pty.data echo head-of-line blocking', () => {
// The echo must not sit behind an unbounded chunk backlog: at most one
// in-flight bulk frame (the write that saturated the sink) plus slack.
expect(queuedBytesAheadOfEcho).toBeLessThan(2 * FRAMED_CHUNK_BYTES)
expect(queuedBytesAheadOfEcho).toBeLessThan(FRAMED_CHUNK_BYTES)
// Un-congest: the stream must still complete with intact content.
harness.startAutoDeliver()
@@ -28,6 +28,7 @@ import {
import { requestGitStreamable } from '../main/ssh/ssh-git-response-stream-reader'
import { RelayDispatcher } from './dispatcher'
import type { SinkWriteSettlement } from './dispatcher'
import { RelayContext } from './context'
import { GitHandler } from './git-handler'
import { GIT_RESPONSE_CHUNK_SIZE } from './protocol'
@@ -91,7 +92,10 @@ function createHarness(opts: { congested: boolean }): Harness {
onClose: () => {}
}
const outQueue: Buffer[] = []
const outQueue: {
data: Buffer
settle: (result: SinkWriteSettlement) => void
}[] = []
let queuedBytes = 0
const drainWaiters = new Set<() => void>()
const fireDrainIfIdle = (): void => {
@@ -105,8 +109,8 @@ function createHarness(opts: { congested: boolean }): Harness {
}
const dispatcher = new RelayDispatcher(
(data: Buffer) => {
outQueue.push(data)
(data: Buffer, settle) => {
outQueue.push({ data, settle })
queuedBytes += data.length
if (!opts.congested) {
return true
@@ -114,6 +118,9 @@ function createHarness(opts: { congested: boolean }): Harness {
return queuedBytes < SINK_HIGH_WATER_MARK
},
{
supportsWriteCallback: true,
writableLength: () => queuedBytes,
writableHighWaterMark: () => SINK_HIGH_WATER_MARK,
waitWriteDrain: (cb: () => void) => {
drainWaiters.add(cb)
fireDrainIfIdle()
@@ -124,11 +131,12 @@ function createHarness(opts: { congested: boolean }): Harness {
const deliverAll = (): void => {
while (outQueue.length > 0) {
const buf = outQueue.shift()!
queuedBytes -= buf.length
const { data, settle } = outQueue.shift()!
queuedBytes -= data.length
for (const cb of clientDataCallbacks) {
cb(buf)
cb(data)
}
settle({ ok: true })
}
fireDrainIfIdle()
}
+21
View File
@@ -41,6 +41,27 @@ function frame(id: number, payload = `${id}`): Buffer {
}
describe('relay FrameDecoder bounded turns', () => {
it('retains at most one maximum frame plus one MiB of partial input', () => {
expect(FRAME_DECODER_MAX_RETAINED_BYTES).toBe(MAX_MESSAGE_SIZE + HEADER_LENGTH + 1024 * 1024)
const maximumFrame = encodeFrame(MessageType.Regular, 1, 0, Buffer.alloc(MAX_MESSAGE_SIZE))
const acceptedError = vi.fn()
const accepted = new FrameDecoder(vi.fn(), acceptedError)
accepted.feed(Buffer.concat([maximumFrame, Buffer.alloc(1024 * 1024)]))
expect(acceptedError).not.toHaveBeenCalled()
expect(accepted.drain()).toHaveLength(1024 * 1024)
const excessError = vi.fn()
const excess = new FrameDecoder(vi.fn(), excessError)
excess.feed(Buffer.concat([maximumFrame, Buffer.alloc(1024 * 1024 + 1)]))
expect(excessError).toHaveBeenCalledWith(
expect.objectContaining({ message: expect.stringContaining('retained-input') })
)
expect(excess.drain()).toHaveLength(0)
})
it('fails closed when one delivered chunk exceeds retained-input capacity', () => {
const onError = vi.fn()
const decoder = new FrameDecoder(vi.fn(), onError)
@@ -0,0 +1,257 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { PTY_STARTUP_INGRESS_VERSION } from '../shared/pty-startup-ingress'
import {
RelayDispatcher,
type RelayClientSessionIdentity,
type SinkWriteSettlement
} from './dispatcher'
import { encodeJsonRpcFrame, MessageType } from './protocol'
import { PtyHandler } from './pty-handler'
import { RelayPtySourcePublication } from './relay-pty-source-publication'
import { SshPtyConsumerSessionAdapter } from './ssh-pty-consumer-session-adapter'
const { mockPtySpawn } = vi.hoisted(() => ({ mockPtySpawn: vi.fn() }))
vi.mock('node-pty', () => ({ spawn: mockPtySpawn }))
const endpointIdentity: RelayClientSessionIdentity = {
principal: 'endpoint-principal',
authenticated: true,
allowSessionOwner: true,
authenticationKind: 'endpoint-credential'
}
type Notification = { method: string; params: Record<string, unknown> }
function requestFrame(id: number, method: string, params: Record<string, unknown>): Buffer {
return encodeJsonRpcFrame({ jsonrpc: '2.0', id, method, params }, id, 0)
}
function notification(buffer: Buffer): Notification | null {
if (buffer[0] !== MessageType.Regular) {
return null
}
const length = buffer.readUInt32BE(9)
const message = JSON.parse(buffer.subarray(13, 13 + length).toString('utf8'))
return typeof message.method === 'string' && message.id === undefined ? message : null
}
function responseResult(buffer: Buffer, id: number): Record<string, unknown> | null {
if (buffer[0] !== MessageType.Regular) {
return null
}
const length = buffer.readUInt32BE(9)
const message = JSON.parse(buffer.subarray(13, 13 + length).toString('utf8'))
return message.id === id ? (message.result ?? null) : null
}
describe('PtyHandler negotiated source publication', () => {
let dispatcher: RelayDispatcher
let handler: PtyHandler
let publication: RelayPtySourcePublication
let dataCallback: ((data: string) => void) | undefined
let originalPlatform: PropertyDescriptor | undefined
let writes: Buffer[]
let heldResponseId: number | null
let heldResponseSettlements: ((result: SinkWriteSettlement) => void)[]
let adapter: SshPtyConsumerSessionAdapter
beforeEach(async () => {
vi.useFakeTimers()
originalPlatform = Object.getOwnPropertyDescriptor(process, 'platform')
writes = []
heldResponseId = null
heldResponseSettlements = []
dataCallback = undefined
mockPtySpawn.mockReset()
mockPtySpawn.mockReturnValue({
pid: process.pid,
onData: vi.fn((callback: (data: string) => void) => {
dataCallback = callback
}),
onExit: vi.fn(),
write: vi.fn(),
resize: vi.fn(),
kill: vi.fn(),
clear: vi.fn(),
pause: vi.fn(),
resume: vi.fn()
})
dispatcher = new RelayDispatcher(
(data, settle) => {
writes.push(Buffer.from(data))
if (heldResponseId !== null && responseResult(data, heldResponseId)) {
heldResponseSettlements.push(settle)
return true
}
settle({ ok: true })
return true
},
{ supportsWriteCallback: true },
endpointIdentity
)
handler = new PtyHandler(dispatcher)
adapter = new SshPtyConsumerSessionAdapter(dispatcher, 'build-a', undefined, (id) =>
publication.onCreditAvailable(id)
)
publication = new RelayPtySourcePublication(dispatcher, adapter, (id) =>
handler.handleSourcePublicationCapacity(id)
)
handler.setSourcePublication(publication)
dispatcher.feed(
requestFrame(1, 'pty.openClient', {
protocolVersion: 1,
clientInstanceId: 'client-1',
requestedRole: 'session-owner',
capabilities: { outputFlowControl: { versions: [1], requestedWindowSu: 1024 } }
})
)
await vi.advanceTimersByTimeAsync(0)
})
afterEach(async () => {
await handler.dispose({ waitForPhysicalExit: false }).catch(() => {})
dispatcher.dispose()
if (originalPlatform) {
Object.defineProperty(process, 'platform', originalPlatform)
}
vi.useRealTimers()
})
async function spawn(params: Record<string, unknown>): Promise<void> {
dispatcher.feed(requestFrame(2, 'pty.spawn', params))
await vi.advanceTimersByTimeAsync(0)
expect(dataCallback).toBeTypeOf('function')
}
function sourceDataFrames(): Notification[] {
return writes
.map(notification)
.filter((frame): frame is Notification => frame?.method === 'pty.data')
}
it('settles a consumed POSIX startup query before publishing its prompt', async () => {
Object.defineProperty(process, 'platform', { configurable: true, value: 'linux' })
await spawn({
startupIngressVersion: PTY_STARTUP_INGRESS_VERSION,
startupIngress: {
colors: { foreground: '#2e3434', background: '#ffffff' },
deadlineMs: 5_000
}
})
const query = '\x1b]10;?\x07'
dataCallback!(query)
dataCallback!('prompt')
await vi.advanceTimersByTimeAsync(9)
expect(sourceDataFrames().map((frame) => frame.params)).toEqual([
expect.objectContaining({
data: '',
rawLength: query.length,
transformed: true,
sourceLengthSu: query.length,
sourceEndSu: query.length
}),
expect.objectContaining({
data: 'prompt',
rawLength: 6,
sourceLengthSu: 6,
sourceEndSu: query.length + 6
})
])
expect(publication.getDebugSnapshot()).toMatchObject({ sendCommitted: 2 })
})
it('settles a suppressed ConPTY query before publishing its prompt', async () => {
Object.defineProperty(process, 'platform', { configurable: true, value: 'win32' })
await spawn({ shellOverride: 'powershell.exe' })
const query = '\x1b]10;?\x07'
dataCallback!(query)
dataCallback!('PS> ')
await vi.advanceTimersByTimeAsync(9)
expect(sourceDataFrames().map((frame) => frame.params)).toEqual([
expect.objectContaining({
data: '',
rawLength: query.length,
transformed: true,
sourceLengthSu: query.length,
sourceEndSu: query.length
}),
expect.objectContaining({
data: 'PS> ',
rawLength: 4,
sourceLengthSu: 4,
sourceEndSu: query.length + 4
})
])
expect(publication.getDebugSnapshot()).toMatchObject({ sendCommitted: 2 })
})
it('activates a fresh source consumer when an operation response is lost and retried', async () => {
const operationId = 'a'.repeat(43)
const grant = writes.map((buffer) => responseResult(buffer, 1)).find(Boolean)!
heldResponseId = 2
dispatcher.feed(
requestFrame(2, 'pty.spawn', {
agentSessionCreateOperationId: operationId
})
)
await vi.advanceTimersByTimeAsync(0)
expect(heldResponseSettlements).toHaveLength(1)
dispatcher.invalidateClient()
expect(adapter.getDebugSnapshot()).toMatchObject({ deliveryTokens: 0 })
const replacementWrites: Buffer[] = []
const replacementClientId = dispatcher.attachClient(
(data, settle) => {
replacementWrites.push(Buffer.from(data))
settle({ ok: true })
return true
},
{ supportsWriteCallback: true },
endpointIdentity
)
dispatcher.feedClient(
replacementClientId,
requestFrame(3, 'pty.openClient', {
protocolVersion: 1,
clientInstanceId: 'client-1',
requestedRole: 'session-owner',
resume: {
ownerGeneration: grant.ownerGeneration,
ownerLease: grant.ownerLease
},
capabilities: { outputFlowControl: { versions: [1], requestedWindowSu: 1024 } }
})
)
await vi.advanceTimersByTimeAsync(0)
dispatcher.feedClient(
replacementClientId,
requestFrame(4, 'pty.spawn', {
agentSessionCreateOperationId: operationId
})
)
await vi.advanceTimersByTimeAsync(0)
dataCallback!('prompt')
await vi.advanceTimersByTimeAsync(8)
expect(mockPtySpawn).toHaveBeenCalledOnce()
expect(
responseResult(replacementWrites.find((buffer) => responseResult(buffer, 4))!, 4)
).toEqual({ id: 'pty-1', incarnationId: expect.any(String) })
expect(
replacementWrites.map(notification).find((frame) => frame?.method === 'pty.data')?.params
).toMatchObject({
id: 'pty-1',
data: 'prompt',
sourceLengthSu: 6,
sourceEndSu: 6
})
expect(adapter.getDebugSnapshot()).toMatchObject({ deliveryTokens: 1 })
})
})
+9 -2
View File
@@ -876,6 +876,8 @@ export class PtyHandler {
const desiredChars = pending.transformed
? pending.data.length
: Math.min(pending.data.length, PTY_OUTPUT_FLUSH_CHUNK_CHARS)
const sourceOnlyEmission =
pending.transformed === true && pending.data.length === 0 && (pending.rawLength ?? 0) > 0
const paramsWithoutData = {
id,
...(pending.seq === undefined ? {} : { seq: pending.seq }),
@@ -894,7 +896,10 @@ export class PtyHandler {
) {
chunkChars--
}
if (chunkChars <= 0 || (pending.transformed && chunkChars !== pending.data.length)) {
if (
(!sourceOnlyEmission && chunkChars <= 0) ||
(pending.transformed && chunkChars !== pending.data.length)
) {
this.pendingOutputByPty.set(id, queue)
this.pausePtyOutput(id)
return false
@@ -1137,7 +1142,9 @@ export class PtyHandler {
}
const existing = this.agentSessionCreateOperations.get(operationId)
if (existing) {
return await existing
const result = await existing
this.sourcePublication?.activate(result.id, result.incarnationId, context)
return result
}
if (this.agentSessionCreateOperations.size >= AGENT_SESSION_CREATE_OPERATION_LIMIT) {
throw new Error('agent_session_operation_capacity')
+61 -1
View File
@@ -355,6 +355,66 @@ describe('RelayPtySourceCreditLedger', () => {
expect(ledger.retentionSnapshot()).toEqual({ sourceSu: 0, dataBytes: 0, spans: 0 })
})
it('charges UTF-16 storage and one record per retained source frame', () => {
const ledger = new RelayPtySourceCreditLedger({
maxRetainedDataBytes: 391,
maxAggregateRetainedDataBytes: 391
})
const owner = identity()
ledger.open(owner, 8)
append(ledger, owner, '\u0000'.repeat(64), 'utf16-heavy')
expect(ledger.retentionSnapshot()).toEqual({ sourceSu: 64, dataBytes: 256, spans: 1 })
expect(() => append(ledger, owner, 'a'.repeat(8), 'fragment')).toThrow('retained encoded-data')
expect(ledger.retentionSnapshot()).toEqual({ sourceSu: 64, dataBytes: 256, spans: 1 })
})
it('enforces per-PTY and aggregate source, charged-byte, and frame caps across fragments', () => {
const limits = {
maxRetainedSourceSu: 6,
maxAggregateRetainedSourceSu: 9,
maxRetainedDataBytes: 500,
maxAggregateRetainedDataBytes: 700,
maxRetainedSpans: 2,
maxAggregateRetainedSpans: 3
}
const first = identity('token-a')
const second = identity('token-b', {
id: 'pty-2',
ptyIncarnation: 'incarnation-2'
})
const ledger = new RelayPtySourceCreditLedger(limits)
ledger.open(first, 8)
ledger.open(second, 8)
append(ledger, first, 'aa', 'first-a')
append(ledger, first, 'bb', 'first-b')
append(ledger, second, 'cc', 'second-a')
expect(ledger.retentionSnapshot()).toEqual({
sourceSu: 6,
dataBytes: 396,
spans: 3
})
expect(() => append(ledger, first, 'c', 'first-frame-cap')).toThrow('retained-span')
expect(() => append(ledger, second, 'dddd', 'aggregate-source-cap')).toThrow('Aggregate')
const byteLimited = new RelayPtySourceCreditLedger({
...limits,
maxRetainedSourceSu: 200,
maxAggregateRetainedSourceSu: 300,
maxAggregateRetainedSpans: 10,
maxRetainedSpans: 10,
maxAggregateRetainedDataBytes: 650
})
byteLimited.open(first, 8)
byteLimited.open(second, 8)
append(byteLimited, first, '\u0000'.repeat(64), 'first-utf16')
append(byteLimited, second, '\u0000'.repeat(64), 'second-utf16')
expect(() => append(byteLimited, second, '\u0000'.repeat(8), 'aggregate-byte-cap')).toThrow(
'Aggregate'
)
})
it('bounds retained frame metadata independently of source units', () => {
const ledger = new RelayPtySourceCreditLedger({
maxRetainedSpans: 2,
@@ -366,7 +426,7 @@ describe('RelayPtySourceCreditLedger', () => {
append(ledger, owner, 'b', 'span-b')
expect(() => append(ledger, owner, 'c', 'span-c')).toThrow('retained-span')
expect(ledger.retentionSnapshot()).toEqual({ sourceSu: 2, dataBytes: 2, spans: 2 })
expect(ledger.retentionSnapshot()).toEqual({ sourceSu: 2, dataBytes: 260, spans: 2 })
})
it('bounds closed token tombstones', () => {
+7 -5
View File
@@ -38,6 +38,7 @@ import {
type PtySourceAppendInput,
type PtySourceSendReservation
} from './pty-source-credit-record'
import { chargedPtyRetainedStringBytes } from '../shared/pty-retained-string-memory'
export type { PtySourceSendReservation } from './pty-source-credit-record'
export type { PtySourceCreditLedgerOptions } from './pty-source-credit-limits'
@@ -78,8 +79,9 @@ export class RelayPtySourceCreditLedger {
append(identity: PtySourceDeliveryIdentity, input: PtySourceAppendInput): PtySourceSpan {
const record = this.requireActive(identity)
const sourceEndSu = record.receivedEndSu + input.transform.rawLengthSu
const dataBytes = Buffer.byteLength(input.data, 'utf8')
if (dataBytes > MAX_SOURCE_SPAN_DATA_BYTES) {
const encodedDataBytes = Buffer.byteLength(input.data, 'utf8')
const retainedDataBytes = chargedPtyRetainedStringBytes(input.data)
if (encodedDataBytes > MAX_SOURCE_SPAN_DATA_BYTES) {
throw new Error('PTY source span encoded-data budget exceeded')
}
if (sourceEndSu - record.creditedEndSu > this.limits.maxRetainedSourceSu) {
@@ -91,10 +93,10 @@ export class RelayPtySourceCreditLedger {
) {
throw new Error('Aggregate PTY source retained-range budget exceeded')
}
if (record.retainedDataBytes + dataBytes > this.limits.maxRetainedDataBytes) {
if (record.retainedDataBytes + retainedDataBytes > this.limits.maxRetainedDataBytes) {
throw new Error('PTY source retained encoded-data budget exceeded')
}
if (this.retainedDataBytes() + dataBytes > this.limits.maxAggregateRetainedDataBytes) {
if (this.retainedDataBytes() + retainedDataBytes > this.limits.maxAggregateRetainedDataBytes) {
throw new Error('Aggregate PTY source retained encoded-data budget exceeded')
}
if (record.spans.length + 1 > this.limits.maxRetainedSpans) {
@@ -105,7 +107,7 @@ export class RelayPtySourceCreditLedger {
}
const span = createAppendedSourceSpan(record, input)
record.spans.push(span)
record.retainedDataBytes += dataBytes
record.retainedDataBytes += retainedDataBytes
record.receivedEndSu = sourceEndSu
return span
}
+3 -2
View File
@@ -14,6 +14,7 @@ import {
assertPtySourceIdentity,
assertPtySourceSpan
} from '../shared/pty-source-credit-validation'
import { chargedPtyRetainedStringBytes } from '../shared/pty-retained-string-memory'
export const DEFAULT_RETAINED_SOURCE_SU = 512 * 1024
export const DEFAULT_AGGREGATE_RETAINED_SOURCE_SU = 48 * 1024 * 1024
@@ -172,7 +173,7 @@ export function snapshotDeliveryRecord(record: DeliveryRecord): PtySourceDeliver
export function reclaimCreditedSpans(record: DeliveryRecord): void {
while (record.spans[0]?.sourceEndSu <= record.creditedEndSu) {
record.retainedDataBytes -= Buffer.byteLength(record.spans.shift()!.data, 'utf8')
record.retainedDataBytes -= chargedPtyRetainedStringBytes(record.spans.shift()!.data)
}
}
@@ -272,7 +273,7 @@ export function createReplacementDeliveryRecord(
.map((span) => sliceAtSourceStart(span, Math.max(span.sourceStartSu, acceptedSourceEndSu)))
.map((span) => Object.freeze({ ...span, ...replacement.identity }))
replacement.retainedDataBytes = replacement.spans.reduce(
(bytes, span) => bytes + Buffer.byteLength(span.data, 'utf8'),
(bytes, span) => bytes + chargedPtyRetainedStringBytes(span.data),
0
)
return replacement
+1 -1
View File
@@ -6,7 +6,7 @@ export const MAX_MESSAGE_SIZE = 16 * 1024 * 1024
export const FRAME_DECODER_MAX_FRAMES_PER_TURN = 64
export const FRAME_DECODER_MAX_BYTES_PER_TURN = MAX_MESSAGE_SIZE + HEADER_LENGTH
export const FRAME_DECODER_MAX_TURN_MS = 4,
FRAME_DECODER_MAX_RETAINED_BYTES = 2 * (MAX_MESSAGE_SIZE + HEADER_LENGTH)
FRAME_DECODER_MAX_RETAINED_BYTES = MAX_MESSAGE_SIZE + HEADER_LENGTH + 1024 * 1024
export class FrameDecoder {
private chunks: Buffer[] = []
+35 -2
View File
@@ -49,7 +49,11 @@ describe('RelayPtySourcePublication', () => {
dispatcher = null
})
async function createHarness(windowSu = 4, settleSourceImmediately = true) {
async function createHarness(
windowSu = 4,
settleSourceImmediately = true,
highWaterMark?: number
) {
const writes: Buffer[] = []
const sourceSettlements: ((result: SinkWriteSettlement) => void)[] = []
dispatcher = new RelayDispatcher(
@@ -66,7 +70,15 @@ describe('RelayPtySourcePublication', () => {
}
return true
},
{ supportsWriteCallback: true },
{
supportsWriteCallback: true,
...(highWaterMark
? {
writableLength: () => 0,
writableHighWaterMark: () => highWaterMark
}
: {})
},
endpointIdentity
)
let publication: RelayPtySourcePublication
@@ -137,6 +149,27 @@ describe('RelayPtySourcePublication', () => {
await flushRequests()
})
it('slices source frames to the encoded HWM-minus-reserve capacity', async () => {
const highWaterMark = 4096
const harness = await createHarness(10_000, true, highWaterMark)
const payload = '\u0000'.repeat(4000)
expect(harness.publication.publish('pty-1', { data: payload }, false)).toBe(true)
for (let turn = 0; turn < 4; turn++) {
await flushRequests()
}
const frames = harness.writes.filter((buffer) => notification(buffer)?.method === 'pty.data')
expect(frames.length).toBeGreaterThan(1)
expect(Math.max(...frames.map((buffer) => buffer.length))).toBeLessThanOrEqual(3072)
expect(
frames
.map(notification)
.map((frame) => frame!.params.data)
.join('')
).toBe(payload)
})
it('keeps mixed legacy and V1 clients on distinct frame authority', async () => {
const harness = await createHarness(8)
const legacyWrites: Buffer[] = []
+23 -1
View File
@@ -35,6 +35,8 @@ export type RelayPtySourcePublicationCounters = {
exitRolledBack: number
}
const PTY_SOURCE_FRAME_MAX_SU = 16 * 1024
export function onceSinkSettlement(
callback: (result: SinkWriteSettlement) => void
): (result: SinkWriteSettlement) => void {
@@ -153,7 +155,27 @@ export class RelayPtySourceSendScheduler {
})
return
}
const reservation = this.session.reserveSourceSend(record.identity)
const snapshot = this.session.sourceDeliverySnapshot(record.identity)
const encodedDataBudget = this.dispatcher.producerDataBudget(
'pty.data',
{
id: record.identity.id,
rawLength: PTY_SOURCE_FRAME_MAX_SU,
transformed: false,
deliveryToken: record.identity.deliveryToken,
clientGeneration: record.identity.clientGeneration,
ownerGeneration: record.identity.ownerGeneration,
ptyIncarnation: record.identity.ptyIncarnation,
sourceEndSu: snapshot.receivedEndSu,
sourceLengthSu: PTY_SOURCE_FRAME_MAX_SU
},
record.clientId
)
const maxSourceSu = Math.min(
PTY_SOURCE_FRAME_MAX_SU,
Math.max(1, Math.floor(Math.max(0, encodedDataBudget - 32) / 6))
)
const reservation = this.session.reserveSourceSend(record.identity, maxSourceSu)
if (!reservation) {
return
}
@@ -1,5 +1,6 @@
export const PTY_CONSUMER_SESSION_PROTOCOL_VERSION = 1
export const PTY_CONSUMER_OWNER_GRACE_MS = 30_000
export const PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR = -32041
export type PtyConsumerRole = 'session-owner' | 'subscriber'
+17
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest'
import {
PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR,
PtyConsumerSession,
type PtyConsumerAuthentication,
type PtyConsumerSessionHello
@@ -37,6 +38,22 @@ function createSession(options: { now?: () => number } = {}): PtyConsumerSession
}
describe('PtyConsumerSession', () => {
it('types a resume against a fresh relay without weakening other owner refusals', () => {
const session = createSession()
expect(() =>
session.admit(
ownerHello({ resume: { ownerGeneration: 1, ownerLease: 'stale' } }),
auth('connection-1')
)
).toThrow(
expect.objectContaining({
code: PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR,
message: expect.stringContaining('stale')
})
)
})
it('activates an authenticated owner only after its publication fence', () => {
const session = createSession()
const first = session.admit(ownerHello(), auth('connection-1'))
+4 -1
View File
@@ -2,6 +2,7 @@ import { randomUUID } from 'node:crypto'
import {
PTY_CONSUMER_OWNER_GRACE_MS,
PTY_CONSUMER_SESSION_PROTOCOL_VERSION,
PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR,
type PtyConsumerAuthentication,
type PtyConsumerSessionAdmission,
type PtyConsumerSessionGrant,
@@ -244,7 +245,9 @@ export class PtyConsumerSession {
const current = this.owner
if (!current) {
if (hello.resume) {
throw new Error('Owner recovery lease is stale')
throw Object.assign(new Error('Owner recovery lease is stale'), {
code: PTY_CONSUMER_STALE_OWNER_RECOVERY_ERROR
})
}
return this.newOwner(hello, authentication, null)
}
+5
View File
@@ -0,0 +1,5 @@
export const PTY_RETAINED_RECORD_BYTES = 128
export function chargedPtyRetainedStringBytes(value: string): number {
return Math.max(Buffer.byteLength(value, 'utf8'), 2 * value.length) + PTY_RETAINED_RECORD_BYTES
}