fix(remote): baseline unsequenced terminal probes

config/reliability-gates.jsonc

@@ -3088,13 +3088,13 @@
       "providers": ["paired-runtime", "ssh"],
       "coveredPlatforms": ["macos"],
       "coveredProviders": ["paired-runtime"],
-      "coverageNotes": "Deterministic headed macOS runs launch an isolated Orca desktop server and a separate paired web client. A byte-identical headless run uses an isolated `orca serve` host. Both create six real paired host PTYs with bounded high-output scrollback, ordinary-park five client xterms without enabling the lossy retention budget, assert bounded cells/heap/timer lag, then restore 5,000 rows on the original PTY including output produced while parked and continued input/output. The headed oracle additionally keeps every remote target workspace and terminal unmounted in the host renderer before and after client recovery. Separate headed/headless ACK-starvation tests recover one stalled stream without replacing its PTY. A headed output-drop oracle proves a successful snapshot probe cannot certify a stale live stream when the authoritative PTY sequence advanced beyond the client high-water. A capability-disabled run proves legacy hosts retain the prior lossy limit/TTL fallback. Unit tests cover exact-owner capability routing, raw-stream release, singleton side-effect facts with timed handoff cleanup, provider-authoritative snapshots, 128 active streams plus retry after a 129th-stream capacity rejection, capacity-pressure backoff, full split-leaf remint reconciliation, truncation, and manual-disconnect queue fencing. Linux, Windows, mixed-version paired hosts, live SSH, and production-scale paired hosts remain gaps.",
+      "coverageNotes": "Deterministic headed macOS runs launch an isolated Orca desktop server and a separate paired web client. A byte-identical headless run uses an isolated `orca serve` host. Both create six real paired host PTYs with bounded high-output scrollback, ordinary-park five client xterms without enabling the lossy retention budget, assert bounded cells/heap/timer lag, then restore 5,000 rows on the original PTY including output produced while parked and continued input/output. The headed oracle additionally keeps every remote target workspace and terminal unmounted in the host renderer before and after client recovery. Separate headed/headless ACK-starvation tests recover one stalled stream without replacing its PTY. A headed output-drop oracle proves a successful snapshot probe cannot certify a stale live stream when the authoritative PTY sequence advanced beyond the client high-water. Unit coverage requires an unsequenced client to establish a first-probe baseline, remain attached at the same sequence, and recover only after a later probe advances. A capability-disabled run proves legacy hosts retain the prior lossy limit/TTL fallback. Unit tests cover exact-owner capability routing, raw-stream release, singleton side-effect facts with timed handoff cleanup, provider-authoritative snapshots, 128 active streams plus retry after a 129th-stream capacity rejection, capacity-pressure backoff, full split-leaf remint reconciliation, truncation, and manual-disconnect queue fencing. Linux, Windows, mixed-version paired hosts, live SSH, and production-scale paired hosts remain gaps.",
       "motivatingLinks": [
         "https://github.com/stablyai/orca/issues/8652",
         "https://github.com/stablyai/orca/pull/10625"
       ],
-      "invariant": "A host advertising terminal.paired-parking.v1 keeps the PTY and bounded authoritative history alive while an ordinary hidden-view park destroys the client xterm and releases its raw per-PTY stream. Reveal must restore up to the requested 5,000 rows, parked-time side effects/output, the same PTY identity, and continued input/output. Hosts without the capability must retain the existing limit/TTL force-parking fallback. A paired terminal stream whose delivery credits stop progressing must replace only that stream; command silence first probes authoritative state and replaces the stream if the probe times out or proves the PTY advanced beyond the client's delivered output sequence. A same-sequence snapshot remains valid proof of a responsive silent command. A successful status probe may replace a pre-ready shared-control socket without rejecting or duplicating calls already waiting for that transport. Manual disconnect must retain pairing while preventing queued or passive calls and subscriptions from recreating transport until explicit Connect.",
-      "oracle": "Run one byte-identical six-terminal oracle against an isolated headed desktop host and an isolated headless `orca serve` host. Stage at least 1,000,000 xterm cells, enable ordinary parking with the lossy retention budget disabled, require exactly one mounted manager and five parked tabs, at most 45% retained cells, no more than 16 MiB heap growth, and under 500 ms timer drift. In headed mode, require the host renderer to remain on its original workspace with zero target terminal managers mounted throughout client park, reveal, and live I/O. While a tab is parked, require authoritative terminal.read to observe new PTY output; reveal it and require the original PTY, a marker within the requested 5,000-row history, the parked marker, and post-reveal input/output. Drop and acknowledge output only for one original paired-client stream, prove the fixture process consumed input while the host model advanced and the client stayed stale, then require the command snapshot probe to replace that stream, repaint exact fixture output, preserve authoritative PTY identity and target-tab cardinality, and resume live I/O. Withhold the first encrypted shared-control ready frame, start one RPC, trigger a status-probe refresh, and require two connections, one host delivery, successful response, and zero retained request bytes. Admit 128 active streams, reject the 129th as retryable, release one stream, then require the retry to attach and publish a snapshot without multiplying retained subscribers. Disable terminal.paired-parking.v1 and require the same oracle to fail before parking, while the legacy limit-one fallback separately passes. Also preserve truncated first paint, ACK-starved same-PTY recovery, responsive silent-command snapshot probes, dead-stream probe timeout recovery, and queued manual-disconnect fencing.",
+      "invariant": "A host advertising terminal.paired-parking.v1 keeps the PTY and bounded authoritative history alive while an ordinary hidden-view park destroys the client xterm and releases its raw per-PTY stream. Reveal must restore up to the requested 5,000 rows, parked-time side effects/output, the same PTY identity, and continued input/output. Hosts without the capability must retain the existing limit/TTL force-parking fallback. A paired terminal stream whose delivery credits stop progressing must replace only that stream; command silence first probes authoritative state and replaces the stream if the probe times out or proves the PTY advanced beyond the client's delivered output sequence. When no comparable delivery high-water exists, the first sequenced probe establishes a baseline and only later advancement proves staleness. A same-sequence snapshot remains valid proof of a responsive silent command. A successful status probe may replace a pre-ready shared-control socket without rejecting or duplicating calls already waiting for that transport. Manual disconnect must retain pairing while preventing queued or passive calls and subscriptions from recreating transport until explicit Connect.",
+      "oracle": "Run one byte-identical six-terminal oracle against an isolated headed desktop host and an isolated headless `orca serve` host. Stage at least 1,000,000 xterm cells, enable ordinary parking with the lossy retention budget disabled, require exactly one mounted manager and five parked tabs, at most 45% retained cells, no more than 16 MiB heap growth, and under 500 ms timer drift. In headed mode, require the host renderer to remain on its original workspace with zero target terminal managers mounted throughout client park, reveal, and live I/O. While a tab is parked, require authoritative terminal.read to observe new PTY output; reveal it and require the original PTY, a marker within the requested 5,000-row history, the parked marker, and post-reveal input/output. Drop and acknowledge output only for one original paired-client stream, prove the fixture process consumed input while the host model advanced and the client stayed stale, then require the command snapshot probe to replace that stream, repaint exact fixture output, preserve authoritative PTY identity and target-tab cardinality, and resume live I/O. For a client without a delivered sequence, require the first numeric probe to establish a baseline without replacement, a same-sequence probe to remain attached, and a later advanced probe to replace only that stream. Withhold the first encrypted shared-control ready frame, start one RPC, trigger a status-probe refresh, and require two connections, one host delivery, successful response, and zero retained request bytes. Admit 128 active streams, reject the 129th as retryable, release one stream, then require the retry to attach and publish a snapshot without multiplying retained subscribers. Disable terminal.paired-parking.v1 and require the same oracle to fail before parking, while the legacy limit-one fallback separately passes. Also preserve truncated first paint, ACK-starved same-PTY recovery, responsive silent-command snapshot probes, dead-stream probe timeout recovery, and queued manual-disconnect fencing.",
       "commands": [
         "pnpm exec vitest run --config config/vitest.config.ts src/main/runtime/orca-runtime.test.ts src/main/runtime/rpc/terminal-multiplex.test.ts src/main/runtime/rpc/terminal-subscribe-buffer.test.ts src/renderer/src/components/terminal-pane/parked-terminal-byte-watcher.test.ts src/renderer/src/components/terminal-pane/terminal-side-effect-facts-handler.test.ts src/renderer/src/components/terminal-pane/pty-connection.test.ts src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts src/renderer/src/components/terminal-pane/terminal-hidden-view-parking.test.ts src/renderer/src/components/terminal-pane/terminal-hidden-worktree-retention.test.ts src/renderer/src/components/terminal-pane/terminal-parked-tab-watchers.test.ts src/renderer/src/components/terminal-pane/terminal-parked-watcher-reconciliation.test.ts src/renderer/src/components/terminal-pane/terminal-parked-watcher-partial-reconciliation.test.ts src/renderer/src/components/terminal-pane/terminal-parking-e2e-overrides.test.ts src/renderer/src/runtime/remote-runtime-terminal-stall-recovery.test.ts src/renderer/src/runtime/runtime-client-events.test.ts src/renderer/src/web/web-preload-api.test.ts src/main/ipc/runtime-environments.test.ts",
         "pnpm exec vitest run --config config/vitest.config.ts src/shared/remote-runtime-shared-control-connection.test.ts",
@@ -3239,7 +3239,7 @@
             "probes then restarts a stream when an entered command receives no frames",
             "keeps a silent responsive stream after its authoritative snapshot probe",
             "restarts a stream when the authoritative snapshot advanced without live output",
-            "restarts when a sequenced probe has no delivered high-water",
+            "establishes a probe baseline before recovering an unsequenced stream",
             "keeps a responsive stream when its probe confirms zero output high-water",
             "classifies a capacity rejection followed by end as recoverable transport pressure"
           ]
@@ -6522,7 +6522,9 @@
         },
         {
           "file": "src/renderer/src/runtime/runtime-terminal-stream.test.ts",
-          "assertions": ["drops output only from the exact stream armed by the E2E fault gate"]
+          "assertions": [
+            "drops output only from the armed stream when its replacement reuses the stream ID"
+          ]
         },
         {
           "file": "src/main/runtime/rpc/terminal-multiplex-flow-control.bench.test.ts",

src/renderer/src/runtime/remote-runtime-terminal-multiplexer.ts

@@ -110,6 +110,7 @@ type RemoteRuntimeMultiplexedTerminalState = {
   // Track it so a gap triggers a self-healing snapshot resync instead of
   // silently rendering corrupt/missing output (frame-drop resync).
   expectedSeq: number | undefined
+  commandProbeBaselineSeq: number | undefined
   recoverySnapshotSeq: number | undefined
   resyncInFlight: boolean
   resyncPendingSend: boolean
@@ -331,6 +332,7 @@ class RemoteRuntimeTerminalMultiplexer {
       initialSnapshotReceived: false,
       pendingSnapshotRequest: null,
       expectedSeq: undefined,
+      commandProbeBaselineSeq: undefined,
       recoverySnapshotSeq: undefined,
       resyncInFlight: false,
       resyncPendingSend: false,
@@ -693,6 +695,7 @@ class RemoteRuntimeTerminalMultiplexer {
         }
         if (typeof seq === 'number') {
           stream.expectedSeq = seq
+          stream.commandProbeBaselineSeq = undefined
         }
         stream.callbacks.onData(data, {
           seq,
@@ -795,6 +798,7 @@ class RemoteRuntimeTerminalMultiplexer {
       if (target === 'initial') {
         clearResyncTimer(stream)
         stream.expectedSeq = typeof info?.seq === 'number' ? info.seq : undefined
+        stream.commandProbeBaselineSeq = undefined
         stream.resyncInFlight = false
         stream.resyncPendingSend = false
         stream.initialSnapshotReceived = true
@@ -805,6 +809,7 @@ class RemoteRuntimeTerminalMultiplexer {
         if (snapshotApplied) {
           clearResyncTimer(stream)
           stream.expectedSeq = typeof info?.seq === 'number' ? info.seq : undefined
+          stream.commandProbeBaselineSeq = undefined
           stream.recoverySnapshotSeq = typeof info?.seq === 'number' ? info.seq : undefined
           stream.resyncAttempts = 0
           stream.resyncInFlight = false
@@ -1049,24 +1054,23 @@ class RemoteRuntimeTerminalMultiplexer {
         if (this.streams.get(stream.streamId) !== stream) {
           return
         }
-        if (
-          typeof snapshot?.seq === 'number' &&
-          snapshot.seq > 0 &&
-          typeof stream.expectedSeq !== 'number'
-        ) {
-          recordRendererCrashBreadcrumb(
-            'remote_terminal_stream_stall_probe_missing_delivery_high_water',
-            {
+        if (typeof snapshot?.seq === 'number' && typeof stream.expectedSeq !== 'number') {
+          if (
+            typeof stream.commandProbeBaselineSeq === 'number' &&
+            snapshot.seq > stream.commandProbeBaselineSeq
+          ) {
+            recordRendererCrashBreadcrumb('remote_terminal_stream_stall_probe_baseline_advanced', {
+              baselineSeq: stream.commandProbeBaselineSeq,
               environmentId: this.environmentId,
               snapshotSeq: snapshot.seq,
               streamId: stream.streamId,
               terminal: stream.terminal
-            }
-          )
-          this.recoverStalledStream(stream)
-          return
-        }
-        if (
+            })
+            this.recoverStalledStream(stream)
+            return
+          }
+          stream.commandProbeBaselineSeq = snapshot.seq
+        } else if (
           typeof snapshot?.seq === 'number' &&
           typeof stream.expectedSeq === 'number' &&
           snapshot.seq > stream.expectedSeq
@@ -1085,6 +1089,7 @@ class RemoteRuntimeTerminalMultiplexer {
         recordRendererCrashBreadcrumb('remote_terminal_stream_stall_probe_succeeded', {
           deliveredSeq: stream.expectedSeq ?? null,
           environmentId: this.environmentId,
+          probeBaselineSeq: stream.commandProbeBaselineSeq ?? null,
           snapshotSeq: snapshot?.seq ?? null,
           streamId: stream.streamId,
           terminal: stream.terminal

src/renderer/src/runtime/remote-runtime-terminal-stall-recovery.test.ts

@@ -142,7 +142,7 @@ describe('remote terminal stalled stream recovery', () => {
       client: { id: 'mac-viewer', type: 'desktop' },
       callbacks: { onData: vi.fn(), onSnapshot: vi.fn(), onTransportClose }
     })
-    emitOutput(stream.streamId, 'baseline')
+    emitSnapshot(stream.streamId, undefined, 'baseline', 8)
     sendBinary.mockClear()
 
     expect(stream.sendInput('secret\r')).toBe(true)
@@ -153,7 +153,7 @@ describe('remote terminal stalled stream recovery', () => {
       : null
     expect(firstRequestPayload?.requestId).toBeTypeOf('number')
 
-    emitRequestedSnapshot(stream.streamId, firstRequestPayload?.requestId ?? 0, 'baseline', 8)
+    emitSnapshot(stream.streamId, firstRequestPayload?.requestId ?? 0, 'baseline', 8)
     await vi.advanceTimersByTimeAsync(0)
 
     expect(onTransportClose).not.toHaveBeenCalled()
@@ -183,7 +183,7 @@ describe('remote terminal stalled stream recovery', () => {
     const payload = request
       ? decodeTerminalStreamJson<{ requestId: number }>(request.payload)
       : null
-    emitRequestedSnapshot(stream.streamId, payload?.requestId ?? 0, 'baselinemissing', 15)
+    emitSnapshot(stream.streamId, payload?.requestId ?? 0, 'baselinemissing', 15)
     await vi.advanceTimersByTimeAsync(0)
 
     expect(onData).toHaveBeenCalledOnce()
@@ -191,7 +191,7 @@ describe('remote terminal stalled stream recovery', () => {
     expect(sentUnsubscribeStreamIds()).toEqual([stream.streamId])
   })
 
-  it('restarts when a sequenced probe has no delivered high-water', async () => {
+  it('establishes a probe baseline before recovering an unsequenced stream', async () => {
     const { getRemoteRuntimeTerminalMultiplexer } =
       await import('./remote-runtime-terminal-multiplexer')
     const onTransportClose = vi.fn()
@@ -208,7 +208,27 @@ describe('remote terminal stalled stream recovery', () => {
     const payload = request
       ? decodeTerminalStreamJson<{ requestId: number }>(request.payload)
       : null
-    emitRequestedSnapshot(stream.streamId, payload?.requestId ?? 0, 'first-output', 12)
+    emitSnapshot(stream.streamId, payload?.requestId ?? 0, 'first-output', 12)
+    await vi.advanceTimersByTimeAsync(0)
+
+    expect(onTransportClose).not.toHaveBeenCalled()
+    expect(stream.sendInput('silent-command\r')).toBe(true)
+    await vi.advanceTimersByTimeAsync(REMOTE_TERMINAL_COMMAND_RESPONSE_TIMEOUT_MS)
+    const sameRequest = sentFrames(TerminalStreamOpcode.SnapshotRequest)[1]
+    const samePayload = sameRequest
+      ? decodeTerminalStreamJson<{ requestId: number }>(sameRequest.payload)
+      : null
+    emitSnapshot(stream.streamId, samePayload?.requestId ?? 0, 'first-output', 12)
+    await vi.advanceTimersByTimeAsync(0)
+
+    expect(onTransportClose).not.toHaveBeenCalled()
+    expect(stream.sendInput('echo second-output\r')).toBe(true)
+    await vi.advanceTimersByTimeAsync(REMOTE_TERMINAL_COMMAND_RESPONSE_TIMEOUT_MS)
+    const advancedRequest = sentFrames(TerminalStreamOpcode.SnapshotRequest)[2]
+    const advancedPayload = advancedRequest
+      ? decodeTerminalStreamJson<{ requestId: number }>(advancedRequest.payload)
+      : null
+    emitSnapshot(stream.streamId, advancedPayload?.requestId ?? 0, 'second-output', 25)
     await vi.advanceTimersByTimeAsync(0)
 
     expect(onTransportClose).toHaveBeenCalledWith({ recoverable: true })
@@ -232,7 +252,7 @@ describe('remote terminal stalled stream recovery', () => {
     const payload = request
       ? decodeTerminalStreamJson<{ requestId: number }>(request.payload)
       : null
-    emitRequestedSnapshot(stream.streamId, payload?.requestId ?? 0, '', 0)
+    emitSnapshot(stream.streamId, payload?.requestId ?? 0, '', 0)
     await vi.advanceTimersByTimeAsync(0)
 
     expect(onTransportClose).not.toHaveBeenCalled()
@@ -317,9 +337,9 @@ describe('remote terminal stalled stream recovery', () => {
     )
   }
 
-  function emitRequestedSnapshot(
+  function emitSnapshot(
     streamId: number,
-    requestId: number,
+    requestId: number | undefined,
     data = '',
     seq?: number
   ): void {

src/renderer/src/runtime/runtime-terminal-stream.test.ts

@@ -462,7 +462,7 @@ describe('remote runtime terminal multiplex ACK gate', () => {
     liveTerminal.close()
   })
 
-  it('drops output only from the exact stream armed by the E2E fault gate', async () => {
+  it('drops output only from the armed stream when its replacement reuses the stream ID', async () => {
     const { getRemoteRuntimeTerminalMultiplexer, resetRemoteRuntimeTerminalMultiplexersForTests } =
       await import('./remote-runtime-terminal-multiplexer')
     resetRemoteRuntimeTerminalMultiplexersForTests()