From fb52c0602a33bdb755d5da5f9a34bb77defffa03 Mon Sep 17 00:00:00 2001 From: Neil <4138956+nwparker@users.noreply.github.com> Date: Tue, 29 Sep 2026 20:27:30 -0700 Subject: [PATCH] fix(terminal): release xterm's DEC 2026 render hold instead of waiting out its 1s timeout (#23920) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(terminal): release xterm's DEC 2026 render hold instead of waiting out its 1s timeout xterm paints nothing while DEC mode 2026 (synchronized output) is open and only force-flushes after 1000ms. Codex wraps every draw in mode 2026, so any byte gap or chunk split that loses the closing \x1b[?2026l freezes the pane for a full second and then repaints in one burst. Orca never emitted \x1b[?2026l anywhere, and three paths could destroy a TUI's: the per-PTY pending cap drops buffered output wholesale (mode 2031 was already salvaged there, 2026 was not), main sliced pending data at a blind 16KB offset that can land inside an open frame or sever the 8-byte marker, and the renderer's backlog warnings replace a queued tail that may hold the close. - salvage the 2026 latch across dropped output, mirroring the existing 2031 salvage, and append the release on both delivery sites - ground 2026 in RESET_AFTER_BYTE_GAP and the replay baseline, and in both backlog warnings, so every drop path is self-healing - make main's 16KB flush split frame-aware instead of a blind byte offset - lift the synchronized-output scanner into shared/ so main and the renderer use one implementation Closing a frame early costs one premature repaint; leaving it open costs a second of blank screen, so the asymmetry favours always closing. Also adds the reproduction this needed: the pre-existing typing bench observes the xterm BUFFER, which the parser fills while rendering is held, so it scored these freezes as fast echoes. * fix(terminal): stop the renderer's queue drain cutting inside an open DEC 2026 frame takeQueuedChunk sliced a queued chunk at a blind byte offset to fit the 16KB coalescing budget, which can strand a frame's closing \x1b[?2026l in the residual until a later drain. Same defect as main's flush split, same fix: reuse the frame-aware split helper. Usually masked because the drain coalesces adjacent chunks and reassembles what main split, but not when the budget boundary falls inside a frame. * fix(relay): keep the SSH path's bounded slice outside an open DEC 2026 frame pty-handler split pending output at a byte offset with a surrogate-pair guard but no synchronized-output awareness, so a frame straddling the 16KB wire slice had its closing \x1b[?2026l stranded in the remainder — the same defect just fixed on the local path, on the path AGENTS.md requires us to consider. Placed before the surrogate guard so that guard keeps the final say, and floored at 2 so frame alignment can never walk a healthy slice into the guard's decrement and then into the chunkChars <= 0 pause-and-retry path. Also drops a dead `splitAt === 0` branch in takeQueuedChunk: both callers pass a positive limit and the helper never returns 0 for one. The two new split tests were each confirmed to fail without their fix. * test(terminal): sweep the DEC 2026 split helper over escape-sequence shapes and every limit Covers OSC 52, DCS, repeated open/close markers and limits 1..len+3, asserting the result never exceeds the limit, never reaches 0, and stays byte-exact. Also pins that a buffer beginning inside an open frame degrades to the blind offset rather than doing something worse, and documents that callers do not thread latch state. * fix(terminal): ground DEC 2026 on the daemon slice, the recovery replays, and the process boundary Four more sites could strand the latch, found by sweeping every path that drops, splits, or replays terminal bytes. - daemon-stream-data-batcher: the 64KB bulk-write slice used a surrogate-only clamp, and its remainder is HELD until 'drain' — "seconds for multi-MB backlogs" per the file's own note. A frame straddling that boundary parked its \x1b[?2026l behind the hold, blanking the pane past xterm's 1s timeout once per frame for as long as the backlog lasted. This is the default daemon-backed pane path, so it is the one users actually hit. The new clampToSafeBulkWriteSplitIndex frame-aligns first and surrogate-clamps last, and lives in daemon-stream-data-split alongside the policy it belongs to. - replay-data-drain and remote-runtime-terminal-binary-snapshots wrote a bare \x1b[2J\x1b[3J\x1b[H, which does not clear mode 2026 — so on the SSH/remote reconnect path, the very event most likely to sever a frame, the whole replay could paint nothing. - ipc-pty-attach: trimIncompleteTerminalControlTail can cut a half-written \x1b[?2026l while its opening marker survives in the replayed prefix. - PROCESS_BOUNDARY_GROUND: the "process that armed these modes is gone" ground omitted 2026, the last unexplained gap in that file. A disable, so it still satisfies the recovery barrier's ownership scan (only ?25h may be an enable). Recovery-path expectations updated where they pin the emitted bytes. Deliberately NOT touched: apply-reattach-payload and ssh-snapshot-prepaint already ground via buildSnapshotReplayPrologue. Still unfixed, deferred with reason: terminal-output-frame-chunks.ts splits the remote wire on accumulated UTF-8 byte width and needs a different shape than the char-index helper; desktop clients reassemble in main's pending buffer, so the exposure is mobile/web only. * fix(terminal): emit the DEC 2026 release before the mode-2031 tail, and stop claiming the drop path writes it Two corrections from adversarial review of the earlier commits. 1. Ordering bug I introduced. getDroppedMode2031RendererData ends with `state.tail`, which extractPrivateModeScanTail deliberately retains as an INCOMPLETE private-mode sequence so the next chunk can resolve it. Appending the 2026 release after it put an ESC behind a dangling CSI, aborting it and silently losing whatever mode spanned the drop boundary. The release now goes first. 2. The drop-path release does not reach xterm in the dominant case, and the comment now says so instead of implying otherwise. live-data-callback's droppedOutput branch discards `data` and salvages only queries (salvageRendererQueriesFromDiscardedRestoreData handles CPR/DA1/OSC colour; \x1b[?2026l is not a query), so for hidden panes and visible panes outside foreground-restore backpressure the synthesized release was dropped. The grounded snapshot replay releases the latch instead. I tried writing it through writePtyOutputToXterm there and reverted: it consumes the pending hidden-output snapshot and broke pty-connection-hidden-snapshot-resize-signals ("re-restores a skipped alt frame"), so the release rides the restore rather than perturbing that state machine. Residual gap, documented: a cap-dropped pane whose restore never arrives. The salvage is still load-bearing on the fall-through path, so it stays. * fix(terminal): release DEC 2026 on the reattach clears, floor the split, and correct the freeze framing Remaining findings from adversarial review. - apply-reattach-payload's three bare-clear branches (:63 daemon snapshot, :229 relay replay, :269 cold restore) had no release anywhere in their sequence: I checked all seven POST_REPLAY_* profiles reachable via chooseReattachReplayReset and none contains \x1b[?2026l. Only the buildMainModelSnapshotReplayWrites branch was grounded, so covering the streamed replay path and not the main reattach path was inconsistent. Verified no production code matches these clear strings — the three test updates are mock equality, and each was confirmed to fail without the source change. - clampToSafeBulkWriteSplitIndex could return 0 (('\u{1F600}aaaa', 1) — alignment returns 1, the surrogate clamp decrements to 0), which would leave a zero-length slice that never shifts the batcher's queue entry and spin its drain loop. Unreachable from today's only caller, but it is exported with an unstated precondition. Floored at 1. - Frame alignment could halve per-PTY flush throughput: main re-queues the remainder with eligibleRound = round + 1, so the shortfall cannot be refilled in the same round, and aligned size is floor(W/F)*F — 50% worst case in the 8-16KB band, which is exactly the full-screen redraw burst that reaches the pending cap. Alignment is now rejected below half the window, preferring throughput and letting the reset profiles release the latch. Framing corrected throughout: bufferRows records a row range and clears nothing, so the pane freezes on its last painted frame — it does not go blank. The real trade is "stale but coherent for <=1s" versus "immediate partial frame", and RESET_AFTER_BYTE_GAP (written alone, with no repaint behind it in the same write) is the one site that can newly flash a partial frame. Said so at the constant instead of implying the release is free. * fix(terminal): rename the shape-flagged symbols the anti-slop audit rejects CI's anti-slop gate rejects "shape" in symbol names as structural rather than domain language: `shapes` -> `outputSamples`, and `writeCodexShapedEchoProbeScript`/`codexShapedEchoProbeScript` -> `writeCodexEchoProbeScript`/`codexEchoProbeScript`. --- .../run-multi-workspace-typing-bench.mjs | 5 + src/main/daemon/daemon-stream-data-batcher.ts | 4 +- src/main/daemon/daemon-stream-data-split.ts | 16 ++ .../ipc/pty-pending-data-drain-contract.ts | 4 + src/main/ipc/pty/delivery/accept.ts | 14 +- src/main/ipc/pty/delivery/flush.ts | 19 +- .../delivery/interactive-typing-burst.test.ts | 92 ++++++++ src/main/ipc/pty/delivery/pending.ts | 25 +- .../pty-handler-output-streaming.test.ts | 35 +++ src/relay/pty-handler.ts | 16 +- .../terminal-pane/ipc-pty-attach.ts | 6 +- ...ty-connection-cold-restore-repaint.test.ts | 5 +- ...-connection-daemon-snapshot-replay.test.ts | 7 +- ...pty-connection-parked-ssh-snapshot.test.ts | 7 +- ...connection-replay-payload-handling.test.ts | 4 +- .../pty-connection/apply-reattach-payload.ts | 15 +- .../pty-connection/foreground-output-scan.ts | 62 +---- .../pty-connection/live-data-callback.ts | 6 + .../pty-connection/replay-data-drain.ts | 7 +- ...e-runtime-pty-snapshot-source-grid.test.ts | 2 +- .../pane-terminal-output-queue-chunks.test.ts | 19 ++ .../pane-terminal-output-queue-chunks.ts | 13 +- .../pane-terminal-output-queue-registry.ts | 7 +- ...emote-runtime-terminal-binary-snapshots.ts | 24 +- ...runtime-terminal-frame-drop-resync.test.ts | 16 +- .../runtime/runtime-terminal-stream.test.ts | 4 +- .../terminal-mode-reset-profiles.test.ts | 2 +- src/shared/terminal-mode-reset-profiles.ts | 23 +- .../terminal-synchronized-output-scan.test.ts | 125 ++++++++++ .../terminal-synchronized-output-scan.ts | 164 +++++++++++++ tests/e2e/git-churn-load.ts | 185 +++++++++++++++ .../sustained-agent-typing-load-scripts.ts | 79 +++++++ .../e2e/synchronized-output-blackout-probe.ts | 151 ++++++++++++ ...nal-multi-workspace-typing-latency.spec.ts | 218 +++++++++++++++++- 34 files changed, 1275 insertions(+), 106 deletions(-) create mode 100644 src/main/ipc/pty/delivery/interactive-typing-burst.test.ts create mode 100644 src/shared/terminal-synchronized-output-scan.test.ts create mode 100644 src/shared/terminal-synchronized-output-scan.ts create mode 100644 tests/e2e/git-churn-load.ts create mode 100644 tests/e2e/synchronized-output-blackout-probe.ts diff --git a/config/scripts/run-multi-workspace-typing-bench.mjs b/config/scripts/run-multi-workspace-typing-bench.mjs index 2aae129c9b2..557453498d3 100644 --- a/config/scripts/run-multi-workspace-typing-bench.mjs +++ b/config/scripts/run-multi-workspace-typing-bench.mjs @@ -23,6 +23,11 @@ const knobByFlag = { '--keys': 'ORCA_TYPING_BENCH_KEYS', '--cadence-ms': 'ORCA_TYPING_BENCH_KEY_CADENCE_MS', '--cpu-workers': 'ORCA_TYPING_BENCH_CPU_WORKERS', + '--git-churn-repos': 'ORCA_TYPING_BENCH_GIT_CHURN_REPOS', + '--git-churn-files': 'ORCA_TYPING_BENCH_GIT_CHURN_FILES', + '--git-churn-concurrency': 'ORCA_TYPING_BENCH_GIT_CHURN_CONCURRENCY', + '--codex-frame-rows': 'ORCA_TYPING_BENCH_CODEX_FRAME_ROWS', + '--codex-split-delay-ms': 'ORCA_TYPING_BENCH_CODEX_SPLIT_DELAY_MS', '--worktrees': 'ORCA_TYPING_BENCH_METADATA_WORKTREES', '--repositories': 'ORCA_TYPING_BENCH_METADATA_REPOSITORIES', '--terminal-tabs': 'ORCA_TYPING_BENCH_METADATA_TERMINAL_TABS', diff --git a/src/main/daemon/daemon-stream-data-batcher.ts b/src/main/daemon/daemon-stream-data-batcher.ts index dfc7c394fdc..a562a60ee9d 100644 --- a/src/main/daemon/daemon-stream-data-batcher.ts +++ b/src/main/daemon/daemon-stream-data-batcher.ts @@ -7,7 +7,7 @@ import { releaseDaemonStreamEntry } from './daemon-stream-entry-accounting' import { DaemonStreamHeldRefill } from './daemon-stream-held-refill' -import { clampToSafeSplitIndex, writeStreamDataEvents } from './daemon-stream-data-split' +import { clampToSafeBulkWriteSplitIndex, writeStreamDataEvents } from './daemon-stream-data-split' import type { PendingStreamDataBatch } from './daemon-stream-keep-tail-drop' import type { DaemonEvent } from './types' import { @@ -213,7 +213,7 @@ export class DaemonStreamDataBatcher { const end = entry.transformed || entry.data.length <= BULK_WRITE_SLICE_CHARS ? entry.data.length - : clampToSafeSplitIndex(entry.data, 0, BULK_WRITE_SLICE_CHARS) + : clampToSafeBulkWriteSplitIndex(entry.data, BULK_WRITE_SLICE_CHARS) const slice = entry.data.slice(0, end) const entrySequenceChars = entry.sequenceChars ?? entry.data.length const sliceSequenceChars = entry.transformed diff --git a/src/main/daemon/daemon-stream-data-split.ts b/src/main/daemon/daemon-stream-data-split.ts index 06ad09e1ded..c31c4437b84 100644 --- a/src/main/daemon/daemon-stream-data-split.ts +++ b/src/main/daemon/daemon-stream-data-split.ts @@ -3,6 +3,7 @@ * chunking (the receiver's parser rejects oversized lines) and the safe-index * clamp shared by the batcher's bulk write slicing and keep-tail dropping. */ +import { resolveSynchronizedOutputSafeSplit } from '../../shared/terminal-synchronized-output-scan' import { encodeNdjson } from './ndjson' export function encodeStreamDataEvent( @@ -38,6 +39,21 @@ function isLowSurrogate(value: number): boolean { return value >= 0xdc00 && value <= 0xdfff } +/** + * Bulk-write split policy: frame-align first so a held remainder cannot strand an + * open DEC 2026 frame's closing \x1b[?2026l (xterm then stops repainting until its + * 1s timeout), then let the surrogate clamp have the final say. + */ +export function clampToSafeBulkWriteSplitIndex(value: string, end: number): number { + // Math.max(1): the surrogate clamp can decrement an aligned index to 0 (e.g. + // ('\u{1F600}aaaa', 1)), and a 0-length slice would never shift the batcher's + // queue entry, spinning its drain loop. + return Math.max( + 1, + clampToSafeSplitIndex(value, 0, resolveSynchronizedOutputSafeSplit(value, end)) + ) +} + export function clampToSafeSplitIndex(value: string, start: number, end: number): number { if (end <= start || end >= value.length) { return end diff --git a/src/main/ipc/pty-pending-data-drain-contract.ts b/src/main/ipc/pty-pending-data-drain-contract.ts index d1d4c5be1a8..ffe0c0e606f 100644 --- a/src/main/ipc/pty-pending-data-drain-contract.ts +++ b/src/main/ipc/pty-pending-data-drain-contract.ts @@ -1,3 +1,4 @@ +import type { SynchronizedOutputLatchState } from '../../shared/terminal-synchronized-output-scan' import type { Mode2031ReplyScanState } from '../../shared/terminal-color-scheme-protocol' export type PendingPtyData = { @@ -9,6 +10,9 @@ export type PendingPtyData = { droppedOutput?: true droppedMode2031Data?: string droppedMode2031ScanState?: Mode2031ReplyScanState + /** Latch state across bytes the renderer never received, so a drop that swallowed + * a closing \x1b[?2026l can still release xterm's render hold. */ + droppedSynchronizedOutputState?: SynchronizedOutputLatchState projectionAdmissionIds?: readonly string[] projectionAdmissionsTransferred?: true } diff --git a/src/main/ipc/pty/delivery/accept.ts b/src/main/ipc/pty/delivery/accept.ts index 979a50d3008..8c68c01f019 100644 --- a/src/main/ipc/pty/delivery/accept.ts +++ b/src/main/ipc/pty/delivery/accept.ts @@ -9,7 +9,11 @@ import { activeRendererPtys } from './visibility-state' import { PTY_BATCH_INTERVAL_MS } from './constants' -import { appendPendingPtyData, getDroppedMode2031RendererData } from './pending' +import { + appendPendingPtyData, + getDroppedMode2031RendererData, + getDroppedSynchronizedOutputRendererData +} from './pending' import { sendModelRestoreNeededMarker, sendPtyDataToRenderer } from './payload' import { shouldSendInteractiveOutputNow } from './interactive' import { requestDeliveryResyncForGatedPty } from './accounting' @@ -98,7 +102,13 @@ export function acceptPtyDataForRenderer( pending.droppedOutput === true && !overflowMarkedBeforeAppend && session.pendingOverflowMarkedPtys.has(payload.id) - const nextData = pending.data + getDroppedMode2031RendererData(pending) + // Why the 2026 release goes BEFORE the 2031 data: that payload ends with a + // deliberately-retained INCOMPLETE private-mode sequence (extractPrivateModeScanTail), + // and an ESC after it would abort the dangling CSI and lose the carried mode. + const nextData = + pending.data + + getDroppedSynchronizedOutputRendererData(pending) + + getDroppedMode2031RendererData(pending) const isInteractiveOutput = shouldSendInteractiveOutputNow( payload.id, nextData, diff --git a/src/main/ipc/pty/delivery/flush.ts b/src/main/ipc/pty/delivery/flush.ts index ffe3c3014e8..ede6a6a4302 100644 --- a/src/main/ipc/pty/delivery/flush.ts +++ b/src/main/ipc/pty/delivery/flush.ts @@ -13,9 +13,11 @@ import { } from './constants' import { getDroppedMode2031RendererData, + getDroppedSynchronizedOutputRendererData, pendingProjectionAdmissionOptions, updatePendingProjectionAdmissions } from './pending' +import { resolveSynchronizedOutputSafeSplit } from '../../../../shared/terminal-synchronized-output-scan' import { makePtyDataPayload, sendModelRestoreNeededMarker, sendPtyDataToRenderer } from './payload' import { warnIfDroppingHiddenBytesForVisiblePty } from './debug-snapshot' import type { PtyIpcSession } from '../session' @@ -144,7 +146,12 @@ export function flushPendingData(session: PtyIpcSession): void { id, { id, - data: pending.data + getDroppedMode2031RendererData(pending), + // 2026 release before the 2031 data: that payload ends with a retained + // partial private-mode sequence an ESC after it would abort. + data: + pending.data + + getDroppedSynchronizedOutputRendererData(pending) + + getDroppedMode2031RendererData(pending), droppedOutput: true }, pending.projectionAdmissionIds @@ -158,8 +165,14 @@ export function flushPendingData(session: PtyIpcSession): void { } const { data } = pending const indivisible = pending.transformed === true - const chunk = indivisible ? data : data.slice(0, PTY_BATCH_FLUSH_CHUNK_CHARS) - const remaining = indivisible ? '' : data.slice(PTY_BATCH_FLUSH_CHUNK_CHARS) + // Why not a blind offset: splitting inside an open DEC 2026 frame strands + // the closing \x1b[?2026l on a later flush, and xterm stops repainting until + // it arrives or its 1000ms timeout fires. + const splitAt = indivisible + ? data.length + : resolveSynchronizedOutputSafeSplit(data, PTY_BATCH_FLUSH_CHUNK_CHARS) + const chunk = indivisible ? data : data.slice(0, splitAt) + const remaining = indivisible ? '' : data.slice(splitAt) let nextPending: PendingPtyData | undefined if (remaining) { nextPending = { data: remaining } diff --git a/src/main/ipc/pty/delivery/interactive-typing-burst.test.ts b/src/main/ipc/pty/delivery/interactive-typing-burst.test.ts new file mode 100644 index 00000000000..db4922a79b5 --- /dev/null +++ b/src/main/ipc/pty/delivery/interactive-typing-burst.test.ts @@ -0,0 +1,92 @@ +import { beforeEach, describe, expect, it } from 'vitest' +import { INTERACTIVE_OUTPUT_BUDGET_CHARS, INTERACTIVE_OUTPUT_WINDOW_MS } from './constants' +import { shouldSendInteractiveOutputNow } from './interactive' +import { interactiveOutputCharsByPty, lastInputAtByPty } from './visibility-state' + +const PTY_ID = 'pty-typing-burst' + +/** + * A Codex composer repaint: one DEC 2026 synchronized frame per keystroke, + * carrying the closing \x1b[?2026l that releases xterm's render latch. Sized + * from a real 204x52 capture of codex-cli 0.158.0, whose per-keystroke frames + * run from a few dozen bytes up to ~1.4KB. + * + * Why the fast path matters for this shape specifically: xterm renders nothing + * while synchronized output is open and force-flushes only after 1000ms, so a + * repaint diverted to the shared batch flush timer — behind every other pane's + * output — leaves the pane visually frozen rather than merely late. + */ +function codexFrame(bytes: number): string { + return `\x1b[?2026h\x1b[1;1H${'x'.repeat(Math.max(0, bytes - 20))}\x1b[?2026l` +} + +/** + * Mirrors the production input path: `noteRendererPtyInput` + * (src/main/ipc/pty/ipc/write-input.ts:152-155) stamps the input time AND zeroes + * the pty's interactive budget on every keystroke. A test that only stamps the + * time would let the budget accumulate across keys and report a fast-path + * divergence that cannot happen in production. + */ +function noteRendererPtyInput(now: number): void { + lastInputAtByPty.set(PTY_ID, now) + interactiveOutputCharsByPty.set(PTY_ID, 0) +} + +function typeKeyAndRepaint(now: number, frame: string): boolean { + noteRendererPtyInput(now) + return shouldSendInteractiveOutputNow(PTY_ID, frame, now + 2) +} + +describe('interactive output fast path during a continuous typing burst', () => { + beforeEach(() => { + lastInputAtByPty.delete(PTY_ID) + interactiveOutputCharsByPty.delete(PTY_ID) + }) + + it('keeps every repaint of a sustained burst on the fast path', () => { + // 90ms apart: inside INTERACTIVE_OUTPUT_WINDOW_MS, i.e. a fast typist who + // never pauses long enough to expire the window. + const cadenceMs = INTERACTIVE_OUTPUT_WINDOW_MS - 10 + const frame = codexFrame(1400) + const batched: number[] = [] + let now = 1_000 + for (let key = 0; key < 200; key++) { + if (!typeKeyAndRepaint(now, frame)) { + batched.push(key) + } + now += cadenceMs + } + + expect( + batched, + `keys diverted off the interactive fast path at 1400-byte frames: ${batched.join(', ')}` + ).toEqual([]) + }) + + it('still cuts off a single keystroke that triggers a flood', () => { + // The budget is per keystroke, so it must still bound one key's blast + // radius: a TUI dumping megabytes after one keypress cannot ride the + // immediate path past the budget and starve main's timers. + const frame = codexFrame(8 * 1024) + let now = 1_000 + noteRendererPtyInput(now) + let admitted = 0 + while (shouldSendInteractiveOutputNow(PTY_ID, frame, now + 2)) { + admitted += 1 + // No new keystroke: the same key's repaints keep arriving. + now += 1 + } + expect(admitted * frame.length).toBeLessThanOrEqual(INTERACTIVE_OUTPUT_BUDGET_CHARS) + expect(admitted).toBeGreaterThan(0) + }) + + it('leaves the fast path when output arrives long after the last keystroke', () => { + const frame = codexFrame(1400) + const now = 1_000 + noteRendererPtyInput(now) + // Unprompted output, well past the input window: throughput work, not echo. + expect( + shouldSendInteractiveOutputNow(PTY_ID, frame, now + INTERACTIVE_OUTPUT_WINDOW_MS + 1) + ).toBe(false) + }) +}) diff --git a/src/main/ipc/pty/delivery/pending.ts b/src/main/ipc/pty/delivery/pending.ts index ff14ccc8c52..b6a7096ca3c 100644 --- a/src/main/ipc/pty/delivery/pending.ts +++ b/src/main/ipc/pty/delivery/pending.ts @@ -4,6 +4,10 @@ import { scanMode2031ReplyDecision, type Mode2031ReplyScanState } from '../../../../shared/terminal-color-scheme-protocol' +import { + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE, + advanceDroppedSynchronizedOutputLatch +} from '../../../../shared/terminal-synchronized-output-scan' import { redactPtyIdForDiagnostics } from '../../../../shared/pty-delivery-diagnostics' import { recordCrashBreadcrumb } from '../../../crash-reporting/crash-breadcrumb-store' import { terminalOutputBacklogCapChars } from '../../../../shared/terminal-scrollback-policy' @@ -52,6 +56,13 @@ export function getDroppedMode2031RendererData(pending: PendingPtyData): string return (pending.droppedMode2031Data ?? '') + pendingSubscribe + state.tail } +/** Releases xterm's DEC 2026 render hold when the dropped span left a frame open. + * Without it the pane freezes on its last painted frame until xterm's 1000ms + * forced flush. */ +export function getDroppedSynchronizedOutputRendererData(pending: PendingPtyData): string { + return pending.droppedSynchronizedOutputState?.active === true ? '\x1b[?2026l' : '' +} + export function pendingProjectionAdmissionOptions(session: PtyIpcSession) { return { isPending: (id: string) => session.sshOutputIntake?.hasUnpublishedProjection(id) ?? false, @@ -117,12 +128,17 @@ export function dropOversizedPendingPtyData( session.sshOutputIntake?.transferProjections(pending.projectionAdmissionIds, 'pending-cap') } const mode2031 = scanDroppedMode2031Data(pending.data, INITIAL_MODE_2031_REPLY_SCAN_STATE) + const synchronizedOutput = advanceDroppedSynchronizedOutputLatch( + pending.data, + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) // Why no trimmed content tail: a mid-stream gap would corrupt the pane; the droppedOutput sentinel repaints from the snapshot and realigns by sequence (only query bytes ride along). return { data: extractDroppedPtyQueryBytes(pending.data).slice(0, DROPPED_QUERY_SALVAGE_MAX_CHARS), droppedOutput: true, droppedMode2031Data: mode2031.data, - droppedMode2031ScanState: mode2031.state + droppedMode2031ScanState: mode2031.state, + droppedSynchronizedOutputState: synchronizedOutput.state } } @@ -147,6 +163,10 @@ export function appendPendingPtyData( data, existing.droppedMode2031ScanState ?? INITIAL_MODE_2031_REPLY_SCAN_STATE ) + const synchronizedOutput = advanceDroppedSynchronizedOutputLatch( + data, + existing.droppedSynchronizedOutputState ?? INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) const remainingQueryCapacity = Math.max( 0, DROPPED_QUERY_SALVAGE_MAX_CHARS - existing.data.length @@ -156,7 +176,8 @@ export function appendPendingPtyData( ...existing, data: existing.data + salvaged, droppedMode2031Data: mode2031.data || existing.droppedMode2031Data, - droppedMode2031ScanState: mode2031.state + droppedMode2031ScanState: mode2031.state, + droppedSynchronizedOutputState: synchronizedOutput.state } } const projectionState = compactPendingProjectionState( diff --git a/src/relay/pty-handler-output-streaming.test.ts b/src/relay/pty-handler-output-streaming.test.ts index 594b30004d3..f9e241e04f0 100644 --- a/src/relay/pty-handler-output-streaming.test.ts +++ b/src/relay/pty-handler-output-streaming.test.ts @@ -713,6 +713,41 @@ describe('PtyHandler', () => { }) }) + it('does not split a bounded slice inside an open DEC 2026 frame', async () => { + let dataCallback: ((data: string) => void) | undefined + mockPtySpawn.mockReturnValue({ + ...mockPtyInstance, + onData: vi.fn((cb: (data: string) => void) => { + dataCallback = cb + }), + onExit: vi.fn() + }) + + await dispatcher.callRequest('pty.spawn', {}) + // A frame that closes just before the 16KB boundary, then a second frame + // that straddles it. Cutting at the raw boundary would strand the second + // frame's \x1b[?2026l, and xterm paints nothing while the latch is open. + const open = '\x1b[?2026h' + const close = '\x1b[?2026l' + const firstFrame = `${open}${'x'.repeat(16 * 1024 - 2 * open.length - close.length)}${close}` + const secondFrame = `${open}${'y'.repeat(64)}${close}` + dataCallback!(`${firstFrame}${secondFrame}`) + + vi.advanceTimersByTime(8) + expect(dispatcher.notify).toHaveBeenCalledTimes(1) + expect(dispatcher.notify).toHaveBeenNthCalledWith(1, 'pty.data', { + id: PTY_1, + data: firstFrame + }) + + vi.advanceTimersByTime(1) + expect(dispatcher.notify).toHaveBeenCalledTimes(2) + expect(dispatcher.notify).toHaveBeenNthCalledWith(2, 'pty.data', { + id: PTY_1, + data: secondFrame + }) + }) + it('writes data to PTY via pty.data notification', async () => { const mockWrite = vi.fn() mockPtySpawn.mockReturnValue({ diff --git a/src/relay/pty-handler.ts b/src/relay/pty-handler.ts index 6090c56a9f3..b6714a956b2 100644 --- a/src/relay/pty-handler.ts +++ b/src/relay/pty-handler.ts @@ -1,6 +1,7 @@ +/* oxlint-disable max-lines */ +import { resolveSynchronizedOutputSafeSplit } from '../shared/terminal-synchronized-output-scan' import { FreebuffStatusProjection } from './freebuff-status-projection' import { applyRelayAgentWorkspaceTrust } from './agent-workspace-trust-spawn' -/* oxlint-disable max-lines */ import type { IPty } from 'node-pty' import { killWithDescendantSweep } from '../main/pty-descendant-termination' import type * as NodePty from 'node-pty' @@ -1348,6 +1349,19 @@ export class PtyHandler { ? desiredChars : (this.dispatcher.maxLegacyPtyDataChars?.(paramsWithoutData, pending.data, desiredChars) ?? desiredChars) + // Why before the surrogate guard: splitting inside an open DEC 2026 frame + // strands the closing \x1b[?2026l in the remainder, and xterm stops repainting + // until it arrives or its 1000ms timeout fires. The surrogate guard keeps the + // final say so a frame boundary can never sever a pair. + if (!pending.transformed && !pending.sourceChunk && chunkChars > 0) { + const frameAligned = resolveSynchronizedOutputSafeSplit(pending.data, chunkChars) + // Why the floor of 2: the surrogate guard below can decrement by one, and + // a chunkChars of 0 takes the pause-and-retry path. Never let frame + // alignment walk a healthy slice into that. + if (frameAligned >= 2) { + chunkChars = frameAligned + } + } if ( chunkChars > 0 && chunkChars < pending.data.length && diff --git a/src/renderer/src/components/terminal-pane/ipc-pty-attach.ts b/src/renderer/src/components/terminal-pane/ipc-pty-attach.ts index 44d3e9639fc..c1640ce293b 100644 --- a/src/renderer/src/components/terminal-pane/ipc-pty-attach.ts +++ b/src/renderer/src/components/terminal-pane/ipc-pty-attach.ts @@ -1,3 +1,4 @@ +import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../shared/terminal-mode-reset-profiles' import { ensurePtyDispatcher, getEagerPtyBufferHandle } from './pty-dispatcher' import { hasTerminalDisplayContent, @@ -53,7 +54,10 @@ function replayEagerPtyBuffer(options: PtyAttachOptions, context: IpcPtyAttachCo const shouldClearBeforeReplay = !options.isAlternateScreen && hasTerminalDisplayContent(replayData) if (shouldClearBeforeReplay && !options.callbacks.onReplayData) { - options.callbacks.onData?.('\x1b[2J\x1b[3J\x1b[H') + // RELEASE_SYNCHRONIZED_OUTPUT: trimIncompleteTerminalControlTail can have cut a + // half-written \x1b[?2026l off the replayed payload while its opening \x1b[?2026h + // survives, and \x1b[2J does not clear the mode. + options.callbacks.onData?.(`${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H`) } context.setSuppressAttentionEvents(true) diff --git a/src/renderer/src/components/terminal-pane/pty-connection-cold-restore-repaint.test.ts b/src/renderer/src/components/terminal-pane/pty-connection-cold-restore-repaint.test.ts index a1c71fdbce4..ca68810aa71 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection-cold-restore-repaint.test.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection-cold-restore-repaint.test.ts @@ -2,7 +2,8 @@ import type * as React from 'react' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { POST_REPLAY_MODE_RESET, - RESET_GRAPHIC_RENDITION + RESET_GRAPHIC_RENDITION, + RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../shared/terminal-mode-reset-profiles' import { Terminal } from '@xterm/headless' import { buildFreshShellViewportBlankingSequence } from './terminal-restored-viewport' @@ -296,7 +297,7 @@ describe('connectPanePty', () => { const recoveredRows = 3 const coldScrollback = '\x1b[1;1HCOLD\x1b[1;15HEND\r\nCOLD_SOURCE_ROW_02' const groundedColdScrollback = `${RESET_GRAPHIC_RENDITION}${coldScrollback}` - const viewportClear = `${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H` + const viewportClear = `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H` transport.connect.mockImplementation(async ({ sessionId }: { sessionId?: string }) => { if (sessionId) { return { diff --git a/src/renderer/src/components/terminal-pane/pty-connection-daemon-snapshot-replay.test.ts b/src/renderer/src/components/terminal-pane/pty-connection-daemon-snapshot-replay.test.ts index 088c2c54905..f03c99fa7dd 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection-daemon-snapshot-replay.test.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection-daemon-snapshot-replay.test.ts @@ -4,7 +4,8 @@ import { POST_REPLAY_DEAD_TUI_RESET, POST_REPLAY_MODE_RESET, POST_REPLAY_REATTACH_RESET, - RESET_GRAPHIC_RENDITION + RESET_GRAPHIC_RENDITION, + RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../shared/terminal-mode-reset-profiles' import { replayEpilogue } from './pty-connection-test-replay-epilogue' import { Terminal } from '@xterm/headless' @@ -185,7 +186,7 @@ describe('connectPanePty', () => { await flushAsyncTicks(20) expect(pane.terminal.write).toHaveBeenCalledWith( - `${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`, + `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`, expect.any(Function) ) expect(pane.terminal.write).toHaveBeenCalledWith( @@ -769,7 +770,7 @@ describe('connectPanePty', () => { await flushAsyncTicks(8) replayCallback.current?.('blocking replay') await flushAsyncTicks(12) - expect(writes).toEqual(['\x1b[2J\x1b[3J\x1b[H']) + expect(writes).toEqual(['\x1b[?2026l\x1b[2J\x1b[3J\x1b[H']) reattachResult.resolve({ id: 'tab-pty', snapshot: 'stale authoritative snapshot' }) await flushAsyncTicks(12) const resizeCallsBeforeReplacement = transport.resize.mock.calls.length diff --git a/src/renderer/src/components/terminal-pane/pty-connection-parked-ssh-snapshot.test.ts b/src/renderer/src/components/terminal-pane/pty-connection-parked-ssh-snapshot.test.ts index 84043771908..4b23e85e389 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection-parked-ssh-snapshot.test.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection-parked-ssh-snapshot.test.ts @@ -2,7 +2,8 @@ import type * as React from 'react' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { POST_REPLAY_REATTACH_RESET, - RESET_GRAPHIC_RENDITION + RESET_GRAPHIC_RENDITION, + RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../shared/terminal-mode-reset-profiles' import { replayEpilogue } from './pty-connection-test-replay-epilogue' import { toAppSshPtyId } from '../../../../shared/ssh-pty-id' @@ -208,7 +209,9 @@ describe('connectPanePty', () => { expect(deps.syncPanePtyLayoutBinding).toHaveBeenCalledWith(1, 'leaf-session') expect(deps.updateTabPtyId).toHaveBeenCalledWith('tab-1', 'leaf-session') // Why: the relay's replay buffer holds full history, so clear xterm before writing to avoid duplicating prior-session content. - expect(writes).toContain(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`) + expect(writes).toContain( + `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H` + ) expect(writes).toContain('restored-ssh-output') expect(writes).toContain(replayEpilogue(POST_REPLAY_REATTACH_RESET)) expect(api.pty.signal).toHaveBeenCalledWith('leaf-session', 'SIGWINCH') diff --git a/src/renderer/src/components/terminal-pane/pty-connection-replay-payload-handling.test.ts b/src/renderer/src/components/terminal-pane/pty-connection-replay-payload-handling.test.ts index dbcc67fa47e..8214432ed82 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection-replay-payload-handling.test.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection-replay-payload-handling.test.ts @@ -619,7 +619,7 @@ describe('connectPanePty', () => { expect(pane.terminal.write).toHaveBeenCalledTimes(1) expect(pane.terminal.write).toHaveBeenNthCalledWith( 1, - '\x1b[2J\x1b[3J\x1b[H', + '\x1b[?2026l\x1b[2J\x1b[3J\x1b[H', expect.any(Function) ) @@ -658,7 +658,7 @@ describe('connectPanePty', () => { callbacksRef.replay?.('authoritative replay') await flushAsyncTicks(8) - expect(writes).toEqual(['\x1b[2J\x1b[3J\x1b[H']) + expect(writes).toEqual(['\x1b[?2026l\x1b[2J\x1b[3J\x1b[H']) const acknowledgeLiveFrame = vi.fn() deliverTerminalDataWithDeferredCredit(acknowledgeLiveFrame, () => { diff --git a/src/renderer/src/components/terminal-pane/pty-connection/apply-reattach-payload.ts b/src/renderer/src/components/terminal-pane/pty-connection/apply-reattach-payload.ts index e8f8e7c1a08..3f33132074d 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection/apply-reattach-payload.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection/apply-reattach-payload.ts @@ -1,7 +1,8 @@ import { waitForTerminalReplayWritesParsed } from '../replay-guard' import { POST_REPLAY_MODE_RESET, - RESET_GRAPHIC_RENDITION + RESET_GRAPHIC_RENDITION, + RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../../shared/terminal-mode-reset-profiles' import { buildMainModelSnapshotReplayWrites, @@ -60,7 +61,9 @@ export function createReattachPayloadHandlers( session.suppressStructuralReplayPtyResize = false } } - session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`) + session.writeReplayData( + `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H` + ) // Why: re-arm the kitty keyboard mirror from the snapshot preamble so Option chords keep their encoding after a window reload. session.applySnapshotKittyKeyboardModes(daemonSnapshotReplay, { kittyKeyboardFlags: ctx.connectResult.snapshotKittyKeyboardFlags, @@ -226,7 +229,9 @@ export function createReattachPayloadHandlers( fullScreenReplay: true }) // Relay replay may overlap xterm's pre-disconnect content; clear first to avoid duplication. - session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`) + session.writeReplayData( + `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H` + ) // Why: raw relay replay may contain the app's own kitty pushes; re-arm with set semantics so redelivery can't grow the stack. // A constructor-fresh mirror (window reload) first demotes to unproven: // the replay window proves nothing about negotiations that predate it. @@ -266,7 +271,9 @@ export function createReattachPayloadHandlers( // The current xterm grid remains a safe lower bound for blanking. } // Why: shrinking first would promote clipped stale viewport rows into scrollback, beyond the reach of a later viewport-only clear. - session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H`) + session.writeReplayData( + `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H` + ) await waitForTerminalReplayWritesParsed(session.pane.terminal) if (!ctx.isCurrentReattachPayload()) { return diff --git a/src/renderer/src/components/terminal-pane/pty-connection/foreground-output-scan.ts b/src/renderer/src/components/terminal-pane/pty-connection/foreground-output-scan.ts index b74f44a6d09..00392c1e1b5 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection/foreground-output-scan.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection/foreground-output-scan.ts @@ -1,4 +1,8 @@ import { isDocumentVisibilityProvenStale } from '../stale-document-visibility' +import { + scanSynchronizedOutput, + type SynchronizedOutputScan +} from '../../../../../shared/terminal-synchronized-output-scan' import { INACTIVE_FOREGROUND_IMMEDIATE_BUDGET_CHARS, consumeForegroundImmediateBudget, @@ -6,9 +10,11 @@ import { } from './foreground-output-budgets' export const TERMINAL_RENDERER_RISK_SCAN_TAIL_CHARS = 256 -export const SYNCHRONIZED_OUTPUT_START_SEQUENCE = '\x1b[?2026h' -export const SYNCHRONIZED_OUTPUT_END_SEQUENCE = '\x1b[?2026l' -export const SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS = SYNCHRONIZED_OUTPUT_START_SEQUENCE.length - 1 +export { + SYNCHRONIZED_OUTPUT_START_SEQUENCE, + SYNCHRONIZED_OUTPUT_END_SEQUENCE, + SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS +} from '../../../../../shared/terminal-synchronized-output-scan' export const CURSOR_SHOW_SEQUENCE = '\x1b[?25h' export const CURSOR_HIDE_SEQUENCE = '\x1b[?25l' export const TERMINAL_FOCUS_IN_SEQUENCE = '\x1b[I' @@ -38,59 +44,15 @@ export function shouldWritePtyOutputForeground(isPaneVisible: boolean): boolean return isDocumentVisibilityProvenStale() } -export type SynchronizedForegroundScan = { - started: boolean - ended: boolean - active: boolean - markerTail: string -} +export type SynchronizedForegroundScan = SynchronizedOutputScan -// Why the carried tail: ConPTY can split \x1b[?2026l across chunks; scanning the raw -// chunk alone left the foreground DEC 2026 latch stuck open so every later chunk was -// held instead of coalesced, freezing the visible pane (#8754). Mirrors the hidden path. +/** Renderer-facing name for the shared latch scan; the logic is protocol, not view. */ export function scanSynchronizedForegroundOutput( data: string, markerTail: string, wasActive: boolean ): SynchronizedForegroundScan { - const scanData = markerTail ? `${markerTail}${data}` : data - const currentChunkStartIndex = scanData.length - data.length - let active = wasActive - let started = false - let ended = false - let startIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_START_SEQUENCE) - let endIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_END_SEQUENCE) - - // Each marker search advances independently, so a missing counterpart is scanned only once. - while (startIndex !== -1 || endIndex !== -1) { - if (endIndex !== -1 && (startIndex === -1 || endIndex < startIndex)) { - active = false - if (endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length > currentChunkStartIndex) { - ended = true - } - endIndex = scanData.indexOf( - SYNCHRONIZED_OUTPUT_END_SEQUENCE, - endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length - ) - continue - } - active = true - if (startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length > currentChunkStartIndex) { - started = true - } - startIndex = scanData.indexOf( - SYNCHRONIZED_OUTPUT_START_SEQUENCE, - startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length - ) - } - - return { - started, - ended, - active, - // Why length-1: a full marker can never hide in the tail, so no marker is counted twice. - markerTail: scanData.slice(-SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS) - } + return scanSynchronizedOutput(data, markerTail, wasActive) } export function containsCursorPositionSequence(data: string): boolean { diff --git a/src/renderer/src/components/terminal-pane/pty-connection/live-data-callback.ts b/src/renderer/src/components/terminal-pane/pty-connection/live-data-callback.ts index 08bb1670043..a523576486f 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection/live-data-callback.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection/live-data-callback.ts @@ -46,6 +46,12 @@ export function bindLiveDataCallback(session: ConnectPanePtySession): void { } else { // Why: main dropped buffered output at the pending cap, so the stream has a gap; repaint from the main-owned snapshot instead of writing on. session.markHiddenOutputRestoreNeeded() + // Why the synthesized \x1b[?2026l is not written here: this branch discards + // `data` deliberately, and the grounded snapshot replay + // (REPLAY_BASELINE_TERMINAL_RESET) releases the latch instead. Writing it + // through writePtyOutputToXterm perturbs the hidden-output-restore state + // machine (it consumes the pending snapshot), so the release rides the + // restore. Residual gap: a pane whose restore never arrives. if (data) { // The sentinel can carry query bytes carved from the bulk drop (extractDroppedPtyQueryBytes in main); replies must still flow. session.salvageRendererQueriesFromDiscardedRestoreData(data) diff --git a/src/renderer/src/components/terminal-pane/pty-connection/replay-data-drain.ts b/src/renderer/src/components/terminal-pane/pty-connection/replay-data-drain.ts index 087326d0f40..693ee4be67a 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection/replay-data-drain.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection/replay-data-drain.ts @@ -1,3 +1,4 @@ +import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../../shared/terminal-mode-reset-profiles' import { waitForTerminalOutputParsed } from '@/lib/pane-manager/pane-terminal-output-scheduler' import { safeFit, safeFitAndThen } from '@/lib/pane-manager/pane-tree-ops' import { getFitOverrideForPty } from '@/lib/pane-manager/mobile-fit-overrides' @@ -118,7 +119,11 @@ export function bindReplayDataDrain(session: ConnectPanePtySession): void { // dropping the scrollback first spares a reflow of history the very next // sequence discards (see use-terminal-container-fit-sync.ts on its cost). if (clearBeforeReplay) { - await session.writeReplayDataAsync('\x1b[2J\x1b[3J\x1b[H') + // RELEASE_SYNCHRONIZED_OUTPUT: a reconnect is exactly the event that severs a + // frame mid-flight, so this xterm may hold an open 2026 latch — and \x1b[2J does + // not clear it, so the pane would stay frozen on its last painted frame and the + // whole replay would go unseen until xterm's 1s timeout. + await session.writeReplayDataAsync(`${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H`) if (!isCurrentPayload()) { continue } diff --git a/src/renderer/src/components/terminal-pane/remote-runtime-pty-snapshot-source-grid.test.ts b/src/renderer/src/components/terminal-pane/remote-runtime-pty-snapshot-source-grid.test.ts index 3f49e6e0b7c..2cd40bd4a33 100644 --- a/src/renderer/src/components/terminal-pane/remote-runtime-pty-snapshot-source-grid.test.ts +++ b/src/renderer/src/components/terminal-pane/remote-runtime-pty-snapshot-source-grid.test.ts @@ -124,7 +124,7 @@ describe('remote transport snapshot source-grid threading', () => { deliverSnapshot({ cols: 154, rows: 68, seq: 9, source: 'headless' }, 'recovered') await expect.poll(() => onReplayData.mock.calls.length, { timeout: 5000 }).toBe(2) expect(onReplayData).toHaveBeenLastCalledWith( - '\x1b[2J\x1b[3J\x1b[Hrecovered', + '\x1b[?2026l\x1b[2J\x1b[3J\x1b[Hrecovered', expect.objectContaining({ snapshotCols: 154, snapshotRows: 68 }) ) diff --git a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.test.ts b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.test.ts index 02bf5798802..1bcc963cb90 100644 --- a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.test.ts +++ b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.test.ts @@ -40,4 +40,23 @@ describe('pane terminal output queue chunks', () => { expect(second?.data).toBe(allData.slice(-2)) expect(entry.queuedChars).toBe(0) }) + + it('does not cut a queued chunk inside an open DEC 2026 frame', () => { + const entry = createEntry() + const open = '\x1b[?2026h' + const close = '\x1b[?2026l' + const firstFrame = `${open}aaaa${close}` + const data = `${firstFrame}${open}bbbbbbbbbbbb${close}` + enqueueChunk(entry, data, { foreground: true }) + + // A limit landing inside the second frame must stop after the first one: + // xterm paints nothing while the latch is open, so stranding the close in + // the residual freezes the pane until a later drain or its 1000ms timeout. + const taken = takeQueuedChunk(entry, firstFrame.length + 6) + expect(taken?.data).toBe(firstFrame) + // The residual keeps the rest, byte-exact, with accounting still balanced. + const rest = takeQueuedChunk(entry, data.length) + expect(`${taken?.data ?? ''}${rest?.data ?? ''}`).toBe(data) + expect(entry.queuedChars).toBe(0) + }) }) diff --git a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.ts b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.ts index 82863fee717..9cfdfcd960e 100644 --- a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.ts +++ b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-chunks.ts @@ -1,4 +1,5 @@ import { flattenRetainedSlice } from '@/lib/flatten-retained-slice' +import { resolveSynchronizedOutputSafeSplit } from '../../../../shared/terminal-synchronized-output-scan' import type { QueueEntry, QueuedWrite, @@ -90,7 +91,13 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite | continue } - const prefix = chunk.data.slice(0, remaining) + // Why not a blind offset: cutting inside an open DEC 2026 frame strands the + // closing \x1b[?2026l in the residual, and xterm stops repainting — the pane holds + // its last frame — until a later drain delivers it or its 1000ms timeout fires. + // Always >= 1 here: `remaining > 0` gates the loop and the helper never + // returns 0 for a positive limit, so the loop cannot stall. + const splitAt = resolveSynchronizedOutputSafeSplit(chunk.data, remaining) + const prefix = chunk.data.slice(0, splitAt) if (dataParts) { dataParts.push(prefix) } else if (dataLength === 0) { @@ -100,7 +107,7 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite | data = '' } dataLength += prefix.length - const residual = chunk.data.slice(remaining) + const residual = chunk.data.slice(splitAt) // Geometric flattening bounds retained parents while keeping total copy work linear. const flatten = residual.length * 2 <= chunk.retainedChars entry.chunks[entry.chunkIndex] = { @@ -108,7 +115,7 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite | data: flatten ? flattenRetainedSlice(residual) : residual, retainedChars: flatten ? residual.length : chunk.retainedChars } - entry.queuedChars -= remaining + entry.queuedChars -= prefix.length remaining = 0 } diff --git a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-registry.ts b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-registry.ts index bb18a47a2a2..73b23bdf11f 100644 --- a/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-registry.ts +++ b/src/renderer/src/lib/pane-manager/pane-terminal-output-queue-registry.ts @@ -108,11 +108,14 @@ export function getTerminalOutputMaxQueueChars(): number { return maxQueueChars } // Why: leading CAN aborts any partial escape sequence before the style reset so the backlog warning renders cleanly. +// Why ?2026l too: the discarded tail may have held the TUI's closing \x1b[?2026l, and +// xterm stops repainting until the latch closes or its 1000ms timeout fires — the +// pane freezes on its last frame, so the warning itself would go unseen for a second. export const BACKGROUND_BACKLOG_WARNING = - '\x18\x1b[0m\r\n[Orca skipped hidden terminal output because the backlog grew too large.]\r\n' + '\x18\x1b[?2026l\x1b[0m\r\n[Orca skipped hidden terminal output because the backlog grew too large.]\r\n' // Why a separate foreground message: a visible pane hitting the cap means the drain couldn't keep up with a flood (starved renderer), not merely output produced while hidden. export const FOREGROUND_BACKLOG_WARNING = - '\x18\x1b[0m\r\n[Orca skipped a burst of terminal output because the backlog grew too large.]\r\n' + '\x18\x1b[?2026l\x1b[0m\r\n[Orca skipped a burst of terminal output because the backlog grew too large.]\r\n' export const ALWAYS_REFRESH_FOREGROUND_SYNCHRONOUSLY = (): boolean => true export const queuedByTerminal = new Map() diff --git a/src/renderer/src/runtime/remote-runtime-terminal-binary-snapshots.ts b/src/renderer/src/runtime/remote-runtime-terminal-binary-snapshots.ts index 61cb242a856..6928267a12a 100644 --- a/src/renderer/src/runtime/remote-runtime-terminal-binary-snapshots.ts +++ b/src/renderer/src/runtime/remote-runtime-terminal-binary-snapshots.ts @@ -1,3 +1,4 @@ +import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../shared/terminal-mode-reset-profiles' import { TerminalStreamOpcode, decodeTerminalStreamText, @@ -105,15 +106,20 @@ export abstract class RemoteRuntimeTerminalBinarySnapshots extends RemoteRuntime // mid-session; clear the screen and scrollback before applying it. // An empty snapshot is still applied so stale dropped output does // not linger on a terminal the model says is blank. - stream.callbacks.onSnapshot(`\x1b[2J\x1b[3J\x1b[H${data ?? ''}`, { - pendingEscapeTailAnsi: info?.pendingEscapeTailAnsi, - seq: info?.seq, - kittyKeyboardFlags: info?.kittyKeyboardFlags, - alternateScreen: info?.alternateScreen, - terminalOwner: info?.terminalOwner, - cols: info?.cols, - rows: info?.rows - }) + // RELEASE_SYNCHRONIZED_OUTPUT: \x1b[2J does not clear mode 2026, so a pane + // holding an open latch would not paint this recovery snapshot at all. + stream.callbacks.onSnapshot( + `${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H${data ?? ''}`, + { + pendingEscapeTailAnsi: info?.pendingEscapeTailAnsi, + seq: info?.seq, + kittyKeyboardFlags: info?.kittyKeyboardFlags, + alternateScreen: info?.alternateScreen, + terminalOwner: info?.terminalOwner, + cols: info?.cols, + rows: info?.rows + } + ) } } else if (matchesPendingRequest) { pendingRequest.resolve({ diff --git a/src/renderer/src/runtime/remote-runtime-terminal-frame-drop-resync.test.ts b/src/renderer/src/runtime/remote-runtime-terminal-frame-drop-resync.test.ts index 03a6a944618..0f591a50412 100644 --- a/src/renderer/src/runtime/remote-runtime-terminal-frame-drop-resync.test.ts +++ b/src/renderer/src/runtime/remote-runtime-terminal-frame-drop-resync.test.ts @@ -243,7 +243,7 @@ describe('remote terminal frame-drop resync', () => { expect(data).toEqual(['aaa']) expect(server.droppedFrames).toBe(1) // Instead, a fresh authoritative snapshot recovers the terminal. - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) server.replaySnapshotCoveredOutput('ccc') server.output('ddd') @@ -270,7 +270,7 @@ describe('remote terminal frame-drop resync', () => { expect(server.snapshotRequests).toEqual([undefined, undefined]) server.output('eee') - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) expect(data).toEqual(['aaa', 'eee']) } finally { vi.useRealTimers() @@ -298,7 +298,7 @@ describe('remote terminal frame-drop resync', () => { server.output('eee') expect(server.snapshotRequests).toEqual([undefined, undefined]) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) expect(data).toEqual(['aaa', 'eee']) }) @@ -321,7 +321,7 @@ describe('remote terminal frame-drop resync', () => { await expect(manualSnapshot).rejects.toThrow('stream failed') expect(server.snapshotRequests).toEqual([expect.any(Number), undefined]) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) server.output('ddd') expect(data).toEqual(['aaa', 'ddd']) @@ -342,7 +342,7 @@ describe('remote terminal frame-drop resync', () => { server.output('eee') expect(server.snapshotRequests).toEqual([undefined, undefined]) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) expect(data).toEqual(['aaa', 'eee']) } finally { vi.useRealTimers() @@ -452,7 +452,7 @@ describe('remote terminal frame-drop resync', () => { await Promise.resolve() expect(data).toEqual([]) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) }) it('uses UTF-16 sequence units when detecting gaps in multibyte output', async () => { @@ -466,7 +466,7 @@ describe('remote terminal frame-drop resync', () => { await Promise.resolve() expect(data).toEqual(['é']) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) }) it('defers recovery until an in-flight manual snapshot finishes', async () => { @@ -489,7 +489,7 @@ describe('remote terminal frame-drop resync', () => { await expect(manualSnapshot).resolves.toMatchObject({ data: 'MANUAL' }) expect(server.snapshotRequests).toHaveLength(2) - expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED']) + expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED']) server.output('ddd') expect(data).toEqual(['aaa', 'ddd']) diff --git a/src/renderer/src/runtime/runtime-terminal-stream.test.ts b/src/renderer/src/runtime/runtime-terminal-stream.test.ts index 2fcc8b2e293..994cdbe5824 100644 --- a/src/renderer/src/runtime/runtime-terminal-stream.test.ts +++ b/src/renderer/src/runtime/runtime-terminal-stream.test.ts @@ -592,7 +592,7 @@ describe('remote runtime terminal multiplex ACK gate', () => { // Why: an unsolicited recovery snapshot replaces terminal state, so it // clears screen and scrollback first and must not replay the subscribe // lifecycle. - expect(onSnapshot).toHaveBeenCalledWith(`\x1b[2J\x1b[3J\x1b[H${'recovered state'}`, { + expect(onSnapshot).toHaveBeenCalledWith(`\x1b[?2026l\x1b[2J\x1b[3J\x1b[H${'recovered state'}`, { pendingEscapeTailAnsi: undefined, cols: 120, rows: 40 @@ -611,7 +611,7 @@ describe('remote runtime terminal multiplex ACK gate', () => { }, '' ) - expect(onSnapshot).toHaveBeenCalledWith('\x1b[2J\x1b[3J\x1b[H', { + expect(onSnapshot).toHaveBeenCalledWith('\x1b[?2026l\x1b[2J\x1b[3J\x1b[H', { pendingEscapeTailAnsi: undefined, cols: 120, rows: 40 diff --git a/src/shared/terminal-mode-reset-profiles.test.ts b/src/shared/terminal-mode-reset-profiles.test.ts index 203d836a3f8..521ceea7237 100644 --- a/src/shared/terminal-mode-reset-profiles.test.ts +++ b/src/shared/terminal-mode-reset-profiles.test.ts @@ -53,7 +53,7 @@ describe('terminal mode reset profiles', () => { // Why: the one reset for a process boundary (cold-restore seed, proven crash). it('pins the process boundary ground', () => { expect(PROCESS_BOUNDARY_GROUND).toBe( - '\x1b[<99u\x1b[=0u\x1b7\x1b[?1049l\x1b[?9l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1006l\x1b[?1016l\x1b[?1005l\x1b[?1015l\x1b[?1004l\x1b[?2004l\x1b[?1l\x1b[?66l\x1b[?25h\x1b[0 q\x1b[<99u\x1b[=0u\x1b[0m\x1b7' + '\x1b[?2026l\x1b[<99u\x1b[=0u\x1b7\x1b[?1049l\x1b[?9l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1006l\x1b[?1016l\x1b[?1005l\x1b[?1015l\x1b[?1004l\x1b[?2004l\x1b[?1l\x1b[?66l\x1b[?25h\x1b[0 q\x1b[<99u\x1b[=0u\x1b[0m\x1b7' ) }) diff --git a/src/shared/terminal-mode-reset-profiles.ts b/src/shared/terminal-mode-reset-profiles.ts index a219c2d70d0..d8c80f32494 100644 --- a/src/shared/terminal-mode-reset-profiles.ts +++ b/src/shared/terminal-mode-reset-profiles.ts @@ -58,6 +58,21 @@ export const POST_REPLAY_LIVE_AGENT_SNAPSHOT_RESET = RESET_TERMINAL_CURSOR_STYLE // writes the clipboard. export const ABORT_TRUNCATED_CONTROL_STRING = '\x18' +// Trade-off, stated because it is not free: the pane was FROZEN on its last +// coherent frame, not blank (bufferRows records a row range and clears nothing). +// Releasing the latch where no repaint follows in the same write — RESET_AFTER_BYTE_GAP +// is written alone — can flash a partial frame in place of that coherent one. A byte +// gap already means the stream is damaged and a restore follows, so a stale frame that +// outlives the damage is the worse option. +// Why this is grounded everywhere a byte gap or a repaint happens: xterm renders +// NOTHING while DEC 2026 is open and only force-flushes after 1000ms, so a gap +// that swallowed a TUI's closing \x1b[?2026l leaves the pane blank for a full +// second per frame — and Orca is otherwise incapable of closing a latch it +// opened. Unlike the modes deliberately left ungrounded below, a snapshot never +// re-asserts 2026, and closing a frame early costs one premature repaint against +// a second of frozen, increasingly stale output. +export const RELEASE_SYNCHRONIZED_OUTPUT = '\x1b[?2026l' + // Why the DECSC first: xterm's `?1049l` runs restoreCursor() even on the normal // buffer, so saving in place keeps the cursor put there; on the alt buffer the // save lands in the alt register and `?1049l` restores the shell's position. @@ -78,7 +93,9 @@ const SHOW_CURSOR = '\x1b[?25h' */ export function buildProcessBoundaryGround(opts: { keepFocusReporting: boolean }): string { const focus = opts.keepFocusReporting ? '' : RESET_FOCUS_REPORTING - return `${RESET_KITTY_KEYBOARD_PROTOCOL}${LEAVE_ALTERNATE_SCREEN_KEEPING_NORMAL_CURSOR}${RESET_MOUSE_REPORTING}${RESET_LEGACY_MOUSE_ENCODINGS}${focus}${RESET_BRACKETED_PASTE}${RESET_APPLICATION_CURSOR_AND_KEYPAD}${SHOW_CURSOR}${RESET_TERMINAL_CURSOR_STYLE}${RESET_KITTY_KEYBOARD_PROTOCOL}${RESET_GRAPHIC_RENDITION}${SAVE_GROUNDED_CURSOR}` + // RELEASE_SYNCHRONIZED_OUTPUT first: the process that opened a 2026 frame is + // gone, so nothing will ever close it, and xterm stops repainting until it does. + return `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_KITTY_KEYBOARD_PROTOCOL}${LEAVE_ALTERNATE_SCREEN_KEEPING_NORMAL_CURSOR}${RESET_MOUSE_REPORTING}${RESET_LEGACY_MOUSE_ENCODINGS}${focus}${RESET_BRACKETED_PASTE}${RESET_APPLICATION_CURSOR_AND_KEYPAD}${SHOW_CURSOR}${RESET_TERMINAL_CURSOR_STYLE}${RESET_KITTY_KEYBOARD_PROTOCOL}${RESET_GRAPHIC_RENDITION}${SAVE_GROUNDED_CURSOR}` } export const PROCESS_BOUNDARY_GROUND = buildProcessBoundaryGround({ keepFocusReporting: false }) @@ -87,7 +104,7 @@ export const PROCESS_BOUNDARY_GROUND = buildProcessBoundaryGround({ keepFocusRep // queued chunks instead of repainting. Parser + pen only — a live TUI keeps // writing here and owns its charset and margins. Not DECSTR: xterm's soft reset // wipes the kitty flags agents negotiate only at startup. -export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RESET_GRAPHIC_RENDITION}` +export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}` // The baseline a serialized snapshot assumes it lands on: SerializeAddon diffs // cells against DEFAULT attributes and emits no charset at all. @@ -100,7 +117,7 @@ export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RESET_GR // resetting is unilateral. `enacs=\E(B\E)0` (screen/tmux/vt100 terminfo) // designates G1 once at init and then uses bare SO/SI, so grounding G1 would // render a live app's box drawing as letters. -const REPLAY_BASELINE_TERMINAL_RESET = `${RESET_GRAPHIC_RENDITION}\x0f\x1b(B\x1b[?6l\x1b[?7h\x1b[?45l\x1b[4l` +const REPLAY_BASELINE_TERMINAL_RESET = `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x0f\x1b(B\x1b[?6l\x1b[?7h\x1b[?45l\x1b[4l` // Buffer-scoped: margins live on the xterm buffer, and `?1049` neither carries // them across nor clears them unless it actually swaps. diff --git a/src/shared/terminal-synchronized-output-scan.test.ts b/src/shared/terminal-synchronized-output-scan.test.ts new file mode 100644 index 00000000000..cc4c5dd2fc7 --- /dev/null +++ b/src/shared/terminal-synchronized-output-scan.test.ts @@ -0,0 +1,125 @@ +import { describe, expect, it } from 'vitest' +import { + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE, + SYNCHRONIZED_OUTPUT_END_SEQUENCE, + SYNCHRONIZED_OUTPUT_START_SEQUENCE, + advanceDroppedSynchronizedOutputLatch, + resolveSynchronizedOutputSafeSplit, + scanSynchronizedOutput +} from './terminal-synchronized-output-scan' + +const OPEN = SYNCHRONIZED_OUTPUT_START_SEQUENCE +const CLOSE = SYNCHRONIZED_OUTPUT_END_SEQUENCE + +describe('advanceDroppedSynchronizedOutputLatch', () => { + it('releases the latch when the dropped span ended mid-frame', () => { + // The close was inside the bytes the renderer will never receive, so xterm + // would paint nothing until its 1000ms forced flush. + const result = advanceDroppedSynchronizedOutputLatch( + `${OPEN}rows`, + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) + expect(result.state.active).toBe(true) + expect(result.data).toBe(CLOSE) + }) + + it('emits nothing when the dropped span closed its own frame', () => { + const result = advanceDroppedSynchronizedOutputLatch( + `${OPEN}rows${CLOSE}`, + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) + expect(result.state.active).toBe(false) + expect(result.data).toBe('') + }) + + it('carries an open latch across successive dropped chunks', () => { + const first = advanceDroppedSynchronizedOutputLatch( + `${OPEN}a`, + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) + const second = advanceDroppedSynchronizedOutputLatch('b', first.state) + expect(second.state.active).toBe(true) + expect(second.data).toBe(CLOSE) + const third = advanceDroppedSynchronizedOutputLatch(CLOSE, second.state) + expect(third.state.active).toBe(false) + expect(third.data).toBe('') + }) + + it('stitches a close marker split across dropped chunks', () => { + const head = CLOSE.slice(0, 4) + const tail = CLOSE.slice(4) + const first = advanceDroppedSynchronizedOutputLatch( + `${OPEN}rows${head}`, + INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE + ) + expect(first.state.active).toBe(true) + const second = advanceDroppedSynchronizedOutputLatch(tail, first.state) + expect(second.state.active).toBe(false) + expect(second.data).toBe('') + }) +}) + +describe('resolveSynchronizedOutputSafeSplit', () => { + it('returns the whole length when it already fits', () => { + expect(resolveSynchronizedOutputSafeSplit('abc', 16)).toBe(3) + }) + + it('splits after a completed frame rather than inside the next one', () => { + const data = `${OPEN}aaaa${CLOSE}${OPEN}bbbbbbbbbb${CLOSE}` + const limit = data.indexOf('bbb') + const splitAt = resolveSynchronizedOutputSafeSplit(data, limit) + // Everything delivered must leave the latch closed. + expect(scanSynchronizedOutput(data.slice(0, splitAt), '', false).active).toBe(false) + expect(splitAt).toBe(`${OPEN}aaaa${CLOSE}`.length) + }) + + it('never severs the close marker itself', () => { + const data = `${OPEN}aaaa${CLOSE}tail` + // Limit lands in the middle of the 8-byte close sequence. + const limit = `${OPEN}aaaa`.length + 4 + const splitAt = resolveSynchronizedOutputSafeSplit(data, limit) + expect(splitAt).toBeLessThanOrEqual(limit) + expect(data.slice(0, splitAt).endsWith('\x1b')).toBe(false) + // The remainder must still contain a complete, parseable close. + expect(data.slice(splitAt)).toContain(CLOSE) + }) + + it('falls back to the limit when one frame is longer than the window', () => { + const data = `${OPEN}${'x'.repeat(100)}${CLOSE}` + expect(resolveSynchronizedOutputSafeSplit(data, 20)).toBe(20) + }) + + it('degrades to the plain limit when the buffer starts inside a frame', () => { + // Callers do not thread prior latch state, so a remainder that begins inside + // an already-open frame is scanned as if closed. It must never be WORSE than + // the blind offset it replaced: same boundary, byte-exact. + const data = `${'z'.repeat(40)}${CLOSE}${'q'.repeat(40)}` + const limit = 20 + const splitAt = resolveSynchronizedOutputSafeSplit(data, limit, '', true) + const naive = resolveSynchronizedOutputSafeSplit(data, limit) + // With the real prior state it can only do better or the same. + expect(splitAt).toBeLessThanOrEqual(limit) + expect(naive).toBeLessThanOrEqual(limit) + expect(data.slice(0, naive) + data.slice(naive)).toBe(data) + }) + + it('never returns past the limit or breaks byte-exactness across many shapes', () => { + const outputSamples = [ + `${OPEN}${'a'.repeat(50)}${CLOSE}`, + `${'a'.repeat(50)}${CLOSE}${'b'.repeat(50)}`, + `${OPEN}${OPEN}${'a'.repeat(30)}${CLOSE}${CLOSE}`, + `${'a'.repeat(30)}\x1b]52;c;SGVsbG8=\x07${'b'.repeat(30)}`, + `${'a'.repeat(30)}\x1bP0;1|payload\x1b\\${'b'.repeat(30)}`, + CLOSE.repeat(10), + `${OPEN.repeat(10)}tail` + ] + for (const data of outputSamples) { + for (let limit = 1; limit <= data.length + 3; limit++) { + const splitAt = resolveSynchronizedOutputSafeSplit(data, limit) + expect(splitAt).toBeGreaterThan(0) + expect(splitAt).toBeLessThanOrEqual(Math.min(limit, data.length)) + expect(data.slice(0, splitAt) + data.slice(splitAt)).toBe(data) + } + } + }) +}) diff --git a/src/shared/terminal-synchronized-output-scan.ts b/src/shared/terminal-synchronized-output-scan.ts new file mode 100644 index 00000000000..b7bed2a89e4 --- /dev/null +++ b/src/shared/terminal-synchronized-output-scan.ts @@ -0,0 +1,164 @@ +/** + * DEC mode 2026 (synchronized output) latch tracking. + * + * Why this module is shared: like terminal-mode-reset-profiles, the latch is a + * terminal-protocol contract rather than a renderer concern. xterm stops + * repainting while the latch is open and force-flushes only after a 1000ms + * timeout, so whichever side of the PTY relay last touched a pane's bytes has + * to know whether it left a frame open — main's drop paths included, not just + * the renderer's foreground coalescer. + */ +export const SYNCHRONIZED_OUTPUT_START_SEQUENCE = '\x1b[?2026h' +export const SYNCHRONIZED_OUTPUT_END_SEQUENCE = '\x1b[?2026l' +export const SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS = SYNCHRONIZED_OUTPUT_START_SEQUENCE.length - 1 + +export type SynchronizedOutputScan = { + /** A start marker landed inside THIS chunk. */ + started: boolean + /** An end marker landed inside THIS chunk. */ + ended: boolean + /** Latch state after the chunk: true means a frame is still open. */ + active: boolean + markerTail: string +} + +// Why the carried tail: a PTY relay can split \x1b[?2026l across chunks; scanning the raw +// chunk alone left the foreground DEC 2026 latch stuck open so every later chunk was +// held instead of coalesced, freezing the visible pane (#8754). +export function scanSynchronizedOutput( + data: string, + markerTail: string, + wasActive: boolean +): SynchronizedOutputScan { + const scanData = markerTail ? `${markerTail}${data}` : data + const currentChunkStartIndex = scanData.length - data.length + let active = wasActive + let started = false + let ended = false + let startIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_START_SEQUENCE) + let endIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_END_SEQUENCE) + + // Each marker search advances independently, so a missing counterpart is scanned only once. + while (startIndex !== -1 || endIndex !== -1) { + if (endIndex !== -1 && (startIndex === -1 || endIndex < startIndex)) { + active = false + if (endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length > currentChunkStartIndex) { + ended = true + } + endIndex = scanData.indexOf( + SYNCHRONIZED_OUTPUT_END_SEQUENCE, + endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length + ) + continue + } + active = true + if (startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length > currentChunkStartIndex) { + started = true + } + startIndex = scanData.indexOf( + SYNCHRONIZED_OUTPUT_START_SEQUENCE, + startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length + ) + } + + return { + started, + ended, + active, + // Why length-1: a full marker can never hide in the tail, so no marker is counted twice. + markerTail: scanData.slice(-SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS) + } +} + +export type SynchronizedOutputLatchState = { + markerTail: string + active: boolean +} + +export const INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE: SynchronizedOutputLatchState = { + markerTail: '', + active: false +} + +/** + * Advances the latch across bytes the renderer will never receive. + * + * Returns the sequence that must ride along with whatever IS delivered so the + * pane is not left mid-frame: a drop that swallowed the closing marker would + * otherwise leave xterm painting nothing until its 1000ms forced flush, once + * per frame, for as long as the condition lasts. Closing a frame early only + * costs one premature repaint; leaving it open costs a full second of blank + * screen, so the asymmetry favours always closing. + */ +export function advanceDroppedSynchronizedOutputLatch( + data: string, + previous: SynchronizedOutputLatchState +): { data: string; state: SynchronizedOutputLatchState } { + const scan = scanSynchronizedOutput(data, previous.markerTail, previous.active) + return { + data: scan.active ? SYNCHRONIZED_OUTPUT_END_SEQUENCE : '', + state: { markerTail: scan.markerTail, active: scan.active } + } +} + +/** + * Chooses a split point that does not leave the delivered half inside an open + * DEC 2026 frame. + * + * A blind byte-offset split puts `\x1b[?2026h` in one chunk and its + * `\x1b[?2026l` in the next, so xterm stops repainting until the remainder is + * delivered on a later flush — behind every other pane's output — or until its + * 1000ms forced flush. It can also sever the 8-byte marker itself. + * + * Returns the largest length <= `limit` that ends outside an open frame, or + * `limit` when no such point exists (a frame genuinely longer than the window; + * the latch release still rides along via the reset profiles). + * + * KNOWN LIMITATION: no caller threads `markerTail`/`wasActive`, so a buffer that + * begins INSIDE an already-open frame is scanned as if closed. That degrades to + * the blind offset this replaced — never worse, and byte-exact either way — but + * it means cross-chunk alignment is best-effort. Threading per-PTY latch state + * through the split sites would close it. + */ +export function resolveSynchronizedOutputSafeSplit( + data: string, + limit: number, + markerTail = '', + wasActive = false +): number { + if (data.length <= limit) { + return data.length + } + // Step 1: never hand over a severed marker. If an ESC near the boundary cannot + // have completed by `limit`, cut before it instead. + let candidate = limit + const guardStart = Math.max(0, limit - SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS) + const escapeIndex = data.lastIndexOf('\x1b', limit - 1) + if ( + escapeIndex >= guardStart && + limit - escapeIndex < SYNCHRONIZED_OUTPUT_START_SEQUENCE.length + ) { + candidate = escapeIndex + } + // Step 2: the delivered half must not end inside an open frame. + if (!scanSynchronizedOutput(data.slice(0, candidate), markerTail, wasActive).active) { + return candidate > 0 ? candidate : limit + } + // Fall back to the last frame close that ENDS at or before the candidate. + const lastClose = data.lastIndexOf( + SYNCHRONIZED_OUTPUT_END_SEQUENCE, + Math.max(0, candidate - SYNCHRONIZED_OUTPUT_END_SEQUENCE.length) + ) + if (lastClose === -1) { + // One frame is longer than the window; deliver the window and let the + // reset profiles release the latch. + return candidate > 0 ? candidate : limit + } + const aligned = lastClose + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length + // Why the floor: a caller that cannot refill the shortfall in the same round + // (main's flush re-queues the remainder with eligibleRound = round + 1) would + // lose up to half its per-PTY throughput when frames land just past the + // midpoint. Below the floor, prefer throughput and let the reset profiles + // release the latch. + return aligned * 2 >= limit ? aligned : candidate > 0 ? candidate : limit +} diff --git a/tests/e2e/git-churn-load.ts b/tests/e2e/git-churn-load.ts new file mode 100644 index 00000000000..1a14f5b9c5c --- /dev/null +++ b/tests/e2e/git-churn-load.ts @@ -0,0 +1,185 @@ +/** + * Real git-subprocess churn for the multi-workspace typing bench. + * + * Why this exists: every pre-existing scenario in + * terminal-multi-workspace-typing-latency.spec.ts loads the app with PTY bytes + * only, and PTY bytes turn out to be cheap — 24 hidden panes at 512 KB/s left + * typing at 17 ms p50 because main drops hidden renderer-bound bytes at the + * hidden-delivery gate. What the field reports actually have and the bench did + * not is dozens of repositories being polled by the sidebar/source-control + * pollers, which spawn a git child per worktree. Those pollers are gated on + * document visibility (isWindowVisible), so a headless bench window never + * starts them and the dominant main-thread load is simply absent. + * + * This drives the same work through the same production path — main's + * `git:status` handler and its GitAdmissionScheduler — from the renderer, so it + * runs headless without depending on window visibility or on the sidebar being + * mounted. + */ +import type { Page } from '@stablyai/playwright-test' +import { execFileSync } from 'node:child_process' +import { mkdirSync, writeFileSync } from 'node:fs' +import path from 'node:path' + +export type GitChurnRepo = { path: string; index: number } + +export type GitChurnStats = { + requested: number + settled: number + failed: number + durationMsTotal: number + durationMsMax: number + repos: number + concurrency: number + /** First failure's message — a churn loop that only throws measures nothing. */ + firstError: string | null +} + +// Matches the sibling probes (runtime-graph-publication-probe): the handle must +// live on `window` to survive between page.evaluate calls, and `declare global` +// would need an `interface` the lint rules forbid. +type GitChurnWindow = Window & { __orcaGitChurnLoad?: { stop: () => GitChurnStats } } + +/** + * Builds `repoCount` real repositories, each with `filesPerRepo` tracked files + * and a dirty working tree, so `git status` does measurable work instead of + * returning instantly on an empty repo. + */ +export function createGitChurnRepos( + rootDirectory: string, + repoCount: number, + filesPerRepo: number +): GitChurnRepo[] { + const repos: GitChurnRepo[] = [] + for (let index = 0; index < repoCount; index++) { + const repoPath = path.join(rootDirectory, `churn-repo-${index}`) + mkdirSync(repoPath, { recursive: true }) + const git = (...args: string[]): void => { + execFileSync('git', args, { cwd: repoPath, stdio: 'ignore' }) + } + git('init', '--quiet') + git('config', 'user.email', 'bench@example.com') + git('config', 'user.name', 'Bench') + // Why nested directories: a flat tree of N files understates the readdir + // and lstat cost that dominates `git status` on a real checkout. + for (let fileIndex = 0; fileIndex < filesPerRepo; fileIndex++) { + const directory = path.join(repoPath, `dir-${fileIndex % 64}`) + mkdirSync(directory, { recursive: true }) + writeFileSync(path.join(directory, `file-${fileIndex}.txt`), `content ${fileIndex}\n`) + } + git('add', '-A') + git('commit', '--quiet', '-m', 'seed') + // Why left dirty: a clean tree lets git short-circuit; the reported setups + // all have modified worktrees. + for (let fileIndex = 0; fileIndex < Math.min(filesPerRepo, 64); fileIndex++) { + const directory = path.join(repoPath, `dir-${fileIndex % 64}`) + writeFileSync(path.join(directory, `file-${fileIndex}.txt`), `modified ${fileIndex}\n`) + } + repos.push({ path: repoPath, index }) + } + return repos +} + +/** + * Registers each repo with Orca. Required: main's `git:status` handler rejects + * any path that is not a known repository or worktree ("Access denied: unknown + * repository or worktree path"), so an unregistered churn loop only measures + * its own rejection path. + */ +export async function registerGitChurnRepos( + page: Page, + repoPaths: string[] +): Promise<{ registered: number; failures: string[] }> { + return page.evaluate(async (paths) => { + const failures: string[] = [] + let registered = 0 + for (const repoPath of paths) { + try { + // Why inspect the result: repos:add REPORTS failure as { error }, it + // does not throw, so a try/catch alone would count a refusal as success. + const result = await window.api.repos.add({ path: repoPath, kind: 'git' }) + if (result && typeof result === 'object' && 'error' in result) { + failures.push(String(result.error).slice(0, 200)) + } else { + registered += 1 + } + } catch (error) { + failures.push((error instanceof Error ? error.message : String(error)).slice(0, 200)) + } + } + return { registered, failures: failures.slice(0, 5) } + }, repoPaths) +} + +/** + * Starts `concurrency` renderer-side loops that keep calling `git.status` + * across `repoPaths` until stopped. Each call routes through main's IPC + * handler and admission scheduler exactly as a sidebar poll does. + */ +export async function startGitChurnLoad( + page: Page, + repoPaths: string[], + options: { concurrency: number; admissionTier: 'interactive' | 'status' | 'background' } +): Promise { + await page.evaluate( + ({ paths, concurrency, admissionTier }) => { + // oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the probe owns this property on its own renderer window; nothing else reads or writes it. + const target = window as GitChurnWindow + target.__orcaGitChurnLoad?.stop() + let running = true + const stats: GitChurnStats = { + requested: 0, + settled: 0, + failed: 0, + durationMsTotal: 0, + durationMsMax: 0, + repos: paths.length, + concurrency, + firstError: null + } + const runLoop = async (lane: number): Promise => { + let cursor = lane + while (running) { + const worktreePath = paths[cursor % paths.length] + cursor += concurrency + stats.requested += 1 + const startedAt = performance.now() + try { + await window.api.git.status({ worktreePath, admissionTier }) + } catch (error) { + stats.failed += 1 + stats.firstError ??= (error instanceof Error ? error.message : String(error)).slice( + 0, + 300 + ) + } + // Why the yield: a loop whose call rejects synchronously would spin the + // renderer thread and measure the spin, not git churn. + await new Promise((resolve) => setTimeout(resolve, 0)) + const durationMs = performance.now() - startedAt + stats.settled += 1 + stats.durationMsTotal += durationMs + stats.durationMsMax = Math.max(stats.durationMsMax, durationMs) + } + } + for (let lane = 0; lane < concurrency; lane++) { + void runLoop(lane) + } + target.__orcaGitChurnLoad = { + stop: () => { + running = false + return { ...stats } + } + } + }, + { paths: repoPaths, concurrency: options.concurrency, admissionTier: options.admissionTier } + ) +} + +export async function stopGitChurnLoad(page: Page): Promise { + return page.evaluate( + () => + // oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: same probe-owned property as startGitChurnLoad installs. + (window as GitChurnWindow).__orcaGitChurnLoad?.stop() ?? null + ) +} diff --git a/tests/e2e/sustained-agent-typing-load-scripts.ts b/tests/e2e/sustained-agent-typing-load-scripts.ts index 139ff6de001..c5681bf6bea 100644 --- a/tests/e2e/sustained-agent-typing-load-scripts.ts +++ b/tests/e2e/sustained-agent-typing-load-scripts.ts @@ -153,6 +153,85 @@ process.stdin.on('data', (chunk) => { ` } +/** + * Codex-shaped echo probe: same sidecar contract as the plain probe, but each + * keystroke repaint is wrapped in a DEC 2026 synchronized frame and split so the + * closing `\x1b[?2026l` leaves the process in a SEPARATE write from the frame + * body. That is what a real Codex draw looks like on the wire (codex-rs + * tui.rs `stdout().sync_update(...)`, up to 120 FPS), and it is the shape the + * plain probe misses — it emits one unwrapped ~50 byte line per key, so it can + * never leave xterm's synchronized-output latch open. + * + * `frameRows` pads the frame body to a realistic repaint size; `splitDelayMs` + * is the gap between body and close, i.e. how long the latch stays open at the + * source before Orca's delivery adds any of its own. + */ +function codexEchoProbeScript( + runId: string, + arrivalSidecarPath: string, + frameRows: number, + splitDelayMs: number +): string { + return ` +import { appendFileSync } from 'node:fs' + +process.stdin.setEncoding('utf8') +if (process.stdin.isTTY) process.stdin.setRawMode(true) +process.stdin.resume() +let seq = 0 +const interrupt = String.fromCharCode(3) +const rows = ${frameRows} +const splitDelayMs = ${splitDelayMs} +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)) +const write = (data) => + new Promise((resolve) => { + if (process.stdout.write(data)) resolve() + else process.stdout.once('drain', resolve) + }) +process.stdout.write('${'MWT_TYPING_READY_'}${runId}\\r\\n') +let queue = Promise.resolve() +process.stdin.on('data', (chunk) => { + const atMs = Date.now() + if (chunk.includes(interrupt)) process.exit(0) + for (const char of chunk) { + if (char === '\\r' || char === '\\n') continue + seq += 1 + const mySeq = seq + appendFileSync( + ${JSON.stringify(arrivalSidecarPath)}, + JSON.stringify({ seq: mySeq, atMs, char }) + '\\n' + ) + queue = queue.then(async () => { + // Frame body: open the latch, repaint rows, but do NOT close yet. + let body = '\\x1b[?2026h' + for (let row = 1; row <= rows; row++) { + body += '\\x1b[' + row + ';1H\\x1b[2Kcodex frame row ' + row + ' seq ' + mySeq + } + body += + '\\x1b[' + (rows + 1) + ';1H\\x1b[2Kmwt prompt ' + + mySeq + ': ' + char + ' ${'MWT_KEY_'}${runId}_' + mySeq + await write(body) + if (splitDelayMs > 0) await sleep(splitDelayMs) + await write('\\x1b[?2026l') + }) + } +}) +` +} + +export function writeCodexEchoProbeScript( + scriptPath: string, + runId: string, + arrivalSidecarPath: string, + options: { frameRows: number; splitDelayMs: number } +): void { + mkdirSync(path.dirname(scriptPath), { recursive: true }) + writeFileSync( + scriptPath, + codexEchoProbeScript(runId, arrivalSidecarPath, options.frameRows, options.splitDelayMs) + ) +} + export function writeSustainedAgentLoadScript( scriptPath: string, runId: string, diff --git a/tests/e2e/synchronized-output-blackout-probe.ts b/tests/e2e/synchronized-output-blackout-probe.ts new file mode 100644 index 00000000000..bbe0508ef9d --- /dev/null +++ b/tests/e2e/synchronized-output-blackout-probe.ts @@ -0,0 +1,151 @@ +/** + * Measures DEC 2026 render blackouts in the focused pane's xterm. + * + * Why buffer polling cannot see this bug: xterm's parser writes into the buffer + * whether or not synchronized output is open, so the existing bench's + * xterm-buffer observation reports a keystroke as "echoed" while the screen is + * still showing the previous frame. What the user sees is the RENDER. + * + * xterm (6.1.0-beta.303) suppresses all row rendering while + * `decPrivateModes.synchronizedOutput` is set — `RenderService.refreshRows` + * returns early into `SyncOutputHandler.bufferRows` — and the only escapes are + * the closing `\x1b[?2026l` or a 1000 ms timeout armed once per buffering + * episode. So a frame whose close is delayed freezes the pane for up to a + * second and then repaints everything at once, which is exactly the reported + * symptom. + * + * This probe records actual `onRender` timestamps and how long the + * synchronized-output latch stays open, so a run can distinguish "echo arrived + * late" from "echo arrived on time but was not painted". + */ +import type { Page } from '@stablyai/playwright-test' + +export type SynchronizedOutputBlackoutSnapshot = { + installed: boolean + reason: string + /** Wall time the probe observed, ms. */ + observedMs: number + renderCount: number + /** Gaps between consecutive renders, ms. */ + maxRenderGapMs: number + p90RenderGapMs: number + /** Episodes where the latch was observed open, ms each. */ + latchOpenEpisodes: number[] + maxLatchOpenMs: number + /** Episodes at/over this are xterm's 1s forced flush rather than a real close. */ + timeoutScaleEpisodes: number + samples: number +} + +type BlackoutProbeTerminal = { + onRender?: (listener: () => void) => { dispose: () => void } + /** Public IModes getter; xterm's internals are minified in the bundle. */ + modes?: { synchronizedOutputMode?: boolean } +} + +type BlackoutProbePane = { terminal?: BlackoutProbeTerminal } + +// Matches the sibling probes (runtime-graph-publication-probe): the handle must +// live on `window` to survive between page.evaluate calls, and `declare global` +// would need an `interface` the lint rules forbid. +type BlackoutProbeWindow = Window & { + __orcaSyncOutputBlackoutProbe?: { stop: () => SynchronizedOutputBlackoutSnapshot } +} + +/** + * Installs the probe on the focused pane of `tabId`. Sampling runs on a short + * interval; a renderer thread blocked for N ms shows up as a sampling gap, + * which is reported rather than hidden. + */ +export async function startSynchronizedOutputBlackoutProbe( + page: Page, + tabId: string +): Promise<{ installed: boolean; reason: string }> { + return page.evaluate((activeTabId) => { + // oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the probe owns this property on its own renderer window; nothing else reads or writes it. + const target = window as BlackoutProbeWindow + target.__orcaSyncOutputBlackoutProbe?.stop() + const manager = window.__paneManagers?.get(activeTabId) + const pane: BlackoutProbePane | null = + manager?.getActivePane?.() ?? manager?.getPanes?.()[0] ?? null + const terminal = pane?.terminal + if (!terminal) { + return { installed: false, reason: 'no focused pane terminal' } + } + if (typeof terminal.modes?.synchronizedOutputMode !== 'boolean') { + // Why fail loudly: a silently absent latch would report zero blackouts + // and read as "not reproduced". + return { installed: false, reason: 'terminal.modes.synchronizedOutputMode not reachable' } + } + if (typeof terminal.onRender !== 'function') { + return { installed: false, reason: 'terminal.onRender unavailable' } + } + + const startedAt = performance.now() + const renderAtMs: number[] = [] + const latchOpenEpisodes: number[] = [] + let samples = 0 + let latchOpenedAt: number | null = null + const subscription = terminal.onRender(() => { + renderAtMs.push(performance.now()) + }) + const sampleLatch = (): void => { + samples += 1 + const open = terminal.modes?.synchronizedOutputMode === true + if (open && latchOpenedAt === null) { + latchOpenedAt = performance.now() + } else if (!open && latchOpenedAt !== null) { + latchOpenEpisodes.push(performance.now() - latchOpenedAt) + latchOpenedAt = null + } + } + const timer = window.setInterval(sampleLatch, 4) + + target.__orcaSyncOutputBlackoutProbe = { + stop: () => { + window.clearInterval(timer) + subscription.dispose() + sampleLatch() + if (latchOpenedAt !== null) { + latchOpenEpisodes.push(performance.now() - latchOpenedAt) + } + const gaps: number[] = [] + for (let index = 1; index < renderAtMs.length; index++) { + gaps.push(renderAtMs[index] - renderAtMs[index - 1]) + } + const sortedGaps = [...gaps].sort((a, b) => a - b) + const round = (value: number): number => Number(value.toFixed(1)) + return { + installed: true, + reason: 'ok', + observedMs: round(performance.now() - startedAt), + renderCount: renderAtMs.length, + maxRenderGapMs: round(sortedGaps.at(-1) ?? 0), + p90RenderGapMs: round( + sortedGaps[Math.min(sortedGaps.length - 1, Math.floor(0.9 * sortedGaps.length))] ?? 0 + ), + latchOpenEpisodes: latchOpenEpisodes + .map(round) + .sort((a, b) => b - a) + .slice(0, 20), + maxLatchOpenMs: round(Math.max(0, ...latchOpenEpisodes)), + // 900ms+ cannot be a real close at Codex's 120 FPS draw rate; it is + // xterm's 1000ms forced flush. + timeoutScaleEpisodes: latchOpenEpisodes.filter((episode) => episode >= 900).length, + samples + } + } + } + return { installed: true, reason: 'ok' } + }, tabId) +} + +export async function stopSynchronizedOutputBlackoutProbe( + page: Page +): Promise { + return page.evaluate( + () => + // oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: same probe-owned property as startSynchronizedOutputBlackoutProbe installs. + (window as BlackoutProbeWindow).__orcaSyncOutputBlackoutProbe?.stop() ?? null + ) +} diff --git a/tests/e2e/terminal-multi-workspace-typing-latency.spec.ts b/tests/e2e/terminal-multi-workspace-typing-latency.spec.ts index 61e5d26a20f..f22ffd346bc 100644 --- a/tests/e2e/terminal-multi-workspace-typing-latency.spec.ts +++ b/tests/e2e/terminal-multi-workspace-typing-latency.spec.ts @@ -43,6 +43,7 @@ import { } from './paced-terminal-typing' import { ensureTerminalVisible, + getActiveTabId, getActiveWorktreeId, getAllWorktreeIds, switchToWorktree, @@ -63,9 +64,22 @@ import { import { sustainedLoadReadyFilePath, typingProbeReadyMarker, + writeCodexEchoProbeScript, writeSustainedAgentLoadScript, writeTypingEchoProbeScript } from './sustained-agent-typing-load-scripts' +import { + startSynchronizedOutputBlackoutProbe, + stopSynchronizedOutputBlackoutProbe, + type SynchronizedOutputBlackoutSnapshot +} from './synchronized-output-blackout-probe' +import { + createGitChurnRepos, + registerGitChurnRepos, + startGitChurnLoad, + stopGitChurnLoad, + type GitChurnStats +} from './git-churn-load' import { cleanupAccumulatedWorkspaceFixture, seedAccumulatedWorkspaceFixture, @@ -104,6 +118,16 @@ const LOAD_RATE_KBPS = readPositiveInt('ORCA_TYPING_BENCH_RATE_KBPS', 256) const KEY_COUNT = readPositiveInt('ORCA_TYPING_BENCH_KEYS', 32) const KEY_CADENCE_MS = readPositiveInt('ORCA_TYPING_BENCH_KEY_CADENCE_MS', 250) const CPU_WORKERS = readPositiveInt('ORCA_TYPING_BENCH_CPU_WORKERS', 0) +// Git churn stands in for the sidebar/source-control pollers, which spawn a git +// child per worktree and are gated on window visibility — so a headless bench +// window never runs them and the field's dominant main-thread load is missing. +const GIT_CHURN_REPOS = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_REPOS', 0) +const GIT_CHURN_FILES = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_FILES', 2000) +const GIT_CHURN_CONCURRENCY = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_CONCURRENCY', 8) +// Codex-shaped foreground: rows repainted per keystroke, and the gap the probe +// itself leaves between the frame body and its closing \x1b[?2026l. +const CODEX_FRAME_ROWS = readPositiveInt('ORCA_TYPING_BENCH_CODEX_FRAME_ROWS', 20) +const CODEX_SPLIT_DELAY_MS = Number(process.env.ORCA_TYPING_BENCH_CODEX_SPLIT_DELAY_MS ?? 0) || 0 const PTY_METADATA = process.env.ORCA_TYPING_BENCH_PTY_METADATA === '1' const BENCH_LABEL = process.env.ORCA_TYPING_BENCH_LABEL ?? 'dev' // Request optional probes by default; the report records when the build does not install them. @@ -227,7 +251,9 @@ function writeBenchReport( scaleCensus?: unknown, accumulatedFixture?: unknown, ptyWorkload?: unknown, - graphProbe?: RuntimeGraphPublicationProbeSnapshot | null + graphProbe?: RuntimeGraphPublicationProbeSnapshot | null, + gitChurn?: GitChurnStats | null, + blackout?: SynchronizedOutputBlackoutSnapshot | null ): void { const { measurement, appliedCpuThrottleRate } = measured const report = { @@ -285,7 +311,9 @@ function writeBenchReport( scaleCensus: scaleCensus ?? null, accumulatedFixture: accumulatedFixture ?? null, ptyWorkload: ptyWorkload ?? null, - graphProbe: graphProbe ?? null + graphProbe: graphProbe ?? null, + gitChurn: gitChurn ?? null, + synchronizedOutputBlackout: blackout ?? null } mkdirSync(RESULTS_DIR, { recursive: true }) const stamp = report.timestamp.replace(/[:.]/g, '-') @@ -698,4 +726,190 @@ test.describe('Multi-workspace sustained typing latency bench', () => { removeLoadReadyFiles(testRepoPath, runId, panes.length) } }) + + /** + * The field shape the PTY-only scenarios miss: many repositories being polled + * for git status while the user types. Each poll spawns a git child from + * main, and PTY input and echo are relayed through that same main event loop. + * + * Requires --git-churn-repos; without it the scenario would silently be the + * visible-split scenario again. + */ + test('typing under sustained git-subprocess churn', async ({ + orcaPage, + testRepoPath + }, testInfo) => { + test.skip( + GIT_CHURN_REPOS === 0, + 'Set --git-churn-repos to run: this scenario measures git-spawn churn, not PTY bytes' + ) + await waitForSessionReady(orcaPage) + await waitForActiveWorktree(orcaPage) + await ensureTerminalVisible(orcaPage) + await waitForActiveTerminalManager(orcaPage, 30_000) + + const runId = randomUUID() + const loadPath = path.join(testRepoPath, `.orca-mwt-load-${runId}.mjs`) + const probePath = path.join(testRepoPath, `.orca-mwt-probe-${runId}.mjs`) + const sidecarPath = path.join(testRepoPath, `.orca-mwt-arrivals-${runId}.jsonl`) + const churnRoot = mkdtempSync(path.join(tmpdir(), 'orca-git-churn-')) + writeSustainedAgentLoadScript(loadPath, runId, testRepoPath) + writeTypingEchoProbeScript(probePath, runId, sidecarPath) + + const cpuWorkers = spawnCpuPressureWorkers() + let panes: TerminalLoadPane[] = [] + let gitChurn: GitChurnStats | null = null + try { + const repos = createGitChurnRepos(churnRoot, GIT_CHURN_REPOS, GIT_CHURN_FILES) + const registration = await registerGitChurnRepos( + orcaPage, + repos.map((repo) => repo.path) + ) + expect( + registration, + `churn repos were not registered with Orca: ${registration.failures.join('; ')}` + ).toMatchObject({ registered: repos.length }) + panes = await ensureActiveWorktreePaneLoad(orcaPage, 2) + const [typingPane, ...loadPanes] = panes + await startSustainedLoadInPanes(orcaPage, loadPanes, loadPath, runId, testRepoPath) + await focusPane(orcaPage, typingPane.paneKey) + + await resetDeliveryDebug(orcaPage) + await startTypingProbe(orcaPage, typingPane.ptyId, probePath, runId) + await startGitChurnLoad( + orcaPage, + repos.map((repo) => repo.path), + { concurrency: GIT_CHURN_CONCURRENCY, admissionTier: 'status' } + ) + const measured = await measureTypingWindow(orcaPage, runId, sidecarPath) + gitChurn = await stopGitChurnLoad(orcaPage) + const { measurement } = measured + writeBenchReport( + testInfo, + `git-churn-${GIT_CHURN_REPOS}repos-x${GIT_CHURN_CONCURRENCY}`, + measured, + await readSchedulerDebug(orcaPage), + await readMainDeliveryDebug(orcaPage), + undefined, + null, + null, + null, + undefined, + undefined, + undefined, + null, + gitChurn + ) + // The churn must have actually spawned git, or a fast result means nothing: + // a loop that only rejects measures the rejection path, not subprocess churn. + expect(gitChurn?.settled ?? 0).toBeGreaterThan(0) + expect( + gitChurn, + `git churn never reached git: ${gitChurn?.firstError ?? 'unknown error'}` + ).toMatchObject({ failed: 0 }) + expect(measurement.inputHalfMs?.count).toBe(KEY_COUNT) + expect(measurement.totalMs?.count).toBe(KEY_COUNT) + } finally { + await stopGitChurnLoad(orcaPage).catch(() => null) + for (const worker of cpuWorkers) { + worker.kill('SIGKILL') + } + await stopPtysQuietly( + orcaPage, + panes.map((pane) => pane.ptyId) + ) + rmSync(loadPath, { force: true }) + rmSync(probePath, { force: true }) + rmSync(sidecarPath, { force: true }) + rmSync(churnRoot, { recursive: true, force: true }) + removeLoadReadyFiles(testRepoPath, runId, panes.length) + } + }) + + /** + * The shape that actually matches the report: a Codex-like foreground TUI that + * wraps every keystroke repaint in a DEC 2026 synchronized frame, under + * background agent load, measured at the RENDER rather than at the xterm + * buffer. + * + * xterm suppresses all row rendering while the synchronized-output latch is + * open and force-flushes on a 1000 ms timeout, so a delayed closing + * `\x1b[?2026l` freezes the pane for up to a second and then repaints in one + * burst. Buffer-observation scenarios score that as a fast echo. + */ + test('typing a DEC 2026 foreground TUI under agent load, measured at the render', async ({ + orcaPage, + testRepoPath + }, testInfo) => { + await waitForSessionReady(orcaPage) + await waitForActiveWorktree(orcaPage) + await ensureTerminalVisible(orcaPage) + await waitForActiveTerminalManager(orcaPage, 30_000) + + const runId = randomUUID() + const loadPath = path.join(testRepoPath, `.orca-mwt-load-${runId}.mjs`) + const probePath = path.join(testRepoPath, `.orca-mwt-probe-${runId}.mjs`) + const sidecarPath = path.join(testRepoPath, `.orca-mwt-arrivals-${runId}.jsonl`) + writeSustainedAgentLoadScript(loadPath, runId, testRepoPath) + writeCodexEchoProbeScript(probePath, runId, sidecarPath, { + frameRows: CODEX_FRAME_ROWS, + splitDelayMs: CODEX_SPLIT_DELAY_MS + }) + + const cpuWorkers = spawnCpuPressureWorkers() + let panes: TerminalLoadPane[] = [] + let blackout: SynchronizedOutputBlackoutSnapshot | null = null + try { + panes = await ensureActiveWorktreePaneLoad(orcaPage, Math.max(2, LOAD_PANES + 1)) + const [typingPane, ...loadPanes] = panes + await startSustainedLoadInPanes(orcaPage, loadPanes, loadPath, runId, testRepoPath) + await focusPane(orcaPage, typingPane.paneKey) + + await resetDeliveryDebug(orcaPage) + await startTypingProbe(orcaPage, typingPane.ptyId, probePath, runId) + const activeTabId = await getActiveTabId(orcaPage) + const probeInstall = await startSynchronizedOutputBlackoutProbe(orcaPage, activeTabId ?? '') + // A probe that did not install would report zero blackouts, which reads + // identically to "no bug". + expect(probeInstall, `blackout probe not installed: ${probeInstall.reason}`).toMatchObject({ + installed: true + }) + + const measured = await measureTypingWindow(orcaPage, runId, sidecarPath) + blackout = await stopSynchronizedOutputBlackoutProbe(orcaPage) + const { measurement } = measured + writeBenchReport( + testInfo, + `dec2026-render-${LOAD_PANES}panes-${LOAD_RATE_KBPS}kbps-rows${CODEX_FRAME_ROWS}`, + measured, + await readSchedulerDebug(orcaPage), + await readMainDeliveryDebug(orcaPage), + undefined, + null, + null, + null, + await readTypingScaleCensus(orcaPage).catch(() => null), + undefined, + undefined, + null, + null, + blackout + ) + expect(measurement.totalMs?.count).toBe(KEY_COUNT) + expect(blackout?.renderCount ?? 0).toBeGreaterThan(0) + } finally { + await stopSynchronizedOutputBlackoutProbe(orcaPage).catch(() => null) + for (const worker of cpuWorkers) { + worker.kill('SIGKILL') + } + await stopPtysQuietly( + orcaPage, + panes.map((pane) => pane.ptyId) + ) + rmSync(loadPath, { force: true }) + rmSync(probePath, { force: true }) + rmSync(sidecarPath, { force: true }) + removeLoadReadyFiles(testRepoPath, runId, panes.length) + } + }) })