fix(terminal): release xterm's DEC 2026 render hold instead of waiting out its 1s timeout (#23920)

* fix(terminal): release xterm's DEC 2026 render hold instead of waiting out its 1s timeout

xterm paints nothing while DEC mode 2026 (synchronized output) is open and only
force-flushes after 1000ms. Codex wraps every draw in mode 2026, so any byte gap
or chunk split that loses the closing \x1b[?2026l freezes the pane for a full
second and then repaints in one burst.

Orca never emitted \x1b[?2026l anywhere, and three paths could destroy a TUI's:
the per-PTY pending cap drops buffered output wholesale (mode 2031 was already
salvaged there, 2026 was not), main sliced pending data at a blind 16KB offset
that can land inside an open frame or sever the 8-byte marker, and the renderer's
backlog warnings replace a queued tail that may hold the close.

- salvage the 2026 latch across dropped output, mirroring the existing 2031
  salvage, and append the release on both delivery sites
- ground 2026 in RESET_AFTER_BYTE_GAP and the replay baseline, and in both
  backlog warnings, so every drop path is self-healing
- make main's 16KB flush split frame-aware instead of a blind byte offset
- lift the synchronized-output scanner into shared/ so main and the renderer
  use one implementation

Closing a frame early costs one premature repaint; leaving it open costs a
second of blank screen, so the asymmetry favours always closing.

Also adds the reproduction this needed: the pre-existing typing bench observes
the xterm BUFFER, which the parser fills while rendering is held, so it scored
these freezes as fast echoes.

* fix(terminal): stop the renderer's queue drain cutting inside an open DEC 2026 frame

takeQueuedChunk sliced a queued chunk at a blind byte offset to fit the 16KB
coalescing budget, which can strand a frame's closing \x1b[?2026l in the residual
until a later drain. Same defect as main's flush split, same fix: reuse the
frame-aware split helper.

Usually masked because the drain coalesces adjacent chunks and reassembles what
main split, but not when the budget boundary falls inside a frame.

* fix(relay): keep the SSH path's bounded slice outside an open DEC 2026 frame

pty-handler split pending output at a byte offset with a surrogate-pair guard but
no synchronized-output awareness, so a frame straddling the 16KB wire slice had
its closing \x1b[?2026l stranded in the remainder — the same defect just fixed on
the local path, on the path AGENTS.md requires us to consider.

Placed before the surrogate guard so that guard keeps the final say, and floored
at 2 so frame alignment can never walk a healthy slice into the guard's
decrement and then into the chunkChars <= 0 pause-and-retry path.

Also drops a dead `splitAt === 0` branch in takeQueuedChunk: both callers pass a
positive limit and the helper never returns 0 for one.

The two new split tests were each confirmed to fail without their fix.

* test(terminal): sweep the DEC 2026 split helper over escape-sequence shapes and every limit

Covers OSC 52, DCS, repeated open/close markers and limits 1..len+3, asserting the
result never exceeds the limit, never reaches 0, and stays byte-exact. Also pins
that a buffer beginning inside an open frame degrades to the blind offset rather
than doing something worse, and documents that callers do not thread latch state.

* fix(terminal): ground DEC 2026 on the daemon slice, the recovery replays, and the process boundary

Four more sites could strand the latch, found by sweeping every path that drops,
splits, or replays terminal bytes.

- daemon-stream-data-batcher: the 64KB bulk-write slice used a surrogate-only
  clamp, and its remainder is HELD until 'drain' — "seconds for multi-MB
  backlogs" per the file's own note. A frame straddling that boundary parked its
  \x1b[?2026l behind the hold, blanking the pane past xterm's 1s timeout once per
  frame for as long as the backlog lasted. This is the default daemon-backed pane
  path, so it is the one users actually hit. The new
  clampToSafeBulkWriteSplitIndex frame-aligns first and surrogate-clamps last,
  and lives in daemon-stream-data-split alongside the policy it belongs to.
- replay-data-drain and remote-runtime-terminal-binary-snapshots wrote a bare
  \x1b[2J\x1b[3J\x1b[H, which does not clear mode 2026 — so on the SSH/remote
  reconnect path, the very event most likely to sever a frame, the whole replay
  could paint nothing.
- ipc-pty-attach: trimIncompleteTerminalControlTail can cut a half-written
  \x1b[?2026l while its opening marker survives in the replayed prefix.
- PROCESS_BOUNDARY_GROUND: the "process that armed these modes is gone" ground
  omitted 2026, the last unexplained gap in that file. A disable, so it still
  satisfies the recovery barrier's ownership scan (only ?25h may be an enable).

Recovery-path expectations updated where they pin the emitted bytes. Deliberately
NOT touched: apply-reattach-payload and ssh-snapshot-prepaint already ground via
buildSnapshotReplayPrologue.

Still unfixed, deferred with reason: terminal-output-frame-chunks.ts splits the
remote wire on accumulated UTF-8 byte width and needs a different shape than the
char-index helper; desktop clients reassemble in main's pending buffer, so the
exposure is mobile/web only.

* fix(terminal): emit the DEC 2026 release before the mode-2031 tail, and stop claiming the drop path writes it

Two corrections from adversarial review of the earlier commits.

1. Ordering bug I introduced. getDroppedMode2031RendererData ends with
   `state.tail`, which extractPrivateModeScanTail deliberately retains as an
   INCOMPLETE private-mode sequence so the next chunk can resolve it. Appending the
   2026 release after it put an ESC behind a dangling CSI, aborting it and silently
   losing whatever mode spanned the drop boundary. The release now goes first.

2. The drop-path release does not reach xterm in the dominant case, and the comment
   now says so instead of implying otherwise. live-data-callback's droppedOutput
   branch discards `data` and salvages only queries
   (salvageRendererQueriesFromDiscardedRestoreData handles CPR/DA1/OSC colour;
   \x1b[?2026l is not a query), so for hidden panes and visible panes outside
   foreground-restore backpressure the synthesized release was dropped. The grounded
   snapshot replay releases the latch instead.

   I tried writing it through writePtyOutputToXterm there and reverted: it consumes
   the pending hidden-output snapshot and broke
   pty-connection-hidden-snapshot-resize-signals ("re-restores a skipped alt frame"),
   so the release rides the restore rather than perturbing that state machine.
   Residual gap, documented: a cap-dropped pane whose restore never arrives.

The salvage is still load-bearing on the fall-through path, so it stays.

* fix(terminal): release DEC 2026 on the reattach clears, floor the split, and correct the freeze framing

Remaining findings from adversarial review.

- apply-reattach-payload's three bare-clear branches (:63 daemon snapshot, :229
  relay replay, :269 cold restore) had no release anywhere in their sequence: I
  checked all seven POST_REPLAY_* profiles reachable via chooseReattachReplayReset
  and none contains \x1b[?2026l. Only the buildMainModelSnapshotReplayWrites branch
  was grounded, so covering the streamed replay path and not the main reattach path
  was inconsistent. Verified no production code matches these clear strings — the
  three test updates are mock equality, and each was confirmed to fail without the
  source change.
- clampToSafeBulkWriteSplitIndex could return 0 (('\u{1F600}aaaa', 1) — alignment
  returns 1, the surrogate clamp decrements to 0), which would leave a zero-length
  slice that never shifts the batcher's queue entry and spin its drain loop.
  Unreachable from today's only caller, but it is exported with an unstated
  precondition. Floored at 1.
- Frame alignment could halve per-PTY flush throughput: main re-queues the
  remainder with eligibleRound = round + 1, so the shortfall cannot be refilled in
  the same round, and aligned size is floor(W/F)*F — 50% worst case in the 8-16KB
  band, which is exactly the full-screen redraw burst that reaches the pending cap.
  Alignment is now rejected below half the window, preferring throughput and
  letting the reset profiles release the latch.

Framing corrected throughout: bufferRows records a row range and clears nothing, so
the pane freezes on its last painted frame — it does not go blank. The real trade is
"stale but coherent for <=1s" versus "immediate partial frame", and
RESET_AFTER_BYTE_GAP (written alone, with no repaint behind it in the same write) is
the one site that can newly flash a partial frame. Said so at the constant instead
of implying the release is free.

* fix(terminal): rename the shape-flagged symbols the anti-slop audit rejects

CI's anti-slop gate rejects "shape" in symbol names as structural rather than
domain language: `shapes` -> `outputSamples`, and
`writeCodexShapedEchoProbeScript`/`codexShapedEchoProbeScript` ->
`writeCodexEchoProbeScript`/`codexEchoProbeScript`.
This commit is contained in:
Neil
2026-09-29 20:27:30 -07:00
committed by GitHub
parent 33351b0085
commit fb52c0602a
34 changed files with 1275 additions and 106 deletions
@@ -23,6 +23,11 @@ const knobByFlag = {
'--keys': 'ORCA_TYPING_BENCH_KEYS',
'--cadence-ms': 'ORCA_TYPING_BENCH_KEY_CADENCE_MS',
'--cpu-workers': 'ORCA_TYPING_BENCH_CPU_WORKERS',
'--git-churn-repos': 'ORCA_TYPING_BENCH_GIT_CHURN_REPOS',
'--git-churn-files': 'ORCA_TYPING_BENCH_GIT_CHURN_FILES',
'--git-churn-concurrency': 'ORCA_TYPING_BENCH_GIT_CHURN_CONCURRENCY',
'--codex-frame-rows': 'ORCA_TYPING_BENCH_CODEX_FRAME_ROWS',
'--codex-split-delay-ms': 'ORCA_TYPING_BENCH_CODEX_SPLIT_DELAY_MS',
'--worktrees': 'ORCA_TYPING_BENCH_METADATA_WORKTREES',
'--repositories': 'ORCA_TYPING_BENCH_METADATA_REPOSITORIES',
'--terminal-tabs': 'ORCA_TYPING_BENCH_METADATA_TERMINAL_TABS',
@@ -7,7 +7,7 @@ import {
releaseDaemonStreamEntry
} from './daemon-stream-entry-accounting'
import { DaemonStreamHeldRefill } from './daemon-stream-held-refill'
import { clampToSafeSplitIndex, writeStreamDataEvents } from './daemon-stream-data-split'
import { clampToSafeBulkWriteSplitIndex, writeStreamDataEvents } from './daemon-stream-data-split'
import type { PendingStreamDataBatch } from './daemon-stream-keep-tail-drop'
import type { DaemonEvent } from './types'
import {
@@ -213,7 +213,7 @@ export class DaemonStreamDataBatcher {
const end =
entry.transformed || entry.data.length <= BULK_WRITE_SLICE_CHARS
? entry.data.length
: clampToSafeSplitIndex(entry.data, 0, BULK_WRITE_SLICE_CHARS)
: clampToSafeBulkWriteSplitIndex(entry.data, BULK_WRITE_SLICE_CHARS)
const slice = entry.data.slice(0, end)
const entrySequenceChars = entry.sequenceChars ?? entry.data.length
const sliceSequenceChars = entry.transformed
@@ -3,6 +3,7 @@
* chunking (the receiver's parser rejects oversized lines) and the safe-index
* clamp shared by the batcher's bulk write slicing and keep-tail dropping.
*/
import { resolveSynchronizedOutputSafeSplit } from '../../shared/terminal-synchronized-output-scan'
import { encodeNdjson } from './ndjson'
export function encodeStreamDataEvent(
@@ -38,6 +39,21 @@ function isLowSurrogate(value: number): boolean {
return value >= 0xdc00 && value <= 0xdfff
}
/**
* Bulk-write split policy: frame-align first so a held remainder cannot strand an
* open DEC 2026 frame's closing \x1b[?2026l (xterm then stops repainting until its
* 1s timeout), then let the surrogate clamp have the final say.
*/
export function clampToSafeBulkWriteSplitIndex(value: string, end: number): number {
// Math.max(1): the surrogate clamp can decrement an aligned index to 0 (e.g.
// ('\u{1F600}aaaa', 1)), and a 0-length slice would never shift the batcher's
// queue entry, spinning its drain loop.
return Math.max(
1,
clampToSafeSplitIndex(value, 0, resolveSynchronizedOutputSafeSplit(value, end))
)
}
export function clampToSafeSplitIndex(value: string, start: number, end: number): number {
if (end <= start || end >= value.length) {
return end
@@ -1,3 +1,4 @@
import type { SynchronizedOutputLatchState } from '../../shared/terminal-synchronized-output-scan'
import type { Mode2031ReplyScanState } from '../../shared/terminal-color-scheme-protocol'
export type PendingPtyData = {
@@ -9,6 +10,9 @@ export type PendingPtyData = {
droppedOutput?: true
droppedMode2031Data?: string
droppedMode2031ScanState?: Mode2031ReplyScanState
/** Latch state across bytes the renderer never received, so a drop that swallowed
* a closing \x1b[?2026l can still release xterm's render hold. */
droppedSynchronizedOutputState?: SynchronizedOutputLatchState
projectionAdmissionIds?: readonly string[]
projectionAdmissionsTransferred?: true
}
+12 -2
View File
@@ -9,7 +9,11 @@ import {
activeRendererPtys
} from './visibility-state'
import { PTY_BATCH_INTERVAL_MS } from './constants'
import { appendPendingPtyData, getDroppedMode2031RendererData } from './pending'
import {
appendPendingPtyData,
getDroppedMode2031RendererData,
getDroppedSynchronizedOutputRendererData
} from './pending'
import { sendModelRestoreNeededMarker, sendPtyDataToRenderer } from './payload'
import { shouldSendInteractiveOutputNow } from './interactive'
import { requestDeliveryResyncForGatedPty } from './accounting'
@@ -98,7 +102,13 @@ export function acceptPtyDataForRenderer(
pending.droppedOutput === true &&
!overflowMarkedBeforeAppend &&
session.pendingOverflowMarkedPtys.has(payload.id)
const nextData = pending.data + getDroppedMode2031RendererData(pending)
// Why the 2026 release goes BEFORE the 2031 data: that payload ends with a
// deliberately-retained INCOMPLETE private-mode sequence (extractPrivateModeScanTail),
// and an ESC after it would abort the dangling CSI and lose the carried mode.
const nextData =
pending.data +
getDroppedSynchronizedOutputRendererData(pending) +
getDroppedMode2031RendererData(pending)
const isInteractiveOutput = shouldSendInteractiveOutputNow(
payload.id,
nextData,
+16 -3
View File
@@ -13,9 +13,11 @@ import {
} from './constants'
import {
getDroppedMode2031RendererData,
getDroppedSynchronizedOutputRendererData,
pendingProjectionAdmissionOptions,
updatePendingProjectionAdmissions
} from './pending'
import { resolveSynchronizedOutputSafeSplit } from '../../../../shared/terminal-synchronized-output-scan'
import { makePtyDataPayload, sendModelRestoreNeededMarker, sendPtyDataToRenderer } from './payload'
import { warnIfDroppingHiddenBytesForVisiblePty } from './debug-snapshot'
import type { PtyIpcSession } from '../session'
@@ -144,7 +146,12 @@ export function flushPendingData(session: PtyIpcSession): void {
id,
{
id,
data: pending.data + getDroppedMode2031RendererData(pending),
// 2026 release before the 2031 data: that payload ends with a retained
// partial private-mode sequence an ESC after it would abort.
data:
pending.data +
getDroppedSynchronizedOutputRendererData(pending) +
getDroppedMode2031RendererData(pending),
droppedOutput: true
},
pending.projectionAdmissionIds
@@ -158,8 +165,14 @@ export function flushPendingData(session: PtyIpcSession): void {
}
const { data } = pending
const indivisible = pending.transformed === true
const chunk = indivisible ? data : data.slice(0, PTY_BATCH_FLUSH_CHUNK_CHARS)
const remaining = indivisible ? '' : data.slice(PTY_BATCH_FLUSH_CHUNK_CHARS)
// Why not a blind offset: splitting inside an open DEC 2026 frame strands
// the closing \x1b[?2026l on a later flush, and xterm stops repainting until
// it arrives or its 1000ms timeout fires.
const splitAt = indivisible
? data.length
: resolveSynchronizedOutputSafeSplit(data, PTY_BATCH_FLUSH_CHUNK_CHARS)
const chunk = indivisible ? data : data.slice(0, splitAt)
const remaining = indivisible ? '' : data.slice(splitAt)
let nextPending: PendingPtyData | undefined
if (remaining) {
nextPending = { data: remaining }
@@ -0,0 +1,92 @@
import { beforeEach, describe, expect, it } from 'vitest'
import { INTERACTIVE_OUTPUT_BUDGET_CHARS, INTERACTIVE_OUTPUT_WINDOW_MS } from './constants'
import { shouldSendInteractiveOutputNow } from './interactive'
import { interactiveOutputCharsByPty, lastInputAtByPty } from './visibility-state'
const PTY_ID = 'pty-typing-burst'
/**
* A Codex composer repaint: one DEC 2026 synchronized frame per keystroke,
* carrying the closing \x1b[?2026l that releases xterm's render latch. Sized
* from a real 204x52 capture of codex-cli 0.158.0, whose per-keystroke frames
* run from a few dozen bytes up to ~1.4KB.
*
* Why the fast path matters for this shape specifically: xterm renders nothing
* while synchronized output is open and force-flushes only after 1000ms, so a
* repaint diverted to the shared batch flush timer — behind every other pane's
* output — leaves the pane visually frozen rather than merely late.
*/
function codexFrame(bytes: number): string {
return `\x1b[?2026h\x1b[1;1H${'x'.repeat(Math.max(0, bytes - 20))}\x1b[?2026l`
}
/**
* Mirrors the production input path: `noteRendererPtyInput`
* (src/main/ipc/pty/ipc/write-input.ts:152-155) stamps the input time AND zeroes
* the pty's interactive budget on every keystroke. A test that only stamps the
* time would let the budget accumulate across keys and report a fast-path
* divergence that cannot happen in production.
*/
function noteRendererPtyInput(now: number): void {
lastInputAtByPty.set(PTY_ID, now)
interactiveOutputCharsByPty.set(PTY_ID, 0)
}
function typeKeyAndRepaint(now: number, frame: string): boolean {
noteRendererPtyInput(now)
return shouldSendInteractiveOutputNow(PTY_ID, frame, now + 2)
}
describe('interactive output fast path during a continuous typing burst', () => {
beforeEach(() => {
lastInputAtByPty.delete(PTY_ID)
interactiveOutputCharsByPty.delete(PTY_ID)
})
it('keeps every repaint of a sustained burst on the fast path', () => {
// 90ms apart: inside INTERACTIVE_OUTPUT_WINDOW_MS, i.e. a fast typist who
// never pauses long enough to expire the window.
const cadenceMs = INTERACTIVE_OUTPUT_WINDOW_MS - 10
const frame = codexFrame(1400)
const batched: number[] = []
let now = 1_000
for (let key = 0; key < 200; key++) {
if (!typeKeyAndRepaint(now, frame)) {
batched.push(key)
}
now += cadenceMs
}
expect(
batched,
`keys diverted off the interactive fast path at 1400-byte frames: ${batched.join(', ')}`
).toEqual([])
})
it('still cuts off a single keystroke that triggers a flood', () => {
// The budget is per keystroke, so it must still bound one key's blast
// radius: a TUI dumping megabytes after one keypress cannot ride the
// immediate path past the budget and starve main's timers.
const frame = codexFrame(8 * 1024)
let now = 1_000
noteRendererPtyInput(now)
let admitted = 0
while (shouldSendInteractiveOutputNow(PTY_ID, frame, now + 2)) {
admitted += 1
// No new keystroke: the same key's repaints keep arriving.
now += 1
}
expect(admitted * frame.length).toBeLessThanOrEqual(INTERACTIVE_OUTPUT_BUDGET_CHARS)
expect(admitted).toBeGreaterThan(0)
})
it('leaves the fast path when output arrives long after the last keystroke', () => {
const frame = codexFrame(1400)
const now = 1_000
noteRendererPtyInput(now)
// Unprompted output, well past the input window: throughput work, not echo.
expect(
shouldSendInteractiveOutputNow(PTY_ID, frame, now + INTERACTIVE_OUTPUT_WINDOW_MS + 1)
).toBe(false)
})
})
+23 -2
View File
@@ -4,6 +4,10 @@ import {
scanMode2031ReplyDecision,
type Mode2031ReplyScanState
} from '../../../../shared/terminal-color-scheme-protocol'
import {
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE,
advanceDroppedSynchronizedOutputLatch
} from '../../../../shared/terminal-synchronized-output-scan'
import { redactPtyIdForDiagnostics } from '../../../../shared/pty-delivery-diagnostics'
import { recordCrashBreadcrumb } from '../../../crash-reporting/crash-breadcrumb-store'
import { terminalOutputBacklogCapChars } from '../../../../shared/terminal-scrollback-policy'
@@ -52,6 +56,13 @@ export function getDroppedMode2031RendererData(pending: PendingPtyData): string
return (pending.droppedMode2031Data ?? '') + pendingSubscribe + state.tail
}
/** Releases xterm's DEC 2026 render hold when the dropped span left a frame open.
* Without it the pane freezes on its last painted frame until xterm's 1000ms
* forced flush. */
export function getDroppedSynchronizedOutputRendererData(pending: PendingPtyData): string {
return pending.droppedSynchronizedOutputState?.active === true ? '\x1b[?2026l' : ''
}
export function pendingProjectionAdmissionOptions(session: PtyIpcSession) {
return {
isPending: (id: string) => session.sshOutputIntake?.hasUnpublishedProjection(id) ?? false,
@@ -117,12 +128,17 @@ export function dropOversizedPendingPtyData(
session.sshOutputIntake?.transferProjections(pending.projectionAdmissionIds, 'pending-cap')
}
const mode2031 = scanDroppedMode2031Data(pending.data, INITIAL_MODE_2031_REPLY_SCAN_STATE)
const synchronizedOutput = advanceDroppedSynchronizedOutputLatch(
pending.data,
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
// Why no trimmed content tail: a mid-stream gap would corrupt the pane; the droppedOutput sentinel repaints from the snapshot and realigns by sequence (only query bytes ride along).
return {
data: extractDroppedPtyQueryBytes(pending.data).slice(0, DROPPED_QUERY_SALVAGE_MAX_CHARS),
droppedOutput: true,
droppedMode2031Data: mode2031.data,
droppedMode2031ScanState: mode2031.state
droppedMode2031ScanState: mode2031.state,
droppedSynchronizedOutputState: synchronizedOutput.state
}
}
@@ -147,6 +163,10 @@ export function appendPendingPtyData(
data,
existing.droppedMode2031ScanState ?? INITIAL_MODE_2031_REPLY_SCAN_STATE
)
const synchronizedOutput = advanceDroppedSynchronizedOutputLatch(
data,
existing.droppedSynchronizedOutputState ?? INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
const remainingQueryCapacity = Math.max(
0,
DROPPED_QUERY_SALVAGE_MAX_CHARS - existing.data.length
@@ -156,7 +176,8 @@ export function appendPendingPtyData(
...existing,
data: existing.data + salvaged,
droppedMode2031Data: mode2031.data || existing.droppedMode2031Data,
droppedMode2031ScanState: mode2031.state
droppedMode2031ScanState: mode2031.state,
droppedSynchronizedOutputState: synchronizedOutput.state
}
}
const projectionState = compactPendingProjectionState(
@@ -713,6 +713,41 @@ describe('PtyHandler', () => {
})
})
it('does not split a bounded slice inside an open DEC 2026 frame', async () => {
let dataCallback: ((data: string) => void) | undefined
mockPtySpawn.mockReturnValue({
...mockPtyInstance,
onData: vi.fn((cb: (data: string) => void) => {
dataCallback = cb
}),
onExit: vi.fn()
})
await dispatcher.callRequest('pty.spawn', {})
// A frame that closes just before the 16KB boundary, then a second frame
// that straddles it. Cutting at the raw boundary would strand the second
// frame's \x1b[?2026l, and xterm paints nothing while the latch is open.
const open = '\x1b[?2026h'
const close = '\x1b[?2026l'
const firstFrame = `${open}${'x'.repeat(16 * 1024 - 2 * open.length - close.length)}${close}`
const secondFrame = `${open}${'y'.repeat(64)}${close}`
dataCallback!(`${firstFrame}${secondFrame}`)
vi.advanceTimersByTime(8)
expect(dispatcher.notify).toHaveBeenCalledTimes(1)
expect(dispatcher.notify).toHaveBeenNthCalledWith(1, 'pty.data', {
id: PTY_1,
data: firstFrame
})
vi.advanceTimersByTime(1)
expect(dispatcher.notify).toHaveBeenCalledTimes(2)
expect(dispatcher.notify).toHaveBeenNthCalledWith(2, 'pty.data', {
id: PTY_1,
data: secondFrame
})
})
it('writes data to PTY via pty.data notification', async () => {
const mockWrite = vi.fn()
mockPtySpawn.mockReturnValue({
+15 -1
View File
@@ -1,6 +1,7 @@
/* oxlint-disable max-lines */
import { resolveSynchronizedOutputSafeSplit } from '../shared/terminal-synchronized-output-scan'
import { FreebuffStatusProjection } from './freebuff-status-projection'
import { applyRelayAgentWorkspaceTrust } from './agent-workspace-trust-spawn'
/* oxlint-disable max-lines */
import type { IPty } from 'node-pty'
import { killWithDescendantSweep } from '../main/pty-descendant-termination'
import type * as NodePty from 'node-pty'
@@ -1348,6 +1349,19 @@ export class PtyHandler {
? desiredChars
: (this.dispatcher.maxLegacyPtyDataChars?.(paramsWithoutData, pending.data, desiredChars) ??
desiredChars)
// Why before the surrogate guard: splitting inside an open DEC 2026 frame
// strands the closing \x1b[?2026l in the remainder, and xterm stops repainting
// until it arrives or its 1000ms timeout fires. The surrogate guard keeps the
// final say so a frame boundary can never sever a pair.
if (!pending.transformed && !pending.sourceChunk && chunkChars > 0) {
const frameAligned = resolveSynchronizedOutputSafeSplit(pending.data, chunkChars)
// Why the floor of 2: the surrogate guard below can decrement by one, and
// a chunkChars of 0 takes the pause-and-retry path. Never let frame
// alignment walk a healthy slice into that.
if (frameAligned >= 2) {
chunkChars = frameAligned
}
}
if (
chunkChars > 0 &&
chunkChars < pending.data.length &&
@@ -1,3 +1,4 @@
import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../shared/terminal-mode-reset-profiles'
import { ensurePtyDispatcher, getEagerPtyBufferHandle } from './pty-dispatcher'
import {
hasTerminalDisplayContent,
@@ -53,7 +54,10 @@ function replayEagerPtyBuffer(options: PtyAttachOptions, context: IpcPtyAttachCo
const shouldClearBeforeReplay =
!options.isAlternateScreen && hasTerminalDisplayContent(replayData)
if (shouldClearBeforeReplay && !options.callbacks.onReplayData) {
options.callbacks.onData?.('\x1b[2J\x1b[3J\x1b[H')
// RELEASE_SYNCHRONIZED_OUTPUT: trimIncompleteTerminalControlTail can have cut a
// half-written \x1b[?2026l off the replayed payload while its opening \x1b[?2026h
// survives, and \x1b[2J does not clear the mode.
options.callbacks.onData?.(`${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H`)
}
context.setSuppressAttentionEvents(true)
@@ -2,7 +2,8 @@ import type * as React from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
POST_REPLAY_MODE_RESET,
RESET_GRAPHIC_RENDITION
RESET_GRAPHIC_RENDITION,
RELEASE_SYNCHRONIZED_OUTPUT
} from '../../../../shared/terminal-mode-reset-profiles'
import { Terminal } from '@xterm/headless'
import { buildFreshShellViewportBlankingSequence } from './terminal-restored-viewport'
@@ -296,7 +297,7 @@ describe('connectPanePty', () => {
const recoveredRows = 3
const coldScrollback = '\x1b[1;1HCOLD\x1b[1;15HEND\r\nCOLD_SOURCE_ROW_02'
const groundedColdScrollback = `${RESET_GRAPHIC_RENDITION}${coldScrollback}`
const viewportClear = `${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H`
const viewportClear = `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H`
transport.connect.mockImplementation(async ({ sessionId }: { sessionId?: string }) => {
if (sessionId) {
return {
@@ -4,7 +4,8 @@ import {
POST_REPLAY_DEAD_TUI_RESET,
POST_REPLAY_MODE_RESET,
POST_REPLAY_REATTACH_RESET,
RESET_GRAPHIC_RENDITION
RESET_GRAPHIC_RENDITION,
RELEASE_SYNCHRONIZED_OUTPUT
} from '../../../../shared/terminal-mode-reset-profiles'
import { replayEpilogue } from './pty-connection-test-replay-epilogue'
import { Terminal } from '@xterm/headless'
@@ -185,7 +186,7 @@ describe('connectPanePty', () => {
await flushAsyncTicks(20)
expect(pane.terminal.write).toHaveBeenCalledWith(
`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`,
`${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`,
expect.any(Function)
)
expect(pane.terminal.write).toHaveBeenCalledWith(
@@ -769,7 +770,7 @@ describe('connectPanePty', () => {
await flushAsyncTicks(8)
replayCallback.current?.('blocking replay')
await flushAsyncTicks(12)
expect(writes).toEqual(['\x1b[2J\x1b[3J\x1b[H'])
expect(writes).toEqual(['\x1b[?2026l\x1b[2J\x1b[3J\x1b[H'])
reattachResult.resolve({ id: 'tab-pty', snapshot: 'stale authoritative snapshot' })
await flushAsyncTicks(12)
const resizeCallsBeforeReplacement = transport.resize.mock.calls.length
@@ -2,7 +2,8 @@ import type * as React from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import {
POST_REPLAY_REATTACH_RESET,
RESET_GRAPHIC_RENDITION
RESET_GRAPHIC_RENDITION,
RELEASE_SYNCHRONIZED_OUTPUT
} from '../../../../shared/terminal-mode-reset-profiles'
import { replayEpilogue } from './pty-connection-test-replay-epilogue'
import { toAppSshPtyId } from '../../../../shared/ssh-pty-id'
@@ -208,7 +209,9 @@ describe('connectPanePty', () => {
expect(deps.syncPanePtyLayoutBinding).toHaveBeenCalledWith(1, 'leaf-session')
expect(deps.updateTabPtyId).toHaveBeenCalledWith('tab-1', 'leaf-session')
// Why: the relay's replay buffer holds full history, so clear xterm before writing to avoid duplicating prior-session content.
expect(writes).toContain(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`)
expect(writes).toContain(
`${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`
)
expect(writes).toContain('restored-ssh-output')
expect(writes).toContain(replayEpilogue(POST_REPLAY_REATTACH_RESET))
expect(api.pty.signal).toHaveBeenCalledWith('leaf-session', 'SIGWINCH')
@@ -619,7 +619,7 @@ describe('connectPanePty', () => {
expect(pane.terminal.write).toHaveBeenCalledTimes(1)
expect(pane.terminal.write).toHaveBeenNthCalledWith(
1,
'\x1b[2J\x1b[3J\x1b[H',
'\x1b[?2026l\x1b[2J\x1b[3J\x1b[H',
expect.any(Function)
)
@@ -658,7 +658,7 @@ describe('connectPanePty', () => {
callbacksRef.replay?.('authoritative replay')
await flushAsyncTicks(8)
expect(writes).toEqual(['\x1b[2J\x1b[3J\x1b[H'])
expect(writes).toEqual(['\x1b[?2026l\x1b[2J\x1b[3J\x1b[H'])
const acknowledgeLiveFrame = vi.fn()
deliverTerminalDataWithDeferredCredit(acknowledgeLiveFrame, () => {
@@ -1,7 +1,8 @@
import { waitForTerminalReplayWritesParsed } from '../replay-guard'
import {
POST_REPLAY_MODE_RESET,
RESET_GRAPHIC_RENDITION
RESET_GRAPHIC_RENDITION,
RELEASE_SYNCHRONIZED_OUTPUT
} from '../../../../../shared/terminal-mode-reset-profiles'
import {
buildMainModelSnapshotReplayWrites,
@@ -60,7 +61,9 @@ export function createReattachPayloadHandlers(
session.suppressStructuralReplayPtyResize = false
}
}
session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`)
session.writeReplayData(
`${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`
)
// Why: re-arm the kitty keyboard mirror from the snapshot preamble so Option chords keep their encoding after a window reload.
session.applySnapshotKittyKeyboardModes(daemonSnapshotReplay, {
kittyKeyboardFlags: ctx.connectResult.snapshotKittyKeyboardFlags,
@@ -226,7 +229,9 @@ export function createReattachPayloadHandlers(
fullScreenReplay: true
})
// Relay replay may overlap xterm's pre-disconnect content; clear first to avoid duplication.
session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`)
session.writeReplayData(
`${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[3J\x1b[H`
)
// Why: raw relay replay may contain the app's own kitty pushes; re-arm with set semantics so redelivery can't grow the stack.
// A constructor-fresh mirror (window reload) first demotes to unproven:
// the replay window proves nothing about negotiations that predate it.
@@ -266,7 +271,9 @@ export function createReattachPayloadHandlers(
// The current xterm grid remains a safe lower bound for blanking.
}
// Why: shrinking first would promote clipped stale viewport rows into scrollback, beyond the reach of a later viewport-only clear.
session.writeReplayData(`${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H`)
session.writeReplayData(
`${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x1b[2J\x1b[H`
)
await waitForTerminalReplayWritesParsed(session.pane.terminal)
if (!ctx.isCurrentReattachPayload()) {
return
@@ -1,4 +1,8 @@
import { isDocumentVisibilityProvenStale } from '../stale-document-visibility'
import {
scanSynchronizedOutput,
type SynchronizedOutputScan
} from '../../../../../shared/terminal-synchronized-output-scan'
import {
INACTIVE_FOREGROUND_IMMEDIATE_BUDGET_CHARS,
consumeForegroundImmediateBudget,
@@ -6,9 +10,11 @@ import {
} from './foreground-output-budgets'
export const TERMINAL_RENDERER_RISK_SCAN_TAIL_CHARS = 256
export const SYNCHRONIZED_OUTPUT_START_SEQUENCE = '\x1b[?2026h'
export const SYNCHRONIZED_OUTPUT_END_SEQUENCE = '\x1b[?2026l'
export const SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS = SYNCHRONIZED_OUTPUT_START_SEQUENCE.length - 1
export {
SYNCHRONIZED_OUTPUT_START_SEQUENCE,
SYNCHRONIZED_OUTPUT_END_SEQUENCE,
SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS
} from '../../../../../shared/terminal-synchronized-output-scan'
export const CURSOR_SHOW_SEQUENCE = '\x1b[?25h'
export const CURSOR_HIDE_SEQUENCE = '\x1b[?25l'
export const TERMINAL_FOCUS_IN_SEQUENCE = '\x1b[I'
@@ -38,59 +44,15 @@ export function shouldWritePtyOutputForeground(isPaneVisible: boolean): boolean
return isDocumentVisibilityProvenStale()
}
export type SynchronizedForegroundScan = {
started: boolean
ended: boolean
active: boolean
markerTail: string
}
export type SynchronizedForegroundScan = SynchronizedOutputScan
// Why the carried tail: ConPTY can split \x1b[?2026l across chunks; scanning the raw
// chunk alone left the foreground DEC 2026 latch stuck open so every later chunk was
// held instead of coalesced, freezing the visible pane (#8754). Mirrors the hidden path.
/** Renderer-facing name for the shared latch scan; the logic is protocol, not view. */
export function scanSynchronizedForegroundOutput(
data: string,
markerTail: string,
wasActive: boolean
): SynchronizedForegroundScan {
const scanData = markerTail ? `${markerTail}${data}` : data
const currentChunkStartIndex = scanData.length - data.length
let active = wasActive
let started = false
let ended = false
let startIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_START_SEQUENCE)
let endIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_END_SEQUENCE)
// Each marker search advances independently, so a missing counterpart is scanned only once.
while (startIndex !== -1 || endIndex !== -1) {
if (endIndex !== -1 && (startIndex === -1 || endIndex < startIndex)) {
active = false
if (endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length > currentChunkStartIndex) {
ended = true
}
endIndex = scanData.indexOf(
SYNCHRONIZED_OUTPUT_END_SEQUENCE,
endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length
)
continue
}
active = true
if (startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length > currentChunkStartIndex) {
started = true
}
startIndex = scanData.indexOf(
SYNCHRONIZED_OUTPUT_START_SEQUENCE,
startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length
)
}
return {
started,
ended,
active,
// Why length-1: a full marker can never hide in the tail, so no marker is counted twice.
markerTail: scanData.slice(-SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS)
}
return scanSynchronizedOutput(data, markerTail, wasActive)
}
export function containsCursorPositionSequence(data: string): boolean {
@@ -46,6 +46,12 @@ export function bindLiveDataCallback(session: ConnectPanePtySession): void {
} else {
// Why: main dropped buffered output at the pending cap, so the stream has a gap; repaint from the main-owned snapshot instead of writing on.
session.markHiddenOutputRestoreNeeded()
// Why the synthesized \x1b[?2026l is not written here: this branch discards
// `data` deliberately, and the grounded snapshot replay
// (REPLAY_BASELINE_TERMINAL_RESET) releases the latch instead. Writing it
// through writePtyOutputToXterm perturbs the hidden-output-restore state
// machine (it consumes the pending snapshot), so the release rides the
// restore. Residual gap: a pane whose restore never arrives.
if (data) {
// The sentinel can carry query bytes carved from the bulk drop (extractDroppedPtyQueryBytes in main); replies must still flow.
session.salvageRendererQueriesFromDiscardedRestoreData(data)
@@ -1,3 +1,4 @@
import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../../../shared/terminal-mode-reset-profiles'
import { waitForTerminalOutputParsed } from '@/lib/pane-manager/pane-terminal-output-scheduler'
import { safeFit, safeFitAndThen } from '@/lib/pane-manager/pane-tree-ops'
import { getFitOverrideForPty } from '@/lib/pane-manager/mobile-fit-overrides'
@@ -118,7 +119,11 @@ export function bindReplayDataDrain(session: ConnectPanePtySession): void {
// dropping the scrollback first spares a reflow of history the very next
// sequence discards (see use-terminal-container-fit-sync.ts on its cost).
if (clearBeforeReplay) {
await session.writeReplayDataAsync('\x1b[2J\x1b[3J\x1b[H')
// RELEASE_SYNCHRONIZED_OUTPUT: a reconnect is exactly the event that severs a
// frame mid-flight, so this xterm may hold an open 2026 latch — and \x1b[2J does
// not clear it, so the pane would stay frozen on its last painted frame and the
// whole replay would go unseen until xterm's 1s timeout.
await session.writeReplayDataAsync(`${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H`)
if (!isCurrentPayload()) {
continue
}
@@ -124,7 +124,7 @@ describe('remote transport snapshot source-grid threading', () => {
deliverSnapshot({ cols: 154, rows: 68, seq: 9, source: 'headless' }, 'recovered')
await expect.poll(() => onReplayData.mock.calls.length, { timeout: 5000 }).toBe(2)
expect(onReplayData).toHaveBeenLastCalledWith(
'\x1b[2J\x1b[3J\x1b[Hrecovered',
'\x1b[?2026l\x1b[2J\x1b[3J\x1b[Hrecovered',
expect.objectContaining({ snapshotCols: 154, snapshotRows: 68 })
)
@@ -40,4 +40,23 @@ describe('pane terminal output queue chunks', () => {
expect(second?.data).toBe(allData.slice(-2))
expect(entry.queuedChars).toBe(0)
})
it('does not cut a queued chunk inside an open DEC 2026 frame', () => {
const entry = createEntry()
const open = '\x1b[?2026h'
const close = '\x1b[?2026l'
const firstFrame = `${open}aaaa${close}`
const data = `${firstFrame}${open}bbbbbbbbbbbb${close}`
enqueueChunk(entry, data, { foreground: true })
// A limit landing inside the second frame must stop after the first one:
// xterm paints nothing while the latch is open, so stranding the close in
// the residual freezes the pane until a later drain or its 1000ms timeout.
const taken = takeQueuedChunk(entry, firstFrame.length + 6)
expect(taken?.data).toBe(firstFrame)
// The residual keeps the rest, byte-exact, with accounting still balanced.
const rest = takeQueuedChunk(entry, data.length)
expect(`${taken?.data ?? ''}${rest?.data ?? ''}`).toBe(data)
expect(entry.queuedChars).toBe(0)
})
})
@@ -1,4 +1,5 @@
import { flattenRetainedSlice } from '@/lib/flatten-retained-slice'
import { resolveSynchronizedOutputSafeSplit } from '../../../../shared/terminal-synchronized-output-scan'
import type {
QueueEntry,
QueuedWrite,
@@ -90,7 +91,13 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite |
continue
}
const prefix = chunk.data.slice(0, remaining)
// Why not a blind offset: cutting inside an open DEC 2026 frame strands the
// closing \x1b[?2026l in the residual, and xterm stops repainting — the pane holds
// its last frame — until a later drain delivers it or its 1000ms timeout fires.
// Always >= 1 here: `remaining > 0` gates the loop and the helper never
// returns 0 for a positive limit, so the loop cannot stall.
const splitAt = resolveSynchronizedOutputSafeSplit(chunk.data, remaining)
const prefix = chunk.data.slice(0, splitAt)
if (dataParts) {
dataParts.push(prefix)
} else if (dataLength === 0) {
@@ -100,7 +107,7 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite |
data = ''
}
dataLength += prefix.length
const residual = chunk.data.slice(remaining)
const residual = chunk.data.slice(splitAt)
// Geometric flattening bounds retained parents while keeping total copy work linear.
const flatten = residual.length * 2 <= chunk.retainedChars
entry.chunks[entry.chunkIndex] = {
@@ -108,7 +115,7 @@ export function takeQueuedChunk(entry: QueueEntry, limit: number): QueuedWrite |
data: flatten ? flattenRetainedSlice(residual) : residual,
retainedChars: flatten ? residual.length : chunk.retainedChars
}
entry.queuedChars -= remaining
entry.queuedChars -= prefix.length
remaining = 0
}
@@ -108,11 +108,14 @@ export function getTerminalOutputMaxQueueChars(): number {
return maxQueueChars
}
// Why: leading CAN aborts any partial escape sequence before the style reset so the backlog warning renders cleanly.
// Why ?2026l too: the discarded tail may have held the TUI's closing \x1b[?2026l, and
// xterm stops repainting until the latch closes or its 1000ms timeout fires — the
// pane freezes on its last frame, so the warning itself would go unseen for a second.
export const BACKGROUND_BACKLOG_WARNING =
'\x18\x1b[0m\r\n[Orca skipped hidden terminal output because the backlog grew too large.]\r\n'
'\x18\x1b[?2026l\x1b[0m\r\n[Orca skipped hidden terminal output because the backlog grew too large.]\r\n'
// Why a separate foreground message: a visible pane hitting the cap means the drain couldn't keep up with a flood (starved renderer), not merely output produced while hidden.
export const FOREGROUND_BACKLOG_WARNING =
'\x18\x1b[0m\r\n[Orca skipped a burst of terminal output because the backlog grew too large.]\r\n'
'\x18\x1b[?2026l\x1b[0m\r\n[Orca skipped a burst of terminal output because the backlog grew too large.]\r\n'
export const ALWAYS_REFRESH_FOREGROUND_SYNCHRONOUSLY = (): boolean => true
export const queuedByTerminal = new Map<TerminalOutputTarget, QueueEntry>()
@@ -1,3 +1,4 @@
import { RELEASE_SYNCHRONIZED_OUTPUT } from '../../../shared/terminal-mode-reset-profiles'
import {
TerminalStreamOpcode,
decodeTerminalStreamText,
@@ -105,15 +106,20 @@ export abstract class RemoteRuntimeTerminalBinarySnapshots extends RemoteRuntime
// mid-session; clear the screen and scrollback before applying it.
// An empty snapshot is still applied so stale dropped output does
// not linger on a terminal the model says is blank.
stream.callbacks.onSnapshot(`\x1b[2J\x1b[3J\x1b[H${data ?? ''}`, {
pendingEscapeTailAnsi: info?.pendingEscapeTailAnsi,
seq: info?.seq,
kittyKeyboardFlags: info?.kittyKeyboardFlags,
alternateScreen: info?.alternateScreen,
terminalOwner: info?.terminalOwner,
cols: info?.cols,
rows: info?.rows
})
// RELEASE_SYNCHRONIZED_OUTPUT: \x1b[2J does not clear mode 2026, so a pane
// holding an open latch would not paint this recovery snapshot at all.
stream.callbacks.onSnapshot(
`${RELEASE_SYNCHRONIZED_OUTPUT}\x1b[2J\x1b[3J\x1b[H${data ?? ''}`,
{
pendingEscapeTailAnsi: info?.pendingEscapeTailAnsi,
seq: info?.seq,
kittyKeyboardFlags: info?.kittyKeyboardFlags,
alternateScreen: info?.alternateScreen,
terminalOwner: info?.terminalOwner,
cols: info?.cols,
rows: info?.rows
}
)
}
} else if (matchesPendingRequest) {
pendingRequest.resolve({
@@ -243,7 +243,7 @@ describe('remote terminal frame-drop resync', () => {
expect(data).toEqual(['aaa'])
expect(server.droppedFrames).toBe(1)
// Instead, a fresh authoritative snapshot recovers the terminal.
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
server.replaySnapshotCoveredOutput('ccc')
server.output('ddd')
@@ -270,7 +270,7 @@ describe('remote terminal frame-drop resync', () => {
expect(server.snapshotRequests).toEqual([undefined, undefined])
server.output('eee')
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(data).toEqual(['aaa', 'eee'])
} finally {
vi.useRealTimers()
@@ -298,7 +298,7 @@ describe('remote terminal frame-drop resync', () => {
server.output('eee')
expect(server.snapshotRequests).toEqual([undefined, undefined])
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(data).toEqual(['aaa', 'eee'])
})
@@ -321,7 +321,7 @@ describe('remote terminal frame-drop resync', () => {
await expect(manualSnapshot).rejects.toThrow('stream failed')
expect(server.snapshotRequests).toEqual([expect.any(Number), undefined])
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
server.output('ddd')
expect(data).toEqual(['aaa', 'ddd'])
@@ -342,7 +342,7 @@ describe('remote terminal frame-drop resync', () => {
server.output('eee')
expect(server.snapshotRequests).toEqual([undefined, undefined])
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(data).toEqual(['aaa', 'eee'])
} finally {
vi.useRealTimers()
@@ -452,7 +452,7 @@ describe('remote terminal frame-drop resync', () => {
await Promise.resolve()
expect(data).toEqual([])
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
})
it('uses UTF-16 sequence units when detecting gaps in multibyte output', async () => {
@@ -466,7 +466,7 @@ describe('remote terminal frame-drop resync', () => {
await Promise.resolve()
expect(data).toEqual(['é'])
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
})
it('defers recovery until an in-flight manual snapshot finishes', async () => {
@@ -489,7 +489,7 @@ describe('remote terminal frame-drop resync', () => {
await expect(manualSnapshot).resolves.toMatchObject({ data: 'MANUAL' })
expect(server.snapshotRequests).toHaveLength(2)
expect(snapshots).toEqual(['INITIAL', '\x1b[2J\x1b[3J\x1b[HRECOVERED'])
expect(snapshots).toEqual(['INITIAL', '\x1b[?2026l\x1b[2J\x1b[3J\x1b[HRECOVERED'])
server.output('ddd')
expect(data).toEqual(['aaa', 'ddd'])
@@ -592,7 +592,7 @@ describe('remote runtime terminal multiplex ACK gate', () => {
// Why: an unsolicited recovery snapshot replaces terminal state, so it
// clears screen and scrollback first and must not replay the subscribe
// lifecycle.
expect(onSnapshot).toHaveBeenCalledWith(`\x1b[2J\x1b[3J\x1b[H${'recovered state'}`, {
expect(onSnapshot).toHaveBeenCalledWith(`\x1b[?2026l\x1b[2J\x1b[3J\x1b[H${'recovered state'}`, {
pendingEscapeTailAnsi: undefined,
cols: 120,
rows: 40
@@ -611,7 +611,7 @@ describe('remote runtime terminal multiplex ACK gate', () => {
},
''
)
expect(onSnapshot).toHaveBeenCalledWith('\x1b[2J\x1b[3J\x1b[H', {
expect(onSnapshot).toHaveBeenCalledWith('\x1b[?2026l\x1b[2J\x1b[3J\x1b[H', {
pendingEscapeTailAnsi: undefined,
cols: 120,
rows: 40
@@ -53,7 +53,7 @@ describe('terminal mode reset profiles', () => {
// Why: the one reset for a process boundary (cold-restore seed, proven crash).
it('pins the process boundary ground', () => {
expect(PROCESS_BOUNDARY_GROUND).toBe(
'\x1b[<99u\x1b[=0u\x1b7\x1b[?1049l\x1b[?9l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1006l\x1b[?1016l\x1b[?1005l\x1b[?1015l\x1b[?1004l\x1b[?2004l\x1b[?1l\x1b[?66l\x1b[?25h\x1b[0 q\x1b[<99u\x1b[=0u\x1b[0m\x1b7'
'\x1b[?2026l\x1b[<99u\x1b[=0u\x1b7\x1b[?1049l\x1b[?9l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1006l\x1b[?1016l\x1b[?1005l\x1b[?1015l\x1b[?1004l\x1b[?2004l\x1b[?1l\x1b[?66l\x1b[?25h\x1b[0 q\x1b[<99u\x1b[=0u\x1b[0m\x1b7'
)
})
+20 -3
View File
@@ -58,6 +58,21 @@ export const POST_REPLAY_LIVE_AGENT_SNAPSHOT_RESET = RESET_TERMINAL_CURSOR_STYLE
// writes the clipboard.
export const ABORT_TRUNCATED_CONTROL_STRING = '\x18'
// Trade-off, stated because it is not free: the pane was FROZEN on its last
// coherent frame, not blank (bufferRows records a row range and clears nothing).
// Releasing the latch where no repaint follows in the same write — RESET_AFTER_BYTE_GAP
// is written alone — can flash a partial frame in place of that coherent one. A byte
// gap already means the stream is damaged and a restore follows, so a stale frame that
// outlives the damage is the worse option.
// Why this is grounded everywhere a byte gap or a repaint happens: xterm renders
// NOTHING while DEC 2026 is open and only force-flushes after 1000ms, so a gap
// that swallowed a TUI's closing \x1b[?2026l leaves the pane blank for a full
// second per frame — and Orca is otherwise incapable of closing a latch it
// opened. Unlike the modes deliberately left ungrounded below, a snapshot never
// re-asserts 2026, and closing a frame early costs one premature repaint against
// a second of frozen, increasingly stale output.
export const RELEASE_SYNCHRONIZED_OUTPUT = '\x1b[?2026l'
// Why the DECSC first: xterm's `?1049l` runs restoreCursor() even on the normal
// buffer, so saving in place keeps the cursor put there; on the alt buffer the
// save lands in the alt register and `?1049l` restores the shell's position.
@@ -78,7 +93,9 @@ const SHOW_CURSOR = '\x1b[?25h'
*/
export function buildProcessBoundaryGround(opts: { keepFocusReporting: boolean }): string {
const focus = opts.keepFocusReporting ? '' : RESET_FOCUS_REPORTING
return `${RESET_KITTY_KEYBOARD_PROTOCOL}${LEAVE_ALTERNATE_SCREEN_KEEPING_NORMAL_CURSOR}${RESET_MOUSE_REPORTING}${RESET_LEGACY_MOUSE_ENCODINGS}${focus}${RESET_BRACKETED_PASTE}${RESET_APPLICATION_CURSOR_AND_KEYPAD}${SHOW_CURSOR}${RESET_TERMINAL_CURSOR_STYLE}${RESET_KITTY_KEYBOARD_PROTOCOL}${RESET_GRAPHIC_RENDITION}${SAVE_GROUNDED_CURSOR}`
// RELEASE_SYNCHRONIZED_OUTPUT first: the process that opened a 2026 frame is
// gone, so nothing will ever close it, and xterm stops repainting until it does.
return `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_KITTY_KEYBOARD_PROTOCOL}${LEAVE_ALTERNATE_SCREEN_KEEPING_NORMAL_CURSOR}${RESET_MOUSE_REPORTING}${RESET_LEGACY_MOUSE_ENCODINGS}${focus}${RESET_BRACKETED_PASTE}${RESET_APPLICATION_CURSOR_AND_KEYPAD}${SHOW_CURSOR}${RESET_TERMINAL_CURSOR_STYLE}${RESET_KITTY_KEYBOARD_PROTOCOL}${RESET_GRAPHIC_RENDITION}${SAVE_GROUNDED_CURSOR}`
}
export const PROCESS_BOUNDARY_GROUND = buildProcessBoundaryGround({ keepFocusReporting: false })
@@ -87,7 +104,7 @@ export const PROCESS_BOUNDARY_GROUND = buildProcessBoundaryGround({ keepFocusRep
// queued chunks instead of repainting. Parser + pen only — a live TUI keeps
// writing here and owns its charset and margins. Not DECSTR: xterm's soft reset
// wipes the kitty flags agents negotiate only at startup.
export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RESET_GRAPHIC_RENDITION}`
export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}`
// The baseline a serialized snapshot assumes it lands on: SerializeAddon diffs
// cells against DEFAULT attributes and emits no charset at all.
@@ -100,7 +117,7 @@ export const RESET_AFTER_BYTE_GAP = `${ABORT_TRUNCATED_CONTROL_STRING}${RESET_GR
// resetting is unilateral. `enacs=\E(B\E)0` (screen/tmux/vt100 terminfo)
// designates G1 once at init and then uses bare SO/SI, so grounding G1 would
// render a live app's box drawing as letters.
const REPLAY_BASELINE_TERMINAL_RESET = `${RESET_GRAPHIC_RENDITION}\x0f\x1b(B\x1b[?6l\x1b[?7h\x1b[?45l\x1b[4l`
const REPLAY_BASELINE_TERMINAL_RESET = `${RELEASE_SYNCHRONIZED_OUTPUT}${RESET_GRAPHIC_RENDITION}\x0f\x1b(B\x1b[?6l\x1b[?7h\x1b[?45l\x1b[4l`
// Buffer-scoped: margins live on the xterm buffer, and `?1049` neither carries
// them across nor clears them unless it actually swaps.
@@ -0,0 +1,125 @@
import { describe, expect, it } from 'vitest'
import {
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE,
SYNCHRONIZED_OUTPUT_END_SEQUENCE,
SYNCHRONIZED_OUTPUT_START_SEQUENCE,
advanceDroppedSynchronizedOutputLatch,
resolveSynchronizedOutputSafeSplit,
scanSynchronizedOutput
} from './terminal-synchronized-output-scan'
const OPEN = SYNCHRONIZED_OUTPUT_START_SEQUENCE
const CLOSE = SYNCHRONIZED_OUTPUT_END_SEQUENCE
describe('advanceDroppedSynchronizedOutputLatch', () => {
it('releases the latch when the dropped span ended mid-frame', () => {
// The close was inside the bytes the renderer will never receive, so xterm
// would paint nothing until its 1000ms forced flush.
const result = advanceDroppedSynchronizedOutputLatch(
`${OPEN}rows`,
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
expect(result.state.active).toBe(true)
expect(result.data).toBe(CLOSE)
})
it('emits nothing when the dropped span closed its own frame', () => {
const result = advanceDroppedSynchronizedOutputLatch(
`${OPEN}rows${CLOSE}`,
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
expect(result.state.active).toBe(false)
expect(result.data).toBe('')
})
it('carries an open latch across successive dropped chunks', () => {
const first = advanceDroppedSynchronizedOutputLatch(
`${OPEN}a`,
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
const second = advanceDroppedSynchronizedOutputLatch('b', first.state)
expect(second.state.active).toBe(true)
expect(second.data).toBe(CLOSE)
const third = advanceDroppedSynchronizedOutputLatch(CLOSE, second.state)
expect(third.state.active).toBe(false)
expect(third.data).toBe('')
})
it('stitches a close marker split across dropped chunks', () => {
const head = CLOSE.slice(0, 4)
const tail = CLOSE.slice(4)
const first = advanceDroppedSynchronizedOutputLatch(
`${OPEN}rows${head}`,
INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE
)
expect(first.state.active).toBe(true)
const second = advanceDroppedSynchronizedOutputLatch(tail, first.state)
expect(second.state.active).toBe(false)
expect(second.data).toBe('')
})
})
describe('resolveSynchronizedOutputSafeSplit', () => {
it('returns the whole length when it already fits', () => {
expect(resolveSynchronizedOutputSafeSplit('abc', 16)).toBe(3)
})
it('splits after a completed frame rather than inside the next one', () => {
const data = `${OPEN}aaaa${CLOSE}${OPEN}bbbbbbbbbb${CLOSE}`
const limit = data.indexOf('bbb')
const splitAt = resolveSynchronizedOutputSafeSplit(data, limit)
// Everything delivered must leave the latch closed.
expect(scanSynchronizedOutput(data.slice(0, splitAt), '', false).active).toBe(false)
expect(splitAt).toBe(`${OPEN}aaaa${CLOSE}`.length)
})
it('never severs the close marker itself', () => {
const data = `${OPEN}aaaa${CLOSE}tail`
// Limit lands in the middle of the 8-byte close sequence.
const limit = `${OPEN}aaaa`.length + 4
const splitAt = resolveSynchronizedOutputSafeSplit(data, limit)
expect(splitAt).toBeLessThanOrEqual(limit)
expect(data.slice(0, splitAt).endsWith('\x1b')).toBe(false)
// The remainder must still contain a complete, parseable close.
expect(data.slice(splitAt)).toContain(CLOSE)
})
it('falls back to the limit when one frame is longer than the window', () => {
const data = `${OPEN}${'x'.repeat(100)}${CLOSE}`
expect(resolveSynchronizedOutputSafeSplit(data, 20)).toBe(20)
})
it('degrades to the plain limit when the buffer starts inside a frame', () => {
// Callers do not thread prior latch state, so a remainder that begins inside
// an already-open frame is scanned as if closed. It must never be WORSE than
// the blind offset it replaced: same boundary, byte-exact.
const data = `${'z'.repeat(40)}${CLOSE}${'q'.repeat(40)}`
const limit = 20
const splitAt = resolveSynchronizedOutputSafeSplit(data, limit, '', true)
const naive = resolveSynchronizedOutputSafeSplit(data, limit)
// With the real prior state it can only do better or the same.
expect(splitAt).toBeLessThanOrEqual(limit)
expect(naive).toBeLessThanOrEqual(limit)
expect(data.slice(0, naive) + data.slice(naive)).toBe(data)
})
it('never returns past the limit or breaks byte-exactness across many shapes', () => {
const outputSamples = [
`${OPEN}${'a'.repeat(50)}${CLOSE}`,
`${'a'.repeat(50)}${CLOSE}${'b'.repeat(50)}`,
`${OPEN}${OPEN}${'a'.repeat(30)}${CLOSE}${CLOSE}`,
`${'a'.repeat(30)}\x1b]52;c;SGVsbG8=\x07${'b'.repeat(30)}`,
`${'a'.repeat(30)}\x1bP0;1|payload\x1b\\${'b'.repeat(30)}`,
CLOSE.repeat(10),
`${OPEN.repeat(10)}tail`
]
for (const data of outputSamples) {
for (let limit = 1; limit <= data.length + 3; limit++) {
const splitAt = resolveSynchronizedOutputSafeSplit(data, limit)
expect(splitAt).toBeGreaterThan(0)
expect(splitAt).toBeLessThanOrEqual(Math.min(limit, data.length))
expect(data.slice(0, splitAt) + data.slice(splitAt)).toBe(data)
}
}
})
})
@@ -0,0 +1,164 @@
/**
* DEC mode 2026 (synchronized output) latch tracking.
*
* Why this module is shared: like terminal-mode-reset-profiles, the latch is a
* terminal-protocol contract rather than a renderer concern. xterm stops
* repainting while the latch is open and force-flushes only after a 1000ms
* timeout, so whichever side of the PTY relay last touched a pane's bytes has
* to know whether it left a frame open — main's drop paths included, not just
* the renderer's foreground coalescer.
*/
export const SYNCHRONIZED_OUTPUT_START_SEQUENCE = '\x1b[?2026h'
export const SYNCHRONIZED_OUTPUT_END_SEQUENCE = '\x1b[?2026l'
export const SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS = SYNCHRONIZED_OUTPUT_START_SEQUENCE.length - 1
export type SynchronizedOutputScan = {
/** A start marker landed inside THIS chunk. */
started: boolean
/** An end marker landed inside THIS chunk. */
ended: boolean
/** Latch state after the chunk: true means a frame is still open. */
active: boolean
markerTail: string
}
// Why the carried tail: a PTY relay can split \x1b[?2026l across chunks; scanning the raw
// chunk alone left the foreground DEC 2026 latch stuck open so every later chunk was
// held instead of coalesced, freezing the visible pane (#8754).
export function scanSynchronizedOutput(
data: string,
markerTail: string,
wasActive: boolean
): SynchronizedOutputScan {
const scanData = markerTail ? `${markerTail}${data}` : data
const currentChunkStartIndex = scanData.length - data.length
let active = wasActive
let started = false
let ended = false
let startIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_START_SEQUENCE)
let endIndex = scanData.indexOf(SYNCHRONIZED_OUTPUT_END_SEQUENCE)
// Each marker search advances independently, so a missing counterpart is scanned only once.
while (startIndex !== -1 || endIndex !== -1) {
if (endIndex !== -1 && (startIndex === -1 || endIndex < startIndex)) {
active = false
if (endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length > currentChunkStartIndex) {
ended = true
}
endIndex = scanData.indexOf(
SYNCHRONIZED_OUTPUT_END_SEQUENCE,
endIndex + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length
)
continue
}
active = true
if (startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length > currentChunkStartIndex) {
started = true
}
startIndex = scanData.indexOf(
SYNCHRONIZED_OUTPUT_START_SEQUENCE,
startIndex + SYNCHRONIZED_OUTPUT_START_SEQUENCE.length
)
}
return {
started,
ended,
active,
// Why length-1: a full marker can never hide in the tail, so no marker is counted twice.
markerTail: scanData.slice(-SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS)
}
}
export type SynchronizedOutputLatchState = {
markerTail: string
active: boolean
}
export const INITIAL_SYNCHRONIZED_OUTPUT_LATCH_STATE: SynchronizedOutputLatchState = {
markerTail: '',
active: false
}
/**
* Advances the latch across bytes the renderer will never receive.
*
* Returns the sequence that must ride along with whatever IS delivered so the
* pane is not left mid-frame: a drop that swallowed the closing marker would
* otherwise leave xterm painting nothing until its 1000ms forced flush, once
* per frame, for as long as the condition lasts. Closing a frame early only
* costs one premature repaint; leaving it open costs a full second of blank
* screen, so the asymmetry favours always closing.
*/
export function advanceDroppedSynchronizedOutputLatch(
data: string,
previous: SynchronizedOutputLatchState
): { data: string; state: SynchronizedOutputLatchState } {
const scan = scanSynchronizedOutput(data, previous.markerTail, previous.active)
return {
data: scan.active ? SYNCHRONIZED_OUTPUT_END_SEQUENCE : '',
state: { markerTail: scan.markerTail, active: scan.active }
}
}
/**
* Chooses a split point that does not leave the delivered half inside an open
* DEC 2026 frame.
*
* A blind byte-offset split puts `\x1b[?2026h` in one chunk and its
* `\x1b[?2026l` in the next, so xterm stops repainting until the remainder is
* delivered on a later flush — behind every other pane's output — or until its
* 1000ms forced flush. It can also sever the 8-byte marker itself.
*
* Returns the largest length <= `limit` that ends outside an open frame, or
* `limit` when no such point exists (a frame genuinely longer than the window;
* the latch release still rides along via the reset profiles).
*
* KNOWN LIMITATION: no caller threads `markerTail`/`wasActive`, so a buffer that
* begins INSIDE an already-open frame is scanned as if closed. That degrades to
* the blind offset this replaced — never worse, and byte-exact either way — but
* it means cross-chunk alignment is best-effort. Threading per-PTY latch state
* through the split sites would close it.
*/
export function resolveSynchronizedOutputSafeSplit(
data: string,
limit: number,
markerTail = '',
wasActive = false
): number {
if (data.length <= limit) {
return data.length
}
// Step 1: never hand over a severed marker. If an ESC near the boundary cannot
// have completed by `limit`, cut before it instead.
let candidate = limit
const guardStart = Math.max(0, limit - SYNCHRONIZED_OUTPUT_MARKER_TAIL_CHARS)
const escapeIndex = data.lastIndexOf('\x1b', limit - 1)
if (
escapeIndex >= guardStart &&
limit - escapeIndex < SYNCHRONIZED_OUTPUT_START_SEQUENCE.length
) {
candidate = escapeIndex
}
// Step 2: the delivered half must not end inside an open frame.
if (!scanSynchronizedOutput(data.slice(0, candidate), markerTail, wasActive).active) {
return candidate > 0 ? candidate : limit
}
// Fall back to the last frame close that ENDS at or before the candidate.
const lastClose = data.lastIndexOf(
SYNCHRONIZED_OUTPUT_END_SEQUENCE,
Math.max(0, candidate - SYNCHRONIZED_OUTPUT_END_SEQUENCE.length)
)
if (lastClose === -1) {
// One frame is longer than the window; deliver the window and let the
// reset profiles release the latch.
return candidate > 0 ? candidate : limit
}
const aligned = lastClose + SYNCHRONIZED_OUTPUT_END_SEQUENCE.length
// Why the floor: a caller that cannot refill the shortfall in the same round
// (main's flush re-queues the remainder with eligibleRound = round + 1) would
// lose up to half its per-PTY throughput when frames land just past the
// midpoint. Below the floor, prefer throughput and let the reset profiles
// release the latch.
return aligned * 2 >= limit ? aligned : candidate > 0 ? candidate : limit
}
+185
View File
@@ -0,0 +1,185 @@
/**
* Real git-subprocess churn for the multi-workspace typing bench.
*
* Why this exists: every pre-existing scenario in
* terminal-multi-workspace-typing-latency.spec.ts loads the app with PTY bytes
* only, and PTY bytes turn out to be cheap — 24 hidden panes at 512 KB/s left
* typing at 17 ms p50 because main drops hidden renderer-bound bytes at the
* hidden-delivery gate. What the field reports actually have and the bench did
* not is dozens of repositories being polled by the sidebar/source-control
* pollers, which spawn a git child per worktree. Those pollers are gated on
* document visibility (isWindowVisible), so a headless bench window never
* starts them and the dominant main-thread load is simply absent.
*
* This drives the same work through the same production path — main's
* `git:status` handler and its GitAdmissionScheduler — from the renderer, so it
* runs headless without depending on window visibility or on the sidebar being
* mounted.
*/
import type { Page } from '@stablyai/playwright-test'
import { execFileSync } from 'node:child_process'
import { mkdirSync, writeFileSync } from 'node:fs'
import path from 'node:path'
export type GitChurnRepo = { path: string; index: number }
export type GitChurnStats = {
requested: number
settled: number
failed: number
durationMsTotal: number
durationMsMax: number
repos: number
concurrency: number
/** First failure's message — a churn loop that only throws measures nothing. */
firstError: string | null
}
// Matches the sibling probes (runtime-graph-publication-probe): the handle must
// live on `window` to survive between page.evaluate calls, and `declare global`
// would need an `interface` the lint rules forbid.
type GitChurnWindow = Window & { __orcaGitChurnLoad?: { stop: () => GitChurnStats } }
/**
* Builds `repoCount` real repositories, each with `filesPerRepo` tracked files
* and a dirty working tree, so `git status` does measurable work instead of
* returning instantly on an empty repo.
*/
export function createGitChurnRepos(
rootDirectory: string,
repoCount: number,
filesPerRepo: number
): GitChurnRepo[] {
const repos: GitChurnRepo[] = []
for (let index = 0; index < repoCount; index++) {
const repoPath = path.join(rootDirectory, `churn-repo-${index}`)
mkdirSync(repoPath, { recursive: true })
const git = (...args: string[]): void => {
execFileSync('git', args, { cwd: repoPath, stdio: 'ignore' })
}
git('init', '--quiet')
git('config', 'user.email', 'bench@example.com')
git('config', 'user.name', 'Bench')
// Why nested directories: a flat tree of N files understates the readdir
// and lstat cost that dominates `git status` on a real checkout.
for (let fileIndex = 0; fileIndex < filesPerRepo; fileIndex++) {
const directory = path.join(repoPath, `dir-${fileIndex % 64}`)
mkdirSync(directory, { recursive: true })
writeFileSync(path.join(directory, `file-${fileIndex}.txt`), `content ${fileIndex}\n`)
}
git('add', '-A')
git('commit', '--quiet', '-m', 'seed')
// Why left dirty: a clean tree lets git short-circuit; the reported setups
// all have modified worktrees.
for (let fileIndex = 0; fileIndex < Math.min(filesPerRepo, 64); fileIndex++) {
const directory = path.join(repoPath, `dir-${fileIndex % 64}`)
writeFileSync(path.join(directory, `file-${fileIndex}.txt`), `modified ${fileIndex}\n`)
}
repos.push({ path: repoPath, index })
}
return repos
}
/**
* Registers each repo with Orca. Required: main's `git:status` handler rejects
* any path that is not a known repository or worktree ("Access denied: unknown
* repository or worktree path"), so an unregistered churn loop only measures
* its own rejection path.
*/
export async function registerGitChurnRepos(
page: Page,
repoPaths: string[]
): Promise<{ registered: number; failures: string[] }> {
return page.evaluate(async (paths) => {
const failures: string[] = []
let registered = 0
for (const repoPath of paths) {
try {
// Why inspect the result: repos:add REPORTS failure as { error }, it
// does not throw, so a try/catch alone would count a refusal as success.
const result = await window.api.repos.add({ path: repoPath, kind: 'git' })
if (result && typeof result === 'object' && 'error' in result) {
failures.push(String(result.error).slice(0, 200))
} else {
registered += 1
}
} catch (error) {
failures.push((error instanceof Error ? error.message : String(error)).slice(0, 200))
}
}
return { registered, failures: failures.slice(0, 5) }
}, repoPaths)
}
/**
* Starts `concurrency` renderer-side loops that keep calling `git.status`
* across `repoPaths` until stopped. Each call routes through main's IPC
* handler and admission scheduler exactly as a sidebar poll does.
*/
export async function startGitChurnLoad(
page: Page,
repoPaths: string[],
options: { concurrency: number; admissionTier: 'interactive' | 'status' | 'background' }
): Promise<void> {
await page.evaluate(
({ paths, concurrency, admissionTier }) => {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the probe owns this property on its own renderer window; nothing else reads or writes it.
const target = window as GitChurnWindow
target.__orcaGitChurnLoad?.stop()
let running = true
const stats: GitChurnStats = {
requested: 0,
settled: 0,
failed: 0,
durationMsTotal: 0,
durationMsMax: 0,
repos: paths.length,
concurrency,
firstError: null
}
const runLoop = async (lane: number): Promise<void> => {
let cursor = lane
while (running) {
const worktreePath = paths[cursor % paths.length]
cursor += concurrency
stats.requested += 1
const startedAt = performance.now()
try {
await window.api.git.status({ worktreePath, admissionTier })
} catch (error) {
stats.failed += 1
stats.firstError ??= (error instanceof Error ? error.message : String(error)).slice(
0,
300
)
}
// Why the yield: a loop whose call rejects synchronously would spin the
// renderer thread and measure the spin, not git churn.
await new Promise((resolve) => setTimeout(resolve, 0))
const durationMs = performance.now() - startedAt
stats.settled += 1
stats.durationMsTotal += durationMs
stats.durationMsMax = Math.max(stats.durationMsMax, durationMs)
}
}
for (let lane = 0; lane < concurrency; lane++) {
void runLoop(lane)
}
target.__orcaGitChurnLoad = {
stop: () => {
running = false
return { ...stats }
}
}
},
{ paths: repoPaths, concurrency: options.concurrency, admissionTier: options.admissionTier }
)
}
export async function stopGitChurnLoad(page: Page): Promise<GitChurnStats | null> {
return page.evaluate(
() =>
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: same probe-owned property as startGitChurnLoad installs.
(window as GitChurnWindow).__orcaGitChurnLoad?.stop() ?? null
)
}
@@ -153,6 +153,85 @@ process.stdin.on('data', (chunk) => {
`
}
/**
* Codex-shaped echo probe: same sidecar contract as the plain probe, but each
* keystroke repaint is wrapped in a DEC 2026 synchronized frame and split so the
* closing `\x1b[?2026l` leaves the process in a SEPARATE write from the frame
* body. That is what a real Codex draw looks like on the wire (codex-rs
* tui.rs `stdout().sync_update(...)`, up to 120 FPS), and it is the shape the
* plain probe misses — it emits one unwrapped ~50 byte line per key, so it can
* never leave xterm's synchronized-output latch open.
*
* `frameRows` pads the frame body to a realistic repaint size; `splitDelayMs`
* is the gap between body and close, i.e. how long the latch stays open at the
* source before Orca's delivery adds any of its own.
*/
function codexEchoProbeScript(
runId: string,
arrivalSidecarPath: string,
frameRows: number,
splitDelayMs: number
): string {
return `
import { appendFileSync } from 'node:fs'
process.stdin.setEncoding('utf8')
if (process.stdin.isTTY) process.stdin.setRawMode(true)
process.stdin.resume()
let seq = 0
const interrupt = String.fromCharCode(3)
const rows = ${frameRows}
const splitDelayMs = ${splitDelayMs}
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
const write = (data) =>
new Promise((resolve) => {
if (process.stdout.write(data)) resolve()
else process.stdout.once('drain', resolve)
})
process.stdout.write('${'MWT_TYPING_READY_'}${runId}\\r\\n')
let queue = Promise.resolve()
process.stdin.on('data', (chunk) => {
const atMs = Date.now()
if (chunk.includes(interrupt)) process.exit(0)
for (const char of chunk) {
if (char === '\\r' || char === '\\n') continue
seq += 1
const mySeq = seq
appendFileSync(
${JSON.stringify(arrivalSidecarPath)},
JSON.stringify({ seq: mySeq, atMs, char }) + '\\n'
)
queue = queue.then(async () => {
// Frame body: open the latch, repaint rows, but do NOT close yet.
let body = '\\x1b[?2026h'
for (let row = 1; row <= rows; row++) {
body += '\\x1b[' + row + ';1H\\x1b[2Kcodex frame row ' + row + ' seq ' + mySeq
}
body +=
'\\x1b[' + (rows + 1) + ';1H\\x1b[2Kmwt prompt ' +
mySeq + ': ' + char + ' ${'MWT_KEY_'}${runId}_' + mySeq
await write(body)
if (splitDelayMs > 0) await sleep(splitDelayMs)
await write('\\x1b[?2026l')
})
}
})
`
}
export function writeCodexEchoProbeScript(
scriptPath: string,
runId: string,
arrivalSidecarPath: string,
options: { frameRows: number; splitDelayMs: number }
): void {
mkdirSync(path.dirname(scriptPath), { recursive: true })
writeFileSync(
scriptPath,
codexEchoProbeScript(runId, arrivalSidecarPath, options.frameRows, options.splitDelayMs)
)
}
export function writeSustainedAgentLoadScript(
scriptPath: string,
runId: string,
@@ -0,0 +1,151 @@
/**
* Measures DEC 2026 render blackouts in the focused pane's xterm.
*
* Why buffer polling cannot see this bug: xterm's parser writes into the buffer
* whether or not synchronized output is open, so the existing bench's
* xterm-buffer observation reports a keystroke as "echoed" while the screen is
* still showing the previous frame. What the user sees is the RENDER.
*
* xterm (6.1.0-beta.303) suppresses all row rendering while
* `decPrivateModes.synchronizedOutput` is set — `RenderService.refreshRows`
* returns early into `SyncOutputHandler.bufferRows` — and the only escapes are
* the closing `\x1b[?2026l` or a 1000 ms timeout armed once per buffering
* episode. So a frame whose close is delayed freezes the pane for up to a
* second and then repaints everything at once, which is exactly the reported
* symptom.
*
* This probe records actual `onRender` timestamps and how long the
* synchronized-output latch stays open, so a run can distinguish "echo arrived
* late" from "echo arrived on time but was not painted".
*/
import type { Page } from '@stablyai/playwright-test'
export type SynchronizedOutputBlackoutSnapshot = {
installed: boolean
reason: string
/** Wall time the probe observed, ms. */
observedMs: number
renderCount: number
/** Gaps between consecutive renders, ms. */
maxRenderGapMs: number
p90RenderGapMs: number
/** Episodes where the latch was observed open, ms each. */
latchOpenEpisodes: number[]
maxLatchOpenMs: number
/** Episodes at/over this are xterm's 1s forced flush rather than a real close. */
timeoutScaleEpisodes: number
samples: number
}
type BlackoutProbeTerminal = {
onRender?: (listener: () => void) => { dispose: () => void }
/** Public IModes getter; xterm's internals are minified in the bundle. */
modes?: { synchronizedOutputMode?: boolean }
}
type BlackoutProbePane = { terminal?: BlackoutProbeTerminal }
// Matches the sibling probes (runtime-graph-publication-probe): the handle must
// live on `window` to survive between page.evaluate calls, and `declare global`
// would need an `interface` the lint rules forbid.
type BlackoutProbeWindow = Window & {
__orcaSyncOutputBlackoutProbe?: { stop: () => SynchronizedOutputBlackoutSnapshot }
}
/**
* Installs the probe on the focused pane of `tabId`. Sampling runs on a short
* interval; a renderer thread blocked for N ms shows up as a sampling gap,
* which is reported rather than hidden.
*/
export async function startSynchronizedOutputBlackoutProbe(
page: Page,
tabId: string
): Promise<{ installed: boolean; reason: string }> {
return page.evaluate((activeTabId) => {
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the probe owns this property on its own renderer window; nothing else reads or writes it.
const target = window as BlackoutProbeWindow
target.__orcaSyncOutputBlackoutProbe?.stop()
const manager = window.__paneManagers?.get(activeTabId)
const pane: BlackoutProbePane | null =
manager?.getActivePane?.() ?? manager?.getPanes?.()[0] ?? null
const terminal = pane?.terminal
if (!terminal) {
return { installed: false, reason: 'no focused pane terminal' }
}
if (typeof terminal.modes?.synchronizedOutputMode !== 'boolean') {
// Why fail loudly: a silently absent latch would report zero blackouts
// and read as "not reproduced".
return { installed: false, reason: 'terminal.modes.synchronizedOutputMode not reachable' }
}
if (typeof terminal.onRender !== 'function') {
return { installed: false, reason: 'terminal.onRender unavailable' }
}
const startedAt = performance.now()
const renderAtMs: number[] = []
const latchOpenEpisodes: number[] = []
let samples = 0
let latchOpenedAt: number | null = null
const subscription = terminal.onRender(() => {
renderAtMs.push(performance.now())
})
const sampleLatch = (): void => {
samples += 1
const open = terminal.modes?.synchronizedOutputMode === true
if (open && latchOpenedAt === null) {
latchOpenedAt = performance.now()
} else if (!open && latchOpenedAt !== null) {
latchOpenEpisodes.push(performance.now() - latchOpenedAt)
latchOpenedAt = null
}
}
const timer = window.setInterval(sampleLatch, 4)
target.__orcaSyncOutputBlackoutProbe = {
stop: () => {
window.clearInterval(timer)
subscription.dispose()
sampleLatch()
if (latchOpenedAt !== null) {
latchOpenEpisodes.push(performance.now() - latchOpenedAt)
}
const gaps: number[] = []
for (let index = 1; index < renderAtMs.length; index++) {
gaps.push(renderAtMs[index] - renderAtMs[index - 1])
}
const sortedGaps = [...gaps].sort((a, b) => a - b)
const round = (value: number): number => Number(value.toFixed(1))
return {
installed: true,
reason: 'ok',
observedMs: round(performance.now() - startedAt),
renderCount: renderAtMs.length,
maxRenderGapMs: round(sortedGaps.at(-1) ?? 0),
p90RenderGapMs: round(
sortedGaps[Math.min(sortedGaps.length - 1, Math.floor(0.9 * sortedGaps.length))] ?? 0
),
latchOpenEpisodes: latchOpenEpisodes
.map(round)
.sort((a, b) => b - a)
.slice(0, 20),
maxLatchOpenMs: round(Math.max(0, ...latchOpenEpisodes)),
// 900ms+ cannot be a real close at Codex's 120 FPS draw rate; it is
// xterm's 1000ms forced flush.
timeoutScaleEpisodes: latchOpenEpisodes.filter((episode) => episode >= 900).length,
samples
}
}
}
return { installed: true, reason: 'ok' }
}, tabId)
}
export async function stopSynchronizedOutputBlackoutProbe(
page: Page
): Promise<SynchronizedOutputBlackoutSnapshot | null> {
return page.evaluate(
() =>
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: same probe-owned property as startSynchronizedOutputBlackoutProbe installs.
(window as BlackoutProbeWindow).__orcaSyncOutputBlackoutProbe?.stop() ?? null
)
}
@@ -43,6 +43,7 @@ import {
} from './paced-terminal-typing'
import {
ensureTerminalVisible,
getActiveTabId,
getActiveWorktreeId,
getAllWorktreeIds,
switchToWorktree,
@@ -63,9 +64,22 @@ import {
import {
sustainedLoadReadyFilePath,
typingProbeReadyMarker,
writeCodexEchoProbeScript,
writeSustainedAgentLoadScript,
writeTypingEchoProbeScript
} from './sustained-agent-typing-load-scripts'
import {
startSynchronizedOutputBlackoutProbe,
stopSynchronizedOutputBlackoutProbe,
type SynchronizedOutputBlackoutSnapshot
} from './synchronized-output-blackout-probe'
import {
createGitChurnRepos,
registerGitChurnRepos,
startGitChurnLoad,
stopGitChurnLoad,
type GitChurnStats
} from './git-churn-load'
import {
cleanupAccumulatedWorkspaceFixture,
seedAccumulatedWorkspaceFixture,
@@ -104,6 +118,16 @@ const LOAD_RATE_KBPS = readPositiveInt('ORCA_TYPING_BENCH_RATE_KBPS', 256)
const KEY_COUNT = readPositiveInt('ORCA_TYPING_BENCH_KEYS', 32)
const KEY_CADENCE_MS = readPositiveInt('ORCA_TYPING_BENCH_KEY_CADENCE_MS', 250)
const CPU_WORKERS = readPositiveInt('ORCA_TYPING_BENCH_CPU_WORKERS', 0)
// Git churn stands in for the sidebar/source-control pollers, which spawn a git
// child per worktree and are gated on window visibility — so a headless bench
// window never runs them and the field's dominant main-thread load is missing.
const GIT_CHURN_REPOS = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_REPOS', 0)
const GIT_CHURN_FILES = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_FILES', 2000)
const GIT_CHURN_CONCURRENCY = readPositiveInt('ORCA_TYPING_BENCH_GIT_CHURN_CONCURRENCY', 8)
// Codex-shaped foreground: rows repainted per keystroke, and the gap the probe
// itself leaves between the frame body and its closing \x1b[?2026l.
const CODEX_FRAME_ROWS = readPositiveInt('ORCA_TYPING_BENCH_CODEX_FRAME_ROWS', 20)
const CODEX_SPLIT_DELAY_MS = Number(process.env.ORCA_TYPING_BENCH_CODEX_SPLIT_DELAY_MS ?? 0) || 0
const PTY_METADATA = process.env.ORCA_TYPING_BENCH_PTY_METADATA === '1'
const BENCH_LABEL = process.env.ORCA_TYPING_BENCH_LABEL ?? 'dev'
// Request optional probes by default; the report records when the build does not install them.
@@ -227,7 +251,9 @@ function writeBenchReport(
scaleCensus?: unknown,
accumulatedFixture?: unknown,
ptyWorkload?: unknown,
graphProbe?: RuntimeGraphPublicationProbeSnapshot | null
graphProbe?: RuntimeGraphPublicationProbeSnapshot | null,
gitChurn?: GitChurnStats | null,
blackout?: SynchronizedOutputBlackoutSnapshot | null
): void {
const { measurement, appliedCpuThrottleRate } = measured
const report = {
@@ -285,7 +311,9 @@ function writeBenchReport(
scaleCensus: scaleCensus ?? null,
accumulatedFixture: accumulatedFixture ?? null,
ptyWorkload: ptyWorkload ?? null,
graphProbe: graphProbe ?? null
graphProbe: graphProbe ?? null,
gitChurn: gitChurn ?? null,
synchronizedOutputBlackout: blackout ?? null
}
mkdirSync(RESULTS_DIR, { recursive: true })
const stamp = report.timestamp.replace(/[:.]/g, '-')
@@ -698,4 +726,190 @@ test.describe('Multi-workspace sustained typing latency bench', () => {
removeLoadReadyFiles(testRepoPath, runId, panes.length)
}
})
/**
* The field shape the PTY-only scenarios miss: many repositories being polled
* for git status while the user types. Each poll spawns a git child from
* main, and PTY input and echo are relayed through that same main event loop.
*
* Requires --git-churn-repos; without it the scenario would silently be the
* visible-split scenario again.
*/
test('typing under sustained git-subprocess churn', async ({
orcaPage,
testRepoPath
}, testInfo) => {
test.skip(
GIT_CHURN_REPOS === 0,
'Set --git-churn-repos to run: this scenario measures git-spawn churn, not PTY bytes'
)
await waitForSessionReady(orcaPage)
await waitForActiveWorktree(orcaPage)
await ensureTerminalVisible(orcaPage)
await waitForActiveTerminalManager(orcaPage, 30_000)
const runId = randomUUID()
const loadPath = path.join(testRepoPath, `.orca-mwt-load-${runId}.mjs`)
const probePath = path.join(testRepoPath, `.orca-mwt-probe-${runId}.mjs`)
const sidecarPath = path.join(testRepoPath, `.orca-mwt-arrivals-${runId}.jsonl`)
const churnRoot = mkdtempSync(path.join(tmpdir(), 'orca-git-churn-'))
writeSustainedAgentLoadScript(loadPath, runId, testRepoPath)
writeTypingEchoProbeScript(probePath, runId, sidecarPath)
const cpuWorkers = spawnCpuPressureWorkers()
let panes: TerminalLoadPane[] = []
let gitChurn: GitChurnStats | null = null
try {
const repos = createGitChurnRepos(churnRoot, GIT_CHURN_REPOS, GIT_CHURN_FILES)
const registration = await registerGitChurnRepos(
orcaPage,
repos.map((repo) => repo.path)
)
expect(
registration,
`churn repos were not registered with Orca: ${registration.failures.join('; ')}`
).toMatchObject({ registered: repos.length })
panes = await ensureActiveWorktreePaneLoad(orcaPage, 2)
const [typingPane, ...loadPanes] = panes
await startSustainedLoadInPanes(orcaPage, loadPanes, loadPath, runId, testRepoPath)
await focusPane(orcaPage, typingPane.paneKey)
await resetDeliveryDebug(orcaPage)
await startTypingProbe(orcaPage, typingPane.ptyId, probePath, runId)
await startGitChurnLoad(
orcaPage,
repos.map((repo) => repo.path),
{ concurrency: GIT_CHURN_CONCURRENCY, admissionTier: 'status' }
)
const measured = await measureTypingWindow(orcaPage, runId, sidecarPath)
gitChurn = await stopGitChurnLoad(orcaPage)
const { measurement } = measured
writeBenchReport(
testInfo,
`git-churn-${GIT_CHURN_REPOS}repos-x${GIT_CHURN_CONCURRENCY}`,
measured,
await readSchedulerDebug(orcaPage),
await readMainDeliveryDebug(orcaPage),
undefined,
null,
null,
null,
undefined,
undefined,
undefined,
null,
gitChurn
)
// The churn must have actually spawned git, or a fast result means nothing:
// a loop that only rejects measures the rejection path, not subprocess churn.
expect(gitChurn?.settled ?? 0).toBeGreaterThan(0)
expect(
gitChurn,
`git churn never reached git: ${gitChurn?.firstError ?? 'unknown error'}`
).toMatchObject({ failed: 0 })
expect(measurement.inputHalfMs?.count).toBe(KEY_COUNT)
expect(measurement.totalMs?.count).toBe(KEY_COUNT)
} finally {
await stopGitChurnLoad(orcaPage).catch(() => null)
for (const worker of cpuWorkers) {
worker.kill('SIGKILL')
}
await stopPtysQuietly(
orcaPage,
panes.map((pane) => pane.ptyId)
)
rmSync(loadPath, { force: true })
rmSync(probePath, { force: true })
rmSync(sidecarPath, { force: true })
rmSync(churnRoot, { recursive: true, force: true })
removeLoadReadyFiles(testRepoPath, runId, panes.length)
}
})
/**
* The shape that actually matches the report: a Codex-like foreground TUI that
* wraps every keystroke repaint in a DEC 2026 synchronized frame, under
* background agent load, measured at the RENDER rather than at the xterm
* buffer.
*
* xterm suppresses all row rendering while the synchronized-output latch is
* open and force-flushes on a 1000 ms timeout, so a delayed closing
* `\x1b[?2026l` freezes the pane for up to a second and then repaints in one
* burst. Buffer-observation scenarios score that as a fast echo.
*/
test('typing a DEC 2026 foreground TUI under agent load, measured at the render', async ({
orcaPage,
testRepoPath
}, testInfo) => {
await waitForSessionReady(orcaPage)
await waitForActiveWorktree(orcaPage)
await ensureTerminalVisible(orcaPage)
await waitForActiveTerminalManager(orcaPage, 30_000)
const runId = randomUUID()
const loadPath = path.join(testRepoPath, `.orca-mwt-load-${runId}.mjs`)
const probePath = path.join(testRepoPath, `.orca-mwt-probe-${runId}.mjs`)
const sidecarPath = path.join(testRepoPath, `.orca-mwt-arrivals-${runId}.jsonl`)
writeSustainedAgentLoadScript(loadPath, runId, testRepoPath)
writeCodexEchoProbeScript(probePath, runId, sidecarPath, {
frameRows: CODEX_FRAME_ROWS,
splitDelayMs: CODEX_SPLIT_DELAY_MS
})
const cpuWorkers = spawnCpuPressureWorkers()
let panes: TerminalLoadPane[] = []
let blackout: SynchronizedOutputBlackoutSnapshot | null = null
try {
panes = await ensureActiveWorktreePaneLoad(orcaPage, Math.max(2, LOAD_PANES + 1))
const [typingPane, ...loadPanes] = panes
await startSustainedLoadInPanes(orcaPage, loadPanes, loadPath, runId, testRepoPath)
await focusPane(orcaPage, typingPane.paneKey)
await resetDeliveryDebug(orcaPage)
await startTypingProbe(orcaPage, typingPane.ptyId, probePath, runId)
const activeTabId = await getActiveTabId(orcaPage)
const probeInstall = await startSynchronizedOutputBlackoutProbe(orcaPage, activeTabId ?? '')
// A probe that did not install would report zero blackouts, which reads
// identically to "no bug".
expect(probeInstall, `blackout probe not installed: ${probeInstall.reason}`).toMatchObject({
installed: true
})
const measured = await measureTypingWindow(orcaPage, runId, sidecarPath)
blackout = await stopSynchronizedOutputBlackoutProbe(orcaPage)
const { measurement } = measured
writeBenchReport(
testInfo,
`dec2026-render-${LOAD_PANES}panes-${LOAD_RATE_KBPS}kbps-rows${CODEX_FRAME_ROWS}`,
measured,
await readSchedulerDebug(orcaPage),
await readMainDeliveryDebug(orcaPage),
undefined,
null,
null,
null,
await readTypingScaleCensus(orcaPage).catch(() => null),
undefined,
undefined,
null,
null,
blackout
)
expect(measurement.totalMs?.count).toBe(KEY_COUNT)
expect(blackout?.renderCount ?? 0).toBeGreaterThan(0)
} finally {
await stopSynchronizedOutputBlackoutProbe(orcaPage).catch(() => null)
for (const worker of cpuWorkers) {
worker.kill('SIGKILL')
}
await stopPtysQuietly(
orcaPage,
panes.map((pane) => pane.ptyId)
)
rmSync(loadPath, { force: true })
rmSync(probePath, { force: true })
rmSync(sidecarPath, { force: true })
removeLoadReadyFiles(testRepoPath, runId, panes.length)
}
})
})