diff --git a/notes/orca-performance-branch-guide.md b/notes/orca-performance-branch-guide.md index 50c243614b4..ca60dc98b4c 100644 --- a/notes/orca-performance-branch-guide.md +++ b/notes/orca-performance-branch-guide.md @@ -3,7 +3,7 @@ Agent-facing map of every optimization on this branch: what it does, why it exists, where it lives, and the invariants you must not break when adding to it. The chronological evidence trail (benchmarks, retractions, A/B protocols) is in -`notes/terminal-performance-initiative.md`; this doc is the *current-state* view. +`notes/terminal-performance-initiative.md`; this doc is the _current-state_ view. **Context**: Orca's terminal was ~300× slower than Terminal.app under agent load (DSR-under-load p50 134ms, p99 292ms on v1.4.91; agent-TUI throughput 2.0 MB/s). @@ -21,13 +21,14 @@ shell → pty → daemon (persistence, headless model) → unix socket Main is on the hot path for every byte (unlike VS Code's ptyHost→renderer MessagePort). The daemon owns sessions so they survive app restarts; it also runs -a headless xterm emulator per pty — the *model* — which is the source of truth -for screen contents. The renderer terminal is a *view* that can be discarded and +a headless xterm emulator per pty — the _model_ — which is the source of truth +for screen contents. The renderer terminal is a _view_ that can be discarded and rebuilt from model snapshots. ## Optimization inventory ### 1. Renderer parse-path fixes (the original 16× on agent TUIs) + - **Parse-clocked scheduler drains** (`pane-terminal-output-scheduler.ts`): drain cadence follows xterm's actual parse completion instead of fixed timers, so the queue never outruns the parser. @@ -40,8 +41,10 @@ rebuilt from model snapshots. per-chunk cost. ### 2. term-speed-2 chain (model/view contract — the architecture) + Revived from ~38 never-merged branches; kill-switched, default ON. Docs: `docs/reference/terminal-model-view-contract.md`. + - **Hidden view parking**: hidden tabs tear down their xterm view entirely (memory: parked panes cost ~0). - **Hidden delivery gate** (main): renderer-bound bytes for hidden ptys are @@ -55,6 +58,7 @@ Revived from ~38 never-merged branches; kill-switched, default ON. Docs: reconciliation drops duplicates already covered by the snapshot baseline. ### 3. Batching & scheduling cadence + - **Batch windows 8ms → 2ms** in both `daemon-stream-data-batcher.ts` (`STREAM_DATA_BATCH_INTERVAL_MS`) and `ipc/pty.ts` (`PTY_BATCH_INTERVAL_MS`). At 9% utilization there is no queue — latency was literally the sum of fixed @@ -67,9 +71,11 @@ Revived from ~38 never-merged branches; kill-switched, default ON. Docs: instead of queuing macrotask-per-keystroke. ### 4. Backpressure (the correctness spine — read before touching delivery) + Three cooperating layers, innermost first: + - **ACK at parse-drain** (`deliverPtyDataWithDeferredAck`, scheduler - `ackCredit`): the renderer credits a chunk when xterm has *parsed* it (or the + `ackCredit`): the renderer credits a chunk when xterm has _parsed_ it (or the chunk is legitimately discarded), not when IPC delivered it. **INVARIANT: every delivered chunk credits exactly once — parsed or discarded.** Every scheduler/pty-connection discard path (backlog replacement, @@ -92,15 +98,32 @@ Three cooperating layers, innermost first: push listeners + pull restore markers through the modelRestoreNeeded router. E2e blackhole harness: `__terminalDeliveryWatchdog`, `terminal-push-delivery-loss-recovery.spec.ts`. +- **Stale-visibility proof for the hidden gate** (`stale-document-visibility.ts` + and the `shouldWritePtyOutputForeground` fallthrough in pty-connection.ts): + recovers the field-confirmed wedge where macOS occlusion tracking pins + `document.visibilityState` at `'hidden'` after display sleep and never fires + another visibilitychange (v1.4.124-rc.2.perf snapshot: 78MB hidden-gate + dropped across 2 pane-level-visible ptys, transport healthy). Real user + input (keydown/pointerdown/window focus) while the document claims hidden is + a physical contradiction — it latches an override, runs each pane's existing + visibilitychange resync (gate unhide + hidden-output restore), and a genuine + visibilitychange hands authority back. No timers; recovery is purely + event-proven, and the failure bias is safe (a wrong override only restores + pre-gate delivery cost, never drops bytes). Hot-path cost: zero when + visible (same single comparison); one property read per user-interaction + event. E2e: `terminal-stuck-occlusion-recovery.spec.ts` (pins both the + freeze repro and the keystroke recovery, plus the + `hiddenDeliveryGatedVisiblePtyCount` field discriminator). - **Producer flow control** (protocol v19 `pausePty`/`resumePty`, 256KB pause / 32KB resume watermarks, keyed off **pendingData only** — never renderer counters; kill switch `PRODUCER_FLOW_CONTROL_ENABLED`, ipc/pty.ts): when main's - buffer grows, the *shell* blocks. For main-hosted ptys pause is synchronous + buffer grows, the _shell_ blocks. For main-hosted ptys pause is synchronous (drops impossible); for daemon ptys the pause notify has ~20-30ms socket latency, so wire-speed bursts can still cross the 2MB cap (known follow-up: daemon-side self-pacing watermark). ### 5. Flood resilience (why bulk output can't wedge or lie anymore) + - **Restore-loop cut** (pty-connection.ts): the hidden-output-restore loop abandons immediately when a foreground pane's live-chunk queue overflows (3-iteration hard cap), and a 2s flood-suppression window stops main's own @@ -110,20 +133,23 @@ Three cooperating layers, innermost first: drop re-arms restore) that caused multi-second renderer stalls. - **Query survival**: if the 2MB cap ever drops bulk output, embedded terminal queries are extracted (`terminal-reply-query-extraction.ts`) and answered by - *synthesizing replies on the input path* (CPR from live buffer, DA1 canned, + _synthesizing replies on the input path_ (CPR from live buffer, DA1 canned, OSC via direct responder) — probes and TUIs never hang on a dropped reply. - Drops are downstream of the model: the daemon ingests every byte, so the post-flood repaint restores complete, correct content. ### 6. Wake/sleep recovery + - powerMonitor resume → `system:resumed` IPC → renderer wake recovery (fixes WebGL-latch blank-after-sleep that DOM focus/visibilitychange missed). - Cumulative ACKs make the historical "lost ACKs across suspend pin the global window forever" wedge (BMW user bug) structurally impossible. ### 7. Snapshot fidelity (the garble fixes — all fuzz-pinned) + Reveal-from-snapshot multiplied exposure of serializer defects ~1000×. Five bugs found by differential fuzzing; four fixed, one tolerated: + - **B: SGR intensity ordering** — upstream `@xterm/addon-serialize` emitted `1;22` (22 clears the bold 1 just set). Patched via pnpm patch (`config/patches/@xterm__addon-serialize@*.patch`): clear-before-set for the @@ -143,6 +169,7 @@ found by differential fuzzing; four fixed, one tolerated: `bufferHasSerializeHostileWrappedRow`, the only remaining tolerance. ## Correctness infrastructure (run these before merging delivery/restore changes) + - `headless-emulator-fidelity.fuzz.test.ts` — differential: HeadlessEmulator vs reference xterm, seeded TUI streams. `FUZZ_ITERATIONS=2000` for deep, `FUZZ_SEED=n` to replay. @@ -158,8 +185,9 @@ found by differential fuzzing; four fixed, one tolerated: (pane-manager / terminal-pane suites). ## Benchmarking protocol (hard-won rules) + - Rig: `tools/benchmarks/terminal-pipeline-bench.mjs` — DSR idle + DSR under - 1MB/s agent-TUI load + DSR-fenced throughput on 4 fixtures. Run *inside* the + 1MB/s agent-TUI load + DSR-fenced throughput on 4 fixtures. Run _inside_ the terminal under test. - **Bench at 10MB** (`--size-mb 10`). The ACK-at-parse bug shipped because dev benches used 3MB and never tripped the cap. @@ -170,8 +198,9 @@ found by differential fuzzing; four fixed, one tolerated: - Packaged builds are truth; dev has ~2× overhead. ## Release mechanics + - Perf RCs: `release-cut.yml` workflow_dispatch, `kind=rc ref=orca-performance - version_suffix=perf` → tags like `v1.4.122-rc.1.perf`. The suffix sorts above +version_suffix=perf` → tags like `v1.4.122-rc.1.perf`. The suffix sorts above its base rc.N but below rc.N+1 (never hijacks the RC channel). The rc counter (`release-rc-history.mjs`), telemetry identity classifier, and build guard are all suffix-aware — a suffixed rc classifies as `rc`. @@ -205,6 +234,7 @@ Conflict pattern, established over ~6 syncs: before the next RC cut. ## Known limits / next levers (in rough priority order) + 1. Daemon self-pacing: daemon-hosted ptys can cross the 2MB cap for ~20-30ms at wire speed before `pausePty` bites. Fix: daemon enforces its own watermark locally (VS Code does this server-side for remotes). @@ -218,12 +248,13 @@ Conflict pattern, established over ~6 syncs: is open against main separately. ## Guardrails for future agents + - The chunk-credit invariant (§4) is the load-bearing one. If you add ANY path that receives, defers, drops, or splits pty data in the renderer, prove it credits exactly once. The credit-invariant unit tests are the gate. - Flow control keys off `pendingData` only. Do not couple it to renderer counters; the two layers compose because they are independent. -- Snapshot changes must keep seq semantics: a snapshot covers *exactly* bytes +- Snapshot changes must keep seq semantics: a snapshot covers _exactly_ bytes ≤ its seq (Bug E made this true; don't regress it). Chunks after restore are reconciled by seq — off-by-N re-triggers duplicate-drop garble. - Hidden panes must receive nothing (delivery gate) but side-effects and query diff --git a/src/renderer/src/components/terminal-pane/pty-connection.test.ts b/src/renderer/src/components/terminal-pane/pty-connection.test.ts index feab3833ae8..85650ad5746 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection.test.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection.test.ts @@ -8652,12 +8652,21 @@ describe('connectPanePty', () => { }) it('does not apply stale background Codex query chunks after hidden snapshot restore', async () => { - const visibilityChangeHandler: { current: (() => void) | null } = { current: null } + // Fire-all like a real event target: the pane resync handler and the + // stale-visibility trust handler both listen for visibilitychange. + const visibilityChangeListeners: (() => void)[] = [] + const visibilityChangeHandler = { + current: (): void => { + for (const listener of visibilityChangeListeners) { + listener() + } + } + } ;(globalThis as { document?: Document }).document = { visibilityState: 'visible', addEventListener: vi.fn((type: string, listener: EventListenerOrEventListenerObject) => { if (type === 'visibilitychange') { - visibilityChangeHandler.current = listener as () => void + visibilityChangeListeners.push(listener as () => void) } }), removeEventListener: vi.fn() @@ -9959,14 +9968,23 @@ describe('connectPanePty', () => { it('keeps recovery pending when hidden output arrives during an in-flight snapshot', async () => { let visibilityState: DocumentVisibilityState = 'visible' - const visibilityChangeHandler: { current: (() => void) | null } = { current: null } + // Fire-all like a real event target: the pane resync handler and the + // stale-visibility trust handler both listen for visibilitychange. + const visibilityChangeListeners: (() => void)[] = [] + const visibilityChangeHandler = { + current: (): void => { + for (const listener of visibilityChangeListeners) { + listener() + } + } + } ;(globalThis as { document?: Document }).document = { get visibilityState() { return visibilityState }, addEventListener: vi.fn((type: string, listener: EventListenerOrEventListenerObject) => { if (type === 'visibilitychange') { - visibilityChangeHandler.current = listener as () => void + visibilityChangeListeners.push(listener as () => void) } }), removeEventListener: vi.fn() diff --git a/src/renderer/src/components/terminal-pane/pty-connection.ts b/src/renderer/src/components/terminal-pane/pty-connection.ts index c93e9b1df5e..f97f7a26310 100644 --- a/src/renderer/src/components/terminal-pane/pty-connection.ts +++ b/src/renderer/src/components/terminal-pane/pty-connection.ts @@ -49,6 +49,10 @@ import { isPtyLocked } from '@/lib/pane-manager/mobile-driver-state' import { reconcilePtySizeAcrossFrames, type PtySizeReconcileHandle } from './pty-size-reconcile' import { createPtySizeReassertion } from './pty-size-reassertion' import { isPaneReplaying, replayIntoTerminal, replayIntoTerminalAsync } from './replay-guard' +import { + isDocumentVisibilityProvenStale, + registerStaleDocumentVisibilityRecovery +} from './stale-document-visibility' import { nativeWindowsRewriteNeedsFollowupRenderRefresh, terminalOutputPrefersRenderRefresh, @@ -766,7 +770,13 @@ function shouldWritePtyOutputForeground(isPaneVisible: boolean): boolean { // Why: Electron can keep visible panes mounted while the whole app is // backgrounded. Treat hidden documents like background tabs so Chromium // timer throttling cannot pin terminal writes on the renderer foreground path. - return document.visibilityState === 'visible' + if (document.visibilityState === 'visible') { + return true + } + // Why: macOS occlusion tracking can wedge visibilityState at 'hidden' after + // display sleep; proven-stale means real user input contradicted it, so the + // hidden-delivery gate must not keep dropping a watched pane's bytes. + return isDocumentVisibilityProvenStale() } function containsSynchronizedOutputStart(data: string): boolean { @@ -5582,8 +5592,17 @@ export function connectPanePty( } } document.addEventListener('visibilitychange', onDocumentVisibilityChange) - unregisterDocumentVisibilityRecovery = () => + // Why: when user input proves visibilityState is wedged at 'hidden' + // (stale macOS occlusion), run the same resync — no visibilitychange + // will ever fire in that state, and the gate would drop watched bytes + // forever. + const unregisterStaleVisibilityRecovery = registerStaleDocumentVisibilityRecovery( + onDocumentVisibilityChange + ) + unregisterDocumentVisibilityRecovery = () => { document.removeEventListener('visibilitychange', onDocumentVisibilityChange) + unregisterStaleVisibilityRecovery() + } } const dataCallback = (data: string, meta?: PtyDataMeta): void => { diff --git a/src/renderer/src/components/terminal-pane/stale-document-visibility.test.ts b/src/renderer/src/components/terminal-pane/stale-document-visibility.test.ts new file mode 100644 index 00000000000..7007995bd0d --- /dev/null +++ b/src/renderer/src/components/terminal-pane/stale-document-visibility.test.ts @@ -0,0 +1,157 @@ +// Why: pins the stale-occlusion proof — user input while +// document.visibilityState claims 'hidden' must latch the override and run +// each pane's recovery exactly once, and a genuine visibilitychange must hand +// authority back to the occlusion tracker. +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import type * as StaleDocumentVisibility from './stale-document-visibility' + +type Handler = () => void + +type EventTargetStub = { + addEventListener: (type: string, handler: Handler, options?: unknown) => void + removeEventListener: (type: string, handler: Handler, options?: unknown) => void + emit: (type: string) => void + listenerCount: (type: string) => number +} + +function createEventTargetStub(): EventTargetStub { + const listeners = new Map>() + return { + addEventListener(type, handler) { + const set = listeners.get(type) ?? new Set() + set.add(handler) + listeners.set(type, set) + }, + removeEventListener(type, handler) { + listeners.get(type)?.delete(handler) + }, + emit(type) { + for (const handler of listeners.get(type) ?? []) { + handler() + } + }, + listenerCount(type) { + return listeners.get(type)?.size ?? 0 + } + } +} + +type DocumentStub = EventTargetStub & { visibilityState: string } + +describe('stale document visibility', () => { + const originalDocument = (globalThis as { document?: unknown }).document + const originalWindow = (globalThis as { window?: unknown }).window + let documentStub: DocumentStub + let windowStub: EventTargetStub + let warnSpy: ReturnType + let mod: typeof StaleDocumentVisibility + + beforeEach(async () => { + vi.resetModules() + documentStub = { ...createEventTargetStub(), visibilityState: 'visible' } + windowStub = createEventTargetStub() + ;(globalThis as { document: unknown }).document = documentStub + ;(globalThis as { window: unknown }).window = windowStub + warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {}) + mod = await import('./stale-document-visibility') + }) + + afterEach(() => { + mod.resetStaleDocumentVisibilityForTesting() + warnSpy.mockRestore() + ;(globalThis as { document: unknown }).document = originalDocument + ;(globalThis as { window: unknown }).window = originalWindow + }) + + it('latches the override and runs recovery once when input arrives while hidden', () => { + const recovery = vi.fn() + mod.registerStaleDocumentVisibilityRecovery(recovery) + documentStub.visibilityState = 'hidden' + + documentStub.emit('keydown') + expect(mod.isDocumentVisibilityProvenStale()).toBe(true) + expect(recovery).toHaveBeenCalledTimes(1) + expect(warnSpy).toHaveBeenCalledTimes(1) + + // Further input during the same stuck episode must not re-run recovery. + documentStub.emit('keydown') + documentStub.emit('pointerdown') + expect(recovery).toHaveBeenCalledTimes(1) + }) + + it('does not latch while the document is genuinely visible', () => { + const recovery = vi.fn() + mod.registerStaleDocumentVisibilityRecovery(recovery) + + documentStub.emit('keydown') + documentStub.emit('pointerdown') + windowStub.emit('focus') + expect(mod.isDocumentVisibilityProvenStale()).toBe(false) + expect(recovery).not.toHaveBeenCalled() + }) + + it('treats pointerdown and window focus as staleness proof too', () => { + const recovery = vi.fn() + mod.registerStaleDocumentVisibilityRecovery(recovery) + documentStub.visibilityState = 'hidden' + + documentStub.emit('pointerdown') + expect(mod.isDocumentVisibilityProvenStale()).toBe(true) + + mod.resetStaleDocumentVisibilityForTesting() + mod.registerStaleDocumentVisibilityRecovery(recovery) + documentStub.visibilityState = 'hidden' + windowStub.emit('focus') + expect(mod.isDocumentVisibilityProvenStale()).toBe(true) + }) + + it('hands authority back to the occlusion tracker on a genuine visibilitychange', () => { + const recovery = vi.fn() + mod.registerStaleDocumentVisibilityRecovery(recovery) + documentStub.visibilityState = 'hidden' + documentStub.emit('keydown') + expect(mod.isDocumentVisibilityProvenStale()).toBe(true) + + documentStub.visibilityState = 'visible' + documentStub.emit('visibilitychange') + expect(mod.isDocumentVisibilityProvenStale()).toBe(false) + + // A fresh wedge after trust was restored must latch (and recover) again. + documentStub.visibilityState = 'hidden' + documentStub.emit('keydown') + expect(mod.isDocumentVisibilityProvenStale()).toBe(true) + expect(recovery).toHaveBeenCalledTimes(2) + }) + + it('keeps running other panes when one recovery listener throws', () => { + const throwing = vi.fn(() => { + throw new Error('pane exploded') + }) + const healthy = vi.fn() + mod.registerStaleDocumentVisibilityRecovery(throwing) + mod.registerStaleDocumentVisibilityRecovery(healthy) + documentStub.visibilityState = 'hidden' + + documentStub.emit('keydown') + expect(throwing).toHaveBeenCalledTimes(1) + expect(healthy).toHaveBeenCalledTimes(1) + }) + + it('removes global listeners when the last pane unregisters', () => { + const unregisterA = mod.registerStaleDocumentVisibilityRecovery(vi.fn()) + const unregisterB = mod.registerStaleDocumentVisibilityRecovery(vi.fn()) + expect(documentStub.listenerCount('keydown')).toBe(1) + + unregisterA() + expect(documentStub.listenerCount('keydown')).toBe(1) + unregisterB() + expect(documentStub.listenerCount('keydown')).toBe(0) + expect(documentStub.listenerCount('pointerdown')).toBe(0) + expect(documentStub.listenerCount('visibilitychange')).toBe(0) + expect(windowStub.listenerCount('focus')).toBe(0) + + // Re-registering must reinstall for a later pane. + mod.registerStaleDocumentVisibilityRecovery(vi.fn()) + expect(documentStub.listenerCount('keydown')).toBe(1) + }) +}) diff --git a/src/renderer/src/components/terminal-pane/stale-document-visibility.ts b/src/renderer/src/components/terminal-pane/stale-document-visibility.ts new file mode 100644 index 00000000000..f443bac3158 --- /dev/null +++ b/src/renderer/src/components/terminal-pane/stale-document-visibility.ts @@ -0,0 +1,100 @@ +// Chromium's macOS window-occlusion tracker can wedge document.visibilityState +// at 'hidden' after display sleep, never firing another visibilitychange. Real +// user input reaching this window while the document claims hidden is a +// physical contradiction — keystrokes and clicks only reach a focused, +// on-screen window. This module latches that proof so terminal delivery can +// treat the document as visible until the next genuine visibilitychange +// restores trust in the occlusion tracker. Without it, the hidden-delivery +// gate stays latched for panes the user is looking at and main drops their +// bytes indefinitely (field: 78MB dropped on 2 visible ptys, v1.4.124-rc.2.perf). + +type StaleVisibilityRecoveryListener = () => void + +const recoveryListeners = new Set() +let visibilityProvenStale = false +let globalListenersInstalled = false + +export function isDocumentVisibilityProvenStale(): boolean { + return visibilityProvenStale +} + +function onUserInteractionWithDocument(): void { + if (visibilityProvenStale || document.visibilityState !== 'hidden') { + return + } + visibilityProvenStale = true + console.warn( + '[terminal] user input arrived while document.visibilityState is hidden — treating occlusion state as stale and re-syncing terminal delivery', + { recoveryListenerCount: recoveryListeners.size } + ) + for (const listener of recoveryListeners) { + try { + listener() + } catch { + // Why: one pane's recovery failure must not starve the other panes'. + } + } +} + +function onDocumentVisibilityChange(): void { + // A genuine visibilitychange means the occlusion tracker is reporting + // again — hand authority back to document.visibilityState. + visibilityProvenStale = false +} + +function installGlobalListeners(): void { + if ( + globalListenersInstalled || + typeof document === 'undefined' || + typeof window === 'undefined' || + typeof document.addEventListener !== 'function' + ) { + return + } + globalListenersInstalled = true + // Capture phase so no stopPropagation in the app can hide the proof; the + // handler is a single property read when visibility is healthy. + document.addEventListener('keydown', onUserInteractionWithDocument, { + capture: true, + passive: true + }) + document.addEventListener('pointerdown', onUserInteractionWithDocument, { + capture: true, + passive: true + }) + window.addEventListener('focus', onUserInteractionWithDocument) + document.addEventListener('visibilitychange', onDocumentVisibilityChange) +} + +function removeGlobalListeners(): void { + if (!globalListenersInstalled) { + return + } + globalListenersInstalled = false + document.removeEventListener('keydown', onUserInteractionWithDocument, { capture: true }) + document.removeEventListener('pointerdown', onUserInteractionWithDocument, { capture: true }) + window.removeEventListener('focus', onUserInteractionWithDocument) + document.removeEventListener('visibilitychange', onDocumentVisibilityChange) +} + +// The listener runs when staleness is first proven; register the same handler +// used for document visibilitychange so recovery reuses the pane's existing +// gate-resync + hidden-output-restore path. +export function registerStaleDocumentVisibilityRecovery( + listener: StaleVisibilityRecoveryListener +): () => void { + installGlobalListeners() + recoveryListeners.add(listener) + return () => { + recoveryListeners.delete(listener) + if (recoveryListeners.size === 0) { + removeGlobalListeners() + } + } +} + +export function resetStaleDocumentVisibilityForTesting(): void { + visibilityProvenStale = false + recoveryListeners.clear() + removeGlobalListeners() +} diff --git a/tests/e2e/terminal-stuck-occlusion-recovery.spec.ts b/tests/e2e/terminal-stuck-occlusion-recovery.spec.ts new file mode 100644 index 00000000000..ae819c07b06 --- /dev/null +++ b/tests/e2e/terminal-stuck-occlusion-recovery.spec.ts @@ -0,0 +1,122 @@ +/** + * Repro + recovery for the stuck-occlusion freeze (field snapshot, + * v1.4.124-rc.2.perf, 2026-07-06): macOS occlusion tracking wedges + * document.visibilityState at 'hidden' after display sleep and never fires + * another visibilitychange. The hidden-delivery gate then marks the pane the + * user is looking at as hidden, and main drops its renderer-bound bytes + * indefinitely (78MB dropped across 2 visible ptys in the field) — a frozen + * terminal with a perfectly healthy transport. + * + * The test emulates the wedge exactly as Chromium produces it: visibilityState + * pinned to 'hidden' with one final visibilitychange, then silence. Recovery + * must come from the staleness proof — a real keystroke while the document + * claims hidden — which unlatches the gate and repaints the missed output from + * the main-owned snapshot, WITHOUT a reload and WITHOUT any visibilitychange. + */ +import type { Page } from '@stablyai/playwright-test' +import { test, expect } from './helpers/orca-app' +import { waitForSessionReady, waitForActiveWorktree, ensureTerminalVisible } from './helpers/store' +import { + waitForActiveTerminalManager, + waitForActivePanePtyId, + execInTerminal, + getTerminalContent +} from './helpers/terminal' + +type DeliverySnapshot = { + hiddenDeliveryGatedPtyCount: number + hiddenDeliveryGatedVisiblePtyCount: number + hiddenDeliveryDroppedChars: number +} + +async function getDeliverySnapshot(page: Page): Promise { + return page.evaluate(async () => { + const snapshot = await window.api.pty.getRendererDeliveryDebugSnapshot() + return { + hiddenDeliveryGatedPtyCount: snapshot.hiddenDeliveryGatedPtyCount, + hiddenDeliveryGatedVisiblePtyCount: snapshot.hiddenDeliveryGatedVisiblePtyCount, + hiddenDeliveryDroppedChars: snapshot.hiddenDeliveryDroppedChars + } + }) +} + +test.describe('terminal stuck-occlusion recovery', () => { + test.afterEach(async ({ orcaPage }) => { + // Drop the instance shadow so the prototype getter (real state) rules + // again, and fire one genuine visibilitychange to restore tracker trust. + await orcaPage.evaluate(() => { + delete (document as { visibilityState?: string }).visibilityState + document.dispatchEvent(new Event('visibilitychange')) + }) + }) + + test('a keystroke unlatches the hidden-delivery gate wedged by stale visibilityState', async ({ + orcaPage + }) => { + test.setTimeout(120_000) + await waitForSessionReady(orcaPage) + await waitForActiveWorktree(orcaPage) + await ensureTerminalVisible(orcaPage) + await waitForActiveTerminalManager(orcaPage) + const ptyId = await waitForActivePanePtyId(orcaPage) + + // Live baseline: foreground delivery works. The $((…)) arithmetic keeps + // the asserted string out of the typed command's local echo. + await execInTerminal(orcaPage, ptyId, 'echo live-before-$((41+1))') + await expect + .poll(async () => getTerminalContent(orcaPage), { timeout: 15_000 }) + .toContain('live-before-42') + + // Emulate the Chromium occlusion wedge: visibilityState pins at 'hidden', + // one last visibilitychange fires, then the tracker goes silent forever. + await orcaPage.evaluate(() => { + Object.defineProperty(document, 'visibilityState', { + get: () => 'hidden', + configurable: true + }) + document.dispatchEvent(new Event('visibilitychange')) + }) + + // The visible pane's pty gets marked hidden in main — the field state: + // gate holding a pty that main's own visibility set says is visible. + await expect + .poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedPtyCount, { + timeout: 15_000 + }) + .toBeGreaterThan(0) + expect( + (await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedVisiblePtyCount + ).toBeGreaterThan(0) + + // The freeze repro: output produced now is dropped by main, not painted. + const droppedBefore = (await getDeliverySnapshot(orcaPage)).hiddenDeliveryDroppedChars + await execInTerminal(orcaPage, ptyId, 'echo occluded-$((70+8))') + await expect + .poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryDroppedChars, { + timeout: 15_000 + }) + .toBeGreaterThan(droppedBefore) + expect(await getTerminalContent(orcaPage)).not.toContain('occluded-78') + + // The staleness proof: one real keystroke while the document claims + // hidden. No visibilitychange fires — recovery must ride the proof alone. + await orcaPage.keyboard.press('Shift') + + // Gate unlatches and the missed output repaints from the main-owned + // snapshot — no reload, visibilityState still reads 'hidden'. + await expect + .poll(async () => getTerminalContent(orcaPage), { timeout: 30_000 }) + .toContain('occluded-78') + await expect + .poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedVisiblePtyCount, { + timeout: 15_000 + }) + .toBe(0) + + // Live delivery continues under the override. + await execInTerminal(orcaPage, ptyId, 'echo live-after-$((200+56))') + await expect + .poll(async () => getTerminalContent(orcaPage), { timeout: 15_000 }) + .toContain('live-after-256') + }) +})