mirror of
https://github.com/stablyai/orca.git
synced 2026-10-18 00:03:20 +00:00
Unlatch the hidden-delivery gate when user input disproves a stuck document.visibilityState
macOS occlusion tracking can wedge document.visibilityState at 'hidden' after display sleep and never fire another visibilitychange. The hidden- delivery gate then keeps dropping renderer-bound bytes for panes the user is looking at (field snapshot 2026-07-06, v1.4.124-rc.2.perf: 78MB dropped across 2 pane-level-visible ptys with a fully healthy transport), and every recovery path (window focus, system-resume relay, backlog recovery) re-ran syncHiddenRendererPtyDelivery only to recompute the same stale predicate — nothing could ever clear the gate. The user sees a frozen terminal; typing echo is dropped in main; only a reload recovers. Real user input while the document claims hidden is a physical contradiction: keystrokes and clicks only reach a focused, on-screen window. stale-document-visibility.ts latches that proof, runs each pane's existing visibilitychange resync (gate unhide + hidden-output snapshot restore), and hands authority back to the occlusion tracker on the next genuine visibilitychange. No timers; the failure bias is safe — a wrong latch can only restore pre-gate delivery cost, never drop bytes. Hot path unchanged: the foreground predicate still returns on the same single comparison while the document is visible. tests/e2e/terminal-stuck-occlusion-recovery.spec.ts pins the wedge (visibilityState pinned hidden -> output dropped, not painted; the hiddenDeliveryGatedVisiblePtyCount field discriminator reads >0) and the recovery (one Shift keypress repaints the missed output from the main-owned snapshot, no reload, while visibilityState still reads hidden). Negative control verified: the spec fails without this fix. Typing-latency perf gate passes; terminal-pane unit suites 366/366. Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
@@ -3,7 +3,7 @@
|
||||
Agent-facing map of every optimization on this branch: what it does, why it exists,
|
||||
where it lives, and the invariants you must not break when adding to it. The
|
||||
chronological evidence trail (benchmarks, retractions, A/B protocols) is in
|
||||
`notes/terminal-performance-initiative.md`; this doc is the *current-state* view.
|
||||
`notes/terminal-performance-initiative.md`; this doc is the _current-state_ view.
|
||||
|
||||
**Context**: Orca's terminal was ~300× slower than Terminal.app under agent load
|
||||
(DSR-under-load p50 134ms, p99 292ms on v1.4.91; agent-TUI throughput 2.0 MB/s).
|
||||
@@ -21,13 +21,14 @@ shell → pty → daemon (persistence, headless model) → unix socket
|
||||
|
||||
Main is on the hot path for every byte (unlike VS Code's ptyHost→renderer
|
||||
MessagePort). The daemon owns sessions so they survive app restarts; it also runs
|
||||
a headless xterm emulator per pty — the *model* — which is the source of truth
|
||||
for screen contents. The renderer terminal is a *view* that can be discarded and
|
||||
a headless xterm emulator per pty — the _model_ — which is the source of truth
|
||||
for screen contents. The renderer terminal is a _view_ that can be discarded and
|
||||
rebuilt from model snapshots.
|
||||
|
||||
## Optimization inventory
|
||||
|
||||
### 1. Renderer parse-path fixes (the original 16× on agent TUIs)
|
||||
|
||||
- **Parse-clocked scheduler drains** (`pane-terminal-output-scheduler.ts`): drain
|
||||
cadence follows xterm's actual parse completion instead of fixed timers, so the
|
||||
queue never outruns the parser.
|
||||
@@ -40,8 +41,10 @@ rebuilt from model snapshots.
|
||||
per-chunk cost.
|
||||
|
||||
### 2. term-speed-2 chain (model/view contract — the architecture)
|
||||
|
||||
Revived from ~38 never-merged branches; kill-switched, default ON. Docs:
|
||||
`docs/reference/terminal-model-view-contract.md`.
|
||||
|
||||
- **Hidden view parking**: hidden tabs tear down their xterm view entirely
|
||||
(memory: parked panes cost ~0).
|
||||
- **Hidden delivery gate** (main): renderer-bound bytes for hidden ptys are
|
||||
@@ -55,6 +58,7 @@ Revived from ~38 never-merged branches; kill-switched, default ON. Docs:
|
||||
reconciliation drops duplicates already covered by the snapshot baseline.
|
||||
|
||||
### 3. Batching & scheduling cadence
|
||||
|
||||
- **Batch windows 8ms → 2ms** in both `daemon-stream-data-batcher.ts`
|
||||
(`STREAM_DATA_BATCH_INTERVAL_MS`) and `ipc/pty.ts` (`PTY_BATCH_INTERVAL_MS`).
|
||||
At 9% utilization there is no queue — latency was literally the sum of fixed
|
||||
@@ -67,9 +71,11 @@ Revived from ~38 never-merged branches; kill-switched, default ON. Docs:
|
||||
instead of queuing macrotask-per-keystroke.
|
||||
|
||||
### 4. Backpressure (the correctness spine — read before touching delivery)
|
||||
|
||||
Three cooperating layers, innermost first:
|
||||
|
||||
- **ACK at parse-drain** (`deliverPtyDataWithDeferredAck`, scheduler
|
||||
`ackCredit`): the renderer credits a chunk when xterm has *parsed* it (or the
|
||||
`ackCredit`): the renderer credits a chunk when xterm has _parsed_ it (or the
|
||||
chunk is legitimately discarded), not when IPC delivered it.
|
||||
**INVARIANT: every delivered chunk credits exactly once — parsed or
|
||||
discarded.** Every scheduler/pty-connection discard path (backlog replacement,
|
||||
@@ -92,15 +98,32 @@ Three cooperating layers, innermost first:
|
||||
push listeners + pull restore markers through the modelRestoreNeeded router.
|
||||
E2e blackhole harness: `__terminalDeliveryWatchdog`,
|
||||
`terminal-push-delivery-loss-recovery.spec.ts`.
|
||||
- **Stale-visibility proof for the hidden gate** (`stale-document-visibility.ts`
|
||||
and the `shouldWritePtyOutputForeground` fallthrough in pty-connection.ts):
|
||||
recovers the field-confirmed wedge where macOS occlusion tracking pins
|
||||
`document.visibilityState` at `'hidden'` after display sleep and never fires
|
||||
another visibilitychange (v1.4.124-rc.2.perf snapshot: 78MB hidden-gate
|
||||
dropped across 2 pane-level-visible ptys, transport healthy). Real user
|
||||
input (keydown/pointerdown/window focus) while the document claims hidden is
|
||||
a physical contradiction — it latches an override, runs each pane's existing
|
||||
visibilitychange resync (gate unhide + hidden-output restore), and a genuine
|
||||
visibilitychange hands authority back. No timers; recovery is purely
|
||||
event-proven, and the failure bias is safe (a wrong override only restores
|
||||
pre-gate delivery cost, never drops bytes). Hot-path cost: zero when
|
||||
visible (same single comparison); one property read per user-interaction
|
||||
event. E2e: `terminal-stuck-occlusion-recovery.spec.ts` (pins both the
|
||||
freeze repro and the keystroke recovery, plus the
|
||||
`hiddenDeliveryGatedVisiblePtyCount` field discriminator).
|
||||
- **Producer flow control** (protocol v19 `pausePty`/`resumePty`, 256KB pause /
|
||||
32KB resume watermarks, keyed off **pendingData only** — never renderer
|
||||
counters; kill switch `PRODUCER_FLOW_CONTROL_ENABLED`, ipc/pty.ts): when main's
|
||||
buffer grows, the *shell* blocks. For main-hosted ptys pause is synchronous
|
||||
buffer grows, the _shell_ blocks. For main-hosted ptys pause is synchronous
|
||||
(drops impossible); for daemon ptys the pause notify has ~20-30ms socket
|
||||
latency, so wire-speed bursts can still cross the 2MB cap (known follow-up:
|
||||
daemon-side self-pacing watermark).
|
||||
|
||||
### 5. Flood resilience (why bulk output can't wedge or lie anymore)
|
||||
|
||||
- **Restore-loop cut** (pty-connection.ts): the hidden-output-restore loop
|
||||
abandons immediately when a foreground pane's live-chunk queue overflows
|
||||
(3-iteration hard cap), and a 2s flood-suppression window stops main's own
|
||||
@@ -110,20 +133,23 @@ Three cooperating layers, innermost first:
|
||||
drop re-arms restore) that caused multi-second renderer stalls.
|
||||
- **Query survival**: if the 2MB cap ever drops bulk output, embedded terminal
|
||||
queries are extracted (`terminal-reply-query-extraction.ts`) and answered by
|
||||
*synthesizing replies on the input path* (CPR from live buffer, DA1 canned,
|
||||
_synthesizing replies on the input path_ (CPR from live buffer, DA1 canned,
|
||||
OSC via direct responder) — probes and TUIs never hang on a dropped reply.
|
||||
- Drops are downstream of the model: the daemon ingests every byte, so the
|
||||
post-flood repaint restores complete, correct content.
|
||||
|
||||
### 6. Wake/sleep recovery
|
||||
|
||||
- powerMonitor resume → `system:resumed` IPC → renderer wake recovery (fixes
|
||||
WebGL-latch blank-after-sleep that DOM focus/visibilitychange missed).
|
||||
- Cumulative ACKs make the historical "lost ACKs across suspend pin the global
|
||||
window forever" wedge (BMW user bug) structurally impossible.
|
||||
|
||||
### 7. Snapshot fidelity (the garble fixes — all fuzz-pinned)
|
||||
|
||||
Reveal-from-snapshot multiplied exposure of serializer defects ~1000×. Five bugs
|
||||
found by differential fuzzing; four fixed, one tolerated:
|
||||
|
||||
- **B: SGR intensity ordering** — upstream `@xterm/addon-serialize` emitted
|
||||
`1;22` (22 clears the bold 1 just set). Patched via pnpm patch
|
||||
(`config/patches/@xterm__addon-serialize@*.patch`): clear-before-set for the
|
||||
@@ -143,6 +169,7 @@ found by differential fuzzing; four fixed, one tolerated:
|
||||
`bufferHasSerializeHostileWrappedRow`, the only remaining tolerance.
|
||||
|
||||
## Correctness infrastructure (run these before merging delivery/restore changes)
|
||||
|
||||
- `headless-emulator-fidelity.fuzz.test.ts` — differential: HeadlessEmulator vs
|
||||
reference xterm, seeded TUI streams. `FUZZ_ITERATIONS=2000` for deep,
|
||||
`FUZZ_SEED=n` to replay.
|
||||
@@ -158,8 +185,9 @@ found by differential fuzzing; four fixed, one tolerated:
|
||||
(pane-manager / terminal-pane suites).
|
||||
|
||||
## Benchmarking protocol (hard-won rules)
|
||||
|
||||
- Rig: `tools/benchmarks/terminal-pipeline-bench.mjs` — DSR idle + DSR under
|
||||
1MB/s agent-TUI load + DSR-fenced throughput on 4 fixtures. Run *inside* the
|
||||
1MB/s agent-TUI load + DSR-fenced throughput on 4 fixtures. Run _inside_ the
|
||||
terminal under test.
|
||||
- **Bench at 10MB** (`--size-mb 10`). The ACK-at-parse bug shipped because dev
|
||||
benches used 3MB and never tripped the cap.
|
||||
@@ -170,8 +198,9 @@ found by differential fuzzing; four fixed, one tolerated:
|
||||
- Packaged builds are truth; dev has ~2× overhead.
|
||||
|
||||
## Release mechanics
|
||||
|
||||
- Perf RCs: `release-cut.yml` workflow_dispatch, `kind=rc ref=orca-performance
|
||||
version_suffix=perf` → tags like `v1.4.122-rc.1.perf`. The suffix sorts above
|
||||
version_suffix=perf` → tags like `v1.4.122-rc.1.perf`. The suffix sorts above
|
||||
its base rc.N but below rc.N+1 (never hijacks the RC channel). The rc counter
|
||||
(`release-rc-history.mjs`), telemetry identity classifier, and build guard are
|
||||
all suffix-aware — a suffixed rc classifies as `rc`.
|
||||
@@ -205,6 +234,7 @@ Conflict pattern, established over ~6 syncs:
|
||||
before the next RC cut.
|
||||
|
||||
## Known limits / next levers (in rough priority order)
|
||||
|
||||
1. Daemon self-pacing: daemon-hosted ptys can cross the 2MB cap for ~20-30ms at
|
||||
wire speed before `pausePty` bites. Fix: daemon enforces its own watermark
|
||||
locally (VS Code does this server-side for remotes).
|
||||
@@ -218,12 +248,13 @@ Conflict pattern, established over ~6 syncs:
|
||||
is open against main separately.
|
||||
|
||||
## Guardrails for future agents
|
||||
|
||||
- The chunk-credit invariant (§4) is the load-bearing one. If you add ANY path
|
||||
that receives, defers, drops, or splits pty data in the renderer, prove it
|
||||
credits exactly once. The credit-invariant unit tests are the gate.
|
||||
- Flow control keys off `pendingData` only. Do not couple it to renderer
|
||||
counters; the two layers compose because they are independent.
|
||||
- Snapshot changes must keep seq semantics: a snapshot covers *exactly* bytes
|
||||
- Snapshot changes must keep seq semantics: a snapshot covers _exactly_ bytes
|
||||
≤ its seq (Bug E made this true; don't regress it). Chunks after restore are
|
||||
reconciled by seq — off-by-N re-triggers duplicate-drop garble.
|
||||
- Hidden panes must receive nothing (delivery gate) but side-effects and query
|
||||
|
||||
@@ -8652,12 +8652,21 @@ describe('connectPanePty', () => {
|
||||
})
|
||||
|
||||
it('does not apply stale background Codex query chunks after hidden snapshot restore', async () => {
|
||||
const visibilityChangeHandler: { current: (() => void) | null } = { current: null }
|
||||
// Fire-all like a real event target: the pane resync handler and the
|
||||
// stale-visibility trust handler both listen for visibilitychange.
|
||||
const visibilityChangeListeners: (() => void)[] = []
|
||||
const visibilityChangeHandler = {
|
||||
current: (): void => {
|
||||
for (const listener of visibilityChangeListeners) {
|
||||
listener()
|
||||
}
|
||||
}
|
||||
}
|
||||
;(globalThis as { document?: Document }).document = {
|
||||
visibilityState: 'visible',
|
||||
addEventListener: vi.fn((type: string, listener: EventListenerOrEventListenerObject) => {
|
||||
if (type === 'visibilitychange') {
|
||||
visibilityChangeHandler.current = listener as () => void
|
||||
visibilityChangeListeners.push(listener as () => void)
|
||||
}
|
||||
}),
|
||||
removeEventListener: vi.fn()
|
||||
@@ -9959,14 +9968,23 @@ describe('connectPanePty', () => {
|
||||
|
||||
it('keeps recovery pending when hidden output arrives during an in-flight snapshot', async () => {
|
||||
let visibilityState: DocumentVisibilityState = 'visible'
|
||||
const visibilityChangeHandler: { current: (() => void) | null } = { current: null }
|
||||
// Fire-all like a real event target: the pane resync handler and the
|
||||
// stale-visibility trust handler both listen for visibilitychange.
|
||||
const visibilityChangeListeners: (() => void)[] = []
|
||||
const visibilityChangeHandler = {
|
||||
current: (): void => {
|
||||
for (const listener of visibilityChangeListeners) {
|
||||
listener()
|
||||
}
|
||||
}
|
||||
}
|
||||
;(globalThis as { document?: Document }).document = {
|
||||
get visibilityState() {
|
||||
return visibilityState
|
||||
},
|
||||
addEventListener: vi.fn((type: string, listener: EventListenerOrEventListenerObject) => {
|
||||
if (type === 'visibilitychange') {
|
||||
visibilityChangeHandler.current = listener as () => void
|
||||
visibilityChangeListeners.push(listener as () => void)
|
||||
}
|
||||
}),
|
||||
removeEventListener: vi.fn()
|
||||
|
||||
@@ -49,6 +49,10 @@ import { isPtyLocked } from '@/lib/pane-manager/mobile-driver-state'
|
||||
import { reconcilePtySizeAcrossFrames, type PtySizeReconcileHandle } from './pty-size-reconcile'
|
||||
import { createPtySizeReassertion } from './pty-size-reassertion'
|
||||
import { isPaneReplaying, replayIntoTerminal, replayIntoTerminalAsync } from './replay-guard'
|
||||
import {
|
||||
isDocumentVisibilityProvenStale,
|
||||
registerStaleDocumentVisibilityRecovery
|
||||
} from './stale-document-visibility'
|
||||
import {
|
||||
nativeWindowsRewriteNeedsFollowupRenderRefresh,
|
||||
terminalOutputPrefersRenderRefresh,
|
||||
@@ -766,7 +770,13 @@ function shouldWritePtyOutputForeground(isPaneVisible: boolean): boolean {
|
||||
// Why: Electron can keep visible panes mounted while the whole app is
|
||||
// backgrounded. Treat hidden documents like background tabs so Chromium
|
||||
// timer throttling cannot pin terminal writes on the renderer foreground path.
|
||||
return document.visibilityState === 'visible'
|
||||
if (document.visibilityState === 'visible') {
|
||||
return true
|
||||
}
|
||||
// Why: macOS occlusion tracking can wedge visibilityState at 'hidden' after
|
||||
// display sleep; proven-stale means real user input contradicted it, so the
|
||||
// hidden-delivery gate must not keep dropping a watched pane's bytes.
|
||||
return isDocumentVisibilityProvenStale()
|
||||
}
|
||||
|
||||
function containsSynchronizedOutputStart(data: string): boolean {
|
||||
@@ -5582,8 +5592,17 @@ export function connectPanePty(
|
||||
}
|
||||
}
|
||||
document.addEventListener('visibilitychange', onDocumentVisibilityChange)
|
||||
unregisterDocumentVisibilityRecovery = () =>
|
||||
// Why: when user input proves visibilityState is wedged at 'hidden'
|
||||
// (stale macOS occlusion), run the same resync — no visibilitychange
|
||||
// will ever fire in that state, and the gate would drop watched bytes
|
||||
// forever.
|
||||
const unregisterStaleVisibilityRecovery = registerStaleDocumentVisibilityRecovery(
|
||||
onDocumentVisibilityChange
|
||||
)
|
||||
unregisterDocumentVisibilityRecovery = () => {
|
||||
document.removeEventListener('visibilitychange', onDocumentVisibilityChange)
|
||||
unregisterStaleVisibilityRecovery()
|
||||
}
|
||||
}
|
||||
|
||||
const dataCallback = (data: string, meta?: PtyDataMeta): void => {
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
// Why: pins the stale-occlusion proof — user input while
|
||||
// document.visibilityState claims 'hidden' must latch the override and run
|
||||
// each pane's recovery exactly once, and a genuine visibilitychange must hand
|
||||
// authority back to the occlusion tracker.
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type * as StaleDocumentVisibility from './stale-document-visibility'
|
||||
|
||||
type Handler = () => void
|
||||
|
||||
type EventTargetStub = {
|
||||
addEventListener: (type: string, handler: Handler, options?: unknown) => void
|
||||
removeEventListener: (type: string, handler: Handler, options?: unknown) => void
|
||||
emit: (type: string) => void
|
||||
listenerCount: (type: string) => number
|
||||
}
|
||||
|
||||
function createEventTargetStub(): EventTargetStub {
|
||||
const listeners = new Map<string, Set<Handler>>()
|
||||
return {
|
||||
addEventListener(type, handler) {
|
||||
const set = listeners.get(type) ?? new Set()
|
||||
set.add(handler)
|
||||
listeners.set(type, set)
|
||||
},
|
||||
removeEventListener(type, handler) {
|
||||
listeners.get(type)?.delete(handler)
|
||||
},
|
||||
emit(type) {
|
||||
for (const handler of listeners.get(type) ?? []) {
|
||||
handler()
|
||||
}
|
||||
},
|
||||
listenerCount(type) {
|
||||
return listeners.get(type)?.size ?? 0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type DocumentStub = EventTargetStub & { visibilityState: string }
|
||||
|
||||
describe('stale document visibility', () => {
|
||||
const originalDocument = (globalThis as { document?: unknown }).document
|
||||
const originalWindow = (globalThis as { window?: unknown }).window
|
||||
let documentStub: DocumentStub
|
||||
let windowStub: EventTargetStub
|
||||
let warnSpy: ReturnType<typeof vi.spyOn>
|
||||
let mod: typeof StaleDocumentVisibility
|
||||
|
||||
beforeEach(async () => {
|
||||
vi.resetModules()
|
||||
documentStub = { ...createEventTargetStub(), visibilityState: 'visible' }
|
||||
windowStub = createEventTargetStub()
|
||||
;(globalThis as { document: unknown }).document = documentStub
|
||||
;(globalThis as { window: unknown }).window = windowStub
|
||||
warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
|
||||
mod = await import('./stale-document-visibility')
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
mod.resetStaleDocumentVisibilityForTesting()
|
||||
warnSpy.mockRestore()
|
||||
;(globalThis as { document: unknown }).document = originalDocument
|
||||
;(globalThis as { window: unknown }).window = originalWindow
|
||||
})
|
||||
|
||||
it('latches the override and runs recovery once when input arrives while hidden', () => {
|
||||
const recovery = vi.fn()
|
||||
mod.registerStaleDocumentVisibilityRecovery(recovery)
|
||||
documentStub.visibilityState = 'hidden'
|
||||
|
||||
documentStub.emit('keydown')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(true)
|
||||
expect(recovery).toHaveBeenCalledTimes(1)
|
||||
expect(warnSpy).toHaveBeenCalledTimes(1)
|
||||
|
||||
// Further input during the same stuck episode must not re-run recovery.
|
||||
documentStub.emit('keydown')
|
||||
documentStub.emit('pointerdown')
|
||||
expect(recovery).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('does not latch while the document is genuinely visible', () => {
|
||||
const recovery = vi.fn()
|
||||
mod.registerStaleDocumentVisibilityRecovery(recovery)
|
||||
|
||||
documentStub.emit('keydown')
|
||||
documentStub.emit('pointerdown')
|
||||
windowStub.emit('focus')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(false)
|
||||
expect(recovery).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('treats pointerdown and window focus as staleness proof too', () => {
|
||||
const recovery = vi.fn()
|
||||
mod.registerStaleDocumentVisibilityRecovery(recovery)
|
||||
documentStub.visibilityState = 'hidden'
|
||||
|
||||
documentStub.emit('pointerdown')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(true)
|
||||
|
||||
mod.resetStaleDocumentVisibilityForTesting()
|
||||
mod.registerStaleDocumentVisibilityRecovery(recovery)
|
||||
documentStub.visibilityState = 'hidden'
|
||||
windowStub.emit('focus')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(true)
|
||||
})
|
||||
|
||||
it('hands authority back to the occlusion tracker on a genuine visibilitychange', () => {
|
||||
const recovery = vi.fn()
|
||||
mod.registerStaleDocumentVisibilityRecovery(recovery)
|
||||
documentStub.visibilityState = 'hidden'
|
||||
documentStub.emit('keydown')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(true)
|
||||
|
||||
documentStub.visibilityState = 'visible'
|
||||
documentStub.emit('visibilitychange')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(false)
|
||||
|
||||
// A fresh wedge after trust was restored must latch (and recover) again.
|
||||
documentStub.visibilityState = 'hidden'
|
||||
documentStub.emit('keydown')
|
||||
expect(mod.isDocumentVisibilityProvenStale()).toBe(true)
|
||||
expect(recovery).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('keeps running other panes when one recovery listener throws', () => {
|
||||
const throwing = vi.fn(() => {
|
||||
throw new Error('pane exploded')
|
||||
})
|
||||
const healthy = vi.fn()
|
||||
mod.registerStaleDocumentVisibilityRecovery(throwing)
|
||||
mod.registerStaleDocumentVisibilityRecovery(healthy)
|
||||
documentStub.visibilityState = 'hidden'
|
||||
|
||||
documentStub.emit('keydown')
|
||||
expect(throwing).toHaveBeenCalledTimes(1)
|
||||
expect(healthy).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('removes global listeners when the last pane unregisters', () => {
|
||||
const unregisterA = mod.registerStaleDocumentVisibilityRecovery(vi.fn())
|
||||
const unregisterB = mod.registerStaleDocumentVisibilityRecovery(vi.fn())
|
||||
expect(documentStub.listenerCount('keydown')).toBe(1)
|
||||
|
||||
unregisterA()
|
||||
expect(documentStub.listenerCount('keydown')).toBe(1)
|
||||
unregisterB()
|
||||
expect(documentStub.listenerCount('keydown')).toBe(0)
|
||||
expect(documentStub.listenerCount('pointerdown')).toBe(0)
|
||||
expect(documentStub.listenerCount('visibilitychange')).toBe(0)
|
||||
expect(windowStub.listenerCount('focus')).toBe(0)
|
||||
|
||||
// Re-registering must reinstall for a later pane.
|
||||
mod.registerStaleDocumentVisibilityRecovery(vi.fn())
|
||||
expect(documentStub.listenerCount('keydown')).toBe(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,100 @@
|
||||
// Chromium's macOS window-occlusion tracker can wedge document.visibilityState
|
||||
// at 'hidden' after display sleep, never firing another visibilitychange. Real
|
||||
// user input reaching this window while the document claims hidden is a
|
||||
// physical contradiction — keystrokes and clicks only reach a focused,
|
||||
// on-screen window. This module latches that proof so terminal delivery can
|
||||
// treat the document as visible until the next genuine visibilitychange
|
||||
// restores trust in the occlusion tracker. Without it, the hidden-delivery
|
||||
// gate stays latched for panes the user is looking at and main drops their
|
||||
// bytes indefinitely (field: 78MB dropped on 2 visible ptys, v1.4.124-rc.2.perf).
|
||||
|
||||
type StaleVisibilityRecoveryListener = () => void
|
||||
|
||||
const recoveryListeners = new Set<StaleVisibilityRecoveryListener>()
|
||||
let visibilityProvenStale = false
|
||||
let globalListenersInstalled = false
|
||||
|
||||
export function isDocumentVisibilityProvenStale(): boolean {
|
||||
return visibilityProvenStale
|
||||
}
|
||||
|
||||
function onUserInteractionWithDocument(): void {
|
||||
if (visibilityProvenStale || document.visibilityState !== 'hidden') {
|
||||
return
|
||||
}
|
||||
visibilityProvenStale = true
|
||||
console.warn(
|
||||
'[terminal] user input arrived while document.visibilityState is hidden — treating occlusion state as stale and re-syncing terminal delivery',
|
||||
{ recoveryListenerCount: recoveryListeners.size }
|
||||
)
|
||||
for (const listener of recoveryListeners) {
|
||||
try {
|
||||
listener()
|
||||
} catch {
|
||||
// Why: one pane's recovery failure must not starve the other panes'.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function onDocumentVisibilityChange(): void {
|
||||
// A genuine visibilitychange means the occlusion tracker is reporting
|
||||
// again — hand authority back to document.visibilityState.
|
||||
visibilityProvenStale = false
|
||||
}
|
||||
|
||||
function installGlobalListeners(): void {
|
||||
if (
|
||||
globalListenersInstalled ||
|
||||
typeof document === 'undefined' ||
|
||||
typeof window === 'undefined' ||
|
||||
typeof document.addEventListener !== 'function'
|
||||
) {
|
||||
return
|
||||
}
|
||||
globalListenersInstalled = true
|
||||
// Capture phase so no stopPropagation in the app can hide the proof; the
|
||||
// handler is a single property read when visibility is healthy.
|
||||
document.addEventListener('keydown', onUserInteractionWithDocument, {
|
||||
capture: true,
|
||||
passive: true
|
||||
})
|
||||
document.addEventListener('pointerdown', onUserInteractionWithDocument, {
|
||||
capture: true,
|
||||
passive: true
|
||||
})
|
||||
window.addEventListener('focus', onUserInteractionWithDocument)
|
||||
document.addEventListener('visibilitychange', onDocumentVisibilityChange)
|
||||
}
|
||||
|
||||
function removeGlobalListeners(): void {
|
||||
if (!globalListenersInstalled) {
|
||||
return
|
||||
}
|
||||
globalListenersInstalled = false
|
||||
document.removeEventListener('keydown', onUserInteractionWithDocument, { capture: true })
|
||||
document.removeEventListener('pointerdown', onUserInteractionWithDocument, { capture: true })
|
||||
window.removeEventListener('focus', onUserInteractionWithDocument)
|
||||
document.removeEventListener('visibilitychange', onDocumentVisibilityChange)
|
||||
}
|
||||
|
||||
// The listener runs when staleness is first proven; register the same handler
|
||||
// used for document visibilitychange so recovery reuses the pane's existing
|
||||
// gate-resync + hidden-output-restore path.
|
||||
export function registerStaleDocumentVisibilityRecovery(
|
||||
listener: StaleVisibilityRecoveryListener
|
||||
): () => void {
|
||||
installGlobalListeners()
|
||||
recoveryListeners.add(listener)
|
||||
return () => {
|
||||
recoveryListeners.delete(listener)
|
||||
if (recoveryListeners.size === 0) {
|
||||
removeGlobalListeners()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function resetStaleDocumentVisibilityForTesting(): void {
|
||||
visibilityProvenStale = false
|
||||
recoveryListeners.clear()
|
||||
removeGlobalListeners()
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
/**
|
||||
* Repro + recovery for the stuck-occlusion freeze (field snapshot,
|
||||
* v1.4.124-rc.2.perf, 2026-07-06): macOS occlusion tracking wedges
|
||||
* document.visibilityState at 'hidden' after display sleep and never fires
|
||||
* another visibilitychange. The hidden-delivery gate then marks the pane the
|
||||
* user is looking at as hidden, and main drops its renderer-bound bytes
|
||||
* indefinitely (78MB dropped across 2 visible ptys in the field) — a frozen
|
||||
* terminal with a perfectly healthy transport.
|
||||
*
|
||||
* The test emulates the wedge exactly as Chromium produces it: visibilityState
|
||||
* pinned to 'hidden' with one final visibilitychange, then silence. Recovery
|
||||
* must come from the staleness proof — a real keystroke while the document
|
||||
* claims hidden — which unlatches the gate and repaints the missed output from
|
||||
* the main-owned snapshot, WITHOUT a reload and WITHOUT any visibilitychange.
|
||||
*/
|
||||
import type { Page } from '@stablyai/playwright-test'
|
||||
import { test, expect } from './helpers/orca-app'
|
||||
import { waitForSessionReady, waitForActiveWorktree, ensureTerminalVisible } from './helpers/store'
|
||||
import {
|
||||
waitForActiveTerminalManager,
|
||||
waitForActivePanePtyId,
|
||||
execInTerminal,
|
||||
getTerminalContent
|
||||
} from './helpers/terminal'
|
||||
|
||||
type DeliverySnapshot = {
|
||||
hiddenDeliveryGatedPtyCount: number
|
||||
hiddenDeliveryGatedVisiblePtyCount: number
|
||||
hiddenDeliveryDroppedChars: number
|
||||
}
|
||||
|
||||
async function getDeliverySnapshot(page: Page): Promise<DeliverySnapshot> {
|
||||
return page.evaluate(async () => {
|
||||
const snapshot = await window.api.pty.getRendererDeliveryDebugSnapshot()
|
||||
return {
|
||||
hiddenDeliveryGatedPtyCount: snapshot.hiddenDeliveryGatedPtyCount,
|
||||
hiddenDeliveryGatedVisiblePtyCount: snapshot.hiddenDeliveryGatedVisiblePtyCount,
|
||||
hiddenDeliveryDroppedChars: snapshot.hiddenDeliveryDroppedChars
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
test.describe('terminal stuck-occlusion recovery', () => {
|
||||
test.afterEach(async ({ orcaPage }) => {
|
||||
// Drop the instance shadow so the prototype getter (real state) rules
|
||||
// again, and fire one genuine visibilitychange to restore tracker trust.
|
||||
await orcaPage.evaluate(() => {
|
||||
delete (document as { visibilityState?: string }).visibilityState
|
||||
document.dispatchEvent(new Event('visibilitychange'))
|
||||
})
|
||||
})
|
||||
|
||||
test('a keystroke unlatches the hidden-delivery gate wedged by stale visibilityState', async ({
|
||||
orcaPage
|
||||
}) => {
|
||||
test.setTimeout(120_000)
|
||||
await waitForSessionReady(orcaPage)
|
||||
await waitForActiveWorktree(orcaPage)
|
||||
await ensureTerminalVisible(orcaPage)
|
||||
await waitForActiveTerminalManager(orcaPage)
|
||||
const ptyId = await waitForActivePanePtyId(orcaPage)
|
||||
|
||||
// Live baseline: foreground delivery works. The $((…)) arithmetic keeps
|
||||
// the asserted string out of the typed command's local echo.
|
||||
await execInTerminal(orcaPage, ptyId, 'echo live-before-$((41+1))')
|
||||
await expect
|
||||
.poll(async () => getTerminalContent(orcaPage), { timeout: 15_000 })
|
||||
.toContain('live-before-42')
|
||||
|
||||
// Emulate the Chromium occlusion wedge: visibilityState pins at 'hidden',
|
||||
// one last visibilitychange fires, then the tracker goes silent forever.
|
||||
await orcaPage.evaluate(() => {
|
||||
Object.defineProperty(document, 'visibilityState', {
|
||||
get: () => 'hidden',
|
||||
configurable: true
|
||||
})
|
||||
document.dispatchEvent(new Event('visibilitychange'))
|
||||
})
|
||||
|
||||
// The visible pane's pty gets marked hidden in main — the field state:
|
||||
// gate holding a pty that main's own visibility set says is visible.
|
||||
await expect
|
||||
.poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedPtyCount, {
|
||||
timeout: 15_000
|
||||
})
|
||||
.toBeGreaterThan(0)
|
||||
expect(
|
||||
(await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedVisiblePtyCount
|
||||
).toBeGreaterThan(0)
|
||||
|
||||
// The freeze repro: output produced now is dropped by main, not painted.
|
||||
const droppedBefore = (await getDeliverySnapshot(orcaPage)).hiddenDeliveryDroppedChars
|
||||
await execInTerminal(orcaPage, ptyId, 'echo occluded-$((70+8))')
|
||||
await expect
|
||||
.poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryDroppedChars, {
|
||||
timeout: 15_000
|
||||
})
|
||||
.toBeGreaterThan(droppedBefore)
|
||||
expect(await getTerminalContent(orcaPage)).not.toContain('occluded-78')
|
||||
|
||||
// The staleness proof: one real keystroke while the document claims
|
||||
// hidden. No visibilitychange fires — recovery must ride the proof alone.
|
||||
await orcaPage.keyboard.press('Shift')
|
||||
|
||||
// Gate unlatches and the missed output repaints from the main-owned
|
||||
// snapshot — no reload, visibilityState still reads 'hidden'.
|
||||
await expect
|
||||
.poll(async () => getTerminalContent(orcaPage), { timeout: 30_000 })
|
||||
.toContain('occluded-78')
|
||||
await expect
|
||||
.poll(async () => (await getDeliverySnapshot(orcaPage)).hiddenDeliveryGatedVisiblePtyCount, {
|
||||
timeout: 15_000
|
||||
})
|
||||
.toBe(0)
|
||||
|
||||
// Live delivery continues under the override.
|
||||
await execInTerminal(orcaPage, ptyId, 'echo live-after-$((200+56))')
|
||||
await expect
|
||||
.poll(async () => getTerminalContent(orcaPage), { timeout: 15_000 })
|
||||
.toContain('live-after-256')
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user