test(terminal): add visible split activation latency benchmark

This commit is contained in:
Neil
2026-08-30 23:02:06 -07:00
parent d453ffcdb7
commit 6dddb7a409
5 changed files with 1026 additions and 14 deletions
+47 -13
View File
@@ -5735,12 +5735,14 @@
"providers": ["local", "local-daemon", "ssh", "wsl", "remote-runtime"],
"coveredPlatforms": ["macos"],
"coveredProviders": ["local", "remote-runtime"],
"coverageNotes": "Deterministic renderer contracts hold CWD resolution behind an explicit promise, require the new pane to be created synchronously, and prove that close cancels the pending connection. Detach contracts reject cwd-pending and cwd-resolved deferred splits before PTY bind without mutation, carry resolved cwd for other unbound panes, and preserve persisted or live PTY handoff. Local transport contracts exercise the real bounded pre-connect buffer and one live input FIFO across ordinary, acknowledged, and immediate writes, concurrent flushes, in-flight teardown, late spawn success or failure, attach failure, failed spawn, same-id reuse, stale-spawn retirement ownership, destroy, and mutable recovery metadata. A mocked direct-SSH authority-rotation contract proves a rejected stale spawn releases its deferred-CWD fence. Remote-runtime coverage proves delegation remains host-owned; physical daemon, SSH, WSL, Linux, and Windows journeys remain gaps.",
"coverageNotes": "Deterministic renderer contracts hold CWD resolution behind an explicit promise, require the new pane to be created synchronously, and prove that close cancels the pending connection. Detach contracts reject cwd-pending and cwd-resolved deferred splits before PTY bind without mutation, carry resolved cwd for other unbound panes, and preserve persisted or live PTY handoff. Local transport contracts exercise the real bounded pre-connect buffer and one live input FIFO across ordinary, acknowledged, and immediate writes, concurrent flushes, in-flight teardown, late spawn success or failure, attach failure, failed spawn, same-id reuse, stale-spawn retirement ownership, destroy, and mutable recovery metadata. A mocked direct-SSH authority-rotation contract proves a rejected stale spawn releases its deferred-CWD fence. The committed headful Electron benchmark drives the real platform shortcut in a visible BrowserWindow and document for 3 warmups followed by 20 measured cold-CWD cycles, requiring a distinct child PTY, first echo, and observed child pty:exit before the next cycle. Remote-runtime coverage proves delegation remains host-owned; physical local-daemon, SSH, WSL, Linux, Windows, and folder-workspace latency journeys remain gaps.",
"motivatingLinks": ["https://github.com/stablyai/orca/commit/572ed1a8882"],
"invariant": "A terminal split creates and activates its renderer pane before an inherited-CWD lookup settles, starts its PTY only after the resolved directory is available, and cannot be externally detached while that deferred spawn remains unbound. Other unbound panes preserve resolved cwd as startupCwd when detached. Bounded pre-connect input and later live local input share byte order, and teardown settles acknowledged writes without creating or rebinding a stale PTY. A disconnected or detached pending connect cannot bind its late fresh spawn, report its late failure through current callbacks, or ID-retire a newer same-ID owner; rejecting a stale direct-SSH spawn also releases the matching deferred-CWD fence. Natural exit cannot deliver queued work into a reused PTY id. Bound and remote-runtime splits remain owned by their execution host.",
"oracle": "Hold CWD resolution behind a controllable promise, invoke the production split path, and require manager.splitPane plus split telemetry before resolving it. Before PTY bind, require both cwd-pending and cwd-resolved deferred detach attempts to return null without layout, pane, tab, ownership, or focus mutation; separately require resolved cwd on an allowed unbound detach and unchanged persisted/live PTY adoption. Rotate a mocked direct-SSH authority while its delayed spawn is in flight, reject and disconnect the stale PTY claim, then require exactly one deferred-CWD cleanup when the delayed connect settles. At the PTY boundary, require zero connect calls while pending, the resolved CWD in spawn and local recovery metadata, one shared FIFO across pre-connect and live ordinary/acknowledged/immediate input, prompt acknowledged-promise settlement on teardown, retirement of an unowned late fresh spawn, preservation of a newer same-ID owner, and zero stale delivery after failure, close, destroy, detach, natural exit, or same-id reuse.",
"oracle": "Hold CWD resolution behind a controllable promise, invoke the production split path, and require manager.splitPane plus split telemetry before resolving it. Before PTY bind, require both cwd-pending and cwd-resolved deferred detach attempts to return null without layout, pane, tab, ownership, or focus mutation; separately require resolved cwd on an allowed unbound detach and unchanged persisted/live PTY adoption. Rotate a mocked direct-SSH authority while its delayed spawn is in flight, reject and disconnect the stale PTY claim, then require exactly one deferred-CWD cleanup when the delayed connect settles. At the PTY boundary, require zero connect calls while pending, the resolved CWD in spawn and local recovery metadata, one shared FIFO across pre-connect and live ordinary/acknowledged/immediate input, prompt acknowledged-promise settlement on teardown, retirement of an unowned late fresh spawn, preservation of a newer same-ID owner, and zero stale delivery after failure, close, destroy, detach, natural exit, or same-id reuse. In visible Electron, press the real split shortcut after each warmup and each cold inherited-CWD measured interval, and require complete focus, distinct-PTY bind, immediate-input echo, pane-count, and child-exit observations for every cycle; a timed-out, missing-event, or cleanup-aborted run must publish no headline latency and fail.",
"commands": [
"pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot"
"pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot",
"ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
"ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1"
],
"testFiles": [
"src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts",
@@ -5749,7 +5751,8 @@
"src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts",
"src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts",
"src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts",
"src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts"
"src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts",
"tests/e2e/terminal-split-activation-latency.spec.ts"
],
"assertionRefs": [
{
@@ -5815,6 +5818,16 @@
"local recovery metadata observes the resolved split CWD",
"a stale fresh-spawn completion cannot retire a newer same-ID owner"
]
},
{
"file": "tests/e2e/terminal-split-activation-latency.spec.ts",
"assertions": [
"requires a visible BrowserWindow and visible document before sampling",
"records 3 warmups followed by 20 measured real-shortcut cycles, each after a cold inherited-CWD interval",
"requires every split to focus, bind a PTY distinct from its source, and echo immediate input",
"observes each closed child PTY exit before starting the next cycle",
"marks timed-out, missing-event, or cleanup-aborted runs invalid and publishes no headline latency"
]
}
],
"evidenceRuns": [
@@ -5826,15 +5839,34 @@
"command": "pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot",
"durationSeconds": 56.59,
"summary": "Seven focused files and 78 tests passed. Vitest reported 49.92 seconds and the measured wall time was 56.59 seconds; coverage includes split creation and focus ordering, nested CWD lineage, promise-identity and SSH authority-rotation cleanup fencing, full pre-bind detach fencing, resolved-CWD detach handoff, close-cancellation, single-FIFO ordering, late-spawn retirement and error suppression, preservation of a newer same-ID owner, generation fencing, in-flight settlement across explicit teardown and natural exit, attach cleanup, bounded retention, and existing input-write contracts."
},
{
"date": "2026-08-30",
"runner": "local",
"platform": "macos",
"result": "passed",
"command": "ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
"durationSeconds": 72,
"summary": "At baseline df14d1a2983d8339e788d0e521f1c4affd9c6d5f, the visible BrowserWindow and document completed 3/3 warmups followed by 20/20 measured cold-CWD cycles with every event present, distinct child PTYs, and observed child exits. Shortcut-to-focus p50/p95/max was 65.7/88.7/102.6 ms, PTY bind was 122.8/154.0/159.7 ms, and first echo was 203.8/264.2/332.7 ms. Artifact SHA-256: 6d860cd0cd210f55f2349a197318248af488042117b20c08b6831160950c3277."
},
{
"date": "2026-08-30",
"runner": "local",
"platform": "macos",
"result": "passed",
"command": "ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
"durationSeconds": 72,
"summary": "At candidate d453ffcdb704764daced1b2917fddee7224389f0, the visible BrowserWindow and document completed 3/3 warmups followed by 20/20 measured cold-CWD cycles with every event present, distinct child PTYs, and observed child exits. Shortcut-to-focus p50/p95/max was 12.8/14.4/16.5 ms, PTY bind was 177.0/316.1/333.3 ms, and first echo was 268.6/627.4/710.7 ms. Artifact SHA-256: 9aef7fa842c732eb74f0066a77b2a0336e8f956a06ca61387f9999d6e76213cc."
}
],
"evidenceProcedure": "Historical artifact procedure: run the benchmark spec in a visible macOS Electron project from the primary worktree, complete 3 warmups followed by 20 measured cycles, save the JSON report, and record its SHA-256. Labels carry the product commit SHA; the stored pair predates the committed failure-report and exit-generation hardening, so rerun both product revisions with one committed harness before promotion.",
"runtimeBudget": {
"p95Seconds": 60,
"scope": "seven deterministic renderer and local-transport unit files"
"p95Seconds": 240,
"scope": "the full listed gate command set: seven deterministic renderer/local-transport unit files plus two opt-in 3-warmup/20-measured visible Electron benchmark invocations"
},
"flakeHistory": {
"status": "not-started",
"evidence": "The focused promise-barrier suite passes locally; routed CI and soak history have not started."
"evidence": "The focused promise-barrier suite and one visible baseline/candidate benchmark pair pass locally; routed CI and soak history have not started. The benchmark commands are recorded invocations from the primary worktree, with product SHAs encoded in their labels; they are evidence records rather than checkout scripts, and the stored pair predates the final harness hardening."
},
"redGreenEvidence": {
"status": "partial",
@@ -5842,18 +5874,20 @@
},
"performanceBudget": {
"required": true,
"evidence": "Pane creation and focus add no timer, polling, provider inventory, or subprocess work. CWD resolution remains one existing bounded request off the visible activation path, and detach admission adds only bounded map and record lookups. Pre-connect input, including an in-flight acknowledged write, is capped at 1,024 entries and a conservative UTF-16 ceiling derived from the existing terminal-input byte limit, drains through one worker in order, and clears on teardown or failed connect. Across 20 fresh-source dev cycles, click-to-focused-first-frame changed from baseline p50/p95/max 63.8/72.1/78.3 ms to 56.0/60.5/63.1 ms; all 20 focused. Cold PTY binding changed from 194.2/255.2/303.1 ms to 231.4/337.2/371.9 ms, so the interaction budget is tracked separately from shell readiness. A second 20-cycle steady-source run measured focus 51.1/88.1/109.1 ms and PTY binding 186.6/285.7/1158.6 ms; the 1.16 s readiness outlier still focused in 85.6 ms."
"evidence": "Pane creation and focus add no timer, polling, provider inventory, or subprocess work. CWD resolution remains one existing bounded request off the visible activation path, and detach admission adds only bounded map and record lookups. Pre-connect input, including an in-flight acknowledged write, is capped at 1,024 entries and a conservative UTF-16 ceiling derived from the existing terminal-input byte limit, drains through one worker in order, and clears on teardown or failed connect. In same-mode visible BrowserWindow runs with 3 warmups and n=20 measured cold-CWD cycles, baseline df14d1a2983d8339e788d0e521f1c4affd9c6d5f versus candidate d453ffcdb704764daced1b2917fddee7224389f0 changed shortcut-to-focus p50/p95/max from 65.7/88.7/102.6 ms to 12.8/14.4/16.5 ms. PTY bind changed from 122.8/154.0/159.7 ms to 177.0/316.1/333.3 ms, and first echo changed from 203.8/264.2/332.7 ms to 268.6/627.4/710.7 ms, so shell readiness regressed in this pair and remains diagnostic rather than part of the activation claim. The n=20 empirical p95 values are descriptive, are not a distribution guarantee, and are not CI-enforced."
},
"promotionCriteria": [
"Record complete red/green evidence for close, mixed-input ordering, metadata, and failure cleanup.",
"Collect 100 consecutive focused CI passes or 14 days without an unexplained flake.",
"Add a real-shortcut Electron sentinel with shortcut-to-focus, first-render, PTY-ready, and immediate typed-marker artifacts.",
"Collect physical SSH or WSL plus Linux and Windows evidence before claiming provider-complete coverage."
"Run the committed real-shortcut Electron benchmark in routed CI or soak before enforcing a latency budget.",
"Collect physical local-daemon, SSH or WSL plus Linux, Windows, and folder-workspace evidence before claiming provider-complete coverage."
],
"knownGaps": [
"Ad-hoc macOS Electron CDP proves real-control focus, pre-bind physical input, inherited cwd, close-before-bind cleanup, and rapid nested splits, but no committed Electron sentinel is registered yet.",
"Live daemon, SSH, WSL, Linux, and Windows split journeys have not run.",
"Local dev measurements establish a numeric first-focused-frame budget, but CI does not enforce it and PTY-ready latency remains diagnostic rather than blocking."
"The visible benchmark pair ran on one Apple-silicon macOS host with a synthetic POSIX echo shell and a git-backed workspace; its n=20 empirical p95 is descriptive and not CI-enforced.",
"No physical local-daemon, SSH, WSL, Linux, Windows, or folder-workspace latency journey has run; the synthetic fixture is currently skipped on Windows because it requires a POSIX shell.",
"The stored artifact pair predates the final harness cleanup/reporting and exit-generation hardening; labels and immutable hashes preserve product provenance, but both product revisions should be rerun with one committed harness before promotion.",
"No forced-failure visible benchmark artifact has been recorded; the invalid-report path is exercised by the harness catch/cleanup logic but remains unsoaked.",
"PTY-bind and first-echo readiness regressed in the recorded candidate pair and remain diagnostic rather than blocking the activation-ordering claim."
],
"demotionRule": "Keep experimental or demote if pane activation waits on CWD, a deferred split can detach before PTY bind, detached cwd is lost, input reorders or remains pending after cleanup, a closed pane can spawn, stale retirement kills a newer same-ID owner, remote-runtime delegation creates a competing local pane, or the focused suite flakes without an identified product or harness cause."
},
@@ -95,6 +95,66 @@ describe('benchmark artifact comparison', () => {
})
})
it('compares terminal split headline metrics in milliseconds', () => {
const dir = makeTempDir()
const baselinePath = writeArtifact(dir, 'split-baseline.json', {
label: 'split baseline',
headlineMs: {
shortcutToFocusP50: 284.2,
shortcutToFocusP95: 676.3
}
})
const candidatePath = writeArtifact(dir, 'split-candidate.json', {
label: 'split candidate',
headlineMs: {
shortcutToFocusP50: 12.7,
shortcutToFocusP95: 13.7
}
})
const comparison = comparePaths(baselinePath, candidatePath)
expect(comparison.baseline.kind).toBe('terminal-split-activation')
expect(comparison.metrics).toEqual(
expect.arrayContaining([
expect.objectContaining({
key: 'shortcutToFocusP50',
unit: 'ms',
baseline: 284.2,
candidate: 12.7,
status: 'improved'
}),
expect.objectContaining({
key: 'shortcutToFocusP95',
unit: 'ms',
baseline: 676.3,
candidate: 13.7,
status: 'improved'
})
])
)
})
it('rejects invalid benchmark artifacts before comparing partial metrics', () => {
const dir = makeTempDir()
const baselinePath = writeArtifact(dir, 'split-invalid.json', {
label: 'invalid split',
status: 'failed',
valid: false,
headlineMs: { shortcutToFocusP50: 0 }
})
const candidatePath = writeArtifact(dir, 'split-valid.json', {
label: 'valid split',
status: 'passed',
valid: true,
headlineMs: { shortcutToFocusP50: 10 }
})
expect(() => comparePaths(baselinePath, candidatePath)).toThrow(
'split-invalid.json: benchmark artifact is marked invalid'
)
})
it('compares numeric Playwright annotation metrics and omits metadata fields', () => {
const dir = makeTempDir()
const baselinePath = writeArtifact(dir, 'baseline-playwright.json', {
+13 -1
View File
@@ -74,6 +74,9 @@ export function readBenchmarkArtifact(path) {
}
export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifact(path)) {
if (artifact?.valid === false || artifact?.status === 'failed') {
throw new Error(`${path}: benchmark artifact is marked invalid`)
}
if (artifact?.summaryMedianMs != null) {
return normalizeNumericObject(path, artifact, 'startup', artifact.summaryMedianMs, () => 'ms')
}
@@ -82,6 +85,15 @@ export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifac
key.endsWith('Count') || key.endsWith('After') ? 'count' : 'ms'
)
}
if (artifact?.headlineMs != null) {
return normalizeNumericObject(
path,
artifact,
'terminal-split-activation',
artifact.headlineMs,
() => 'ms'
)
}
if (artifact?.suites != null) {
return normalizePlaywrightArtifact(path, artifact)
}
@@ -89,7 +101,7 @@ export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifac
return normalizeSummaryArtifact(path, artifact)
}
throw new Error(
`${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary`
`${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, headlineMs, Playwright suites, or top-level summary`
)
}
@@ -0,0 +1,186 @@
import { summarizeLatencies, type LatencyDistribution } from './codex-composer-echo-latency-probe'
export type RendererPhaseStamps = {
marker: string
sourcePaneId: number
sourcePtyId: string
newPaneId: number | null
newPtyId: string | null
keydownAtMs: number | null
focusAtMs: number | null
ptyBoundAtMs: number | null
inputAtMs: number | null
firstEchoAtMs: number | null
}
export type SplitLatencySample = RendererPhaseStamps & {
phase: 'warmup' | 'measured'
iteration: number
completedWithinTimeout: boolean
paneCountAfterProbe: number
ptyExitObserved: boolean
cleanupError: string | null
shortcutToFocusMs: number | null
shortcutToPtyBindMs: number | null
shortcutToFirstEchoMs: number | null
ptyBindToFirstEchoMs: number | null
inputToFirstEchoMs: number | null
missing: string[]
success: boolean
}
export type SampleSummary = {
counts: {
requested: number
attempted: number
success: number
missing: number
unattempted: number
missingEvents: {
keydown: number
focus: number
ptyBind: number
input: number
firstEcho: number
paneCount: number
ptyIdentity: number
ptyExit: number
cleanup: number
}
}
distributions: {
shortcutToFocusMs: LatencyDistribution
shortcutToPtyBindMs: LatencyDistribution
shortcutToFirstEchoMs: LatencyDistribution
ptyBindToFirstEchoMs: LatencyDistribution
inputToFirstEchoMs: LatencyDistribution
}
}
export type BrowserWindowState = {
browserWindowVisible: boolean
windowCount: number
}
export type TerminalSplitLatencyReportConfig = {
warmupCycles: number
measuredCycles: number
maxMeasuredCycles: number
testTimeoutMs: number
splitChord: string
closeChord: string
sampleTimeoutMs: number
cleanupTimeoutMs: number
processCwdCacheExpiryWaitMs: number
}
export type BenchmarkReportResult = {
report: Record<string, unknown>
warmupSummary: SampleSummary
measuredSummary: SampleSummary
}
function valuesFor(
samples: SplitLatencySample[],
key:
| 'shortcutToFocusMs'
| 'shortcutToPtyBindMs'
| 'shortcutToFirstEchoMs'
| 'ptyBindToFirstEchoMs'
| 'inputToFirstEchoMs'
): number[] {
return samples.flatMap((sample) => {
const value = sample[key]
return value === null ? [] : [value]
})
}
export function summarizeSamples(samples: SplitLatencySample[], requested: number): SampleSummary {
const missingEvents = {
keydown: samples.filter((sample) => sample.keydownAtMs === null).length,
focus: samples.filter((sample) => sample.focusAtMs === null).length,
ptyBind: samples.filter((sample) => sample.ptyBoundAtMs === null).length,
input: samples.filter((sample) => sample.inputAtMs === null).length,
firstEcho: samples.filter((sample) => sample.firstEchoAtMs === null).length,
paneCount: samples.filter((sample) => sample.paneCountAfterProbe !== 2).length,
ptyIdentity: samples.filter((sample) => sample.newPtyId === sample.sourcePtyId).length,
ptyExit: samples.filter((sample) => !sample.ptyExitObserved).length,
cleanup: samples.filter((sample) => sample.cleanupError !== null).length
}
const success = samples.filter((sample) => sample.success).length
return {
counts: {
requested,
attempted: samples.length,
success,
missing: samples.length - success,
unattempted: Math.max(0, requested - samples.length),
missingEvents
},
distributions: {
shortcutToFocusMs: summarizeLatencies(valuesFor(samples, 'shortcutToFocusMs')),
shortcutToPtyBindMs: summarizeLatencies(valuesFor(samples, 'shortcutToPtyBindMs')),
shortcutToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'shortcutToFirstEchoMs')),
ptyBindToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'ptyBindToFirstEchoMs')),
inputToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'inputToFirstEchoMs'))
}
}
}
export function buildBenchmarkReport(args: {
label: string
headfulRun: boolean
windowState: BrowserWindowState
documentVisibility: string
testRepoPath: string
warmupSamples: SplitLatencySample[]
measuredSamples: SplitLatencySample[]
abortError: Error | null
config: TerminalSplitLatencyReportConfig
}): BenchmarkReportResult {
const warmupSummary = summarizeSamples(args.warmupSamples, args.config.warmupCycles)
const measuredSummary = summarizeSamples(args.measuredSamples, args.config.measuredCycles)
const runComplete =
warmupSummary.counts.success === args.config.warmupCycles &&
measuredSummary.counts.success === args.config.measuredCycles &&
args.abortError === null
const headlineMs = runComplete
? {
shortcutToFocusP50: measuredSummary.distributions.shortcutToFocusMs.p50,
shortcutToFocusP95: measuredSummary.distributions.shortcutToFocusMs.p95,
shortcutToFocusMax: measuredSummary.distributions.shortcutToFocusMs.max,
shortcutToPtyBindP50: measuredSummary.distributions.shortcutToPtyBindMs.p50,
shortcutToPtyBindP95: measuredSummary.distributions.shortcutToPtyBindMs.p95,
shortcutToPtyBindMax: measuredSummary.distributions.shortcutToPtyBindMs.max,
shortcutToFirstEchoP50: measuredSummary.distributions.shortcutToFirstEchoMs.p50,
shortcutToFirstEchoP95: measuredSummary.distributions.shortcutToFirstEchoMs.p95,
shortcutToFirstEchoMax: measuredSummary.distributions.shortcutToFirstEchoMs.max
}
: null
return {
report: {
schemaVersion: 1,
benchmark: 'terminal-split-activation-latency',
label: args.label,
status: runComplete ? 'passed' : 'failed',
valid: runComplete,
abortReason: args.abortError?.message ?? null,
timestamp: new Date().toISOString(),
platform: process.platform,
arch: process.arch,
nodeVersion: process.version,
headful: args.headfulRun,
browserWindowVisible: args.windowState.browserWindowVisible,
documentVisibility: args.documentVisibility,
testRepoPath: args.testRepoPath,
config: args.config,
headlineMs,
warmupSummary,
measuredSummary,
warmupSamples: args.warmupSamples,
measuredSamples: args.measuredSamples
},
warmupSummary,
measuredSummary
}
}
@@ -0,0 +1,720 @@
import { randomUUID } from 'node:crypto'
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import path from 'node:path'
import type { Page, TestInfo } from '@stablyai/playwright-test'
import { test, expect } from './helpers/orca-app'
import {
countVisibleTerminalPanes,
focusActiveTerminalInput,
sendToTerminal,
waitForActivePanePtyId,
waitForActiveTerminalManager,
waitForPaneCount,
waitForTerminalOutput
} from './helpers/terminal'
import { ensureTerminalVisible, waitForActiveWorktree, waitForSessionReady } from './helpers/store'
import {
buildBenchmarkReport,
type BrowserWindowState,
type RendererPhaseStamps,
type SplitLatencySample,
type TerminalSplitLatencyReportConfig
} from './terminal-split-activation-latency-report'
const BENCH_ENABLED = process.env.ORCA_TERMINAL_SPLIT_LATENCY_BENCH === '1'
const BENCH_LABEL = process.env.ORCA_TERMINAL_SPLIT_LATENCY_LABEL?.trim() || 'local'
const BENCH_OUTPUT_PATH = process.env.ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT?.trim() || null
const WARMUP_CYCLES = 3
const MIN_MEASURED_CYCLES = 20
const MAX_MEASURED_CYCLES = 200
const SAMPLE_TIMEOUT_MS = 15_000
const CLEANUP_TIMEOUT_MS = 15_000
const CONFIRM_CLICK_TIMEOUT_MS = 2_000
const BENCH_SETUP_TIMEOUT_MS = 5 * 60 * 1000
// Why: process-cwd caches each pid for 1500ms; this wait isolates cold lookups, not correctness.
const PROCESS_CWD_CACHE_EXPIRY_WAIT_MS = 1_650
const SOURCE_READY_MARKER = 'ORCA_SPLIT_LATENCY_SOURCE_READY'
const IS_MAC = process.platform === 'darwin'
const SPLIT_CHORD = IS_MAC ? 'Meta+d' : 'Control+Shift+d'
const CLOSE_CHORD = IS_MAC ? 'Meta+w' : 'Control+w'
function readPositiveInt(name: string, fallback: number): number {
const value = Number(process.env[name])
return Number.isInteger(value) && value > 0 ? value : fallback
}
const MEASURED_CYCLES = Math.min(
MAX_MEASURED_CYCLES,
Math.max(
MIN_MEASURED_CYCLES,
readPositiveInt('ORCA_TERMINAL_SPLIT_LATENCY_CYCLES', MIN_MEASURED_CYCLES)
)
)
const BENCH_TIMEOUT_MS =
BENCH_SETUP_TIMEOUT_MS +
WARMUP_CYCLES * (SAMPLE_TIMEOUT_MS + 4 * CLEANUP_TIMEOUT_MS + CONFIRM_CLICK_TIMEOUT_MS) +
MEASURED_CYCLES *
(SAMPLE_TIMEOUT_MS +
4 * CLEANUP_TIMEOUT_MS +
CONFIRM_CLICK_TIMEOUT_MS +
PROCESS_CWD_CACHE_EXPIRY_WAIT_MS)
const REPORT_CONFIG = {
warmupCycles: WARMUP_CYCLES,
measuredCycles: MEASURED_CYCLES,
maxMeasuredCycles: MAX_MEASURED_CYCLES,
testTimeoutMs: BENCH_TIMEOUT_MS,
splitChord: SPLIT_CHORD,
closeChord: CLOSE_CHORD,
sampleTimeoutMs: SAMPLE_TIMEOUT_MS,
cleanupTimeoutMs: CLEANUP_TIMEOUT_MS,
processCwdCacheExpiryWaitMs: PROCESS_CWD_CACHE_EXPIRY_WAIT_MS
} satisfies TerminalSplitLatencyReportConfig
type RendererProbe = {
report: () => RendererPhaseStamps
dispose: () => void
}
type SplitLatencyProbeWindow = Window & {
__terminalSplitLatencyProbe?: RendererProbe
__terminalSplitLatencyPtyExitIds?: string[]
__terminalSplitLatencyPtyExitDispose?: () => void
}
function createEchoShellFixture(): { root: string; shellPath: string } {
const root = mkdtempSync(path.join(tmpdir(), 'orca-split-latency-'))
const shellPath = path.join(root, 'split-echo-shell')
writeFileSync(
shellPath,
[
'#!/bin/sh',
'stty raw -echo',
'dd bs=1 count=1 of=/dev/null 2>/dev/null',
`printf '%s' '${SOURCE_READY_MARKER}'`,
'exec /bin/cat',
''
].join('\n'),
'utf8'
)
chmodSync(shellPath, 0o755)
return { root, shellPath }
}
async function createSourceTab(
page: Page,
shellOverride: string
): Promise<{ tabId: string; ptyId: string }> {
const tabId = await page.evaluate((shellOverride) => {
const store = window.__store
if (!store) {
throw new Error('Store unavailable')
}
const state = store.getState()
const worktreeId = state.activeWorktreeId
if (!worktreeId) {
throw new Error('No active worktree')
}
const tab = state.createTab(worktreeId, undefined, shellOverride, { activate: true })
store.getState().setActiveTab(tab.id)
store.getState().setActiveTabType('terminal')
return tab.id
}, shellOverride)
await waitForActiveTerminalManager(page, 30_000)
await waitForPaneCount(page, 1, 30_000)
const ptyId = await waitForActivePanePtyId(page, 30_000)
await sendToTerminal(page, ptyId, '\r')
await waitForTerminalOutput(page, SOURCE_READY_MARKER, 30_000)
return { tabId, ptyId }
}
async function readActivePaneId(page: Page, tabId: string): Promise<number> {
const paneId = await page.evaluate((tabId) => {
const manager = window.__paneManagers?.get(tabId)
return manager?.getActivePane?.()?.id ?? null
}, tabId)
if (paneId === null) {
throw new Error(`No active pane for source tab ${tabId}`)
}
return paneId
}
async function installRendererProbe(
page: Page,
args: {
tabId: string
sourcePaneId: number
sourcePtyId: string
marker: string
readyMarker: string
isMac: boolean
}
): Promise<void> {
await page.evaluate(({ tabId, sourcePaneId, sourcePtyId, marker, readyMarker, isMac }) => {
const targetWindow = window as SplitLatencyProbeWindow
targetWindow.__terminalSplitLatencyProbe?.dispose()
const stamps: RendererPhaseStamps = {
marker,
sourcePaneId,
sourcePtyId,
newPaneId: null,
newPtyId: null,
keydownAtMs: null,
focusAtMs: null,
ptyBoundAtMs: null,
inputAtMs: null,
firstEchoAtMs: null
}
let ptyBindingObserver: MutationObserver | null = null
let parsedDisposable: { dispose: () => void } | null = null
let fixtureReady = false
let markerFeedQueued = false
const originalStopImmediatePropagation = Event.prototype.stopImmediatePropagation
const onKeyDown = (event: KeyboardEvent): void => {
const matches = isMac
? event.code === 'KeyD' && event.metaKey && !event.shiftKey && !event.altKey
: event.code === 'KeyD' && event.ctrlKey && event.shiftKey && !event.altKey
if (matches && stamps.keydownAtMs === null) {
stamps.keydownAtMs = performance.now()
}
}
const patchedStopImmediatePropagation = function (this: Event): void {
// Why: terminal shortcuts stop same-target listeners before split work starts.
if (this instanceof KeyboardEvent) {
onKeyDown(this)
}
originalStopImmediatePropagation.call(this)
}
const onFocusIn = (event: FocusEvent): void => {
if (stamps.keydownAtMs === null || stamps.focusAtMs !== null) {
return
}
const target = event.target
if (!(target instanceof HTMLElement) || !target.matches('.xterm-helper-textarea')) {
return
}
const paneElement = target.closest<HTMLElement>('.pane[data-pane-id]')
const manager = window.__paneManagers?.get(tabId)
const pane = manager?.getPanes?.().find((candidate) => candidate.container === paneElement)
if (!pane || pane.id === sourcePaneId) {
return
}
stamps.newPaneId = pane.id
stamps.focusAtMs = performance.now()
const maybeFeedMarker = (): void => {
if (!fixtureReady || stamps.ptyBoundAtMs === null || markerFeedQueued) {
return
}
markerFeedQueued = true
queueMicrotask(() => {
stamps.inputAtMs = performance.now()
pane.terminal.input(marker, true)
})
}
const observeParsedOutput = (): void => {
const buffer = pane.terminal.buffer.active
let text = ''
for (let row = 0; row < buffer.length; row += 1) {
text += buffer.getLine(row)?.translateToString(true) ?? ''
}
if (!fixtureReady && text.includes(readyMarker)) {
fixtureReady = true
maybeFeedMarker()
}
if (stamps.firstEchoAtMs === null && text.includes(marker)) {
stamps.firstEchoAtMs = performance.now()
}
}
parsedDisposable = pane.terminal.onWriteParsed(observeParsedOutput)
observeParsedOutput()
const observePtyBinding = (): void => {
const ptyId = pane.container.dataset.ptyId
if (!ptyId || stamps.ptyBoundAtMs !== null) {
return
}
stamps.newPtyId = ptyId
stamps.ptyBoundAtMs = performance.now()
ptyBindingObserver?.disconnect()
queueMicrotask(() => pane.terminal.input('\r', true))
}
if (pane.container.dataset.ptyId) {
observePtyBinding()
return
}
ptyBindingObserver = new MutationObserver(observePtyBinding)
ptyBindingObserver.observe(pane.container, {
attributes: true,
attributeFilter: ['data-pty-id']
})
}
Event.prototype.stopImmediatePropagation = patchedStopImmediatePropagation
window.addEventListener('keydown', onKeyDown, { capture: true })
document.addEventListener('focusin', onFocusIn, { capture: true })
targetWindow.__terminalSplitLatencyProbe = {
report: () => ({ ...stamps }),
dispose: () => {
window.removeEventListener('keydown', onKeyDown, { capture: true })
document.removeEventListener('focusin', onFocusIn, { capture: true })
if (Event.prototype.stopImmediatePropagation === patchedStopImmediatePropagation) {
Event.prototype.stopImmediatePropagation = originalStopImmediatePropagation
}
ptyBindingObserver?.disconnect()
parsedDisposable?.dispose()
}
}
}, args)
}
async function waitForRendererProbe(page: Page): Promise<boolean> {
try {
await page.waitForFunction(
() =>
(window as SplitLatencyProbeWindow).__terminalSplitLatencyProbe?.report().firstEchoAtMs !==
null,
null,
{ timeout: SAMPLE_TIMEOUT_MS }
)
return true
} catch {
return false
}
}
async function collectRendererProbe(page: Page): Promise<RendererPhaseStamps> {
return page.evaluate(() => {
const targetWindow = window as SplitLatencyProbeWindow
const probe = targetWindow.__terminalSplitLatencyProbe
if (!probe) {
throw new Error('Terminal split latency probe was not installed')
}
const report = probe.report()
probe.dispose()
delete targetWindow.__terminalSplitLatencyProbe
return report
})
}
function elapsed(start: number | null, end: number | null): number | null {
return start === null || end === null ? null : end - start
}
function createSample(
phase: SplitLatencySample['phase'],
iteration: number,
stamps: RendererPhaseStamps,
completedWithinTimeout: boolean,
paneCountAfterProbe: number,
ptyExitObserved: boolean,
cleanupError: string | null
): SplitLatencySample {
const missing = [
...(stamps.keydownAtMs === null ? ['keydown'] : []),
...(stamps.focusAtMs === null ? ['focus'] : []),
...(stamps.ptyBoundAtMs === null ? ['pty-bind'] : []),
...(stamps.inputAtMs === null ? ['input'] : []),
...(stamps.firstEchoAtMs === null ? ['first-echo'] : []),
...(stamps.newPtyId === stamps.sourcePtyId ? ['pty-identity'] : []),
...(paneCountAfterProbe !== 2 ? [`pane-count:${paneCountAfterProbe}`] : []),
...(!ptyExitObserved ? ['pty-exit'] : []),
...(cleanupError ? ['cleanup'] : [])
]
return {
...stamps,
phase,
iteration,
completedWithinTimeout,
paneCountAfterProbe,
ptyExitObserved,
cleanupError,
shortcutToFocusMs: elapsed(stamps.keydownAtMs, stamps.focusAtMs),
shortcutToPtyBindMs: elapsed(stamps.keydownAtMs, stamps.ptyBoundAtMs),
shortcutToFirstEchoMs: elapsed(stamps.keydownAtMs, stamps.firstEchoAtMs),
ptyBindToFirstEchoMs: elapsed(stamps.ptyBoundAtMs, stamps.firstEchoAtMs),
inputToFirstEchoMs: elapsed(stamps.inputAtMs, stamps.firstEchoAtMs),
missing,
success: completedWithinTimeout && missing.length === 0
}
}
async function closeSplitsAndRefocusSource(
page: Page,
tabId: string,
sourcePaneId: number,
closedPtyIds: string[]
): Promise<{ closeCompletedAt: number; ptyExitObserved: boolean; cleanupError: string | null }> {
let paneCount = await countVisibleTerminalPanes(page)
if (paneCount < 1) {
throw new Error('Source terminal disappeared during split benchmark')
}
while (paneCount > 1) {
const expectedCount = paneCount - 1
await focusActiveTerminalInput(page)
await page.keyboard.press(CLOSE_CHORD)
const confirmButton = page
.locator(
'[data-slot="dialog-content"][data-state="open"] [data-slot="dialog-footer"] [data-slot="button"][data-variant="destructive"]'
)
.last()
await expect
.poll(
async () => {
if (await confirmButton.isVisible().catch(() => false)) {
await confirmButton.click({ timeout: CONFIRM_CLICK_TIMEOUT_MS })
}
return countVisibleTerminalPanes(page)
},
{
timeout: CLEANUP_TIMEOUT_MS,
message: `Split pane did not close to ${expectedCount} pane(s)`
}
)
.toBe(expectedCount)
paneCount = expectedCount
}
await waitForPaneCount(page, 1, CLEANUP_TIMEOUT_MS)
await expect
.poll(
() =>
page.evaluate(
({ tabId, sourcePaneId }) =>
window.__paneManagers?.get(tabId)?.getActivePane?.()?.id === sourcePaneId,
{ tabId, sourcePaneId }
),
{
timeout: CLEANUP_TIMEOUT_MS,
message: 'Source pane did not regain active ownership after close'
}
)
.toBe(true)
const ptyExitResults = await Promise.all(
closedPtyIds.map(async (ptyId) => ({ ptyId, observed: await waitForPtyExit(page, ptyId) }))
)
const missingPtyExitIds = ptyExitResults
.filter((result) => !result.observed)
.map((result) => result.ptyId)
await focusActiveTerminalInput(page)
const cleanupError =
closedPtyIds.length === 0
? 'Split cleanup could not identify a child PTY to verify its exit'
: missingPtyExitIds.length > 0
? `Closed split PTY did not emit exit: ${missingPtyExitIds.join(', ')}`
: null
return {
closeCompletedAt: Date.now(),
ptyExitObserved: cleanupError === null,
cleanupError
}
}
async function readChildPtyIds(page: Page, tabId: string, sourcePtyId: string): Promise<string[]> {
return page.evaluate(
({ tabId, sourcePtyId }) => {
const manager = window.__paneManagers?.get(tabId)
return (manager?.getPanes?.() ?? [])
.map((pane) => pane.container.dataset.ptyId ?? null)
.filter((ptyId): ptyId is string => Boolean(ptyId) && ptyId !== sourcePtyId)
},
{ tabId, sourcePtyId }
)
}
async function runSplitCycle(
page: Page,
args: {
tabId: string
sourcePaneId: number
sourcePtyId: string
phase: SplitLatencySample['phase']
iteration: number
}
): Promise<{ sample: SplitLatencySample; closeCompletedAt: number; fatalError: Error | null }> {
const marker = `ORCA_SPLIT_ECHO_${args.phase}_${args.iteration}_${randomUUID().replaceAll('-', '')}`
await focusActiveTerminalInput(page)
// Prevent an ID reused by a later PTY lifetime from matching an earlier exit.
await resetPtyExitProbe(page)
await installRendererProbe(page, {
tabId: args.tabId,
sourcePaneId: args.sourcePaneId,
sourcePtyId: args.sourcePtyId,
marker,
readyMarker: SOURCE_READY_MARKER,
isMac: IS_MAC
})
await page.keyboard.press(SPLIT_CHORD)
const completedWithinTimeout = await waitForRendererProbe(page)
const stamps = await collectRendererProbe(page)
let paneCountAfterProbe = -1
let closeCompletedAt = Date.now()
let ptyExitObserved = false
let cleanupError: Error | null = null
try {
paneCountAfterProbe = await countVisibleTerminalPanes(page)
const childPtyIds = await readChildPtyIds(page, args.tabId, args.sourcePtyId)
const closeResult = await closeSplitsAndRefocusSource(
page,
args.tabId,
args.sourcePaneId,
childPtyIds
)
closeCompletedAt = closeResult.closeCompletedAt
ptyExitObserved = closeResult.ptyExitObserved
cleanupError = closeResult.cleanupError ? new Error(closeResult.cleanupError) : null
} catch (error) {
cleanupError = error instanceof Error ? error : new Error(String(error))
closeCompletedAt = Date.now()
}
const sample = createSample(
args.phase,
args.iteration,
stamps,
completedWithinTimeout,
paneCountAfterProbe,
ptyExitObserved,
cleanupError?.message ?? null
)
return { sample, closeCompletedAt, fatalError: cleanupError }
}
async function waitForColdProcessCwdLookup(
page: Page,
priorCloseCompletedAt: number
): Promise<void> {
const remaining = PROCESS_CWD_CACHE_EXPIRY_WAIT_MS - (Date.now() - priorCloseCompletedAt)
if (remaining > 0) {
await page.waitForTimeout(remaining)
}
expect(Date.now() - priorCloseCompletedAt).toBeGreaterThanOrEqual(
PROCESS_CWD_CACHE_EXPIRY_WAIT_MS
)
}
function logSample(sample: SplitLatencySample): void {
console.log(
`[terminal-split-activation-latency] ${sample.phase} ${sample.iteration + 1} ` +
`success=${sample.success} missing=${sample.missing.join(',') || 'none'}`
)
}
async function installPtyExitProbe(page: Page): Promise<void> {
await page.evaluate(() => {
const targetWindow = window as SplitLatencyProbeWindow
targetWindow.__terminalSplitLatencyPtyExitDispose?.()
targetWindow.__terminalSplitLatencyPtyExitIds = []
targetWindow.__terminalSplitLatencyPtyExitDispose = window.api.pty.onExit(({ id }) => {
const ids = targetWindow.__terminalSplitLatencyPtyExitIds ?? []
if (!ids.includes(id)) {
ids.push(id)
}
targetWindow.__terminalSplitLatencyPtyExitIds = ids
})
})
}
async function resetPtyExitProbe(page: Page): Promise<void> {
await page.evaluate(() => {
;(window as SplitLatencyProbeWindow).__terminalSplitLatencyPtyExitIds = []
})
}
async function waitForPtyExit(page: Page, ptyId: string): Promise<boolean> {
try {
await expect
.poll(
() =>
page.evaluate((expectedPtyId) => {
const targetWindow = window as SplitLatencyProbeWindow
return targetWindow.__terminalSplitLatencyPtyExitIds?.includes(expectedPtyId) ?? false
}, ptyId),
{ timeout: CLEANUP_TIMEOUT_MS, message: `Closed split PTY did not emit exit: ${ptyId}` }
)
.toBe(true)
return true
} catch {
return false
}
}
async function disposePtyExitProbe(page: Page): Promise<void> {
await page.evaluate(() => {
const targetWindow = window as SplitLatencyProbeWindow
targetWindow.__terminalSplitLatencyPtyExitDispose?.()
delete targetWindow.__terminalSplitLatencyPtyExitDispose
delete targetWindow.__terminalSplitLatencyPtyExitIds
})
}
async function attachReport(testInfo: TestInfo, report: Record<string, unknown>): Promise<void> {
const body = `${JSON.stringify(report, null, 2)}\n`
await testInfo.attach('terminal-split-activation-latency.json', {
body,
contentType: 'application/json'
})
if (BENCH_OUTPUT_PATH) {
try {
writeFileSync(BENCH_OUTPUT_PATH, body, 'utf8')
} catch (error) {
const message = error instanceof Error ? error.message : String(error)
console.error(
`[terminal-split-activation-latency] unable to write ${BENCH_OUTPUT_PATH}: ${message}`
)
}
}
console.log(`[terminal-split-activation-latency] ${JSON.stringify(report)}`)
}
test.describe('Terminal split activation latency benchmark @headful', () => {
test.skip(!BENCH_ENABLED, 'One-off benchmark: set ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1')
test.skip(process.platform === 'win32', 'Deterministic echo-shell fixture is POSIX-only')
test.setTimeout(BENCH_TIMEOUT_MS)
test('records shortcut, focus, PTY bind, and first echo phases', async ({
electronApp,
orcaPage,
testRepoPath
}, testInfo) => {
const headfulRun =
process.env.ORCA_E2E_FORCE_HEADFUL === '1' || testInfo.project.metadata.orcaHeadful === true
const windowState: BrowserWindowState = {
browserWindowVisible: false,
windowCount: 0
}
let documentVisibility = 'unavailable'
let fixture: { root: string; shellPath: string } | null = null
const warmupSamples: SplitLatencySample[] = []
const measuredSamples: SplitLatencySample[] = []
let abortError: Error | null = null
let reportAttached = false
try {
expect(headfulRun, 'The latency benchmark must run with a visible BrowserWindow').toBe(true)
const observedWindowState = await electronApp.evaluate(({ BrowserWindow }) => ({
browserWindowVisible: BrowserWindow.getAllWindows()[0]?.isVisible() ?? false,
windowCount: BrowserWindow.getAllWindows().length
}))
windowState.browserWindowVisible = observedWindowState.browserWindowVisible
windowState.windowCount = observedWindowState.windowCount
expect(windowState.windowCount).toBeGreaterThan(0)
expect(windowState.browserWindowVisible).toBe(true)
await expect
.poll(
async () => {
documentVisibility = await orcaPage.evaluate(() => document.visibilityState)
return documentVisibility
},
{
timeout: 15_000,
message: 'Visible latency benchmark renderer remained hidden'
}
)
.toBe('visible')
await waitForSessionReady(orcaPage)
await waitForActiveWorktree(orcaPage)
await ensureTerminalVisible(orcaPage)
fixture = createEchoShellFixture()
const source = await createSourceTab(orcaPage, fixture.shellPath)
const { tabId, ptyId: sourcePtyId } = source
const sourcePaneId = await readActivePaneId(orcaPage, tabId)
await installPtyExitProbe(orcaPage)
let priorCloseCompletedAt = Date.now()
for (let iteration = 0; iteration < WARMUP_CYCLES; iteration += 1) {
const result = await runSplitCycle(orcaPage, {
tabId,
sourcePaneId,
sourcePtyId,
phase: 'warmup',
iteration
})
warmupSamples.push(result.sample)
logSample(result.sample)
priorCloseCompletedAt = result.closeCompletedAt
if (result.fatalError) {
abortError = result.fatalError
break
}
}
for (let iteration = 0; iteration < MEASURED_CYCLES && abortError === null; iteration += 1) {
await waitForColdProcessCwdLookup(orcaPage, priorCloseCompletedAt)
const result = await runSplitCycle(orcaPage, {
tabId,
sourcePaneId,
sourcePtyId,
phase: 'measured',
iteration
})
measuredSamples.push(result.sample)
logSample(result.sample)
priorCloseCompletedAt = result.closeCompletedAt
if (result.fatalError) {
abortError = result.fatalError
}
}
documentVisibility = await orcaPage
.evaluate(() => document.visibilityState)
.catch(() => 'unavailable' as const)
const reportResult = buildBenchmarkReport({
label: BENCH_LABEL,
headfulRun,
windowState,
documentVisibility,
testRepoPath,
warmupSamples,
measuredSamples,
abortError,
config: REPORT_CONFIG
})
await attachReport(testInfo, reportResult.report)
reportAttached = true
testInfo.annotations.push({
type: 'terminal-split-activation-latency',
description:
`success=${reportResult.measuredSummary.counts.success}/${MEASURED_CYCLES} ` +
`focusP50=${reportResult.measuredSummary.distributions.shortcutToFocusMs.p50.toFixed(1)}ms ` +
`echoP50=${reportResult.measuredSummary.distributions.shortcutToFirstEchoMs.p50.toFixed(1)}ms`
})
if (abortError) {
throw abortError
}
expect(reportResult.warmupSummary.counts.success).toBe(WARMUP_CYCLES)
expect(reportResult.measuredSummary.counts.success).toBe(MEASURED_CYCLES)
} catch (error) {
const failure = error instanceof Error ? error : new Error(String(error))
if (!reportAttached) {
abortError ??= failure
const failureReport = buildBenchmarkReport({
label: BENCH_LABEL,
headfulRun,
windowState,
documentVisibility,
testRepoPath,
warmupSamples,
measuredSamples,
abortError,
config: REPORT_CONFIG
})
await attachReport(testInfo, failureReport.report).catch((attachError) => {
const message = attachError instanceof Error ? attachError.message : String(attachError)
console.error(
`[terminal-split-activation-latency] unable to attach failure report: ${message}`
)
})
}
throw error
} finally {
await disposePtyExitProbe(orcaPage).catch(() => undefined)
if (fixture) {
rmSync(fixture.root, { recursive: true, force: true })
}
}
})
})