mirror of
https://github.com/stablyai/orca.git
synced 2026-10-02 16:02:15 +00:00
test(terminal): add visible split activation latency benchmark
This commit is contained in:
@@ -5735,12 +5735,14 @@
|
||||
"providers": ["local", "local-daemon", "ssh", "wsl", "remote-runtime"],
|
||||
"coveredPlatforms": ["macos"],
|
||||
"coveredProviders": ["local", "remote-runtime"],
|
||||
"coverageNotes": "Deterministic renderer contracts hold CWD resolution behind an explicit promise, require the new pane to be created synchronously, and prove that close cancels the pending connection. Detach contracts reject cwd-pending and cwd-resolved deferred splits before PTY bind without mutation, carry resolved cwd for other unbound panes, and preserve persisted or live PTY handoff. Local transport contracts exercise the real bounded pre-connect buffer and one live input FIFO across ordinary, acknowledged, and immediate writes, concurrent flushes, in-flight teardown, late spawn success or failure, attach failure, failed spawn, same-id reuse, stale-spawn retirement ownership, destroy, and mutable recovery metadata. A mocked direct-SSH authority-rotation contract proves a rejected stale spawn releases its deferred-CWD fence. Remote-runtime coverage proves delegation remains host-owned; physical daemon, SSH, WSL, Linux, and Windows journeys remain gaps.",
|
||||
"coverageNotes": "Deterministic renderer contracts hold CWD resolution behind an explicit promise, require the new pane to be created synchronously, and prove that close cancels the pending connection. Detach contracts reject cwd-pending and cwd-resolved deferred splits before PTY bind without mutation, carry resolved cwd for other unbound panes, and preserve persisted or live PTY handoff. Local transport contracts exercise the real bounded pre-connect buffer and one live input FIFO across ordinary, acknowledged, and immediate writes, concurrent flushes, in-flight teardown, late spawn success or failure, attach failure, failed spawn, same-id reuse, stale-spawn retirement ownership, destroy, and mutable recovery metadata. A mocked direct-SSH authority-rotation contract proves a rejected stale spawn releases its deferred-CWD fence. The committed headful Electron benchmark drives the real platform shortcut in a visible BrowserWindow and document for 3 warmups followed by 20 measured cold-CWD cycles, requiring a distinct child PTY, first echo, and observed child pty:exit before the next cycle. Remote-runtime coverage proves delegation remains host-owned; physical local-daemon, SSH, WSL, Linux, Windows, and folder-workspace latency journeys remain gaps.",
|
||||
"motivatingLinks": ["https://github.com/stablyai/orca/commit/572ed1a8882"],
|
||||
"invariant": "A terminal split creates and activates its renderer pane before an inherited-CWD lookup settles, starts its PTY only after the resolved directory is available, and cannot be externally detached while that deferred spawn remains unbound. Other unbound panes preserve resolved cwd as startupCwd when detached. Bounded pre-connect input and later live local input share byte order, and teardown settles acknowledged writes without creating or rebinding a stale PTY. A disconnected or detached pending connect cannot bind its late fresh spawn, report its late failure through current callbacks, or ID-retire a newer same-ID owner; rejecting a stale direct-SSH spawn also releases the matching deferred-CWD fence. Natural exit cannot deliver queued work into a reused PTY id. Bound and remote-runtime splits remain owned by their execution host.",
|
||||
"oracle": "Hold CWD resolution behind a controllable promise, invoke the production split path, and require manager.splitPane plus split telemetry before resolving it. Before PTY bind, require both cwd-pending and cwd-resolved deferred detach attempts to return null without layout, pane, tab, ownership, or focus mutation; separately require resolved cwd on an allowed unbound detach and unchanged persisted/live PTY adoption. Rotate a mocked direct-SSH authority while its delayed spawn is in flight, reject and disconnect the stale PTY claim, then require exactly one deferred-CWD cleanup when the delayed connect settles. At the PTY boundary, require zero connect calls while pending, the resolved CWD in spawn and local recovery metadata, one shared FIFO across pre-connect and live ordinary/acknowledged/immediate input, prompt acknowledged-promise settlement on teardown, retirement of an unowned late fresh spawn, preservation of a newer same-ID owner, and zero stale delivery after failure, close, destroy, detach, natural exit, or same-id reuse.",
|
||||
"oracle": "Hold CWD resolution behind a controllable promise, invoke the production split path, and require manager.splitPane plus split telemetry before resolving it. Before PTY bind, require both cwd-pending and cwd-resolved deferred detach attempts to return null without layout, pane, tab, ownership, or focus mutation; separately require resolved cwd on an allowed unbound detach and unchanged persisted/live PTY adoption. Rotate a mocked direct-SSH authority while its delayed spawn is in flight, reject and disconnect the stale PTY claim, then require exactly one deferred-CWD cleanup when the delayed connect settles. At the PTY boundary, require zero connect calls while pending, the resolved CWD in spawn and local recovery metadata, one shared FIFO across pre-connect and live ordinary/acknowledged/immediate input, prompt acknowledged-promise settlement on teardown, retirement of an unowned late fresh spawn, preservation of a newer same-ID owner, and zero stale delivery after failure, close, destroy, detach, natural exit, or same-id reuse. In visible Electron, press the real split shortcut after each warmup and each cold inherited-CWD measured interval, and require complete focus, distinct-PTY bind, immediate-input echo, pane-count, and child-exit observations for every cycle; a timed-out, missing-event, or cleanup-aborted run must publish no headline latency and fail.",
|
||||
"commands": [
|
||||
"pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot"
|
||||
"pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot",
|
||||
"ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
|
||||
"ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1"
|
||||
],
|
||||
"testFiles": [
|
||||
"src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts",
|
||||
@@ -5749,7 +5751,8 @@
|
||||
"src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts",
|
||||
"src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts",
|
||||
"src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts",
|
||||
"src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts"
|
||||
"src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts",
|
||||
"tests/e2e/terminal-split-activation-latency.spec.ts"
|
||||
],
|
||||
"assertionRefs": [
|
||||
{
|
||||
@@ -5815,6 +5818,16 @@
|
||||
"local recovery metadata observes the resolved split CWD",
|
||||
"a stale fresh-spawn completion cannot retire a newer same-ID owner"
|
||||
]
|
||||
},
|
||||
{
|
||||
"file": "tests/e2e/terminal-split-activation-latency.spec.ts",
|
||||
"assertions": [
|
||||
"requires a visible BrowserWindow and visible document before sampling",
|
||||
"records 3 warmups followed by 20 measured real-shortcut cycles, each after a cold inherited-CWD interval",
|
||||
"requires every split to focus, bind a PTY distinct from its source, and echo immediate input",
|
||||
"observes each closed child PTY exit before starting the next cycle",
|
||||
"marks timed-out, missing-event, or cleanup-aborted runs invalid and publishes no headline latency"
|
||||
]
|
||||
}
|
||||
],
|
||||
"evidenceRuns": [
|
||||
@@ -5826,15 +5839,34 @@
|
||||
"command": "pnpm exec vitest run --config config/vitest.config.ts src/renderer/src/components/terminal-pane/terminal-pane-split-with-inherited-cwd.test.ts src/renderer/src/lib/pane-manager/pane-split-close.test.ts src/renderer/src/components/terminal-pane/resolve-split-cwd.test.ts src/renderer/src/components/terminal-pane/pty-connection-split-cwd-resolution.test.ts src/renderer/src/components/terminal-pane/terminal-pane-tab-detach.test.ts src/renderer/src/components/terminal-pane/pty-preconnect-input-buffer.test.ts src/renderer/src/components/terminal-pane/pty-transport-input-write.test.ts --reporter=dot",
|
||||
"durationSeconds": 56.59,
|
||||
"summary": "Seven focused files and 78 tests passed. Vitest reported 49.92 seconds and the measured wall time was 56.59 seconds; coverage includes split creation and focus ordering, nested CWD lineage, promise-identity and SSH authority-rotation cleanup fencing, full pre-bind detach fencing, resolved-CWD detach handoff, close-cancellation, single-FIFO ordering, late-spawn retirement and error suppression, preservation of a newer same-ID owner, generation fencing, in-flight settlement across explicit teardown and natural exit, attach cleanup, bounded retention, and existing input-write contracts."
|
||||
},
|
||||
{
|
||||
"date": "2026-08-30",
|
||||
"runner": "local",
|
||||
"platform": "macos",
|
||||
"result": "passed",
|
||||
"command": "ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-baseline-df14d1a2983d8339e788d0e521f1c4affd9c6d5f-headful-run1.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
|
||||
"durationSeconds": 72,
|
||||
"summary": "At baseline df14d1a2983d8339e788d0e521f1c4affd9c6d5f, the visible BrowserWindow and document completed 3/3 warmups followed by 20/20 measured cold-CWD cycles with every event present, distinct child PTYs, and observed child exits. Shortcut-to-focus p50/p95/max was 65.7/88.7/102.6 ms, PTY bind was 122.8/154.0/159.7 ms, and first echo was 203.8/264.2/332.7 ms. Artifact SHA-256: 6d860cd0cd210f55f2349a197318248af488042117b20c08b6831160950c3277."
|
||||
},
|
||||
{
|
||||
"date": "2026-08-30",
|
||||
"runner": "local",
|
||||
"platform": "macos",
|
||||
"result": "passed",
|
||||
"command": "ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1 ORCA_TERMINAL_SPLIT_LATENCY_LABEL=candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2 ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT=/private/tmp/orca-terminal-split-activation-candidate-d453ffcdb704764daced1b2917fddee7224389f0-headful-run2.json pnpm exec playwright test tests/e2e/terminal-split-activation-latency.spec.ts --config tests/playwright.config.ts --project electron-headful --workers=1",
|
||||
"durationSeconds": 72,
|
||||
"summary": "At candidate d453ffcdb704764daced1b2917fddee7224389f0, the visible BrowserWindow and document completed 3/3 warmups followed by 20/20 measured cold-CWD cycles with every event present, distinct child PTYs, and observed child exits. Shortcut-to-focus p50/p95/max was 12.8/14.4/16.5 ms, PTY bind was 177.0/316.1/333.3 ms, and first echo was 268.6/627.4/710.7 ms. Artifact SHA-256: 9aef7fa842c732eb74f0066a77b2a0336e8f956a06ca61387f9999d6e76213cc."
|
||||
}
|
||||
],
|
||||
"evidenceProcedure": "Historical artifact procedure: run the benchmark spec in a visible macOS Electron project from the primary worktree, complete 3 warmups followed by 20 measured cycles, save the JSON report, and record its SHA-256. Labels carry the product commit SHA; the stored pair predates the committed failure-report and exit-generation hardening, so rerun both product revisions with one committed harness before promotion.",
|
||||
"runtimeBudget": {
|
||||
"p95Seconds": 60,
|
||||
"scope": "seven deterministic renderer and local-transport unit files"
|
||||
"p95Seconds": 240,
|
||||
"scope": "the full listed gate command set: seven deterministic renderer/local-transport unit files plus two opt-in 3-warmup/20-measured visible Electron benchmark invocations"
|
||||
},
|
||||
"flakeHistory": {
|
||||
"status": "not-started",
|
||||
"evidence": "The focused promise-barrier suite passes locally; routed CI and soak history have not started."
|
||||
"evidence": "The focused promise-barrier suite and one visible baseline/candidate benchmark pair pass locally; routed CI and soak history have not started. The benchmark commands are recorded invocations from the primary worktree, with product SHAs encoded in their labels; they are evidence records rather than checkout scripts, and the stored pair predates the final harness hardening."
|
||||
},
|
||||
"redGreenEvidence": {
|
||||
"status": "partial",
|
||||
@@ -5842,18 +5874,20 @@
|
||||
},
|
||||
"performanceBudget": {
|
||||
"required": true,
|
||||
"evidence": "Pane creation and focus add no timer, polling, provider inventory, or subprocess work. CWD resolution remains one existing bounded request off the visible activation path, and detach admission adds only bounded map and record lookups. Pre-connect input, including an in-flight acknowledged write, is capped at 1,024 entries and a conservative UTF-16 ceiling derived from the existing terminal-input byte limit, drains through one worker in order, and clears on teardown or failed connect. Across 20 fresh-source dev cycles, click-to-focused-first-frame changed from baseline p50/p95/max 63.8/72.1/78.3 ms to 56.0/60.5/63.1 ms; all 20 focused. Cold PTY binding changed from 194.2/255.2/303.1 ms to 231.4/337.2/371.9 ms, so the interaction budget is tracked separately from shell readiness. A second 20-cycle steady-source run measured focus 51.1/88.1/109.1 ms and PTY binding 186.6/285.7/1158.6 ms; the 1.16 s readiness outlier still focused in 85.6 ms."
|
||||
"evidence": "Pane creation and focus add no timer, polling, provider inventory, or subprocess work. CWD resolution remains one existing bounded request off the visible activation path, and detach admission adds only bounded map and record lookups. Pre-connect input, including an in-flight acknowledged write, is capped at 1,024 entries and a conservative UTF-16 ceiling derived from the existing terminal-input byte limit, drains through one worker in order, and clears on teardown or failed connect. In same-mode visible BrowserWindow runs with 3 warmups and n=20 measured cold-CWD cycles, baseline df14d1a2983d8339e788d0e521f1c4affd9c6d5f versus candidate d453ffcdb704764daced1b2917fddee7224389f0 changed shortcut-to-focus p50/p95/max from 65.7/88.7/102.6 ms to 12.8/14.4/16.5 ms. PTY bind changed from 122.8/154.0/159.7 ms to 177.0/316.1/333.3 ms, and first echo changed from 203.8/264.2/332.7 ms to 268.6/627.4/710.7 ms, so shell readiness regressed in this pair and remains diagnostic rather than part of the activation claim. The n=20 empirical p95 values are descriptive, are not a distribution guarantee, and are not CI-enforced."
|
||||
},
|
||||
"promotionCriteria": [
|
||||
"Record complete red/green evidence for close, mixed-input ordering, metadata, and failure cleanup.",
|
||||
"Collect 100 consecutive focused CI passes or 14 days without an unexplained flake.",
|
||||
"Add a real-shortcut Electron sentinel with shortcut-to-focus, first-render, PTY-ready, and immediate typed-marker artifacts.",
|
||||
"Collect physical SSH or WSL plus Linux and Windows evidence before claiming provider-complete coverage."
|
||||
"Run the committed real-shortcut Electron benchmark in routed CI or soak before enforcing a latency budget.",
|
||||
"Collect physical local-daemon, SSH or WSL plus Linux, Windows, and folder-workspace evidence before claiming provider-complete coverage."
|
||||
],
|
||||
"knownGaps": [
|
||||
"Ad-hoc macOS Electron CDP proves real-control focus, pre-bind physical input, inherited cwd, close-before-bind cleanup, and rapid nested splits, but no committed Electron sentinel is registered yet.",
|
||||
"Live daemon, SSH, WSL, Linux, and Windows split journeys have not run.",
|
||||
"Local dev measurements establish a numeric first-focused-frame budget, but CI does not enforce it and PTY-ready latency remains diagnostic rather than blocking."
|
||||
"The visible benchmark pair ran on one Apple-silicon macOS host with a synthetic POSIX echo shell and a git-backed workspace; its n=20 empirical p95 is descriptive and not CI-enforced.",
|
||||
"No physical local-daemon, SSH, WSL, Linux, Windows, or folder-workspace latency journey has run; the synthetic fixture is currently skipped on Windows because it requires a POSIX shell.",
|
||||
"The stored artifact pair predates the final harness cleanup/reporting and exit-generation hardening; labels and immutable hashes preserve product provenance, but both product revisions should be rerun with one committed harness before promotion.",
|
||||
"No forced-failure visible benchmark artifact has been recorded; the invalid-report path is exercised by the harness catch/cleanup logic but remains unsoaked.",
|
||||
"PTY-bind and first-echo readiness regressed in the recorded candidate pair and remain diagnostic rather than blocking the activation-ordering claim."
|
||||
],
|
||||
"demotionRule": "Keep experimental or demote if pane activation waits on CWD, a deferred split can detach before PTY bind, detached cwd is lost, input reorders or remains pending after cleanup, a closed pane can spawn, stale retirement kills a newer same-ID owner, remote-runtime delegation creates a competing local pane, or the focused suite flakes without an identified product or harness cause."
|
||||
},
|
||||
|
||||
@@ -95,6 +95,66 @@ describe('benchmark artifact comparison', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('compares terminal split headline metrics in milliseconds', () => {
|
||||
const dir = makeTempDir()
|
||||
const baselinePath = writeArtifact(dir, 'split-baseline.json', {
|
||||
label: 'split baseline',
|
||||
headlineMs: {
|
||||
shortcutToFocusP50: 284.2,
|
||||
shortcutToFocusP95: 676.3
|
||||
}
|
||||
})
|
||||
const candidatePath = writeArtifact(dir, 'split-candidate.json', {
|
||||
label: 'split candidate',
|
||||
headlineMs: {
|
||||
shortcutToFocusP50: 12.7,
|
||||
shortcutToFocusP95: 13.7
|
||||
}
|
||||
})
|
||||
|
||||
const comparison = comparePaths(baselinePath, candidatePath)
|
||||
|
||||
expect(comparison.baseline.kind).toBe('terminal-split-activation')
|
||||
expect(comparison.metrics).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
key: 'shortcutToFocusP50',
|
||||
unit: 'ms',
|
||||
baseline: 284.2,
|
||||
candidate: 12.7,
|
||||
status: 'improved'
|
||||
}),
|
||||
expect.objectContaining({
|
||||
key: 'shortcutToFocusP95',
|
||||
unit: 'ms',
|
||||
baseline: 676.3,
|
||||
candidate: 13.7,
|
||||
status: 'improved'
|
||||
})
|
||||
])
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects invalid benchmark artifacts before comparing partial metrics', () => {
|
||||
const dir = makeTempDir()
|
||||
const baselinePath = writeArtifact(dir, 'split-invalid.json', {
|
||||
label: 'invalid split',
|
||||
status: 'failed',
|
||||
valid: false,
|
||||
headlineMs: { shortcutToFocusP50: 0 }
|
||||
})
|
||||
const candidatePath = writeArtifact(dir, 'split-valid.json', {
|
||||
label: 'valid split',
|
||||
status: 'passed',
|
||||
valid: true,
|
||||
headlineMs: { shortcutToFocusP50: 10 }
|
||||
})
|
||||
|
||||
expect(() => comparePaths(baselinePath, candidatePath)).toThrow(
|
||||
'split-invalid.json: benchmark artifact is marked invalid'
|
||||
)
|
||||
})
|
||||
|
||||
it('compares numeric Playwright annotation metrics and omits metadata fields', () => {
|
||||
const dir = makeTempDir()
|
||||
const baselinePath = writeArtifact(dir, 'baseline-playwright.json', {
|
||||
|
||||
@@ -74,6 +74,9 @@ export function readBenchmarkArtifact(path) {
|
||||
}
|
||||
|
||||
export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifact(path)) {
|
||||
if (artifact?.valid === false || artifact?.status === 'failed') {
|
||||
throw new Error(`${path}: benchmark artifact is marked invalid`)
|
||||
}
|
||||
if (artifact?.summaryMedianMs != null) {
|
||||
return normalizeNumericObject(path, artifact, 'startup', artifact.summaryMedianMs, () => 'ms')
|
||||
}
|
||||
@@ -82,6 +85,15 @@ export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifac
|
||||
key.endsWith('Count') || key.endsWith('After') ? 'count' : 'ms'
|
||||
)
|
||||
}
|
||||
if (artifact?.headlineMs != null) {
|
||||
return normalizeNumericObject(
|
||||
path,
|
||||
artifact,
|
||||
'terminal-split-activation',
|
||||
artifact.headlineMs,
|
||||
() => 'ms'
|
||||
)
|
||||
}
|
||||
if (artifact?.suites != null) {
|
||||
return normalizePlaywrightArtifact(path, artifact)
|
||||
}
|
||||
@@ -89,7 +101,7 @@ export function normalizeBenchmarkArtifact(path, artifact = readBenchmarkArtifac
|
||||
return normalizeSummaryArtifact(path, artifact)
|
||||
}
|
||||
throw new Error(
|
||||
`${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, Playwright suites, or top-level summary`
|
||||
`${path}: unsupported benchmark artifact; expected summaryMedianMs, summaryMedian, headlineMs, Playwright suites, or top-level summary`
|
||||
)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
import { summarizeLatencies, type LatencyDistribution } from './codex-composer-echo-latency-probe'
|
||||
|
||||
export type RendererPhaseStamps = {
|
||||
marker: string
|
||||
sourcePaneId: number
|
||||
sourcePtyId: string
|
||||
newPaneId: number | null
|
||||
newPtyId: string | null
|
||||
keydownAtMs: number | null
|
||||
focusAtMs: number | null
|
||||
ptyBoundAtMs: number | null
|
||||
inputAtMs: number | null
|
||||
firstEchoAtMs: number | null
|
||||
}
|
||||
|
||||
export type SplitLatencySample = RendererPhaseStamps & {
|
||||
phase: 'warmup' | 'measured'
|
||||
iteration: number
|
||||
completedWithinTimeout: boolean
|
||||
paneCountAfterProbe: number
|
||||
ptyExitObserved: boolean
|
||||
cleanupError: string | null
|
||||
shortcutToFocusMs: number | null
|
||||
shortcutToPtyBindMs: number | null
|
||||
shortcutToFirstEchoMs: number | null
|
||||
ptyBindToFirstEchoMs: number | null
|
||||
inputToFirstEchoMs: number | null
|
||||
missing: string[]
|
||||
success: boolean
|
||||
}
|
||||
|
||||
export type SampleSummary = {
|
||||
counts: {
|
||||
requested: number
|
||||
attempted: number
|
||||
success: number
|
||||
missing: number
|
||||
unattempted: number
|
||||
missingEvents: {
|
||||
keydown: number
|
||||
focus: number
|
||||
ptyBind: number
|
||||
input: number
|
||||
firstEcho: number
|
||||
paneCount: number
|
||||
ptyIdentity: number
|
||||
ptyExit: number
|
||||
cleanup: number
|
||||
}
|
||||
}
|
||||
distributions: {
|
||||
shortcutToFocusMs: LatencyDistribution
|
||||
shortcutToPtyBindMs: LatencyDistribution
|
||||
shortcutToFirstEchoMs: LatencyDistribution
|
||||
ptyBindToFirstEchoMs: LatencyDistribution
|
||||
inputToFirstEchoMs: LatencyDistribution
|
||||
}
|
||||
}
|
||||
|
||||
export type BrowserWindowState = {
|
||||
browserWindowVisible: boolean
|
||||
windowCount: number
|
||||
}
|
||||
|
||||
export type TerminalSplitLatencyReportConfig = {
|
||||
warmupCycles: number
|
||||
measuredCycles: number
|
||||
maxMeasuredCycles: number
|
||||
testTimeoutMs: number
|
||||
splitChord: string
|
||||
closeChord: string
|
||||
sampleTimeoutMs: number
|
||||
cleanupTimeoutMs: number
|
||||
processCwdCacheExpiryWaitMs: number
|
||||
}
|
||||
|
||||
export type BenchmarkReportResult = {
|
||||
report: Record<string, unknown>
|
||||
warmupSummary: SampleSummary
|
||||
measuredSummary: SampleSummary
|
||||
}
|
||||
|
||||
function valuesFor(
|
||||
samples: SplitLatencySample[],
|
||||
key:
|
||||
| 'shortcutToFocusMs'
|
||||
| 'shortcutToPtyBindMs'
|
||||
| 'shortcutToFirstEchoMs'
|
||||
| 'ptyBindToFirstEchoMs'
|
||||
| 'inputToFirstEchoMs'
|
||||
): number[] {
|
||||
return samples.flatMap((sample) => {
|
||||
const value = sample[key]
|
||||
return value === null ? [] : [value]
|
||||
})
|
||||
}
|
||||
|
||||
export function summarizeSamples(samples: SplitLatencySample[], requested: number): SampleSummary {
|
||||
const missingEvents = {
|
||||
keydown: samples.filter((sample) => sample.keydownAtMs === null).length,
|
||||
focus: samples.filter((sample) => sample.focusAtMs === null).length,
|
||||
ptyBind: samples.filter((sample) => sample.ptyBoundAtMs === null).length,
|
||||
input: samples.filter((sample) => sample.inputAtMs === null).length,
|
||||
firstEcho: samples.filter((sample) => sample.firstEchoAtMs === null).length,
|
||||
paneCount: samples.filter((sample) => sample.paneCountAfterProbe !== 2).length,
|
||||
ptyIdentity: samples.filter((sample) => sample.newPtyId === sample.sourcePtyId).length,
|
||||
ptyExit: samples.filter((sample) => !sample.ptyExitObserved).length,
|
||||
cleanup: samples.filter((sample) => sample.cleanupError !== null).length
|
||||
}
|
||||
const success = samples.filter((sample) => sample.success).length
|
||||
return {
|
||||
counts: {
|
||||
requested,
|
||||
attempted: samples.length,
|
||||
success,
|
||||
missing: samples.length - success,
|
||||
unattempted: Math.max(0, requested - samples.length),
|
||||
missingEvents
|
||||
},
|
||||
distributions: {
|
||||
shortcutToFocusMs: summarizeLatencies(valuesFor(samples, 'shortcutToFocusMs')),
|
||||
shortcutToPtyBindMs: summarizeLatencies(valuesFor(samples, 'shortcutToPtyBindMs')),
|
||||
shortcutToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'shortcutToFirstEchoMs')),
|
||||
ptyBindToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'ptyBindToFirstEchoMs')),
|
||||
inputToFirstEchoMs: summarizeLatencies(valuesFor(samples, 'inputToFirstEchoMs'))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function buildBenchmarkReport(args: {
|
||||
label: string
|
||||
headfulRun: boolean
|
||||
windowState: BrowserWindowState
|
||||
documentVisibility: string
|
||||
testRepoPath: string
|
||||
warmupSamples: SplitLatencySample[]
|
||||
measuredSamples: SplitLatencySample[]
|
||||
abortError: Error | null
|
||||
config: TerminalSplitLatencyReportConfig
|
||||
}): BenchmarkReportResult {
|
||||
const warmupSummary = summarizeSamples(args.warmupSamples, args.config.warmupCycles)
|
||||
const measuredSummary = summarizeSamples(args.measuredSamples, args.config.measuredCycles)
|
||||
const runComplete =
|
||||
warmupSummary.counts.success === args.config.warmupCycles &&
|
||||
measuredSummary.counts.success === args.config.measuredCycles &&
|
||||
args.abortError === null
|
||||
const headlineMs = runComplete
|
||||
? {
|
||||
shortcutToFocusP50: measuredSummary.distributions.shortcutToFocusMs.p50,
|
||||
shortcutToFocusP95: measuredSummary.distributions.shortcutToFocusMs.p95,
|
||||
shortcutToFocusMax: measuredSummary.distributions.shortcutToFocusMs.max,
|
||||
shortcutToPtyBindP50: measuredSummary.distributions.shortcutToPtyBindMs.p50,
|
||||
shortcutToPtyBindP95: measuredSummary.distributions.shortcutToPtyBindMs.p95,
|
||||
shortcutToPtyBindMax: measuredSummary.distributions.shortcutToPtyBindMs.max,
|
||||
shortcutToFirstEchoP50: measuredSummary.distributions.shortcutToFirstEchoMs.p50,
|
||||
shortcutToFirstEchoP95: measuredSummary.distributions.shortcutToFirstEchoMs.p95,
|
||||
shortcutToFirstEchoMax: measuredSummary.distributions.shortcutToFirstEchoMs.max
|
||||
}
|
||||
: null
|
||||
return {
|
||||
report: {
|
||||
schemaVersion: 1,
|
||||
benchmark: 'terminal-split-activation-latency',
|
||||
label: args.label,
|
||||
status: runComplete ? 'passed' : 'failed',
|
||||
valid: runComplete,
|
||||
abortReason: args.abortError?.message ?? null,
|
||||
timestamp: new Date().toISOString(),
|
||||
platform: process.platform,
|
||||
arch: process.arch,
|
||||
nodeVersion: process.version,
|
||||
headful: args.headfulRun,
|
||||
browserWindowVisible: args.windowState.browserWindowVisible,
|
||||
documentVisibility: args.documentVisibility,
|
||||
testRepoPath: args.testRepoPath,
|
||||
config: args.config,
|
||||
headlineMs,
|
||||
warmupSummary,
|
||||
measuredSummary,
|
||||
warmupSamples: args.warmupSamples,
|
||||
measuredSamples: args.measuredSamples
|
||||
},
|
||||
warmupSummary,
|
||||
measuredSummary
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,720 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import path from 'node:path'
|
||||
import type { Page, TestInfo } from '@stablyai/playwright-test'
|
||||
import { test, expect } from './helpers/orca-app'
|
||||
import {
|
||||
countVisibleTerminalPanes,
|
||||
focusActiveTerminalInput,
|
||||
sendToTerminal,
|
||||
waitForActivePanePtyId,
|
||||
waitForActiveTerminalManager,
|
||||
waitForPaneCount,
|
||||
waitForTerminalOutput
|
||||
} from './helpers/terminal'
|
||||
import { ensureTerminalVisible, waitForActiveWorktree, waitForSessionReady } from './helpers/store'
|
||||
import {
|
||||
buildBenchmarkReport,
|
||||
type BrowserWindowState,
|
||||
type RendererPhaseStamps,
|
||||
type SplitLatencySample,
|
||||
type TerminalSplitLatencyReportConfig
|
||||
} from './terminal-split-activation-latency-report'
|
||||
|
||||
const BENCH_ENABLED = process.env.ORCA_TERMINAL_SPLIT_LATENCY_BENCH === '1'
|
||||
const BENCH_LABEL = process.env.ORCA_TERMINAL_SPLIT_LATENCY_LABEL?.trim() || 'local'
|
||||
const BENCH_OUTPUT_PATH = process.env.ORCA_TERMINAL_SPLIT_LATENCY_OUTPUT?.trim() || null
|
||||
const WARMUP_CYCLES = 3
|
||||
const MIN_MEASURED_CYCLES = 20
|
||||
const MAX_MEASURED_CYCLES = 200
|
||||
const SAMPLE_TIMEOUT_MS = 15_000
|
||||
const CLEANUP_TIMEOUT_MS = 15_000
|
||||
const CONFIRM_CLICK_TIMEOUT_MS = 2_000
|
||||
const BENCH_SETUP_TIMEOUT_MS = 5 * 60 * 1000
|
||||
// Why: process-cwd caches each pid for 1500ms; this wait isolates cold lookups, not correctness.
|
||||
const PROCESS_CWD_CACHE_EXPIRY_WAIT_MS = 1_650
|
||||
const SOURCE_READY_MARKER = 'ORCA_SPLIT_LATENCY_SOURCE_READY'
|
||||
const IS_MAC = process.platform === 'darwin'
|
||||
const SPLIT_CHORD = IS_MAC ? 'Meta+d' : 'Control+Shift+d'
|
||||
const CLOSE_CHORD = IS_MAC ? 'Meta+w' : 'Control+w'
|
||||
|
||||
function readPositiveInt(name: string, fallback: number): number {
|
||||
const value = Number(process.env[name])
|
||||
return Number.isInteger(value) && value > 0 ? value : fallback
|
||||
}
|
||||
|
||||
const MEASURED_CYCLES = Math.min(
|
||||
MAX_MEASURED_CYCLES,
|
||||
Math.max(
|
||||
MIN_MEASURED_CYCLES,
|
||||
readPositiveInt('ORCA_TERMINAL_SPLIT_LATENCY_CYCLES', MIN_MEASURED_CYCLES)
|
||||
)
|
||||
)
|
||||
const BENCH_TIMEOUT_MS =
|
||||
BENCH_SETUP_TIMEOUT_MS +
|
||||
WARMUP_CYCLES * (SAMPLE_TIMEOUT_MS + 4 * CLEANUP_TIMEOUT_MS + CONFIRM_CLICK_TIMEOUT_MS) +
|
||||
MEASURED_CYCLES *
|
||||
(SAMPLE_TIMEOUT_MS +
|
||||
4 * CLEANUP_TIMEOUT_MS +
|
||||
CONFIRM_CLICK_TIMEOUT_MS +
|
||||
PROCESS_CWD_CACHE_EXPIRY_WAIT_MS)
|
||||
const REPORT_CONFIG = {
|
||||
warmupCycles: WARMUP_CYCLES,
|
||||
measuredCycles: MEASURED_CYCLES,
|
||||
maxMeasuredCycles: MAX_MEASURED_CYCLES,
|
||||
testTimeoutMs: BENCH_TIMEOUT_MS,
|
||||
splitChord: SPLIT_CHORD,
|
||||
closeChord: CLOSE_CHORD,
|
||||
sampleTimeoutMs: SAMPLE_TIMEOUT_MS,
|
||||
cleanupTimeoutMs: CLEANUP_TIMEOUT_MS,
|
||||
processCwdCacheExpiryWaitMs: PROCESS_CWD_CACHE_EXPIRY_WAIT_MS
|
||||
} satisfies TerminalSplitLatencyReportConfig
|
||||
|
||||
type RendererProbe = {
|
||||
report: () => RendererPhaseStamps
|
||||
dispose: () => void
|
||||
}
|
||||
|
||||
type SplitLatencyProbeWindow = Window & {
|
||||
__terminalSplitLatencyProbe?: RendererProbe
|
||||
__terminalSplitLatencyPtyExitIds?: string[]
|
||||
__terminalSplitLatencyPtyExitDispose?: () => void
|
||||
}
|
||||
|
||||
function createEchoShellFixture(): { root: string; shellPath: string } {
|
||||
const root = mkdtempSync(path.join(tmpdir(), 'orca-split-latency-'))
|
||||
const shellPath = path.join(root, 'split-echo-shell')
|
||||
writeFileSync(
|
||||
shellPath,
|
||||
[
|
||||
'#!/bin/sh',
|
||||
'stty raw -echo',
|
||||
'dd bs=1 count=1 of=/dev/null 2>/dev/null',
|
||||
`printf '%s' '${SOURCE_READY_MARKER}'`,
|
||||
'exec /bin/cat',
|
||||
''
|
||||
].join('\n'),
|
||||
'utf8'
|
||||
)
|
||||
chmodSync(shellPath, 0o755)
|
||||
return { root, shellPath }
|
||||
}
|
||||
|
||||
async function createSourceTab(
|
||||
page: Page,
|
||||
shellOverride: string
|
||||
): Promise<{ tabId: string; ptyId: string }> {
|
||||
const tabId = await page.evaluate((shellOverride) => {
|
||||
const store = window.__store
|
||||
if (!store) {
|
||||
throw new Error('Store unavailable')
|
||||
}
|
||||
const state = store.getState()
|
||||
const worktreeId = state.activeWorktreeId
|
||||
if (!worktreeId) {
|
||||
throw new Error('No active worktree')
|
||||
}
|
||||
const tab = state.createTab(worktreeId, undefined, shellOverride, { activate: true })
|
||||
store.getState().setActiveTab(tab.id)
|
||||
store.getState().setActiveTabType('terminal')
|
||||
return tab.id
|
||||
}, shellOverride)
|
||||
|
||||
await waitForActiveTerminalManager(page, 30_000)
|
||||
await waitForPaneCount(page, 1, 30_000)
|
||||
const ptyId = await waitForActivePanePtyId(page, 30_000)
|
||||
await sendToTerminal(page, ptyId, '\r')
|
||||
await waitForTerminalOutput(page, SOURCE_READY_MARKER, 30_000)
|
||||
return { tabId, ptyId }
|
||||
}
|
||||
|
||||
async function readActivePaneId(page: Page, tabId: string): Promise<number> {
|
||||
const paneId = await page.evaluate((tabId) => {
|
||||
const manager = window.__paneManagers?.get(tabId)
|
||||
return manager?.getActivePane?.()?.id ?? null
|
||||
}, tabId)
|
||||
if (paneId === null) {
|
||||
throw new Error(`No active pane for source tab ${tabId}`)
|
||||
}
|
||||
return paneId
|
||||
}
|
||||
|
||||
async function installRendererProbe(
|
||||
page: Page,
|
||||
args: {
|
||||
tabId: string
|
||||
sourcePaneId: number
|
||||
sourcePtyId: string
|
||||
marker: string
|
||||
readyMarker: string
|
||||
isMac: boolean
|
||||
}
|
||||
): Promise<void> {
|
||||
await page.evaluate(({ tabId, sourcePaneId, sourcePtyId, marker, readyMarker, isMac }) => {
|
||||
const targetWindow = window as SplitLatencyProbeWindow
|
||||
targetWindow.__terminalSplitLatencyProbe?.dispose()
|
||||
|
||||
const stamps: RendererPhaseStamps = {
|
||||
marker,
|
||||
sourcePaneId,
|
||||
sourcePtyId,
|
||||
newPaneId: null,
|
||||
newPtyId: null,
|
||||
keydownAtMs: null,
|
||||
focusAtMs: null,
|
||||
ptyBoundAtMs: null,
|
||||
inputAtMs: null,
|
||||
firstEchoAtMs: null
|
||||
}
|
||||
let ptyBindingObserver: MutationObserver | null = null
|
||||
let parsedDisposable: { dispose: () => void } | null = null
|
||||
let fixtureReady = false
|
||||
let markerFeedQueued = false
|
||||
const originalStopImmediatePropagation = Event.prototype.stopImmediatePropagation
|
||||
|
||||
const onKeyDown = (event: KeyboardEvent): void => {
|
||||
const matches = isMac
|
||||
? event.code === 'KeyD' && event.metaKey && !event.shiftKey && !event.altKey
|
||||
: event.code === 'KeyD' && event.ctrlKey && event.shiftKey && !event.altKey
|
||||
if (matches && stamps.keydownAtMs === null) {
|
||||
stamps.keydownAtMs = performance.now()
|
||||
}
|
||||
}
|
||||
const patchedStopImmediatePropagation = function (this: Event): void {
|
||||
// Why: terminal shortcuts stop same-target listeners before split work starts.
|
||||
if (this instanceof KeyboardEvent) {
|
||||
onKeyDown(this)
|
||||
}
|
||||
originalStopImmediatePropagation.call(this)
|
||||
}
|
||||
|
||||
const onFocusIn = (event: FocusEvent): void => {
|
||||
if (stamps.keydownAtMs === null || stamps.focusAtMs !== null) {
|
||||
return
|
||||
}
|
||||
const target = event.target
|
||||
if (!(target instanceof HTMLElement) || !target.matches('.xterm-helper-textarea')) {
|
||||
return
|
||||
}
|
||||
const paneElement = target.closest<HTMLElement>('.pane[data-pane-id]')
|
||||
const manager = window.__paneManagers?.get(tabId)
|
||||
const pane = manager?.getPanes?.().find((candidate) => candidate.container === paneElement)
|
||||
if (!pane || pane.id === sourcePaneId) {
|
||||
return
|
||||
}
|
||||
|
||||
stamps.newPaneId = pane.id
|
||||
stamps.focusAtMs = performance.now()
|
||||
const maybeFeedMarker = (): void => {
|
||||
if (!fixtureReady || stamps.ptyBoundAtMs === null || markerFeedQueued) {
|
||||
return
|
||||
}
|
||||
markerFeedQueued = true
|
||||
queueMicrotask(() => {
|
||||
stamps.inputAtMs = performance.now()
|
||||
pane.terminal.input(marker, true)
|
||||
})
|
||||
}
|
||||
const observeParsedOutput = (): void => {
|
||||
const buffer = pane.terminal.buffer.active
|
||||
let text = ''
|
||||
for (let row = 0; row < buffer.length; row += 1) {
|
||||
text += buffer.getLine(row)?.translateToString(true) ?? ''
|
||||
}
|
||||
if (!fixtureReady && text.includes(readyMarker)) {
|
||||
fixtureReady = true
|
||||
maybeFeedMarker()
|
||||
}
|
||||
if (stamps.firstEchoAtMs === null && text.includes(marker)) {
|
||||
stamps.firstEchoAtMs = performance.now()
|
||||
}
|
||||
}
|
||||
parsedDisposable = pane.terminal.onWriteParsed(observeParsedOutput)
|
||||
observeParsedOutput()
|
||||
|
||||
const observePtyBinding = (): void => {
|
||||
const ptyId = pane.container.dataset.ptyId
|
||||
if (!ptyId || stamps.ptyBoundAtMs !== null) {
|
||||
return
|
||||
}
|
||||
stamps.newPtyId = ptyId
|
||||
stamps.ptyBoundAtMs = performance.now()
|
||||
ptyBindingObserver?.disconnect()
|
||||
queueMicrotask(() => pane.terminal.input('\r', true))
|
||||
}
|
||||
|
||||
if (pane.container.dataset.ptyId) {
|
||||
observePtyBinding()
|
||||
return
|
||||
}
|
||||
ptyBindingObserver = new MutationObserver(observePtyBinding)
|
||||
ptyBindingObserver.observe(pane.container, {
|
||||
attributes: true,
|
||||
attributeFilter: ['data-pty-id']
|
||||
})
|
||||
}
|
||||
|
||||
Event.prototype.stopImmediatePropagation = patchedStopImmediatePropagation
|
||||
window.addEventListener('keydown', onKeyDown, { capture: true })
|
||||
document.addEventListener('focusin', onFocusIn, { capture: true })
|
||||
targetWindow.__terminalSplitLatencyProbe = {
|
||||
report: () => ({ ...stamps }),
|
||||
dispose: () => {
|
||||
window.removeEventListener('keydown', onKeyDown, { capture: true })
|
||||
document.removeEventListener('focusin', onFocusIn, { capture: true })
|
||||
if (Event.prototype.stopImmediatePropagation === patchedStopImmediatePropagation) {
|
||||
Event.prototype.stopImmediatePropagation = originalStopImmediatePropagation
|
||||
}
|
||||
ptyBindingObserver?.disconnect()
|
||||
parsedDisposable?.dispose()
|
||||
}
|
||||
}
|
||||
}, args)
|
||||
}
|
||||
|
||||
async function waitForRendererProbe(page: Page): Promise<boolean> {
|
||||
try {
|
||||
await page.waitForFunction(
|
||||
() =>
|
||||
(window as SplitLatencyProbeWindow).__terminalSplitLatencyProbe?.report().firstEchoAtMs !==
|
||||
null,
|
||||
null,
|
||||
{ timeout: SAMPLE_TIMEOUT_MS }
|
||||
)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function collectRendererProbe(page: Page): Promise<RendererPhaseStamps> {
|
||||
return page.evaluate(() => {
|
||||
const targetWindow = window as SplitLatencyProbeWindow
|
||||
const probe = targetWindow.__terminalSplitLatencyProbe
|
||||
if (!probe) {
|
||||
throw new Error('Terminal split latency probe was not installed')
|
||||
}
|
||||
const report = probe.report()
|
||||
probe.dispose()
|
||||
delete targetWindow.__terminalSplitLatencyProbe
|
||||
return report
|
||||
})
|
||||
}
|
||||
|
||||
function elapsed(start: number | null, end: number | null): number | null {
|
||||
return start === null || end === null ? null : end - start
|
||||
}
|
||||
|
||||
function createSample(
|
||||
phase: SplitLatencySample['phase'],
|
||||
iteration: number,
|
||||
stamps: RendererPhaseStamps,
|
||||
completedWithinTimeout: boolean,
|
||||
paneCountAfterProbe: number,
|
||||
ptyExitObserved: boolean,
|
||||
cleanupError: string | null
|
||||
): SplitLatencySample {
|
||||
const missing = [
|
||||
...(stamps.keydownAtMs === null ? ['keydown'] : []),
|
||||
...(stamps.focusAtMs === null ? ['focus'] : []),
|
||||
...(stamps.ptyBoundAtMs === null ? ['pty-bind'] : []),
|
||||
...(stamps.inputAtMs === null ? ['input'] : []),
|
||||
...(stamps.firstEchoAtMs === null ? ['first-echo'] : []),
|
||||
...(stamps.newPtyId === stamps.sourcePtyId ? ['pty-identity'] : []),
|
||||
...(paneCountAfterProbe !== 2 ? [`pane-count:${paneCountAfterProbe}`] : []),
|
||||
...(!ptyExitObserved ? ['pty-exit'] : []),
|
||||
...(cleanupError ? ['cleanup'] : [])
|
||||
]
|
||||
return {
|
||||
...stamps,
|
||||
phase,
|
||||
iteration,
|
||||
completedWithinTimeout,
|
||||
paneCountAfterProbe,
|
||||
ptyExitObserved,
|
||||
cleanupError,
|
||||
shortcutToFocusMs: elapsed(stamps.keydownAtMs, stamps.focusAtMs),
|
||||
shortcutToPtyBindMs: elapsed(stamps.keydownAtMs, stamps.ptyBoundAtMs),
|
||||
shortcutToFirstEchoMs: elapsed(stamps.keydownAtMs, stamps.firstEchoAtMs),
|
||||
ptyBindToFirstEchoMs: elapsed(stamps.ptyBoundAtMs, stamps.firstEchoAtMs),
|
||||
inputToFirstEchoMs: elapsed(stamps.inputAtMs, stamps.firstEchoAtMs),
|
||||
missing,
|
||||
success: completedWithinTimeout && missing.length === 0
|
||||
}
|
||||
}
|
||||
|
||||
async function closeSplitsAndRefocusSource(
|
||||
page: Page,
|
||||
tabId: string,
|
||||
sourcePaneId: number,
|
||||
closedPtyIds: string[]
|
||||
): Promise<{ closeCompletedAt: number; ptyExitObserved: boolean; cleanupError: string | null }> {
|
||||
let paneCount = await countVisibleTerminalPanes(page)
|
||||
if (paneCount < 1) {
|
||||
throw new Error('Source terminal disappeared during split benchmark')
|
||||
}
|
||||
while (paneCount > 1) {
|
||||
const expectedCount = paneCount - 1
|
||||
await focusActiveTerminalInput(page)
|
||||
await page.keyboard.press(CLOSE_CHORD)
|
||||
const confirmButton = page
|
||||
.locator(
|
||||
'[data-slot="dialog-content"][data-state="open"] [data-slot="dialog-footer"] [data-slot="button"][data-variant="destructive"]'
|
||||
)
|
||||
.last()
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
if (await confirmButton.isVisible().catch(() => false)) {
|
||||
await confirmButton.click({ timeout: CONFIRM_CLICK_TIMEOUT_MS })
|
||||
}
|
||||
return countVisibleTerminalPanes(page)
|
||||
},
|
||||
{
|
||||
timeout: CLEANUP_TIMEOUT_MS,
|
||||
message: `Split pane did not close to ${expectedCount} pane(s)`
|
||||
}
|
||||
)
|
||||
.toBe(expectedCount)
|
||||
paneCount = expectedCount
|
||||
}
|
||||
await waitForPaneCount(page, 1, CLEANUP_TIMEOUT_MS)
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
page.evaluate(
|
||||
({ tabId, sourcePaneId }) =>
|
||||
window.__paneManagers?.get(tabId)?.getActivePane?.()?.id === sourcePaneId,
|
||||
{ tabId, sourcePaneId }
|
||||
),
|
||||
{
|
||||
timeout: CLEANUP_TIMEOUT_MS,
|
||||
message: 'Source pane did not regain active ownership after close'
|
||||
}
|
||||
)
|
||||
.toBe(true)
|
||||
const ptyExitResults = await Promise.all(
|
||||
closedPtyIds.map(async (ptyId) => ({ ptyId, observed: await waitForPtyExit(page, ptyId) }))
|
||||
)
|
||||
const missingPtyExitIds = ptyExitResults
|
||||
.filter((result) => !result.observed)
|
||||
.map((result) => result.ptyId)
|
||||
await focusActiveTerminalInput(page)
|
||||
const cleanupError =
|
||||
closedPtyIds.length === 0
|
||||
? 'Split cleanup could not identify a child PTY to verify its exit'
|
||||
: missingPtyExitIds.length > 0
|
||||
? `Closed split PTY did not emit exit: ${missingPtyExitIds.join(', ')}`
|
||||
: null
|
||||
return {
|
||||
closeCompletedAt: Date.now(),
|
||||
ptyExitObserved: cleanupError === null,
|
||||
cleanupError
|
||||
}
|
||||
}
|
||||
|
||||
async function readChildPtyIds(page: Page, tabId: string, sourcePtyId: string): Promise<string[]> {
|
||||
return page.evaluate(
|
||||
({ tabId, sourcePtyId }) => {
|
||||
const manager = window.__paneManagers?.get(tabId)
|
||||
return (manager?.getPanes?.() ?? [])
|
||||
.map((pane) => pane.container.dataset.ptyId ?? null)
|
||||
.filter((ptyId): ptyId is string => Boolean(ptyId) && ptyId !== sourcePtyId)
|
||||
},
|
||||
{ tabId, sourcePtyId }
|
||||
)
|
||||
}
|
||||
|
||||
async function runSplitCycle(
|
||||
page: Page,
|
||||
args: {
|
||||
tabId: string
|
||||
sourcePaneId: number
|
||||
sourcePtyId: string
|
||||
phase: SplitLatencySample['phase']
|
||||
iteration: number
|
||||
}
|
||||
): Promise<{ sample: SplitLatencySample; closeCompletedAt: number; fatalError: Error | null }> {
|
||||
const marker = `ORCA_SPLIT_ECHO_${args.phase}_${args.iteration}_${randomUUID().replaceAll('-', '')}`
|
||||
await focusActiveTerminalInput(page)
|
||||
// Prevent an ID reused by a later PTY lifetime from matching an earlier exit.
|
||||
await resetPtyExitProbe(page)
|
||||
await installRendererProbe(page, {
|
||||
tabId: args.tabId,
|
||||
sourcePaneId: args.sourcePaneId,
|
||||
sourcePtyId: args.sourcePtyId,
|
||||
marker,
|
||||
readyMarker: SOURCE_READY_MARKER,
|
||||
isMac: IS_MAC
|
||||
})
|
||||
await page.keyboard.press(SPLIT_CHORD)
|
||||
const completedWithinTimeout = await waitForRendererProbe(page)
|
||||
const stamps = await collectRendererProbe(page)
|
||||
let paneCountAfterProbe = -1
|
||||
let closeCompletedAt = Date.now()
|
||||
let ptyExitObserved = false
|
||||
let cleanupError: Error | null = null
|
||||
try {
|
||||
paneCountAfterProbe = await countVisibleTerminalPanes(page)
|
||||
const childPtyIds = await readChildPtyIds(page, args.tabId, args.sourcePtyId)
|
||||
const closeResult = await closeSplitsAndRefocusSource(
|
||||
page,
|
||||
args.tabId,
|
||||
args.sourcePaneId,
|
||||
childPtyIds
|
||||
)
|
||||
closeCompletedAt = closeResult.closeCompletedAt
|
||||
ptyExitObserved = closeResult.ptyExitObserved
|
||||
cleanupError = closeResult.cleanupError ? new Error(closeResult.cleanupError) : null
|
||||
} catch (error) {
|
||||
cleanupError = error instanceof Error ? error : new Error(String(error))
|
||||
closeCompletedAt = Date.now()
|
||||
}
|
||||
const sample = createSample(
|
||||
args.phase,
|
||||
args.iteration,
|
||||
stamps,
|
||||
completedWithinTimeout,
|
||||
paneCountAfterProbe,
|
||||
ptyExitObserved,
|
||||
cleanupError?.message ?? null
|
||||
)
|
||||
return { sample, closeCompletedAt, fatalError: cleanupError }
|
||||
}
|
||||
|
||||
async function waitForColdProcessCwdLookup(
|
||||
page: Page,
|
||||
priorCloseCompletedAt: number
|
||||
): Promise<void> {
|
||||
const remaining = PROCESS_CWD_CACHE_EXPIRY_WAIT_MS - (Date.now() - priorCloseCompletedAt)
|
||||
if (remaining > 0) {
|
||||
await page.waitForTimeout(remaining)
|
||||
}
|
||||
expect(Date.now() - priorCloseCompletedAt).toBeGreaterThanOrEqual(
|
||||
PROCESS_CWD_CACHE_EXPIRY_WAIT_MS
|
||||
)
|
||||
}
|
||||
|
||||
function logSample(sample: SplitLatencySample): void {
|
||||
console.log(
|
||||
`[terminal-split-activation-latency] ${sample.phase} ${sample.iteration + 1} ` +
|
||||
`success=${sample.success} missing=${sample.missing.join(',') || 'none'}`
|
||||
)
|
||||
}
|
||||
|
||||
async function installPtyExitProbe(page: Page): Promise<void> {
|
||||
await page.evaluate(() => {
|
||||
const targetWindow = window as SplitLatencyProbeWindow
|
||||
targetWindow.__terminalSplitLatencyPtyExitDispose?.()
|
||||
targetWindow.__terminalSplitLatencyPtyExitIds = []
|
||||
targetWindow.__terminalSplitLatencyPtyExitDispose = window.api.pty.onExit(({ id }) => {
|
||||
const ids = targetWindow.__terminalSplitLatencyPtyExitIds ?? []
|
||||
if (!ids.includes(id)) {
|
||||
ids.push(id)
|
||||
}
|
||||
targetWindow.__terminalSplitLatencyPtyExitIds = ids
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
async function resetPtyExitProbe(page: Page): Promise<void> {
|
||||
await page.evaluate(() => {
|
||||
;(window as SplitLatencyProbeWindow).__terminalSplitLatencyPtyExitIds = []
|
||||
})
|
||||
}
|
||||
|
||||
async function waitForPtyExit(page: Page, ptyId: string): Promise<boolean> {
|
||||
try {
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
page.evaluate((expectedPtyId) => {
|
||||
const targetWindow = window as SplitLatencyProbeWindow
|
||||
return targetWindow.__terminalSplitLatencyPtyExitIds?.includes(expectedPtyId) ?? false
|
||||
}, ptyId),
|
||||
{ timeout: CLEANUP_TIMEOUT_MS, message: `Closed split PTY did not emit exit: ${ptyId}` }
|
||||
)
|
||||
.toBe(true)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
async function disposePtyExitProbe(page: Page): Promise<void> {
|
||||
await page.evaluate(() => {
|
||||
const targetWindow = window as SplitLatencyProbeWindow
|
||||
targetWindow.__terminalSplitLatencyPtyExitDispose?.()
|
||||
delete targetWindow.__terminalSplitLatencyPtyExitDispose
|
||||
delete targetWindow.__terminalSplitLatencyPtyExitIds
|
||||
})
|
||||
}
|
||||
|
||||
async function attachReport(testInfo: TestInfo, report: Record<string, unknown>): Promise<void> {
|
||||
const body = `${JSON.stringify(report, null, 2)}\n`
|
||||
await testInfo.attach('terminal-split-activation-latency.json', {
|
||||
body,
|
||||
contentType: 'application/json'
|
||||
})
|
||||
if (BENCH_OUTPUT_PATH) {
|
||||
try {
|
||||
writeFileSync(BENCH_OUTPUT_PATH, body, 'utf8')
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error)
|
||||
console.error(
|
||||
`[terminal-split-activation-latency] unable to write ${BENCH_OUTPUT_PATH}: ${message}`
|
||||
)
|
||||
}
|
||||
}
|
||||
console.log(`[terminal-split-activation-latency] ${JSON.stringify(report)}`)
|
||||
}
|
||||
|
||||
test.describe('Terminal split activation latency benchmark @headful', () => {
|
||||
test.skip(!BENCH_ENABLED, 'One-off benchmark: set ORCA_TERMINAL_SPLIT_LATENCY_BENCH=1')
|
||||
test.skip(process.platform === 'win32', 'Deterministic echo-shell fixture is POSIX-only')
|
||||
test.setTimeout(BENCH_TIMEOUT_MS)
|
||||
|
||||
test('records shortcut, focus, PTY bind, and first echo phases', async ({
|
||||
electronApp,
|
||||
orcaPage,
|
||||
testRepoPath
|
||||
}, testInfo) => {
|
||||
const headfulRun =
|
||||
process.env.ORCA_E2E_FORCE_HEADFUL === '1' || testInfo.project.metadata.orcaHeadful === true
|
||||
const windowState: BrowserWindowState = {
|
||||
browserWindowVisible: false,
|
||||
windowCount: 0
|
||||
}
|
||||
let documentVisibility = 'unavailable'
|
||||
let fixture: { root: string; shellPath: string } | null = null
|
||||
const warmupSamples: SplitLatencySample[] = []
|
||||
const measuredSamples: SplitLatencySample[] = []
|
||||
let abortError: Error | null = null
|
||||
let reportAttached = false
|
||||
try {
|
||||
expect(headfulRun, 'The latency benchmark must run with a visible BrowserWindow').toBe(true)
|
||||
const observedWindowState = await electronApp.evaluate(({ BrowserWindow }) => ({
|
||||
browserWindowVisible: BrowserWindow.getAllWindows()[0]?.isVisible() ?? false,
|
||||
windowCount: BrowserWindow.getAllWindows().length
|
||||
}))
|
||||
windowState.browserWindowVisible = observedWindowState.browserWindowVisible
|
||||
windowState.windowCount = observedWindowState.windowCount
|
||||
expect(windowState.windowCount).toBeGreaterThan(0)
|
||||
expect(windowState.browserWindowVisible).toBe(true)
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
documentVisibility = await orcaPage.evaluate(() => document.visibilityState)
|
||||
return documentVisibility
|
||||
},
|
||||
{
|
||||
timeout: 15_000,
|
||||
message: 'Visible latency benchmark renderer remained hidden'
|
||||
}
|
||||
)
|
||||
.toBe('visible')
|
||||
await waitForSessionReady(orcaPage)
|
||||
await waitForActiveWorktree(orcaPage)
|
||||
await ensureTerminalVisible(orcaPage)
|
||||
|
||||
fixture = createEchoShellFixture()
|
||||
const source = await createSourceTab(orcaPage, fixture.shellPath)
|
||||
const { tabId, ptyId: sourcePtyId } = source
|
||||
const sourcePaneId = await readActivePaneId(orcaPage, tabId)
|
||||
await installPtyExitProbe(orcaPage)
|
||||
let priorCloseCompletedAt = Date.now()
|
||||
|
||||
for (let iteration = 0; iteration < WARMUP_CYCLES; iteration += 1) {
|
||||
const result = await runSplitCycle(orcaPage, {
|
||||
tabId,
|
||||
sourcePaneId,
|
||||
sourcePtyId,
|
||||
phase: 'warmup',
|
||||
iteration
|
||||
})
|
||||
warmupSamples.push(result.sample)
|
||||
logSample(result.sample)
|
||||
priorCloseCompletedAt = result.closeCompletedAt
|
||||
if (result.fatalError) {
|
||||
abortError = result.fatalError
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
for (let iteration = 0; iteration < MEASURED_CYCLES && abortError === null; iteration += 1) {
|
||||
await waitForColdProcessCwdLookup(orcaPage, priorCloseCompletedAt)
|
||||
const result = await runSplitCycle(orcaPage, {
|
||||
tabId,
|
||||
sourcePaneId,
|
||||
sourcePtyId,
|
||||
phase: 'measured',
|
||||
iteration
|
||||
})
|
||||
measuredSamples.push(result.sample)
|
||||
logSample(result.sample)
|
||||
priorCloseCompletedAt = result.closeCompletedAt
|
||||
if (result.fatalError) {
|
||||
abortError = result.fatalError
|
||||
}
|
||||
}
|
||||
|
||||
documentVisibility = await orcaPage
|
||||
.evaluate(() => document.visibilityState)
|
||||
.catch(() => 'unavailable' as const)
|
||||
const reportResult = buildBenchmarkReport({
|
||||
label: BENCH_LABEL,
|
||||
headfulRun,
|
||||
windowState,
|
||||
documentVisibility,
|
||||
testRepoPath,
|
||||
warmupSamples,
|
||||
measuredSamples,
|
||||
abortError,
|
||||
config: REPORT_CONFIG
|
||||
})
|
||||
await attachReport(testInfo, reportResult.report)
|
||||
reportAttached = true
|
||||
testInfo.annotations.push({
|
||||
type: 'terminal-split-activation-latency',
|
||||
description:
|
||||
`success=${reportResult.measuredSummary.counts.success}/${MEASURED_CYCLES} ` +
|
||||
`focusP50=${reportResult.measuredSummary.distributions.shortcutToFocusMs.p50.toFixed(1)}ms ` +
|
||||
`echoP50=${reportResult.measuredSummary.distributions.shortcutToFirstEchoMs.p50.toFixed(1)}ms`
|
||||
})
|
||||
if (abortError) {
|
||||
throw abortError
|
||||
}
|
||||
expect(reportResult.warmupSummary.counts.success).toBe(WARMUP_CYCLES)
|
||||
expect(reportResult.measuredSummary.counts.success).toBe(MEASURED_CYCLES)
|
||||
} catch (error) {
|
||||
const failure = error instanceof Error ? error : new Error(String(error))
|
||||
if (!reportAttached) {
|
||||
abortError ??= failure
|
||||
const failureReport = buildBenchmarkReport({
|
||||
label: BENCH_LABEL,
|
||||
headfulRun,
|
||||
windowState,
|
||||
documentVisibility,
|
||||
testRepoPath,
|
||||
warmupSamples,
|
||||
measuredSamples,
|
||||
abortError,
|
||||
config: REPORT_CONFIG
|
||||
})
|
||||
await attachReport(testInfo, failureReport.report).catch((attachError) => {
|
||||
const message = attachError instanceof Error ? attachError.message : String(attachError)
|
||||
console.error(
|
||||
`[terminal-split-activation-latency] unable to attach failure report: ${message}`
|
||||
)
|
||||
})
|
||||
}
|
||||
throw error
|
||||
} finally {
|
||||
await disposePtyExitProbe(orcaPage).catch(() => undefined)
|
||||
if (fixture) {
|
||||
rmSync(fixture.root, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user