mirror of
https://github.com/stablyai/orca.git
synced 2026-10-17 00:03:08 +00:00
Add terminal interaction latency benchmark
CDP-driven harness measuring tab creation, tab switching, and workspace switching against the real dev app with renderer-clock phase timings (store -> PTY bind -> first output -> paint settle), longtask counts, and WebGL context diagnostics. Reuses the windows-apphang-repro session helpers; records ORCA_PTY_SPAWN_TIMING attribution when enabled.
This commit is contained in:
Jinwoo Hong
@@ -4,6 +4,48 @@ Goal: (1) significantly improve Windows startup time (~1 min cold start reported
|
||||
(2) fix OpenCode-driven UI freezes, (3) improve overall Windows performance.
|
||||
All changes must be proven with before/after benchmark numbers.
|
||||
|
||||
## Phase 2 (2026-07-02, branch Jinwoo-H/windows-performance-improvement) — terminal interaction latency
|
||||
|
||||
Complaints: slow workspace switching, slow tab create/switch (terminal-related), occasional crashes.
|
||||
Harness: `tools/benchmarks/terminal-perf-bench.mjs` (CDP-driven dev app, renderer-clock phase
|
||||
timings; scenarios tab-create / tab-switch / workspace-switch; local git fixture).
|
||||
Main-process spawn attribution: `ORCA_PTY_SPAWN_TIMING=1` → `[pty-spawn-timing]` lines
|
||||
(pty.ts handler phases: preflight/auth/host_env/options/provider_spawn).
|
||||
|
||||
Findings (baseline, this machine):
|
||||
- Workspace switch: every hide disposed each pane's WebGL context; resume recreated it —
|
||||
~5ms macOS, 100-500ms/pane Windows ANGLE (the comment in terminal-visibility-resume.ts
|
||||
admitted this). Premise (16-context budget) stale since #7064 raised budget to 128.
|
||||
- Tab create: ~550ms steady state; main handler only ~115ms (host_env≈50ms, daemon
|
||||
provider_spawn≈68ms). Remainder is renderer-side (xterm open + WebGL context for the new
|
||||
pane + React mount). First-ever spawn paid +2.7s inside provider_spawn = daemon's first
|
||||
ConPTY (native module + conpty.dll + OpenConsole + Defender), lazily on the user's first terminal.
|
||||
- Tab switch: paint settle median 80-99ms; longtasks 64-151ms — every light tab resume runs
|
||||
scheduleTerminalWebglAtlasRecovery: 3× (frame/120ms/500ms) global shared-atlas clear +
|
||||
refresh of EVERY pane in EVERY manager. Parse-time recovery (pty-connection.ts
|
||||
recoverWebglAtlasAfterParse / hiddenOutputNeedsAtlasRecoveryAfterParse) already covers
|
||||
risky output including hidden. CAUTION: #7058 changed this area and was reverted (#7073) —
|
||||
left as follow-up.
|
||||
- LocalPtyProvider spawned without useConptyDll while the daemon path used it (legacy system
|
||||
ConPTY corruption + perf differences on degraded-mode/fresh-local spawns).
|
||||
|
||||
Fixes on this branch (PR #7080 merged in — WebGL release on dispose + stale pty:exit synthesis):
|
||||
- A: WebGL context retention across hide/show (pane-webgl-context-retention.ts, LRU cap 16;
|
||||
suspend keeps live contexts; resume repaints instead of recreating).
|
||||
- B: useConptyDll for LocalPtyProvider spawns (local-pty-utils.ts) — parity with daemon.
|
||||
- D: atlas recovery bursts skip suspended panes (scoped to visible); retained panes repaint
|
||||
on resume (shared-atlas invariant preserved).
|
||||
- F: daemon boots a throwaway `cmd.exe /c exit` ConPTY (windows-conpty-warmup.ts) so the
|
||||
first user terminal doesn't pay the ~2.7s first-ConPTY cost.
|
||||
|
||||
Follow-ups (documented, not in this branch): gate/scope the light-tab-switch atlas burst
|
||||
(see #7058/#7073 history first); renderer-side tab-create cost (defer WebGL attach for
|
||||
brand-new panes?); cold-restore respawn fan-out concurrency cap; node-pty ≥1.2.0-beta
|
||||
deferred conpty connect (spawn returns pid=0 fast, stops daemon-loop serialization).
|
||||
|
||||
Pre-existing Windows-only test failures (also on main, CI is ubuntu-only): 5 attribution-shim
|
||||
PATH assertions in src/main/ipc/pty.test.ts (path-separator artifacts).
|
||||
|
||||
## Status
|
||||
|
||||
- [x] Benchmark harness for startup time (`tools/benchmarks/startup-time-bench.mjs`)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,600 @@
|
||||
#!/usr/bin/env node
|
||||
// Terminal interaction latency benchmark: tab creation, tab switching, and
|
||||
// workspace switching against the real Electron dev app over CDP. Produces
|
||||
// median/p95 phase timings so Windows terminal slowness can be attributed to
|
||||
// store work, PTY spawn, shell startup, or renderer paint — not guessed at.
|
||||
|
||||
import { execFileSync } from 'node:child_process'
|
||||
import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs'
|
||||
import os from 'node:os'
|
||||
import path from 'node:path'
|
||||
import process from 'node:process'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import {
|
||||
collectRendererDiagnostics,
|
||||
connectToApp,
|
||||
createGpuUserDataDirectory,
|
||||
installRendererProbe,
|
||||
launchDevApp,
|
||||
pickFreePort,
|
||||
stopDevApp,
|
||||
waitForStoreReady
|
||||
} from '../../config/scripts/windows-apphang-repro/electron-dev-session.mjs'
|
||||
import {
|
||||
pollUntil,
|
||||
rendererActionTimeoutMs,
|
||||
runWithTimeout,
|
||||
setupTimeoutMs
|
||||
} from '../../config/scripts/windows-apphang-repro/repro-timing.mjs'
|
||||
import { safeRemoveLocalDirectory } from '../../config/scripts/windows-apphang-repro/wsl-workspace-fixture.mjs'
|
||||
|
||||
const rootDir = path.resolve(fileURLToPath(new URL('../..', import.meta.url)))
|
||||
const scenarioTimeoutMs = 300_000
|
||||
const defaultIterations = 8
|
||||
const defaultSwitches = 24
|
||||
const defaultCycles = 12
|
||||
|
||||
function parseArgs() {
|
||||
const args = {
|
||||
label: 'run',
|
||||
iterations: defaultIterations,
|
||||
switches: defaultSwitches,
|
||||
cycles: defaultCycles,
|
||||
shell: null,
|
||||
scenarios: ['tab-create', 'tab-switch', 'workspace-switch'],
|
||||
reportPath: null,
|
||||
keep: false
|
||||
}
|
||||
for (const arg of process.argv.slice(2)) {
|
||||
if (arg === '--help' || arg === '-h') {
|
||||
printHelp()
|
||||
process.exit(0)
|
||||
}
|
||||
if (arg === '--keep') {
|
||||
args.keep = true
|
||||
continue
|
||||
}
|
||||
const [name, value] = arg.split('=', 2)
|
||||
if (name === '--label') {
|
||||
args.label = value?.trim() || 'run'
|
||||
continue
|
||||
}
|
||||
if (name === '--iterations') {
|
||||
args.iterations = parsePositiveInt(name, value)
|
||||
continue
|
||||
}
|
||||
if (name === '--switches') {
|
||||
args.switches = parsePositiveInt(name, value)
|
||||
continue
|
||||
}
|
||||
if (name === '--cycles') {
|
||||
args.cycles = parsePositiveInt(name, value)
|
||||
continue
|
||||
}
|
||||
if (name === '--shell') {
|
||||
args.shell = value?.trim() || null
|
||||
continue
|
||||
}
|
||||
if (name === '--scenarios') {
|
||||
const scenarios = (value ?? '')
|
||||
.split(',')
|
||||
.map((entry) => entry.trim())
|
||||
.filter(Boolean)
|
||||
const known = new Set(['tab-create', 'tab-switch', 'workspace-switch'])
|
||||
if (scenarios.length === 0 || scenarios.some((scenario) => !known.has(scenario))) {
|
||||
throw new Error(
|
||||
`Unsupported --scenarios=${value}. Use tab-create,tab-switch,workspace-switch.`
|
||||
)
|
||||
}
|
||||
args.scenarios = scenarios
|
||||
continue
|
||||
}
|
||||
if (name === '--report') {
|
||||
args.reportPath = value?.trim() || null
|
||||
continue
|
||||
}
|
||||
throw new Error(`Unknown argument: ${arg}`)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log(`Usage:
|
||||
node tools/benchmarks/terminal-perf-bench.mjs [options]
|
||||
|
||||
Options:
|
||||
--label=NAME Label recorded in the report filename/JSON. Default: run.
|
||||
--iterations=N Terminal tab create/close iterations. Default: ${defaultIterations}.
|
||||
--switches=N Tab switch alternations. Default: ${defaultSwitches}.
|
||||
--cycles=N Workspace switch cycles. Default: ${defaultCycles}.
|
||||
--shell=PATH Shell override for created tabs (e.g. cmd.exe). Default: app default.
|
||||
--scenarios=a,b Subset of tab-create,tab-switch,workspace-switch.
|
||||
--report=PATH Report JSON path. Default: tools/benchmarks/results/terminal-perf-<label>-<ts>.json.
|
||||
--keep Keep the temp fixture and user data dir.`)
|
||||
}
|
||||
|
||||
function parsePositiveInt(name, value) {
|
||||
const parsed = Number.parseInt(value ?? '', 10)
|
||||
if (!Number.isInteger(parsed) || parsed <= 0 || String(parsed) !== value) {
|
||||
throw new Error(`${name} requires a positive integer.`)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
function git(cwd, ...cmd) {
|
||||
execFileSync('git', cmd, { cwd, stdio: 'pipe' })
|
||||
}
|
||||
|
||||
/** Local (native-path) git repo with two external worktrees — three workspace
|
||||
* cards total. Native paths keep the benchmark portable and representative of
|
||||
* the default local workflow, unlike the WSL-specific apphang fixture. */
|
||||
function createLocalRepoFixture() {
|
||||
const baseDir = mkdtempSync(path.join(os.tmpdir(), 'orca-termperf-'))
|
||||
const repoPath = path.join(baseDir, 'repo')
|
||||
mkdirSync(repoPath, { recursive: true })
|
||||
git(repoPath, 'init', '--initial-branch=main')
|
||||
git(repoPath, 'config', 'user.email', 'bench@orca.local')
|
||||
git(repoPath, 'config', 'user.name', 'Orca Bench')
|
||||
writeFileSync(path.join(repoPath, 'README.md'), '# terminal perf fixture\n')
|
||||
git(repoPath, 'add', '.')
|
||||
git(repoPath, 'commit', '-m', 'init', '--no-gpg-sign')
|
||||
const worktreePaths = []
|
||||
for (const name of ['wt-one', 'wt-two']) {
|
||||
const worktreePath = path.join(baseDir, name)
|
||||
git(repoPath, 'worktree', 'add', worktreePath, '-b', name)
|
||||
worktreePaths.push(worktreePath)
|
||||
}
|
||||
return { baseDir, repoPath, worktreePaths }
|
||||
}
|
||||
|
||||
async function setupWorkspaces(page, fixture) {
|
||||
return await runWithTimeout(
|
||||
'fixture registration in Orca',
|
||||
() =>
|
||||
page.evaluate(
|
||||
async ({ repoPath, importedWorktreePaths }) => {
|
||||
const store = window.__store
|
||||
if (!store) {
|
||||
throw new Error('window.__store is unavailable.')
|
||||
}
|
||||
await store.getState().fetchSettings?.()
|
||||
const addResult = await window.api.repos.add({ path: repoPath, kind: 'git' })
|
||||
if ('error' in addResult) {
|
||||
throw new Error(addResult.error)
|
||||
}
|
||||
await store.getState().fetchRepos()
|
||||
const state = store.getState()
|
||||
const repo =
|
||||
state.repos.find((candidate) => candidate.path === repoPath) ?? addResult.repo
|
||||
await state.updateRepo(repo.id, {
|
||||
externalWorktreeVisibility: 'show',
|
||||
externalWorktreeVisibilityPromptDismissedAt: Date.now(),
|
||||
importedExternalWorktreePaths: importedWorktreePaths,
|
||||
externalWorktreeInboxBaselinePaths: importedWorktreePaths
|
||||
})
|
||||
await store.getState().fetchWorktrees(repo.id, { requireAuthoritative: true })
|
||||
const nextState = store.getState()
|
||||
nextState.setSidebarOpen(true)
|
||||
nextState.setGroupBy('none')
|
||||
nextState.setSortBy('recent')
|
||||
nextState.setShowActiveOnly(false)
|
||||
nextState.setActiveView('terminal')
|
||||
const worktrees = nextState.worktreesByRepo[repo.id] ?? []
|
||||
return {
|
||||
repoId: repo.id,
|
||||
worktrees: worktrees.map((worktree) => ({
|
||||
id: worktree.id,
|
||||
path: worktree.path,
|
||||
displayName: worktree.displayName,
|
||||
isMainWorktree: worktree.isMainWorktree
|
||||
}))
|
||||
}
|
||||
},
|
||||
{ repoPath: fixture.repoPath, importedWorktreePaths: fixture.worktreePaths }
|
||||
),
|
||||
setupTimeoutMs
|
||||
)
|
||||
}
|
||||
|
||||
async function clickWorktreeCard(page, worktreeId) {
|
||||
const rect = await runWithTimeout(
|
||||
`locate worktree card ${worktreeId}`,
|
||||
() =>
|
||||
page.evaluate((id) => {
|
||||
const rows = Array.from(document.querySelectorAll('[data-worktree-id]'))
|
||||
const row = rows.find((candidate) => candidate.getAttribute('data-worktree-id') === id)
|
||||
if (!row) {
|
||||
return null
|
||||
}
|
||||
row.scrollIntoView({ block: 'center', inline: 'nearest' })
|
||||
const surface = row.querySelector('[data-worktree-card-surface="true"]') ?? row
|
||||
const bounds = surface.getBoundingClientRect()
|
||||
if (bounds.width <= 0 || bounds.height <= 0) {
|
||||
return null
|
||||
}
|
||||
return { x: bounds.left + bounds.width / 2, y: bounds.top + bounds.height / 2 }
|
||||
}, worktreeId),
|
||||
rendererActionTimeoutMs
|
||||
)
|
||||
if (!rect) {
|
||||
throw new Error(`Could not find rendered worktree card for ${worktreeId}`)
|
||||
}
|
||||
await runWithTimeout(
|
||||
`click worktree card ${worktreeId}`,
|
||||
() => page.mouse.click(rect.x, rect.y),
|
||||
rendererActionTimeoutMs
|
||||
)
|
||||
}
|
||||
|
||||
async function activateWorktree(page, worktreeId) {
|
||||
await clickWorktreeCard(page, worktreeId)
|
||||
await pollUntil(
|
||||
`active worktree ${worktreeId}`,
|
||||
() =>
|
||||
page.evaluate((id) => {
|
||||
const state = window.__store?.getState?.()
|
||||
return state?.activeWorktreeId === id && state.activeTabType === 'terminal'
|
||||
}, worktreeId),
|
||||
Boolean,
|
||||
30_000
|
||||
)
|
||||
}
|
||||
|
||||
/** Runs entirely in the renderer so phase timestamps come from one
|
||||
* performance.now() clock with no CDP round-trip skew. */
|
||||
async function runTabCreateScenario(page, worktreeId, iterations, shell) {
|
||||
return await runWithTimeout(
|
||||
'tab-create scenario',
|
||||
() =>
|
||||
page.evaluate(
|
||||
async ({ worktreeId, iterations, shell }) => {
|
||||
const store = window.__store
|
||||
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
|
||||
const raf2 = () =>
|
||||
new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)))
|
||||
const longtasks = []
|
||||
const observer = new PerformanceObserver((list) => {
|
||||
for (const entry of list.getEntries()) {
|
||||
longtasks.push(entry.duration)
|
||||
}
|
||||
})
|
||||
observer.observe({ entryTypes: ['longtask'] })
|
||||
const samples = []
|
||||
for (let index = 0; index < iterations; index++) {
|
||||
const t0 = performance.now()
|
||||
const tab = store.getState().createTab(worktreeId, undefined, shell ?? undefined)
|
||||
const tCreated = performance.now()
|
||||
const readPane = () => {
|
||||
const manager = window.__paneManagers?.get(tab.id)
|
||||
return manager?.getActivePane?.() ?? manager?.getPanes?.()?.[0] ?? null
|
||||
}
|
||||
let ptyId = null
|
||||
while (!ptyId && performance.now() - t0 < 60_000) {
|
||||
ptyId = readPane()?.container?.dataset?.ptyId ?? null
|
||||
if (!ptyId) {
|
||||
await sleep(5)
|
||||
}
|
||||
}
|
||||
const tPty = performance.now()
|
||||
let sawOutput = false
|
||||
while (!sawOutput && performance.now() - t0 < 60_000) {
|
||||
const content = readPane()?.serializeAddon?.serialize?.() ?? ''
|
||||
sawOutput = content.trim().length > 0
|
||||
if (!sawOutput) {
|
||||
await sleep(10)
|
||||
}
|
||||
}
|
||||
const tOutput = performance.now()
|
||||
await raf2()
|
||||
const tPainted = performance.now()
|
||||
samples.push({
|
||||
index,
|
||||
storeCreateMs: tCreated - t0,
|
||||
ptyBindMs: tPty - t0,
|
||||
firstOutputMs: tOutput - t0,
|
||||
paintSettleMs: tPainted - t0,
|
||||
timedOut: !ptyId || !sawOutput
|
||||
})
|
||||
store.getState().closeTab(tab.id)
|
||||
await sleep(400)
|
||||
}
|
||||
observer.disconnect()
|
||||
return { samples, longtasks }
|
||||
},
|
||||
{ worktreeId, iterations, shell }
|
||||
),
|
||||
scenarioTimeoutMs
|
||||
)
|
||||
}
|
||||
|
||||
/** K tabs are created up-front (PTY bound), then activation alternates across
|
||||
* them. switchMs = setActiveTab call → active pane manager/pane for the new
|
||||
* tab present → two RAFs (paint settled). */
|
||||
async function runTabSwitchScenario(page, worktreeId, switches, shell) {
|
||||
return await runWithTimeout(
|
||||
'tab-switch scenario',
|
||||
() =>
|
||||
page.evaluate(
|
||||
async ({ worktreeId, switches, shell }) => {
|
||||
const store = window.__store
|
||||
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms))
|
||||
const raf2 = () =>
|
||||
new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)))
|
||||
const tabCount = 4
|
||||
const tabs = []
|
||||
for (let index = 0; index < tabCount; index++) {
|
||||
const tab = store.getState().createTab(worktreeId, undefined, shell ?? undefined)
|
||||
tabs.push(tab.id)
|
||||
const started = performance.now()
|
||||
let bound = false
|
||||
while (!bound && performance.now() - started < 60_000) {
|
||||
const manager = window.__paneManagers?.get(tab.id)
|
||||
const pane = manager?.getActivePane?.() ?? manager?.getPanes?.()?.[0] ?? null
|
||||
bound = Boolean(pane?.container?.dataset?.ptyId)
|
||||
if (!bound) {
|
||||
await sleep(10)
|
||||
}
|
||||
}
|
||||
}
|
||||
await sleep(500)
|
||||
const longtasks = []
|
||||
const observer = new PerformanceObserver((list) => {
|
||||
for (const entry of list.getEntries()) {
|
||||
longtasks.push(entry.duration)
|
||||
}
|
||||
})
|
||||
observer.observe({ entryTypes: ['longtask'] })
|
||||
const samples = []
|
||||
for (let index = 0; index < switches; index++) {
|
||||
const targetTabId = tabs[index % tabs.length]
|
||||
if (store.getState().activeTabId === targetTabId) {
|
||||
continue
|
||||
}
|
||||
const t0 = performance.now()
|
||||
store.getState().setActiveTab(targetTabId)
|
||||
const tStore = performance.now()
|
||||
let visible = false
|
||||
while (!visible && performance.now() - t0 < 10_000) {
|
||||
const manager = window.__paneManagers?.get(targetTabId)
|
||||
const pane = manager?.getActivePane?.() ?? manager?.getPanes?.()?.[0] ?? null
|
||||
visible = Boolean(pane?.container?.isConnected)
|
||||
if (!visible) {
|
||||
await sleep(2)
|
||||
}
|
||||
}
|
||||
const tVisible = performance.now()
|
||||
await raf2()
|
||||
const tPainted = performance.now()
|
||||
samples.push({
|
||||
index,
|
||||
targetTabId,
|
||||
storeMs: tStore - t0,
|
||||
paneVisibleMs: tVisible - t0,
|
||||
paintSettleMs: tPainted - t0,
|
||||
timedOut: !visible
|
||||
})
|
||||
await sleep(120)
|
||||
}
|
||||
observer.disconnect()
|
||||
for (const tabId of tabs) {
|
||||
store.getState().closeTab(tabId)
|
||||
}
|
||||
return { samples, longtasks }
|
||||
},
|
||||
{ worktreeId, switches, shell }
|
||||
),
|
||||
scenarioTimeoutMs
|
||||
)
|
||||
}
|
||||
|
||||
async function runWorkspaceSwitchScenario(page, worktreeIds, cycles) {
|
||||
const samples = []
|
||||
for (let index = 0; index < cycles; index++) {
|
||||
const worktreeId = worktreeIds[index % worktreeIds.length]
|
||||
const t0 = Date.now()
|
||||
await clickWorktreeCard(page, worktreeId)
|
||||
await pollUntil(
|
||||
`workspace activation ${worktreeId}`,
|
||||
() =>
|
||||
page.evaluate((id) => {
|
||||
const state = window.__store?.getState?.()
|
||||
return state?.activeWorktreeId === id && state.activeTabType === 'terminal'
|
||||
}, worktreeId),
|
||||
Boolean,
|
||||
30_000,
|
||||
10
|
||||
)
|
||||
const activationMs = Date.now() - t0
|
||||
const pty = await pollUntil(
|
||||
`workspace pty ${worktreeId}`,
|
||||
() =>
|
||||
page.evaluate((id) => {
|
||||
const state = window.__store?.getState?.()
|
||||
const tabId =
|
||||
state?.activeWorktreeId === id && state.activeTabType === 'terminal'
|
||||
? state.activeTabId
|
||||
: (state?.activeTabIdByWorktree?.[id] ?? null)
|
||||
const manager = tabId ? window.__paneManagers?.get(tabId) : null
|
||||
const pane = manager?.getActivePane?.() ?? manager?.getPanes?.()?.[0] ?? null
|
||||
return pane?.container?.dataset?.ptyId ?? null
|
||||
}, worktreeId),
|
||||
Boolean,
|
||||
30_000,
|
||||
10
|
||||
)
|
||||
const ptyBindMs = Date.now() - t0
|
||||
await runWithTimeout(
|
||||
'paint settle',
|
||||
() =>
|
||||
page.evaluate(
|
||||
() =>
|
||||
new Promise((resolve) => requestAnimationFrame(() => requestAnimationFrame(resolve)))
|
||||
),
|
||||
rendererActionTimeoutMs
|
||||
)
|
||||
const paintSettleMs = Date.now() - t0
|
||||
samples.push({ index, worktreeId, ptyId: pty, activationMs, ptyBindMs, paintSettleMs })
|
||||
}
|
||||
return { samples }
|
||||
}
|
||||
|
||||
function summarize(values) {
|
||||
const sorted = values.filter((value) => Number.isFinite(value)).sort((a, b) => a - b)
|
||||
if (sorted.length === 0) {
|
||||
return null
|
||||
}
|
||||
const at = (fraction) =>
|
||||
sorted[Math.min(sorted.length - 1, Math.round(fraction * (sorted.length - 1)))]
|
||||
return {
|
||||
count: sorted.length,
|
||||
median: Math.round(at(0.5)),
|
||||
p95: Math.round(at(0.95)),
|
||||
max: Math.round(sorted.at(-1))
|
||||
}
|
||||
}
|
||||
|
||||
function summarizeScenario(name, result) {
|
||||
if (!result) {
|
||||
return null
|
||||
}
|
||||
const fields = {}
|
||||
const keys = new Set()
|
||||
for (const sample of result.samples) {
|
||||
for (const [key, value] of Object.entries(sample)) {
|
||||
if (typeof value === 'number' && key !== 'index') {
|
||||
keys.add(key)
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const key of keys) {
|
||||
fields[key] = summarize(result.samples.map((sample) => sample[key]))
|
||||
}
|
||||
const longtasks = result.longtasks ?? []
|
||||
return {
|
||||
name,
|
||||
samples: result.samples.length,
|
||||
timedOut: result.samples.filter((sample) => sample.timedOut).length,
|
||||
fields,
|
||||
longtaskCount: longtasks.length,
|
||||
longtaskMaxMs: longtasks.length ? Math.round(Math.max(...longtasks)) : 0
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArgs()
|
||||
const fixture = createLocalRepoFixture()
|
||||
const cdpPort = await pickFreePort()
|
||||
const userDataDir = createGpuUserDataDirectory('bench')
|
||||
console.log(`[terminal-perf] fixture=${fixture.baseDir} userData=${userDataDir} cdp=${cdpPort}`)
|
||||
const launched = launchDevApp({ cdpPort, userDataDir })
|
||||
let browser = null
|
||||
let page = null
|
||||
const startedAt = Date.now()
|
||||
const report = {
|
||||
label: args.label,
|
||||
startedAt: new Date(startedAt).toISOString(),
|
||||
platform: `${process.platform} ${os.release()}`,
|
||||
shell: args.shell ?? 'default',
|
||||
args: { iterations: args.iterations, switches: args.switches, cycles: args.cycles },
|
||||
scenarios: {},
|
||||
summaries: [],
|
||||
finalDiagnostics: null,
|
||||
cleanupErrors: []
|
||||
}
|
||||
try {
|
||||
const connected = await connectToApp(cdpPort)
|
||||
browser = connected.browser
|
||||
page = connected.page
|
||||
await waitForStoreReady(page)
|
||||
await installRendererProbe(page)
|
||||
const setup = await setupWorkspaces(page, fixture)
|
||||
const worktrees = setup.worktrees
|
||||
if (worktrees.length < 2) {
|
||||
throw new Error(`Expected >=2 worktrees, got ${worktrees.length}`)
|
||||
}
|
||||
const primary = worktrees.find((worktree) => worktree.isMainWorktree) ?? worktrees[0]
|
||||
await activateWorktree(page, primary.id)
|
||||
|
||||
if (args.scenarios.includes('tab-create')) {
|
||||
console.log(`[terminal-perf] scenario=tab-create iterations=${args.iterations}`)
|
||||
report.scenarios['tab-create'] = await runTabCreateScenario(
|
||||
page,
|
||||
primary.id,
|
||||
args.iterations,
|
||||
args.shell
|
||||
)
|
||||
}
|
||||
if (args.scenarios.includes('tab-switch')) {
|
||||
console.log(`[terminal-perf] scenario=tab-switch switches=${args.switches}`)
|
||||
report.scenarios['tab-switch'] = await runTabSwitchScenario(
|
||||
page,
|
||||
primary.id,
|
||||
args.switches,
|
||||
args.shell
|
||||
)
|
||||
}
|
||||
if (args.scenarios.includes('workspace-switch')) {
|
||||
console.log(`[terminal-perf] scenario=workspace-switch cycles=${args.cycles}`)
|
||||
const targets = worktrees.slice(0, 3).map((worktree) => worktree.id)
|
||||
report.scenarios['workspace-switch'] = await runWorkspaceSwitchScenario(
|
||||
page,
|
||||
targets,
|
||||
args.cycles
|
||||
)
|
||||
}
|
||||
report.finalDiagnostics = await collectRendererDiagnostics(page)
|
||||
} finally {
|
||||
report.elapsedMs = Date.now() - startedAt
|
||||
for (const [name, result] of Object.entries(report.scenarios)) {
|
||||
const summary = summarizeScenario(name, result)
|
||||
if (summary) {
|
||||
report.summaries.push(summary)
|
||||
}
|
||||
}
|
||||
if (browser) {
|
||||
await browser.close().catch(() => undefined)
|
||||
}
|
||||
await stopDevApp(launched.child)
|
||||
report.appLogsTail = launched.logs.slice(-80)
|
||||
// Main-process phase attribution (requires ORCA_PTY_SPAWN_TIMING=1 in the
|
||||
// benchmark's environment; launchDevApp inherits it into the app).
|
||||
report.ptySpawnTimings = launched.logs
|
||||
.filter((entry) => entry.line.includes('[pty-spawn-timing]'))
|
||||
.map((entry) => entry.line.slice(entry.line.indexOf('[pty-spawn-timing]')))
|
||||
if (!args.keep) {
|
||||
safeRemoveLocalDirectory(fixture.baseDir, report.cleanupErrors)
|
||||
safeRemoveLocalDirectory(userDataDir, report.cleanupErrors)
|
||||
}
|
||||
const stamp = new Date(startedAt).toISOString().replace(/[:.]/g, '-')
|
||||
const reportPath = path.resolve(
|
||||
args.reportPath ??
|
||||
path.join(
|
||||
rootDir,
|
||||
'tools',
|
||||
'benchmarks',
|
||||
'results',
|
||||
`terminal-perf-${args.label}-${stamp}.json`
|
||||
)
|
||||
)
|
||||
mkdirSync(path.dirname(reportPath), { recursive: true })
|
||||
writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`)
|
||||
console.log(`[terminal-perf] report=${reportPath}`)
|
||||
for (const summary of report.summaries) {
|
||||
console.log(
|
||||
`[terminal-perf] ${summary.name}: samples=${summary.samples} timedOut=${summary.timedOut} longtasks=${summary.longtaskCount} (max ${summary.longtaskMaxMs}ms)`
|
||||
)
|
||||
for (const [field, stats] of Object.entries(summary.fields)) {
|
||||
if (stats) {
|
||||
console.log(
|
||||
`[terminal-perf] ${field}: median=${stats.median}ms p95=${stats.p95}ms max=${stats.max}ms`
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.stack || error.message : String(error))
|
||||
process.exit(1)
|
||||
})
|
||||
Reference in New Issue
Block a user