mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
Jinwoo Hong
a634bf9b49
* test(bench): count runtime-graph publications from main The build-provided `__orcaBenchmarkInstrumentation` is gone from the tree, so the typing bench could no longer report graph-publication counts at all. The renderer cannot supply them either: `window.api` is frozen by contextBridge, so `runtime.syncWindowGraph` is not wrappable. Count them where they land instead — main's `runtime:syncWindowGraph` invoke handler — behind ORCA_TYPING_BENCH_GRAPH_PROBE=1, and record the result in the bench report. Measured on an 870-worktree fixture: 21 publications over a 50 s metadata-only window versus ~1,205 with recurring OSC title/status traffic. The long-task fields ship unproven: an injected 250 ms renderer busy-wait produced zero entries even though `longtask` is in `supportedEntryTypes`, so their zeros mean "oracle unverified", not "no long task". The self-test knob exists to make that falsifiable, and the file says so; per-publication build time still needs a separate --cpu-profile run. * test(bench): optional CDP CPU throttle around the typing window * test(bench): report the throttle that ran and the long task the self-test caused Two ways the bench could misreport its own conditions. `cpuThrottleRate` was the requested rate, written into every report, but only two of the three scenarios wrapped their typing window in the throttle — a `--cpu-throttle 4` visible-split run claimed a 4x throttle it never applied. Recording the rate per scenario would have made the report honest; it would also have left one scenario silently ignoring the flag, and a fourth scenario would inherit the same omission. So both: every scenario now goes through one `measureTypingWindow` helper, and the value it returns is the rate the throttle actually applied. `writeBenchReport` takes that composite instead of a bare measurement, so a scenario cannot produce a report without saying what it ran under. Unthrottled runs are unchanged — rate 1 still opens no CDP session. `selfTestLongTaskMs` took the *earliest* long task starting before a cutoff captured after the busy-wait. The observer has been live since probe start, so any unrelated long task from fixture setup satisfied it — the field whose whole job is to prove the oracle is live was the easiest one to fake. The busy-wait now reports its own renderer-clock bounds and the matching entry is the one containing their midpoint: main-thread tasks never overlap, so at most one can, and it is the task the busy-wait ran in. That entry is then withheld from `longTasks`, `longestLongTasks`, and `longTasksAroundPublication`, which had been counting the oracle's injected 250 ms as workload. A zero still means "oracle unproven" — it now also means it honestly. * test(bench): stop the graph probe when the typing run throws * test(e2e): drain queued long-task records before the probe disconnects
86 lines
3.0 KiB
JavaScript
86 lines
3.0 KiB
JavaScript
/**
|
|
* Entry point for the multi-workspace typing-latency bench
|
|
* (tests/e2e/terminal-multi-workspace-typing-latency.spec.ts).
|
|
*
|
|
* Usage:
|
|
* pnpm bench:multi-workspace-typing [-- --panes 8 --rate-kbps 512 \
|
|
* --keys 48 --cadence-ms 250 --cpu-workers 4 --worktrees 870 \
|
|
* --repositories 27 --terminal-tabs 1410 --unified-tabs 2000 --label before-fix]
|
|
*
|
|
* Results land in tests/tools/benchmarks/results/multi-workspace-typing-*.json.
|
|
* Run once per build/config with distinct --label values, then diff the
|
|
* totalMs/inputHalfMs/echoHalfMs percentiles.
|
|
*/
|
|
import { spawn } from 'node:child_process'
|
|
|
|
const npxCommand = process.platform === 'win32' ? 'npx.cmd' : 'npx'
|
|
|
|
const knobByFlag = {
|
|
'--visited-workspaces': 'ORCA_TYPING_BENCH_VISITED_WORKSPACES',
|
|
'--load-workspaces': 'ORCA_TYPING_BENCH_LOAD_WORKSPACES',
|
|
'--panes': 'ORCA_TYPING_BENCH_LOAD_PANES',
|
|
'--rate-kbps': 'ORCA_TYPING_BENCH_RATE_KBPS',
|
|
'--keys': 'ORCA_TYPING_BENCH_KEYS',
|
|
'--cadence-ms': 'ORCA_TYPING_BENCH_KEY_CADENCE_MS',
|
|
'--cpu-workers': 'ORCA_TYPING_BENCH_CPU_WORKERS',
|
|
'--worktrees': 'ORCA_TYPING_BENCH_METADATA_WORKTREES',
|
|
'--repositories': 'ORCA_TYPING_BENCH_METADATA_REPOSITORIES',
|
|
'--terminal-tabs': 'ORCA_TYPING_BENCH_METADATA_TERMINAL_TABS',
|
|
'--unified-tabs': 'ORCA_TYPING_BENCH_METADATA_UNIFIED_TABS',
|
|
'--sleepers': 'ORCA_TYPING_BENCH_METADATA_SLEEPERS',
|
|
'--live-statuses': 'ORCA_TYPING_BENCH_METADATA_LIVE_STATUSES',
|
|
'--panes-per-tab': 'ORCA_TYPING_BENCH_METADATA_PANES',
|
|
'--status-history': 'ORCA_TYPING_BENCH_METADATA_STATUS_HISTORY',
|
|
'--status-interval-ms': 'ORCA_TYPING_BENCH_METADATA_STATUS_INTERVAL_MS',
|
|
'--agent-rows': 'ORCA_TYPING_BENCH_AGENT_ROWS',
|
|
'--title-change-ms': 'ORCA_TYPING_BENCH_TITLE_CHANGE_MS',
|
|
'--lifecycle-ms': 'ORCA_TYPING_BENCH_LIFECYCLE_MS',
|
|
'--cpu-profile': 'ORCA_TYPING_BENCH_CPU_PROFILE',
|
|
'--pty-metadata': 'ORCA_TYPING_BENCH_PTY_METADATA',
|
|
'--metadata-status': 'ORCA_TYPING_BENCH_METADATA_STATUS',
|
|
'--metadata-titles': 'ORCA_TYPING_BENCH_METADATA_TITLES',
|
|
'--instrumentation': 'ORCA_TYPING_BENCH_INSTRUMENTATION',
|
|
'--graph-probe': 'ORCA_TYPING_BENCH_GRAPH_PROBE',
|
|
'--cpu-throttle': 'ORCA_TYPING_BENCH_CPU_THROTTLE',
|
|
'--label': 'ORCA_TYPING_BENCH_LABEL'
|
|
}
|
|
|
|
const env = { ...process.env, ORCA_BACKGROUND_LAUNCH: '1', ORCA_TYPING_BENCH: '1' }
|
|
const passthroughArgs = []
|
|
const argv = process.argv.slice(2)
|
|
for (let i = 0; i < argv.length; i++) {
|
|
if (argv[i] === '--') {
|
|
continue
|
|
}
|
|
const knob = knobByFlag[argv[i]]
|
|
if (knob) {
|
|
env[knob] = argv[++i]
|
|
} else {
|
|
passthroughArgs.push(argv[i])
|
|
}
|
|
}
|
|
|
|
const child = spawn(
|
|
npxCommand,
|
|
[
|
|
'playwright',
|
|
'test',
|
|
'tests/e2e/terminal-multi-workspace-typing-latency.spec.ts',
|
|
'--config',
|
|
'tests/playwright.config.ts',
|
|
'--project',
|
|
'electron-headless',
|
|
'--workers=1',
|
|
...passthroughArgs
|
|
],
|
|
{ stdio: 'inherit', env }
|
|
)
|
|
|
|
child.on('exit', (code, signal) => {
|
|
if (signal) {
|
|
process.kill(process.pid, signal)
|
|
return
|
|
}
|
|
process.exit(code ?? 1)
|
|
})
|