From 8b7f8c88ce53cc18cb3cdb56cfb58ecf7fa0ec07 Mon Sep 17 00:00:00 2001 From: OrcaWin Date: Tue, 27 Oct 2026 21:48:08 -0700 Subject: [PATCH] ci: run terminal perf on demand only, on a hosted runner (#34726) * ci: run terminal perf on demand only, on a hosted runner The nightly failed every run: the presented-frame guard accepts only GitHub-hosted Xvfb displays, but the workflow ran on Blacksmith. Drop the schedule, move dispatch runs to ubuntu-24.04 and document local use. * docs: record the first hosted terminal perf result --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> --- .github/workflows/terminal-perf.yml | 8 ++-- config/performance-audit.md | 2 +- docs/readme/terminal-perf.md | 67 +++++++++++++++++++++++++++++ 3 files changed, 71 insertions(+), 6 deletions(-) create mode 100644 docs/readme/terminal-perf.md diff --git a/.github/workflows/terminal-perf.yml b/.github/workflows/terminal-perf.yml index dd6cffe0911..6d46bc984e3 100644 --- a/.github/workflows/terminal-perf.yml +++ b/.github/workflows/terminal-perf.yml @@ -1,5 +1,6 @@ name: Terminal Perf +# On demand only; see docs/readme/terminal-perf.md. on: workflow_dispatch: inputs: @@ -43,10 +44,6 @@ on: description: Comma-separated hidden real-PTY pressure pane counts required: false type: string - schedule: - # Why: keep the expensive scale run off PRs, but exercise it daily so - # terminal throughput regressions are caught before users report typing lag. - - cron: '30 8 * * *' permissions: contents: read @@ -57,7 +54,8 @@ cache-mode: read jobs: terminal-perf: name: terminal scale perf - runs-on: blacksmith-4vcpu-ubuntu-2404 + # Presented-frame measurement only accepts GitHub-hosted Xvfb displays. + runs-on: ubuntu-24.04 timeout-minutes: 45 env: NODE_OPTIONS: --max-old-space-size=4096 diff --git a/config/performance-audit.md b/config/performance-audit.md index 161f35a0dcc..3a11b03e7e8 100644 --- a/config/performance-audit.md +++ b/config/performance-audit.md @@ -46,7 +46,7 @@ Incremental Claude scans and persisted Claude cache checkpoints are also include macOS and Windows, and on PRs changing this tooling or any listed contract file. It uploads per-OS JSON test results, plus the source inventory once from Linux because that scan is OS-independent. Its schedule starts after merge. Run the existing -`test:e2e:terminal-perf:scale:report` for rendered typing/frame budgets. Unit tests do not +`test:e2e:terminal-perf:scale:report` ([guide](../docs/readme/terminal-perf.md)) for rendered typing/frame budgets. Unit tests do not measure SSH RTT, WSL scheduling or a packaged Electron renderer. To extend coverage, select a production-path regression with an operation-count, diff --git a/docs/readme/terminal-perf.md b/docs/readme/terminal-perf.md new file mode 100644 index 00000000000..a6f0ed53521 --- /dev/null +++ b/docs/readme/terminal-perf.md @@ -0,0 +1,67 @@ +# Terminal perf suite + +An on-demand Electron benchmark of terminal typing latency, frame timing, +output backpressure and hidden-pane restore. It drives the real renderer with +synthetic OpenCode-style redraw load across many panes and workspaces. It does +not run on PRs or on a schedule; run it before and after terminal rendering, +PTY, scheduler or xterm changes. + +## Run locally + +```sh +# Smaller suite: typing latency, foreground redraw, output scheduler, hidden TUI restore. +ORCA_BACKGROUND_LAUNCH=1 pnpm test:e2e:terminal-perf + +# Scale suite with JSON report, budget check and HTML report. +npx electron-vite build --mode e2e +ORCA_BACKGROUND_LAUNCH=1 SKIP_BUILD=1 pnpm test:e2e:terminal-perf:scale:report +``` + +On Linux, wrap the commands in `xvfb-run --auto-servernum` to keep windows off +your desktop. Do not set `ORCA_E2E_TERMINAL_PERF_XVFB` locally: it presents +frames on a GitHub-hosted Xvfb display and throws anywhere else. + +Pass Playwright arguments after `--`, for example +`pnpm test:e2e:terminal-perf:scale:report -- --grep "cross-workspace"`. + +## Outputs + +- `test-results/terminal-scale-perf-report.json`: the Playwright JSON report + (change with `--report ` or `ORCA_E2E_TERMINAL_PERF_REPORT_PATH`). +- A summary table on stdout (`test:e2e:terminal-perf:summarize `). +- A budget check (`test:e2e:terminal-perf:check-report `); budgets live + in `config/scripts/terminal-perf-report-budgets.mjs`. +- `test-results/terminal-perf-impact-report.html` + (`ORCA_E2E_TERMINAL_PERF_HTML_REPORT_PATH`). + +Compare reports from the base and candidate commits on the same machine; the +absolute numbers depend on hardware and display. + +## Load knobs + +| Variable | Default | +| ----------------------------------------------- | -------------- | +| `ORCA_E2E_OPENCODE_SCALE_PANES` | `10,25,50,100` | +| `ORCA_E2E_OPENCODE_SCALE_CROSS_WORKSPACE_PANES` | `10,25,50,100` | +| `ORCA_E2E_OPENCODE_SCALE_PRESSURE_PANES` | `25,50` | +| `ORCA_E2E_OPENCODE_SCALE_HIDDEN_PRESSURE_PANES` | `25` | +| `ORCA_E2E_OPENCODE_FRAME_COUNT` | `60` | +| `ORCA_E2E_OPENCODE_FRAME_INTERVAL_MS` | spec default | +| `ORCA_E2E_OPENCODE_PRESSURE_OUTPUT_CHARS` | spec default | + +## Run in CI + +Dispatch **Terminal Perf** (`.github/workflows/terminal-perf.yml`) from the +Actions tab or with `gh workflow run terminal-perf.yml -f ref=`. Its +inputs mirror the knobs above. It runs on a free GitHub-hosted runner and +uploads the JSON report, plus Playwright traces on failure. + +The nightly schedule was removed on 2026-10-28. It had failed every night since +at least 2026-10-18 because the workflow ran on Blacksmith, which the +presented-frame guard rejects, and the JSON report hid the error. + +The first hosted dispatch afterwards (run 40632739161) passed five scenarios. +It then failed "keeps active interactions responsive at 50 ACK-backpressured +OpenCode PTYs": the median typing latency was 104 ms against a 75 ms budget, +and the remaining scenarios were skipped. Reproduce locally before treating it +as a regression rather than a 4-vCPU runner limit.