ci: run terminal perf on demand only, on a hosted runner (#34726)

* ci: run terminal perf on demand only, on a hosted runner

The nightly failed every run: the presented-frame guard accepts only
GitHub-hosted Xvfb displays, but the workflow ran on Blacksmith. Drop the
schedule, move dispatch runs to ubuntu-24.04 and document local use.

* docs: record the first hosted terminal perf result

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
This commit is contained in:
OrcaWin
2026-10-27 21:48:08 -07:00
committed by GitHub
co-authored by OrcaWin
parent 32a50856ac
commit 8b7f8c88ce
3 changed files with 71 additions and 6 deletions
+3 -5
View File
@@ -1,5 +1,6 @@
name: Terminal Perf
# On demand only; see docs/readme/terminal-perf.md.
on:
workflow_dispatch:
inputs:
@@ -43,10 +44,6 @@ on:
description: Comma-separated hidden real-PTY pressure pane counts
required: false
type: string
schedule:
# Why: keep the expensive scale run off PRs, but exercise it daily so
# terminal throughput regressions are caught before users report typing lag.
- cron: '30 8 * * *'
permissions:
contents: read
@@ -57,7 +54,8 @@ cache-mode: read
jobs:
terminal-perf:
name: terminal scale perf
runs-on: blacksmith-4vcpu-ubuntu-2404
# Presented-frame measurement only accepts GitHub-hosted Xvfb displays.
runs-on: ubuntu-24.04
timeout-minutes: 45
env:
NODE_OPTIONS: --max-old-space-size=4096
+1 -1
View File
@@ -46,7 +46,7 @@ Incremental Claude scans and persisted Claude cache checkpoints are also include
macOS and Windows, and on PRs changing this tooling or any listed contract file.
It uploads per-OS JSON test results, plus the source inventory once from Linux
because that scan is OS-independent. Its schedule starts after merge. Run the existing
`test:e2e:terminal-perf:scale:report` for rendered typing/frame budgets. Unit tests do not
`test:e2e:terminal-perf:scale:report` ([guide](../docs/readme/terminal-perf.md)) for rendered typing/frame budgets. Unit tests do not
measure SSH RTT, WSL scheduling or a packaged Electron renderer.
To extend coverage, select a production-path regression with an operation-count,
+67
View File
@@ -0,0 +1,67 @@
# Terminal perf suite
An on-demand Electron benchmark of terminal typing latency, frame timing,
output backpressure and hidden-pane restore. It drives the real renderer with
synthetic OpenCode-style redraw load across many panes and workspaces. It does
not run on PRs or on a schedule; run it before and after terminal rendering,
PTY, scheduler or xterm changes.
## Run locally
```sh
# Smaller suite: typing latency, foreground redraw, output scheduler, hidden TUI restore.
ORCA_BACKGROUND_LAUNCH=1 pnpm test:e2e:terminal-perf
# Scale suite with JSON report, budget check and HTML report.
npx electron-vite build --mode e2e
ORCA_BACKGROUND_LAUNCH=1 SKIP_BUILD=1 pnpm test:e2e:terminal-perf:scale:report
```
On Linux, wrap the commands in `xvfb-run --auto-servernum` to keep windows off
your desktop. Do not set `ORCA_E2E_TERMINAL_PERF_XVFB` locally: it presents
frames on a GitHub-hosted Xvfb display and throws anywhere else.
Pass Playwright arguments after `--`, for example
`pnpm test:e2e:terminal-perf:scale:report -- --grep "cross-workspace"`.
## Outputs
- `test-results/terminal-scale-perf-report.json`: the Playwright JSON report
(change with `--report <path>` or `ORCA_E2E_TERMINAL_PERF_REPORT_PATH`).
- A summary table on stdout (`test:e2e:terminal-perf:summarize <report>`).
- A budget check (`test:e2e:terminal-perf:check-report <report>`); budgets live
in `config/scripts/terminal-perf-report-budgets.mjs`.
- `test-results/terminal-perf-impact-report.html`
(`ORCA_E2E_TERMINAL_PERF_HTML_REPORT_PATH`).
Compare reports from the base and candidate commits on the same machine; the
absolute numbers depend on hardware and display.
## Load knobs
| Variable | Default |
| ----------------------------------------------- | -------------- |
| `ORCA_E2E_OPENCODE_SCALE_PANES` | `10,25,50,100` |
| `ORCA_E2E_OPENCODE_SCALE_CROSS_WORKSPACE_PANES` | `10,25,50,100` |
| `ORCA_E2E_OPENCODE_SCALE_PRESSURE_PANES` | `25,50` |
| `ORCA_E2E_OPENCODE_SCALE_HIDDEN_PRESSURE_PANES` | `25` |
| `ORCA_E2E_OPENCODE_FRAME_COUNT` | `60` |
| `ORCA_E2E_OPENCODE_FRAME_INTERVAL_MS` | spec default |
| `ORCA_E2E_OPENCODE_PRESSURE_OUTPUT_CHARS` | spec default |
## Run in CI
Dispatch **Terminal Perf** (`.github/workflows/terminal-perf.yml`) from the
Actions tab or with `gh workflow run terminal-perf.yml -f ref=<branch>`. Its
inputs mirror the knobs above. It runs on a free GitHub-hosted runner and
uploads the JSON report, plus Playwright traces on failure.
The nightly schedule was removed on 2026-10-28. It had failed every night since
at least 2026-10-18 because the workflow ran on Blacksmith, which the
presented-frame guard rejects, and the JSON report hid the error.
The first hosted dispatch afterwards (run 40632739161) passed five scenarios.
It then failed "keeps active interactions responsive at 50 ACK-backpressured
OpenCode PTYs": the median typing latency was 104 ms against a 75 ms budget,
and the remaining scenarios were skipped. Reproduce locally before treating it
as a regression rather than a 4-vCPU runner limit.