mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
ci: run terminal perf on demand only, on a hosted runner (#34726)
* ci: run terminal perf on demand only, on a hosted runner The nightly failed every run: the presented-frame guard accepts only GitHub-hosted Xvfb displays, but the workflow ran on Blacksmith. Drop the schedule, move dispatch runs to ubuntu-24.04 and document local use. * docs: record the first hosted terminal perf result --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
This commit is contained in:
@@ -1,5 +1,6 @@
|
||||
name: Terminal Perf
|
||||
|
||||
# On demand only; see docs/readme/terminal-perf.md.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
@@ -43,10 +44,6 @@ on:
|
||||
description: Comma-separated hidden real-PTY pressure pane counts
|
||||
required: false
|
||||
type: string
|
||||
schedule:
|
||||
# Why: keep the expensive scale run off PRs, but exercise it daily so
|
||||
# terminal throughput regressions are caught before users report typing lag.
|
||||
- cron: '30 8 * * *'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -57,7 +54,8 @@ cache-mode: read
|
||||
jobs:
|
||||
terminal-perf:
|
||||
name: terminal scale perf
|
||||
runs-on: blacksmith-4vcpu-ubuntu-2404
|
||||
# Presented-frame measurement only accepts GitHub-hosted Xvfb displays.
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 45
|
||||
env:
|
||||
NODE_OPTIONS: --max-old-space-size=4096
|
||||
|
||||
@@ -46,7 +46,7 @@ Incremental Claude scans and persisted Claude cache checkpoints are also include
|
||||
macOS and Windows, and on PRs changing this tooling or any listed contract file.
|
||||
It uploads per-OS JSON test results, plus the source inventory once from Linux
|
||||
because that scan is OS-independent. Its schedule starts after merge. Run the existing
|
||||
`test:e2e:terminal-perf:scale:report` for rendered typing/frame budgets. Unit tests do not
|
||||
`test:e2e:terminal-perf:scale:report` ([guide](../docs/readme/terminal-perf.md)) for rendered typing/frame budgets. Unit tests do not
|
||||
measure SSH RTT, WSL scheduling or a packaged Electron renderer.
|
||||
|
||||
To extend coverage, select a production-path regression with an operation-count,
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Terminal perf suite
|
||||
|
||||
An on-demand Electron benchmark of terminal typing latency, frame timing,
|
||||
output backpressure and hidden-pane restore. It drives the real renderer with
|
||||
synthetic OpenCode-style redraw load across many panes and workspaces. It does
|
||||
not run on PRs or on a schedule; run it before and after terminal rendering,
|
||||
PTY, scheduler or xterm changes.
|
||||
|
||||
## Run locally
|
||||
|
||||
```sh
|
||||
# Smaller suite: typing latency, foreground redraw, output scheduler, hidden TUI restore.
|
||||
ORCA_BACKGROUND_LAUNCH=1 pnpm test:e2e:terminal-perf
|
||||
|
||||
# Scale suite with JSON report, budget check and HTML report.
|
||||
npx electron-vite build --mode e2e
|
||||
ORCA_BACKGROUND_LAUNCH=1 SKIP_BUILD=1 pnpm test:e2e:terminal-perf:scale:report
|
||||
```
|
||||
|
||||
On Linux, wrap the commands in `xvfb-run --auto-servernum` to keep windows off
|
||||
your desktop. Do not set `ORCA_E2E_TERMINAL_PERF_XVFB` locally: it presents
|
||||
frames on a GitHub-hosted Xvfb display and throws anywhere else.
|
||||
|
||||
Pass Playwright arguments after `--`, for example
|
||||
`pnpm test:e2e:terminal-perf:scale:report -- --grep "cross-workspace"`.
|
||||
|
||||
## Outputs
|
||||
|
||||
- `test-results/terminal-scale-perf-report.json`: the Playwright JSON report
|
||||
(change with `--report <path>` or `ORCA_E2E_TERMINAL_PERF_REPORT_PATH`).
|
||||
- A summary table on stdout (`test:e2e:terminal-perf:summarize <report>`).
|
||||
- A budget check (`test:e2e:terminal-perf:check-report <report>`); budgets live
|
||||
in `config/scripts/terminal-perf-report-budgets.mjs`.
|
||||
- `test-results/terminal-perf-impact-report.html`
|
||||
(`ORCA_E2E_TERMINAL_PERF_HTML_REPORT_PATH`).
|
||||
|
||||
Compare reports from the base and candidate commits on the same machine; the
|
||||
absolute numbers depend on hardware and display.
|
||||
|
||||
## Load knobs
|
||||
|
||||
| Variable | Default |
|
||||
| ----------------------------------------------- | -------------- |
|
||||
| `ORCA_E2E_OPENCODE_SCALE_PANES` | `10,25,50,100` |
|
||||
| `ORCA_E2E_OPENCODE_SCALE_CROSS_WORKSPACE_PANES` | `10,25,50,100` |
|
||||
| `ORCA_E2E_OPENCODE_SCALE_PRESSURE_PANES` | `25,50` |
|
||||
| `ORCA_E2E_OPENCODE_SCALE_HIDDEN_PRESSURE_PANES` | `25` |
|
||||
| `ORCA_E2E_OPENCODE_FRAME_COUNT` | `60` |
|
||||
| `ORCA_E2E_OPENCODE_FRAME_INTERVAL_MS` | spec default |
|
||||
| `ORCA_E2E_OPENCODE_PRESSURE_OUTPUT_CHARS` | spec default |
|
||||
|
||||
## Run in CI
|
||||
|
||||
Dispatch **Terminal Perf** (`.github/workflows/terminal-perf.yml`) from the
|
||||
Actions tab or with `gh workflow run terminal-perf.yml -f ref=<branch>`. Its
|
||||
inputs mirror the knobs above. It runs on a free GitHub-hosted runner and
|
||||
uploads the JSON report, plus Playwright traces on failure.
|
||||
|
||||
The nightly schedule was removed on 2026-10-28. It had failed every night since
|
||||
at least 2026-10-18 because the workflow ran on Blacksmith, which the
|
||||
presented-frame guard rejects, and the JSON report hid the error.
|
||||
|
||||
The first hosted dispatch afterwards (run 40632739161) passed five scenarios.
|
||||
It then failed "keeps active interactions responsive at 50 ACK-backpressured
|
||||
OpenCode PTYs": the median typing latency was 104 ms against a 75 ms budget,
|
||||
and the remaining scenarios were skipped. Reproduce locally before treating it
|
||||
as a regression rather than a 4-vCPU runner limit.
|
||||
Reference in New Issue
Block a user