Orca test inventory · snapshot 2026-08-12
E2E & integration coverage
A repository-based view of executable test inventory, CI platform execution, and recorded reliability-gate evidence. It separates tests that exist from tests that actually run on a platform.
Bottom line: the largest exposure is cross-platform live evidence. The standard Electron E2E suite is broad but Linux-only in CI. Windows has valuable targeted native, ConPTY, crash-survival, update, and computer-E2E lanes, but not broad desktop E2E parity.
Playwright Electron specs
246
Headless by default; headful for tagged specs
Vitest-included files
4,764
4,643 src · 101 config · 1 tool · 19 E2E unit
Computer E2E files
5
macOS 2 · Linux 1 · Windows 2
Reliability gates
77
64 partial protection · 13 no protection
Platform evidence
Counts overlap because a gate may target several platforms. “Evidence” means the gate metadata records coverage; it does not imply complete product coverage.
macOS62 / 71 gates
Linux4 / 69 gates
Windows3 / 73 gates
iOS1 / 2 gates
Android0 / 2 gates
Mobile0 / 4 gates
What runs where
| Layer | Current execution | Limitation |
|---|---|---|
| Standard Electron E2E | 246 tests/e2e/*.spec.ts; Playwright headless; 10 Ubuntu CI shards | Linux-biased macOS and Windows are not part of the default E2E workflow. |
| Headful Electron E2E | Tagged @headful specs through a separate Playwright project | Targeted Used for pointer capture and focus-sensitive paths. |
| Computer/accessibility E2E | Dedicated macOS, Linux, and Windows jobs | All 3 desktop OSes Specialized coverage, not full desktop-app parity. |
| Vitest contracts/integration | 4,764 included files; real Git, PTY, sockets, HTTP, SQLite, and relay fixtures appear in the suite | Mostly deterministic No formal integration-test taxonomy or source coverage percentage. |
| Windows native/recovery | Windows PR smoke, ConPTY tests, crash survival, update, and terminal restart workflows | Targeted Strongest Windows confidence is concentrated in terminal and recovery behavior. |
| SSH/WSL/paired runtime | Direct SSH, Docker relay, paired-runtime, and WSL fixtures | Scenario coverage Live evidence is frequently macOS-hosted; Linux/Windows clients and WSL remain gaps. |
Highest-value missing coverage
Windows behavior
- WSL restore and shell-resolution parity for PowerShell 5/7, cmd, Git Bash, WSL, and missing shells.
- ConPTY keyboard reset after agents/TUIs and modified-Enter routing.
- Windows CJK/wide-glyph repaint and pixel-refresh correctness.
- Windows input throughput and terminal backpressure budgets.
- Startup color-query leakage, restore convergence, and persisted-session recovery.
Cross-platform live journeys
- Paired-runtime and remote browser/terminal journeys with Linux and Windows clients.
- Real sleep/wake, network partitions, multi-client churn, and reconnect behavior.
- WSL-specific live process and PTY restore evidence.
- Mixed-version client/host journeys in both skew directions.
- CI soak history and per-OS flake tracking.
Registered gaps with no executable protection
These are the clearest candidates for adding tests, based on config/reliability-gates.jsonc.
| Area | Missing invariant | Target |
|---|---|---|
| Startup upgrade | Recover old production persisted sessions | macOS · Linux · Windows |
| Terminal provider | Preserve WSL launch identity, liveness, and restore boundaries | Windows |
| Terminal input | Reset Windows ConPTY keyboard protocol after agents/TUIs exit | Windows |
| Terminal rendering | Repaint Windows CJK/wide glyphs without stale cells | Windows |
| Terminal shell | Resolve Windows local/daemon shells and startup commands consistently | Windows |
| Terminal performance | Keep input latency and throughput within explicit budgets | macOS · Linux · Windows |
| Terminal capability | Prevent startup color queries leaking into streams | macOS · Linux · Windows |
| Observability | Detect renderer/mirror restore divergence | macOS · Linux · Windows |
| Terminal rendering | Prove forced repaint does not alter revealed-pane pixels | macOS · Linux · Windows |
The gate file contains 13 no-protection entries; nine are explicitly marked “Registered gap only.” The remaining four are tracked as pending-stack or contract gaps.
Recommended order
- Add a Windows Playwright smoke lane covering golden flows, terminal restore/input, file-browser drives, titlebar, settings, and onboarding.
- Wire ConPTY keyboard reset, shell parity, and CJK repaint into executable reliability gates.
- Promote Linux and Windows paired/SSH journeys from contracts to live CI evidence.
- Add explicit unit/integration/E2E/performance classification so coverage reports do not rely on filename heuristics.
- Track pass history and flake rate per OS.
Sources
- tests/playwright.config.ts — Playwright projects and matching.
- config/vitest.config.ts — Vitest inclusion.
- .github/workflows/e2e.yml — standard E2E CI.
- .github/workflows/computer-e2e.yml — platform-specific computer E2E.
- config/reliability-gates.jsonc — gates, evidence, and known gaps.