Files
orca/.github/workflows/e2e.yml
T
Jinjing 7a695c70f1 test(e2e): harden triaged CI failures (#14656)
* test(e2e): harden triaged failures

* test(e2e): ship relay bundle to reusable shards

* test(e2e): tolerate expected IPC closures in daemon shutdown

A normal client exit can close the IPC channel before the finish ack
lands. Distinguish this from real failures by checking error codes,
only throwing if forced cleanup occurred or the error is not an IPC
closure.

* rm doc

* test(e2e): return termination status from legacy close handler

- terminateLegacyCloseClient now returns a discriminated union indicating
  whether the process had already exited ('already-exited') or termination
  was actually attempted ('termination-attempted')
- Allows finishLegacyCloseClient to only set forcedCleanup when termination
  was genuinely needed, not when the process exited cleanly on its own

* test(e2e): fix dispatch contract and voice mic locator

Point the release E2E contract at the renamed build step, and assert the
relabeled microphone through the Voice pane combobox even when Radix
leaves the listbox open.

* test(e2e): add contract test for relay artifact dispatch

Validate that the relay artifact built in CI is properly uploaded,
downloaded, and passed via ORCA_RELAY_PATH to E2E test runs.

* Distinguish between terminated and already-exited processes

Detect when processes have already exited instead of always reporting
termination success. Return booleans from cleanup functions to indicate
whether they actually signalled a process, catch tree-capture failures
when the root process exits before recording completes, and use these
signals to return accurate exit status from termination handlers.

* test(e2e): stabilize file creation and voice microphone tests

Use stable locators (aria-autocomplete, named triggers) and add retry
logic to handle file scans and device events that can interfere with
listbox state. Increase timeouts to allow async operations to complete.

* Add retry logic for transient GitHub API errors in PR body updates

GitHub API occasionally returns transient 5xx errors. Retry up to 3 times
with exponential backoff (1s, 2s, 4s) to improve reliability during
temporary service disruptions. Export updatePullRequest and add sleepImpl
parameter for test injection.

* Add tab search result retention during typing

Keep search results on screen while the deferred query catches up with
the live query. Re-validates results against the current input without
dropping rows prematurely, ensuring the user can select from what they see.

* Add proper types to tab search mock

Replace `unknown` with concrete types (`OpenTabSearchResult`,
`OpenTabSearchEntries`, `SearchableWorkspaceTab`) and use type guards
for discriminated unions to improve test type safety.
2026-08-17 23:28:38 -07:00

351 lines
13 KiB
YAML

name: E2E
run-name: E2E ${{ inputs.ref || github.ref }}
# Why: checkout + artifact upload only; callers can only further restrict.
permissions:
contents: read
on:
workflow_call:
inputs:
ref:
description: Ref to check out (defaults to the calling workflow's ref)
required: false
type: string
test_files:
description: JSON array of changed specs; empty runs the full suite
required: false
type: string
workflow_dispatch:
inputs:
ref:
description: Ref to check out (defaults to the workflow ref)
required: false
type: string
schedule:
# Why: GitHub cron uses UTC; these slots map to 10am and 3pm
# America/Phoenix for the default-branch E2E run.
- cron: '0 17,22 * * *'
jobs:
build:
name: build e2e app
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
# Why: the E2E build compiles native modules via node-gyp. Mirrors the
# install step in pr.yml's verify job so E2E doesn't hit missing-toolchain
# errors.
- name: Install native build tools
run: sudo apt-get update && sudo apt-get install -y build-essential python3
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
# Without that cache every E2E job re-downloaded the whole dependency set.
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
run_install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
cache: pnpm
# Why: this job runs the same pnpm install path as pr.yml's verify
# job, so it needs the same pinned node-gyp override to avoid pnpm's
# broken bundled gyp_main.py on Linux.
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
if: runner.os == 'Linux'
run: |
npm install -g node-gyp@11.5.0
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
- name: Install dependencies
run: pnpm install --frozen-lockfile
# Why: building here avoids parallel builds inside Playwright globalSetup;
# paired-browser specs also need the standalone web bundle.
- name: Build E2E outputs
env:
VITE_EXPOSE_STORE: 'true'
run: |
npx electron-vite build --mode e2e
pnpm run build:web-from-renderer
pnpm run build:relay
- name: Upload E2E build output
uses: actions/upload-artifact@v7
with:
name: e2e-build-out
path: out/
retention-days: 1
if-no-files-found: error
e2e:
name: e2e ${{ matrix.shard_name }}
needs: build
if: inputs.test_files == ''
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- shard: '1/10'
shard_name: 1-of-10
- shard: '2/10'
shard_name: 2-of-10
- shard: '3/10'
shard_name: 3-of-10
- shard: '4/10'
shard_name: 4-of-10
- shard: '5/10'
shard_name: 5-of-10
- shard: '6/10'
shard_name: 6-of-10
- shard: '7/10'
shard_name: 7-of-10
- shard: '8/10'
shard_name: 8-of-10
- shard: '9/10'
shard_name: 9-of-10
- shard: '10/10'
shard_name: 10-of-10
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
# Why: pnpm install rebuilds native modules, and those postinstall
# scripts still need the Linux toolchain even though this shard reuses
# the prebuilt Electron output.
- name: Install native build tools
run: sudo apt-get update && sudo apt-get install -y build-essential python3 zsh
# Why: Electron on Linux needs an X display even when the app
# suppresses mainWindow.show() via ORCA_E2E_HEADLESS. xvfb provides a
# virtual framebuffer so Chromium can initialize without a real display.
- name: Install xvfb
run: sudo apt-get install -y xvfb
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
# Without that cache every E2E job re-downloaded the whole dependency set.
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
run_install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
cache: pnpm
# Why: this job runs the same pnpm install path as pr.yml's verify
# job, so it needs the same pinned node-gyp override to avoid pnpm's
# broken bundled gyp_main.py on Linux. Gate on runner.os matches
# release.yml so the invariant "this workaround is Linux-only" is
# consistent across all three workflows, even though this job
# currently pins runs-on: ubuntu-latest.
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
if: runner.os == 'Linux'
run: |
npm install -g node-gyp@11.5.0
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Download E2E build output
uses: actions/download-artifact@v8
with:
name: e2e-build-out
path: out/
# Why: the Electron suite is wall-clock constrained on OSS runners, but
# multiple Electron apps on one Xvfb VM contend on git/Chromium resources.
# Sharding keeps each VM at one Playwright worker while splitting the
# headless suite across separate runners.
# SKIP_BUILD makes Playwright globalSetup reuse the single build job's
# artifact instead of starting five concurrent electron-vite builds.
# ORCA_E2E_FORWARD_APP_LOGS keeps startup failures visible when Electron
# launches but never creates a BrowserWindow.
- name: Run E2E tests (${{ matrix.shard_name }})
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay" pnpm run test:e2e --shard=${{ matrix.shard }}
# Why: Playwright retains traces/screenshots only on failure. Uploading
# them as an artifact makes post-mortem debugging on CI possible without
# re-running locally.
- name: Upload Playwright traces
if: failure()
uses: actions/upload-artifact@v7
with:
name: playwright-traces-${{ matrix.shard_name }}
path: test-results/
retention-days: 7
if-no-files-found: ignore
changed-e2e:
name: changed e2e specs
needs: build
if: inputs.test_files != ''
runs-on: ubuntu-latest
# Why 45: pr.yml now maps SSH source edits onto Docker-backed specs, so this lane can
# pay a container image build plus ~22 serial SSH tests on top of the changed specs.
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- name: Install native build and headless UI tools
# Why openssh-client: the Docker-SSH fixture shells out to ssh/ssh-keygen, and this
# lane now receives those specs from pr.yml's SSH source mapping.
run: sudo apt-get update && sudo apt-get install -y build-essential openssh-client python3 xvfb zsh
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
# Without that cache every E2E job re-downloaded the whole dependency set.
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
run_install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
cache: pnpm
- name: Use external node-gyp to avoid pnpm's bundled copy
run: |
npm install -g node-gyp@11.5.0
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Download E2E build output
uses: actions/download-artifact@v8
with:
name: e2e-build-out
path: out/
- name: Run changed E2E specs
env:
TEST_FILES_JSON: ${{ inputs.test_files }}
run: |
mapfile -t TEST_FILES < <(jq -r '.[] | select(
. != "tests/e2e/ssh-startup-exec-readiness.spec.ts" and
. != "tests/e2e/paired-startup-exec-readiness.spec.ts"
)' <<<"$TEST_FILES_JSON")
if [ "${#TEST_FILES[@]}" -eq 0 ]; then
echo "Changed startup-readiness specs are owned by the dedicated live lane."
exit 0
fi
E2E_ENV=(SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay")
# Second clause: a spec that reads ORCA_E2E_SSH_DOCKER test.skip()s itself without it, so
# naming only one trigger silently skipped every other Docker-SSH spec in this lane.
# The first clause stays because that spec needs Docker without referencing the variable.
if printf '%s\n' "${TEST_FILES[@]}" | grep -qx 'tests/e2e/ephemeral-vm-provisioned-root.spec.ts' \
|| grep -l 'ORCA_E2E_SSH_DOCKER' "${TEST_FILES[@]}" >/dev/null 2>&1; then
E2E_ENV+=(ORCA_E2E_SSH_DOCKER=1)
fi
E2E_PROJECT_ARGS=()
if grep -l '@headful' "${TEST_FILES[@]}" >/dev/null; then
E2E_PROJECT_ARGS+=(--project=electron-headful)
fi
xvfb-run --auto-servernum env "${E2E_ENV[@]}" \
pnpm run test:e2e "${TEST_FILES[@]}" --workers=1 "${E2E_PROJECT_ARGS[@]}"
- name: Upload Playwright traces
if: failure()
uses: actions/upload-artifact@v7
with:
name: playwright-traces-changed
path: test-results/
retention-days: 7
if-no-files-found: ignore
ssh-docker-watcher-isolation:
name: ssh docker watcher isolation
needs: build
if: >-
inputs.test_files == '' ||
contains(inputs.test_files, 'tests/e2e/ssh-startup-exec-readiness.spec.ts') ||
contains(inputs.test_files, 'tests/e2e/paired-startup-exec-readiness.spec.ts')
runs-on: ubuntu-latest
# Why 35: the parking, retention, startup-exec, and paired parity specs run
# serially on isolated Electron/SSH fixtures after watcher isolation.
timeout-minutes: 35
steps:
- name: Checkout
uses: actions/checkout@v6
with:
ref: ${{ inputs.ref || github.ref }}
- name: Install native build and headless UI tools
run: sudo apt-get update && sudo apt-get install -y build-essential openssh-client python3 xvfb zsh
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
# Without that cache every E2E job re-downloaded the whole dependency set.
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
run_install: false
- name: Setup Node.js
uses: actions/setup-node@v6
with:
node-version-file: package.json
cache: pnpm
# Why: same Linux-only node-gyp pin as build/e2e jobs so the workaround
# stays consistent across workflows even while this job is ubuntu-latest.
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
if: runner.os == 'Linux'
run: |
npm install -g node-gyp@11.5.0
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Download E2E build output
uses: actions/download-artifact@v8
with:
name: e2e-build-out
path: out/
# Why: this is the release-path proof that the deployed Linux relay keeps
# its PTY and explorer live across a real watcher SIGSEGV.
- name: Run Docker SSH watcher isolation E2E
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-watcher-isolation
# Why always(): this lane gates SSH parking/retention plus startup-exec
# readiness across live SSH, headed paired, and headless serve topologies.
- name: Run Docker SSH terminal parking + startup readiness E2E
if: always()
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-terminal-parking
- name: Upload watcher isolation traces
if: failure()
uses: actions/upload-artifact@v7
with:
name: playwright-traces-ssh-docker-watcher-isolation
path: test-results/
retention-days: 7
if-no-files-found: ignore