mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
* test(e2e): harden triaged failures
* test(e2e): ship relay bundle to reusable shards
* test(e2e): tolerate expected IPC closures in daemon shutdown
A normal client exit can close the IPC channel before the finish ack
lands. Distinguish this from real failures by checking error codes,
only throwing if forced cleanup occurred or the error is not an IPC
closure.
* rm doc
* test(e2e): return termination status from legacy close handler
- terminateLegacyCloseClient now returns a discriminated union indicating
whether the process had already exited ('already-exited') or termination
was actually attempted ('termination-attempted')
- Allows finishLegacyCloseClient to only set forcedCleanup when termination
was genuinely needed, not when the process exited cleanly on its own
* test(e2e): fix dispatch contract and voice mic locator
Point the release E2E contract at the renamed build step, and assert the
relabeled microphone through the Voice pane combobox even when Radix
leaves the listbox open.
* test(e2e): add contract test for relay artifact dispatch
Validate that the relay artifact built in CI is properly uploaded,
downloaded, and passed via ORCA_RELAY_PATH to E2E test runs.
* Distinguish between terminated and already-exited processes
Detect when processes have already exited instead of always reporting
termination success. Return booleans from cleanup functions to indicate
whether they actually signalled a process, catch tree-capture failures
when the root process exits before recording completes, and use these
signals to return accurate exit status from termination handlers.
* test(e2e): stabilize file creation and voice microphone tests
Use stable locators (aria-autocomplete, named triggers) and add retry
logic to handle file scans and device events that can interfere with
listbox state. Increase timeouts to allow async operations to complete.
* Add retry logic for transient GitHub API errors in PR body updates
GitHub API occasionally returns transient 5xx errors. Retry up to 3 times
with exponential backoff (1s, 2s, 4s) to improve reliability during
temporary service disruptions. Export updatePullRequest and add sleepImpl
parameter for test injection.
* Add tab search result retention during typing
Keep search results on screen while the deferred query catches up with
the live query. Re-validates results against the current input without
dropping rows prematurely, ensuring the user can select from what they see.
* Add proper types to tab search mock
Replace `unknown` with concrete types (`OpenTabSearchResult`,
`OpenTabSearchEntries`, `SearchableWorkspaceTab`) and use type guards
for discriminated unions to improve test type safety.
351 lines
13 KiB
YAML
351 lines
13 KiB
YAML
name: E2E
|
|
|
|
run-name: E2E ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: checkout + artifact upload only; callers can only further restrict.
|
|
permissions:
|
|
contents: read
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the calling workflow's ref)
|
|
required: false
|
|
type: string
|
|
test_files:
|
|
description: JSON array of changed specs; empty runs the full suite
|
|
required: false
|
|
type: string
|
|
workflow_dispatch:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the workflow ref)
|
|
required: false
|
|
type: string
|
|
schedule:
|
|
# Why: GitHub cron uses UTC; these slots map to 10am and 3pm
|
|
# America/Phoenix for the default-branch E2E run.
|
|
- cron: '0 17,22 * * *'
|
|
|
|
jobs:
|
|
build:
|
|
name: build e2e app
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: the E2E build compiles native modules via node-gyp. Mirrors the
|
|
# install step in pr.yml's verify job so E2E doesn't hit missing-toolchain
|
|
# errors.
|
|
- name: Install native build tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
|
|
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
|
|
# Without that cache every E2E job re-downloaded the whole dependency set.
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
cache: pnpm
|
|
|
|
# Why: this job runs the same pnpm install path as pr.yml's verify
|
|
# job, so it needs the same pinned node-gyp override to avoid pnpm's
|
|
# broken bundled gyp_main.py on Linux.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
# Why: building here avoids parallel builds inside Playwright globalSetup;
|
|
# paired-browser specs also need the standalone web bundle.
|
|
- name: Build E2E outputs
|
|
env:
|
|
VITE_EXPOSE_STORE: 'true'
|
|
run: |
|
|
npx electron-vite build --mode e2e
|
|
pnpm run build:web-from-renderer
|
|
pnpm run build:relay
|
|
|
|
- name: Upload E2E build output
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
retention-days: 1
|
|
if-no-files-found: error
|
|
|
|
e2e:
|
|
name: e2e ${{ matrix.shard_name }}
|
|
needs: build
|
|
if: inputs.test_files == ''
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard: '1/10'
|
|
shard_name: 1-of-10
|
|
- shard: '2/10'
|
|
shard_name: 2-of-10
|
|
- shard: '3/10'
|
|
shard_name: 3-of-10
|
|
- shard: '4/10'
|
|
shard_name: 4-of-10
|
|
- shard: '5/10'
|
|
shard_name: 5-of-10
|
|
- shard: '6/10'
|
|
shard_name: 6-of-10
|
|
- shard: '7/10'
|
|
shard_name: 7-of-10
|
|
- shard: '8/10'
|
|
shard_name: 8-of-10
|
|
- shard: '9/10'
|
|
shard_name: 9-of-10
|
|
- shard: '10/10'
|
|
shard_name: 10-of-10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: pnpm install rebuilds native modules, and those postinstall
|
|
# scripts still need the Linux toolchain even though this shard reuses
|
|
# the prebuilt Electron output.
|
|
- name: Install native build tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential python3 zsh
|
|
|
|
# Why: Electron on Linux needs an X display even when the app
|
|
# suppresses mainWindow.show() via ORCA_E2E_HEADLESS. xvfb provides a
|
|
# virtual framebuffer so Chromium can initialize without a real display.
|
|
- name: Install xvfb
|
|
run: sudo apt-get install -y xvfb
|
|
|
|
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
|
|
# Without that cache every E2E job re-downloaded the whole dependency set.
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
cache: pnpm
|
|
|
|
# Why: this job runs the same pnpm install path as pr.yml's verify
|
|
# job, so it needs the same pinned node-gyp override to avoid pnpm's
|
|
# broken bundled gyp_main.py on Linux. Gate on runner.os matches
|
|
# release.yml so the invariant "this workaround is Linux-only" is
|
|
# consistent across all three workflows, even though this job
|
|
# currently pins runs-on: ubuntu-latest.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: the Electron suite is wall-clock constrained on OSS runners, but
|
|
# multiple Electron apps on one Xvfb VM contend on git/Chromium resources.
|
|
# Sharding keeps each VM at one Playwright worker while splitting the
|
|
# headless suite across separate runners.
|
|
# SKIP_BUILD makes Playwright globalSetup reuse the single build job's
|
|
# artifact instead of starting five concurrent electron-vite builds.
|
|
# ORCA_E2E_FORWARD_APP_LOGS keeps startup failures visible when Electron
|
|
# launches but never creates a BrowserWindow.
|
|
- name: Run E2E tests (${{ matrix.shard_name }})
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay" pnpm run test:e2e --shard=${{ matrix.shard }}
|
|
|
|
# Why: Playwright retains traces/screenshots only on failure. Uploading
|
|
# them as an artifact makes post-mortem debugging on CI possible without
|
|
# re-running locally.
|
|
- name: Upload Playwright traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-${{ matrix.shard_name }}
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
changed-e2e:
|
|
name: changed e2e specs
|
|
needs: build
|
|
if: inputs.test_files != ''
|
|
runs-on: ubuntu-latest
|
|
# Why 45: pr.yml now maps SSH source edits onto Docker-backed specs, so this lane can
|
|
# pay a container image build plus ~22 serial SSH tests on top of the changed specs.
|
|
timeout-minutes: 45
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Install native build and headless UI tools
|
|
# Why openssh-client: the Docker-SSH fixture shells out to ssh/ssh-keygen, and this
|
|
# lane now receives those specs from pr.yml's SSH source mapping.
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential openssh-client python3 xvfb zsh
|
|
|
|
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
|
|
# Without that cache every E2E job re-downloaded the whole dependency set.
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
cache: pnpm
|
|
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
- name: Run changed E2E specs
|
|
env:
|
|
TEST_FILES_JSON: ${{ inputs.test_files }}
|
|
run: |
|
|
mapfile -t TEST_FILES < <(jq -r '.[] | select(
|
|
. != "tests/e2e/ssh-startup-exec-readiness.spec.ts" and
|
|
. != "tests/e2e/paired-startup-exec-readiness.spec.ts"
|
|
)' <<<"$TEST_FILES_JSON")
|
|
if [ "${#TEST_FILES[@]}" -eq 0 ]; then
|
|
echo "Changed startup-readiness specs are owned by the dedicated live lane."
|
|
exit 0
|
|
fi
|
|
E2E_ENV=(SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay")
|
|
# Second clause: a spec that reads ORCA_E2E_SSH_DOCKER test.skip()s itself without it, so
|
|
# naming only one trigger silently skipped every other Docker-SSH spec in this lane.
|
|
# The first clause stays because that spec needs Docker without referencing the variable.
|
|
if printf '%s\n' "${TEST_FILES[@]}" | grep -qx 'tests/e2e/ephemeral-vm-provisioned-root.spec.ts' \
|
|
|| grep -l 'ORCA_E2E_SSH_DOCKER' "${TEST_FILES[@]}" >/dev/null 2>&1; then
|
|
E2E_ENV+=(ORCA_E2E_SSH_DOCKER=1)
|
|
fi
|
|
E2E_PROJECT_ARGS=()
|
|
if grep -l '@headful' "${TEST_FILES[@]}" >/dev/null; then
|
|
E2E_PROJECT_ARGS+=(--project=electron-headful)
|
|
fi
|
|
xvfb-run --auto-servernum env "${E2E_ENV[@]}" \
|
|
pnpm run test:e2e "${TEST_FILES[@]}" --workers=1 "${E2E_PROJECT_ARGS[@]}"
|
|
|
|
- name: Upload Playwright traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-changed
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
ssh-docker-watcher-isolation:
|
|
name: ssh docker watcher isolation
|
|
needs: build
|
|
if: >-
|
|
inputs.test_files == '' ||
|
|
contains(inputs.test_files, 'tests/e2e/ssh-startup-exec-readiness.spec.ts') ||
|
|
contains(inputs.test_files, 'tests/e2e/paired-startup-exec-readiness.spec.ts')
|
|
runs-on: ubuntu-latest
|
|
# Why 35: the parking, retention, startup-exec, and paired parity specs run
|
|
# serially on isolated Electron/SSH fixtures after watcher isolation.
|
|
timeout-minutes: 35
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Install native build and headless UI tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential openssh-client python3 xvfb zsh
|
|
|
|
# Why pnpm first: setup-node needs pnpm on PATH to locate the store it caches.
|
|
# Without that cache every E2E job re-downloaded the whole dependency set.
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
cache: pnpm
|
|
|
|
# Why: same Linux-only node-gyp pin as build/e2e jobs so the workaround
|
|
# stays consistent across workflows even while this job is ubuntu-latest.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: this is the release-path proof that the deployed Linux relay keeps
|
|
# its PTY and explorer live across a real watcher SIGSEGV.
|
|
- name: Run Docker SSH watcher isolation E2E
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-watcher-isolation
|
|
|
|
# Why always(): this lane gates SSH parking/retention plus startup-exec
|
|
# readiness across live SSH, headed paired, and headless serve topologies.
|
|
- name: Run Docker SSH terminal parking + startup readiness E2E
|
|
if: always()
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-terminal-parking
|
|
|
|
- name: Upload watcher isolation traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-ssh-docker-watcher-isolation
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|