mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
* ci(pr): run E2E when a PR touches tests/e2e paths Regression specs under tests/e2e never ran on PR CI — only schedule and release called e2e.yml — so a red regression test could merge green. Path-filter and workflow_call the E2E suite when E2E-relevant files change. Use merge-base diffs so base-branch drift does not false-trigger E2E, fail the detector when git diff cannot compute the PR range, and pin least-privilege contents:read on both the detector and reusable E2E workflow. Closes #10518 Co-authored-by: Wooseong Kim <innocarpe@gmail.com> Co-authored-by: Orca <help@stably.ai> * ci(pr): make the E2E path gate actually block, and match the real config path Two fixes to the new path-filtered E2E job. The gate did not gate. pr.yml's `verify` job is the required check, and it enumerates its dependencies explicitly — `e2e` was in neither `needs` nor the result list, so a failing shard left `verify` green. That reproduces the exact hole this job exists to close: a red spec merges green, just with a red box further down the page. Add `e2e` to both. Because the job is path-filtered, `skipped` is the normal result on a PR that touches no E2E files and has to keep passing. That allowance is checked after the strict loop rather than inside it, so it can never leak to the six jobs that are always required. The `playwright.` pattern matched nothing. The config is tests/playwright.config.ts — beside tests/e2e/, not inside it — so no tracked file starts with `playwright.` and editing the runner config would silently skip E2E. Anchor it at `tests/playwright.`. Adds a contract test alongside the existing release-e2e one. Verified it fails when either fix is reverted, and simulated the gate across success/skipped/failure/cancelled plus the skip-must-not-mask-a-real-failure case. * test(ci): close two gaps in the E2E gate contract CodeRabbit was right on both counts — verified by reverting each and watching the contract stay green. The path filter was unasserted, so `e2e` could lose its `if:` and run on every PR — the cost the filter exists to avoid — without failing anything. The strict-loop check hardcoded four of the six required jobs, so dropping GIT_COMPATIBILITY or SHELL_CONTRACTS left them unenforced while the contract passed. Derive the list from verify.needs instead, so a newly added required job that misses the loop fails here rather than silently going unchecked. * ci(pr): land the E2E path gate advisory instead of blocking The E2E suite is currently failing every scheduled run on main — 22 of the last 22 — so making verify depend on it would block any PR touching tests/e2e/**, including the PRs that fix the suite. This PR's own run reproduced that: 3 of 12 shards failed on specs unrelated to it (agent-session resume, Jira linking, plugin containment, terminal artifacts). So the job runs and reports on E2E-path PRs but is left out of verify.needs for now. The detector, the tests/playwright. path fix, and the contract tests are unaffected — those stand on their own and were the substance of the review. Flipping to blocking is a three-line change once the suite is green; the exact wiring, including why the skipped allowance must sit outside the strict loop, is recorded on verify's Require-successful-checks step. The contract test pins the advisory choice so it reads as deliberate rather than as the unwired-gate bug it originally caught, and still fails if the path filter, the strict-loop coverage, or the config path regress. --------- Co-authored-by: Wooseong Kim <innocarpe@gmail.com> Co-authored-by: Orca <help@stably.ai>
238 lines
8.0 KiB
YAML
238 lines
8.0 KiB
YAML
name: E2E
|
|
|
|
run-name: E2E ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: checkout + artifact upload only; callers can only further restrict.
|
|
permissions:
|
|
contents: read
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the calling workflow's ref)
|
|
required: false
|
|
type: string
|
|
workflow_dispatch:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the workflow ref)
|
|
required: false
|
|
type: string
|
|
schedule:
|
|
# Why: GitHub cron uses UTC; these slots map to 10am and 3pm
|
|
# America/Phoenix for the default-branch E2E run.
|
|
- cron: '0 17,22 * * *'
|
|
|
|
jobs:
|
|
build:
|
|
name: build e2e app
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: the E2E build compiles native modules via node-gyp. Mirrors the
|
|
# install step in pr.yml's verify job so E2E doesn't hit missing-toolchain
|
|
# errors.
|
|
- name: Install native build tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
# Why: this job runs the same pnpm install path as pr.yml's verify
|
|
# job, so it needs the same pinned node-gyp override to avoid pnpm's
|
|
# broken bundled gyp_main.py on Linux.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
# Why: building once avoids five parallel electron-vite builds inside
|
|
# Playwright globalSetup, which otherwise contends for CPU/RAM on OSS
|
|
# runners before the sharded tests even start.
|
|
- name: Build Electron app for E2E
|
|
run: npx electron-vite build --mode e2e
|
|
|
|
- name: Upload E2E build output
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
retention-days: 1
|
|
if-no-files-found: error
|
|
|
|
e2e:
|
|
name: e2e ${{ matrix.shard_name }}
|
|
needs: build
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard: '1/10'
|
|
shard_name: 1-of-10
|
|
- shard: '2/10'
|
|
shard_name: 2-of-10
|
|
- shard: '3/10'
|
|
shard_name: 3-of-10
|
|
- shard: '4/10'
|
|
shard_name: 4-of-10
|
|
- shard: '5/10'
|
|
shard_name: 5-of-10
|
|
- shard: '6/10'
|
|
shard_name: 6-of-10
|
|
- shard: '7/10'
|
|
shard_name: 7-of-10
|
|
- shard: '8/10'
|
|
shard_name: 8-of-10
|
|
- shard: '9/10'
|
|
shard_name: 9-of-10
|
|
- shard: '10/10'
|
|
shard_name: 10-of-10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: pnpm install rebuilds native modules, and those postinstall
|
|
# scripts still need the Linux toolchain even though this shard reuses
|
|
# the prebuilt Electron output.
|
|
- name: Install native build tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential python3
|
|
|
|
# Why: Electron on Linux needs an X display even when the app
|
|
# suppresses mainWindow.show() via ORCA_E2E_HEADLESS. xvfb provides a
|
|
# virtual framebuffer so Chromium can initialize without a real display.
|
|
- name: Install xvfb
|
|
run: sudo apt-get install -y xvfb
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
# Why: this job runs the same pnpm install path as pr.yml's verify
|
|
# job, so it needs the same pinned node-gyp override to avoid pnpm's
|
|
# broken bundled gyp_main.py on Linux. Gate on runner.os matches
|
|
# release.yml so the invariant "this workaround is Linux-only" is
|
|
# consistent across all three workflows, even though this job
|
|
# currently pins runs-on: ubuntu-latest.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: the Electron suite is wall-clock constrained on OSS runners, but
|
|
# multiple Electron apps on one Xvfb VM contend on git/Chromium resources.
|
|
# Sharding keeps each VM at one Playwright worker while splitting the
|
|
# headless suite across separate runners.
|
|
# SKIP_BUILD makes Playwright globalSetup reuse the single build job's
|
|
# artifact instead of starting five concurrent electron-vite builds.
|
|
# ORCA_E2E_FORWARD_APP_LOGS keeps startup failures visible when Electron
|
|
# launches but never creates a BrowserWindow.
|
|
- name: Run E2E tests (${{ matrix.shard_name }})
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e --shard=${{ matrix.shard }}
|
|
|
|
# Why: Playwright retains traces/screenshots only on failure. Uploading
|
|
# them as an artifact makes post-mortem debugging on CI possible without
|
|
# re-running locally.
|
|
- name: Upload Playwright traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-${{ matrix.shard_name }}
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
ssh-docker-watcher-isolation:
|
|
name: ssh docker watcher isolation
|
|
needs: build
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Install native build and headless UI tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential openssh-client python3 xvfb
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version-file: package.json
|
|
|
|
- name: Setup pnpm
|
|
uses: pnpm/action-setup@v6
|
|
with:
|
|
run_install: false
|
|
|
|
# Why: same Linux-only node-gyp pin as build/e2e jobs so the workaround
|
|
# stays consistent across workflows even while this job is ubuntu-latest.
|
|
- name: Use external node-gyp to avoid pnpm's bundled copy (Linux only)
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
npm install -g node-gyp@11.5.0
|
|
echo "npm_config_node_gyp=$(npm root -g)/node-gyp/bin/node-gyp.js" >> "$GITHUB_ENV"
|
|
|
|
- name: Install dependencies
|
|
run: pnpm install --frozen-lockfile
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: this is the release-path proof that the deployed Linux relay keeps
|
|
# its PTY and explorer live across a real watcher SIGSEGV.
|
|
- name: Run Docker SSH watcher isolation E2E
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-watcher-isolation
|
|
|
|
- name: Upload watcher isolation traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-ssh-docker-watcher-isolation
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|