mirror of
https://github.com/stablyai/orca.git
synced 2026-09-23 00:02:29 +00:00
The Docker-SSH e2e lane only ran when a PR's changed specs happened to include
`ssh-startup-exec-readiness.spec.ts` or `paired-startup-exec-readiness.spec.ts`.
Editing SSH source itself did not trigger it, and pruning either spec from a
route's list would have silently retired the whole lane. Meanwhile the sharded
lanes set no `ORCA_E2E_SSH_DOCKER`, so every Docker-gated spec skipped itself
while the shard still reported green -- the exact silent-skip shape
`docs/reference/ssh-reconnect-source-recovery.md` blames for four regressions
that reached users.
Separately, the modules that actually own direct-SSH workspace and tab restore
carry no "ssh" in their names, so the `ssh-terminal-source` route never reached
them. Measured on the real script before this change:
printf '%s\n' src/renderer/src/hooks/remote-workspace-session-merge.ts \
src/main/ipc/remote-workspace-snapshot-normalization.ts \
src/renderer/src/lib/worktree-initial-terminal-seeding.ts \
src/shared/remote-workspace-session-projection.ts \
| node config/scripts/pr-e2e-source-routing.mjs
=> []
Three changes, all pinned by the executable gate contract:
- `hasSshSourceChange` derives an `ssh_source_changed` signal from the SSH
routes themselves, plumbed pr.yml -> e2e.yml, so the lane triggers on source
rather than on a spec name surviving in a list. One list, so the two cannot
drift.
- A sibling `ssh-workspace-session-restore` route names the restore seams
(`remote-workspace-*`, `worktree-initial-terminal-seeding`,
`worktree-default-terminal-tabs`, `initial-terminal`) and routes them to the
two restore specs -- a sibling rather than more paths on `ssh-terminal-source`
so a tab-tombstone edit does not run the whole SSH terminal list.
- A new `test:e2e:ssh-docker` runner claims the remaining Docker-gated specs on
the one VM that sets the flag, and the contract now fails by name when any
Docker-gated spec is claimed by no runner. `ssh-docker-relay-perf` and
`ssh-codex-display-artifacts-repro` are recorded exemptions (wall-clock
budgets; needs a real remote codex binary) and the contract asserts each
exemption still corresponds to a real gated spec, so a stale one cannot
quietly excuse a gap. Lane timeout raised 35 -> 60 minutes for the added
serial specs.
The lane's first act was to surface four latent bugs in a spec that had been
silently skipping. `ssh-docker-bulk-open-freeze-repro.spec.ts` is four call sites
out of date against `tests/e2e/helpers/terminal.ts`: `startDockerSshRelayTarget()`
is called with no argument though the helper dereferences `testInfo.workerIndex`
(a 100% failure, not a flake), `execInTerminal` gained a `ptyId` parameter, and
`splitActiveTerminalPane` gained a direction. It was invisible because it ran
nowhere and `typecheck:e2e` is red on main with 240 pre-existing errors, so four
more could not be seen.
The `testInfo` bug is fixed here -- correct on its own, and it removes one real
error from `typecheck:e2e` (240 -> 239). The other three are not, because they
are not argument plumbing: repairing them requires choosing which ptyId to
capture and which split direction to use, and both change what the repro
measures.
The spec is therefore added to the exemption list rather than repaired, for two
independent reasons recorded in the runner: it is a perf oracle, not a
correctness one (`SOFT_FREEZE_LAG_MS=2500` / `HARD_FREEZE_LAG_MS=5000` measured
under a deliberate 5-pane flood on a 420s budget -- the same rule already applied
to `ssh-docker-relay-perf.spec.ts`), and it is known-rotted. Repair is tracked in
stablyai/orca#16764. Applying an existing written rule to a sibling that plainly
meets it is consistency; inventing a new exemption to dodge a red would not be.
Three hardening fixes to the contract itself:
- Runner text is comment-stripped before the claimed-by-a-lane scan. A substring
scan over raw text lets a spec merely *discussed* in a runner comment count as
claimed -- the silent skip this assertion exists to catch, re-entering through
the documentation. Not live today only because the existing comments write the
spec names without their `tests/e2e/` prefix.
- An exempt spec must not be invoked by any runner. `unreachableSpecs`
short-circuits the unclaimed check, so a spec could be documented as exempt
while a runner still ran it -- an exemption that reads as coverage removal but
changes nothing, leaving the lane red for a reason the file says it excluded.
This is not hypothetical: adding the bulk-open exemption without removing it
from the runner's spec list produced exactly that state, and this assertion is
what caught it.
- The Docker-gate detector is now `/ORCA_E2E_SSH_DOCKER\s*[!=]==\s*['"]1['"]/`
rather than one fixed string, so a double-quoted or `!==` spelling can no
longer escape the contract.
`ssh-restart-tab-accumulation.spec.ts` is a new three-cycle restart fence
asserting tab-id set identity, not just the active pane's reclaimed ptyId as
`ssh-cold-activation-restore.spec.ts:241` did. It passes today; it was validated
by a negative control that injected one tab after cycle 1 and correctly failed.
290 lines
11 KiB
YAML
290 lines
11 KiB
YAML
name: E2E
|
|
|
|
run-name: E2E ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: checkout + artifact upload only; callers can only further restrict.
|
|
permissions:
|
|
contents: read
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the calling workflow's ref)
|
|
required: false
|
|
type: string
|
|
test_files:
|
|
description: JSON array of changed specs; empty runs the full suite
|
|
required: false
|
|
type: string
|
|
ssh_source_changed:
|
|
description: '"true" when the PR touches SSH execution source; gates the Docker-SSH lane'
|
|
required: false
|
|
type: string
|
|
workflow_dispatch:
|
|
inputs:
|
|
ref:
|
|
description: Ref to check out (defaults to the workflow ref)
|
|
required: false
|
|
type: string
|
|
schedule:
|
|
# Why: GitHub cron uses UTC; these slots map to 10am and 3pm
|
|
# America/Phoenix for the default-branch E2E run.
|
|
- cron: '0 17,22 * * *'
|
|
|
|
jobs:
|
|
build:
|
|
name: build e2e app
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: the build's plain-Node daemon smoke load resolves node-pty.
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: node
|
|
|
|
# Why: building here avoids parallel builds inside Playwright globalSetup;
|
|
# paired-browser specs also need the standalone web bundle.
|
|
- name: Build E2E outputs
|
|
env:
|
|
VITE_EXPOSE_STORE: 'true'
|
|
run: |
|
|
npx electron-vite build --mode e2e
|
|
pnpm run build:web-from-renderer
|
|
pnpm run build:relay
|
|
|
|
- name: Upload E2E build output
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
# Why: build-relay.mjs writes each relay's marker as `out/relay/<platform>/.version`,
|
|
# and upload-artifact drops dotfiles by default — consumers then fail SSH specs with
|
|
# "local relay build is missing its version marker".
|
|
include-hidden-files: true
|
|
retention-days: 1
|
|
if-no-files-found: error
|
|
|
|
e2e:
|
|
name: e2e ${{ matrix.shard_name }}
|
|
needs: build
|
|
if: inputs.test_files == ''
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- shard: '1/10'
|
|
shard_name: 1-of-10
|
|
- shard: '2/10'
|
|
shard_name: 2-of-10
|
|
- shard: '3/10'
|
|
shard_name: 3-of-10
|
|
- shard: '4/10'
|
|
shard_name: 4-of-10
|
|
- shard: '5/10'
|
|
shard_name: 5-of-10
|
|
- shard: '6/10'
|
|
shard_name: 6-of-10
|
|
- shard: '7/10'
|
|
shard_name: 7-of-10
|
|
- shard: '8/10'
|
|
shard_name: 8-of-10
|
|
- shard: '9/10'
|
|
shard_name: 9-of-10
|
|
- shard: '10/10'
|
|
shard_name: 10-of-10
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
# Why: pnpm install used to rebuild native modules here; the composite
|
|
# action restores them from cache and only compiles on a miss. The
|
|
# toolchain is still required for that miss path, and for paired Quick
|
|
# Open coverage which exercises the resource-bounded host search.
|
|
- name: Install native build tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential fonts-noto-cjk python3 ripgrep zsh
|
|
|
|
# Why: Electron on Linux needs an X display even when the app
|
|
# suppresses mainWindow.show() via ORCA_E2E_HEADLESS. xvfb provides a
|
|
# virtual framebuffer so Chromium can initialize without a real display.
|
|
- name: Install xvfb
|
|
run: sudo apt-get install -y xvfb
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: electron
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: the Electron suite is wall-clock constrained on OSS runners, but
|
|
# multiple Electron apps on one Xvfb VM contend on git/Chromium resources.
|
|
# Sharding keeps each VM at one Playwright worker while splitting the
|
|
# headless suite across separate runners.
|
|
# SKIP_BUILD makes Playwright globalSetup reuse the single build job's
|
|
# artifact instead of starting five concurrent electron-vite builds.
|
|
# ORCA_E2E_FORWARD_APP_LOGS keeps startup failures visible when Electron
|
|
# launches but never creates a BrowserWindow.
|
|
- name: Run E2E tests (${{ matrix.shard_name }})
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay" pnpm run test:e2e --shard=${{ matrix.shard }}
|
|
|
|
# Why: Playwright retains traces/screenshots only on failure. Uploading
|
|
# them as an artifact makes post-mortem debugging on CI possible without
|
|
# re-running locally.
|
|
- name: Upload Playwright traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-${{ matrix.shard_name }}
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
changed-e2e:
|
|
name: changed e2e specs
|
|
needs: build
|
|
if: inputs.test_files != ''
|
|
runs-on: ubuntu-latest
|
|
# Why 45: pr.yml now maps SSH source edits onto Docker-backed specs, so this lane can
|
|
# pay a container image build plus ~22 serial SSH tests on top of the changed specs.
|
|
timeout-minutes: 45
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Install native build and headless UI tools
|
|
# Why ripgrep: Quick Open's bounded host-side search requires rg instead of an
|
|
# unbounded inventory fallback; the paired fixture exercises that real boundary.
|
|
# Why openssh-client: the Docker-SSH fixture shells out to ssh/ssh-keygen, and this
|
|
# lane now receives those specs from pr.yml's SSH source mapping.
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential fonts-noto-cjk openssh-client python3 ripgrep xvfb zsh
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: electron
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
- name: Run changed E2E specs
|
|
env:
|
|
TEST_FILES_JSON: ${{ inputs.test_files }}
|
|
run: |
|
|
mapfile -t TEST_FILES < <(jq -r '.[] | select(
|
|
. != "tests/e2e/ssh-startup-exec-readiness.spec.ts" and
|
|
. != "tests/e2e/paired-startup-exec-readiness.spec.ts"
|
|
)' <<<"$TEST_FILES_JSON")
|
|
if [ "${#TEST_FILES[@]}" -eq 0 ]; then
|
|
echo "Changed startup-readiness specs are owned by the dedicated live lane."
|
|
exit 0
|
|
fi
|
|
E2E_ENV=(SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 ORCA_E2E_WEB_CLIENT=1 ORCA_RELAY_PATH="$GITHUB_WORKSPACE/out/relay")
|
|
# Second clause: a spec that reads ORCA_E2E_SSH_DOCKER test.skip()s itself without it, so
|
|
# naming only one trigger silently skipped every other Docker-SSH spec in this lane.
|
|
# The first clause stays because that spec needs Docker without referencing the variable.
|
|
if printf '%s\n' "${TEST_FILES[@]}" | grep -qx 'tests/e2e/ephemeral-vm-provisioned-root.spec.ts' \
|
|
|| grep -l 'ORCA_E2E_SSH_DOCKER' "${TEST_FILES[@]}" >/dev/null 2>&1; then
|
|
E2E_ENV+=(ORCA_E2E_SSH_DOCKER=1)
|
|
fi
|
|
E2E_PROJECT_ARGS=()
|
|
if grep -l '@headful' "${TEST_FILES[@]}" >/dev/null; then
|
|
E2E_PROJECT_ARGS+=(--project=electron-headful)
|
|
fi
|
|
xvfb-run --auto-servernum env "${E2E_ENV[@]}" \
|
|
pnpm run test:e2e "${TEST_FILES[@]}" --workers=1 "${E2E_PROJECT_ARGS[@]}"
|
|
|
|
- name: Upload Playwright traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-changed
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|
|
|
|
ssh-docker-watcher-isolation:
|
|
name: ssh docker watcher isolation
|
|
needs: build
|
|
# Why ssh_source_changed first: this lane used to trigger on SSH source only as a side
|
|
# effect of one route listing a startup-readiness spec — pruning that spec would have
|
|
# silently retired the whole lane. The signal is now derived from the SSH routes directly.
|
|
# The two spec clauses stay for their honest purpose: changed-e2e hands these specs to this
|
|
# lane, so editing one must still run it here.
|
|
if: >-
|
|
inputs.test_files == '' ||
|
|
inputs.ssh_source_changed == 'true' ||
|
|
contains(inputs.test_files, 'tests/e2e/ssh-startup-exec-readiness.spec.ts') ||
|
|
contains(inputs.test_files, 'tests/e2e/paired-startup-exec-readiness.spec.ts')
|
|
runs-on: ubuntu-latest
|
|
# Why 60: this lane now also runs the remaining Docker-SSH specs serially. They average
|
|
# ~18s but several budget 4-10 minutes per test, so a slow run lands far above the old 35
|
|
# — and the sharded lanes already show that a lane which times out is a lane nobody trusts.
|
|
timeout-minutes: 60
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Install native build and headless UI tools
|
|
run: sudo apt-get update && sudo apt-get install -y build-essential fonts-noto-cjk openssh-client python3 xvfb zsh
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: electron
|
|
|
|
- name: Download E2E build output
|
|
uses: actions/download-artifact@v8
|
|
with:
|
|
name: e2e-build-out
|
|
path: out/
|
|
|
|
# Why: this is the release-path proof that the deployed Linux relay keeps
|
|
# its PTY and explorer live across a real watcher SIGSEGV.
|
|
- name: Run Docker SSH watcher isolation E2E
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-watcher-isolation
|
|
|
|
# Why always(): this lane gates SSH parking/retention plus startup-exec
|
|
# readiness across live SSH, headed paired, and headless serve topologies.
|
|
- name: Run Docker SSH terminal parking + startup readiness E2E
|
|
if: always()
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker-terminal-parking
|
|
|
|
# Why here rather than the sharded lanes: the shards set no ORCA_E2E_SSH_DOCKER, so every
|
|
# spec below skipped itself while the shard still reported green. Running them on this one
|
|
# VM pays the fixture image build once instead of ten times, and keeps an SSH regression
|
|
# legible as an SSH-named failure.
|
|
- name: Run remaining Docker SSH E2E
|
|
if: always()
|
|
run: xvfb-run --auto-servernum env SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 pnpm run test:e2e:ssh-docker
|
|
|
|
- name: Upload watcher isolation traces
|
|
if: failure()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: playwright-traces-ssh-docker-watcher-isolation
|
|
path: test-results/
|
|
retention-days: 7
|
|
if-no-files-found: ignore
|