mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 16:02:32 +00:00
* ci(pr): run E2E when a PR touches tests/e2e paths Regression specs under tests/e2e never ran on PR CI — only schedule and release called e2e.yml — so a red regression test could merge green. Path-filter and workflow_call the E2E suite when E2E-relevant files change. Use merge-base diffs so base-branch drift does not false-trigger E2E, fail the detector when git diff cannot compute the PR range, and pin least-privilege contents:read on both the detector and reusable E2E workflow. Closes #10518 Co-authored-by: Wooseong Kim <innocarpe@gmail.com> Co-authored-by: Orca <help@stably.ai> * ci(pr): make the E2E path gate actually block, and match the real config path Two fixes to the new path-filtered E2E job. The gate did not gate. pr.yml's `verify` job is the required check, and it enumerates its dependencies explicitly — `e2e` was in neither `needs` nor the result list, so a failing shard left `verify` green. That reproduces the exact hole this job exists to close: a red spec merges green, just with a red box further down the page. Add `e2e` to both. Because the job is path-filtered, `skipped` is the normal result on a PR that touches no E2E files and has to keep passing. That allowance is checked after the strict loop rather than inside it, so it can never leak to the six jobs that are always required. The `playwright.` pattern matched nothing. The config is tests/playwright.config.ts — beside tests/e2e/, not inside it — so no tracked file starts with `playwright.` and editing the runner config would silently skip E2E. Anchor it at `tests/playwright.`. Adds a contract test alongside the existing release-e2e one. Verified it fails when either fix is reverted, and simulated the gate across success/skipped/failure/cancelled plus the skip-must-not-mask-a-real-failure case. * test(ci): close two gaps in the E2E gate contract CodeRabbit was right on both counts — verified by reverting each and watching the contract stay green. The path filter was unasserted, so `e2e` could lose its `if:` and run on every PR — the cost the filter exists to avoid — without failing anything. The strict-loop check hardcoded four of the six required jobs, so dropping GIT_COMPATIBILITY or SHELL_CONTRACTS left them unenforced while the contract passed. Derive the list from verify.needs instead, so a newly added required job that misses the loop fails here rather than silently going unchecked. * ci(pr): land the E2E path gate advisory instead of blocking The E2E suite is currently failing every scheduled run on main — 22 of the last 22 — so making verify depend on it would block any PR touching tests/e2e/**, including the PRs that fix the suite. This PR's own run reproduced that: 3 of 12 shards failed on specs unrelated to it (agent-session resume, Jira linking, plugin containment, terminal artifacts). So the job runs and reports on E2E-path PRs but is left out of verify.needs for now. The detector, the tests/playwright. path fix, and the contract tests are unaffected — those stand on their own and were the substance of the review. Flipping to blocking is a three-line change once the suite is green; the exact wiring, including why the skipped allowance must sit outside the strict loop, is recorded on verify's Require-successful-checks step. The contract test pins the advisory choice so it reads as deliberate rather than as the unwired-gate bug it originally caught, and still fails if the path filter, the strict-loop coverage, or the config path regress. --------- Co-authored-by: Wooseong Kim <innocarpe@gmail.com> Co-authored-by: Orca <help@stably.ai>
357 lines
12 KiB
YAML
357 lines
12 KiB
YAML
name: PR Checks
|
|
|
|
on:
|
|
pull_request:
|
|
types:
|
|
- opened
|
|
- synchronize
|
|
- reopened
|
|
- ready_for_review
|
|
|
|
concurrency:
|
|
group: pr-checks-${{ github.event.pull_request.number }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
static_analysis:
|
|
name: static analysis
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
persist-credentials: false
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
|
|
- name: Lint
|
|
run: pnpm exec oxlint --format github
|
|
|
|
- name: Enforce focused code-quality plugins
|
|
run: pnpm run audit:code-quality:native
|
|
|
|
- name: Enforce type-aware code-quality baseline
|
|
run: pnpm run audit:code-quality:type-aware
|
|
|
|
- name: Enforce changed-code quality
|
|
run: pnpm run check:code-quality:changed -- "${{ github.event.pull_request.base.sha }}"
|
|
|
|
- name: Enforce React Doctor on changed lines
|
|
run: pnpm run check:react-doctor:changed -- "${{ github.event.pull_request.base.sha }}"
|
|
|
|
- name: Check Zustand selector fan-out budget
|
|
run: pnpm run check:zustand-selector-fanout
|
|
|
|
- name: Check reliability gate manifest
|
|
run: pnpm run check:reliability-gates
|
|
|
|
- name: Enforce max-lines ratchet
|
|
run: pnpm run check:max-lines-ratchet
|
|
|
|
- name: Verify bundled skill guides
|
|
run: pnpm run verify:bundled-skill-guides
|
|
|
|
- name: Verify skill freshness manifest
|
|
run: pnpm run verify:skill-bundle-manifest
|
|
|
|
- name: Verify localization catalog
|
|
run: pnpm run verify:localization-catalog
|
|
|
|
- name: Verify localization coverage
|
|
run: pnpm run verify:localization-coverage
|
|
|
|
# Why: project-owned type declarations must live in .ts so tsc
|
|
# actually checks them. TypeScript's skipLibCheck: true (inherited
|
|
# from @electron-toolkit/tsconfig) silently widens unresolved names
|
|
# in .d.ts to `any`, which is how #1186 shipped a broken IPC signature
|
|
# past typecheck. See docs/preload-typecheck-hole.md.
|
|
- name: Guard against project-owned .d.ts in preload/shared
|
|
run: |
|
|
matches=$(find src/preload src/shared -name '*.d.ts' 2>/dev/null || true)
|
|
if [ -n "$matches" ]; then
|
|
echo "::error::Project-owned .d.ts files are not allowed under src/preload or src/shared."
|
|
echo "Move type declarations into a .ts file so skipLibCheck does not hide errors."
|
|
echo "See docs/preload-typecheck-hole.md."
|
|
echo "Found:"
|
|
echo "$matches"
|
|
exit 1
|
|
fi
|
|
|
|
- name: Check feature wall asset budget
|
|
run: pnpm check:feature-wall-assets
|
|
|
|
- name: Verify macOS entitlements
|
|
run: pnpm verify:macos-entitlements
|
|
|
|
typecheck:
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
|
|
- run: pnpm typecheck
|
|
|
|
git_compatibility:
|
|
name: Git compatibility
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
|
|
- name: Verify Git binary compatibility matrix
|
|
run: |
|
|
pids=()
|
|
(
|
|
archive="$RUNNER_TEMP/git-2.25.5.tar.gz"
|
|
source="$RUNNER_TEMP/git-2.25.5"
|
|
curl -fsSL https://www.kernel.org/pub/software/scm/git/git-2.25.5.tar.gz -o "$archive"
|
|
echo "41662c52fc16fec4963bfc41075e71f8ead6b5e386797eb6f9a1111ff95a8ddf $archive" \
|
|
| sha256sum --check
|
|
mkdir -p "$source"
|
|
tar -xzf "$archive" -C "$source" --strip-components=1
|
|
make -C "$source" -j"$(nproc)" \
|
|
NO_GETTEXT=YesPlease NO_TCLTK=YesPlease NO_PYTHON=YesPlease git
|
|
ORCA_GIT_COMPAT_BINARY="$source/git" ORCA_GIT_COMPAT_VERSION="2.25.5" \
|
|
pnpm exec vitest run --config config/vitest.config.ts \
|
|
src/shared/git-binary-compatibility.test.ts
|
|
) &
|
|
pids+=("$!")
|
|
|
|
for spec in \
|
|
"alpine/git:edge-2.38.1|2.38.1" \
|
|
"alpine/git:v2.49.1|2.49.1"; do
|
|
(
|
|
image="${spec%%|*}"
|
|
version="${spec#*|}"
|
|
ORCA_GIT_COMPAT_IMAGE="$image" ORCA_GIT_COMPAT_VERSION="$version" \
|
|
pnpm exec vitest run --config config/vitest.config.ts \
|
|
src/shared/git-binary-compatibility.test.ts
|
|
) &
|
|
pids+=("$!")
|
|
done
|
|
|
|
status=0
|
|
for pid in "${pids[@]}"; do
|
|
wait "$pid" || status=1
|
|
done
|
|
exit "$status"
|
|
|
|
shell_contracts:
|
|
name: shell contracts
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Install zsh
|
|
run: sudo apt-get update && sudo apt-get install -y zsh
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: node
|
|
|
|
- name: Test real shell contracts
|
|
run: |
|
|
pnpm exec vitest run --config config/vitest.config.ts \
|
|
src/main/daemon/shell-ready.test.ts \
|
|
src/main/daemon/node-pty-fd-leak.test.ts \
|
|
src/main/providers/local-pty-shell-ready.test.ts \
|
|
src/main/providers/__tests__/shell-ready-framework-example.test.ts \
|
|
src/main/pty/omp-shell-wrapper.node-pty.test.ts \
|
|
src/shared/posix-command-path-lookup.test.ts
|
|
|
|
test:
|
|
name: tests ${{ matrix.shard }}/${{ strategy.job-total }}
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: node
|
|
|
|
- name: Install Electron package binary for tests
|
|
run: node config/scripts/install-electron-package-binary.mjs
|
|
|
|
- name: Test shard
|
|
run: |
|
|
pnpm exec vitest run --config config/vitest.config.ts \
|
|
--exclude=src/main/daemon/shell-ready.test.ts \
|
|
--exclude=src/main/daemon/node-pty-fd-leak.test.ts \
|
|
--exclude=src/main/providers/local-pty-shell-ready.test.ts \
|
|
--exclude=src/main/providers/__tests__/shell-ready-framework-example.test.ts \
|
|
--exclude=src/main/pty/omp-shell-wrapper.node-pty.test.ts \
|
|
--exclude=src/shared/posix-command-path-lookup.test.ts \
|
|
--shard=${{ matrix.shard }}/${{ strategy.job-total }}
|
|
|
|
package:
|
|
name: package
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Cache electron-builder downloads
|
|
uses: actions/cache@v5
|
|
with:
|
|
path: |
|
|
~/.cache/electron
|
|
~/.cache/electron-builder
|
|
key: electron-builder-linux-${{ hashFiles('pnpm-lock.yaml') }}
|
|
restore-keys: |
|
|
electron-builder-linux-
|
|
|
|
- uses: ./.github/actions/install-node-dependencies
|
|
with:
|
|
native-runtime: electron
|
|
|
|
- name: Build package inputs
|
|
run: |
|
|
status=0
|
|
pnpm run build:cli || status=1
|
|
|
|
scripts=(build:relay build:electron-vite:parallel)
|
|
pids=()
|
|
for script in "${scripts[@]}"; do
|
|
pnpm run "$script" &
|
|
pids+=("$!")
|
|
done
|
|
|
|
for pid in "${pids[@]}"; do
|
|
wait "$pid" || status=1
|
|
done
|
|
exit "$status"
|
|
|
|
- name: Project web client from renderer build
|
|
run: pnpm run build:web-from-renderer
|
|
|
|
- name: Build native components
|
|
run: pnpm run build:native
|
|
|
|
- name: Package unpacked app
|
|
env:
|
|
ORCA_REUSE_PREPARED_NATIVE_RUNTIME: '1'
|
|
run: pnpm exec electron-builder --config config/electron-builder.config.cjs --dir
|
|
|
|
- name: Smoke packaged CLI
|
|
run: node config/scripts/smoke-packaged-cli.mjs --app-dir=dist/linux-unpacked
|
|
|
|
# Why: regression specs under tests/e2e/** used to merge green without ever
|
|
# running — e2e.yml only fired on schedule/release (#10518). Path-filter so
|
|
# ordinary PRs stay light; any E2E suite change still gets a full shard run.
|
|
e2e-paths:
|
|
name: detect e2e path changes
|
|
runs-on: ubuntu-latest
|
|
if: github.event.pull_request.draft != true
|
|
# Why: detector only needs to read the checkout; do not inherit repo defaults.
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
should_run: ${{ steps.filter.outputs.should_run }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v6
|
|
with:
|
|
fetch-depth: 0
|
|
persist-credentials: false
|
|
|
|
- name: Filter E2E-relevant paths
|
|
id: filter
|
|
run: |
|
|
set -euo pipefail
|
|
BASE="${{ github.event.pull_request.base.sha }}"
|
|
HEAD="${{ github.event.pull_request.head.sha }}"
|
|
# Why: capture first so a failed git diff does not look like "no matches"
|
|
# (pipeline status in `if` is not aborted by set -e). Merge-base limits the
|
|
# list to files this PR introduced, not base-branch drift.
|
|
CHANGED="$(git diff --name-only --merge-base "$BASE" "$HEAD")"
|
|
# Why: tests/playwright.config.ts sits beside tests/e2e/, not inside it, so
|
|
# it needs its own pattern — a bare `playwright.` prefix matches no tracked
|
|
# file and would silently skip E2E when the runner config changes.
|
|
if printf '%s\n' "$CHANGED" | grep -E '^(tests/e2e/|tests/playwright\.|\.github/workflows/e2e\.yml$)' >/dev/null; then
|
|
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
|
echo "E2E path changes detected"
|
|
else
|
|
echo "should_run=false" >> "$GITHUB_OUTPUT"
|
|
echo "No E2E path changes"
|
|
fi
|
|
|
|
e2e:
|
|
name: e2e
|
|
needs: e2e-paths
|
|
if: needs.e2e-paths.outputs.should_run == 'true'
|
|
# Why: reusable e2e.yml only checkouts, builds, and uploads artifacts.
|
|
permissions:
|
|
contents: read
|
|
uses: ./.github/workflows/e2e.yml
|
|
|
|
verify:
|
|
if: always()
|
|
needs:
|
|
- static_analysis
|
|
- typecheck
|
|
- git_compatibility
|
|
- shell_contracts
|
|
- test
|
|
- package
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
# Why: e2e is deliberately absent from needs. The suite is currently red on
|
|
# main (every scheduled run), so gating merges on it would block any PR that
|
|
# touches tests/e2e/** — including the ones fixing the suite. Until it is
|
|
# green the job runs and reports for E2E-path PRs without blocking. To flip
|
|
# it on: add `e2e` to needs, add E2E to the env below, and require
|
|
# `"$E2E" = success || skipped` after the loop — skipped is the normal
|
|
# result for a path-filtered job and must keep passing, so it has to be
|
|
# checked outside the loop or it would excuse the jobs above.
|
|
- name: Require successful checks
|
|
env:
|
|
STATIC_ANALYSIS: ${{ needs.static_analysis.result }}
|
|
TYPECHECK: ${{ needs.typecheck.result }}
|
|
GIT_COMPATIBILITY: ${{ needs.git_compatibility.result }}
|
|
SHELL_CONTRACTS: ${{ needs.shell_contracts.result }}
|
|
TEST: ${{ needs.test.result }}
|
|
PACKAGE: ${{ needs.package.result }}
|
|
run: |
|
|
for result in \
|
|
"$STATIC_ANALYSIS" \
|
|
"$TYPECHECK" \
|
|
"$GIT_COMPATIBILITY" \
|
|
"$SHELL_CONTRACTS" \
|
|
"$TEST" \
|
|
"$PACKAGE"; do
|
|
if [ "$result" != "success" ]; then
|
|
exit 1
|
|
fi
|
|
done
|