mirror of
https://github.com/daijro/camoufox.git
synced 2026-10-03 08:00:19 +00:00
f36390a19e
The default GeoIP source was MaxMind GeoLite2 via sapics/ip-location-db, whose URLs kept serving the 2026-06-17 build after that project moved to GitHub Releases (found in #815). GeoIP AIO (daijro/geoip-all-in-one) resolves timezones more accurately on real proxy IPs and is rebuilt weekly. - repos.yml: AIO is the default; GeoLite2 is `deprecated: true`, with the Releases URLs from #815 so it still works when picked by name. - A cache holding a deprecated source it was not explicitly given (`camoufox set --geoip` or the GUI) moves to the default and drops the old database. An explicit choice is kept, with a FutureWarning. - needs_update() reads the database's build date instead of the file age: refresh once the build is over 8 days old, re-checking at most daily, and warn when a fresh download is over 30 days old (a frozen source). - get_geolocation(geoip_db=...) now reads that source's own database rather than the active one's, and no longer makes it the active one. - tests/test_geoip_sources.py (from #815) downloads every non-deprecated source and fails when its build is stale; tests.yml installs the geoip extra so it runs, and so gates every release. - TypeScript twin updated to match; goldens answer in both layouts. Co-authored-by: lp177 <57773165+lp177@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1260 lines
57 KiB
YAML
1260 lines
57 KiB
YAML
name: Tests
|
|
|
|
# The repository's test pipeline. Runs on every pull request, on demand, and --
|
|
# via workflow_call -- from release.yml on every push to main (a release is only
|
|
# built from a commit this pipeline passed) and from any workflow that needs to
|
|
# test a specific browser version, so a contributor's pull request, a release and
|
|
# an automated Firefox bump are judged by exactly the same checks.
|
|
#
|
|
# The browser version comes from upstream.sh unless a caller passes one in,
|
|
# which is what lets one pipeline test both a pull request and a version bump.
|
|
# ci/versions.py then picks the newest released Playwright suite that is not
|
|
# ahead of that browser -- Playwright trails Firefox and skips generations, so
|
|
# the suite's own Firefox pin is usually a release or two behind the browser
|
|
# under test, and that is expected rather than a mismatch. ci/run_playwright.py
|
|
# fetches it fresh, applies ci/skiplist.yml, overlays tests/camoufox/ and runs
|
|
# it with world isolation ON -- the configuration Camoufox ships -- re-running
|
|
# only what fails with isolation off, and counting those as main-world
|
|
# fallbacks. A test that needs the fallback still passes; the size of that set
|
|
# is reported, because it is the isolated-world conformance gap.
|
|
#
|
|
# The stealth check reports a letter grade and a count. Its per-vector detail
|
|
# never leaves ci/run_sundial.py, because this repository is public. It depends
|
|
# on a service outside this repository, so an outage there records a SKIP with
|
|
# its reason rather than blocking every merge (see `--allow-skip` below); a bad
|
|
# credential or a bad score still fails.
|
|
#
|
|
# Ordering: cheapest first, and every tier gates the next, so a pull request that
|
|
# fails a two-second lint never reaches a seventy-minute build.
|
|
#
|
|
# 0 static lint, self-tests, settled decisions seconds
|
|
# 1 unit pythonlib ~1 min
|
|
# 2 browser BUILD if patches/additions changed,
|
|
# otherwise FETCH the released binary ~70 min / ~1 min
|
|
# 3a smoke patch guards, skiplist audit, build-tester ~15 min
|
|
# 3b full Playwright (6 shards), leaks, stealth ~40 min
|
|
# 4 gate the single required status check
|
|
#
|
|
# A driver-only pull request never builds: there is nothing new to compile, so it
|
|
# is tested against the published browser its users actually run. Tier 3b waits
|
|
# on 3a so a browser that fails its guards does not also burn six Playwright
|
|
# shards proving the same thing.
|
|
|
|
on:
|
|
pull_request:
|
|
# Pushes to main are tested by release.yml, which calls this workflow.
|
|
schedule:
|
|
# Keeps the ccache alive. GitHub evicts a cache after 7 days unused, and a
|
|
# cold Firefox build is over an hour; twice a week keeps pull-request builds
|
|
# restoring a warm one from main.
|
|
- cron: "0 5 * * 1,4"
|
|
workflow_dispatch:
|
|
inputs:
|
|
browser_version:
|
|
description: "Firefox version to test. Must match upstream.sh -- the build follows that, not this."
|
|
required: false
|
|
type: string
|
|
playwright_tag:
|
|
description: "Pin the Playwright suite (default: resolved from the browser)"
|
|
required: false
|
|
type: string
|
|
shards:
|
|
description: "How many runners to split the upstream suite across"
|
|
required: false
|
|
default: "6"
|
|
type: string
|
|
workflow_call:
|
|
inputs:
|
|
browser_version:
|
|
required: false
|
|
type: string
|
|
playwright_tag:
|
|
required: false
|
|
type: string
|
|
shards:
|
|
required: false
|
|
default: "6"
|
|
type: string
|
|
ref:
|
|
description: "Commit to test; defaults to the calling workflow's ref"
|
|
required: false
|
|
type: string
|
|
secrets:
|
|
SUNDIAL_USERNAME:
|
|
required: false
|
|
SUNDIAL_AUTOMATION_KEY:
|
|
required: false
|
|
outputs:
|
|
verdict:
|
|
description: "pass or fail"
|
|
value: ${{ jobs.summary.outputs.verdict }}
|
|
browser_version:
|
|
value: ${{ jobs.resolve.outputs.browser_version }}
|
|
playwright_tag:
|
|
value: ${{ jobs.resolve.outputs.playwright_tag }}
|
|
|
|
permissions: {}
|
|
|
|
concurrency:
|
|
group: tests-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
env:
|
|
PYTHON_VERSION: "3.12"
|
|
CI_WORK_DIR: ${{ github.workspace }}/.ci-work
|
|
CI_RESULTS_DIR: ${{ github.workspace }}/.ci-work/results
|
|
|
|
jobs:
|
|
# ---------------------------------------------------------------------------
|
|
resolve:
|
|
name: Resolve versions
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
outputs:
|
|
browser_version: ${{ steps.versions.outputs.browser_version }}
|
|
browser_release: ${{ steps.versions.outputs.browser_release }}
|
|
playwright_tag: ${{ steps.versions.outputs.playwright_tag }}
|
|
playwright_firefox: ${{ steps.versions.outputs.playwright_firefox }}
|
|
version_note: ${{ steps.versions.outputs.note }}
|
|
browser_changed: ${{ steps.scope.outputs.browser_changed }}
|
|
browser_tag: ${{ steps.scope.outputs.browser_tag }}
|
|
has_sundial: ${{ steps.sundial.outputs.has_sundial }}
|
|
shard_matrix: ${{ steps.shards.outputs.matrix }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
fetch-depth: 0
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: pip install -r ci/requirements.txt
|
|
|
|
- id: versions
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: |
|
|
# --check-upstream: only suite SELECTION follows browser_version.
|
|
# The build reads upstream.sh, and the fetch path downloads the release
|
|
# built from this tree's sources, so a browser_version the branch does
|
|
# not pin would test the OLD browser against the NEW suite --
|
|
# silently. A real Firefox bump edits upstream.sh, and then
|
|
# resolution reads it by default and the two cannot disagree.
|
|
python3 -m ci.versions --check-upstream \
|
|
${{ inputs.browser_version && format('--browser-version {0}', inputs.browser_version) || '' }} \
|
|
${{ inputs.playwright_tag && format('--playwright-tag {0}', inputs.playwright_tag) || '' }}
|
|
|
|
- name: Does this change the browser?
|
|
id: scope
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
# Only a change that can alter the binary justifies compiling one. A
|
|
# pull request that touches pythonlib/ or ci/ is a driver change: it
|
|
# still gets the full browser suite, but against the published build its
|
|
# users are running, which takes a minute instead of seventy -- as long
|
|
# as that build matches this tree's browser sources (see below).
|
|
run: |
|
|
if [ "${{ github.event_name }}" != "pull_request" ]; then
|
|
echo "browser_changed=true" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::Not a pull request -- building, which also refreshes the shared ccache."
|
|
exit 0
|
|
fi
|
|
sources='^(patches/|additions/|settings/|assets/|upstream\.sh|Makefile|scripts/)'
|
|
base="${{ github.event.pull_request.base.sha }}"
|
|
changed=$(git diff --name-only "$base"...HEAD || echo "")
|
|
echo "changed files:"; echo "$changed" | sed 's/^/ /'
|
|
if echo "$changed" | grep -qE "$sources"; then
|
|
echo "browser_changed=true" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::Browser sources changed -- rebuilding from source."
|
|
exit 0
|
|
fi
|
|
# The pull request leaves the browser alone -- but a published release
|
|
# is only the right browser to test it on if it was built from the
|
|
# SAME browser sources as this tree. The patch guards and suites come
|
|
# from this checkout, so when the base branch has moved past every
|
|
# release (a merged browser change not built yet), testing a release
|
|
# pairs new guards with an old browser, and every guard for the
|
|
# unbuilt change fails on a pull request that never touched it. So
|
|
# test the release built from exactly these sources -- the one this
|
|
# tree's library would be paired with (ci/release.py) -- or build.
|
|
if python3 -m ci.release paired; then
|
|
echo "browser_changed=false" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::A published release was built from these browser sources -- testing against it."
|
|
else
|
|
echo "browser_changed=true" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::No published release was built from these browser sources -- building (a cache hit when the base branch already built it)."
|
|
fi
|
|
|
|
- name: May the stealth check run?
|
|
id: sundial
|
|
# Two conditions, and the config one is checked FIRST and without the
|
|
# secret in scope. While ci/sundial.yml says `enabled: false` the job is
|
|
# not scheduled at all, so SUNDIAL_AUTOMATION_KEY never enters a runner
|
|
# environment and no request is made. That ordering is what made the
|
|
# kill switch real while the live sundial still predated score mode and
|
|
# would have posted the whole report back; it stays that way round so
|
|
# the switch keeps working the next time it is needed.
|
|
#
|
|
# Secrets are absent for pull requests from forks, so the credential
|
|
# check is resolved here once rather than from a job-level `if`, which
|
|
# the secrets context is not available in.
|
|
env:
|
|
KEY: ${{ secrets.SUNDIAL_AUTOMATION_KEY }}
|
|
run: |
|
|
if [ "$(python3 -m ci.run_sundial status)" != "true" ]; then
|
|
echo "has_sundial=false" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::Stealth check not run: disabled in ci/sundial.yml. See the comment there."
|
|
exit 0
|
|
fi
|
|
# Only the password gates the job. The username names an account, not
|
|
# a secret, so ci/run_sundial.py defaults it (to `guest`, the least
|
|
# privileged role the deployment has -- sundial refuses it the
|
|
# private-vector bundle) instead of demanding a second secret.
|
|
if [ -n "$KEY" ]; then
|
|
echo "has_sundial=true" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "has_sundial=false" >> "$GITHUB_OUTPUT"
|
|
echo "::notice::Stealth check skipped: no sundial credential on this run (normal for a fork PR)."
|
|
fi
|
|
|
|
- name: Build the shard matrix
|
|
id: shards
|
|
run: |
|
|
python3 -c "
|
|
import json, os
|
|
n = max(1, int('${{ inputs.shards || '6' }}'))
|
|
print('matrix=' + json.dumps([f'{i}/{n}' for i in range(1, n + 1)]))
|
|
" >> "$GITHUB_OUTPUT"
|
|
cat "$GITHUB_OUTPUT"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
static:
|
|
name: Static checks
|
|
needs: resolve
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
# -e pythonlib because the settled-decisions tests import camoufox.* to
|
|
# assert against it. It is a pure-Python install; no browser involved.
|
|
- run: |
|
|
pip install -r ci/requirements.txt pytest -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
|
|
- name: Synthesized input goes through one chokepoint
|
|
run: python3 scripts/check-input-dispatch.py
|
|
|
|
- name: CI pipeline self-tests
|
|
# These assert that the pipeline reports honestly: skips carry reasons,
|
|
# shards partition exactly, and nothing identifying a sundial vector
|
|
# survives redaction.
|
|
run: python3 -m pytest ci/tests -q
|
|
|
|
- name: Settled decisions are still in force
|
|
# No browser needed: these read the source. They fail a pull request in
|
|
# seconds rather than after a 40-minute build, which matters because the
|
|
# thing they catch is usually a well-meaning change that looks obviously
|
|
# correct until you read the closed PR that rejected it.
|
|
run: python3 -m ci.run_native --subset rules
|
|
|
|
- name: Skiplist is valid
|
|
run: |
|
|
python3 -c "
|
|
from ci.summarize import validate_skiplist
|
|
from ci.pw_camoufox_plugin import load_skiplist
|
|
problems = validate_skiplist()
|
|
if problems:
|
|
raise SystemExit('\n'.join(problems))
|
|
print(f'skiplist OK: {len(load_skiplist())} entries, every one with a reason')
|
|
"
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-static
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
pythonlib:
|
|
name: pythonlib
|
|
# Tier 1. Waits on the static checks so an obvious mistake costs seconds.
|
|
needs: [resolve, static]
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
# The geoip extra: tests/test_geoip_sources.py downloads every GeoIP
|
|
# source and fails when one stops publishing, which gates releases.
|
|
pip install -r ci/requirements.txt pytest -e 'pythonlib[geoip]'
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
- name: Run
|
|
# pythonlib resolves the published release through the GitHub API
|
|
# (pkgman.py honours GITHUB_TOKEN). Unauthenticated, a runner shares the
|
|
# 60-requests-an-hour anonymous quota for its whole IP range and the job
|
|
# fails on `403 rate limit exceeded` having tested nothing.
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: python3 -m ci.run_pythonlib
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-pythonlib
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
typescript:
|
|
name: typescript
|
|
# Tier 1, beside pythonlib. The npm package's type check, lint and vitest
|
|
# suite, including the golden tests that hold it to pythonlib's output byte
|
|
# for byte -- so a pythonlib change that typescript/ does not mirror fails
|
|
# here, in a minute, not after the build.
|
|
needs: [resolve, static]
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
env:
|
|
# The fpgen model the TS port downloads (sha256-pinned by
|
|
# scripts/data/fpgen-model.json). Kept in the workspace so it can be cached.
|
|
CAMOUFOX_FPGEN_DATA: ${{ github.workspace }}/.ci-work/fpgen
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
# Not PYTHON_VERSION: the golden fixtures are regenerated from this
|
|
# interpreter, and pycompat.ts's pySum() reproduces sum() as 3.14
|
|
# computes it (3.12/3.13 round mixed int/float sums differently in
|
|
# the last bit).
|
|
python-version: "3.14"
|
|
- run: |
|
|
# A venv at the repo root: tests/golden-setup.ts records the golden
|
|
# fixtures from its pythonlib before the suite runs.
|
|
python3 -m venv .venv
|
|
.venv/bin/pip install -r ci/requirements.txt -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
.venv/bin/python scripts/pin-fpgen-model.py
|
|
- name: Test prerequisites
|
|
# Everything tests/prereq.ts may ask for. In CI a missing prerequisite
|
|
# FAILS its tests rather than skipping them, so this list is the job's
|
|
# contract. xvfb: the virtual-display lifecycle.
|
|
run: |
|
|
sudo apt-get update -qq
|
|
sudo apt-get install -y --no-install-recommends xvfb
|
|
- uses: pnpm/action-setup@v4
|
|
with:
|
|
package_json_file: typescript/package.json
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: "22"
|
|
cache: pnpm
|
|
cache-dependency-path: typescript/pnpm-lock.yaml
|
|
- uses: actions/cache@v4
|
|
with:
|
|
path: .ci-work/fpgen
|
|
key: fpgen-model-${{ hashFiles('scripts/data/fpgen-model.json') }}
|
|
- name: Run
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: .venv/bin/python -m ci.run_typescript
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-typescript
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
build:
|
|
name: Build (linux x86_64)
|
|
# Tier 2. Nothing gets compiled until the cheap tiers are green -- this is
|
|
# over an hour cold, and a lint failure should never cost that.
|
|
needs: [resolve, static, pythonlib]
|
|
if: needs.resolve.outputs.browser_changed == 'true'
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 330
|
|
permissions:
|
|
contents: read
|
|
env:
|
|
# scripts/patch.py writes the mozconfig from BUILD_TARGET and defaults to
|
|
# macos,arm64 when it is unset -- sensible for a developer on a Mac,
|
|
# wrong here. Without this, configure goes looking for the macOS SDK and
|
|
# dies with "No such file or directory: MacOSX26.5.sdk/SDKSettings.plist"
|
|
# three minutes in, which reads like a missing dependency rather than a
|
|
# cross-compile nobody asked for. multibuild.py sets this itself; `make
|
|
# dir` + `make build` do not.
|
|
BUILD_TARGET: linux,x86_64
|
|
steps:
|
|
# Checkout first, because the cache key below is a hash of the tree.
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
|
|
- name: Hash the inputs that can change compiled output
|
|
id: native
|
|
# Not hashFiles(): this has to EXCLUDE the files jar.mn packages as
|
|
# resources, and hashFiles has no way to say "everything except".
|
|
# ci/browser_inputs.py is stdlib-only on purpose -- it runs here, before
|
|
# the pip install that a cache hit skips.
|
|
run: echo "hash=$(python3 -m ci.browser_inputs --digest)" >> "$GITHUB_OUTPUT"
|
|
|
|
- name: Is a browser with this compiled half already built?
|
|
id: prebuilt
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: camoufox-dist.tar.zst
|
|
# Keyed on the COMPILED inputs only, so a Juggler JavaScript change
|
|
# still hits: the .js files jar.mn packages are laid over the restored
|
|
# browser below instead of relinking libxul to deliver them.
|
|
#
|
|
# The hash is the classification. There is no "did only JS change?"
|
|
# diff, because a diff compares against the pull request's base while
|
|
# the question is whether THIS cached browser has the same native
|
|
# sources -- and if the hash matches, it does, whatever the diff says.
|
|
#
|
|
# `browser_changed` (see resolve) is a different question and stays as
|
|
# it is: it decides build-versus-fetch against the base, and is true
|
|
# for every push to a branch that touched the browser once.
|
|
key: browser-${{ needs.resolve.outputs.browser_version }}-${{ needs.resolve.outputs.browser_release }}-native-${{ steps.native.outputs.hash }}
|
|
# No restore-keys, deliberately. A prefix match would serve a browser
|
|
# whose compiled half was built from different sources, and every
|
|
# suite downstream would report on it looking perfectly healthy.
|
|
|
|
- name: Maximize build space
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
uses: AdityaGarg8/remove-unwanted-software@8831c82abf29b34eb2caac48d5f999ecfc0d8eef # v4.1
|
|
with:
|
|
remove-dotnet: "true"
|
|
remove-android: "true"
|
|
remove-haskell: "true"
|
|
remove-codeql: "true"
|
|
remove-docker-images: "true"
|
|
remove-cached-tools: "true"
|
|
remove-swapfile: "true"
|
|
|
|
- name: Remove unwanted tools
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
run: |
|
|
sudo apt-get remove -y '^aspnetcore-.*' '^dotnet-.*' '^llvm-.*' 'php.*' \
|
|
'^mongodb-.*' '^mysql-.*' > /dev/null 2>&1 || true
|
|
sudo apt-get remove -y azure-cli google-chrome-stable firefox mono-devel \
|
|
libgl1-mesa-dri --fix-missing > /dev/null 2>&1 || true
|
|
sudo apt-get autoremove -y > /dev/null 2>&1 || true
|
|
sudo apt-get clean > /dev/null 2>&1 || true
|
|
df -h /
|
|
|
|
- uses: actions/setup-python@v5
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
|
|
- name: Install build dependencies
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
run: |
|
|
wget -q https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && sudo ./llvm.sh 18
|
|
sudo apt-get install -y lld-18 clang-18
|
|
sudo update-alternatives --install /usr/bin/ld.lld ld.lld /usr/bin/ld.lld-18 100
|
|
sudo apt-get update
|
|
# The mozconfig sets --with-ccache; configure fails hard without it
|
|
# rather than degrading.
|
|
sudo apt-get install -y msitools p7zip-full aria2 ccache libsqlite3-dev
|
|
|
|
- name: Restore ccache
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
uses: actions/cache@v4
|
|
with:
|
|
path: ~/.ccache
|
|
# Keyed on the browser version and the patch set, so a pull request
|
|
# that does not touch patches/ starts from a fully warm cache.
|
|
key: ccache-${{ needs.resolve.outputs.browser_version }}-${{ hashFiles('patches/**', 'additions/**', 'assets/*.mozconfig') }}
|
|
restore-keys: |
|
|
ccache-${{ needs.resolve.outputs.browser_version }}-
|
|
ccache-
|
|
|
|
- name: Create swap
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
run: |
|
|
# The link step is OOM-killed on a standard runner without this, and
|
|
# reports as a bare SIGTERM that looks nothing like out-of-memory.
|
|
sudo fallocate -l 24G /swapfile && sudo chmod 600 /swapfile
|
|
sudo mkswap /swapfile && sudo swapon /swapfile
|
|
free -h
|
|
|
|
- run: pip install -r ci/requirements.txt
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
|
|
- name: Prepare the source tree
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
run: |
|
|
ccache -M 8G && ccache -z
|
|
echo "CCACHE_DIR=$HOME/.ccache" >> "$GITHUB_ENV"
|
|
# ci.run_prepare runs setup-minimal -> dir -> mozbootstrap, retrying
|
|
# only the two that download things and only when the failure reads as
|
|
# transient. `mach bootstrap` pulls toolchains from Taskcluster, and a
|
|
# connection reset there used to fail the pull request outright.
|
|
#
|
|
# setup-minimal, not `make dir` alone: `dir` falls through to `make
|
|
# setup`, which git-inits the source tree and commits -- and a bare
|
|
# runner has no git identity, so that dies with "empty ident name".
|
|
# The local dev repo is only needed by the patch-repair loop, which
|
|
# sets an identity of its own.
|
|
python3 -m ci.run_prepare
|
|
|
|
- name: Build
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
env:
|
|
CARGO_BUILD_JOBS: "1"
|
|
run: python3 -m ci.run_build
|
|
|
|
- run: ccache -s
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
|
|
- name: Package the binary for the test jobs
|
|
if: steps.prebuilt.outputs.cache-hit != 'true'
|
|
run: |
|
|
set -euo pipefail
|
|
src="camoufox-${{ needs.resolve.outputs.browser_version }}-${{ needs.resolve.outputs.browser_release }}/obj-x86_64-pc-linux-gnu/dist/bin"
|
|
test -x "$src/camoufox-bin" || { echo "::error::no camoufox-bin at $src"; exit 1; }
|
|
|
|
# `mach build` produces an *unpackaged* tree. Two things scripts/package.py
|
|
# would add are load-bearing for the test jobs and are missing here:
|
|
#
|
|
# fonts/ + fontconfig/ -- without them every glyph in page content
|
|
# renders as tofu, silently, because the
|
|
# browser chrome still has system fonts.
|
|
# properties.json -- the Python API resolves it next to the
|
|
# binary, so AsyncCamoufox dies with
|
|
# FileNotFoundError without it. That breaks
|
|
# patch-guards, the leak suite and sundial.
|
|
make stage-fonts
|
|
for f in properties.json chrome.css; do
|
|
[ -f "$src/$f" ] || cp -v "settings/$f" "$src/$f"
|
|
done
|
|
|
|
# Fail here, once, rather than in five browser jobs with five
|
|
# different confusing errors.
|
|
# fonts/ holds the bundle's group directories (L, M, W, LM, ... --
|
|
# bundle/fonts/groups.json), not a copy per OS; groups.json is what
|
|
# utils._generate_fontconfig reads to decide which of them an identity
|
|
# may see, so its absence is the failure that matters.
|
|
for required in camoufox-bin properties.json camoufox.cfg fonts/groups.json fonts/LMW fontconfig/linux; do
|
|
[ -e "$src/$required" ] || { echo "::error::artifact is missing $required"; exit 1; }
|
|
done
|
|
|
|
# -h (--dereference) is load-bearing, not tidiness. mach builds dist/bin
|
|
# out of symlinks -- 17 of them here, and properties.json and
|
|
# camoufox.cfg are ABSOLUTE links into the source tree. Archiving the
|
|
# links means they resolve on this runner, where the tree exists, and
|
|
# dangle on every runner that only downloads the artifact. The browser
|
|
# then starts with no config at all, which is where every spoofing pref
|
|
# lives, and the failure surfaces as "properties.json missing" three
|
|
# jobs later.
|
|
#
|
|
# It also defeats the check above: `[ -e ]` follows a symlink, so the
|
|
# file looked present right up until it was unpacked somewhere else.
|
|
tar -C "$(dirname "$src")" -chf - bin | zstd -3 -T0 -o camoufox-dist.tar.zst
|
|
ls -lh camoufox-dist.tar.zst
|
|
|
|
# A restored browser still has to produce the `build` result, or the
|
|
# summary reports a required suite that never ran and the gate goes red --
|
|
# which is the same trap as requiring `build` on a driver-only pull
|
|
# request, arrived at from the other direction. It records WHERE the
|
|
# binary came from, so "this run did not compile anything" is a fact in
|
|
# the evidence rather than an absence in it.
|
|
- name: Lay this branch's resources over the restored browser
|
|
if: steps.prebuilt.outputs.cache-hit == 'true'
|
|
# The compiled half is identical by construction -- that is what the key
|
|
# asserts. What can still differ is the JavaScript, and in the
|
|
# unpackaged dist/bin that CI archives there is no omni.ja to rebuild:
|
|
# Juggler is loose files under chrome/juggler/, so delivering new
|
|
# JavaScript is a copy. ci/browser_inputs.py reads the destinations out
|
|
# of jar.mn rather than assuming a prefix, because two files in the same
|
|
# source directory land at different depths.
|
|
run: |
|
|
set -euo pipefail
|
|
command -v zstd >/dev/null 2>&1 || { sudo apt-get update -qq && sudo apt-get install -y zstd; }
|
|
mkdir -p restored
|
|
zstd -d -c camoufox-dist.tar.zst | tar -C restored -xf -
|
|
python3 -m ci.browser_inputs --overlay restored/bin
|
|
test -f restored/bin/camoufox-bin || { echo "::error::restored artifact has no camoufox-bin"; exit 1; }
|
|
# Via a temporary name, not over the input. `zstd -o` refuses an
|
|
# existing destination ("already exists; stdin is an input - not
|
|
# proceeding") and exits 1, which failed every cache-hit build as soon
|
|
# as one actually hit. `mv` also means a repack that dies partway
|
|
# cannot leave a truncated archive where the restored one was.
|
|
tar -C restored -chf - bin | zstd -3 -T0 -o camoufox-dist.tar.zst.new
|
|
mv -f camoufox-dist.tar.zst.new camoufox-dist.tar.zst
|
|
rm -rf restored
|
|
|
|
- name: Record that the browser was restored, not built
|
|
if: steps.prebuilt.outputs.cache-hit == 'true'
|
|
run: |
|
|
python3 -c "
|
|
from ci import results
|
|
from ci.browser_inputs import jar_entries
|
|
r = results.GateResult(gate='build')
|
|
r.metrics['from_cache'] = True
|
|
r.metrics['resources_overlaid'] = len(jar_entries())
|
|
r.metrics['cache_key'] = '''${{ steps.prebuilt.outputs.cache-primary-key }}'''
|
|
r.note('restored a browser whose compiled half was built from identical '
|
|
'sources, and laid this branch\\'s resources over it; nothing was compiled')
|
|
r.finish(results.PASS).save()
|
|
"
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
with:
|
|
name: camoufox-dist
|
|
path: camoufox-dist.tar.zst
|
|
retention-days: 3
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-build
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
fetch-browser:
|
|
name: Fetch the released browser
|
|
# The other half of tier 2. A driver-only change has nothing new to compile,
|
|
# so it is tested against the build its users are actually running. Uploads
|
|
# the same artifact name as `build`, so every downstream job is identical
|
|
# whichever way the browser arrived.
|
|
needs: [resolve, static, pythonlib]
|
|
if: needs.resolve.outputs.browser_changed == 'false'
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 30
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -r ci/requirements.txt -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
|
|
- name: Download
|
|
# Same GitHub API path as the pythonlib job above, and the same anonymous
|
|
# rate limit if the token is missing.
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: |
|
|
set -euo pipefail
|
|
# The release built from this tree's sources, installed the way a
|
|
# released library installs its paired browser: stamp the pin, fetch.
|
|
python3 -m ci.release stamp --browser-tag "${{ needs.resolve.outputs.browser_tag }}" \
|
|
--py-version "$(sed -n 's/^version = "\(.*\)"/\1/p' pythonlib/pyproject.toml)" \
|
|
--npm-version "$(python3 -c "import json; print(json.load(open('typescript/package.json'))['version'])")"
|
|
python -m camoufox fetch
|
|
# The ACTIVE build's directory, resolved the way the launcher resolves
|
|
# it. `camoufox path` prints the cache root, and multiversion installs
|
|
# each build under browsers/<channel>/<version>/ -- so reading the
|
|
# binary from the root failed every driver-only run since #772.
|
|
install_dir="$(python -c 'from camoufox.pkgman import camoufox_path; print(camoufox_path(download_if_missing=False))')"
|
|
echo "install dir: $install_dir"
|
|
# Which browser did we actually get? This path does not build: it
|
|
# downloads the release built from exactly these sources, so its
|
|
# Firefox is upstream.sh's by construction (the source digest covers
|
|
# the version line). The suite comes from upstream.sh too; assert that
|
|
# the two agree rather than assume it.
|
|
active="$(python -m camoufox active)"
|
|
echo "fetched: $active"
|
|
python3 -m ci.versions --check-fetched "$active" \
|
|
${{ inputs.browser_version && format('--browser-version {0}', inputs.browser_version) || '' }}
|
|
test -x "$install_dir/camoufox-bin" || {
|
|
echo "::error::no camoufox-bin under $install_dir after fetch"; exit 1; }
|
|
# The published build is packaged, so properties.json and the font
|
|
# bundles are already beside the binary -- the two things the Python
|
|
# API resolves there. Assert rather than assume.
|
|
for required in properties.json fonts; do
|
|
[ -e "$install_dir/$required" ] || {
|
|
echo "::error::the published build has no $required beside the binary"; exit 1; }
|
|
done
|
|
mkdir -p pack/bin
|
|
cp -a "$install_dir/." pack/bin/
|
|
tar -C pack -cf - bin | zstd -3 -T0 -o camoufox-dist.tar.zst
|
|
ls -lh camoufox-dist.tar.zst
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
with:
|
|
name: camoufox-dist
|
|
path: camoufox-dist.tar.zst
|
|
retention-days: 3
|
|
|
|
# ---------------------------------------------------------------------------
|
|
playwright:
|
|
name: Playwright ${{ matrix.shard }}
|
|
# Tier 3b. Gated on 3a: a browser that fails its patch guards is broken, and
|
|
# six shards would take forty minutes to reach the same conclusion.
|
|
needs: [resolve, build, fetch-browser, patch-guards, build-tester]
|
|
if: always() && needs.patch-guards.result == 'success' && needs.build-tester.result == 'success'
|
|
runs-on: ubuntu-24.04
|
|
# A healthy shard is 5-9 minutes. At 120 a wedged shard sat on a runner for
|
|
# two hours before anyone found out, four shards at a time -- which is also
|
|
# a queueing problem for everything behind it. ci/run_playwright.py bounds
|
|
# each pytest invocation at 20 minutes, so this only has to be comfortably
|
|
# clear of one backstop firing and still reporting.
|
|
timeout-minutes: 40
|
|
permissions:
|
|
contents: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: ${{ fromJson(needs.resolve.outputs.shard_matrix) }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- name: Run
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: |
|
|
xvfb-run -a python3 -m ci.run_playwright \
|
|
--shard "${{ matrix.shard }}" \
|
|
--binary "$CAMOUFOX_BINARY" \
|
|
--browser-version "${{ needs.resolve.outputs.browser_version }}" \
|
|
--playwright-tag "${{ needs.resolve.outputs.playwright_tag }}"
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-playwright-${{ strategy.job-index }}
|
|
# One path, and no glob. The summary merges every results-* artifact
|
|
# into one directory and load_all() globs a single level, so these
|
|
# must arrive at the artifact's top level. A second path would move
|
|
# upload-artifact's common root and nest them under results/.
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: diagnostics-playwright-${{ strategy.job-index }}
|
|
path: |
|
|
.ci-work/junit-*.xml
|
|
include-hidden-files: true
|
|
if-no-files-found: ignore
|
|
|
|
# ---------------------------------------------------------------------------
|
|
patch-guards:
|
|
# Tier 3a: the cheap checks on a fresh browser. If these fail the browser is
|
|
# broken in an obvious way and tier 3b would only say so more slowly.
|
|
#
|
|
# Whether the patches APPLY is the build's job; these check that what they
|
|
# do still works. One leg per kind of guarantee (ci/run_patch_guards.py
|
|
# GROUPS), plus the skiplist audit, so a failure names which kind broke and
|
|
# the legs run side by side. One job id, so everything that waits on the
|
|
# guards keeps `needs: patch-guards`.
|
|
if: always() && (needs.build.result == 'success' || needs.fetch-browser.result == 'success')
|
|
name: ${{ matrix.title }}
|
|
needs: [resolve, build, fetch-browser]
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- {leg: spoofing, title: "Patch guards: spoofing"}
|
|
- {leg: automation, title: "Patch guards: automation"}
|
|
- {leg: parity, title: "Patch guards: stock parity"}
|
|
- {leg: skiplist, title: "Skiplist audit"}
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 60
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
# Fonts and properties.json are staged into the artifact by the build job;
|
|
# `make stage-fonts` here would find no source tree and do nothing.
|
|
- if: matrix.leg != 'skiplist'
|
|
run: xvfb-run -a python3 -m ci.run_patch_guards --group "${{ matrix.leg }}" --binary "$CAMOUFOX_BINARY"
|
|
- name: Every skiplist entry is still failing
|
|
if: matrix.leg == 'skiplist'
|
|
# Seconds, because a correct skiplist is short. This is what stops
|
|
# ci/skiplist.yml turning into a list of tests that would now pass --
|
|
# which is what it was: 193 of the 202 tests it skipped passed.
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: |
|
|
xvfb-run -a python3 -m ci.run_skiplist_audit \
|
|
--binary "$CAMOUFOX_BINARY" \
|
|
--browser-version "${{ needs.resolve.outputs.browser_version }}" \
|
|
--playwright-tag "${{ needs.resolve.outputs.playwright_tag }}"
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-patch-guards-${{ matrix.leg }}
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
build-tester:
|
|
# Tier 3a: the cheap checks on a fresh browser. If these fail the browser is
|
|
# broken in an obvious way and tier 3b would only say so more slowly.
|
|
if: always() && (needs.build.result == 'success' || needs.fetch-browser.result == 'success')
|
|
name: build-tester
|
|
needs: [resolve, build, fetch-browser]
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 90
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
cd build-tester && npm install && pip install -r requirements.txt
|
|
# build-tester pulls pythonlib -- and so fpgen -- through its own
|
|
# requirements.txt (`-e ../pythonlib`) rather than `pip install -e
|
|
# pythonlib`, so the pin the other jobs get by matching that line has
|
|
# to be spelled out here. Without it this job was still downloading
|
|
# fpgen's model itself: TLS verification off, no checksum, and only
|
|
# ever the April-2025 release (observed in run 36050915401).
|
|
python3 "$GITHUB_WORKSPACE/scripts/pin-fpgen-model.py"
|
|
- run: xvfb-run -a python3 -m ci.run_build_tester --binary "$CAMOUFOX_BINARY"
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-build-tester
|
|
# One path, and no glob. The summary merges every results-* artifact
|
|
# into one directory and load_all() globs a single level, so these
|
|
# must arrive at the artifact's top level. A second path would move
|
|
# upload-artifact's common root and nest them under results/.
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: diagnostics-build-tester
|
|
path: |
|
|
.ci-work/build-tester-result.json
|
|
include-hidden-files: true
|
|
if-no-files-found: ignore
|
|
|
|
# ---------------------------------------------------------------------------
|
|
native:
|
|
name: Leaks and context semantics
|
|
needs: [resolve, build, fetch-browser, patch-guards, build-tester]
|
|
if: always() && needs.patch-guards.result == 'success' && needs.build-tester.result == 'success'
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 90
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
- name: Run
|
|
# Launches browsers, kills them, and proves nothing survived -- the
|
|
# failure a long-running scraper hits after six hours and no Playwright
|
|
# test can see. Also checks that a context and a browser mean what the
|
|
# project says they mean.
|
|
run: |
|
|
xvfb-run -a python3 -m ci.run_native \
|
|
--subset browser \
|
|
--binary "$CAMOUFOX_BINARY" \
|
|
--rounds 4 --browsers 3
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-native
|
|
# One path, and no glob. The summary merges every results-* artifact
|
|
# into one directory and load_all() globs a single level, so these
|
|
# must arrive at the artifact's top level. A second path would move
|
|
# upload-artifact's common root and nest them under results/.
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: diagnostics-native
|
|
path: |
|
|
.ci-work/junit-*.xml
|
|
include-hidden-files: true
|
|
if-no-files-found: ignore
|
|
|
|
# ---------------------------------------------------------------------------
|
|
typescript-browser:
|
|
name: typescript (browser)
|
|
# Tier 3a. Launches the browser under test through the TS API -- headless,
|
|
# persistent context, and launchServer -- and checks that a page sees the
|
|
# same identity pythonlib's launch of it shows.
|
|
needs: [resolve, build, fetch-browser, typescript]
|
|
if: >-
|
|
always() && needs.typescript.result == 'success' &&
|
|
(needs.build.result == 'success' || needs.fetch-browser.result == 'success')
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 45
|
|
permissions:
|
|
contents: read
|
|
env:
|
|
CAMOUFOX_FPGEN_DATA: ${{ github.workspace }}/.ci-work/fpgen
|
|
# The browser under test is the build job's unpackaged dist/bin, which
|
|
# packages en-US only: scripts/package.py adds the langpacks, and CI never
|
|
# runs it. A de-DE/fr-FR identity therefore presents en-US here though a
|
|
# packaged release presents de-DE. The e2e locale assertions skip on that
|
|
# (named) gap; everything else in the page-vs-config checks still runs.
|
|
# Remove this once the test artifact carries the langpacks.
|
|
CAMOUFOX_TEST_ALLOW_MISSING: packaged-locales
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
- uses: pnpm/action-setup@v4
|
|
with:
|
|
package_json_file: typescript/package.json
|
|
- uses: actions/setup-node@v4
|
|
with:
|
|
node-version: "22"
|
|
cache: pnpm
|
|
cache-dependency-path: typescript/pnpm-lock.yaml
|
|
- uses: actions/cache@v4
|
|
with:
|
|
path: .ci-work/fpgen
|
|
key: fpgen-model-${{ hashFiles('scripts/data/fpgen-model.json') }}
|
|
- name: Run
|
|
env:
|
|
GITHUB_TOKEN: ${{ github.token }}
|
|
run: xvfb-run -a python3 -m ci.run_typescript --browser "$CAMOUFOX_BINARY"
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-typescript-browser
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
growth:
|
|
name: Memory growth ${{ matrix.shard }}
|
|
# Every mechanism is churned twice -- at n and 4n -- to measure whether
|
|
# growth scales with the count: ~38 minutes in one process, which kept this
|
|
# off pull requests. One test per runner (ci.run_native --shard) brings it
|
|
# to the length of a Playwright shard, so it runs on every pull request and
|
|
# is part of the merge gate. Seven shards for the seven tests in
|
|
# native-tests/test_memory_growth.py; with more tests than shards a shard
|
|
# simply runs two.
|
|
needs: [resolve, build, fetch-browser, patch-guards, build-tester]
|
|
if: >-
|
|
always()
|
|
&& needs.patch-guards.result == 'success'
|
|
&& needs.build-tester.result == 'success'
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 60
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: ["1/7", "2/7", "3/7", "4/7", "5/7", "6/7", "7/7"]
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
- run: xvfb-run -a python3 -m ci.run_native --subset growth --shard "${{ matrix.shard }}" --binary "$CAMOUFOX_BINARY"
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
# A shard's "3/7" cannot go in an artifact name.
|
|
name: results-growth-${{ strategy.job-index }}
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
sundial:
|
|
name: Stealth check
|
|
needs: [resolve, build, fetch-browser, patch-guards, build-tester]
|
|
if: always() && needs.resolve.outputs.has_sundial == 'true' && needs.patch-guards.result == 'success' && needs.build-tester.result == 'success'
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 45
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: ./.github/actions/prepare-browser
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: |
|
|
pip install -e pythonlib
|
|
# fpgen downloads its model on first import with TLS verification
|
|
# OFF and no checksum, and its release picker can only ever reach the
|
|
# April-2025 model. Install the pinned one first: see
|
|
# scripts/pin-fpgen-model.py.
|
|
python3 scripts/pin-fpgen-model.py
|
|
- name: Run
|
|
# Exits 0 with a SKIP result if sundial itself is unreachable -- the
|
|
# browser was never measured, so neither a pass nor a failure would be
|
|
# true, and an outage on someone else's host must not block this
|
|
# repository. Anything sundial actually answers -- rejected credential,
|
|
# a role that would be served the vectors, a full report where a score
|
|
# was asked for, a pass rate under the floor -- still fails.
|
|
#
|
|
# The only step in this workflow that sees the sundial credential. It
|
|
# asks for `?auto=1&score=1`, so what comes back is already counts
|
|
# rather than a report; ci/run_sundial.py checks the session's role
|
|
# against sundial's own /__auth/me and refuses to open the browser at
|
|
# all unless the vectors are withheld from it, and refuses to process
|
|
# anything that is not a score. The artifact below is a grade and counts.
|
|
env:
|
|
SUNDIAL_USERNAME: ${{ secrets.SUNDIAL_USERNAME }}
|
|
SUNDIAL_AUTOMATION_KEY: ${{ secrets.SUNDIAL_AUTOMATION_KEY }}
|
|
run: xvfb-run -a python3 -m ci.run_sundial --binary "$CAMOUFOX_BINARY"
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: results-sundial
|
|
path: .ci-work/results/
|
|
include-hidden-files: true
|
|
if-no-files-found: warn
|
|
|
|
# ---------------------------------------------------------------------------
|
|
summary:
|
|
name: Summary
|
|
needs: [resolve, static, pythonlib, typescript, build, fetch-browser, playwright,
|
|
patch-guards, build-tester, native, typescript-browser, growth, sundial]
|
|
if: always() && needs.resolve.result == 'success'
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
pull-requests: write
|
|
outputs:
|
|
verdict: ${{ steps.summarize.outputs.verdict }}
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ inputs.ref || github.ref }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: ${{ env.PYTHON_VERSION }}
|
|
- run: pip install -r ci/requirements.txt
|
|
|
|
- uses: actions/download-artifact@v4
|
|
with:
|
|
pattern: results-*
|
|
merge-multiple: true
|
|
path: .ci-work/results
|
|
continue-on-error: true
|
|
|
|
- name: Summarize
|
|
id: summarize
|
|
run: |
|
|
set +e
|
|
# Suite names, not job names. `static` is a job; the suites it runs are
|
|
# the pipeline self-tests, which write no result, and native_rules,
|
|
# which is named here. The gate separately fails if the job itself did
|
|
# not succeed, so nothing is lost by leaving it out.
|
|
#
|
|
# The browser suites are required either way -- they run against a
|
|
# fetched release just as they do against a fresh build. `build` is
|
|
# the one that is not: on a driver-only pull request that job is
|
|
# skipped by design and writes no result, and requiring it there made
|
|
# summarize report "produced no result file" and fail the gate on
|
|
# every pull request that did not touch the browser. Which is most of
|
|
# them, and exactly the cheap path this pipeline advertises.
|
|
required="pythonlib typescript typescript_browser native_rules patch_guards_spoofing patch_guards_automation patch_guards_parity skiplist_audit build_tester playwright native_browser native_growth"
|
|
if [ "${{ needs.resolve.outputs.browser_changed }}" = "true" ]; then
|
|
required="$required build"
|
|
fi
|
|
if [ "${{ needs.resolve.outputs.has_sundial }}" = "true" ]; then
|
|
required="$required sundial"
|
|
fi
|
|
python3 -m ci.summarize \
|
|
--results-dir .ci-work/results \
|
|
--require $required \
|
|
--allow-skip sundial \
|
|
--markdown summary.md \
|
|
--out summary.json \
|
|
--browser-version "${{ needs.resolve.outputs.browser_version }}" \
|
|
--browser-release "${{ needs.resolve.outputs.browser_release }}" \
|
|
--playwright-tag "${{ needs.resolve.outputs.playwright_tag }}" \
|
|
--playwright-firefox "${{ needs.resolve.outputs.playwright_firefox }}" \
|
|
--version-note "${{ needs.resolve.outputs.version_note }}"
|
|
code=$?
|
|
echo "verdict=$([ $code -eq 0 ] && echo pass || echo fail)" >> "$GITHUB_OUTPUT"
|
|
exit $code
|
|
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: test-summary
|
|
path: |
|
|
summary.md
|
|
summary.json
|
|
.ci-work/results/
|
|
include-hidden-files: true
|
|
|
|
- name: Comment on the pull request
|
|
if: always() && github.event_name == 'pull_request'
|
|
env:
|
|
GH_TOKEN: ${{ github.token }}
|
|
continue-on-error: true
|
|
run: |
|
|
# One rolling comment rather than a new one per push.
|
|
marker="<!-- camoufox-tests -->"
|
|
{ echo "$marker"; cat summary.md; } > body.md
|
|
existing=$(gh pr view "${{ github.event.pull_request.number }}" \
|
|
--json comments --jq "[.comments[] | select(.body | startswith(\"$marker\"))][0].id" 2>/dev/null || true)
|
|
if [ -n "$existing" ] && [ "$existing" != "null" ]; then
|
|
gh api -X PATCH "repos/${{ github.repository }}/issues/comments/${existing#*_}" \
|
|
-f body="$(cat body.md)" >/dev/null 2>&1 \
|
|
|| gh pr comment "${{ github.event.pull_request.number }}" --body-file body.md
|
|
else
|
|
gh pr comment "${{ github.event.pull_request.number }}" --body-file body.md
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
gate:
|
|
name: All tests passed
|
|
# THE required status check. Branch protection points at this one job rather
|
|
# than at a dozen, so the required-check list does not have to be edited
|
|
# every time a suite is added, renamed, or sharded differently.
|
|
#
|
|
# A job that was legitimately not applicable is allowed to be skipped -- the
|
|
# build when the browser was fetched instead, the fetch when it was built,
|
|
# the stealth check on a fork pull request with no credentials or while it is
|
|
# disabled in ci/sundial.yml. Anything else
|
|
# that is not `success`, including `skipped`, fails the gate: a suite that
|
|
# did not run has not passed, and silently skipping one is the cheapest way
|
|
# to a green tick.
|
|
needs: [resolve, static, pythonlib, typescript, build, fetch-browser, playwright,
|
|
patch-guards, build-tester, native, typescript-browser, growth, sundial, summary]
|
|
if: always()
|
|
runs-on: ubuntu-24.04
|
|
permissions:
|
|
contents: read
|
|
steps:
|
|
- name: Check every tier
|
|
env:
|
|
RESULTS: ${{ toJSON(needs) }}
|
|
BROWSER_CHANGED: ${{ needs.resolve.outputs.browser_changed }}
|
|
HAS_SUNDIAL: ${{ needs.resolve.outputs.has_sundial }}
|
|
run: |
|
|
python3 - <<'PY'
|
|
import json, os, sys
|
|
|
|
results = {name: job["result"] for name, job in json.loads(os.environ["RESULTS"]).items()}
|
|
built = os.environ.get("BROWSER_CHANGED") == "true"
|
|
|
|
# The only jobs allowed to be skipped, and only for these reasons.
|
|
may_skip = {
|
|
"build": not built,
|
|
"fetch-browser": built,
|
|
"sundial": os.environ.get("HAS_SUNDIAL") != "true",
|
|
}
|
|
|
|
problems = []
|
|
for name, result in sorted(results.items()):
|
|
if result == "success":
|
|
continue
|
|
if result == "skipped" and may_skip.get(name):
|
|
print(f" - {name}: skipped (not applicable to this run)")
|
|
continue
|
|
problems.append(f"{name}: {result}")
|
|
|
|
width = max(len(n) for n in results)
|
|
print("\ntier results:")
|
|
for name, result in sorted(results.items()):
|
|
mark = "ok " if result == "success" else "FAIL"
|
|
print(f" {mark} {name:<{width}} {result}")
|
|
|
|
if problems:
|
|
print("\nnot mergeable:")
|
|
for problem in problems:
|
|
print(f" - {problem}")
|
|
sys.exit(1)
|
|
print("\nevery tier passed; this pull request is mergeable.")
|
|
PY
|