Files
Cole W. LeavittandClaude Opus 5.5 947685a0c0 patch-guards: check that a bare launch on Linux renders the host's fonts (#1044)
camoufox-bin started without the launcher's config claims no OS, so its
fonts must be the host's plus Twemoji Mozilla, the font stock Firefox
bundles. Up to 158.0.1-beta.41, Linux added fonts/ to fontconfig
recursively and so registered every OS's font set. A bare launch then
resolved Segoe UI, Lucida Console and Cambria Math under a Linux
User-Agent. Akamai's detectOSFontsSignal reads that as Windows (1 where
stock gives 5), and sans-serif measured 99 px high where stock measures
98. #997 changed font-hijacker.patch to add only the directory's own
files, which fixed this, but no test covers it.

The guard launches the binary twice: once as shipped, and once on a
profile with gfx.bundled-fonts.activate=0, which registers nothing from
fonts/. Every family in fonts.json and each generic's 72 px span must
measure the same in both. The only exception is Twemoji Mozilla, which
must render only in the default launch. The pref goes in prefs.js
because the font list reads it once, at startup.

On 158.0.1-beta.41 it fails: 646 extra families, and sans-serif is 99
px high. On a main build (159.0-beta.35) it passes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 21:13:20 +00:00
..

The test pipeline

Everything that runs a suite against Camoufox. The same definition runs on a pull request, on a push to main, before every release (through release.yml), and for any caller of the workflow_call trigger that wants a specific browser version, so there is one meaning of "the tests pass". Every job can be run locally (Running a piece by hand).

resolve ── static ─────────────── lint, tribal rules, skiplist, self-tests  (seconds)
             ├─ typescript ────── type check, lint, vitest, golden parity
             └─ pythonlib ─────── the package's own tests                   (a minute)
                  └─ build or fetch ─┬─ patch guards × 3 + skiplist audit
                                     ├─ build-tester ─────── 8 fingerprint profiles
                                     ├─ typescript-browser ─ the npm launcher end to end
                                     ├─ install ──────────── packed packages: fetch, launch, click, type
                                     └─ once guards and build-tester pass:
                                          ├─ playwright × 6 shards   (upstream suite + our own)
                                          ├─ native ───────── leaks, contexts, crash recovery
                                          ├─ sundial ──────── stealth grade
                                          └─ growth × 7 shards  memory growth, one test per runner
                                  │
                               summary ──► one comment on the PR ──► gate: "All tests passed"

A push to main whose tree a passing pull request run already tested (nothing else merged in between) runs none of that again. resolve finds the run (ci.carry), carry-browser takes over the browser it tested, sundial runs against it, since a pull request from a fork has no credential for it, and summary reports the carried results with this run's grade. Each pull request run records the tree it tested as a tested-tree-<tree> artifact. A push that lands any other tree, or whose run's browser has expired (three days), is tested in full.

Jobs

Job Runner What it proves
static ci.run_native --subset rules, pytest ci/tests, browser/scripts/check-input-dispatch.py Settled decisions still hold, the skiplist is valid, input goes through one chokepoint, the pipeline reports honestly
pythonlib ci.run_pythonlib python/'s own tests
typescript ci.run_typescript Type check, lint, vitest, byte-for-byte parity with python/
install ci.run_install The packed packages install into clean environments, fetch a real browser, and launch it and the browser under test through the installed package: the User-Agent claims the launched Firefox, the page and the request agree on it, the bundled fonts render, and a click and typed text reach the page as trusted events
build / fetch-browser ci.run_prepare, ci.run_build The browser under test: built and packaged from this branch by the command a release's build runs, or the published release built from the same sources
preflight ci.run_prepare, ci.run_build --preflight Each target only a release builds (linux arm64, windows x86_64 and i686, macos x86_64 and arm64) configures, passes the export tiers, and compiles every source file a patch or an addition touches (below)
patch-guards ci.run_patch_guards --group spoofing|automation|parity, ci.run_skiplist_audit One guard per shipped behaviour (browser/tests/playwright/); every skiplist entry still fails
build-tester ci.run_build_tester The raw binary against the anti-detect checks (README)
typescript-browser ci.run_typescript --browser The npm launcher against a real browser
playwright ci.run_playwright --shard i/6 Upstream playwright-python's suite, plus browser/tests/playwright/camoufox/
native ci.run_native --subset browser Leaks, context isolation, crash recovery (below)
growth ci.run_native --subset growth --shard i/7 Memory growth, one test per runner
carry-browser A carried push to main: the pull request run's browser, cached for main when it was built
sundial ci.run_sundial The stealth grade (below)
summary ci.summarize Folds every result into one table and a pull request comment
gate The required check, All tests passed

Which browser, which suite

ci/versions.py answers both, and every entry point uses it:

Question Answer
Browser browser/upstream.sh, or the version a caller passes
Suite The newest released playwright-python tag whose pinned Firefox is not ahead of that browser, and which is below the Playwright ceiling in python/pyproject.toml

Playwright trails Firefox and skips versions, so the suite's Firefox is often a release or two behind the browser. That is the normal case; the summary says which rule picked the tag. The ceiling exists because camoufox.server imports a private Playwright API and every Playwright minor may change Juggler.

python3 -m ci.versions --json
python3 -m ci.versions --browser-version 153.0.4 --json

Only suite selection follows --browser-version; the build always reads upstream.sh. --check-upstream, which the workflow passes, refuses a version the branch does not pin, so an old browser is never judged against a new suite. An upgrade to a new Firefox is therefore a branch that edits upstream.sh.

The Playwright suite

Upstream playwright-python at the resolved tag, fetched fresh every run and run unmodified. ci/pw_camoufox_plugin.py adapts the environment around it, and ci/suite.py overlays browser/tests/playwright/camoufox/. Why there is no vendored copy: browser/tests/playwright/README.md.

ci/run_playwright.py names its targets explicitly, so nothing is left out silently:

Targets How they run
tests/async/, tests/sync/ One pytest process
tests/common/, tests/test_reference_count_async.py Their own process: they start their own event loop
tests/test_installation.py Excluded, with a reason

unclaimed() fails the run if upstream adds a test path that is neither in TARGETS nor in EXCLUDED.

Isolated world first

Camoufox ships with Playwright's code in an isolated world, but upstream's tests assume the main world (they read globals their own page scripts defined). Each group therefore runs in up to three passes:

Pass CI_WORLD What it establishes
1 isolated The browser as users run it
2 main The pass-1 failures again, isolation off
3 main What failed in both, once more; normally empty

A test that passes only in pass 2 is a main-world fallback: it counts as a pass, and the summary reports main_world_fallback_count. That number is the isolated-world conformance gap; a jump means the isolation boundary moved. Both rerun passes run only when the failure set is non-empty, because pytest's --last-failed with nothing to filter reruns the whole group.

Pass 1 is cost-bounded: a 90 s per-test timeout, and upstream's own reruns off (CI="", because upstream's conftest overrides --reruns).

Some modules hang rather than fail under isolation. A Playwright feature that works by replacing a page global, such as route_web_socket (which replaces window.WebSocket) or a binding the page calls, lands in the isolated world, so anything the page itself does never reaches it. In the sync API no timeout can stop such a hang cleanly. Those modules are listed in ISOLATION_HANGS in ci/run_playwright.py and run directly in the main world, counted as fallbacks. The route_web_socket gap is real for users too: #775.

The skiplist

ci/skiplist.yml deselects the upstream tests Camoufox fails on purpose, each with a reason (ci/summarize.py rejects an entry without one). A reason is not evidence, so ci/run_skiplist_audit.py runs every entry in the main world with the skiplist off and fails the build if a skipped test passes. An entry whose test passes or fails with machine speed names, in checked_by, the patch guard that checks its reason instead. ISOLATION_HANGS is a different list: those tests pass in the main world, so they would fail the audit, and test_isolation_hangs_are_not_in_the_skiplist keeps the two apart.

The entries cover keyboard input Camoufox presses as a real keyboard would, client certificates the browser must present during the TLS handshake, HTTPS-First through upstream's GET-only test proxy, the popup blocker Camoufox keeps on, and scrollbars Camoufox does not hide in headless mode. CI is the authority on what fails; a local run is a hypothesis.

Camoufox's own suite

browser/tests/native/ covers what the Playwright suite cannot ask about:

Area What it checks
Leaks Launching and killing browsers leaves no file descriptors, sockets, child processes or X11 lock files, and the cost does not grow with launch count
Contexts versus browsers Two contexts in one browser get different fingerprints; two pages in one context get the same one
Crashes Killing the browser, the X server, a content process or the driver mid-run leaves nothing behind, and the next launch works (test_crash_recovery.py)
Memory growth One mechanism (iframes, canvas readback, WebGL contexts, workers, script compilation, font measurement) driven N and 4N times grows by a bounded amount (test_memory_growth.py, the growth job)
Settled decisions Every mechanically checkable rule in ci/tribal-rules.yml (test_tribal_rules.py)
GPU display headless="virtual-gpu" renders on the card CAMOUFOX_GPU names, the identity claims that card, and the Xorg server exits with the browser (test_gpu_display.py). No runner has a GPU, so the gpu subset is run by hand and is not part of all

Stealth grade

ci/run_sundial.py reports a letter grade and a count, nothing else: no vector name, value or source, and no per-category breakdown. Everything that leaves redact() is checked against a whitelist (_PUBLISHABLE), so adding a field fails the run. The request asks for the score only (?auto=1&score=1), and a full report arriving anyway fails the run.

{ "grade": "A", "checks_total": 412, "checks_passed": 403, "pass_rate": 0.978,
  "out_of_scope_failed": 6, "cross_os_total": 24, "cross_os_passed": 5,
  "os": "linux", "sundial_version": "0.3.1", "schema_version": 1 }
Setting Where
On/off switch, gated categories, min_pass_rate ci/sundial.yml
Credential SUNDIAL_AUTOMATION_KEY secret; SUNDIAL_USERNAME is optional
Keep an encrypted full report for yourself SUNDIAL_REPORT_AGE_RECIPIENT (an age public key)

Cross-OS checks read the host rather than the disguise, so they are counted separately and never scored. Without a credential (a fork pull request) the job is skipped and the summary says so.

To see which category fails, run locally; --explain is refused under GitHub Actions:

python3 -m ci.run_sundial --explain --binary /path/to/camoufox-bin

Blocking a merge

Branch protection on main requires one check, All tests passed (the gate job), so the list does not change when a suite is added or resharded. Anything that is not success fails it, including skipped, except:

May be skipped When
build, preflight The browser was fetched, not compiled
fetch-browser The browser was compiled, not fetched
sundial Disabled in ci/sundial.yml, or no credential
carry-browser The push to main was not carried
every suite but sundial The push to main was carried; summary requires their results from the pull request run

sundial may also record a skip result when the sundial service is unreachable. A rejected credential, a full report, or a score under the floor still fails.

The settings are in ci/branch-protection.json. To apply them (needs admin):

gh api -X PUT repos/<owner>/<repo>/branches/main/protection --input ci/branch-protection.json
Setting Why
enforce_admins: false A maintainer can still merge when CI itself is broken
strict: false A pull request need not be rebased onto main before merging; otherwise every push to main would force a rebuild of every open pull request
No required reviews A solo maintainer cannot approve their own pull request

Cost control

Each tier gates the next, so a lint failure never reaches the build:

Tier Jobs Time
0 static lint, self-tests, settled decisions seconds
1 unit pythonlib, typescript about a minute
2 browser build and five preflights (browser sources changed) or fetch (they did not) minutes to hours
3a smoke patch guards, skiplist audit, build-tester, typescript-browser, install about 15 min
3b full Playwright × 6, native, growth × 7, stealth about 40 min
4 gate the required check seconds
Mechanism Effect
Fetch instead of build A pull request that does not touch browser sources tests the published release built from exactly this tree's sources (ci.release paired). With no such release, it builds.
Compiled-input cache The build cache is keyed on a hash of the compiled inputs only (ci/browser_inputs.py). On a hit the browser is restored and this branch's Juggler JavaScript and other resources, the files Juggler's moz.build lists in FINAL_TARGET_FILES, are laid over it. Everything else, including any new file type, counts as native and forces a build.
No partial restore That cache has no restore-keys: a partial match would hand the tests a browser built from different sources. A self-test checks the key covers every browser-affecting path.
Warm ccache Pushes to main that build, and a twice-weekly schedule, keep the ccache from being evicted; pull requests restore it. A carried push builds nothing, so it caches the pull request's browser under the compiled-input key instead.
Carried pushes A push to main that lands the tree its pull request run tested reuses that run's results (above).
PGO and LTO Every build uses Mozilla's PGO profile for the same Firefox and links libxul with cross-language LTO, as a release does (builds-are-optimized-like-stock). ccache caches the compiles, not that link, so a warm build still pays for it.
Network retries only ci.run_prepare (setup-minimal, dir, mozbootstrap) retries only failures that read as transient network errors. A failed hunk or compile error fails at once.

Pushing to a branch cancels its running build (cancel-in-progress).

What a release builds

A release builds six targets and a pull request builds one. Every step of a release's build therefore also runs on a pull request (one-build-entry-point):

Pull request Release
Runner setup build-host the same action
Source tree ci.run_prepare the same
linux x86_64 ci.run_build: multibuild.py retargets, builds and packages the same command
The other five targets ci.run_build --preflight: multibuild.py retargets, then preflight.py configures, runs the pre-export, export and pre-compile tiers and compiles the objects of every patched or added source file ci.run_build: the full build

A preflight does not link or package, so a link error on a cross-compiled target is still found by a release. Its compiles are cached per target, and a target that passed is not preflighted again until the browser's compiled inputs, build-host or the runner scripts change; the twice-weekly scheduled run preflights every target regardless.

Releases

One workflow, release.yml, makes every release, on a train: on the 1st and 15th of each month the newest prerelease is promoted to stable, and a new prerelease is cut from main. Merges are tested by tests.yml but not released, so users see two releases a month rather than one per merge. Each library release is paired with exactly one browser build.

1st and 15th ─► plan ─► build-library ─► promote-browser ─► publish-pypi ─► publish-npm ─► tag-library ─► cut
 (schedule)    (newest prerelease,        paired → stable     0.5.8            0.5.8 @latest   v0.5.8         │
                no open regression)                                                                         │
     ┌──────────────────────────────────────────────────────────────────────────────────────────────────────┘
     ▼
dispatch on main ─► ubo ─► tests ─► plan ─┬─► build-browser ─► publish-browser ─┐
                  (AMO's  (tests.yml,     │   (only if its      v156.0.1-beta.N  │
                   newest) unless main's  │    sources changed)                  ├─► publish-pypi ─► publish-npm ─► tag-library
                           run passed it) │                                      │
                                          └─► build-library ─────────────────────┘   0.5.9bN          0.5.9-beta.N    v0.5.9bN
                                              (only if what it ships changed)        (pip: --pre)     @next

git tag v0.5.8 <tested main commit> && git push origin v0.5.8
               ─► plan ─► build-library ─► promote-browser ─► publish-pypi ─► publish-npm
                  (checks)                 paired → stable       0.5.8           0.5.8 @latest
Rule Detail
The train The scheduled run promotes the newest library prerelease tag (v0.5.8bN) at the commit it was cut from, as v0.5.8, then its cut job dispatches release.yml on main. With no prerelease since the last promotion it promotes nothing; a cut with nothing changed publishes nothing.
A regression holds the train While an issue labelled regression is open, the promotion fails and says which issues hold it; the cut still runs. The label must exist, so an empty answer means "none open", not "nobody could mark one".
Out of cycle Run the Release workflow by hand on main (gh workflow run release.yml --ref main) to cut a prerelease now. Push a tag to promote by hand.
Only a tested commit is released A cut calls tests.yml on its commit and continues only if it passes. A promotion requires All tests passed on its commit.
The browser is built only when its sources changed ci.browser_inputs.source_digest() hashes the browser's sources. Each browser release carries a manifest.json with that digest, its commit, and the browser interface it speaks (CONSTRAINTS.INTERFACE in python/src/camoufox/__version__.py). A matching release is reused; otherwise plan takes the next unused beta.N. Nothing is committed: ci.release set-build writes the number into the build's tree.
uBlock Origin Every browser package bundles the uBO release pinned in browser/scripts/data/ubo.json under addons/UBO/ (browser/scripts/fetch-ubo.py), and both launchers load it from there. A cut reads the pin once, in the ubo job before tests, and hands it to the tests and every build, so the browser ships the uBO it was tested with. A bump is a reviewed commit (fetch-ubo.py newest prints the candidate). Its manifest.json (ubo) and release notes name it. A reused browser keeps the uBO it was built with.
Builds are attested gh attestation verify <file> --repo daijro/camoufox
Packages are built once publish-pypi and publish-npm upload the files build-library made: 0.5.8b2 on PyPI, 0.5.8-beta.2 on npm. pip install camoufox skips prereleases without --pre; npm installs latest, not next.
The pairing ci.release stamp writes the paired browser release into python/src/camoufox/browser-pin.json, which both launchers read. On main the file is {}, so a development checkout follows its channel.
Promotion Refused unless the commit is on main, All tests passed succeeded on it, X.Y.Z is newer than every release, and a browser release was built from its sources. The paired browser becomes the stable release without a rebuild.
Failed publish Re-run the failed jobs; every version and tag comes from plan.
Credentials None stored. PyPI and npm use trusted publishing (OIDC) from release.yml.

Releases cut before manifests existed (up to v156.0.1-beta.32) pair by the tag upstream.sh names.

Running a piece by hand

Command Needs a browser What it runs
python3 -m ci.run_prepare make setup-minimal, dir, mozbootstrap
python3 -m ci.run_build The browser build and package for linux x86_64; --target, --arch for another
python3 -m ci.run_build --target windows --arch x86_64 --preflight Preflight one target
python3 -m ci.run_pythonlib python/'s tests
python3 -m ci.run_typescript typescript/'s tests
python3 -m ci.run_typescript --browser <bin> yes The npm launcher end to end
python3 -m ci.run_install --tmpdir /mnt/tmpfs --browser <bin> yes Packed packages, a real fetch, and a launch, click and typing through each installed package; the tmpdir must be its own filesystem
python3 -m ci.run_patch_guards --binary <bin> yes Patch guards; --group spoofing|automation|parity, --only <name>
python3 -m ci.run_patch_guards --binary <bin> --only stock-gpu-parity yes WebGL and WebGPU against stock Firefox of the same version, downloaded into .ci-work (upgrade step)
python3 -m ci.run_build_tester --binary <bin> yes build-tester
python3 -m ci.run_skiplist_audit --binary <bin> yes Every skiplist entry still fails
python3 -m ci.run_playwright --binary <bin> yes The Playwright suite; --shard 3/6 for one shard
python3 -m ci.run_native --subset rules Settled decisions
python3 -m ci.run_native --subset browser --binary <bin> yes Leaks, contexts, crashes
python3 -m ci.run_native --subset growth --binary <bin> yes Memory growth; --shard 3/7 for one test
CAMOUFOX_GPU=<pci> python3 -m ci.run_native --subset gpu --binary <bin> yes headless="virtual-gpu" on a real GPU
python3 -m ci.run_sundial --binary <bin> yes The stealth grade (needs the credential)
python3 -m ci.summarize --results-dir .ci-work/results The summary table
GITHUB_EVENT_NAME=schedule python3 -m ci.release target What the train would promote
python3 -m ci.release browser-plan Whether a browser would be built or reused
python3 -m ci.release paired The release built from this tree
python3 -m ci.release lib-plan --channel prerelease The next library version
python3 browser/scripts/fetch-ubo.py resolve The uBlock Origin release a cut would bundle
make -C browser stage-ubo Bundle it into your build tree's dist/bin, as packaging does
python3 -m pytest ci/tests -q The pipeline's own tests

<bin> is a camoufox-bin; make -C browser path prints the one in your build tree. Each runner writes one result file to .ci-work/results/ (run_prepare writes none). A required suite with no result file is a failure, never a skip.

Self-tests

ci/tests/ asserts the pipeline reports honestly: redaction leaks nothing, skips carry reasons, shards partition exactly once, and version resolution never picks a suite newer than the browser. They run in the static job.