camoufox-bin started without the launcher's config claims no OS, so its fonts must be the host's plus Twemoji Mozilla, the font stock Firefox bundles. Up to 158.0.1-beta.41, Linux added fonts/ to fontconfig recursively and so registered every OS's font set. A bare launch then resolved Segoe UI, Lucida Console and Cambria Math under a Linux User-Agent. Akamai's detectOSFontsSignal reads that as Windows (1 where stock gives 5), and sans-serif measured 99 px high where stock measures 98. #997 changed font-hijacker.patch to add only the directory's own files, which fixed this, but no test covers it. The guard launches the binary twice: once as shipped, and once on a profile with gfx.bundled-fonts.activate=0, which registers nothing from fonts/. Every family in fonts.json and each generic's 72 px span must measure the same in both. The only exception is Twemoji Mozilla, which must render only in the default launch. The pref goes in prefs.js because the font list reads it once, at startup. On 158.0.1-beta.41 it fails: 646 extra families, and sans-serif is 99 px high. On a main build (159.0-beta.35) it passes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test pipeline
Everything that runs a suite against Camoufox. The same definition runs on a
pull request, on a push to main, before every release (through
release.yml), and for any caller of the workflow_call trigger that wants a
specific browser version, so there is one meaning of "the tests pass". Every job can be run locally
(Running a piece by hand).
resolve ── static ─────────────── lint, tribal rules, skiplist, self-tests (seconds)
├─ typescript ────── type check, lint, vitest, golden parity
└─ pythonlib ─────── the package's own tests (a minute)
└─ build or fetch ─┬─ patch guards × 3 + skiplist audit
├─ build-tester ─────── 8 fingerprint profiles
├─ typescript-browser ─ the npm launcher end to end
├─ install ──────────── packed packages: fetch, launch, click, type
└─ once guards and build-tester pass:
├─ playwright × 6 shards (upstream suite + our own)
├─ native ───────── leaks, contexts, crash recovery
├─ sundial ──────── stealth grade
└─ growth × 7 shards memory growth, one test per runner
│
summary ──► one comment on the PR ──► gate: "All tests passed"
A push to main whose tree a passing pull request run already tested (nothing
else merged in between) runs none of that again. resolve finds the run
(ci.carry), carry-browser takes over the browser it tested, sundial runs
against it, since a pull request from a fork has no credential for it, and
summary reports the carried results with this run's grade. Each pull request
run records the tree it tested as a tested-tree-<tree> artifact. A push that
lands any other tree, or whose run's browser has expired (three days), is tested
in full.
Jobs
| Job | Runner | What it proves |
|---|---|---|
static |
ci.run_native --subset rules, pytest ci/tests, browser/scripts/check-input-dispatch.py |
Settled decisions still hold, the skiplist is valid, input goes through one chokepoint, the pipeline reports honestly |
pythonlib |
ci.run_pythonlib |
python/'s own tests |
typescript |
ci.run_typescript |
Type check, lint, vitest, byte-for-byte parity with python/ |
install |
ci.run_install |
The packed packages install into clean environments, fetch a real browser, and launch it and the browser under test through the installed package: the User-Agent claims the launched Firefox, the page and the request agree on it, the bundled fonts render, and a click and typed text reach the page as trusted events |
build / fetch-browser |
ci.run_prepare, ci.run_build |
The browser under test: built and packaged from this branch by the command a release's build runs, or the published release built from the same sources |
preflight |
ci.run_prepare, ci.run_build --preflight |
Each target only a release builds (linux arm64, windows x86_64 and i686, macos x86_64 and arm64) configures, passes the export tiers, and compiles every source file a patch or an addition touches (below) |
patch-guards |
ci.run_patch_guards --group spoofing|automation|parity, ci.run_skiplist_audit |
One guard per shipped behaviour (browser/tests/playwright/); every skiplist entry still fails |
build-tester |
ci.run_build_tester |
The raw binary against the anti-detect checks (README) |
typescript-browser |
ci.run_typescript --browser |
The npm launcher against a real browser |
playwright |
ci.run_playwright --shard i/6 |
Upstream playwright-python's suite, plus browser/tests/playwright/camoufox/ |
native |
ci.run_native --subset browser |
Leaks, context isolation, crash recovery (below) |
growth |
ci.run_native --subset growth --shard i/7 |
Memory growth, one test per runner |
carry-browser |
A carried push to main: the pull request run's browser, cached for main when it was built |
|
sundial |
ci.run_sundial |
The stealth grade (below) |
summary |
ci.summarize |
Folds every result into one table and a pull request comment |
gate |
The required check, All tests passed |
Which browser, which suite
ci/versions.py answers both, and every entry point uses it:
| Question | Answer |
|---|---|
| Browser | browser/upstream.sh, or the version a caller passes |
| Suite | The newest released playwright-python tag whose pinned Firefox is not ahead of that browser, and which is below the Playwright ceiling in python/pyproject.toml |
Playwright trails Firefox and skips versions, so the suite's Firefox is often a
release or two behind the browser. That is the normal case; the summary says
which rule picked the tag. The ceiling exists because camoufox.server imports
a private Playwright API and every Playwright minor may change Juggler.
python3 -m ci.versions --json
python3 -m ci.versions --browser-version 153.0.4 --json
Only suite selection follows --browser-version; the build always reads
upstream.sh. --check-upstream, which the workflow passes, refuses a version
the branch does not pin, so an old browser is never judged against a new suite.
An upgrade to a new Firefox is therefore a branch that edits upstream.sh.
The Playwright suite
Upstream playwright-python at the resolved tag, fetched fresh every run and run
unmodified. ci/pw_camoufox_plugin.py adapts the environment around it, and
ci/suite.py overlays browser/tests/playwright/camoufox/.
Why there is no vendored copy: browser/tests/playwright/README.md.
ci/run_playwright.py names its targets explicitly, so nothing is left out
silently:
| Targets | How they run |
|---|---|
tests/async/, tests/sync/ |
One pytest process |
tests/common/, tests/test_reference_count_async.py |
Their own process: they start their own event loop |
tests/test_installation.py |
Excluded, with a reason |
unclaimed() fails the run if upstream adds a test path that is neither in
TARGETS nor in EXCLUDED.
Isolated world first
Camoufox ships with Playwright's code in an isolated world, but upstream's tests assume the main world (they read globals their own page scripts defined). Each group therefore runs in up to three passes:
| Pass | CI_WORLD |
What it establishes |
|---|---|---|
| 1 | isolated |
The browser as users run it |
| 2 | main |
The pass-1 failures again, isolation off |
| 3 | main |
What failed in both, once more; normally empty |
A test that passes only in pass 2 is a main-world fallback: it counts as a
pass, and the summary reports main_world_fallback_count. That number is the
isolated-world conformance gap; a jump means the isolation boundary moved. Both
rerun passes run only when the failure set is non-empty, because pytest's
--last-failed with nothing to filter reruns the whole group.
Pass 1 is cost-bounded: a 90 s per-test timeout, and upstream's own reruns off
(CI="", because upstream's conftest overrides --reruns).
Some modules hang rather than fail under isolation. A Playwright feature
that works by replacing a page global, such as route_web_socket (which
replaces window.WebSocket) or a binding the page calls, lands in the isolated
world, so anything the page itself does never reaches it. In the sync API no
timeout can stop such a hang cleanly. Those modules are listed in
ISOLATION_HANGS in ci/run_playwright.py and run directly in the main world,
counted as fallbacks. The route_web_socket gap is real for users too:
#775.
The skiplist
ci/skiplist.yml deselects the upstream tests Camoufox fails on
purpose, each with a reason (ci/summarize.py rejects an entry without one).
A reason is not evidence, so ci/run_skiplist_audit.py runs every entry in the
main world with the skiplist off and fails the build if a skipped test
passes. An entry whose test passes or fails with machine speed names, in
checked_by, the patch guard that checks its reason instead.
ISOLATION_HANGS is a different list: those tests pass in the main
world, so they would fail the audit, and test_isolation_hangs_are_not_in_the_skiplist
keeps the two apart.
The entries cover keyboard input Camoufox presses as a real keyboard would, client certificates the browser must present during the TLS handshake, HTTPS-First through upstream's GET-only test proxy, the popup blocker Camoufox keeps on, and scrollbars Camoufox does not hide in headless mode. CI is the authority on what fails; a local run is a hypothesis.
Camoufox's own suite
browser/tests/native/ covers what the Playwright suite cannot ask about:
| Area | What it checks |
|---|---|
| Leaks | Launching and killing browsers leaves no file descriptors, sockets, child processes or X11 lock files, and the cost does not grow with launch count |
| Contexts versus browsers | Two contexts in one browser get different fingerprints; two pages in one context get the same one |
| Crashes | Killing the browser, the X server, a content process or the driver mid-run leaves nothing behind, and the next launch works (test_crash_recovery.py) |
| Memory growth | One mechanism (iframes, canvas readback, WebGL contexts, workers, script compilation, font measurement) driven N and 4N times grows by a bounded amount (test_memory_growth.py, the growth job) |
| Settled decisions | Every mechanically checkable rule in ci/tribal-rules.yml (test_tribal_rules.py) |
| GPU display | headless="virtual-gpu" renders on the card CAMOUFOX_GPU names, the identity claims that card, and the Xorg server exits with the browser (test_gpu_display.py). No runner has a GPU, so the gpu subset is run by hand and is not part of all |
Stealth grade
ci/run_sundial.py reports a letter grade and a count, nothing else: no
vector name, value or source, and no per-category breakdown. Everything that
leaves redact() is checked against a whitelist (_PUBLISHABLE), so adding a
field fails the run. The request asks for the score only (?auto=1&score=1),
and a full report arriving anyway fails the run.
{ "grade": "A", "checks_total": 412, "checks_passed": 403, "pass_rate": 0.978,
"out_of_scope_failed": 6, "cross_os_total": 24, "cross_os_passed": 5,
"os": "linux", "sundial_version": "0.3.1", "schema_version": 1 }
| Setting | Where |
|---|---|
On/off switch, gated categories, min_pass_rate |
ci/sundial.yml |
| Credential | SUNDIAL_AUTOMATION_KEY secret; SUNDIAL_USERNAME is optional |
| Keep an encrypted full report for yourself | SUNDIAL_REPORT_AGE_RECIPIENT (an age public key) |
Cross-OS checks read the host rather than the disguise, so they are counted separately and never scored. Without a credential (a fork pull request) the job is skipped and the summary says so.
To see which category fails, run locally; --explain is refused under GitHub
Actions:
python3 -m ci.run_sundial --explain --binary /path/to/camoufox-bin
Blocking a merge
Branch protection on main requires one check, All tests passed (the
gate job), so the list does not change when a suite is added or resharded.
Anything that is not success fails it, including skipped, except:
| May be skipped | When |
|---|---|
build, preflight |
The browser was fetched, not compiled |
fetch-browser |
The browser was compiled, not fetched |
sundial |
Disabled in ci/sundial.yml, or no credential |
carry-browser |
The push to main was not carried |
every suite but sundial |
The push to main was carried; summary requires their results from the pull request run |
sundial may also record a skip result when the sundial service is
unreachable. A rejected credential, a full report, or a score under the floor
still fails.
The settings are in ci/branch-protection.json. To
apply them (needs admin):
gh api -X PUT repos/<owner>/<repo>/branches/main/protection --input ci/branch-protection.json
| Setting | Why |
|---|---|
enforce_admins: false |
A maintainer can still merge when CI itself is broken |
strict: false |
A pull request need not be rebased onto main before merging; otherwise every push to main would force a rebuild of every open pull request |
| No required reviews | A solo maintainer cannot approve their own pull request |
Cost control
Each tier gates the next, so a lint failure never reaches the build:
| Tier | Jobs | Time |
|---|---|---|
| 0 static | lint, self-tests, settled decisions | seconds |
| 1 unit | pythonlib, typescript | about a minute |
| 2 browser | build and five preflights (browser sources changed) or fetch (they did not) | minutes to hours |
| 3a smoke | patch guards, skiplist audit, build-tester, typescript-browser, install | about 15 min |
| 3b full | Playwright × 6, native, growth × 7, stealth | about 40 min |
| 4 gate | the required check | seconds |
| Mechanism | Effect |
|---|---|
| Fetch instead of build | A pull request that does not touch browser sources tests the published release built from exactly this tree's sources (ci.release paired). With no such release, it builds. |
| Compiled-input cache | The build cache is keyed on a hash of the compiled inputs only (ci/browser_inputs.py). On a hit the browser is restored and this branch's Juggler JavaScript and other resources, the files Juggler's moz.build lists in FINAL_TARGET_FILES, are laid over it. Everything else, including any new file type, counts as native and forces a build. |
| No partial restore | That cache has no restore-keys: a partial match would hand the tests a browser built from different sources. A self-test checks the key covers every browser-affecting path. |
| Warm ccache | Pushes to main that build, and a twice-weekly schedule, keep the ccache from being evicted; pull requests restore it. A carried push builds nothing, so it caches the pull request's browser under the compiled-input key instead. |
| Carried pushes | A push to main that lands the tree its pull request run tested reuses that run's results (above). |
| PGO and LTO | Every build uses Mozilla's PGO profile for the same Firefox and links libxul with cross-language LTO, as a release does (builds-are-optimized-like-stock). ccache caches the compiles, not that link, so a warm build still pays for it. |
| Network retries only | ci.run_prepare (setup-minimal, dir, mozbootstrap) retries only failures that read as transient network errors. A failed hunk or compile error fails at once. |
Pushing to a branch cancels its running build (cancel-in-progress).
What a release builds
A release builds six targets and a pull request builds one. Every step of a
release's build therefore also runs on a pull request
(one-build-entry-point):
| Pull request | Release | |
|---|---|---|
| Runner setup | build-host |
the same action |
| Source tree | ci.run_prepare |
the same |
| linux x86_64 | ci.run_build: multibuild.py retargets, builds and packages |
the same command |
| The other five targets | ci.run_build --preflight: multibuild.py retargets, then preflight.py configures, runs the pre-export, export and pre-compile tiers and compiles the objects of every patched or added source file |
ci.run_build: the full build |
A preflight does not link or package, so a link error on a cross-compiled
target is still found by a release. Its compiles are cached per target, and a
target that passed is not preflighted again until the browser's compiled
inputs, build-host or the runner scripts change; the twice-weekly scheduled
run preflights every target regardless.
Releases
One workflow, release.yml, makes every
release, on a train: on the 1st and 15th of each month the newest
prerelease is promoted to stable, and a new prerelease is cut from
main. Merges are tested by tests.yml but not released, so users see two
releases a month rather than one per merge. Each library release is paired with
exactly one browser build.
1st and 15th ─► plan ─► build-library ─► promote-browser ─► publish-pypi ─► publish-npm ─► tag-library ─► cut
(schedule) (newest prerelease, paired → stable 0.5.8 0.5.8 @latest v0.5.8 │
no open regression) │
┌──────────────────────────────────────────────────────────────────────────────────────────────────────┘
▼
dispatch on main ─► ubo ─► tests ─► plan ─┬─► build-browser ─► publish-browser ─┐
(AMO's (tests.yml, │ (only if its v156.0.1-beta.N │
newest) unless main's │ sources changed) ├─► publish-pypi ─► publish-npm ─► tag-library
run passed it) │ │
└─► build-library ─────────────────────┘ 0.5.9bN 0.5.9-beta.N v0.5.9bN
(only if what it ships changed) (pip: --pre) @next
git tag v0.5.8 <tested main commit> && git push origin v0.5.8
─► plan ─► build-library ─► promote-browser ─► publish-pypi ─► publish-npm
(checks) paired → stable 0.5.8 0.5.8 @latest
| Rule | Detail |
|---|---|
| The train | The scheduled run promotes the newest library prerelease tag (v0.5.8bN) at the commit it was cut from, as v0.5.8, then its cut job dispatches release.yml on main. With no prerelease since the last promotion it promotes nothing; a cut with nothing changed publishes nothing. |
| A regression holds the train | While an issue labelled regression is open, the promotion fails and says which issues hold it; the cut still runs. The label must exist, so an empty answer means "none open", not "nobody could mark one". |
| Out of cycle | Run the Release workflow by hand on main (gh workflow run release.yml --ref main) to cut a prerelease now. Push a tag to promote by hand. |
| Only a tested commit is released | A cut calls tests.yml on its commit and continues only if it passes. A promotion requires All tests passed on its commit. |
| The browser is built only when its sources changed | ci.browser_inputs.source_digest() hashes the browser's sources. Each browser release carries a manifest.json with that digest, its commit, and the browser interface it speaks (CONSTRAINTS.INTERFACE in python/src/camoufox/__version__.py). A matching release is reused; otherwise plan takes the next unused beta.N. Nothing is committed: ci.release set-build writes the number into the build's tree. |
| uBlock Origin | Every browser package bundles the uBO release pinned in browser/scripts/data/ubo.json under addons/UBO/ (browser/scripts/fetch-ubo.py), and both launchers load it from there. A cut reads the pin once, in the ubo job before tests, and hands it to the tests and every build, so the browser ships the uBO it was tested with. A bump is a reviewed commit (fetch-ubo.py newest prints the candidate). Its manifest.json (ubo) and release notes name it. A reused browser keeps the uBO it was built with. |
| Builds are attested | gh attestation verify <file> --repo daijro/camoufox |
| Packages are built once | publish-pypi and publish-npm upload the files build-library made: 0.5.8b2 on PyPI, 0.5.8-beta.2 on npm. pip install camoufox skips prereleases without --pre; npm installs latest, not next. |
| The pairing | ci.release stamp writes the paired browser release into python/src/camoufox/browser-pin.json, which both launchers read. On main the file is {}, so a development checkout follows its channel. |
| Promotion | Refused unless the commit is on main, All tests passed succeeded on it, X.Y.Z is newer than every release, and a browser release was built from its sources. The paired browser becomes the stable release without a rebuild. |
| Failed publish | Re-run the failed jobs; every version and tag comes from plan. |
| Credentials | None stored. PyPI and npm use trusted publishing (OIDC) from release.yml. |
Releases cut before manifests existed (up to v156.0.1-beta.32) pair by the tag
upstream.sh names.
Running a piece by hand
| Command | Needs a browser | What it runs |
|---|---|---|
python3 -m ci.run_prepare |
make setup-minimal, dir, mozbootstrap |
|
python3 -m ci.run_build |
The browser build and package for linux x86_64; --target, --arch for another |
|
python3 -m ci.run_build --target windows --arch x86_64 --preflight |
Preflight one target | |
python3 -m ci.run_pythonlib |
python/'s tests |
|
python3 -m ci.run_typescript |
typescript/'s tests |
|
python3 -m ci.run_typescript --browser <bin> |
yes | The npm launcher end to end |
python3 -m ci.run_install --tmpdir /mnt/tmpfs --browser <bin> |
yes | Packed packages, a real fetch, and a launch, click and typing through each installed package; the tmpdir must be its own filesystem |
python3 -m ci.run_patch_guards --binary <bin> |
yes | Patch guards; --group spoofing|automation|parity, --only <name> |
python3 -m ci.run_patch_guards --binary <bin> --only stock-gpu-parity |
yes | WebGL and WebGPU against stock Firefox of the same version, downloaded into .ci-work (upgrade step) |
python3 -m ci.run_build_tester --binary <bin> |
yes | build-tester |
python3 -m ci.run_skiplist_audit --binary <bin> |
yes | Every skiplist entry still fails |
python3 -m ci.run_playwright --binary <bin> |
yes | The Playwright suite; --shard 3/6 for one shard |
python3 -m ci.run_native --subset rules |
Settled decisions | |
python3 -m ci.run_native --subset browser --binary <bin> |
yes | Leaks, contexts, crashes |
python3 -m ci.run_native --subset growth --binary <bin> |
yes | Memory growth; --shard 3/7 for one test |
CAMOUFOX_GPU=<pci> python3 -m ci.run_native --subset gpu --binary <bin> |
yes | headless="virtual-gpu" on a real GPU |
python3 -m ci.run_sundial --binary <bin> |
yes | The stealth grade (needs the credential) |
python3 -m ci.summarize --results-dir .ci-work/results |
The summary table | |
GITHUB_EVENT_NAME=schedule python3 -m ci.release target |
What the train would promote | |
python3 -m ci.release browser-plan |
Whether a browser would be built or reused | |
python3 -m ci.release paired |
The release built from this tree | |
python3 -m ci.release lib-plan --channel prerelease |
The next library version | |
python3 browser/scripts/fetch-ubo.py resolve |
The uBlock Origin release a cut would bundle | |
make -C browser stage-ubo |
Bundle it into your build tree's dist/bin, as packaging does |
|
python3 -m pytest ci/tests -q |
The pipeline's own tests |
<bin> is a camoufox-bin; make -C browser path prints the one in your
build tree. Each runner writes one result file to .ci-work/results/
(run_prepare writes none). A required suite with no result file is a
failure, never a skip.
Self-tests
ci/tests/ asserts the pipeline reports honestly: redaction leaks nothing,
skips carry reasons, shards partition exactly once, and version resolution never
picks a suite newer than the browser. They run in the static job.