Four prerelease cuts failed on commits whose tests had passed (runs
41280578561, 43352392915, 43395652803, 43631788944), each in a step only
release.yml ran: midl.exe under Wine (twice), verify-fonts.py, and
make set-target. tests.yml built with `make build`; release.yml built with
multibuild.py after installing its own packages. A step only a release runs
is run for the first time by a release.
- make set-target (scripts/patch.py --mozconfig-only) writes the mozconfig
and nothing else. It used to reset the tree and copy the additions over
it, which in a tree without a git repository undid any patch hunk to an
addition while leaving _READY in place, so the build went on with the
reverted files. The mozconfig it writes is byte for byte the same.
- ci.run_build runs multibuild.py for one target. tests.yml's build job runs
it for linux x86_64 and release.yml's for every target, so the pull
request retargets, builds and packages exactly as a release does.
- multibuild.py --preflight (make preflight, scripts/preflight.py) checks a
target without building it: configure, the pre-export, export and
pre-compile tiers, then the object of every source file a patch or an
addition touches. tests.yml preflights the five targets only a release
builds, beside the build; the gate requires them. A target that passed is
not run again for the same compiled inputs, except by the scheduled run.
- .github/actions/build-host frees the disk, installs the packages and adds
the swap for all three jobs, so no build depends on a package only one of
them installs.
- browser/multibuild.py is a browser input: changing it rebuilds.
The preflight does not link or package, so a link error on a cross-compiled
target is still first seen by a release.
Tests: ci/tests/test_set_target.py fails on main ('as added' != 'as patched');
test_preflight.py covers the source-to-object mapping and that an object left
by an earlier run is compiled again; test_ci.py holds the preflight matrix to
release.yml's and all three jobs to build-host; the tribal rule
one-build-entry-point holds all three to ci.run_build.
Run locally on Firefox 159 with CI's tree (make setup-minimal, make dir):
- All five preflights pass, in 2 to 6.5 minutes each on 16 cores.
- With the four additions as run 43631788944's tree had them, the linux
arm64 preflight fails with that run's error,
SkBitmapProcState_opts.cpp:41:13: use of undeclared identifier
'sk_cpu_emulates_x86'.
- python3 -m ci.run_build --target linux --arch arm64, the release's
command for a target that run never linked, builds and packages
camoufox-159.0-beta.35-lin.arm64.zip in 41 minutes.
No workflow here has run on a GitHub runner yet; this pull request's run is
the first.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
18 KiB
Firefox Patch Upgrading Guide
How to update Camoufox's patches when browser/upstream.sh moves to a new Firefox
version. Patches break because Firefox renames APIs, moves code and shifts line
numbers; this guide covers finding and fixing those rejects.
Table of Contents
- Understanding the Patch System
- The Source Tree and Its Make Targets
- General Workflow
- Fixing Common Reject Types
- Per-Context Machinery
- Porting Playwright
- Testing and Validation
- Pitfalls
Understanding the Patch System
Patch Categories
All patches live under browser/patches/, and browser/scripts/patch.py applies every
*.patch in it (subdirectories included), sorted by file name:
- Playwright:
playwright/0-playwright.patch(Juggler integration) andplaywright/1-leak-fixes.patch. Their names sort first, and every other patch is written against a tree that already has them. - Feature patches:
webrtc-ip-spoofing.patch,anti-font-fingerprinting.patch, etc. Per-user-context (per-Playwright-context) support is built into each one. librewolf/,ghostery/: patches taken from those projects.
Compile-time dependencies between patches (MaskConfig, RoverfoxStorageManager)
are listed in browser/patches/patch-dependencies.md.
Key Infrastructure Files
- RoverfoxStorageManager.cpp/h: Thread-safe key-value storage for per-context data
- Manager Classes: FontListManager, WebRTCIPManager, etc.
- Window.webidl: Exposes the per-context setters to Playwright
The Source Tree and Its Make Targets
The Firefox tree is browser/camoufox-<version>-<release>/ (from
browser/upstream.sh). It is a git repository whose unpatched tag is plain
Firefox plus browser/additions/ and browser/settings/. Run every target, and
every command in this guide that does not say otherwise, from browser/:
| Target | What it does |
|---|---|
make dir |
Fetches and extracts Firefox if the tree is missing. Otherwise resets it to unpatched, runs mach clobber and git clean -fdx (the object directory goes too), re-copies additions, then applies every patch and lists the ones that left rejects. |
make revert |
git reset --hard unpatched. Untracked files stay, including new files that patches created. |
make clean |
mach clobber, git clean -fdx, then make revert: unpatched Firefox with nothing left over, without re-fetching. |
make patch ./patches/x.patch |
Applies one patch (patch -p1). |
make unpatch ./patches/x.patch |
Reverses one patch. |
make first-checkpoint |
Commits the current tree and tags it first-checkpoint. |
make workspace ./patches/x.patch |
Unapplies x if it is applied, runs first-checkpoint, then applies x again, so the working tree differs from the checkpoint by exactly that patch. |
make diff |
git diff first-checkpoint. Redirect it into the patch file. |
make checkpoint |
Commits tracked changes, keeping the first-checkpoint tag where it is. |
make grep <text> |
Searches the top-level patches for a string. |
make build |
./mach build (runs make dir first if the tree was never prepared). |
make set-target |
Rewrites the mozconfig for BUILD_TARGET. The sources and every object directory are left as they are. |
make preflight |
Checks that BUILD_TARGET builds without building it: configure, the export tiers, and the object of every source file a patch or an addition touches. |
make run |
Runs the build with debug on in CAMOU_CONFIG; wipes ~/.camoufox. args="..." passes arguments. |
make path |
Prints the built camoufox-bin. |
make stage-fonts |
Stages the font bundle into the unpackaged build, which tests through the launchers need. |
make tests |
The Playwright suite against the build (headful=true for headed). |
make help |
Lists every target. |
git diff does not show untracked files. Before make diff, mark new files
with git add -N <file> inside the source tree, or they will be missing from
the patch.
General Workflow
Step 1: Bump the Version and Find the Broken Patches
Update version and release in browser/upstream.sh, pin the PGO profiles
Mozilla trained for that release and the Remote Settings collections a page can
see of it, then:
python3 scripts/pgo.py pin
python3 scripts/remote_settings.py pin
make dir
patch.py applies every patch and ends with a list of the ones that failed and
their reject files. It deletes the .rej files after listing them, so reproduce
each failure one patch at a time (Step 2).
Step 2: Set Up One Patch
Start from a clean unpatched tree, apply what the patch builds on (at least the
Playwright patches, plus anything from browser/patches/patch-dependencies.md),
checkpoint, then apply the broken patch. Use make clean rather than
make revert here: files that other patches created survive a revert and make
patch stop on "previously applied" prompts.
make clean
make patch ./patches/playwright/0-playwright.patch
make patch ./patches/playwright/1-leak-fixes.patch
make first-checkpoint
make patch ./patches/patch-name.patch # fails, leaving .rej files
Find the reject files:
cd camoufox-<version>-<release>
find . -name '*.rej' -type f
Step 3: Analyze Each Reject File
Read the reject file to understand what failed:
cat path/to/file.cpp.rej
Reject files show:
@@lines: Line numbers where patch expected to apply-lines: What the patch expected to find (old code)+lines: What the patch wanted to add (new code)
Step 4: Locate the Correct Position in Firefox Code
The line numbers in rejects are usually wrong for the new Firefox version. You need to:
- Search for unique context around the reject
- Understand what the patch is doing
- Find equivalent location in new Firefox code
Step 5: Apply Changes Manually
Edit the file to make the rejected change at the correct location.
Step 6: Remove Reject Files
After fixing all rejects, delete the .rej files and any .orig backups
patch left, so they do not end up in the diff:
find . -name '*.rej' -o -name '*.orig' | xargs rm -f
Step 7: Write the Updated Patch
From browser/:
(cd camoufox-<version>-<release> && git add -N path/to/new/file.cpp) # new files only
make diff > patches/patch-name.patch
Step 8: Verify
Run make dir again. The patch should no longer be listed as failing.
Fixing Common Reject Types
Type 1: Include Directive Rejects
Symptom: Reject shows failed #include additions
Example Reject:
@@ -325,6 +325,7 @@
#include "xpcpublic.h"
+#include "WebRTCIPManager.h"
#include "nsDocShell.h"
How to Fix:
- Read the actual file to find the includes section
- Search for nearby includes (e.g.,
xpcpublic.h) - Add the new include in the appropriate location
- Firefox include order: system headers, then Mozilla headers, alphabetically within groups
Example:
// Find this in the actual file:
#include "xpcpublic.h"
// Add the missing includes after it:
#include "xpcpublic.h"
#include "WebRTCIPManager.h"
#include "nsDocShell.h"
#include "mozilla/OriginAttributes.h"
Type 2: Function Signature Changes
Symptom: Reject shows function call with changed parameters
Example Reject:
- mouseOrPointerEvent.mButton = aButton;
+ mouseOrPointerEvent.mJugglerEventId = aMouseEventData.mJugglerEventId;
Common Causes:
- Firefox refactored the API
- Parameters moved from individual args to struct/data object
- Parameter order changed
How to Fix:
- Search for the function definition in Firefox source
- Understand the new API structure
- Port the patch logic to the new API
Example - Firefox 146 Mouse Event Refactoring:
Old Firefox 144 API (individual parameters):
void SynthesizeMouseEvent(int x, int y, int button, ...)
New Firefox 146 API (structured data):
void SynthesizeMouseEvent(SynthesizeMouseEventData& aData,
SynthesizeMouseEventOptions& aOptions)
Port the patch:
// Old patch code:
mouseEvent.mButton = aButton;
mouseEvent.jugglerEventId = aJugglerEventId;
// New patch code for Firefox 146:
mouseOrPointerEvent.mButton = aMouseEventData.mButton;
mouseOrPointerEvent.mJugglerEventId = aMouseEventData.mJugglerEventId;
mouseOrPointerEvent.convertToPointer = aOptions.mConvertToPointer;
Type 3: Missing Context - Code Moved
Symptom: Reject shows context that doesn't exist in the file
How to Fix:
- Use grep to search for unique function names or variables in the reject
- Find where Firefox moved the code
- Apply the patch to the new location
# Search across the codebase
grep -r "FunctionName" camoufox-<version>/ --include="*.cpp"
Type 4: New Parameter Added to Function Calls
Symptom: Reject shows function call, but Firefox added/removed parameters
Example - MakeTextRun userContextId:
Old call:
MakeTextRun(text, len, drawTarget, appUnitsPerDevPixel, flags, recorder);
New Firefox expects:
MakeTextRun(text, len, drawTarget, appUnitsPerDevPixel, flags, recorder, userContextId);
How to Fix:
- Extract userContextId from available context (Document, PresContext, etc.)
- Add proper extraction code before the call
- Pass userContextId as the last parameter
userContextId extraction: the per-context patches resolve it from the
BrowsingContext, which exists before the document's attributes are set:
uint32_t userContextId = 0;
if (BrowsingContext* bc = win->GetBrowsingContext()) {
userContextId = bc->OriginAttributesRef().mUserContextId;
}
MakeTextRun(..., userContextId);
Workers use WorkerPrivate::GetOriginAttributes() instead.
Type 5: Line Number Shifts (No Code Changes)
Symptom: Reject shows patch tried to apply at wrong line number, but code is identical
How to Fix:
Simply apply the patch manually at the correct line number. The code hasn't changed, just the location.
Per-Context Machinery
Most spoofing patches carry per-context support. When porting one, expect these pieces:
-
Manager classes (e.g., FontListManager, WebRTCIPManager):
- Store per-context settings using RoverfoxStorageManager
- Gate their setter with
RoverfoxStorageManager::IsSetterOfferedandMarkSetterUsed
-
Window.webidl functions:
- JavaScript APIs exposed to Playwright
- Examples:
setFontList(),setWebRTCIPv4()
-
nsGlobalWindowInner.cpp implementations:
- Extract userContextId from window/document/docshell
- Call manager classes
- Remove the setter after use with
RoverfoxStorageManager::RemoveSetter
-
Core logic changes:
- Consult the per-context manager before the global config (MaskConfig)
- Pass userContextId through call chains
See per-context-patches.md for the full list.
Porting Playwright
The two Playwright patches (browser/patches/playwright/)
and Camoufox's Juggler (browser/additions/juggler/) both started from upstream
Playwright and both carry Camoufox changes: fixes to Juggler's input and
navigation paths, the isolated-world page agent, and the human cursor
(input/). Port upstream changes into them; never copy upstream over them.
The Bootstrap Patch
Compare what changed in Playwright's bootstrap.diff since the last sync, then port it like any other patch:
make dir
make workspace ./patches/playwright/0-playwright.patch
# port the upstream changes into camoufox-<version>-<release>/
make diff > patches/playwright/0-playwright.patch
Juggler
git clone https://github.com/microsoft/playwright.git /tmp/playwright
diff -r additions/juggler/ /tmp/playwright/browser_patches/firefox/juggler/
Port the upstream hunks one by one, keeping the Camoufox changes. Then check:
-
components/Juggler.jsstill exportsJugglerFactory. -
components/components.confregisters it withesModule(Firefox no longer supportsjsm) and"constructor": "JugglerFactory":{ "esModule": "resource://gre/modules/juggler/components/Juggler.js", "constructor": "JugglerFactory", } -
Every file upstream added or renamed is listed in
FINAL_TARGET_FILESinjuggler/moz.build, and everyChromeUtils.importESModule()path still exists. Upstream registers ajugglerchrome package injar.mn; Camoufox does not, because a page can detect one (seejuggler/moz.build), so upstream'schrome://juggler/content/paths becomeresource://gre/modules/juggler/.
Then make dir && make build && make tests. A wrong components.conf shows up
as one of these:
| Error | Cause |
|---|---|
undefined symbol: ... mozCreateComponent<nsICommandLineHandler>() at link time |
components.conf is not an ESM component |
Externally-constructed components may not specify 'constructor' or 'legacy_constructor' properties |
It uses jsm instead of esModule |
Externally-constructed components must specify a type other than nsISupports |
It is an external component with no type; make it an ESM component |
JavaScript components must specify a constructor |
The constructor field is missing |
Testing and Validation
Minimal Verification
After updating a patch, always verify:
-
Every patch applies cleanly:
make dirlists no failures. -
Build compiles (if feasible):
make build
Full Testing
Run the suites that cover the patch (see ci/README.md).
From the repository root,
python3 -m ci.run_patch_guards --binary <camoufox-bin> is the most direct
evidence that a patch which still applies was not neutered by the upgrade.
Browser-Owned GPU Values
The launchers build WebGL from fpgen's recordings, which an older Firefox made.
Some values are the browser's rather than the GPU's, and an upgrade can change
them on every device: Firefox 156 added EXT_depth_clamp on Windows and fixed
WebGL1's ALIASED_LINE_WIDTH_RANGE at [1, 1] on Linux
(webgl-browser-owned-values-follow-firefox).
Firefox 158 added EXT_polygon_offset_clamp on Windows devices with WebGL2
(ANGLE's D3D11 backend at feature level 10_0 and above) and on Linux desktop GL,
but not on macOS, where release builds use CGL's GL 4.1. Firefox 159 exposes
EXT_texture_norm16 on WebGL2 wherever the GL has 16-bit normalized textures,
which is every macOS and x86_64 Linux WebGL2 context; on Windows, ANGLE decides
per device. It also adds EXT_clip_control and EXT_texture_mirror_clamp_to_edge
to both contexts on Linux desktop GL and on Windows (clip control from feature
level 10_0), but not on macOS.
The stock-gpu-parity guard finds them. It runs stock Firefox of the new version,
stock Firefox of the release fpgen's model recorded, and Camoufox claiming the
machine's GPU, and fails where the two stock releases differ and Camoufox does
not follow the new one. From the repository root:
python3 -m ci.run_patch_guards --binary <camoufox-bin> --only stock-gpu-parity
CI runs it on Linux with a software renderer. Run it on Windows and macOS too, passing the stock binaries by hand, because each OS has its own graphics backend. From the repository root:
python3 browser/tests/playwright/patches/stock-gpu-parity.py --binary <camoufox-bin> \
--stock <Firefox of upstream.sh's version> --recorded-stock <Firefox of fpgen's release>
For each failure, find in the Firefox source why the value changed and whether
it holds on every device of that backend, then follow it where the launchers
convert a recording (to_config in python/src/camoufox/webgl.py and
typescript/src/webgl.ts).
Pitfalls
| Pitfall | Avoid it by |
|---|---|
| Trusting reject line numbers | Searching for the context lines instead |
Hand-editing a .patch file |
Editing the tree and running make diff |
| Missing new files in the patch | git add -N them before make diff |
| A diff that includes the patch's dependencies | Running make first-checkpoint after applying them and before the patch being fixed |
.rej and .orig files in the diff |
Deleting them first (Step 6) |
| Guessing a value a refactored API now needs | Reading the new Firefox code, or the release notes |
| Fixing several patches in one pass | One patch at a time, each from make clean |
Appendix: Firefox Source Navigation
Finding Files
# Find files by name
find . -name "Navigator.cpp" -type f
# Find files containing a symbol
grep -r "GetAcceptLanguages" . --include="*.cpp"
# Find class definitions
grep -r "class Navigator" . --include="*.h"
Understanding Firefox Code Structure
dom/: DOM implementationdom/base/: Core DOM classes (Window, Document, Navigator, etc.)dom/webidl/: WebIDL interface definitionsdom/media/webrtc/: WebRTC implementation
gfx/: Graphics and font renderinggfx/thebes/: Text rendering (fonts, glyphs, shaping)
layout/: Layout enginelayout/generic/: Text frameslayout/mathml/: MathML rendering
Checklist
- Bump
browser/upstream.sh, runpython3 scripts/pgo.py pinandpython3 scripts/remote_settings.py pin, and runmake dirto list the failing patches - Delete the backports the new Firefox ships
- For each:
make clean, apply its dependencies,make first-checkpoint,make patchit - Port each reject to the new Firefox code
- Remove
.rejand.origfiles,git add -Nnew files make diff > patches/<name>.patchmake dirapplies the whole stack cleanly, with no fuzzmake build, then run the patch guards;firefox-version-pinchecks that the build and every identity are the new Firefox- Run
stock-gpu-parityon Linux, Windows and macOS (Browser-Owned GPU Values)
Additional Resources
- Firefox source: https://searchfox.org/
- Firefox API documentation: https://firefox-source-docs.mozilla.org/
- Mercurial repository: https://hg.mozilla.org/mozilla-central/