From d3d9d73c2dcb5aa8f4b4fac90632fe1b32eec486 Mon Sep 17 00:00:00 2001 From: Jake Writer Date: Fri, 25 Sep 2026 14:20:18 -0600 Subject: [PATCH] docs: one AGENTS.md for every agent, a roadmap, and docs that match the code - AGENTS.md holds the engineering rules for any coding agent, plus the repo map, build, patch and test commands that CLAUDE.md used to carry. CLAUDE.md now only imports it, so there is one set of rules. ci/tribal-rules.yml is the record of settled decisions it points to. - ROADMAP.md lists planned work, each item linked to its issue. - README: - fpgen and the coherence check replace BrowserForge; - the patch workflow uses the make targets instead of the removed developer UI; - letter-spacing noise is described as off by default, as it is. - docs/: - beta-testing-ff146.md removed; - patch-upgrading-guide rewritten around the make targets; - per-context-patches without the canvas patch that no longer exists, and with measured preset counts; - playwright-maintenance without the JSM wrapper that does not exist; - smaller fixes in MEDIA-DEVICES, input-dispatch and FONTS. - ci/README: every job, and the real shard, skiplist and entry-point lists. - pythonlib, tester and patch-dependency READMEs corrected against the code. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 187 +++++++++++++++++++ CLAUDE.md | 107 +---------- CONTRIBUTING.md | 12 +- README.md | 65 +++---- ROADMAP.md | 61 +++++++ build-tester/README.md | 12 +- build-tester/run_tests.sh | 2 +- ci/README.md | 75 +++++--- ci/run_prepare.py | 2 +- docs/FONTS.md | 2 +- docs/MEDIA-DEVICES.md | 6 +- docs/beta-testing-ff146.md | 64 ------- docs/input-dispatch.md | 2 +- docs/patch-upgrading-guide.md | 320 ++++++++++----------------------- docs/per-context-patches.md | 122 +++++-------- docs/playwright-maintenance.md | 147 +++++++-------- patches/patch-dependencies.md | 60 +++++-- patches/playwright/README.md | 2 +- pythonlib/README.md | 23 +-- service-tester/README.md | 16 +- service-tester/run_tests.ps1 | 4 +- service-tester/run_tests.py | 2 +- tests/camoufox/README.md | 6 +- 23 files changed, 633 insertions(+), 666 deletions(-) create mode 100644 AGENTS.md create mode 100644 ROADMAP.md delete mode 100644 docs/beta-testing-ff146.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..6e58c6c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,187 @@ +# AGENTS.md + +Instructions for any coding agent working in this repository: Claude Code, +Codex, Cursor, Copilot, Gemini or anything else. `CLAUDE.md` only imports this +file, so there is one set of rules. Edit them here. + +--- + +## Before you touch anything + +1. **This repository is public.** The professionalism rules below govern every + line, commit message, PR title and comment you write. +2. **Read [`ci/tribal-rules.yml`](ci/tribal-rules.yml).** It lists the decisions + this repo has already made, each with its evidence. Many are checked by + `native-tests/test_tribal_rules.py`. Do not reopen one without new evidence. + When a new decision is argued and settled, add it there with its citation. +3. **Branch off `main`. Never commit or push to `main`.** Every change is a pull + request tied to a GitHub issue ([`CONTRIBUTING.md`](CONTRIBUTING.md)). + +## Before you write a function + +Search for an existing implementation first: `pythonlib/camoufox/`, `ci/`, +`scripts/` and `additions/`. If something close +exists, extend it rather than forking it. Duplicated logic is a serious defect +here, because the copies drift and a fingerprint built from two drifting copies +is detectable. + +## Professionalism + +A public repository is a finished product. People judge the project by what +is in it, and they will not ask what a file was for. + +- **No garbage.** No scratch files, diagnostic scripts, saved test output, + logs, debug logging, commented-out code, dead code, placeholder text, stub + docs, or `TODO`s left as notes to self. +- **Nothing private reaches this repo.** That includes proprietary results, + private repo names, and per-vector stealth detail. `ci/run_sundial.py` reports + a grade and a count; the vectors never leave that module. +- Write commits and PR titles for a stranger reading them in a year. +- **Documentation is never stale.** Update the README, `docs/` and the package + READMEs in the same pull request as the code that changed them, never + "later". Every snippet must run as written. + +## Code + +- **Less code.** More code is a cost, not an achievement. Make minimal, general + changes, and delete dead code when you find it. +- **No band-aids.** Fix the main flow. A hard-coded value or a special case at + the call site is a bug relocated, not fixed. +- **Fallbacks are a last resort.** They turn a loud failure into a silent wrong + answer, and in an anti-detect browser a silent wrong answer is a fingerprint. + Fail loudly instead. +- **Check real data before inventing a value.** Every spoofed value must be + something a real device reports. Take it from the recorded distributions + (fpgen, `pythonlib/camoufox/*.json`), not from memory. +- **Read the provider's docs** before writing against a third-party API or a + Firefox internal. Never infer an endpoint, pref or field from memory. +- **Comments say why, in a sentence or two.** Simple code needs none. A longer + explanation is a decision: put it in `ci/tribal-rules.yml` or `docs/`. + +## Tests + +- **Run the failing test, not the whole suite.** CI runs the full pipeline on + every pull request. +- **Write test output to a log file and grep the file.** Piping a run straight + into `grep` throws away output you will need. Delete the log afterwards. +- **Every bug gets a regression test, in this order:** reproduce it with a test + and confirm the test fails, fix the bug, confirm the test passes, then land + both. +- **No flakes.** An intermittent failure means something is non-deterministic. + Fix that behaviour. Never retry, loosen a threshold or skip the test to get + green. +- **A missing prerequisite fails in CI; it never passes as a skip.** A skip + reads as green, so a job that forgot to install something passes having + tested nothing. +- **When local and CI disagree, name the mechanism and fix it in the repo.** + Typical causes are git-ignored inputs, skipped prerequisites and cold-cache + races. A fresh clone is a sanity check, not a fix. +- **Never write a test just to pass, or code just to pass a test.** + +## Security + +- **Never commit a credential.** That covers code, config, fixtures, logs and + commit messages. CI secrets live in GitHub Actions secrets. +- **Least privilege** for workflows and tokens. Grant `permissions:` per job, + and only what that job needs. +- **Adding a dependency is a decision.** Say why in the pull request, and commit + the lockfile. + +## Working with the maintainers + +- Say when a direction is wrong, before starting, and give the reason. +- Review your own diff the way a strict senior reviewer would. You are biased + toward what you just wrote. +- Explain plainly and briefly. + +## Parallel work + +- **Separate pull requests:** one agent per task, each in its own git worktree, + running at the same time. +- **One pull request with independent slow parts:** use subagents in worktrees. + Merge each part into your branch as it finishes, then delete the worktrees. +- **Shared files or an ordering requirement:** use one agent. + +--- + +# Repository + +## What this is + +Camoufox is an anti-detect Firefox for web scraping and automation. This repo +is **not the Firefox source**. It is a build system that fetches upstream +Firefox, applies `patches/` and copies in `additions/`, and produces the +browser. Fingerprint spoofing happens in C++ and in Juggler, not in injected +JavaScript, so a page cannot see it. + +`upstream.sh` pins `version` and `release`. The `Makefile` sources and exports +it, so every script sees them. The Firefox tree lives in +`camoufox--/`. It is generated: persist a change there as a +patch, never as an edit to that tree. + +| Path | What it is | +|---|---| +| `patches/` | Diffs applied to Firefox (44 top level, plus `playwright/`, `librewolf/`, `ghostery/`). Browser behaviour changes here. | +| `additions/camoucfg/` | The C++ config layer. `MaskConfig.hpp` reads `CAMOU_CONFIG`, which the patches consult. | +| `additions/juggler/` | Camoufox's Juggler, Playwright's Firefox protocol. The page agent runs in an isolated world. `input/` holds the Cursory cursor trajectories. | +| `settings/` | `camoufox.cfg` (prefs), `properties.json` (every config key and its type), `chrome.css`, policies. | +| `scripts/` | `patch.py` applies patches and writes the mozconfig; `copy-additions.sh`, `package.py`, `install-deps.sh`, font tooling. | +| `pythonlib/` | The `camoufox` PyPI package, the reference launcher. It draws identities with [fpgen](https://github.com/scrapfly/fingerprint-generator), checks them with `coherence.py`, and launches the binary. | +| `ci/` | The test pipeline (below). `ci/tribal-rules.yml` holds the settled decisions. | +| `build-tester/`, `tests/`, `native-tests/`, `service-tester/` | Test suites (below). | +| `bundle/` | Font bundle manifests. The fonts themselves are a release asset: `make fonts-extract`. | + +## Building + +The build runs on **Linux**. Windows and macOS binaries are cross-compiled from +it. `mach` needs Python 3.11 or newer. + +```bash +bash scripts/install-deps.sh # host build dependencies +make dir # fetch Firefox, copy additions, apply every patch +make bootstrap # one time: system packages + mach bootstrap +make build # ./mach build +make run # run the build (wipes ~/.camoufox) +make package-linux arch=x86_64 # or package-macos / package-windows +python3 multibuild.py --target linux windows macos --arch x86_64 arm64 +``` + +Install `ccache`. A cold build takes about 40 minutes, and an incremental one +about 5. + +## Changing a patch + +Never hand-edit a `.patch` file. Edit the tree, then write the diff: + +```bash +make dir && make first-checkpoint # a new patch: checkpoint, edit, then +make diff > patches/my-change.patch # (git add -N new files first) + +make dir && make workspace ./patches/x.patch # an existing patch: edit, then +make diff > patches/x.patch +``` + +`make patch` and `make unpatch` apply or reverse one patch. `make revert` resets +the tree to unpatched Firefox. Keep the `Makefile` diff minimal: host +dependencies belong in `scripts/install-deps.sh`. + +## Testing + +`ci/` is the whole pipeline. It runs the same way locally and on a pull request +([`ci/README.md`](ci/README.md)). Branch protection requires one check, **`All +tests passed`**. Run the suite that covers your change while you work: + +| You changed | Run | +|---|---| +| Patches, C++, Juggler | `python3 -m ci.run_build_tester --binary ` (the anti-detect suite) and `python3 -m ci.run_patch_guards --binary ` (one guard per spoofing behaviour) | +| Automation behaviour | `make tests`, the upstream Playwright suite with `ci/skiplist.yml` applied | +| `pythonlib/` | `python3 -m ci.run_pythonlib` | +| `ci/` itself | `python3 -m pytest ci/tests -q` | + +- **`tests/` is not a fork of Playwright's tests.** It holds only `patches/` + and `camoufox/`. A deliberate difference from upstream goes in + `ci/skiplist.yml`, with its reason. +- **Tests against an unpackaged build** need `make stage-fonts` first. Without + it the browser has no content fonts. +- **The stealth grade** (`ci/run_sundial.py`) reports a letter and a count, never + the individual vectors. diff --git a/CLAUDE.md b/CLAUDE.md index 936eb51..43c994c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,106 +1 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## What this is - -Camoufox is an anti-detect fork of Firefox for web scraping and automation. This repo is **not the Firefox source** — it is a *build system* that fetches upstream Firefox, applies a stack of patches + code additions, and produces a hardened, fingerprint-spoofing browser. The distinguishing design choice is that fingerprint spoofing happens at the **C++/Juggler implementation level**, not via injected JavaScript, so it is invisible to page-side inspection. - -The actual Firefox tree lives in `camoufox--/` (e.g. `camoufox-150.0.2-beta.25/`), created by the build. That directory is generated — never edit it directly to make lasting changes; changes there are captured as patches (see "Making patches" below). - -`upstream.sh` pins `version` / `release`, and is sourced+exported by the `Makefile`, so those variables flow into every script. - -## Build commands - -The build system is designed for **Linux**. Windows and macOS binaries are **cross-compiled from Linux** — they are never built natively. (`scripts/install-deps.sh` covers macOS/Linux host dependencies for local `make dir` + bootstrap experimentation; a full production build path is Linux/Docker.) - -```bash -bash scripts/install-deps.sh # install host build deps (Python ≥3.11, Rust, aria2, p7zip, go, msitools, wget, sqlite) -make dir # fetch Firefox source, extract, copy additions/settings, apply all patches → touches _READY -make bootstrap # install system deps (apt/dnf/pacman) + run `mach bootstrap` (one-time) -make build # ./mach build in the source dir -make run # run the built browser (wipes ~/.camoufox profile) -make run args="--headless https://test.com" -python3 multibuild.py --target linux windows macos --arch x86_64 arm64 i686 # full cross-platform build + package -``` - -`make dir` is the pipeline that matters: `setup` (fetch tarball via `aria2c` → extract → `copy-additions.sh`) → `python3 scripts/patch.py` (applies every patch, writes `mozconfig`) → `_READY`. `mach` requires **Python ≥ 3.11** (stdlib `tomllib`); older `python3` crashes with `ModuleNotFoundError: No module named 'tomllib'`. - -Docker is the portable path: `docker build -t camoufox-builder .` then `docker run -v "$(pwd)/dist:/app/dist" camoufox-builder --target --arch `. - -Packaging: `make package-linux|package-macos|package-windows arch=` (wraps `scripts/package.py`). Launcher (Go): `make build-launcher arch= os=`. - -## Working with patches (the core workflow) - -Almost all browser-behavior changes are `patches/*.patch` (~49 patches: `fingerprint-injection.patch`, `webgl-spoofing.patch`, `navigator-spoofing.patch`, `webrtc-ip-spoofing.patch`, the `playwright/` and `librewolf/` and `ghostery/` subdirs, etc.). Do not hand-edit patch files. - -Use the developer UI instead: - -```bash -make edits # launches scripts/developer.py — apply/undo/create/manage patches -``` - -- **New patch:** in the UI "Reset workspace" → edit files in `camoufox-*/` → `make build` / `make run` to test → "Write workspace to patch". -- **Edit existing patch:** "Edit a patch" (resets workspace to that patch's state) → edit → "Write workspace to patch" to overwrite. - -Low-level equivalents: `make patch ./patches/x.patch`, `make unpatch ./patches/x.patch`, `make workspace ./patches/x.patch`, `make revert` (reset to `unpatched` tag), `make diff` (diff against `first-checkpoint`). The source dir is a git repo with `unpatched` / `first-checkpoint` / `checkpoint` tags used by these targets. - -## Repository layout (the parts that require cross-file understanding) - -- **`patches/`** — the diffs applied to Firefox source. This is where browser behavior is changed. -- **`additions/`** — whole files copied *into* the source tree (not diffs) by `scripts/copy-additions.sh`: - - `additions/camoucfg/` — the C++ config layer. `MaskConfig.hpp` reads the spoofing config (from `CAMOU_CONFIG` env var / `camoufox.cfg`) that the patches consult at the C++ level. (The human-cursor algorithm used to live here too; it is now `additions/juggler/input/CursorTrajectory.js` and the vendored Cursory beside it.) - - `additions/juggler/` — Camoufox's patched **Juggler** (Firefox's Playwright automation protocol, the Firefox analog of CDP). This is where Playwright is made undetectable — the page agent runs in an isolated scope so injected automation JS is not visible to the page. -- **`settings/`** — `camoufox.cfg`, `chrome.css`, `properties.json`, `camoucfg.jvv`, prefs/policies. Copied into the source's `lw/` dir by `copy-additions.sh`. Edit the built config with `make edit-cfg`. -- **`scripts/`** — `patch.py` (the patcher, LibreWolf-derived), `developer.py` (the `make edits` UI), `package.py`, `copy-additions.sh`, `install-deps.sh`. -- **`pythonlib/`** — the `camoufox` PyPI package: the Playwright-compatible Python interface that generates + injects fingerprints via [fpgen](https://github.com/scrapfly/fingerprint-generator) and launches the binary. `fingerprint-presets-v150.json` holds real scraped fingerprints; `coherence.py` checks the assembled identity (the pools are sampled independently, so an impossible machine can be built from individually plausible parts), and `scripts/clean-fingerprint-data.py` applies the same rules to the shipped data files. This is the user-facing API; the browser binary is the backend. -- **`jsonvv/`** — JSON-with-validation format library used for `camoucfg.jvv` (config schema). -- **`legacy/launcher/`** — Go launcher binary. -- **`assets/`** — `base.mozconfig` and other build inputs. - -## Testing - -`ci/` is the whole pipeline, and it runs identically locally and on a pull -request — see [`ci/README.md`](ci/README.md). Every gate below must pass before a -PR can merge; the workflow is `.github/workflows/tests.yml`. - -- **`build-tester/`** — the raw binary directly, bypassing the Python package: - eight fingerprint profiles, injected via `generate_context_fingerprint` + - `addInitScript` and `CAMOU_CONFIG`. **This is the anti-detect suite** — run it - when changing patches, C++, or the JS browser layer. - ```bash - python3 -m ci.run_build_tester --binary /path/to/camoufox-bin - ``` -- **`tests/patches/`** — one standalone guard per shipped spoofing behaviour - (isolated evaluate, trusted events, fonts, mouse trajectories, touchscreen). - The most direct evidence a Firefox bump did not neuter a patch that still - *applies* cleanly. - ```bash - python3 -m ci.run_patch_guards --binary /path/to/camoufox-bin - ``` -- **Playwright** — upstream playwright-python, fetched fresh at the tag - `ci/versions.py` resolves for the browser, with `ci/skiplist.yml` applied and - `tests/camoufox/` overlaid. This is the automation-contract check, not the - stealth check. It runs **isolated-world first** (what users ship) and re-runs - only the failures with isolation off; those are counted and named as - main-world fallbacks rather than hidden, so the size of the isolated-world - gap is visible per run. - ```bash - make tests # or: python3 -m ci.run_playwright --binary ... - ``` -- **`native-tests/`** — leaks, context lifetime, and the repo's own conventions. -- **`pythonlib/`**, **`service-tester/`** — the Python package and service layer. -- **stealth grade** — `ci/run_sundial.py` reports a letter grade and a count. - Its per-vector detail never leaves that module, because this repo is public. - -The Playwright suite is **not** a fork: `tests/` holds only `patches/` and -`camoufox/`. Do not vendor upstream tests back into it — a deliberate difference -from upstream belongs in `ci/skiplist.yml` with a stated reason. - -`ccache` is enabled in the build config — install it for fast incremental rebuilds (cold ~40 min, incremental ~5 min). - -## Constraints when editing this repo - -- The `camoufox-*/` source directory is regenerated — persist changes as patches, never as edits committed to that tree. -- Keep the `Makefile` diff clean against `main` unless a change genuinely belongs there — dependency setup lives in `scripts/install-deps.sh`, not the Makefile. -- Every PR must be tied to a GitHub issue and pass the full pipeline (see `CONTRIBUTING.md` and `ci/README.md`). +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 48cd46b..f5ead31 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,6 +9,8 @@ Thanks for your interest in contributing! Here's how to get started. - **Code contributions** — Fork the repo, make your changes, and open a pull request. - **Documentation** — Fixes and improvements to docs are always welcome. +Planned work is in [`ROADMAP.md`](ROADMAP.md). Comment on an item's issue before you start on it. + ## Development Setup See README.md for general setup. For iterative development with frequent rebuilds, install [ccache](https://ccache.dev/) to cache compiled objects: @@ -26,6 +28,8 @@ ccache is already enabled in the build config. A cold build takes the usual ~40 ## Pull Request Rules +The engineering rules in [`AGENTS.md`](AGENTS.md) apply to every change, whether a person or an agent wrote it. They cover less code, no band-aids, no flakes, and docs that ship with the code. + 1. Each pull request must be associated with a Github issue 2. Follow the pull request template 3. Keep commits focused — one logical change per commit. @@ -34,7 +38,7 @@ ccache is already enabled in the build config. A cold build takes the usual ~40 ## Testing Requirements -**CI runs everything, on every pull request.** [`.github/workflows/tests.yml`](.github/workflows/tests.yml) builds the browser from your branch when you touch browser sources (and tests against the published release when you do not), then runs the patch guards, build-tester, the upstream Playwright suite, the leak suite and the stealth check. Branch protection requires exactly one check, **`All tests passed`**, which is green only when every applicable suite is. +**CI runs everything, on every pull request.** [`.github/workflows/tests.yml`](.github/workflows/tests.yml) builds the browser from your branch when you touch browser sources (and tests against the published release when you do not), then runs the Python package tests, the patch guards, build-tester, the upstream Playwright suite, the leak suite and the stealth check. Branch protection requires exactly one check, **`All tests passed`**, which is green only when every applicable suite is. So there is nothing to attach to the pull request by hand. The old process — run the suites locally, screenshot the output, paste it in — was unenforceable: nothing checked that the browser in the screenshot was built from the branch under review. If you want a report in the description anyway, CI leaves one as a comment on the pull request. @@ -45,6 +49,7 @@ python3 -m ci.run_patch_guards --binary /path/to/camoufox-bin python3 -m ci.run_build_tester --binary /path/to/camoufox-bin python3 -m ci.run_playwright --binary /path/to/camoufox-bin # or --shard 3/6 python3 -m ci.run_skiplist_audit --binary /path/to/camoufox-bin +python3 -m ci.run_pythonlib # pythonlib/ python3 -m pytest ci/tests -q # the pipeline's own tests ``` @@ -77,8 +82,7 @@ Tests the **full stack** — the binary and the Python package together — usin ```bash cd service-tester -# Add proxies (one per line, format: user:pass@domain:port) -cp proxies.txt.example proxies.txt # or create manually +# Create proxies.txt: one user:pass@domain:port per line (# comments allowed) ./run_tests.sh ``` @@ -102,7 +106,7 @@ See [`service-tester/README.md`](service-tester/README.md) for full details. Please search existing issues before opening a new one. Include: - Camoufox version -- OS and Python version +- OS, and your Python or Node version - A minimal reproducible example ## Questions diff --git a/README.md b/README.md index be32a7f..16125d8 100644 --- a/README.md +++ b/README.md @@ -362,7 +362,7 @@ Camoufox is a Firefox fork engineered for web scraping and AI agents. It is head * No CSS animations 💨 - Debloated & optimized for memory efficiency ⚡ -- [PyPi package](https://pypi.org/project/camoufox/) for updates & auto fingerprint injection 📦 +- [PyPI package](https://pypi.org/project/camoufox/) for updates & auto fingerprint injection 📦 - Stays up to date with the latest Firefox version 🕓 --- @@ -371,13 +371,13 @@ Camoufox is a Firefox fork engineered for web scraping and AI agents. It is head In Camoufox, data is intercepted at the C++ implementation level, making the changes undetectable through JavaScript inspection. -To spoof individual fingerprint properties, pass a JSON containing properties to spoof to the [Python interface](https://github.com/daijro/camoufox/tree/main/pythonlib#camoufox-python-interface): +To spoof individual fingerprint properties, pass a JSON containing properties to spoof to the [Python interface](pythonlib/): ```py >>> with Camoufox(config={"property": "value"}) as browser: ``` -Config data not set by the user will be automatically populated using [BrowserForge](https://github.com/daijro/browserforge) fingerprints, which mimic the statistical distribution of device characteristics in real-world traffic. +Config data not set by the user is populated from [fpgen](https://github.com/scrapfly/fingerprint-generator), a model of the statistical distribution of device characteristics in real-world traffic. The assembled identity is then checked for coherence, so parts that are each plausible cannot combine into a machine that does not exist. [[See implemented properties](https://camoufox.com/fingerprint/)] @@ -447,7 +447,7 @@ Below is a list of patches and features implemented in Camoufox. - Automatically uses the correct system fonts for your User Agent - Bundled with Windows, Mac, and Linux system fonts -- Prevents font metrics fingerprinting by randomly offsetting letter spacing +- Letter-spacing noise is available (`fonts:spacing_seed`) but off by default, because no real machine produces it ### Playwright support @@ -508,7 +508,9 @@ However, even with hiding its automation library, Camoufox is not immune to inco Anti-bot systems also run client-side scripts to monitor your behavior. For example, they look for patterns in mouse movements, clicks, scrolling, and the timing between actions. - +Cursor paths Camoufox produced with humanize=True, replayed at their recorded speed + +Every dot above is a `mousemove` event the page received from six `page.mouse.move()` calls, replayed at the speed it arrived. Close dots mean the hand slowed down. `scripts/cursor-demo.py` regenerates the figure from a build. Camoufox does not draw its cursor paths. With `humanize=True` it uses [**Cursory**](https://github.com/Vinyzu/cursory) by [Vinyzu](https://github.com/Vinyzu), which holds 2357 mouse movements recorded from real people: it picks a recording whose direction, distance and wander suit the move being made, morphs it onto the requested start and end points, and replays it with that recording's own timing — pauses, overshoots and all. @@ -528,13 +530,13 @@ AI agents need to operate across many sessions without getting flagged or rate-l Even if you are rotating your IP for each running bot instance, web access firewalls can still use machine learning to analyze incoming web traffic to detect if it's abnormal. If the Linux market share was 5%, then suddenly it's 20%, it's a red flag. They will unconditionally require all Linux users to complete a captcha. -Camoufox uses [BrowserForge](https://github.com/daijro/browserforge)'s fingerprint generator to mimic the statistical distribution of device data in real-world traffic. For example, Camoufox will make your browser look like a Linux user 5% of the time. Of that 5%, it will spoof a 2560x1440 screen resolution 9.5% of the time and an Intel HD GPU 27.5% of the time. +Camoufox draws identities from [fpgen](https://github.com/scrapfly/fingerprint-generator), a Bayesian network trained on live traffic, so each device characteristic appears about as often as it does in the real world, and in the combinations real devices produce. ### How can Camoufox be detected? Camoufox can spoof fingerprints with a correct market share. However, **fingerprints must also be internally consistent.** A Windows user agent with an Apple M1 GPU, a MacOS user agent with a Windows DirectX renderer, and a mobile device with a desktop screen resolution are all impossible, and will be flagged for being suspicious. -Of the thousands of possible datapoints that must be changed to create a believable spoofed fingerprint, where each change must be consistent with the others, Camoufox doesn't always succeed. Anti-bot providers test Camoufox over and over again to find even 1 unique inconsistency, then they immediately update their background scripts to test for it. +Every drawn identity passes a coherence check (`pythonlib/camoufox/coherence.py`) that rejects impossible combinations before launch. But of the thousands of datapoints that must agree with each other, Camoufox doesn't always get every one right. Anti-bot providers test Camoufox over and over again to find even 1 unique inconsistency, then they immediately update their background scripts to test for it. --- @@ -550,7 +552,7 @@ Additionally, all injected JavaScript is detectable in some way. Anti-bot system Since Camoufox intercepts calls in the browser's C++ implementation level, all of the hijacked objects and properties appear native. There is no JavaScript hijacking to be detected. -Camoufox also attempts to generate consistent and believable fingerprints with Browserforge as well. However, this can still be detected by complex fingerprint detection methods like mismatching data (as described earlier). +Camoufox also generates consistent and believable fingerprints with fpgen and its coherence check. However, this can still be detected by complex fingerprint detection methods like mismatching data (as described earlier).
@@ -571,7 +573,7 @@ Additionally, Juggler sends its inputs directly through the Firefox's original u

Build System

> [!WARNING] -> The content below is intended for those interested in building & debugging Camoufox. For Playwright usage instructions, see [here](https://github.com/daijro/camoufox/tree/main/pythonlib#camoufox-python-interface). +> The content below is intended for those interested in building & debugging Camoufox. For usage instructions, see [pythonlib](pythonlib/). ### Overview @@ -583,7 +585,7 @@ graph TD subgraph REPO[Camoufox Repository] PATCHES[Fingerprint masking patches] - ADDONS[uBlock & B.P.C.] + ADDONS[uBlock Origin] DEBLOAT[Debloat/optimizations] SYSTEM_FONTS[Win, Mac, Linux fonts] JUGGLER[Patched Juggler] @@ -620,7 +622,7 @@ make dir Before bootstrapping, install the system build dependencies with the helper script. It detects your platform and installs everything the build needs -(Python ≥ 3.11, Rust, `aria2`, `p7zip`, `go`, `msitools`, `wget`, `sqlite`, and +(Python ≥ 3.11, Rust, `aria2`, `p7zip`, `msitools`, `wget`, `sqlite`, and the core build tools) using the appropriate package manager — Homebrew on macOS, or `apt`/`dnf`/`pacman` on Linux: @@ -725,29 +727,27 @@ Build artifacts will now appear written under the `dist/` folder. --- -## Development Tools +## Working on patches -This repo comes with a developer UI under scripts/developer.py: +`make dir` leaves `camoufox--/` as a git repository with every patch applied. A patch is a diff against a checkpoint in that repository: -``` -make edits +```bash +# A new patch +make dir # apply every existing patch +make first-checkpoint # mark the starting point +# ...edit files in camoufox-*/, test with `make build` and `make run`... +make diff > patches/my-change.patch + +# An existing patch +make dir +make workspace ./patches/x.patch # unapply x, checkpoint, reapply x +# ...edit... +make diff > patches/x.patch ``` -Patches can be edited, created, removed, and managed through here. +`make diff` shows only tracked files, so `git add -N ` any new file first. `make workspace` needs every later patch to leave the hunks of `x.patch` alone. `make patch` and `make unpatch` apply or reverse one patch, and `make revert` resets the tree to unpatched Firefox. - - -### How to make a patch - -1. In the developer UI, click **Reset workspace**. -2. Make changes in the `camoufox-*/` folder as needed. You can test your changes with `make build` and `make run`. -3. After you're done making changes, click **Write workspace to patch** and save the patch file. - -### How to work on an existing patch - -1. In the developer UI, click **Edit a patch**. -2. Select the patch you'd like to edit. Your workspace will be reset to the state of the selected patch. -3. After you're done making changes, hit **Write workspace to patch** and overwrite the existing patch file. +Then run the suites that cover what you changed. [`CONTRIBUTING.md`](CONTRIBUTING.md) says which ones, and [`ci/README.md`](ci/README.md) has the whole pipeline. --- @@ -768,12 +768,12 @@ flowchart TD B -->|Yes| C[Likely bad IP/rate-limiting. If the website fails on both headless and headful mode on the official Firefox distribution, the issue is not with the browser.] B -->|No| D["Run make ff-dbg(1) and build(2) a clean distribution of Firefox. Does the website flag in Firefox **headless** mode(4)?"] D -->|Yes| E["Does the website flag in headful mode(3) AND headless mode(4)?"] - D -->|No| F["Open the developer UI(5), apply config.patch, then rebuild(2). Does the website still flag(3)?"] + D -->|No| F["Apply config.patch(5), then rebuild(2). Does the website still flag(3)?"] E -->|No| G["Enable privacy.resistFingerprinting in the config(6). Does the website still flag(3)?"] E -->|Yes| C G -->|No| H["In the config(6), enable FPP and start omitting overrides until you find the one that fixed the leak."] G -->|Yes| I[If you get to this point, you may need to deobfuscate the Javascript behind the website to identify what it's testing.] - F -->|Yes| K["Open the developer UI, apply the playwright bootstrap patch, then rebuild. Does it still flag?"] + F -->|Yes| K["Apply playwright/0-playwright.patch(5), then rebuild. Does it still flag?"] F -->|No| J["Omit options from camoufox.cfg(6) and rerun(3) until you find the one causing the leak."] K -->|No| M[Juggler needs to be debugged to locate the leak.] K -->|Yes| L[The issue has nothing to do with Playwright. Apply the rest of the Camoufox patches one by one until the one causing the leak is found.] @@ -788,7 +788,7 @@ flowchart TD | (2) | `make build` | Build the source code. | | (3) | `make run` | Runs the built browser. | | (4) | `make run args="--headless https://test.com"` | Run a URL in headless mode. All redirects will be printed to the console to determine if the test passed. | -| (5) | `make edits` | Opens the developer UI. Allows the user to apply/undo patches, and see which patches are currently applied. | +| (5) | `make patch ./patches/.patch` | Apply one patch. `make unpatch` reverses it. | | (6) | `make edit-cfg` | Edit camoufox.cfg in the default system editor. | @@ -807,6 +807,7 @@ Web scraping & testing: - [Vinyzu/cursory](https://github.com/Vinyzu/cursory): The recorded human mouse trajectories behind `humanize=True`, vendored via [cursory-js](https://github.com/JWriter20/cursory-js) (LGPLv3-or-later — see `additions/juggler/input/cursory/NOTICE`) - [riflosnake/HumanCursor](https://github.com/riflosnake/HumanCursor): The Bézier cursor algorithm Camoufox used before Cursory +- [scrapfly/fingerprint-generator](https://github.com/scrapfly/fingerprint-generator) (fpgen): The device distribution identities are drawn from - [CreepJS](https://github.com/abrahamjuliot/creepjs), [Browserleaks](https://browserleaks.com), [BrowserScan](https://www.browserscan.net/) - Valuable leak testing sites UI theming: diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..7beecdb --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,61 @@ +# Roadmap + +What is planned, grouped by area. Items link to their issue. There are no +dates: an item ships when its tests prove it. To pick one up, comment on its +issue first. + +## In progress + +- **TypeScript/JavaScript package on npm**, at parity with `pythonlib` + ([#784](https://github.com/daijro/camoufox/issues/784)). +- **Firefox 155** ([#764](https://github.com/daijro/camoufox/issues/764)). + +## Identity + +- **Draw more of the identity from fpgen.** Navigator, screen, window and + headers come from fpgen today. Its fonts, voices, WebGL parameters, + permissions, WebRTC capabilities and audio hashes are still drawn elsewhere. +- **Reproducible identities across relaunches**: the same seed gives the same + device, including its canvas and audio output + ([#442](https://github.com/daijro/camoufox/issues/442), + [#765](https://github.com/daijro/camoufox/issues/765)). +- **Per-context fonts everywhere**: the default context + ([#757](https://github.com/daijro/camoufox/issues/757)), and `measureText` + agreeing with `@font-face local()` + ([#759](https://github.com/daijro/camoufox/issues/759)). +- **Platform-consistent APIs**: + - speech voices ([#717](https://github.com/daijro/camoufox/issues/717)); + - WebAuthn platform-authenticator availability + ([#718](https://github.com/daijro/camoufox/issues/718)); + - audio output devices ([#768](https://github.com/daijro/camoufox/issues/768)); + - favicon caching ([#577](https://github.com/daijro/camoufox/issues/577)). + +## Automation + +- **Routing in the isolated world**: `route_web_socket` + ([#775](https://github.com/daijro/camoufox/issues/775)) and server-sent events + ([#786](https://github.com/daijro/camoufox/issues/786)). +- **Protocol resilience**: interrupting a runaway `evaluate` + ([#720](https://github.com/daijro/camoufox/issues/720)), and recovering from a + dead Juggler pipe ([#719](https://github.com/daijro/camoufox/issues/719)). + +## Platforms + +- Sandboxes without `ARCH_SET_GS`, such as gVisor + ([#740](https://github.com/daijro/camoufox/issues/740)), and read-only + filesystems ([#572](https://github.com/daijro/camoufox/issues/572)). +- **macOS**: headed windows must not take focus when shown + ([#739](https://github.com/daijro/camoufox/issues/739)). +- **Windows**: + - deterministic generic-font resolution at startup + ([#783](https://github.com/daijro/camoufox/issues/783)); + - clean repaint while resizing + ([#734](https://github.com/daijro/camoufox/issues/734)). + +## Testing + +- **Test the packaged browser in CI, langpacks included.** CI tests an + unpackaged build that carries only `en-US`, so no suite can check another + locale end to end. +- **Run the stealth grade on pull requests from forks.** Forks get no secrets + today, so that check is skipped for them. diff --git a/build-tester/README.md b/build-tester/README.md index 960eda1..26aa061 100644 --- a/build-tester/README.md +++ b/build-tester/README.md @@ -4,7 +4,7 @@ Tests a raw Camoufox binary (Firefox) directly against the same antibot-detectio ## Prerequisites -- Python 3.9+ +- Python 3.10+ (what `pythonlib/` requires) - Node.js (for building the TypeScript checks bundle via `esbuild`, first run only) ## Setup @@ -28,6 +28,13 @@ python scripts/run_tests.py [options] python scripts/run_tests.py /path/to/camoufox-bin/camoufox ``` +`./run_tests.sh [options]` does the setup for you: it installs the +npm dependencies and creates `.venv/` with this tree's `pythonlib/` on first +run, then calls `run_tests.py`. It forwards every option below except `--json`. + +In CI the suite runs through `python3 -m ci.run_build_tester --binary ` +from the repository root (see [`ci/README.md`](../ci/README.md)). + ## Options ``` @@ -36,6 +43,7 @@ python scripts/run_tests.py /path/to/camoufox-bin/camoufox --secret KEY HMAC signing key for certificate --save-cert PATH Save certificate text to a file --no-cert Skip certificate generation + --json PATH Write the full machine-readable result tree to PATH ``` ## What It Tests @@ -56,7 +64,7 @@ Each profile is scored across: | Firefox APIs | Firefox-specific API presence | | Cross-Signal | Consistency across navigator, screen, etc. | | CSS Fingerprint | CSS rendering fingerprint | -| Canvas Noise | Canvas hash uniqueness and stability | +| Canvas Noise | Canvas output is identical across renders (no random noise) | | WebGL Render | WebGL rendering hash | | Audio Integrity | AudioContext fingerprint | | Font Platform | OS-consistent font availability | diff --git a/build-tester/run_tests.sh b/build-tester/run_tests.sh index fee02fd..01bd430 100755 --- a/build-tester/run_tests.sh +++ b/build-tester/run_tests.sh @@ -19,7 +19,7 @@ while [[ $# -gt 0 ]]; do ;; -h|--help) echo "Usage: $0 [--profile-count N] [--secret KEY] [--save-cert PATH] [--no-cert]" - echo " e.g. $0 ../camoufox-146.0.1-beta.25/obj-aarch64-apple-darwin/dist/Camoufox.app" + echo " e.g. $0 ../camoufox-152.0.4-beta.31/obj-x86_64-pc-linux-gnu/dist/bin/camoufox-bin" exit 0 ;; -*) diff --git a/ci/README.md b/ci/README.md index c17cfcf..7217ad2 100644 --- a/ci/README.md +++ b/ci/README.md @@ -6,16 +6,17 @@ to test a specific browser version, so there is one definition of "the tests pass", not two. ``` -resolve ──┬─ static ────────── tribal rules, skiplist, self-tests (seconds) - ├─ pythonlib ─────── the package's own tests (a minute) - └─ build ──┬─ playwright × 6 shards (conformance + our own) - ├─ skiplist audit ───── every skip must still fail - ├─ native ───────────── leaks, contexts (ours) - ├─ patch guards ─────── one per spoofing patch - ├─ build-tester ─────── 8 fingerprint profiles - └─ sundial ──────────── stealth grade (off: see below) - │ - summary ──► one comment on the PR +resolve ── static ─────────────── lint, tribal rules, skiplist, self-tests (seconds) + └─ pythonlib ─────── the package's own tests (a minute) + └─ build or fetch ─┬─ patch guards ─────── one per spoofing patch, + skiplist audit + ├─ build-tester ─────── 8 fingerprint profiles + └─ once guards and build-tester pass: + ├─ playwright × 6 shards (conformance + our own) + ├─ native ───────── leaks, contexts, crash recovery + ├─ sundial ──────── stealth grade + └─ growth ───────── memory growth (scheduled only) + │ + summary ──► one comment on the PR ``` ## Which browser, which suite @@ -260,7 +261,7 @@ isolation itself regresses. entry has to claim a test cannot pass in *either* world, or the suite would have counted it as a fallback rather than a failure. -Ten tests are deselected outright by [`ci/skiplist.yml`](skiplist.yml), which +Fourteen tests are deselected outright by [`ci/skiplist.yml`](skiplist.yml), which requires a stated reason per entry — `ci/summarize.py` fails the run on an unreasoned one. @@ -273,19 +274,22 @@ leaving them bare. `ci/run_skiplist_audit.py` now runs every entry with the skiplist disabled and **fails the build if a skipped test passes**. It is cheap precisely because a -correct skiplist is short — ten tests, a few seconds — and it is what keeps the +correct skiplist is short — fourteen tests, a few seconds — and it is what keeps the list from drifting back into a place failing tests go to disappear. ```bash python3 -m ci.run_skiplist_audit --binary /path/to/camoufox-bin ``` -What remains after the audit, 10 tests: two `test_click.py` tests where -Playwright's stable-position wait races the humanized travel time; six -client-certificate tests (async and sync) that need the **browser** to present a -certificate during the TLS handshake — the two that go through the Node driver's -own request context instead pass, and are not skipped; and the two upstream -expectations that encode a stock-Firefox quirk, replaced by `tests/camoufox/`. +What remains, 14 tests: two `test_click.py` tests where Playwright's +stable-position wait races the humanized travel time; two `test_keyboard.py` +tests that assert a shifted character arrives without Shift, which Camoufox +presses as a real keyboard would; six client-certificate tests (async and sync) +that need the **browser** to present a certificate during the TLS handshake — +the two that go through the Node driver's own request context instead pass, and +are not skipped; two upstream expectations that encode a stock-Firefox quirk, +replaced by `tests/camoufox/`; and two popup tests that rely on Playwright +shipping Firefox's popup blocker off, which Camoufox keeps on. That client-certificate split is the audit earning its place. The entry was first written as a whole module, because on a local machine all five fail — @@ -308,6 +312,14 @@ authority for what fails; a local run is a hypothesis.** and per-context injection silently degrades to process-global — which passes every single-context test there is. It has happened here before (commit `d17c887`, "fix screen size leak in contexts"). +- **Crashes.** Kill the browser, the X server, a content process or the driver + mid-run, then check that teardown does not hang, nothing leaks, and a fresh + launch still works (`test_crash_recovery.py`). +- **Memory growth.** Drive one mechanism (iframes, canvas readback, WebGL + contexts, workers, script compilation, font measurement) N and 4N times and + compare the growth: a bounded cost stays flat, a per-iteration leak scales + (`test_memory_growth.py`). It takes over half an hour, so it runs on the + schedule and on demand (the `growth` job, `--subset growth`), not in the gate. - **Settled decisions.** `ci/tribal-rules.yml` lists choices this project already made, each with the issue or PR that made it, and `native-tests/test_tribal_rules.py` asserts them. A comment explaining a @@ -529,8 +541,8 @@ Each tier gates the next, so a two-second lint failure never reaches the build: 1 unit pythonlib ~1 min 2 browser build (patches/additions/settings/assets/upstream.sh/Makefile/scripts changed) fetch (anything else -- driver changes test against the published release) -3a smoke patch guards, build-tester ~15 min -3b full Playwright x2, leaks, stealth ~40 min +3a smoke patch guards, skiplist audit, build-tester ~15 min +3b full Playwright x6, leaks, stealth ~40 min 4 gate the required check ``` @@ -607,16 +619,23 @@ into a slow red build. ## Running a piece by hand ```bash -python3 -m ci.run_playwright --binary path/to/camoufox-bin -python3 -m ci.run_playwright --binary path/to/camoufox-bin --shard 3/6 -python3 -m ci.run_native --subset rules # no browser needed -python3 -m ci.run_native --subset browser --binary path/to/camoufox-bin -python3 -m ci.run_sundial --binary path/to/camoufox-bin -python3 -m ci.summarize --results-dir .ci-work/results +python3 -m ci.run_prepare # make setup-minimal, dir, mozbootstrap +python3 -m ci.run_build +python3 -m ci.run_pythonlib # no browser needed +python3 -m ci.run_patch_guards --binary path/to/camoufox-bin +python3 -m ci.run_build_tester --binary path/to/camoufox-bin +python3 -m ci.run_skiplist_audit --binary path/to/camoufox-bin +python3 -m ci.run_playwright --binary path/to/camoufox-bin +python3 -m ci.run_playwright --binary path/to/camoufox-bin --shard 3/6 +python3 -m ci.run_native --subset rules # no browser needed +python3 -m ci.run_native --subset browser --binary path/to/camoufox-bin +python3 -m ci.run_native --subset growth --binary path/to/camoufox-bin +python3 -m ci.run_sundial --binary path/to/camoufox-bin +python3 -m ci.summarize --results-dir .ci-work/results ``` -Each writes one result file to `.ci-work/results/`. `ci/summarize.py` folds the -shards, decides, and renders the table. A required suite that produced no result +Each suite runner writes one result file to `.ci-work/results/` (`run_prepare` +writes none). `ci/summarize.py` folds the shards, decides, and renders the table. A required suite that produced no result file is a **failure**, never a skip — otherwise deleting a job would be the cheapest way to a green tick. diff --git a/ci/run_prepare.py b/ci/run_prepare.py index 3608b1a..9d8925b 100644 --- a/ci/run_prepare.py +++ b/ci/run_prepare.py @@ -16,7 +16,7 @@ failures that look transient: a compile error or a patch that will not apply must still fail on the first try, because retrying those only wastes a runner. The retry lives here rather than in the Makefile so the Makefile diff stays -clean against upstream (see CLAUDE.md) and so the auto-update harness gets the +clean against upstream (see AGENTS.md) and so the auto-update harness gets the same behaviour without duplicating it in a workflow. Run: diff --git a/docs/FONTS.md b/docs/FONTS.md index dbb65de..1a30ac0 100644 --- a/docs/FONTS.md +++ b/docs/FONTS.md @@ -214,7 +214,7 @@ and `font-hijacker.patch` does not activate the bundle. On a Windows host the Win11 marker families are *subtracted* when the host cannot render them, rather than added when it can; claiming a marker the host lacks is the leak. -`pythonlib/tests/test_font_distribution.py` (29 tests) covers this: base +`pythonlib/tests/test_font_distribution.py` covers this: base completeness and weights, per-unit probability, bundle atomicity, à-la-carte sizing, locale gating, determinism, renderable-only, and that the draw actually varies (distinct lists, no single list dominating). Tolerances are binomial, at diff --git a/docs/MEDIA-DEVICES.md b/docs/MEDIA-DEVICES.md index e82bd32..ebdddb1 100644 --- a/docs/MEDIA-DEVICES.md +++ b/docs/MEDIA-DEVICES.md @@ -24,8 +24,10 @@ Camoufox reproduces exactly that for a spoofed machine: - `getUserMedia()` therefore succeeds iff the identity has the requested device kind (a claimed camera captures the fake engine's test pattern; a camera-less identity gets `NotFoundError`, like a real machine without one). -- `media.navigator.permission.fake = true` (camoufox.cfg) makes the fake - devices count as capturing, which is what exposes labels after a grant. +- The patch treats the identity's devices (`LocalMediaDevice::IsIdentityDevice()`) + as real hardware, so they get the permission prompt, count as capturing + and expose their labels after a grant. `media.navigator.permission.fake` + stays off, as in stock Firefox, because a page can detect it. ## Config keys diff --git a/docs/beta-testing-ff146.md b/docs/beta-testing-ff146.md deleted file mode 100644 index fc8810f..0000000 --- a/docs/beta-testing-ff146.md +++ /dev/null @@ -1,64 +0,0 @@ -# Testing Firefox 146 Beta - -This guide explains how to test the experimental Firefox 146 build of Camoufox. - -> **Note:** The FF146 build is experimental and may contain bugs. For a stable production version, use branch `releases/135`. - -## Build from Source - -1. Clone the repository: -```bash -git clone --depth 1 https://github.com/daijro/camoufox -cd camoufox -``` - -2. Set up the build environment: -```bash -make dir -make bootstrap # only needed once -``` - -3. Build for your target platform: -```bash -python3 multibuild.py --target --arch -``` - -| Parameter | Options | -|-----------|---------| -| `--target` | `linux`, `windows`, `macos` | -| `--arch` | `x86_64`, `arm64`, `i686` | - -Build artifacts will appear in the `dist/` folder. - -### Default Install Directories - -When using the Python library (`camoufox fetch`), the default install directory is: - -| OS | Install Directory | -|------|-------------------| -| **Linux** | `~/.cache/camoufox/` | -| **macOS** | `~/Library/Caches/camoufox/` | -| **Windows** | `C:\Users\\AppData\Local\camoufox\camoufox\Cache\` | - -## Replacing the Binary - -To test FF146 with an existing Camoufox installation: - -1. Build from source using the instructions above -2. Extract the built zip from `dist/` -3. Replace the binary at the corresponding path for your OS: - -**Linux:** -```bash -cp /path/to/built/camoufox-bin ~/.cache/camoufox/camoufox-bin -``` - -**macOS:** -```bash -cp /path/to/built/Camoufox.app ~/Library/Caches/camoufox/Camoufox.app -``` - -**Windows:** -```powershell -copy C:\path\to\built\camoufox.exe C:\Users\\AppData\Local\camoufox\camoufox\Cache\camoufox.exe -``` diff --git a/docs/input-dispatch.md b/docs/input-dispatch.md index e31ac00..cabaac8 100644 --- a/docs/input-dispatch.md +++ b/docs/input-dispatch.md @@ -75,7 +75,7 @@ reachable from the same slot (`apz-repaints-flushed`, `TabSwitchDone`, the drag path's waits), none of which has failed yet. **The static check.** `scripts/check-input-dispatch.py`, wired into -`.github/workflows/lint.yml`. Two exemptions, both content-process: +the `static` job of `.github/workflows/tests.yml`. Two exemptions, both content-process: `PageAgent.js` (drag events, already content-relative, no ack) and `FrameTree.js` (the ack *producer*). diff --git a/docs/patch-upgrading-guide.md b/docs/patch-upgrading-guide.md index 457b145..33a10ee 100644 --- a/docs/patch-upgrading-guide.md +++ b/docs/patch-upgrading-guide.md @@ -1,16 +1,16 @@ -# Firefox Patch Upgrading Guide for LLMs +# Firefox Patch Upgrading Guide -This guide provides step-by-step instructions for updating Camoufox patches when upgrading Firefox versions. Patches frequently break due to Firefox API changes, file reorganizations, and line number shifts. - -**All patches are located in the `patches/` directory.** There are no separate context patches to merge—per-context functionality is already built into the patches. +How to update Camoufox's patches when `upstream.sh` moves to a new Firefox +version. Patches break because Firefox renames APIs, moves code and shifts line +numbers; this guide covers finding and fixing those rejects. ## Table of Contents 1. [Understanding the Patch System](#understanding-the-patch-system) -2. [Preparation](#preparation) +2. [The Source Tree and Its Make Targets](#the-source-tree-and-its-make-targets) 3. [General Workflow](#general-workflow) 4. [Fixing Common Reject Types](#fixing-common-reject-types) -5. [Context Patch Merging](#context-patch-merging) *(Historical - skip for future updates)* +5. [Per-Context Machinery](#per-context-machinery) 6. [Testing and Validation](#testing-and-validation) 7. [Best Practices](#best-practices) @@ -20,66 +20,89 @@ This guide provides step-by-step instructions for updating Camoufox patches when ### Patch Categories -All Camoufox patches are in the `patches/` directory: +All patches live under `patches/`, and `scripts/patch.py` applies every +`*.patch` in it (subdirectories included), sorted by file name: -- **Core Patches**: `0-playwright.patch`, `1-leak-fixes.patch`, etc. -- **Feature Patches**: `webrtc-ip-spoofing.patch`, `anti-font-fingerprinting.patch`, etc. -- All patches now include per-user-context (per-Playwright-context) support built-in +- **Playwright**: `playwright/0-playwright.patch` (Juggler integration) and + `playwright/1-leak-fixes.patch`. Their names sort first, and every other + patch is written against a tree that already has them. +- **Feature patches**: `webrtc-ip-spoofing.patch`, + `anti-font-fingerprinting.patch`, etc. Per-user-context (per-Playwright-context) + support is built into each one. +- **`librewolf/`, `ghostery/`**: patches taken from those projects. -**Historical Note**: Context patches (e.g., `font-fingerprinting.context.patch`, `webrtc.context.patch`) were previously separate but have been merged into their base patches as of Firefox 146. You will not find `.context.patch` files in the repository. +Compile-time dependencies between patches (MaskConfig, RoverfoxStorageManager) +are listed in [`patches/patch-dependencies.md`](../patches/patch-dependencies.md). ### Key Infrastructure Files - **RoverfoxStorageManager.cpp/h**: Thread-safe key-value storage for per-context data - **Manager Classes**: FontSpacingSeedManager, WebRTCIPManager, etc. -- **Window.webidl**: Exposes JavaScript APIs to Playwright +- **Window.webidl**: Exposes the per-context setters to Playwright --- -## Preparation +## The Source Tree and Its Make Targets -### 1. Reset to Clean State +The Firefox tree is `camoufox--/` (from `upstream.sh`). It is +a git repository whose `unpatched` tag is plain Firefox plus `additions/` and +`settings/`. Run every target from the repository root: -**IMPORTANT**: Always use `make clean` to reset to fresh Firefox source: +| Target | What it does | +|---|---| +| `make dir` | Fetches and extracts Firefox if the tree is missing. Otherwise resets it to `unpatched`, runs `mach clobber` and `git clean -fdx` (the object directory goes too), re-copies additions, then applies every patch and lists the ones that left rejects. | +| `make revert` | `git reset --hard unpatched`. Untracked files stay, including new files that patches created. | +| `make clean` | `mach clobber`, `git clean -fdx`, then `make revert`: unpatched Firefox with nothing left over, without re-fetching. | +| `make patch ./patches/x.patch` | Applies one patch (`patch -p1`). | +| `make unpatch ./patches/x.patch` | Reverses one patch. | +| `make first-checkpoint` | Commits the current tree and tags it `first-checkpoint`. | +| `make workspace ./patches/x.patch` | Unapplies `x` if it is applied, runs `first-checkpoint`, then applies `x` again, so the working tree differs from the checkpoint by exactly that patch. | +| `make diff` | `git diff first-checkpoint`. Redirect it into the patch file. | -```bash -make clean -``` - -**DO NOT** use `git reset` or `git clean` commands directly in the Firefox source directory - these can delete untracked files needed for the build. - -### 2. Identify Patches to Update - -Check which patches exist: - -```bash -ls patches/*.patch -``` - -### 3. Understand Patch Dependencies - -Some patches depend on others being applied first: -- `1-leak-fixes.patch` requires `0-playwright.patch` -- Check the Makefile or patch comments for dependency chains +`git diff` does not show untracked files. Before `make diff`, mark new files +with `git add -N ` inside the source tree, or they will be missing from +the patch. --- ## General Workflow -### Step 1: Apply Base Patch and Identify Rejects +### Step 1: Bump the Version and Find the Broken Patches + +Update `version` and `release` in `upstream.sh`, then: ```bash -cd camoufox- -patch -p1 < ../patches/patch-name.patch +make dir ``` -Find reject files: +`patch.py` applies every patch and ends with a list of the ones that failed and +their reject files. It deletes the `.rej` files after listing them, so reproduce +each failure one patch at a time (Step 2). + +### Step 2: Set Up One Patch + +Start from a clean unpatched tree, apply what the patch builds on (at least the +Playwright patches, plus anything from `patches/patch-dependencies.md`), +checkpoint, then apply the broken patch. Use `make clean` rather than +`make revert` here: files that other patches created survive a revert and make +`patch` stop on "previously applied" prompts. ```bash +make clean +make patch ./patches/playwright/0-playwright.patch +make patch ./patches/playwright/1-leak-fixes.patch +make first-checkpoint +make patch ./patches/patch-name.patch # fails, leaving .rej files +``` + +Find the reject files: + +```bash +cd camoufox-- find . -name '*.rej' -type f ``` -### Step 2: Analyze Each Reject File +### Step 3: Analyze Each Reject File Read the reject file to understand what failed: @@ -92,7 +115,7 @@ Reject files show: - `-` lines: What the patch expected to find (old code) - `+` lines: What the patch wanted to add (new code) -### Step 3: Locate the Correct Position in Firefox Code +### Step 4: Locate the Correct Position in Firefox Code The line numbers in rejects are usually wrong for the new Firefox version. You need to: @@ -100,42 +123,31 @@ The line numbers in rejects are usually wrong for the new Firefox version. You n 2. **Understand what the patch is doing** 3. **Find equivalent location** in new Firefox code -### Step 4: Apply Changes Manually +### Step 5: Apply Changes Manually -Use the Edit tool to apply the rejected changes to the correct location. +Edit the file to make the rejected change at the correct location. -### Step 5: Remove Reject Files +### Step 6: Remove Reject Files -After fixing all rejects: +After fixing all rejects, delete the `.rej` files and any `.orig` backups +`patch` left, so they do not end up in the diff: ```bash -rm -f path/to/file.cpp.rej -find . -name '*.rej' -type f # Verify all removed +find . -name '*.rej' -o -name '*.orig' | xargs rm -f ``` -### Step 6: Generate Updated Patch +### Step 7: Write the Updated Patch + +From the repository root: ```bash -# Add any new files first -git add new/file.cpp new/file.h - -# Generate patch with both staged and unstaged changes -git diff --cached --binary > /tmp/patch-name.patch -git diff --binary >> /tmp/patch-name.patch - -# Copy to patches directory -cp /tmp/patch-name.patch ../patches/patch-name.patch +(cd camoufox-- && git add -N path/to/new/file.cpp) # new files only +make diff > patches/patch-name.patch ``` -### Step 7: Verify Patch Applies Cleanly +### Step 8: Verify -```bash -cd .. -make clean -cd camoufox- -patch -p1 < ../patches/patch-name.patch -find . -name '*.rej' -type f # Should return nothing -``` +Run `make dir` again. The patch should no longer be listed as failing. --- @@ -285,68 +297,10 @@ Simply apply the patch manually at the correct line number. The code hasn't chan --- -## Context Patch Merging (Historical - Not Applicable for Future Updates) +## Per-Context Machinery -**NOTE**: As of Firefox 146, all context patches have been merged into their base patches. This section is kept for historical reference and understanding how the patches evolved. Future Firefox updates will only need to update patches in the `patches/` directory. - ---- - -**Historical Context**: Context patches previously added per-user-context functionality to base patches. The workflow was different from simple patch updates. - -### Historical Goal - -Merge all changes from `*.context.patch` into the corresponding base patch so there's only one comprehensive patch file. - -### Historical Example: font-fingerprinting.context.patch → anti-font-fingerprinting.patch - -### Workflow - -1. **Reset to clean Firefox**: - ```bash - make clean - ``` - -2. **Apply base patch first**: - ```bash - cd camoufox- - patch -p1 < ../patches/anti-font-fingerprinting.patch - ``` - -3. **Apply context patch on top**: - ```bash - patch -p1 < ../font-fingerprinting.context.patch - ``` - -4. **Fix any rejects** (usually include conflicts since base patch may have some overlapping changes) - -5. **Generate combined patch**: - ```bash - # Add new files (e.g., FontSpacingSeedManager.cpp/h) - git add dom/base/FontSpacingSeedManager.cpp - git add dom/base/FontSpacingSeedManager.h - git add dom/base/RoverfoxStorageManager.cpp - git add dom/base/RoverfoxStorageManager.h - - # Generate combined patch - git diff --cached --binary > /tmp/anti-font-fingerprinting.patch - git diff --binary >> /tmp/anti-font-fingerprinting.patch - - # Replace base patch - cp /tmp/anti-font-fingerprinting.patch ../patches/anti-font-fingerprinting.patch - ``` - -6. **Verify combined patch**: - ```bash - cd .. - make clean - cd camoufox- - patch -p1 < ../patches/anti-font-fingerprinting.patch - find . -name '*.rej' -type f # Should be empty - ``` - -### What Context Patches Add - -Context patches typically add: +Most spoofing patches carry per-context support. When porting one, expect these +pieces: 1. **Manager classes** (e.g., FontSpacingSeedManager, WebRTCIPManager): - Store per-context settings using RoverfoxStorageManager @@ -363,9 +317,10 @@ Context patches typically add: - Self-destruct logic (remove function after first use) 4. **Core logic changes**: - - Replace global config (MaskConfig) with per-context manager + - Consult the per-context manager before the global config (MaskConfig) - Pass userContextId through call chains - - Query manager for per-context values + +See [`per-context-patches.md`](per-context-patches.md) for the full list. --- @@ -375,25 +330,18 @@ Context patches typically add: After updating a patch, always verify: -1. **Patch applies cleanly**: - ```bash - make clean - cd camoufox- - patch -p1 < ../patches/patch-name.patch - find . -name '*.rej' -type f - ``` +1. **Every patch applies cleanly**: `make dir` lists no failures. -2. **No reject files remain** - -3. **Build compiles** (if feasible): +2. **Build compiles** (if feasible): ```bash - cd .. make build ``` ### Full Testing -For critical patches, test with actual Playwright scenarios after building. +Run the suites that cover the patch (see [`ci/README.md`](../ci/README.md)): +`python3 -m ci.run_patch_guards --binary ` is the most direct +evidence that a patch which still applies was not neutered by the upgrade. --- @@ -401,18 +349,18 @@ For critical patches, test with actual Playwright scenarios after building. ### DO: -1. ✅ **Always use `make clean`** to reset Firefox source +1. ✅ **Start each patch from `make clean`** plus the patches it builds on 2. ✅ **Read and understand** what the patch is trying to do before fixing rejects 3. ✅ **Search for API changes** in Firefox release notes when functions have changed 4. ✅ **Use grep/search** extensively to find where code moved 5. ✅ **Extract userContextId properly** using the standard pattern -6. ✅ **Test patches apply cleanly** before considering them done +6. ✅ **Check with `make dir`** that the whole stack applies before considering a patch done 7. ✅ **Keep commits atomic** - one patch fix per session 8. ✅ **Document major API changes** you discover ### DON'T: -1. ❌ **Don't use `git reset` or `git clean`** on Firefox source directory +1. ❌ **Don't hand-edit `.patch` files** - edit the tree and regenerate with `make diff` 2. ❌ **Don't leave TODO comments** - fix things properly as you go 3. ❌ **Don't guess parameter values** - extract them properly or investigate 4. ❌ **Don't skip verification** - always test the patch applies cleanly @@ -424,75 +372,9 @@ For critical patches, test with actual Playwright scenarios after building. 1. **Assuming reject line numbers are accurate**: They're usually wrong in new Firefox versions 2. **Not understanding API changes**: Firefox refactors often - read the new code -3. **Forgetting to add new files**: Use `git add` before generating patch -4. **Not testing on clean source**: Always verify with `make clean` -5. **Leaving reject files**: Remove all `.rej` files after fixing - ---- - -## Example: Complete Patch Update Session - -Here's a complete example of updating `0-playwright.patch` from Firefox 144 to Firefox 146: - -### 1. Reset and Apply - -```bash -make clean -cd camoufox-146.0.1-beta.25 -patch -p1 < ../patches/0-playwright.patch -``` - -### 2. Find Rejects - -```bash -find . -name '*.rej' -type f -``` - -Output shows 20 reject files. - -### 3. Analyze First Reject - -```bash -cat dom/base/Navigator.cpp.rej -``` - -Shows parameter order changed in `GetAcceptLanguages`. - -### 4. Fix the Reject - -Search for the function in the actual file, understand the new signature, apply changes manually. - -### 5. Repeat for All Rejects - -Work through each reject systematically. - -### 6. Discover API Change - -Firefox 146 refactored mouse events from individual parameters to `SynthesizeMouseEventData` and `SynthesizeMouseEventOptions`. Port all mouse event logic to new API. - -### 7. Remove Rejects - -```bash -rm -f dom/base/Navigator.cpp.rej dom/base/Element.cpp.rej ... -find . -name '*.rej' -type f # Verify empty -``` - -### 8. Generate New Patch - -```bash -git diff --binary > /tmp/0-playwright.patch -cp /tmp/0-playwright.patch ../patches/0-playwright.patch -``` - -### 9. Verify - -```bash -cd .. -make clean -cd camoufox-146.0.1-beta.25 -patch -p1 < ../patches/0-playwright.patch -find . -name '*.rej' -type f # Should be empty -``` +3. **Forgetting new files**: `git diff` skips untracked files; `git add -N` them before `make diff` +4. **Diffing against the wrong base**: `make first-checkpoint` before applying the patch you are fixing, or `make diff` will include its dependencies +5. **Leaving reject files**: Remove all `.rej` and `.orig` files after fixing --- @@ -570,17 +452,17 @@ if (userContextId == 0) { When updating patches for a new Firefox version: -- [ ] Use `make clean` to reset to fresh Firefox source -- [ ] Apply patch and identify all reject files +- [ ] Bump `upstream.sh` and run `make dir` to list the failing patches +- [ ] For each: `make clean`, apply its dependencies, `make first-checkpoint`, `make patch` it - [ ] Analyze each reject to understand what changed - [ ] Search Firefox source for moved/refactored code - [ ] Fix rejects by porting logic to new Firefox APIs - [ ] Extract userContextId properly using standard patterns - [ ] Don't leave TODO comments - fix everything immediately -- [ ] Remove all `.rej` files after fixing -- [ ] Add any new files with `git add` -- [ ] Generate new patch with `git diff --cached --binary` + `git diff --binary` -- [ ] Verify patch applies cleanly to fresh source +- [ ] Remove all `.rej` and `.orig` files after fixing +- [ ] `git add -N` any new files +- [ ] `make diff > patches/.patch` +- [ ] `make dir` applies the whole stack cleanly - [ ] Document any major API changes discovered --- @@ -590,7 +472,3 @@ When updating patches for a new Firefox version: - Firefox source: https://searchfox.org/ - Firefox API documentation: https://firefox-source-docs.mozilla.org/ - Mercurial repository: https://hg.mozilla.org/mozilla-central/ - ---- - -**Last Updated**: December 2025 (Firefox 146 upgrade) diff --git a/docs/per-context-patches.md b/docs/per-context-patches.md index 27bef41..277016f 100644 --- a/docs/per-context-patches.md +++ b/docs/per-context-patches.md @@ -2,24 +2,24 @@ Camoufox spoofs fingerprints globally via `CAMOU_CONFIG` — every browser context shares the same identity. These patches add **per-context isolation**, so each Playwright context can have a unique, deterministic fingerprint. This lets you run multiple concurrent sessions from a single Camoufox process without cross-context correlation. -### What's New +### The Patches -**New patches (8):** +**Per-context patches (with a `window.setXxx()` API):** +- `anti-font-fingerprinting.patch` — per-context `measureText()` spacing seed; also adds `RoverfoxStorageManager` (the shared per-context store) and puts the userContextId in `WordCacheKey` so the glyph cache never serves one context's result to another - `audio-fingerprint-manager.patch` — per-context audio fingerprint seeding (all 6 AudioBuffer + AnalyserNode methods) - `timezone-spoofing.patch` — true per-realm timezone isolation via SpiderMonkey DateTimeInfo +- `screen-spoofing.patch` — per-context screen dimensions and color depth via `ScreenDimensionManager` - `navigator-spoofing.patch` — per-context platform, oscpu, hardwareConcurrency, userAgent +- `webrtc-ip-spoofing.patch` — per-context WebRTC IP, including `getStats()` sanitization and IPv6 - `webgl-spoofing.patch` — per-context UNMASKED_VENDOR/RENDERER_WEBGL -- `canvas-spoofing.patch` — per-context canvas 2D fingerprint noise - `font-list-spoofing.patch` — per-context installed font list filtering via thread-local propagation - `speech-voices-spoofing.patch` — per-context `speechSynthesis.getVoices()` filtering -- `cross-process-storage.patch` — IPDL message for content-to-parent pref writes, enabling cross-process fingerprint storage -**Enhanced existing patches (5):** -- `anti-font-fingerprinting.patch` — added `RoverfoxStorageManager` (cross-process Preferences-based storage), `WordCacheKey` fix (userContextId in glyph cache to prevent cross-context cache hits), random font subset generation -- `screen-spoofing.patch` — replaces old `screen-hijacker.patch` with full per-context support via `ScreenDimensionManager` -- `webrtc-ip-spoofing.patch` — added `getStats()` API sanitization, per-context IP storage, comprehensive IPv6 regex -- `geolocation-spoofing.patch` — updated for Firefox 146, fixed malformed hunks and moz.build line offsets -- `locale-spoofing.patch` — updated for Firefox 146 compatibility +**Infrastructure:** +- `cross-process-storage.patch` — IPDL messages for content-to-parent storage writes, so per-context values reach every process + +There is no canvas pixel noise: Camoufox leaves `toDataURL()`/`getImageData()` +output as the GPU and fonts produce it. ## Quick Reference @@ -38,11 +38,10 @@ Camoufox spoofs fingerprints globally via `CAMOU_CONFIG` — every browser conte | `window.setWebRTCIPv6(ip)` | `webrtc-ip-spoofing.patch` | WebRTC IPv6 addresses | | `window.setWebGLVendor(vendor)` | `webgl-spoofing.patch` | `UNMASKED_VENDOR_WEBGL` parameter | | `window.setWebGLRenderer(renderer)` | `webgl-spoofing.patch` | `UNMASKED_RENDERER_WEBGL` parameter | -| `window.setCanvasSeed(seed)` | `canvas-spoofing.patch` | Canvas 2D `toDataURL()`/`getImageData()` hash | | `window.setFontList(fonts)` | `font-list-spoofing.patch` | Which fonts appear "installed" to fingerprinters | | `window.setSpeechVoices(voices)` | `speech-voices-spoofing.patch` | `speechSynthesis.getVoices()` filtering | -All 16 functions **self-destruct after the first call** — page JavaScript cannot detect them via `typeof window.setTimezone`. +All 15 functions **self-destruct after the first call** — page JavaScript cannot detect them via `typeof window.setTimezone`. --- @@ -101,9 +100,6 @@ await context.addInitScript((values) => { if (typeof w.setWebGLRenderer === 'function') { w.setWebGLRenderer(values.webglRenderer); } - if (typeof w.setCanvasSeed === 'function') { - w.setCanvasSeed(values.canvasSeed); - } if (values.fontList && values.fontList.length > 0 && typeof w.setFontList === 'function') { w.setFontList(values.fontList.join(',')); } @@ -121,10 +117,9 @@ await context.addInitScript((values) => { navigatorPlatform: 'MacIntel', navigatorOscpu: 'Intel Mac OS X 10.15', hardwareConcurrency: 8, - userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0', + userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0', webglVendor: 'Intel Inc.', webglRenderer: 'Intel Iris OpenGL Engine', - canvasSeed: 55555555, fontList: ['Arial', 'Helvetica', 'Georgia', 'Courier New', 'Verdana', 'Times New Roman'], speechVoices: 'Microsoft David,Microsoft Zira,Google US English', }); @@ -147,9 +142,8 @@ await ctxA.addInitScript((v) => { if (typeof window.setTimezone === 'function') window.setTimezone(v.tz); if (typeof window.setAudioFingerprintSeed === 'function') window.setAudioFingerprintSeed(v.audio); if (typeof window.setScreenDimensions === 'function') window.setScreenDimensions(v.sw, v.sh); - if (typeof window.setCanvasSeed === 'function') window.setCanvasSeed(v.canvas); if (typeof window.setWebGLRenderer === 'function') window.setWebGLRenderer(v.gpu); -}, { tz: 'America/New_York', audio: 11111, sw: 1920, sh: 1080, canvas: 44444, gpu: 'Intel Iris OpenGL Engine' }); +}, { tz: 'America/New_York', audio: 11111, sw: 1920, sh: 1080, gpu: 'Intel Iris OpenGL Engine' }); // Context B — appears as a Tokyo user with Apple GPU (fully isolated from A) const ctxB = await browser.newContext(); @@ -157,9 +151,8 @@ await ctxB.addInitScript((v) => { if (typeof window.setTimezone === 'function') window.setTimezone(v.tz); if (typeof window.setAudioFingerprintSeed === 'function') window.setAudioFingerprintSeed(v.audio); if (typeof window.setScreenDimensions === 'function') window.setScreenDimensions(v.sw, v.sh); - if (typeof window.setCanvasSeed === 'function') window.setCanvasSeed(v.canvas); if (typeof window.setWebGLRenderer === 'function') window.setWebGLRenderer(v.gpu); -}, { tz: 'Asia/Tokyo', audio: 99999, sw: 2560, sh: 1440, canvas: 88888, gpu: 'Apple M1' }); +}, { tz: 'Asia/Tokyo', audio: 99999, sw: 2560, sh: 1440, gpu: 'Apple M1' }); ``` --- @@ -176,7 +169,7 @@ All patches share `RoverfoxStorageManager`, a thread-safe C++ key-value store ke 3. The value is stored in RoverfoxStorageManager's local HashMap cache (thread-safe `nsTHashMap` protected by `Mutex`) 4. The value is also written to Firefox Preferences (`Preferences::SetCString`) with a `roverfox.s.` prefix — all value types (uint32, bool, string) are serialized as CString internally 5. In content processes, values are sent to the parent (browser) process via sync IPC (`SendRoverfoxStoragePut`) -6. Some patches also store the value under ucid=0 as a global fallback — this ensures workers that cannot resolve a specific `userContextId` can still read the value. Patches with ucid=0 fallback: **audio**, **canvas**, **navigator** (all 4 functions), **timezone**, **webgl**. Patches without: font-spacing, screen, font-list, speech-voices, webrtc-ip +6. Some patches also store the value under ucid=0 as a global fallback — this ensures workers that cannot resolve a specific `userContextId` can still read the value. Patches with ucid=0 fallback: **audio**, **navigator** (all 4 functions), **screen**, **timezone**, **webgl**. Patches without: font-spacing, font-list, speech-voices, webrtc-ip **Read path (3-tier fallback):** 1. **Local cache** — in-process `nsTHashMap` protected by `Mutex` (fastest, same-process reads) @@ -229,7 +222,7 @@ Workers resolve `userContextId` via `WorkerPrivate::GetOriginAttributes()`, whic The `camoufox.cfg` file sets Firefox preferences at startup (before `prefs.js` is loaded). Key settings: - `fission.autostart = true` — keeps Fission (site isolation) enabled. Some WAFs can detect disabled Fission. With the cross-process storage patch, Fission works correctly because values are synced across all content processes. -- `fission.webContentIsolationStrategy = 1` — standard isolation strategy. +- `fission.webContentIsolationStrategy = 0` — no site isolation: cross-site iframes stay in their parent's process (no out-of-process iframes); COOP handling and BFCache-in-parent stay active. - `dom.ipc.processPrelaunch.enabled = false` — prevents Firefox from reusing pre-launched content processes that may have stale overridden values (locale, timezone). Ensures each new content process starts clean. - No `dom.ipc.processCount` override — Firefox uses its default multi-process behavior. The cross-process storage patch eliminates the need for `processCount=1`. @@ -239,7 +232,7 @@ The `camoufox.cfg` file sets Firefox preferences at startup (before `prefs.js` i ### 1. anti-font-fingerprinting.patch -**Controls:** Canvas `measureText()` letter spacing — makes text width measurements unique per context. +**Controls:** Canvas `measureText()` letter spacing — makes text width measurements unique per context. The Python library sets the seed to 0 (off) by default: perturbed widths are something no stock Firefox produces. Pass `fonts:spacing_seed` to opt in. **How it works:** Stores a seed per context, then applies a deterministic spacing transformation in HarfBuzz (the text shaping engine). The seed is propagated through the entire text rendering pipeline: `nsTextFrame` → `gfxFont` → `gfxTextRun` → `gfxHarfBuzzShaper`. @@ -322,7 +315,7 @@ window.setTimezone('America/New_York'); // IANA timezone ID Also hooks `nsMediaFeatures.cpp` so CSS media queries like `matchMedia('(device-width: 1920px)')` return results consistent with `screen.width`. Without this, fingerprinters can detect a mismatch between the JavaScript API and CSS media queries. -This replaces the old `screen-hijacker.patch` (which only supported global config). It includes the same global `CAMOU_CONFIG` fallback, so it works for both single-context and multi-context use cases. +The global `CAMOU_CONFIG` fallback means it works for both single-context and multi-context use cases. **API:** ```javascript @@ -389,7 +382,7 @@ window.setWebRTCIPv6('2001:db8::1'); // proxy exit IPv6 (optional) window.setNavigatorPlatform('Win32'); // navigator.platform window.setNavigatorOscpu('Windows NT 10.0; Win64; x64'); // navigator.oscpu window.setNavigatorHardwareConcurrency(8); // navigator.hardwareConcurrency -window.setNavigatorUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:146.0) Gecko/20100101 Firefox/146.0'); +window.setNavigatorUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0'); ``` **Global config fallback (no JavaScript needed):** @@ -427,35 +420,7 @@ window.setWebGLRenderer('Intel Iris OpenGL Engine'); // UNMASKED_RENDERER_WEBGL --- -### 8. canvas-spoofing.patch - -**Controls:** Canvas 2D fingerprint hash — websites draw text, shapes, and gradients on a canvas, then call `toDataURL()` or `getImageData()` to hash the pixel output. GPU, driver, and font rendering differences make this hash highly unique. - -**How it works:** Stores a seed per context via `CanvasFingerprintManager`, then hooks both canvas data extraction paths in `CanvasRenderingContext2D.cpp`: -- `GetImageBuffer()` — used by `toDataURL()` and `toBlob()`, returns pixels in **BGRA** format -- `GetImageData()` — used by `ctx.getImageData()`, returns pixels in **RGBA** format - -The noise algorithm is **format-agnostic**: for each selected pixel, it iterates RGB channels (skipping alpha) and modifies the first non-zero channel by +/-1. This works correctly regardless of whether byte 0 is Red (RGBA) or Blue (BGRA). - -**Zero-pixel preservation:** Channels with value 0 are skipped. This means `clearRect()` followed by `getImageData()` returns all zeros — no false noise on transparent pixels. This is important because CreepJS specifically tests for noise in cleared canvas regions as a detection vector. - -**Noise is deterministic, not random:** Same seed always produces the same pixel modifications, so fingerprinters calling `toDataURL()` multiple times get identical results. This is critical — random noise is trivially detected by calling the API twice and comparing outputs. - -**Worker support:** Includes a `WorkerPrivate` fallback for resolving `userContextId` when canvas operations happen in a Web Worker via OffscreenCanvas. - -**MaskConfig fallback:** If no per-context seed is set, checks `MaskConfig::GetUint32("canvas:seed")` from `CAMOU_CONFIG`. - -**API:** -```javascript -window.setCanvasSeed(55555555); // uint32 seed -``` - -**New C++ files:** `CanvasFingerprintManager.h/cpp` -**Modified Firefox files:** `nsGlobalWindowInner.cpp/h`, `CanvasRenderingContext2D.cpp`, `Window.webidl`, `moz.build` - ---- - -### 9. font-list-spoofing.patch +### 8. font-list-spoofing.patch **Controls:** Which fonts appear "installed" to fingerprinting scripts. Websites detect fonts by measuring text widths (canvas `measureText()`) — if the width changes compared to a fallback font, the font is present. Each context can have a different subset of fonts. @@ -493,7 +458,7 @@ window.setFontList('Arial,Helvetica,Georgia,Courier New,Verdana'); --- -### 10. speech-voices-spoofing.patch +### 9. speech-voices-spoofing.patch **Controls:** `speechSynthesis.getVoices()` — the list of installed text-to-speech voices. This varies by OS and installed language packs, making it a fingerprinting vector. Each context can expose a different subset of voices. @@ -512,7 +477,7 @@ window.setSpeechVoices('Microsoft David,Samantha,Alex'); --- -### 11. cross-process-storage.patch +### 10. cross-process-storage.patch **Controls:** Cross-process synchronization of all per-context fingerprint values. This is an infrastructure patch — it has no JavaScript API of its own. It enables all other per-context patches to work correctly when Firefox runs content in multiple processes (Fission). @@ -570,7 +535,7 @@ For per-context geolocation, use Playwright's built-in `context.setGeolocation() ## Build Notes -**SOURCES vs UNIFIED_SOURCES:** Most new `.cpp` manager files use `SOURCES` (separate compilation) in `moz.build` to avoid namespace pollution (`mozilla::dom::mozilla::dom::`) that occurs when files including `RoverfoxStorageManager.h` are concatenated in unified builds. Currently in `SOURCES`: `AudioFingerprintManager.cpp`, `WebRTCIPManager.cpp`, `NavigatorManager.cpp`, `WebGLParamsManager.cpp`, `CanvasFingerprintManager.cpp`, `FontListManager.cpp`, `SpeechVoicesManager.cpp`. Four files use `UNIFIED_SOURCES` instead: `FontSpacingSeedManager.cpp`, `RoverfoxStorageManager.cpp` (both from `anti-font-fingerprinting.patch`), `TimezoneManager.cpp` (from `timezone-spoofing.patch`), and `ScreenDimensionManager.cpp` (from `screen-spoofing.patch`) — these were written before the SOURCES pattern was established and happen to compile without namespace issues in their alphabetical position. +**SOURCES vs UNIFIED_SOURCES:** Most new `.cpp` manager files use `SOURCES` (separate compilation) in `moz.build` to avoid namespace pollution (`mozilla::dom::mozilla::dom::`) that occurs when files including `RoverfoxStorageManager.h` are concatenated in unified builds. Currently in `SOURCES`: `AudioFingerprintManager.cpp`, `WebRTCIPManager.cpp`, `NavigatorManager.cpp`, `WebGLParamsManager.cpp`, `FontListManager.cpp`, `SpeechVoicesManager.cpp`, `ScreenDimensionManager.cpp`. Three files use `UNIFIED_SOURCES` and compile without namespace issues in their alphabetical position: `FontSpacingSeedManager.cpp`, `RoverfoxStorageManager.cpp` (both from `anti-font-fingerprinting.patch`) and `TimezoneManager.cpp` (from `timezone-spoofing.patch`). **EXPORTS sort conflicts:** Each patch uses a separate `EXPORTS.mozilla.dom += ["Header.h"]` statement near its `SOURCES` block, rather than inserting into the main sorted EXPORTS list. This avoids sort conflicts when multiple patches add headers at similar alphabetical positions. @@ -578,7 +543,7 @@ For per-context geolocation, use Playwright's built-in `context.setGeolocation() **Patch independence:** All patches apply independently to vanilla Firefox. Context lines in hunks reference unpatched source files. Patches apply alphabetically and use fuzzy matching for line shifts caused by other patches. -**camoufox.cfg:** The `settings/camoufox.cfg` file sets `fission.autostart=true`, `fission.webContentIsolationStrategy=1`, and `dom.ipc.processPrelaunch.enabled=false`. No `dom.ipc.processCount` override is needed — the cross-process storage patch enables all per-context values to sync across Firefox's default multi-process architecture. +**camoufox.cfg:** The `settings/camoufox.cfg` file sets `fission.autostart=true`, `fission.webContentIsolationStrategy=0`, and `dom.ipc.processPrelaunch.enabled=false`. No `dom.ipc.processCount` override is needed — the cross-process storage patch enables all per-context values to sync across Firefox's default multi-process architecture. --- @@ -608,16 +573,16 @@ what differs per identity is which of them are on the search path. **What each `fonts.conf` defines:** - **Generic family defaults** — `sans-serif`, `serif`, `monospace`, `cursive`, `fantasy`, `system-ui` mapped to OS-appropriate fonts - **TTC weight-variant aliases** — macOS TrueType Collections register with weight suffixes ("PingFang HK Light") but Linux fontconfig only sees the base name. Aliases rewrite queries so CreepJS marker font detection works cross-platform. -- **MONO redirect** — "MONO" is a Linux-only font. Redirected to the OS-appropriate monospace (Menlo on macOS, Cousine on Linux) to prevent host OS leakage. +- **MONO redirect** — "MONO" is a Linux marker font. The Linux and Windows configs redirect it to `monospace` so it measures like the monospace baseline; the macOS config leaves it unmatched, so it falls through to Menlo as on a real Mac. - **Rendering settings** — Standardized antialias, hinting, and lcdfilter across all configs. **Runtime path rewriting:** At launch time, `utils._generate_fontconfig()` reads the bundled `fonts.conf` and replaces its single `fonts` with one absolute `` per group the claimed OS reads (from `fonts/groups.json`). This is what prevents cross-OS font leakage — a face the claimed OS must not see is simply not on the search path — and it also avoids CWD-dependent path issues. The parent `fonts/` directory is never named: fontconfig scans `` **recursively**, so naming it would make every other OS's faces reachable for glyph fallback even though the allowlist hides them from direct lookup. `scripts/verify-fonts.py` asserts that no file outside an OS's own groups is reachable under its conf. -**`FONTCONFIG_PATH` environment variable:** Must be set when launching Camoufox on Linux. Points to the correct OS-specific fontconfig directory (e.g. `camoufox/fontconfig/macos/`). The Go launcher sets this dynamically based on the target OS. +**`FONTCONFIG_FILE` environment variable:** On a Linux host, `utils.get_env_vars()` points `FONTCONFIG_FILE` at the generated `fonts.conf` for the claimed OS. A browser launched without the Python (or TypeScript) wrapper does not get it and uses the system fontconfig. --- -## Python Library Changes +## Python Library The Camoufox Python package (`pythonlib/`) generates fingerprints for both `NewBrowser` (global CAMOU_CONFIG) and `NewContext` (per-context init script). **fpgen is the default for both paths.** Real fingerprint presets are available as an opt-in alternative. @@ -630,9 +595,9 @@ The Camoufox Python package (`pythonlib/`) generates fingerprints for both `NewB **Recommended for v149+ binaries:** opt into bundled real fingerprints via `fingerprint_preset=True`. The library auto-selects the v150 preset bundle -(`fingerprint-presets-v150.json`, 312 real fingerprints scraped from v149–v152 +(`fingerprint-presets-v150.json`, 288 real fingerprints scraped from v149–v152 browsers) for any binary at Firefox ≥ 149, and falls back to the original -bundle (`fingerprint-presets.json`, 123 presets) for older binaries. UA strings +bundle (`fingerprint-presets.json`, 109 presets) for older binaries. UA strings are rewritten to match the active binary's Firefox version, so opting in costs nothing for compatibility. @@ -659,13 +624,12 @@ bundles are shipped in the wheel. |----------|--------|-------| | UA, platform, HWC, oscpu | fpgen or preset | UA version patched to match Camoufox Firefox version | | Screen dims, colorDepth | fpgen or preset | Viewport adjusted by -28px for browser chrome | -| WebGL vendor/renderer | `sample_webgl()` from `webgl_data.db` | OS-weighted probability sampling. The generator's own GPU fields are not mapped in `fpgen.yml` yet, so both paths call `sample_webgl()` for WebGL. fpgen does carry a full WebGL set (999 renderers against webgl_data.db's 33) -- wiring it through is the follow-up. | -| Font list | `_generate_random_font_subset()` | Random 30-78% of OS fonts. Essential + marker fonts always included. NOT from presets — generated fresh per call. | -| Font spacing seed | `randint(1, 2^32-1)` | Excludes 0 (0 = no-op in C++) | -| Audio seed | `randint(1, 2^32-1)` | Excludes 0 | -| Canvas seed | `randint(1, 2^32-1)` | Excludes 0 | -| Timezone | From preset, or Intl.DateTimeFormat fallback in init script | NewBrowser: from preset or geolocation detection. NewContext: preset or browser default. | -| Speech voices | `_generate_random_voice_subset()` | Random 40-80% of OS voices. Essential voices always included. macOS: 6 essentials + random subset of ~184. Windows: all voices (too few to subset). Linux: empty (no native voices). NOT from presets — generated fresh per call. | +| WebGL vendor/renderer | `sample_webgl()` from `webgl_data.db` | OS-weighted probability sampling. fpgen's own WebGL fields are not mapped in `fpgen.yml` yet, so both paths call `sample_webgl()`. | +| Font list | `_generate_random_font_subset()` | One weighted OS-version base in full, plus each addition unit at its measured probability; marker fonts always included. See [FONTS.md](FONTS.md). NOT from presets. | +| Font spacing seed | `0` | Off by default (0 = no-op in C++); pass `fonts:spacing_seed` to opt in | +| Audio seed | Derived from the identity (NewBrowser) or `randint(1, 2^32-1)` (NewContext) | Never 0 | +| Timezone | From preset, or `timezone` in `CAMOU_CONFIG` | The init script calls `setTimezone()` only for an explicit value; otherwise the C++ side falls back to `CAMOU_CONFIG` (set from geoip at launch) or the browser default. | +| Speech voices | `_generate_random_voice_subset()` | Follows the measured model in `voice-manifests.json`: Windows gets the display language's OneCore pack plus its legacy Desktop voices at their measured rate; macOS the compact + Eloquence base plus rare downloads; Linux speech-dispatcher's espeak-ng list. Seeded by the identity. NOT from presets. | | WebRTC IP | Not set by default | User sets via `window.setWebRTCIPv4()`. NewContext init script defaults to empty string `""` | | Geolocation | User parameter or geoip detection | Via Playwright `context.setGeolocation()` | @@ -675,9 +639,9 @@ bundles are shipped in the wheel. - `generate_context_fingerprint()` — main API. Returns `{init_script, context_options, config, preset}` - `from_preset()` — converts real preset to CAMOU_CONFIG format - `from_fpgen()` — converts an fpgen fingerprint dict to CAMOU_CONFIG using `fpgen.yml` mappings -- `_build_init_script()` — generates JavaScript IIFE calling 15 `window.setXxx()` functions with `typeof` guards (`setWebRTCIPv6` is not included — IPv6 is optional and rarely set) -- `_generate_random_font_subset()` — unique random font subset per call (Fisher-Yates, essential + marker fonts always included) -- `_generate_random_voice_subset()` — unique random voice subset per call (essential voices always included, OS-aware) +- `_build_init_script()` — generates a JavaScript IIFE calling the `window.setXxx()` functions with `typeof` guards (every setter except `setWebRTCIPv6` — IPv6 is optional and rarely set) +- `_generate_random_font_subset()` — the font list of one plausible machine of the OS (weighted base + per-unit draws, marker fonts always included) +- `_generate_random_voice_subset()` — the voice list of one plausible machine of the OS, as MaskConfig voice objects **`utils.py`** — Global browser launch configuration: - `launch_options()` — builds CAMOU_CONFIG env var, Playwright args, and Firefox prefs @@ -686,15 +650,15 @@ bundles are shipped in the wheel. - WebGL sampled via same `sample_webgl()` function - Config validated against `properties.json` before serialization -**`fingerprint-presets.json`** — Original bundled real fingerprints organized by OS (macOS 30, Windows 75, Linux 18). Each preset includes navigator properties, screen dimensions, WebGL params, and speech voices. Used for Firefox < 149 binaries. Font and voice data not used from presets — generated fresh per launch. +**`fingerprint-presets.json`** — Original bundled real fingerprints organized by OS (macOS 18, Windows 73, Linux 18). Each preset includes navigator properties, screen dimensions, WebGL params, and speech voices. Used for Firefox < 149 binaries. Font and voice data not used from presets — generated fresh per launch. -**`fingerprint-presets-v150.json`** — Newer bundle covering Firefox v149–v152 (macOS 67, Windows 180, Linux 65; 312 total). Same schema as the original. Auto-selected by `load_presets()` when the active binary reports Firefox ≥ 149. +**`fingerprint-presets-v150.json`** — Newer bundle covering Firefox v149–v152 (macOS 45, Windows 178, Linux 65; 288 total). Same schema as the original. Auto-selected by `load_presets()` when the active binary reports Firefox ≥ 149. -**`fonts.json`** — Complete OS-specific font lists for random font subset generation. +**`fonts.json`, `font-bases.json`, `font-groups.json`** — OS font lists, the OS-version bases and the addition units with their probabilities (see [FONTS.md](FONTS.md)). -**`voices.json`** — Complete OS-specific speech voice lists for random voice subset generation. macOS: 190 voices, Windows: 53 voices, Linux: empty. Format: `"Name:locale:type"` — names extracted at load time. +**`voices.json`** — OS-specific speech voice lists (macOS 190, Windows 53, Linux 131). Format: `"Name:locale:type"`. **`voice-manifests.json`** holds the per-OS model the voice draw follows. -**`properties.json`** — Includes `audio:seed` and `canvas:seed` as `CAMOU_CONFIG` properties (uint type). These enable the MaskConfig fallback in the audio and canvas patches when using global config without per-context JavaScript. +**`properties.json`** — Includes `audio:seed` as a `CAMOU_CONFIG` property (uint type), the MaskConfig fallback for the audio patch when using global config without per-context JavaScript. **`camoufox.cfg`** — Sets `fission.autostart=true` and `dom.ipc.processPrelaunch.enabled=false`. No `dom.ipc.processCount` override needed with cross-process storage. diff --git a/docs/playwright-maintenance.md b/docs/playwright-maintenance.md index a4aeb92..b7e09de 100644 --- a/docs/playwright-maintenance.md +++ b/docs/playwright-maintenance.md @@ -4,7 +4,10 @@ This document describes how to maintain Playwright integration in Camoufox. ## Overview -Camoufox integrates Playwright's browser automation capabilities through patches and additional files. These need to be kept in sync with upstream Playwright development. +Camoufox integrates Playwright's browser automation through a patch and a copy of +Playwright's Juggler protocol. Both started from upstream Playwright and both +carry Camoufox changes, so upstream updates are ported into them, never copied +over them. ## Patch Files @@ -12,106 +15,78 @@ Location: `patches/playwright/` | File | Purpose | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `0-playwright.patch` | Playwright's upstream patches. Must be kept up to date with [bootstrap.diff](https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/patches/bootstrap.diff) | -| `1-leak-fixes.patch` | Undos certain patches from `0-playwright.patch` to fix memory leaks | +| `0-playwright.patch` | Playwright's [bootstrap.diff](https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/patches/bootstrap.diff), ported to the Firefox version in `upstream.sh`, plus Camoufox fixes to the Juggler input and navigation paths | +| `1-leak-fixes.patch` | Undoes two changes from `0-playwright.patch` that expose automation: `navigator.webdriver` always reports `false`, and enterprise policies load from Firefox's normal provider instead of Playwright's | + +Both sort ahead of every other patch, so `scripts/patch.py` applies them first. ## Addition Files Location: `additions/juggler/` -The `juggler` directory contains Playwright's Juggler protocol implementation. These files must be kept in sync with: - -**Upstream Source:** https://github.com/microsoft/playwright/tree/main/browser_patches/firefox/juggler +Camoufox's Juggler, started from +[upstream](https://github.com/microsoft/playwright/tree/main/browser_patches/firefox/juggler). +Camoufox changes include the isolated-world page agent and the human cursor +(`input/`), so a straight copy from upstream would remove them. ### Key Files -- **`components/Juggler.js`** - Main Juggler component (legacy JSM format) -- **`components/Juggler.sys.mjs`** - ESM wrapper for Firefox 146+ compatibility +- **`components/Juggler.js`** - Main Juggler component, an ES module that exports `JugglerFactory` - **`components/components.conf`** - XPCOM component registration +- **`jar.mn`** - What gets packaged into `chrome://juggler/content/` -### Firefox 146 ESM Migration +`components.conf` registers the component with the `esModule` field (Firefox +no longer supports `jsm`): -Firefox 146 removed JSM (JavaScript Module) support in favor of ESM (ES Modules). To maintain compatibility: - -1. **`components.conf`** uses `esModule` field instead of deprecated `jsm` field: - ```python - { - "esModule": "chrome://juggler/content/components/Juggler.sys.mjs", - "constructor": "JugglerFactory", - } - ``` - -2. **`Juggler.sys.mjs`** acts as an ESM wrapper that imports the legacy JSM file: - ```javascript - const { JugglerFactory } = ChromeUtils.import( - "chrome://juggler/content/components/Juggler.js" - ); - export { JugglerFactory }; - ``` - -This maintains backward compatibility while satisfying Firefox 146's static component generator requirements. +```python +{ + "esModule": "chrome://juggler/content/components/Juggler.js", + "constructor": "JugglerFactory", +} +``` ## Updating Playwright Integration -### 1. Update Upstream Patches +### 1. Port Upstream Patch Changes -Compare the current `patches/playwright/0-playwright.patch` with Playwright's [bootstrap.diff](https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/patches/bootstrap.diff). - -If changes are needed: -```bash -# Download latest bootstrap.diff -curl -o patches/playwright/0-playwright.patch \ - https://raw.githubusercontent.com/microsoft/playwright/main/browser_patches/firefox/patches/bootstrap.diff - -# Test the build -make clean && make dir && make build -``` - -### 2. Update Juggler Files - -Sync `additions/juggler/` with upstream: +Compare what changed in Playwright's +[bootstrap.diff](https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/patches/bootstrap.diff) +since the last sync and apply those changes to the tree, then regenerate the +patch with the make targets described in +[patch-upgrading-guide.md](patch-upgrading-guide.md): ```bash -# Clone Playwright repository -git clone https://github.com/microsoft/playwright.git /tmp/playwright - -# Compare directories -diff -r additions/juggler/ /tmp/playwright/browser_patches/firefox/juggler/ - -# Copy updated files (example) -cp -r /tmp/playwright/browser_patches/firefox/juggler/* additions/juggler/ - -# IMPORTANT: Preserve Firefox 146 ESM compatibility -# - Keep additions/juggler/components/Juggler.sys.mjs -# - Keep additions/juggler/components/components.conf with esModule field -``` - -### 3. Verify ESM Wrapper Compatibility - -After updating from upstream, ensure the ESM wrapper remains functional: - -1. Check that `Juggler.js` still exports `JugglerFactory`: - ```javascript - var EXPORTED_SYMBOLS = ["Juggler", "JugglerFactory"]; - var JugglerFactory = function() { /* ... */ }; - ``` - -2. If upstream changed the export name, update `Juggler.sys.mjs` accordingly. - -3. Verify `components.conf` matches the format above (not upstream's format). - -### 4. Test Build - -```bash -# Clean build to verify component registration -cd camoufox-146.0.1-beta.25 -make clean -cd .. make dir -cd camoufox-146.0.1-beta.25 -./mach build +make workspace ./patches/playwright/0-playwright.patch +# port the upstream changes into camoufox--/ +make diff > patches/playwright/0-playwright.patch ``` +### 2. Port Juggler Changes + +```bash +git clone https://github.com/microsoft/playwright.git /tmp/playwright +diff -r additions/juggler/ /tmp/playwright/browser_patches/firefox/juggler/ +``` + +Port the upstream hunks one by one, keeping the Camoufox changes. After an +update, check that: + +1. `Juggler.js` still exports `JugglerFactory`, and `components.conf` still + names it as the `constructor`. +2. `components.conf` uses `esModule`, not `jsm`. +3. Every file upstream added or renamed is listed in `jar.mn`. + +### 3. Test + +```bash +make dir && make build +make tests +``` + +`make dir` resets the tree, clobbers the object directory and reapplies every +patch, so this is a clean build. + **Expected:** No linker errors about `mozCreateComponent`. **Common Error:** If you see: @@ -129,7 +104,7 @@ This means `components.conf` is not using ESM format. Fix by ensuring it has: **Error:** `Externally-constructed components may not specify 'constructor' or 'legacy_constructor' properties` -**Cause:** Using `"jsm"` field which is unsupported in Firefox 146. +**Cause:** Using the `"jsm"` field, which Firefox no longer supports. **Fix:** Use `"esModule"` field instead. @@ -153,13 +128,11 @@ This means `components.conf` is not using ESM format. Fix by ensuring it has: If the build fails after updating Juggler files: -1. Check that all JSM imports in `Juggler.js` are still valid -2. Verify the ESM wrapper exports match what's imported -3. Ensure no file paths changed in upstream -4. Check for Firefox API changes that might require patches +1. Check that every `ChromeUtils.importESModule()` path in the Juggler files still exists +2. Check that new or renamed files are listed in `jar.mn` +3. Check for Firefox API changes that might require patches ## References - [Playwright Firefox Patches](https://github.com/microsoft/playwright/tree/main/browser_patches/firefox) -- [Firefox 146 Component Registration](https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/basics.html) - [Firefox ESM Migration Guide](https://firefox-source-docs.mozilla.org/dom/script_loader/index.html) diff --git a/patches/patch-dependencies.md b/patches/patch-dependencies.md index 3233058..0fdcf68 100644 --- a/patches/patch-dependencies.md +++ b/patches/patch-dependencies.md @@ -1,24 +1,62 @@ # Patch Dependencies -Quick reference for which patches depend on shared infrastructure. +Quick reference for the shared infrastructure patches build on. `scripts/patch.py` +applies every `patches/**/*.patch` in order of file name, so the +`playwright/0-*` and `playwright/1-*` patches go first and the rest follow +alphabetically. The dependencies below are compile-time: a patch applies without +them, but the tree will not build. ## camoucfg (MaskConfig) -Most patches read config via `MaskConfig::GetBool()`, `MaskConfig::GetString()`, etc. from `/camoucfg`. Any patch that adds `LOCAL_INCLUDES += ["/camoucfg"]` to a `moz.build` file depends on `config.patch` being applied first (which provides the `camoucfg` directory). +`additions/camoucfg/MaskConfig.hpp` reads the spoofing config (`CAMOU_CONFIG` / +`camoufox.cfg`) through `MaskConfig::GetBool()`, `GetString()`, `GetUint32()` +and friends. `scripts/copy-additions.sh` copies it into the source tree before +any patch applies. A patch that calls MaskConfig from a directory whose +`moz.build` does not already see `/camoucfg` must add +`LOCAL_INCLUDES += ["/camoucfg"]` itself. -### Patches using MaskConfig +Config keys are declared in `settings/properties.json`; a key a patch reads must +be listed there. -| Patch | Config keys | What it does | -|-------|-------------|--------------| -| `media-codec-spoofing.patch` | `media:spoof_codecs` | Bypasses `PDMFactory::Supports()` checks in `MP4Decoder` and `MatroskaDecoder` so `canPlayType()`/`isTypeSupported()` don't leak system codec libraries | -| `navigator-spoofing.patch` | Various `navigator:*` keys | Per-context navigator property spoofing | -| `geolocation-spoofing.patch` | `geo:*` keys | Geolocation coordinate spoofing | -| `locale-spoofing.patch` | `locale:*` keys | Language/locale spoofing | +### Patches that read config + +| Patch | Config keys | +|-------|-------------| +| `anti-font-fingerprinting.patch` | `fonts:spacing_seed` | +| `audio-context-spoofing.patch` | `AudioContext:outputLatency` | +| `audio-fingerprint-manager.patch` | `audio:seed` | +| `chromeutil.patch` | `debug` | +| `fingerprint-injection.patch` | `navigator.*`, `screen.*`, `window.*`, `battery:*` | +| `font-hijacker.patch` | `navigator.platform` | +| `font-system-fonts-css2.patch` | `navigator.platform`, `window.devicePixelRatio` | +| `force-default-pointer.patch` | `navigator.maxTouchPoints` | +| `geolocation-spoofing.patch` | `geolocation:*` | +| `global-style-sheets.patch` | `disableTheming` | +| `locale-spoofing.patch` | `locale:*`, `navigator.language` | +| `media-codec-spoofing.patch` | `media:spoof_codecs` (bypasses `PDMFactory::Supports()` in `MP4Decoder`/`MatroskaDecoder` so `canPlayType()`/`isTypeSupported()` don't leak system codec libraries) | +| `media-device-spoofing.patch` | `mediaDevices:*` | +| `navigator-spoofing.patch` | `navigator.*`, `timezone` | +| `network-patches.patch` | `headers.*`, `navigator.userAgent` | +| `no-css-animations.patch` | `disableInstantAnimations` | +| `screen-spoofing.patch` | `screen.width`, `screen.height` | +| `system-ui-font-spoofing.patch` | `navigator.platform` | +| `timezone-spoofing.patch` | `timezone` | +| `touchscreen-fingerprint-spoofing.patch` | `navigator.maxTouchPoints` | +| `voice-spoofing.patch` | `voices:*` | +| `webgl-spoofing.patch` | `webGl:*` | +| `webrtc-ip-spoofing.patch` | `webrtc:ipv4`, `webrtc:ipv6`, `navigator.platform` | + +To regenerate the list: `grep -l 'MaskConfig::' patches/*.patch`. ## RoverfoxStorageManager -Per-context patches that use cross-process storage depend on `cross-process-storage.patch`. +Per-context values set from Playwright (font spacing seed, WebRTC IP, timezone, +screen, navigator, voices, ...) are kept in `RoverfoxStorageManager`, which +`anti-font-fingerprinting.patch` adds under `dom/base/`. Its cross-process +put/get IPC lives in `cross-process-storage.patch`. Any patch that uses the +storage manager needs both. ## Playwright -All patches should be applied after `0-playwright.patch` and `1-leak-fixes.patch`. +Everything else is written against a tree that already has +`playwright/0-playwright.patch` and `playwright/1-leak-fixes.patch` applied. diff --git a/patches/playwright/README.md b/patches/playwright/README.md index d3899e3..c5a3b55 100644 --- a/patches/playwright/README.md +++ b/patches/playwright/README.md @@ -3,4 +3,4 @@ | File | Purpose | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `0-playwright.patch` | Playwright's upstream patches. Needs to be kept up to date with [this file](https://github.com/microsoft/playwright/blob/main/browser_patches/firefox/patches/bootstrap.diff). Will branch off if upstream is out of date. | -| `1-leak-fixes.patch` | Undos certain patches from `0-playwright.patch`. | +| `1-leak-fixes.patch` | Undoes certain patches from `0-playwright.patch`. | diff --git a/pythonlib/README.md b/pythonlib/README.md index 10c6572..d66e9b2 100644 --- a/pythonlib/README.md +++ b/pythonlib/README.md @@ -7,7 +7,7 @@ > [!NOTE] -> All the the latest documentation is avaliable [here](https://camoufox.com/python). +> All the latest documentation is available [here](https://camoufox.com/python). --- @@ -33,18 +33,10 @@ The `geoip` parameter is optional, but heavily recommended if you are using prox Next, download the Camoufox browser: -**Windows** - ```bash camoufox fetch ``` -**MacOS & Linux** - -```bash -python3 -m camoufox fetch -``` - To uninstall, run `camoufox remove`. --- @@ -75,7 +67,7 @@ camoufox gui --- -## CLI Mananger +## CLI Manager #### Demonstration @@ -246,13 +238,12 @@ Display the Python package version, active browser version, channel, and update ```bash > camoufox version Python Packages - Camoufox v0.5.0 + Camoufox v0.5.6 fpgen v1.3.0 - Apify Fingerprints v0.10.0 - Playwright v1.57.1.dev0+g732639b35.d20251217 + Playwright v1.62.0 Browser - Active official/stable/135.0.1-beta.24 - Current browser v135.0.1-beta.24 + Active official/stable/152.0.4-beta.31 + Current browser v152.0.4-beta.31 Installed Yes Latest in official/stable? Yes Last Sync 2026-03-07 00:23 @@ -303,4 +294,4 @@ Launch a remote Playwright server. ## Usage -All of the latest stable documentation is avaliable at [camoufox.com/python](https://camoufox.com/python). +All of the latest stable documentation is available at [camoufox.com/python](https://camoufox.com/python). diff --git a/service-tester/README.md b/service-tester/README.md index abe8afe..253a765 100644 --- a/service-tester/README.md +++ b/service-tester/README.md @@ -4,7 +4,7 @@ End-to-end antibot-detection tests that verify a pip-installed camoufox release ## Prerequisites -- Python 3.9+ +- Python 3.10+ (what `pythonlib/` requires) - Node.js (for building the TypeScript checks bundle via `esbuild`) - At least one proxy in `proxies.txt` @@ -30,7 +30,12 @@ End-to-end antibot-detection tests that verify a pip-installed camoufox release Use `--binary local` or `--binary fetched` to run only one phase. -> **Heads-up on Phase 2:** if the latest `official/stable` is much older than the local pythonlib (e.g. v135 binary vs pythonlib targeting v149+), the binary may not understand newer fingerprint patches and the test page can fail to produce results. Pin a newer binary with `--browser-version` to avoid this. +On Windows, `.\run_tests.ps1` does the same in PowerShell, with the local build +looked up at `..\camoufox-*\obj-*-windows-msvc\dist\bin\camoufox.exe`. Its +options are the ones below in PowerShell form (`-BrowserVersion`, +`-ProfileCount`, `-Proxies`, `-Headful`, `-NoCert`, `-SaveCert`, `-Binary`). + +> **Heads-up on Phase 2:** if the latest `official/stable` is older than the Firefox version the local pythonlib targets, the binary may not understand newer fingerprint patches and the test page can fail to produce results. Pin a newer binary with `--browser-version` to avoid this. ## Proxies @@ -85,13 +90,14 @@ python run_tests.py python run_tests.py [options] --browser-version VER Camoufox version specifier (default: official/stable) - e.g. official/prerelease/146.0.1-beta.50 + e.g. official/prerelease/- --profile-count N Number of profiles to test (1-6, default: 6) --proxies PATH Path to proxies file (default: proxies.txt) --headful Run with visible browser window --no-cert Skip certificate generation --save-cert PATH Save certificate text to a file --secret KEY HMAC signing key for the certificate + (run_tests.py only) --binary MODE Which binary to test: local | fetched | both (default: both) (run_tests.sh only — orchestrates the two phases) --executable-path PATH Run against a specific binary path @@ -102,7 +108,7 @@ python run_tests.py [options] 6 browser contexts run simultaneously — 3 macOS profiles and 3 Linux profiles — each with: -- A unique fingerprint generated by camoufox via fpgen (navigator, screen, WebGL, fonts, voices, audio/canvas seeds) +- A unique fingerprint generated by camoufox via fpgen (navigator, screen, WebGL, fonts, voices, audio seed) - A distinct timezone - Its own proxy, with WebRTC ICE candidates spoofed to the proxy's IP @@ -116,7 +122,7 @@ Each context is scored across these categories: | Firefox APIs | Firefox-specific API presence | | Cross-Signal | Consistency across navigator, screen, etc. | | CSS Fingerprint | CSS rendering fingerprint | -| Canvas Noise | Canvas hash uniqueness and stability | +| Canvas Noise | Canvas output is identical across renders (no random noise) | | WebGL Render | WebGL rendering hash | | Audio Integrity | AudioContext fingerprint | | Font Platform | OS-consistent font availability | diff --git a/service-tester/run_tests.ps1 b/service-tester/run_tests.ps1 index 92b1fca..1087e00 100644 --- a/service-tester/run_tests.ps1 +++ b/service-tester/run_tests.ps1 @@ -8,7 +8,7 @@ .PARAMETER BrowserVersion Camoufox version specifier (default: official/stable) - e.g. official/prerelease/146.0.1-beta.50 + e.g. official/stable/152.0.4-beta.31 .PARAMETER ProfileCount Number of profiles to test (1-6, default: 6) @@ -30,7 +30,7 @@ .EXAMPLE .\run_tests.ps1 - .\run_tests.ps1 -BrowserVersion official/prerelease/146.0.1-beta.50 -Headful + .\run_tests.ps1 -BrowserVersion official/stable/152.0.4-beta.31 -Headful .\run_tests.ps1 -Binary local .\run_tests.ps1 -Binary fetched -ProfileCount 3 #> diff --git a/service-tester/run_tests.py b/service-tester/run_tests.py index da0175c..8216eb8 100644 --- a/service-tester/run_tests.py +++ b/service-tester/run_tests.py @@ -11,7 +11,7 @@ Usage: Options: --browser-version VER Camoufox version specifier (default: official/stable) - e.g. official/prerelease/146.0.1-beta.50 + e.g. official/stable/152.0.4-beta.31 --profile-count N Number of profiles to test (1-6, default: 6) --headful Run with visible browser window --proxies PATH Path to proxies file (default: proxies.txt next to this script) diff --git a/tests/camoufox/README.md b/tests/camoufox/README.md index 2ed3e3c..0740c59 100644 --- a/tests/camoufox/README.md +++ b/tests/camoufox/README.md @@ -23,7 +23,11 @@ Write a test here when, and only when, one of these is true: to *ignore* the context locale (microsoft/playwright#38919); Camoufox sets the locale below that layer, so its workers agree with the main thread. The upstream test is skiplisted in `ci/skiplist.yml` and the version here takes - over guarding the behaviour. + over guarding the behaviour. `test_user_agent_token.py` is the other one: + upstream expects `Firefox` in the User-Agent, while the bare binary + advertises `Camoufox/` until the Python package injects a + fingerprint, so it asserts a well-formed Gecko UA that matches between the + request header and `navigator.userAgent`. In the second case the `ci/skiplist.yml` entry must name the test that replaces it, so a skip can never quietly mean "nothing checks this any more".