Files
archived-camoufox/ci/browser_inputs.py
T
Jake WriterandClaude Opus 5.5 e3f9a20e55 Hide Juggler from pages, and show them an official Firefox build (#847)
hCaptcha's hsw.js runs new Worker("chrome://juggler/content/content/WorkerMain.js").
Content processes receive every registered chrome package, and the Worker
constructor returns for a registered name but throws a SecurityError for an
absent one, so any build that registered a `juggler` package got a puzzle and
had its solved tokens rejected, with or without Playwright attached.

Juggler is no longer a chrome package. Its files are installed with
FINAL_TARGET_FILES under modules/juggler/ and loaded from
resource://gre/modules/juggler/. resource://gre exists in every Firefox, and
inside it a page cannot tell which files exist. ci/browser_inputs.py reads the
resource mapping from that moz.build instead of jar.mn.

Camoufox also builds with MOZILLA_OFFICIAL, which drops
browser-development-helpers.js from a content-accessible package as stock
does, and serves Firefox's official logos from chrome://branding/content/.
Telemetry reporting stays off, and dump() is re-enabled because Playwright
waits for Juggler's readiness line.

contentaccessible-parity.py now compares every content-reachable file's
SHA-256 with stock, compares the registered chrome packages and resource hosts,
and runs hCaptcha's Worker probe. It fails on the previous build and passes on
this one. viewport-no-rdm.py fails with one line, instead of a traceback, on a
Playwright older than 1.61, whose driver rejects is_mobile for Firefox.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 15:41:40 -06:00

246 lines
9.8 KiB
Python

#!/usr/bin/env python3
"""Split the browser's inputs into what needs a compiler and what does not.
Juggler is mostly JavaScript, and JavaScript does not need libxul relinked. But
a full build is what every change to `browser/additions/` used to cost: 24
minutes, of which ccache reported 98.63% hits -- so almost none of it was
compiling C++. It was Rust, linking, and packaging, none of which a `.js` file
affects.
The saving is real and the trap is sharp, so this module is deliberately small
and deliberately paranoid.
**The hash IS the classification.** There is no "did only JS change?" diff here,
because a diff answers the wrong question -- it compares against the pull
request's base, while what matters is whether the cached browser was built from
the same native sources. So: hash every input that can change compiled output.
If that hash matches a cached browser, the compiled half is identical *by
construction*, whatever the diff says, and the resources can simply be laid over
it.
**Two things it would be easy to get wrong, and how they are closed:**
`browser/additions/juggler/` is not all JavaScript. It also holds the
screencast encoder and the remote-debugging pipe -- 5 .cpp, 5 .h, 2 .idl, 3
components.conf, 4 moz.build -- which are compiled into libxul. Only the files
its moz.build lists in FINAL_TARGET_FILES are treated as resources.
Everything else, including anything with an extension nobody has thought about
yet, counts as native and forces a build. Fail closed.
The source-to-destination mapping is per-file, not a prefix rule. moz.build
installs `content/FrameTree.js` at `modules/juggler/content/FrameTree.js`
(depth preserved) but `content/JugglerFrameChild.sys.mjs` at
`modules/juggler/JugglerFrameChild.sys.mjs` (a level dropped). Two files in
the same source directory land at different depths. Assuming a prefix would
write one of them to the wrong path, leave the old copy in place, and run a
browser with stale Juggler -- while every suite reported green, because the
browser works fine, it is just not the one under review. So the mapping is
read from moz.build, never inferred.
That moz.build is itself native: it decides both the mapping and what is
packaged at all, so changing it must force a real build rather than a
re-overlay.
Run:
python3 -m ci.browser_inputs --digest # the native-inputs hash
python3 -m ci.browser_inputs --list # what feeds that hash
python3 -m ci.browser_inputs --resources # source -> path in the dist
python3 -m ci.browser_inputs --overlay DIR # lay resources over a dist/bin
"""
from __future__ import annotations
import argparse
import hashlib
import re
import shutil
import sys
from pathlib import Path
from typing import Dict, List, Optional
REPO_ROOT = Path(__file__).resolve().parent.parent
# The same set `resolve` greps to decide whether the browser changed at all.
# Kept in step by ci/tests/test_ci.py, because a path that can change the binary
# and is not hashed here would be served a stale browser. The fontconfig files
# are not compiled, but `make package-*` ships them, so they pair a release.
BROWSER_DIRS = (
"browser/patches",
"browser/additions",
"browser/settings",
"browser/assets",
"browser/scripts",
"browser/bundle/fontconfig",
)
BROWSER_FILES = ("browser/upstream.sh", "browser/Makefile")
# Tools under browser/scripts/ that no build runs, so touching one must not
# invalidate a cached browser and cost an hour of compiling. ci/tests/test_ci.py
# checks each entry against BUILD_ENTRY_POINTS: excluding a script a build does
# run would serve a stale binary to every suite downstream.
NON_NATIVE_SCRIPTS = frozenset(
{
"browser/scripts/_font_bundle.py",
"browser/scripts/check-input-dispatch.py",
"browser/scripts/cursor-demo.py",
"browser/scripts/gen-contentaccessible-manifest.py",
"browser/scripts/gen-font-groups.py",
"browser/scripts/gen-fonts-json.py",
"browser/scripts/verify-fonts.py",
}
)
# Where a build can reach a script from. Used by the test above, not here.
BUILD_ENTRY_POINTS = (
"browser/Makefile",
"browser/multibuild.py",
"browser/scripts/patch.py",
"browser/scripts/package.py",
"browser/scripts/copy-additions.sh",
"browser/scripts/_mixin.py",
)
JUGGLER = Path("browser") / "additions" / "juggler"
MOZ_BUILD = JUGGLER / "moz.build"
# `FINAL_TARGET_FILES.modules.juggler.content += [ "content/main.js", ... ]`: the
# attribute path is the destination directory, each string a source file.
_FILES = re.compile(r"^FINAL_TARGET_FILES\.([\w.]+)\s*\+=\s*\[(.*?)\]", re.M | re.S)
_STRING = re.compile(r'"([^"]+)"')
def resource_entries(root: Optional[Path] = None) -> Dict[str, str]:
"""{repo-relative source: path inside the built dist}, straight from moz.build.
FINAL_TARGET_FILES installs each file under its own name in the directory
the attribute path names: `FINAL_TARGET_FILES.modules.juggler.content`
puts `content/main.js` at `dist/bin/modules/juggler/content/main.js`.
"""
root = root or REPO_ROOT
text = (root / MOZ_BUILD).read_text(encoding="utf-8")
entries: Dict[str, str] = {}
for target, body in _FILES.findall(text):
directory = target.replace(".", "/")
for source in _STRING.findall(body):
entries[str(JUGGLER / source)] = f"{directory}/{Path(source).name}"
return entries
def resource_sources(root: Optional[Path] = None) -> set:
"""Repo-relative paths that are packaged as-is and never compiled."""
return set(resource_entries(root))
def native_inputs(root: Optional[Path] = None) -> List[str]:
"""Every browser input that is not a packaged resource, sorted.
Anything unrecognised lands here rather than being skipped: a new file type
under browser/additions/ forces a build until somebody decides otherwise,
which is the safe direction to be wrong in.
"""
root = root or REPO_ROOT
resources = resource_sources(root)
found: List[str] = []
for name in BROWSER_DIRS:
base = root / name
if not base.is_dir():
continue
for path in base.rglob("*"):
if not path.is_file():
continue
rel = path.relative_to(root).as_posix()
if rel not in resources and rel not in NON_NATIVE_SCRIPTS:
found.append(rel)
for name in BROWSER_FILES:
if (root / name).is_file():
found.append(name)
return sorted(found)
def native_digest(root: Optional[Path] = None) -> str:
"""A hash of the compiled browser's inputs. Same hash, same binary."""
root = root or REPO_ROOT
digest = hashlib.sha256()
for rel in native_inputs(root):
digest.update(rel.encode("utf-8"))
digest.update(b"\0")
digest.update(hashlib.sha256((root / rel).read_bytes()).digest())
return digest.hexdigest()[:32]
def source_inputs(root: Optional[Path] = None) -> List[str]:
"""Every file that goes into the browser, compiled or packaged, sorted."""
root = root or REPO_ROOT
return sorted(set(native_inputs(root)) | resource_sources(root))
def source_digest(root: Optional[Path] = None) -> str:
"""A hash of everything the browser is built from, except its release number.
This is what pairs a library release with a browser release: two commits
with the same source digest produce the same browser, so a library built
from either may name that browser's release. The release number is left
out because the release workflow writes it into the build's working tree
from the release tag (ci/release.py set-build), and that must not make the
browser look different from the main commit it was built from.
"""
root = root or REPO_ROOT
digest = hashlib.sha256()
for rel in source_inputs(root):
data = (root / rel).read_bytes()
if rel == "browser/upstream.sh":
data = b"\n".join(
line for line in data.splitlines() if not line.strip().startswith(b"release=")
)
digest.update(rel.encode("utf-8"))
digest.update(b"\0")
digest.update(hashlib.sha256(data).digest())
return digest.hexdigest()[:32]
def overlay(dist_bin: Path, root: Optional[Path] = None) -> List[str]:
"""Lay the current resources over an already-built dist/bin.
Every entry is written, not just the changed ones: the set is then always
exactly what moz.build says, so a stale file cannot survive. (A resource
*removed* from moz.build is handled by moz.build being a native input --
that forces a real build rather than an overlay.)
"""
root = root or REPO_ROOT
written = []
for source, destination in sorted(resource_entries(root).items()):
target = dist_bin / destination
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(root / source, target)
written.append(destination)
return written
def main(argv: Optional[List[str]] = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--digest", action="store_true")
parser.add_argument("--source-digest", action="store_true")
parser.add_argument("--list", action="store_true")
parser.add_argument("--resources", action="store_true")
parser.add_argument("--overlay", type=Path, metavar="DIST_BIN")
args = parser.parse_args(argv)
if args.digest:
print(native_digest())
if args.source_digest:
print(source_digest())
if args.list:
for rel in native_inputs():
print(rel)
if args.resources:
for source, destination in sorted(resource_entries().items()):
print(f"{source} -> {destination}")
if args.overlay:
written = overlay(args.overlay)
print(f"overlaid {len(written)} resource(s) onto {args.overlay}")
return 0
if __name__ == "__main__":
sys.exit(main())