Files
archived-camoufox/browser/scripts/patch.py
T
6e9a1de13b Build with cross-language LTO and PGO, as stock Firefox does (#886)
* Build with cross-language LTO and PGO, as stock Firefox does

Without them Gecko's C++ ran 1.3-1.8x stock while JIT-compiled script
did not, and a page can time that ratio (JSON.parse over a regexp
replace read ~3.3 here, ~2.2 in stock). Stock's recipe is two stages:
an instrumented build trains on Mozilla's own pages, then the release
links with --enable-lto=cross and --enable-profile-use=cross.

The "PGO profile" workflow is stage one: it publishes the merged
profile as a release asset, pinned by sha256 in
browser/assets/optimized-build.json. ci/pgo.py exports the switches
base.mozconfig reads; a local build without them is unchanged. The CI
build job pins Rust, checks the pairing, and builds LTO-only with a
warning while no profile is pinned for the tree's Firefox. None is
pinned for 156.0.1 yet.

Rust is pinned to 1.95.0. Cross-language LTO hands Rust bitcode to lld,
so rustc's LLVM may not be newer than the bootstrapped clang's. Firefox
156 bootstraps clang 22 (linux64-clang in toolchain/clang.yml) and
Mozilla builds it with Rust 1.95.0 (linux64-rust in toolchain/rust.yml),
whose LLVM is 22. 1.90.0, the pin for 152, is still 156's minimum but
no longer what stock compiles with.

ccache is restored and saved in separate steps, so a build that fails
late keeps what it compiled, under opt- keys that include the profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Link every build with cross-language ThinLTO, from the one mozconfig

LTO had entered only the tests.yml build, through an environment
variable, so a release would have been tested on an LTO Linux binary and
shipped Linux, Windows and macOS binaries built without it. base.mozconfig
now has --enable-lto=cross and --enable-rust-simd unconditionally, and
every build copies it: the tests.yml build, multibuild.py and the
release matrix.

Firefox 158's lto-pgo.configure makes --enable-lto=cross full LTO only
when PGO and MOZ_AUTOMATION are on and the target is x86_64 Linux or
Windows, so without a profile it is ThinLTO across C++ and Rust on every
target. That is what stock links outside those two, and PGO is the
remaining difference. ./mach configure on 158 with the generated mozconfig
gives MOZ_LTO_RUST_CROSS=thin, -Clinker-plugin-lto and an lld linker for
linux x86_64/arm64, windows x86_64/i686 (clang-cl, lld-link) and macOS
x86_64/arm64.

lld cannot read Rust bitcode from an LLVM newer than its own, and the
runner's stable Rust can be. scripts/patch.py now installs the Rust release
taskcluster/kinds/toolchain/rust.yml aliases as linux64-rust (1.95.0,
LLVM 22, for 158's clang 22) and writes RUSTUP_TOOLCHAIN into the
mozconfig, so the pin follows each Firefox bump by itself. The build jobs
no longer install a system LLVM 18 from apt.llvm.org: mach bootstrap
already fetches the clang and lld the tree builds with.

The PGO training workflow, ci/pgo.py, the profile pin and the separate
ccache keys are removed; PGO needs a profile to pin first. The commented
out mozconfig lines go too, and linux.mozconfig, which held only one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Document the LTO build: what it costs and where its Rust comes from

AGENTS.md and README.md said an incremental build takes about five
minutes; with LTO every build, incremental ones included, relinks
libxul across C++ and Rust, and ccache cannot cache that link. ci/README.md
lists it under cost control. On CI the libxul ThinLTO link took about
nine minutes, and the cold build step 57 minutes against 54-99 for
recent cold non-LTO builds, since rustc no longer runs its own fat LTO
over gkrust.

mozilla_rust_version's docstring now says what the pairing is: Rust's
LLVM has the bootstrapped clang's major version (1.95.0 has LLVM 22.1.2,
158's clang is 22.1.8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Build with stock's PGO profile, and full LTO where stock uses it

Stock builds every desktop release with a profile Mozilla trains on that
release (taskcluster kind generate-profile), and with full cross-language LTO
on x86_64 Linux and Windows. LTO alone left the remaining difference from stock
in how Gecko's C++ is optimized, which a page can time against script.

scripts/pgo.py fetches, for each target, the profile Mozilla trained for the
stock release of the same Firefox: Taskcluster indexes the generate-profile
task by the release commit, and the task's chain of trust records the
archive's sha256. data/pgo-profiles.json pins all five (Linux arm64 uses the
x86_64 Linux profile, as stock's does). `make dir` fetches and verifies the
profile, and base.mozconfig builds with it, with full LTO on x86_64 Linux and
Windows and ThinLTO elsewhere, as stock's configure picks under MOZ_AUTOMATION.
A function a patch changes loses its profile data; configure makes that a
warning. A pin for another Firefox fails `make dir` and says to run
`scripts/pgo.py pin`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Writer <jake_writer@jobharvest.com>
2026-10-31 06:20:35 +00:00

245 lines
7.8 KiB
Python

#!/usr/bin/env python3
"""
The script that patches the Firefox source into the Camoufox source.
Based on LibreWolf's patch script:
https://gitlab.com/librewolf-community/browser/source/-/blob/main/scripts/librewolf-patches.py
Run:
python3 scripts/patch.py <version> <release>
"""
import os
import shutil
import subprocess
import sys
from dataclasses import dataclass
from _mixin import (
find_src_dir,
get_moz_target,
get_options,
list_patches,
mozilla_rust_version,
run,
temp_cd,
)
import pgo
options, args = get_options()
"""
Main patcher functions
"""
def reset_to_unpatched():
"""Reset this source repository without discovering a parent repository."""
if not os.path.exists('.git'):
return
print("Resetting to unpatched state...")
run('git reset --hard unpatched && ./mach clobber && git clean -fdx')
@dataclass
class Patcher:
"""Patch and prepare the Camoufox source"""
moz_target: str
target: str
rust: str
pgo_profile: str
def camoufox_patches(self):
"""
Apply all patches
"""
version, release = extract_args()
with temp_cd(find_src_dir('.', version, release)):
# Reset only when the source tree has its own local repository.
reset_to_unpatched()
# Re-copy additions and settings after reset
print("Re-copying additions and settings...")
run(f'bash ../scripts/copy-additions.sh {version} {release}')
# Create the base mozconfig file
run('cp -v ../assets/base.mozconfig mozconfig')
# Set cross building target
print(f'Using target: {self.moz_target}')
self._update_mozconfig()
if not options.mozconfig_only:
failed_patches = []
for patch_file in list_patches():
rejects = self._apply_and_check(patch_file)
if rejects:
failed_patches.append((patch_file, rejects))
# Report failures
if failed_patches:
print('\n' + '='*70)
print(f'ERROR: {len(failed_patches)} patch(es) failed to apply cleanly:')
print('='*70)
for patch_file, rejects in failed_patches:
print(f'\n{patch_file}:')
for reject in rejects:
print(f' - {reject}')
print('='*70)
sys.exit(1)
print('Complete!')
def _apply_and_check(self, patch_file):
"""
Apply a patch and check for reject files.
Returns the reject files and any non-zero exit status, empty if it applied cleanly.
"""
import time
print(f"\n*** -> patch -p1 -i {patch_file}")
sys.stdout.flush()
# Record time before applying so we only detect .rej files from this patch
start_time = time.time()
# Apply patch interactively - don't capture stdout/stderr at all
# This allows prompts to show immediately and user can respond
# --forward flag: skip patches that appear to be already applied
# --binary flag: preserve line endings (helps with CRLF vs LF differences)
# -l flag: ignore whitespace differences
result = subprocess.run(
['patch', '-p1', '--forward', '-l', '--binary', '-i', patch_file],
stdin=sys.stdin,
stdout=sys.stdout,
stderr=sys.stderr,
text=True
)
# After patch completes, search for any .rej files created during this patch
rejects = []
for root, dirs, files in os.walk('.'):
for file in files:
if file.endswith('.rej'):
reject_path = os.path.join(root, file)
if os.path.exists(reject_path):
# Only include if created after this patch started
if os.path.getmtime(reject_path) >= start_time:
rejects.append(reject_path)
# Clean up .rej files so they don't interfere with subsequent patches
for rej in rejects:
os.remove(rej)
if result.returncode != 0:
rejects.append(f'patch exited with status {result.returncode}')
return rejects
def _update_mozconfig(self):
"""
Helper for adding additional mozconfig code from assets/<target>.mozconfig
"""
mozconfig_backup = "mozconfig.backup"
mozconfig = "mozconfig"
# Create backup if it doesn't exist
if not os.path.exists(mozconfig_backup):
if os.path.exists(mozconfig):
shutil.copy2(mozconfig, mozconfig_backup)
else:
with open(mozconfig_backup, 'w', encoding='utf-8') as f:
pass
# Read backup content
with open(mozconfig_backup, 'r', encoding='utf-8') as f:
content = f.read()
# Set first: base.mozconfig picks stock's LTO mode for the target and
# reads the profile. configure resolves rustc and cargo through rustup,
# which honours RUSTUP_TOOLCHAIN.
content = (f"CAMOUFOX_TARGET={self.moz_target}\n"
f"CAMOUFOX_PGO_PROFILE={self.pgo_profile}\n"
"ac_add_options --target=$CAMOUFOX_TARGET\n"
f"export RUSTUP_TOOLCHAIN={self.rust}\n\n") + content
# Add target-specific mozconfig if it exists
target_mozconfig = os.path.join("..", "assets", f"{self.target}.mozconfig")
if os.path.exists(target_mozconfig):
with open(target_mozconfig, 'r', encoding='utf-8') as f:
content += f.read()
# Update mozconfig
print(f"-> Updating mozconfig, target is {self.moz_target}")
with open(mozconfig, 'w', encoding='utf-8') as f:
f.write(content)
RUST_TARGETS = {
"linux": ("x86_64-unknown-linux-gnu", "aarch64-unknown-linux-gnu"),
"windows": ("x86_64-pc-windows-msvc", "aarch64-pc-windows-msvc", "i686-pc-windows-msvc"),
"macos": ("x86_64-apple-darwin", "aarch64-apple-darwin"),
}
def install_rust(rust, target):
"""Install the Rust release Mozilla builds this tree with, and its std for the target"""
targets = ' '.join(f'--target {t}' for t in RUST_TARGETS[target])
run(f'~/.cargo/bin/rustup toolchain install {rust} --profile minimal {targets}')
"""
Preparation
"""
def extract_args():
"""Get version and release from args"""
if len(args) != 2:
sys.stderr.write('error: please specify version and release of camoufox source')
sys.exit(1)
return args[0], args[1]
AVAILABLE_TARGETS = ["linux", "windows", "macos"]
AVAILABLE_ARCHS = ["x86_64", "arm64", "i686"]
def extract_build_target():
"""Get moz_target if passed to BUILD_TARGET environment variable"""
if os.environ.get('BUILD_TARGET'):
target, arch = os.environ['BUILD_TARGET'].split(',')
assert target in AVAILABLE_TARGETS, f"Unsupported target: {target}"
assert arch in AVAILABLE_ARCHS, f"Unsupported architecture: {arch}"
else:
target, arch = "macos", "arm64"
return target, arch
"""
Launcher
"""
if __name__ == "__main__":
# Extract args
VERSION, RELEASE = extract_args()
TARGET, ARCH = extract_build_target()
MOZ_TARGET = get_moz_target(TARGET, ARCH)
# Check if the folder exists
if not os.path.exists(f'camoufox-{VERSION}-{RELEASE}/configure.py'):
sys.stderr.write('error: folder doesn\'t look like a Firefox folder.')
sys.exit(1)
RUST = mozilla_rust_version(f'camoufox-{VERSION}-{RELEASE}')
install_rust(RUST, TARGET)
PGO_PROFILE = pgo.fetch(TARGET, ARCH, VERSION)
# Apply the patches
patcher = Patcher(MOZ_TARGET, TARGET, RUST, PGO_PROFILE)
patcher.camoufox_patches()
sys.exit(0) # ensure 0 exit code