mirror of
https://github.com/daijro/camoufox.git
synced 2026-11-19 00:07:28 +00:00
* Build with cross-language LTO and PGO, as stock Firefox does Without them Gecko's C++ ran 1.3-1.8x stock while JIT-compiled script did not, and a page can time that ratio (JSON.parse over a regexp replace read ~3.3 here, ~2.2 in stock). Stock's recipe is two stages: an instrumented build trains on Mozilla's own pages, then the release links with --enable-lto=cross and --enable-profile-use=cross. The "PGO profile" workflow is stage one: it publishes the merged profile as a release asset, pinned by sha256 in browser/assets/optimized-build.json. ci/pgo.py exports the switches base.mozconfig reads; a local build without them is unchanged. The CI build job pins Rust, checks the pairing, and builds LTO-only with a warning while no profile is pinned for the tree's Firefox. None is pinned for 156.0.1 yet. Rust is pinned to 1.95.0. Cross-language LTO hands Rust bitcode to lld, so rustc's LLVM may not be newer than the bootstrapped clang's. Firefox 156 bootstraps clang 22 (linux64-clang in toolchain/clang.yml) and Mozilla builds it with Rust 1.95.0 (linux64-rust in toolchain/rust.yml), whose LLVM is 22. 1.90.0, the pin for 152, is still 156's minimum but no longer what stock compiles with. ccache is restored and saved in separate steps, so a build that fails late keeps what it compiled, under opt- keys that include the profile. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Link every build with cross-language ThinLTO, from the one mozconfig LTO had entered only the tests.yml build, through an environment variable, so a release would have been tested on an LTO Linux binary and shipped Linux, Windows and macOS binaries built without it. base.mozconfig now has --enable-lto=cross and --enable-rust-simd unconditionally, and every build copies it: the tests.yml build, multibuild.py and the release matrix. Firefox 158's lto-pgo.configure makes --enable-lto=cross full LTO only when PGO and MOZ_AUTOMATION are on and the target is x86_64 Linux or Windows, so without a profile it is ThinLTO across C++ and Rust on every target. That is what stock links outside those two, and PGO is the remaining difference. ./mach configure on 158 with the generated mozconfig gives MOZ_LTO_RUST_CROSS=thin, -Clinker-plugin-lto and an lld linker for linux x86_64/arm64, windows x86_64/i686 (clang-cl, lld-link) and macOS x86_64/arm64. lld cannot read Rust bitcode from an LLVM newer than its own, and the runner's stable Rust can be. scripts/patch.py now installs the Rust release taskcluster/kinds/toolchain/rust.yml aliases as linux64-rust (1.95.0, LLVM 22, for 158's clang 22) and writes RUSTUP_TOOLCHAIN into the mozconfig, so the pin follows each Firefox bump by itself. The build jobs no longer install a system LLVM 18 from apt.llvm.org: mach bootstrap already fetches the clang and lld the tree builds with. The PGO training workflow, ci/pgo.py, the profile pin and the separate ccache keys are removed; PGO needs a profile to pin first. The commented out mozconfig lines go too, and linux.mozconfig, which held only one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document the LTO build: what it costs and where its Rust comes from AGENTS.md and README.md said an incremental build takes about five minutes; with LTO every build, incremental ones included, relinks libxul across C++ and Rust, and ccache cannot cache that link. ci/README.md lists it under cost control. On CI the libxul ThinLTO link took about nine minutes, and the cold build step 57 minutes against 54-99 for recent cold non-LTO builds, since rustc no longer runs its own fat LTO over gkrust. mozilla_rust_version's docstring now says what the pairing is: Rust's LLVM has the bootstrapped clang's major version (1.95.0 has LLVM 22.1.2, 158's clang is 22.1.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Build with stock's PGO profile, and full LTO where stock uses it Stock builds every desktop release with a profile Mozilla trains on that release (taskcluster kind generate-profile), and with full cross-language LTO on x86_64 Linux and Windows. LTO alone left the remaining difference from stock in how Gecko's C++ is optimized, which a page can time against script. scripts/pgo.py fetches, for each target, the profile Mozilla trained for the stock release of the same Firefox: Taskcluster indexes the generate-profile task by the release commit, and the task's chain of trust records the archive's sha256. data/pgo-profiles.json pins all five (Linux arm64 uses the x86_64 Linux profile, as stock's does). `make dir` fetches and verifies the profile, and base.mozconfig builds with it, with full LTO on x86_64 Linux and Windows and ThinLTO elsewhere, as stock's configure picks under MOZ_AUTOMATION. A function a patch changes loses its profile data; configure makes that a warning. A pin for another Firefox fails `make dir` and says to run `scripts/pgo.py pin`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Jake Writer <jake_writer@jobharvest.com>
245 lines
7.8 KiB
Python
245 lines
7.8 KiB
Python
#!/usr/bin/env python3
|
|
|
|
"""
|
|
The script that patches the Firefox source into the Camoufox source.
|
|
Based on LibreWolf's patch script:
|
|
https://gitlab.com/librewolf-community/browser/source/-/blob/main/scripts/librewolf-patches.py
|
|
|
|
Run:
|
|
python3 scripts/patch.py <version> <release>
|
|
"""
|
|
|
|
import os
|
|
import shutil
|
|
import subprocess
|
|
import sys
|
|
from dataclasses import dataclass
|
|
|
|
from _mixin import (
|
|
find_src_dir,
|
|
get_moz_target,
|
|
get_options,
|
|
list_patches,
|
|
mozilla_rust_version,
|
|
run,
|
|
temp_cd,
|
|
)
|
|
import pgo
|
|
|
|
options, args = get_options()
|
|
|
|
"""
|
|
Main patcher functions
|
|
"""
|
|
|
|
|
|
def reset_to_unpatched():
|
|
"""Reset this source repository without discovering a parent repository."""
|
|
if not os.path.exists('.git'):
|
|
return
|
|
|
|
print("Resetting to unpatched state...")
|
|
run('git reset --hard unpatched && ./mach clobber && git clean -fdx')
|
|
|
|
|
|
@dataclass
|
|
class Patcher:
|
|
"""Patch and prepare the Camoufox source"""
|
|
|
|
moz_target: str
|
|
target: str
|
|
rust: str
|
|
pgo_profile: str
|
|
|
|
def camoufox_patches(self):
|
|
"""
|
|
Apply all patches
|
|
"""
|
|
version, release = extract_args()
|
|
with temp_cd(find_src_dir('.', version, release)):
|
|
# Reset only when the source tree has its own local repository.
|
|
reset_to_unpatched()
|
|
|
|
# Re-copy additions and settings after reset
|
|
print("Re-copying additions and settings...")
|
|
run(f'bash ../scripts/copy-additions.sh {version} {release}')
|
|
|
|
# Create the base mozconfig file
|
|
run('cp -v ../assets/base.mozconfig mozconfig')
|
|
# Set cross building target
|
|
print(f'Using target: {self.moz_target}')
|
|
self._update_mozconfig()
|
|
|
|
if not options.mozconfig_only:
|
|
failed_patches = []
|
|
for patch_file in list_patches():
|
|
rejects = self._apply_and_check(patch_file)
|
|
if rejects:
|
|
failed_patches.append((patch_file, rejects))
|
|
|
|
# Report failures
|
|
if failed_patches:
|
|
print('\n' + '='*70)
|
|
print(f'ERROR: {len(failed_patches)} patch(es) failed to apply cleanly:')
|
|
print('='*70)
|
|
for patch_file, rejects in failed_patches:
|
|
print(f'\n{patch_file}:')
|
|
for reject in rejects:
|
|
print(f' - {reject}')
|
|
print('='*70)
|
|
sys.exit(1)
|
|
|
|
print('Complete!')
|
|
|
|
def _apply_and_check(self, patch_file):
|
|
"""
|
|
Apply a patch and check for reject files.
|
|
Returns the reject files and any non-zero exit status, empty if it applied cleanly.
|
|
"""
|
|
import time
|
|
|
|
print(f"\n*** -> patch -p1 -i {patch_file}")
|
|
sys.stdout.flush()
|
|
|
|
# Record time before applying so we only detect .rej files from this patch
|
|
start_time = time.time()
|
|
|
|
# Apply patch interactively - don't capture stdout/stderr at all
|
|
# This allows prompts to show immediately and user can respond
|
|
# --forward flag: skip patches that appear to be already applied
|
|
# --binary flag: preserve line endings (helps with CRLF vs LF differences)
|
|
# -l flag: ignore whitespace differences
|
|
result = subprocess.run(
|
|
['patch', '-p1', '--forward', '-l', '--binary', '-i', patch_file],
|
|
stdin=sys.stdin,
|
|
stdout=sys.stdout,
|
|
stderr=sys.stderr,
|
|
text=True
|
|
)
|
|
|
|
# After patch completes, search for any .rej files created during this patch
|
|
rejects = []
|
|
for root, dirs, files in os.walk('.'):
|
|
for file in files:
|
|
if file.endswith('.rej'):
|
|
reject_path = os.path.join(root, file)
|
|
if os.path.exists(reject_path):
|
|
# Only include if created after this patch started
|
|
if os.path.getmtime(reject_path) >= start_time:
|
|
rejects.append(reject_path)
|
|
|
|
# Clean up .rej files so they don't interfere with subsequent patches
|
|
for rej in rejects:
|
|
os.remove(rej)
|
|
|
|
if result.returncode != 0:
|
|
rejects.append(f'patch exited with status {result.returncode}')
|
|
return rejects
|
|
|
|
def _update_mozconfig(self):
|
|
"""
|
|
Helper for adding additional mozconfig code from assets/<target>.mozconfig
|
|
"""
|
|
mozconfig_backup = "mozconfig.backup"
|
|
mozconfig = "mozconfig"
|
|
|
|
# Create backup if it doesn't exist
|
|
if not os.path.exists(mozconfig_backup):
|
|
if os.path.exists(mozconfig):
|
|
shutil.copy2(mozconfig, mozconfig_backup)
|
|
else:
|
|
with open(mozconfig_backup, 'w', encoding='utf-8') as f:
|
|
pass
|
|
|
|
# Read backup content
|
|
with open(mozconfig_backup, 'r', encoding='utf-8') as f:
|
|
content = f.read()
|
|
|
|
# Set first: base.mozconfig picks stock's LTO mode for the target and
|
|
# reads the profile. configure resolves rustc and cargo through rustup,
|
|
# which honours RUSTUP_TOOLCHAIN.
|
|
content = (f"CAMOUFOX_TARGET={self.moz_target}\n"
|
|
f"CAMOUFOX_PGO_PROFILE={self.pgo_profile}\n"
|
|
"ac_add_options --target=$CAMOUFOX_TARGET\n"
|
|
f"export RUSTUP_TOOLCHAIN={self.rust}\n\n") + content
|
|
|
|
# Add target-specific mozconfig if it exists
|
|
target_mozconfig = os.path.join("..", "assets", f"{self.target}.mozconfig")
|
|
if os.path.exists(target_mozconfig):
|
|
with open(target_mozconfig, 'r', encoding='utf-8') as f:
|
|
content += f.read()
|
|
|
|
# Update mozconfig
|
|
print(f"-> Updating mozconfig, target is {self.moz_target}")
|
|
with open(mozconfig, 'w', encoding='utf-8') as f:
|
|
f.write(content)
|
|
|
|
|
|
RUST_TARGETS = {
|
|
"linux": ("x86_64-unknown-linux-gnu", "aarch64-unknown-linux-gnu"),
|
|
"windows": ("x86_64-pc-windows-msvc", "aarch64-pc-windows-msvc", "i686-pc-windows-msvc"),
|
|
"macos": ("x86_64-apple-darwin", "aarch64-apple-darwin"),
|
|
}
|
|
|
|
|
|
def install_rust(rust, target):
|
|
"""Install the Rust release Mozilla builds this tree with, and its std for the target"""
|
|
targets = ' '.join(f'--target {t}' for t in RUST_TARGETS[target])
|
|
run(f'~/.cargo/bin/rustup toolchain install {rust} --profile minimal {targets}')
|
|
|
|
|
|
"""
|
|
Preparation
|
|
"""
|
|
|
|
|
|
def extract_args():
|
|
"""Get version and release from args"""
|
|
if len(args) != 2:
|
|
sys.stderr.write('error: please specify version and release of camoufox source')
|
|
sys.exit(1)
|
|
return args[0], args[1]
|
|
|
|
|
|
AVAILABLE_TARGETS = ["linux", "windows", "macos"]
|
|
AVAILABLE_ARCHS = ["x86_64", "arm64", "i686"]
|
|
|
|
|
|
def extract_build_target():
|
|
"""Get moz_target if passed to BUILD_TARGET environment variable"""
|
|
|
|
if os.environ.get('BUILD_TARGET'):
|
|
target, arch = os.environ['BUILD_TARGET'].split(',')
|
|
assert target in AVAILABLE_TARGETS, f"Unsupported target: {target}"
|
|
assert arch in AVAILABLE_ARCHS, f"Unsupported architecture: {arch}"
|
|
else:
|
|
target, arch = "macos", "arm64"
|
|
return target, arch
|
|
|
|
|
|
"""
|
|
Launcher
|
|
"""
|
|
|
|
if __name__ == "__main__":
|
|
# Extract args
|
|
VERSION, RELEASE = extract_args()
|
|
|
|
TARGET, ARCH = extract_build_target()
|
|
MOZ_TARGET = get_moz_target(TARGET, ARCH)
|
|
|
|
# Check if the folder exists
|
|
if not os.path.exists(f'camoufox-{VERSION}-{RELEASE}/configure.py'):
|
|
sys.stderr.write('error: folder doesn\'t look like a Firefox folder.')
|
|
sys.exit(1)
|
|
|
|
RUST = mozilla_rust_version(f'camoufox-{VERSION}-{RELEASE}')
|
|
install_rust(RUST, TARGET)
|
|
PGO_PROFILE = pgo.fetch(TARGET, ARCH, VERSION)
|
|
|
|
# Apply the patches
|
|
patcher = Patcher(MOZ_TARGET, TARGET, RUST, PGO_PROFILE)
|
|
patcher.camoufox_patches()
|
|
|
|
sys.exit(0) # ensure 0 exit code
|