mirror of
https://github.com/daijro/camoufox.git
synced 2026-11-19 00:07:28 +00:00
* Build with cross-language LTO and PGO, as stock Firefox does Without them Gecko's C++ ran 1.3-1.8x stock while JIT-compiled script did not, and a page can time that ratio (JSON.parse over a regexp replace read ~3.3 here, ~2.2 in stock). Stock's recipe is two stages: an instrumented build trains on Mozilla's own pages, then the release links with --enable-lto=cross and --enable-profile-use=cross. The "PGO profile" workflow is stage one: it publishes the merged profile as a release asset, pinned by sha256 in browser/assets/optimized-build.json. ci/pgo.py exports the switches base.mozconfig reads; a local build without them is unchanged. The CI build job pins Rust, checks the pairing, and builds LTO-only with a warning while no profile is pinned for the tree's Firefox. None is pinned for 156.0.1 yet. Rust is pinned to 1.95.0. Cross-language LTO hands Rust bitcode to lld, so rustc's LLVM may not be newer than the bootstrapped clang's. Firefox 156 bootstraps clang 22 (linux64-clang in toolchain/clang.yml) and Mozilla builds it with Rust 1.95.0 (linux64-rust in toolchain/rust.yml), whose LLVM is 22. 1.90.0, the pin for 152, is still 156's minimum but no longer what stock compiles with. ccache is restored and saved in separate steps, so a build that fails late keeps what it compiled, under opt- keys that include the profile. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Link every build with cross-language ThinLTO, from the one mozconfig LTO had entered only the tests.yml build, through an environment variable, so a release would have been tested on an LTO Linux binary and shipped Linux, Windows and macOS binaries built without it. base.mozconfig now has --enable-lto=cross and --enable-rust-simd unconditionally, and every build copies it: the tests.yml build, multibuild.py and the release matrix. Firefox 158's lto-pgo.configure makes --enable-lto=cross full LTO only when PGO and MOZ_AUTOMATION are on and the target is x86_64 Linux or Windows, so without a profile it is ThinLTO across C++ and Rust on every target. That is what stock links outside those two, and PGO is the remaining difference. ./mach configure on 158 with the generated mozconfig gives MOZ_LTO_RUST_CROSS=thin, -Clinker-plugin-lto and an lld linker for linux x86_64/arm64, windows x86_64/i686 (clang-cl, lld-link) and macOS x86_64/arm64. lld cannot read Rust bitcode from an LLVM newer than its own, and the runner's stable Rust can be. scripts/patch.py now installs the Rust release taskcluster/kinds/toolchain/rust.yml aliases as linux64-rust (1.95.0, LLVM 22, for 158's clang 22) and writes RUSTUP_TOOLCHAIN into the mozconfig, so the pin follows each Firefox bump by itself. The build jobs no longer install a system LLVM 18 from apt.llvm.org: mach bootstrap already fetches the clang and lld the tree builds with. The PGO training workflow, ci/pgo.py, the profile pin and the separate ccache keys are removed; PGO needs a profile to pin first. The commented out mozconfig lines go too, and linux.mozconfig, which held only one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Document the LTO build: what it costs and where its Rust comes from AGENTS.md and README.md said an incremental build takes about five minutes; with LTO every build, incremental ones included, relinks libxul across C++ and Rust, and ccache cannot cache that link. ci/README.md lists it under cost control. On CI the libxul ThinLTO link took about nine minutes, and the cold build step 57 minutes against 54-99 for recent cold non-LTO builds, since rustc no longer runs its own fat LTO over gkrust. mozilla_rust_version's docstring now says what the pairing is: Rust's LLVM has the bootstrapped clang's major version (1.95.0 has LLVM 22.1.2, 158's clang is 22.1.8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Build with stock's PGO profile, and full LTO where stock uses it Stock builds every desktop release with a profile Mozilla trains on that release (taskcluster kind generate-profile), and with full cross-language LTO on x86_64 Linux and Windows. LTO alone left the remaining difference from stock in how Gecko's C++ is optimized, which a page can time against script. scripts/pgo.py fetches, for each target, the profile Mozilla trained for the stock release of the same Firefox: Taskcluster indexes the generate-profile task by the release commit, and the task's chain of trust records the archive's sha256. data/pgo-profiles.json pins all five (Linux arm64 uses the x86_64 Linux profile, as stock's does). `make dir` fetches and verifies the profile, and base.mozconfig builds with it, with full LTO on x86_64 Linux and Windows and ThinLTO elsewhere, as stock's configure picks under MOZ_AUTOMATION. A function a patch changes loses its profile data; configure makes that a warning. A pin for another Firefox fails `make dir` and says to run `scripts/pgo.py pin`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Jake Writer <jake_writer@jobharvest.com>
140 lines
4.3 KiB
Python
140 lines
4.3 KiB
Python
#!/usr/bin/env python3
|
|
|
|
"""
|
|
Common functions used across the Camoufox build system.
|
|
Not meant to be called directly.
|
|
"""
|
|
|
|
import contextlib
|
|
import fnmatch
|
|
import optparse
|
|
import os
|
|
import re
|
|
import sys
|
|
import time
|
|
|
|
start_time = time.time()
|
|
|
|
|
|
@contextlib.contextmanager
|
|
def temp_cd(path):
|
|
"""Temporarily change to a different working directory"""
|
|
_old_cwd = os.getcwd()
|
|
abs_path = os.path.abspath(path)
|
|
assert os.path.exists(abs_path), f'{abs_path} does not exist.'
|
|
os.chdir(abs_path)
|
|
|
|
try:
|
|
yield
|
|
finally:
|
|
os.chdir(_old_cwd)
|
|
|
|
|
|
def get_options():
|
|
"""Get options"""
|
|
parser = optparse.OptionParser()
|
|
parser.add_option('--mozconfig-only', dest='mozconfig_only', default=False, action="store_true")
|
|
return parser.parse_args()
|
|
|
|
|
|
def find_src_dir(root_dir='.', version=None, release=None):
|
|
"""Get the source directory"""
|
|
if version and release:
|
|
name = os.path.join(root_dir, f'camoufox-{version}-{release}')
|
|
assert os.path.exists(name), f'{name} does not exist.'
|
|
return name
|
|
folders = os.listdir(root_dir)
|
|
for folder in folders:
|
|
if os.path.isdir(folder) and folder.startswith('camoufox-'):
|
|
return os.path.join(root_dir, folder)
|
|
raise FileNotFoundError('No camoufox-* folder found')
|
|
|
|
|
|
def get_moz_target(target, arch):
|
|
"""Get moz_target from target and arch"""
|
|
if arch == "i686" and target != "windows":
|
|
raise ValueError(f"Unsupported architecture for {target}: {arch}")
|
|
if target == "linux":
|
|
return "aarch64-unknown-linux-gnu" if arch == "arm64" else f"{arch}-pc-linux-gnu"
|
|
if target == "windows":
|
|
return f"{arch}-pc-windows-msvc"
|
|
if target == "macos":
|
|
return "aarch64-apple-darwin" if arch == "arm64" else f"{arch}-apple-darwin"
|
|
raise ValueError(f"Unsupported target: {target}")
|
|
|
|
|
|
def mozilla_rust_version(src_dir):
|
|
"""The Rust release Mozilla builds this Firefox with: the toolchain its
|
|
taskcluster aliases as linux64-rust. Its LLVM is the major version of the
|
|
clang `mach bootstrap` fetches for the same tree, which cross-language LTO
|
|
needs: lld cannot read Rust bitcode from a newer LLVM than its own."""
|
|
path = os.path.join(src_dir, 'taskcluster', 'kinds', 'toolchain', 'rust.yml')
|
|
with open(path, encoding='utf-8') as f:
|
|
text = f.read()
|
|
versions = set()
|
|
# One top-level entry per toolchain; the alias is either a plain value or
|
|
# a by-project mapping whose default is the alias.
|
|
for entry in re.split(r'^(?=\S)', text, flags=re.MULTILINE):
|
|
if re.search(r'^\s+(?:toolchain-alias|default): linux64-rust$', entry, re.MULTILINE):
|
|
versions.update(re.findall(r"'--channel', '(\d+\.\d+\.\d+)'", entry))
|
|
if len(versions) != 1:
|
|
raise ValueError(f'{path}: expected one Rust release aliased linux64-rust, '
|
|
f'found {sorted(versions) or "none"}')
|
|
return versions.pop()
|
|
|
|
|
|
def list_files(root_dir, suffix):
|
|
"""List files in a directory"""
|
|
for root, _, files in os.walk(root_dir):
|
|
for file in fnmatch.filter(files, suffix):
|
|
full_path = os.path.join(root, file)
|
|
relative_path = os.path.relpath(full_path, root_dir)
|
|
yield os.path.join(root_dir, relative_path).replace('\\', '/')
|
|
|
|
|
|
def list_patches(root_dir='../patches', suffix='*.patch'):
|
|
"""List all patch files"""
|
|
return sorted(list_files(root_dir, suffix), key=os.path.basename)
|
|
|
|
|
|
def script_exit(statuscode):
|
|
"""Exit the script"""
|
|
if (time.time() - start_time) > 60:
|
|
# print elapsed time
|
|
elapsed = time.strftime("%H:%M:%S", time.gmtime(time.time() - start_time))
|
|
print(f"\n\aElapsed time: {elapsed}")
|
|
sys.stdout.flush()
|
|
|
|
sys.exit(statuscode)
|
|
|
|
|
|
def run(cmd, exit_on_fail=True, do_print=True):
|
|
"""Run a command"""
|
|
if not cmd:
|
|
return
|
|
if do_print:
|
|
print(cmd)
|
|
sys.stdout.flush()
|
|
retval = os.system(cmd)
|
|
if retval != 0 and exit_on_fail:
|
|
print(f"fatal error: command '{cmd}' failed")
|
|
sys.stdout.flush()
|
|
script_exit(1)
|
|
return retval
|
|
|
|
|
|
__all__ = [
|
|
'get_moz_target',
|
|
'list_patches',
|
|
'mozilla_rust_version',
|
|
'run',
|
|
'script_exit',
|
|
'temp_cd',
|
|
'get_options',
|
|
]
|
|
|
|
|
|
if __name__ == '__main__':
|
|
print('This is a module, not meant to be called directly.')
|
|
sys.exit(1)
|