Files
6e9a1de13b Build with cross-language LTO and PGO, as stock Firefox does (#886)
* Build with cross-language LTO and PGO, as stock Firefox does

Without them Gecko's C++ ran 1.3-1.8x stock while JIT-compiled script
did not, and a page can time that ratio (JSON.parse over a regexp
replace read ~3.3 here, ~2.2 in stock). Stock's recipe is two stages:
an instrumented build trains on Mozilla's own pages, then the release
links with --enable-lto=cross and --enable-profile-use=cross.

The "PGO profile" workflow is stage one: it publishes the merged
profile as a release asset, pinned by sha256 in
browser/assets/optimized-build.json. ci/pgo.py exports the switches
base.mozconfig reads; a local build without them is unchanged. The CI
build job pins Rust, checks the pairing, and builds LTO-only with a
warning while no profile is pinned for the tree's Firefox. None is
pinned for 156.0.1 yet.

Rust is pinned to 1.95.0. Cross-language LTO hands Rust bitcode to lld,
so rustc's LLVM may not be newer than the bootstrapped clang's. Firefox
156 bootstraps clang 22 (linux64-clang in toolchain/clang.yml) and
Mozilla builds it with Rust 1.95.0 (linux64-rust in toolchain/rust.yml),
whose LLVM is 22. 1.90.0, the pin for 152, is still 156's minimum but
no longer what stock compiles with.

ccache is restored and saved in separate steps, so a build that fails
late keeps what it compiled, under opt- keys that include the profile.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Link every build with cross-language ThinLTO, from the one mozconfig

LTO had entered only the tests.yml build, through an environment
variable, so a release would have been tested on an LTO Linux binary and
shipped Linux, Windows and macOS binaries built without it. base.mozconfig
now has --enable-lto=cross and --enable-rust-simd unconditionally, and
every build copies it: the tests.yml build, multibuild.py and the
release matrix.

Firefox 158's lto-pgo.configure makes --enable-lto=cross full LTO only
when PGO and MOZ_AUTOMATION are on and the target is x86_64 Linux or
Windows, so without a profile it is ThinLTO across C++ and Rust on every
target. That is what stock links outside those two, and PGO is the
remaining difference. ./mach configure on 158 with the generated mozconfig
gives MOZ_LTO_RUST_CROSS=thin, -Clinker-plugin-lto and an lld linker for
linux x86_64/arm64, windows x86_64/i686 (clang-cl, lld-link) and macOS
x86_64/arm64.

lld cannot read Rust bitcode from an LLVM newer than its own, and the
runner's stable Rust can be. scripts/patch.py now installs the Rust release
taskcluster/kinds/toolchain/rust.yml aliases as linux64-rust (1.95.0,
LLVM 22, for 158's clang 22) and writes RUSTUP_TOOLCHAIN into the
mozconfig, so the pin follows each Firefox bump by itself. The build jobs
no longer install a system LLVM 18 from apt.llvm.org: mach bootstrap
already fetches the clang and lld the tree builds with.

The PGO training workflow, ci/pgo.py, the profile pin and the separate
ccache keys are removed; PGO needs a profile to pin first. The commented
out mozconfig lines go too, and linux.mozconfig, which held only one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Document the LTO build: what it costs and where its Rust comes from

AGENTS.md and README.md said an incremental build takes about five
minutes; with LTO every build, incremental ones included, relinks
libxul across C++ and Rust, and ccache cannot cache that link. ci/README.md
lists it under cost control. On CI the libxul ThinLTO link took about
nine minutes, and the cold build step 57 minutes against 54-99 for
recent cold non-LTO builds, since rustc no longer runs its own fat LTO
over gkrust.

mozilla_rust_version's docstring now says what the pairing is: Rust's
LLVM has the bootstrapped clang's major version (1.95.0 has LLVM 22.1.2,
158's clang is 22.1.8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Build with stock's PGO profile, and full LTO where stock uses it

Stock builds every desktop release with a profile Mozilla trains on that
release (taskcluster kind generate-profile), and with full cross-language LTO
on x86_64 Linux and Windows. LTO alone left the remaining difference from stock
in how Gecko's C++ is optimized, which a page can time against script.

scripts/pgo.py fetches, for each target, the profile Mozilla trained for the
stock release of the same Firefox: Taskcluster indexes the generate-profile
task by the release commit, and the task's chain of trust records the
archive's sha256. data/pgo-profiles.json pins all five (Linux arm64 uses the
x86_64 Linux profile, as stock's does). `make dir` fetches and verifies the
profile, and base.mozconfig builds with it, with full LTO on x86_64 Linux and
Windows and ThinLTO elsewhere, as stock's configure picks under MOZ_AUTOMATION.
A function a patch changes loses its profile data; configure makes that a
warning. A pin for another Firefox fails `make dir` and says to run
`scripts/pgo.py pin`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Writer <jake_writer@jobharvest.com>
2026-10-31 06:20:35 +00:00

140 lines
4.3 KiB
Python

#!/usr/bin/env python3
"""
Common functions used across the Camoufox build system.
Not meant to be called directly.
"""
import contextlib
import fnmatch
import optparse
import os
import re
import sys
import time
start_time = time.time()
@contextlib.contextmanager
def temp_cd(path):
"""Temporarily change to a different working directory"""
_old_cwd = os.getcwd()
abs_path = os.path.abspath(path)
assert os.path.exists(abs_path), f'{abs_path} does not exist.'
os.chdir(abs_path)
try:
yield
finally:
os.chdir(_old_cwd)
def get_options():
"""Get options"""
parser = optparse.OptionParser()
parser.add_option('--mozconfig-only', dest='mozconfig_only', default=False, action="store_true")
return parser.parse_args()
def find_src_dir(root_dir='.', version=None, release=None):
"""Get the source directory"""
if version and release:
name = os.path.join(root_dir, f'camoufox-{version}-{release}')
assert os.path.exists(name), f'{name} does not exist.'
return name
folders = os.listdir(root_dir)
for folder in folders:
if os.path.isdir(folder) and folder.startswith('camoufox-'):
return os.path.join(root_dir, folder)
raise FileNotFoundError('No camoufox-* folder found')
def get_moz_target(target, arch):
"""Get moz_target from target and arch"""
if arch == "i686" and target != "windows":
raise ValueError(f"Unsupported architecture for {target}: {arch}")
if target == "linux":
return "aarch64-unknown-linux-gnu" if arch == "arm64" else f"{arch}-pc-linux-gnu"
if target == "windows":
return f"{arch}-pc-windows-msvc"
if target == "macos":
return "aarch64-apple-darwin" if arch == "arm64" else f"{arch}-apple-darwin"
raise ValueError(f"Unsupported target: {target}")
def mozilla_rust_version(src_dir):
"""The Rust release Mozilla builds this Firefox with: the toolchain its
taskcluster aliases as linux64-rust. Its LLVM is the major version of the
clang `mach bootstrap` fetches for the same tree, which cross-language LTO
needs: lld cannot read Rust bitcode from a newer LLVM than its own."""
path = os.path.join(src_dir, 'taskcluster', 'kinds', 'toolchain', 'rust.yml')
with open(path, encoding='utf-8') as f:
text = f.read()
versions = set()
# One top-level entry per toolchain; the alias is either a plain value or
# a by-project mapping whose default is the alias.
for entry in re.split(r'^(?=\S)', text, flags=re.MULTILINE):
if re.search(r'^\s+(?:toolchain-alias|default): linux64-rust$', entry, re.MULTILINE):
versions.update(re.findall(r"'--channel', '(\d+\.\d+\.\d+)'", entry))
if len(versions) != 1:
raise ValueError(f'{path}: expected one Rust release aliased linux64-rust, '
f'found {sorted(versions) or "none"}')
return versions.pop()
def list_files(root_dir, suffix):
"""List files in a directory"""
for root, _, files in os.walk(root_dir):
for file in fnmatch.filter(files, suffix):
full_path = os.path.join(root, file)
relative_path = os.path.relpath(full_path, root_dir)
yield os.path.join(root_dir, relative_path).replace('\\', '/')
def list_patches(root_dir='../patches', suffix='*.patch'):
"""List all patch files"""
return sorted(list_files(root_dir, suffix), key=os.path.basename)
def script_exit(statuscode):
"""Exit the script"""
if (time.time() - start_time) > 60:
# print elapsed time
elapsed = time.strftime("%H:%M:%S", time.gmtime(time.time() - start_time))
print(f"\n\aElapsed time: {elapsed}")
sys.stdout.flush()
sys.exit(statuscode)
def run(cmd, exit_on_fail=True, do_print=True):
"""Run a command"""
if not cmd:
return
if do_print:
print(cmd)
sys.stdout.flush()
retval = os.system(cmd)
if retval != 0 and exit_on_fail:
print(f"fatal error: command '{cmd}' failed")
sys.stdout.flush()
script_exit(1)
return retval
__all__ = [
'get_moz_target',
'list_patches',
'mozilla_rust_version',
'run',
'script_exit',
'temp_cd',
'get_options',
]
if __name__ == '__main__':
print('This is a module, not meant to be called directly.')
sys.exit(1)