Files
Jake WriterandClaude Fable 5.2 340cf16170 CI: build pull requests with the release's command, and preflight its other targets (#1055)
Four prerelease cuts failed on commits whose tests had passed (runs
41280578561, 43352392915, 43395652803, 43631788944), each in a step only
release.yml ran: midl.exe under Wine (twice), verify-fonts.py, and
make set-target. tests.yml built with `make build`; release.yml built with
multibuild.py after installing its own packages. A step only a release runs
is run for the first time by a release.

- make set-target (scripts/patch.py --mozconfig-only) writes the mozconfig
  and nothing else. It used to reset the tree and copy the additions over
  it, which in a tree without a git repository undid any patch hunk to an
  addition while leaving _READY in place, so the build went on with the
  reverted files. The mozconfig it writes is byte for byte the same.
- ci.run_build runs multibuild.py for one target. tests.yml's build job runs
  it for linux x86_64 and release.yml's for every target, so the pull
  request retargets, builds and packages exactly as a release does.
- multibuild.py --preflight (make preflight, scripts/preflight.py) checks a
  target without building it: configure, the pre-export, export and
  pre-compile tiers, then the object of every source file a patch or an
  addition touches. tests.yml preflights the five targets only a release
  builds, beside the build; the gate requires them. A target that passed is
  not run again for the same compiled inputs, except by the scheduled run.
- .github/actions/build-host frees the disk, installs the packages and adds
  the swap for all three jobs, so no build depends on a package only one of
  them installs.
- browser/multibuild.py is a browser input: changing it rebuilds.

The preflight does not link or package, so a link error on a cross-compiled
target is still first seen by a release.

Tests: ci/tests/test_set_target.py fails on main ('as added' != 'as patched');
test_preflight.py covers the source-to-object mapping and that an object left
by an earlier run is compiled again; test_ci.py holds the preflight matrix to
release.yml's and all three jobs to build-host; the tribal rule
one-build-entry-point holds all three to ci.run_build.

Run locally on Firefox 159 with CI's tree (make setup-minimal, make dir):
- All five preflights pass, in 2 to 6.5 minutes each on 16 cores.
- With the four additions as run 43631788944's tree had them, the linux
  arm64 preflight fails with that run's error,
  SkBitmapProcState_opts.cpp:41:13: use of undeclared identifier
  'sk_cpu_emulates_x86'.
- python3 -m ci.run_build --target linux --arch arm64, the release's
  command for a target that run never linked, builds and packages
  camoufox-159.0-beta.35-lin.arm64.zip in 41 minutes.
No workflow here has run on a GitHub runner yet; this pull request's run is
the first.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-15 06:54:41 +00:00

246 lines
9.8 KiB
Python

#!/usr/bin/env python3
"""Split the browser's inputs into what needs a compiler and what does not.
Juggler is mostly JavaScript, and JavaScript does not need libxul relinked. But
a full build is what every change to `browser/additions/` used to cost: 24
minutes, of which ccache reported 98.63% hits -- so almost none of it was
compiling C++. It was Rust, linking, and packaging, none of which a `.js` file
affects.
The saving is real and the trap is sharp, so this module is deliberately small
and deliberately paranoid.
**The hash IS the classification.** There is no "did only JS change?" diff here,
because a diff answers the wrong question -- it compares against the pull
request's base, while what matters is whether the cached browser was built from
the same native sources. So: hash every input that can change compiled output.
If that hash matches a cached browser, the compiled half is identical *by
construction*, whatever the diff says, and the resources can simply be laid over
it.
**Two things it would be easy to get wrong, and how they are closed:**
`browser/additions/juggler/` is not all JavaScript. It also holds the
screencast encoder and the remote-debugging pipe -- 5 .cpp, 5 .h, 2 .idl, 3
components.conf, 4 moz.build -- which are compiled into libxul. Only the files
its moz.build lists in FINAL_TARGET_FILES are treated as resources.
Everything else, including anything with an extension nobody has thought about
yet, counts as native and forces a build. Fail closed.
The source-to-destination mapping is per-file, not a prefix rule. moz.build
installs `content/FrameTree.js` at `modules/juggler/content/FrameTree.js`
(depth preserved) but `content/JugglerFrameChild.sys.mjs` at
`modules/juggler/JugglerFrameChild.sys.mjs` (a level dropped). Two files in
the same source directory land at different depths. Assuming a prefix would
write one of them to the wrong path, leave the old copy in place, and run a
browser with stale Juggler -- while every suite reported green, because the
browser works fine, it is just not the one under review. So the mapping is
read from moz.build, never inferred.
That moz.build is itself native: it decides both the mapping and what is
packaged at all, so changing it must force a real build rather than a
re-overlay.
Run:
python3 -m ci.browser_inputs --digest # the native-inputs hash
python3 -m ci.browser_inputs --list # what feeds that hash
python3 -m ci.browser_inputs --resources # source -> path in the dist
python3 -m ci.browser_inputs --overlay DIR # lay resources over a dist/bin
"""
from __future__ import annotations
import argparse
import hashlib
import re
import shutil
import sys
from pathlib import Path
from typing import Dict, List, Optional
REPO_ROOT = Path(__file__).resolve().parent.parent
# The same set `resolve` greps to decide whether the browser changed at all.
# Kept in step by ci/tests/test_ci.py, because a path that can change the binary
# and is not hashed here would be served a stale browser. The fontconfig files
# are not compiled, but `make package-*` ships them, so they pair a release.
BROWSER_DIRS = (
"browser/patches",
"browser/additions",
"browser/settings",
"browser/assets",
"browser/scripts",
"browser/bundle/fontconfig",
)
BROWSER_FILES = ("browser/upstream.sh", "browser/Makefile", "browser/multibuild.py")
# Tools under browser/scripts/ that no build runs, so touching one must not
# invalidate a cached browser and cost an hour of compiling. ci/tests/test_ci.py
# checks each entry against BUILD_ENTRY_POINTS: excluding a script a build does
# run would serve a stale binary to every suite downstream.
NON_NATIVE_SCRIPTS = frozenset(
{
"browser/scripts/_font_bundle.py",
"browser/scripts/check-input-dispatch.py",
"browser/scripts/cursor-demo.py",
"browser/scripts/gen-contentaccessible-manifest.py",
"browser/scripts/gen-font-groups.py",
"browser/scripts/gen-fonts-json.py",
"browser/scripts/verify-fonts.py",
}
)
# Where a build can reach a script from. Used by the test above, not here.
BUILD_ENTRY_POINTS = (
"browser/Makefile",
"browser/multibuild.py",
"browser/scripts/patch.py",
"browser/scripts/package.py",
"browser/scripts/copy-additions.sh",
"browser/scripts/_mixin.py",
)
JUGGLER = Path("browser") / "additions" / "juggler"
MOZ_BUILD = JUGGLER / "moz.build"
# `FINAL_TARGET_FILES.modules.juggler.content += [ "content/main.js", ... ]`: the
# attribute path is the destination directory, each string a source file.
_FILES = re.compile(r"^FINAL_TARGET_FILES\.([\w.]+)\s*\+=\s*\[(.*?)\]", re.M | re.S)
_STRING = re.compile(r'"([^"]+)"')
def resource_entries(root: Optional[Path] = None) -> Dict[str, str]:
"""{repo-relative source: path inside the built dist}, straight from moz.build.
FINAL_TARGET_FILES installs each file under its own name in the directory
the attribute path names: `FINAL_TARGET_FILES.modules.juggler.content`
puts `content/main.js` at `dist/bin/modules/juggler/content/main.js`.
"""
root = root or REPO_ROOT
text = (root / MOZ_BUILD).read_text(encoding="utf-8")
entries: Dict[str, str] = {}
for target, body in _FILES.findall(text):
directory = target.replace(".", "/")
for source in _STRING.findall(body):
entries[str(JUGGLER / source)] = f"{directory}/{Path(source).name}"
return entries
def resource_sources(root: Optional[Path] = None) -> set:
"""Repo-relative paths that are packaged as-is and never compiled."""
return set(resource_entries(root))
def native_inputs(root: Optional[Path] = None) -> List[str]:
"""Every browser input that is not a packaged resource, sorted.
Anything unrecognised lands here rather than being skipped: a new file type
under browser/additions/ forces a build until somebody decides otherwise,
which is the safe direction to be wrong in.
"""
root = root or REPO_ROOT
resources = resource_sources(root)
found: List[str] = []
for name in BROWSER_DIRS:
base = root / name
if not base.is_dir():
continue
for path in base.rglob("*"):
if not path.is_file():
continue
rel = path.relative_to(root).as_posix()
if rel not in resources and rel not in NON_NATIVE_SCRIPTS:
found.append(rel)
for name in BROWSER_FILES:
if (root / name).is_file():
found.append(name)
return sorted(found)
def native_digest(root: Optional[Path] = None) -> str:
"""A hash of the compiled browser's inputs. Same hash, same binary."""
root = root or REPO_ROOT
digest = hashlib.sha256()
for rel in native_inputs(root):
digest.update(rel.encode("utf-8"))
digest.update(b"\0")
digest.update(hashlib.sha256((root / rel).read_bytes()).digest())
return digest.hexdigest()[:32]
def source_inputs(root: Optional[Path] = None) -> List[str]:
"""Every file that goes into the browser, compiled or packaged, sorted."""
root = root or REPO_ROOT
return sorted(set(native_inputs(root)) | resource_sources(root))
def source_digest(root: Optional[Path] = None) -> str:
"""A hash of everything the browser is built from, except its release number.
This is what pairs a library release with a browser release: two commits
with the same source digest produce the same browser, so a library built
from either may name that browser's release. The release number is left
out because the release workflow writes it into the build's working tree
from the release tag (ci/release.py set-build), and that must not make the
browser look different from the main commit it was built from.
"""
root = root or REPO_ROOT
digest = hashlib.sha256()
for rel in source_inputs(root):
data = (root / rel).read_bytes()
if rel == "browser/upstream.sh":
data = b"\n".join(
line for line in data.splitlines() if not line.strip().startswith(b"release=")
)
digest.update(rel.encode("utf-8"))
digest.update(b"\0")
digest.update(hashlib.sha256(data).digest())
return digest.hexdigest()[:32]
def overlay(dist_bin: Path, root: Optional[Path] = None) -> List[str]:
"""Lay the current resources over an already-built dist/bin.
Every entry is written, not just the changed ones: the set is then always
exactly what moz.build says, so a stale file cannot survive. (A resource
*removed* from moz.build is handled by moz.build being a native input --
that forces a real build rather than an overlay.)
"""
root = root or REPO_ROOT
written = []
for source, destination in sorted(resource_entries(root).items()):
target = dist_bin / destination
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(root / source, target)
written.append(destination)
return written
def main(argv: Optional[List[str]] = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--digest", action="store_true")
parser.add_argument("--source-digest", action="store_true")
parser.add_argument("--list", action="store_true")
parser.add_argument("--resources", action="store_true")
parser.add_argument("--overlay", type=Path, metavar="DIST_BIN")
args = parser.parse_args(argv)
if args.digest:
print(native_digest())
if args.source_digest:
print(source_digest())
if args.list:
for rel in native_inputs():
print(rel)
if args.resources:
for source, destination in sorted(resource_entries().items()):
print(f"{source} -> {destination}")
if args.overlay:
written = overlay(args.overlay)
print(f"overlaid {len(written)} resource(s) onto {args.overlay}")
return 0
if __name__ == "__main__":
sys.exit(main())