mirror of
https://github.com/daijro/camoufox.git
synced 2026-11-19 00:07:28 +00:00
Four prerelease cuts failed on commits whose tests had passed (runs
41280578561, 43352392915, 43395652803, 43631788944), each in a step only
release.yml ran: midl.exe under Wine (twice), verify-fonts.py, and
make set-target. tests.yml built with `make build`; release.yml built with
multibuild.py after installing its own packages. A step only a release runs
is run for the first time by a release.
- make set-target (scripts/patch.py --mozconfig-only) writes the mozconfig
and nothing else. It used to reset the tree and copy the additions over
it, which in a tree without a git repository undid any patch hunk to an
addition while leaving _READY in place, so the build went on with the
reverted files. The mozconfig it writes is byte for byte the same.
- ci.run_build runs multibuild.py for one target. tests.yml's build job runs
it for linux x86_64 and release.yml's for every target, so the pull
request retargets, builds and packages exactly as a release does.
- multibuild.py --preflight (make preflight, scripts/preflight.py) checks a
target without building it: configure, the pre-export, export and
pre-compile tiers, then the object of every source file a patch or an
addition touches. tests.yml preflights the five targets only a release
builds, beside the build; the gate requires them. A target that passed is
not run again for the same compiled inputs, except by the scheduled run.
- .github/actions/build-host frees the disk, installs the packages and adds
the swap for all three jobs, so no build depends on a package only one of
them installs.
- browser/multibuild.py is a browser input: changing it rebuilds.
The preflight does not link or package, so a link error on a cross-compiled
target is still first seen by a release.
Tests: ci/tests/test_set_target.py fails on main ('as added' != 'as patched');
test_preflight.py covers the source-to-object mapping and that an object left
by an earlier run is compiled again; test_ci.py holds the preflight matrix to
release.yml's and all three jobs to build-host; the tribal rule
one-build-entry-point holds all three to ci.run_build.
Run locally on Firefox 159 with CI's tree (make setup-minimal, make dir):
- All five preflights pass, in 2 to 6.5 minutes each on 16 cores.
- With the four additions as run 43631788944's tree had them, the linux
arm64 preflight fails with that run's error,
SkBitmapProcState_opts.cpp:41:13: use of undeclared identifier
'sk_cpu_emulates_x86'.
- python3 -m ci.run_build --target linux --arch arm64, the release's
command for a target that run never linked, builds and packages
camoufox-159.0-beta.35-lin.arm64.zip in 41 minutes.
No workflow here has run on a GitHub runner yet; this pull request's run is
the first.
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
246 lines
9.8 KiB
Python
246 lines
9.8 KiB
Python
#!/usr/bin/env python3
|
|
"""Split the browser's inputs into what needs a compiler and what does not.
|
|
|
|
Juggler is mostly JavaScript, and JavaScript does not need libxul relinked. But
|
|
a full build is what every change to `browser/additions/` used to cost: 24
|
|
minutes, of which ccache reported 98.63% hits -- so almost none of it was
|
|
compiling C++. It was Rust, linking, and packaging, none of which a `.js` file
|
|
affects.
|
|
|
|
The saving is real and the trap is sharp, so this module is deliberately small
|
|
and deliberately paranoid.
|
|
|
|
**The hash IS the classification.** There is no "did only JS change?" diff here,
|
|
because a diff answers the wrong question -- it compares against the pull
|
|
request's base, while what matters is whether the cached browser was built from
|
|
the same native sources. So: hash every input that can change compiled output.
|
|
If that hash matches a cached browser, the compiled half is identical *by
|
|
construction*, whatever the diff says, and the resources can simply be laid over
|
|
it.
|
|
|
|
**Two things it would be easy to get wrong, and how they are closed:**
|
|
|
|
`browser/additions/juggler/` is not all JavaScript. It also holds the
|
|
screencast encoder and the remote-debugging pipe -- 5 .cpp, 5 .h, 2 .idl, 3
|
|
components.conf, 4 moz.build -- which are compiled into libxul. Only the files
|
|
its moz.build lists in FINAL_TARGET_FILES are treated as resources.
|
|
Everything else, including anything with an extension nobody has thought about
|
|
yet, counts as native and forces a build. Fail closed.
|
|
|
|
The source-to-destination mapping is per-file, not a prefix rule. moz.build
|
|
installs `content/FrameTree.js` at `modules/juggler/content/FrameTree.js`
|
|
(depth preserved) but `content/JugglerFrameChild.sys.mjs` at
|
|
`modules/juggler/JugglerFrameChild.sys.mjs` (a level dropped). Two files in
|
|
the same source directory land at different depths. Assuming a prefix would
|
|
write one of them to the wrong path, leave the old copy in place, and run a
|
|
browser with stale Juggler -- while every suite reported green, because the
|
|
browser works fine, it is just not the one under review. So the mapping is
|
|
read from moz.build, never inferred.
|
|
|
|
That moz.build is itself native: it decides both the mapping and what is
|
|
packaged at all, so changing it must force a real build rather than a
|
|
re-overlay.
|
|
|
|
Run:
|
|
python3 -m ci.browser_inputs --digest # the native-inputs hash
|
|
python3 -m ci.browser_inputs --list # what feeds that hash
|
|
python3 -m ci.browser_inputs --resources # source -> path in the dist
|
|
python3 -m ci.browser_inputs --overlay DIR # lay resources over a dist/bin
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import hashlib
|
|
import re
|
|
import shutil
|
|
import sys
|
|
from pathlib import Path
|
|
from typing import Dict, List, Optional
|
|
|
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
|
|
|
# The same set `resolve` greps to decide whether the browser changed at all.
|
|
# Kept in step by ci/tests/test_ci.py, because a path that can change the binary
|
|
# and is not hashed here would be served a stale browser. The fontconfig files
|
|
# are not compiled, but `make package-*` ships them, so they pair a release.
|
|
BROWSER_DIRS = (
|
|
"browser/patches",
|
|
"browser/additions",
|
|
"browser/settings",
|
|
"browser/assets",
|
|
"browser/scripts",
|
|
"browser/bundle/fontconfig",
|
|
)
|
|
BROWSER_FILES = ("browser/upstream.sh", "browser/Makefile", "browser/multibuild.py")
|
|
|
|
# Tools under browser/scripts/ that no build runs, so touching one must not
|
|
# invalidate a cached browser and cost an hour of compiling. ci/tests/test_ci.py
|
|
# checks each entry against BUILD_ENTRY_POINTS: excluding a script a build does
|
|
# run would serve a stale binary to every suite downstream.
|
|
NON_NATIVE_SCRIPTS = frozenset(
|
|
{
|
|
"browser/scripts/_font_bundle.py",
|
|
"browser/scripts/check-input-dispatch.py",
|
|
"browser/scripts/cursor-demo.py",
|
|
"browser/scripts/gen-contentaccessible-manifest.py",
|
|
"browser/scripts/gen-font-groups.py",
|
|
"browser/scripts/gen-fonts-json.py",
|
|
"browser/scripts/verify-fonts.py",
|
|
}
|
|
)
|
|
|
|
# Where a build can reach a script from. Used by the test above, not here.
|
|
BUILD_ENTRY_POINTS = (
|
|
"browser/Makefile",
|
|
"browser/multibuild.py",
|
|
"browser/scripts/patch.py",
|
|
"browser/scripts/package.py",
|
|
"browser/scripts/copy-additions.sh",
|
|
"browser/scripts/_mixin.py",
|
|
)
|
|
|
|
JUGGLER = Path("browser") / "additions" / "juggler"
|
|
MOZ_BUILD = JUGGLER / "moz.build"
|
|
|
|
# `FINAL_TARGET_FILES.modules.juggler.content += [ "content/main.js", ... ]`: the
|
|
# attribute path is the destination directory, each string a source file.
|
|
_FILES = re.compile(r"^FINAL_TARGET_FILES\.([\w.]+)\s*\+=\s*\[(.*?)\]", re.M | re.S)
|
|
_STRING = re.compile(r'"([^"]+)"')
|
|
|
|
|
|
def resource_entries(root: Optional[Path] = None) -> Dict[str, str]:
|
|
"""{repo-relative source: path inside the built dist}, straight from moz.build.
|
|
|
|
FINAL_TARGET_FILES installs each file under its own name in the directory
|
|
the attribute path names: `FINAL_TARGET_FILES.modules.juggler.content`
|
|
puts `content/main.js` at `dist/bin/modules/juggler/content/main.js`.
|
|
"""
|
|
root = root or REPO_ROOT
|
|
text = (root / MOZ_BUILD).read_text(encoding="utf-8")
|
|
entries: Dict[str, str] = {}
|
|
for target, body in _FILES.findall(text):
|
|
directory = target.replace(".", "/")
|
|
for source in _STRING.findall(body):
|
|
entries[str(JUGGLER / source)] = f"{directory}/{Path(source).name}"
|
|
return entries
|
|
|
|
|
|
def resource_sources(root: Optional[Path] = None) -> set:
|
|
"""Repo-relative paths that are packaged as-is and never compiled."""
|
|
return set(resource_entries(root))
|
|
|
|
|
|
def native_inputs(root: Optional[Path] = None) -> List[str]:
|
|
"""Every browser input that is not a packaged resource, sorted.
|
|
|
|
Anything unrecognised lands here rather than being skipped: a new file type
|
|
under browser/additions/ forces a build until somebody decides otherwise,
|
|
which is the safe direction to be wrong in.
|
|
"""
|
|
root = root or REPO_ROOT
|
|
resources = resource_sources(root)
|
|
found: List[str] = []
|
|
for name in BROWSER_DIRS:
|
|
base = root / name
|
|
if not base.is_dir():
|
|
continue
|
|
for path in base.rglob("*"):
|
|
if not path.is_file():
|
|
continue
|
|
rel = path.relative_to(root).as_posix()
|
|
if rel not in resources and rel not in NON_NATIVE_SCRIPTS:
|
|
found.append(rel)
|
|
for name in BROWSER_FILES:
|
|
if (root / name).is_file():
|
|
found.append(name)
|
|
return sorted(found)
|
|
|
|
|
|
def native_digest(root: Optional[Path] = None) -> str:
|
|
"""A hash of the compiled browser's inputs. Same hash, same binary."""
|
|
root = root or REPO_ROOT
|
|
digest = hashlib.sha256()
|
|
for rel in native_inputs(root):
|
|
digest.update(rel.encode("utf-8"))
|
|
digest.update(b"\0")
|
|
digest.update(hashlib.sha256((root / rel).read_bytes()).digest())
|
|
return digest.hexdigest()[:32]
|
|
|
|
|
|
def source_inputs(root: Optional[Path] = None) -> List[str]:
|
|
"""Every file that goes into the browser, compiled or packaged, sorted."""
|
|
root = root or REPO_ROOT
|
|
return sorted(set(native_inputs(root)) | resource_sources(root))
|
|
|
|
|
|
def source_digest(root: Optional[Path] = None) -> str:
|
|
"""A hash of everything the browser is built from, except its release number.
|
|
|
|
This is what pairs a library release with a browser release: two commits
|
|
with the same source digest produce the same browser, so a library built
|
|
from either may name that browser's release. The release number is left
|
|
out because the release workflow writes it into the build's working tree
|
|
from the release tag (ci/release.py set-build), and that must not make the
|
|
browser look different from the main commit it was built from.
|
|
"""
|
|
root = root or REPO_ROOT
|
|
digest = hashlib.sha256()
|
|
for rel in source_inputs(root):
|
|
data = (root / rel).read_bytes()
|
|
if rel == "browser/upstream.sh":
|
|
data = b"\n".join(
|
|
line for line in data.splitlines() if not line.strip().startswith(b"release=")
|
|
)
|
|
digest.update(rel.encode("utf-8"))
|
|
digest.update(b"\0")
|
|
digest.update(hashlib.sha256(data).digest())
|
|
return digest.hexdigest()[:32]
|
|
|
|
|
|
def overlay(dist_bin: Path, root: Optional[Path] = None) -> List[str]:
|
|
"""Lay the current resources over an already-built dist/bin.
|
|
|
|
Every entry is written, not just the changed ones: the set is then always
|
|
exactly what moz.build says, so a stale file cannot survive. (A resource
|
|
*removed* from moz.build is handled by moz.build being a native input --
|
|
that forces a real build rather than an overlay.)
|
|
"""
|
|
root = root or REPO_ROOT
|
|
written = []
|
|
for source, destination in sorted(resource_entries(root).items()):
|
|
target = dist_bin / destination
|
|
target.parent.mkdir(parents=True, exist_ok=True)
|
|
shutil.copyfile(root / source, target)
|
|
written.append(destination)
|
|
return written
|
|
|
|
|
|
def main(argv: Optional[List[str]] = None) -> int:
|
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
parser.add_argument("--digest", action="store_true")
|
|
parser.add_argument("--source-digest", action="store_true")
|
|
parser.add_argument("--list", action="store_true")
|
|
parser.add_argument("--resources", action="store_true")
|
|
parser.add_argument("--overlay", type=Path, metavar="DIST_BIN")
|
|
args = parser.parse_args(argv)
|
|
|
|
if args.digest:
|
|
print(native_digest())
|
|
if args.source_digest:
|
|
print(source_digest())
|
|
if args.list:
|
|
for rel in native_inputs():
|
|
print(rel)
|
|
if args.resources:
|
|
for source, destination in sorted(resource_entries().items()):
|
|
print(f"{source} -> {destination}")
|
|
if args.overlay:
|
|
written = overlay(args.overlay)
|
|
print(f"overlaid {len(written)} resource(s) onto {args.overlay}")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|