3 Commits

Include renames

Author	SHA1	Message	Date
Neil	4fc1855bda	build: share one pnpm virtual store across worktrees (off in CI and on Windows) (#30254) ... * build: share one pnpm virtual store across worktrees, off under CI Each worktree carried a full node_modules; the global virtual store links one slot per package version instead. pnpm keeps an explicit setting on under CI, so a pnpmfile hook turns it off there: CI packages from, caches and verifies a project-local node_modules. * fix(build): keep shared electron and windows-process-tree dirs untouched under GVS Every worktree runs Electron from the one shared electron package, so the per-repo dist cache must not swap that dist and a foreign-arch install must not land in it. The relay addon builder now repairs and compiles a private copy instead of the shared windows-process-tree dir. * fix(build): accept CLI runtime imports reached through an in-artifact store link Under the global virtual store node_modules/<pkg> realpaths into a store outside the repo. The check still rejects anything not reached through a node_modules link inside the artifact, such as a stray ~/node_modules. * docs: explain the global virtual store layout and the ~/node_modules hazard * test: compare CLI closure paths with POSIX separators so the suite passes on Windows * build: keep Windows on the normal pnpm layout unless ORCA_PNPM_GLOBAL_VIRTUAL_STORE=1 Store links are true symlinks on Windows, and installs without admin or Developer Mode are unverified; CI still forces the normal layout first. * fix(build): close the remaining shared-slot gaps under the global virtual store - verify-skills-cli-runtime resolves each import from the importer's realpath, as Node does, so an artifact-local copy can't mask a ~/node_modules load. - A concurrent sibling install of the shared Electron dist is kept, not swapped out. - rebuild:node goes through ensure-native-runtime, which builds a shared node-pty beside the slot instead of node-gyp cleaning its build/ dir. * fix(build): keep pnpm rebuild:node a forced rebuild It replaced pnpm rebuild node-pty, which always rebuilt; without --force it only rebuilt a node-pty that failed to load. * docs: note that the store switches must stay set, since pnpm run relinks * build: move the pnpmfile under config/ to keep the repo root allowlisted pnpm-workspace.yaml points pnpm at it; the lockfile checksum is unchanged because the content is. * fix(build): enable the global virtual store from the pnpmfile, not the workspace file pnpm exec derives NODE_PATH from the file setting, so even with the store forced off under CI every child process got the root node_modules on its NODE_PATH; a Windows rebuild fixture then resolved the real @orca/windows-registry it deliberately omits. The hook now sets the store both ways, and ORCA_PNPM_GLOBAL_VIRTUAL_STORE=0/1 is the one switch. * test(pty): keep F3's concurrent spawns on the node-pty mock Two concurrent first import('node-pty') calls race vitest's mock registry and the loser loads the real addon under the mocked linux platform. It passed only where a platform-agnostic build/Release happened to exist, so it failed on every shared-store worktree. * ci: fail PRs whose packages need a dependency they never declared A new dependency_isolation job installs with the global virtual store forced on (ORCA_PNPM_FORCE_GLOBAL_VIRTUAL_STORE_IN_CI), HOME and the store in an empty temp dir, then runs pnpm tc and the undeclared-dependency test. Postinstall now warns when a node_modules above the shared store, such as ~/node_modules, would mask the same class of bug locally. * ci: set the isolation job's HOME and store through GITHUB_ENV Job-level env cannot read the runner context, which made pr.yml invalid.	2026-10-18 20:36:24 -07:00
Neil	df0c44571b	fix(build): shared native packages get one immutable build instead of in-place rebuilds (#30226) ... * refactor(build): extract the lock-free shared tree publish from the Electron dist cache Native builds in package dirs shared across worktrees need the same stage, validate, rename-into-place publish; one implementation keeps the never-replace-a-good-entry and quarantine-restore rules in one place. * fix(build): never rebuild a native addon in a package dir other worktrees load Under pnpm's global virtual store node_modules/node-pty is one store slot shared by every worktree, so an in-place node-gyp or @electron/rebuild run in one worktree replaced the pty.node another was running. A shared package dir now gets an immutable per-ABI build/orca-napi-<platform>-<arch> compiled in a scratch copy, load-checked in a fresh process, and published atomically; a usable entry is reused, never rebuilt. The patched node-pty and windows-process-tree loaders try that dir before build/Release, and packaging renames it to build/Release so packaged layouts are unchanged. Private (non-GVS, CI) installs keep the existing in-place path. * fix(build): find shared native packages through the project's own node_modules link require.resolve with paths fell back to another install under Bun, so a test fixture's private node-pty was judged shared. * ci: run native checks when the shared native build modules change * fix(build): pin the shared native slot and keep sibling publishes intact Resolve the package's physical dir once so a concurrent relink cannot publish one slot's build into another, verify the quarantined tree is the one judged unusable before replacing it, and keep in-flight staging/quarantine trees out of the packaged resource copy. * fix(build): never move a sibling's fresh native entry, and refuse off-platform builds Re-check the entry's identity right before quarantining it, and fail instead of publishing a host-OS binary under another platform's entry name.	2026-10-18 20:21:50 -07:00
Neil	fe0f2f9be7	perf(dev): share one Electron dist per repo instead of per worktree (#17664) ... * perf(dev): clone one Electron dist per repo instead of per worktree Every worktree extracted its own ~295MB node_modules/electron/dist, measured at 69GB across 241 worktrees on one machine. Extract once per repository into <git-common-dir>/orca-cache/electron, then APFS-clone it into each worktree: copy-on-write, so the second worktree allocates ~0 bytes and still gets a real, private, writable directory. Hangs off install-electron-package-binary.mjs, inside the transaction it already uses to swap dist. Every cache path returns a boolean and false means "install normally", so non-APFS, cross-volume, corrupt entry, no Git, folder workspace and CI all keep today's behavior. No symlinks, no lifecycle changes. out/electron-dev's per-branch Electron.app copy clones too, via the same helper. Refs #13709 * perf(dev): share the Electron dist on Linux and Windows too Extends the shared dist cache beyond macOS APFS. Three mechanisms, strongest isolation first: macOS APFS cp -c private copy-on-write Linux btrfs cp --reflink private copy-on-write ext4 / NTFS hardlink + 0555 shared inodes, forced read-only Reflinks cover btrfs/XFS/bcachefs/ZFS but not ext4, and Windows block cloning is ReFS-only, so most Linux and effectively all Windows developers need hardlinks to get any saving at all. Extracted dist is 327MB on linux-x64 and 374MB on win32-x64, both larger than macOS. Hardlinks share inodes, so a write through one worktree would rewrite every sibling and the cache. Nothing in this repo writes inside dist -- every mutation replaces the directory via rename -- but Electron's own install.js extracts over an existing dist with O_TRUNC, and is reachable through `pnpm rebuild electron`. Publishing the entry read-only turns that from silent cross-worktree corruption into EPERM. Directories stay writable so the install transaction's renames and unlinks still work. out/electron-dev's per-branch Electron.app is patched and codesigned after it is copied, so it uses copyPrivateTree, which never hardlinks. Refs #13709 * test(dev): keep shared-dist tests honest across ext4 and NTFS Verified on real hardware: Ubuntu 24.04/ext4 (no reflink support, so the hardlink tier is the only thing that helps there) and Windows/NTFS. Three tests faked platform: 'darwin' while invoking the real mechanism, so they failed on Linux where /bin/cp -c does not exist. Mechanism selection is now asserted with injected stubs; real filesystem behavior is asserted against whatever the host actually supports. Windows maps chmod onto the read-only attribute alone, so a directory never reports 0o755 and a read-only file reports 0o444. Mode-bit assertions that encoded POSIX semantics are now behavioral (the tree stays removable), and the executable-bit assertion is POSIX-only -- confirmed on NTFS that a read-only hardlinked .exe still runs. * fix(dev): stop a losing publisher from discarding a good cache entry Greptile caught a TOCTOU in the shared Electron dist cache. Quarantining an invalid entry happened before sharing the replacement tree, which takes seconds -- long enough for a sibling worktree to publish a good entry that this one would then rename away. If the follow-up publish also failed, the cache was left empty and every worktree re-downloaded. Stage first, then re-validate immediately before the destructive rename, so an entry that became good during the share is kept. On a failed swap, restore the quarantined entry instead of leaving no entry at all: a stale entry still beats an empty cache, because the next publisher re-validates and replaces it. An entry that cannot be validated is never displaced, matching the pre-staging rule. Also covers the Electron upgrade path end to end: a version bump gets its own cache entry and leaves the previous one for worktrees still on the old branch. * feat(dev): add a script to share existing worktrees' Electron dists An install only shares when Electron is (re)installed, and rebuild-native-deps returns early when the package is already usable -- so a worktree that already has a working dist never reaches the sharing path and keeps its own copy until the next Electron upgrade. pnpm reclaim:electron-dists reports what it would share; --apply does it. Each worktree is converted behind a rename, so an interrupted run leaves a working dist either way, and any worktree that fails is left untouched. Measured on one machine: 677 worktrees, ~195 GiB reclaimable. * fix(dev): keep the reclaim script's error formatting type-safe	2026-08-31 20:35:54 -07:00