starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-22 16:02:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
853afdf80eb044df2db43e5db8e37ea2d5226963
orca/docs/reference/xterm-patch-regeneration.md
T
Neil 49752477a6 build(xterm): restore the patch regeneration harness and gate it in CI (#15223) 
* build(xterm): restore the patch regeneration harness and gate it in CI

docs/reference/ime-architecture.md says "Never hand-edit the bundles in
the patch" and links to docs/reference/xterm-patch-regeneration.md. That
doc does not exist, and neither does the harness it describes.

Both landed in 29117bf776 and were deleted by 17cfc968cf, a revert of
the composition-ownership change, which swept up a build tool and a CI
gate as collateral. The rule survived; its enforcement did not. Every
xterm patch since has had to hand-edit minified bundles to comply with
the surrounding architecture, because everything resolves to
lib/xterm.mjs at runtime and under vitest, so a src-only edit is inert.

The shipped bundles were therefore not the output of any build, and this
restores them to build output. Comparing identifier multisets against a
pristine build of the pinned commit finds hand-written names a minifier
never emits ($rl, $hp, $tid), const in an otherwise let-only esbuild
bundle, !! where the source reads Boolean(), an escaped LRM where esbuild
emits the literal, and a return block esbuild collapses to void(...).
Every remaining token difference is a minifier local reallocating.

The old source patch could not be reused. It described the reverted
composition-ownership architecture, so restoring it would have re-applied
an abandoned design on top of dropping three accumulated fixes. It is
re-derived from the shipped patch instead, and the derivation is a fixed
point.

Two deliberate departures from the deleted version. Sourcemaps are
included rather than deleted, because a live test reads lib/*.map and
asserts the mapped version matches the runtime version. The source-patch
superset carve-out is gone, so a source hunk the shipped patch cannot
name now fails loudly instead of being carved out silently.

The doc's claim that the webgl and serialize addons reproduce byte for
byte was half wrong. Their ESM output does reproduce at the pinned
commit, but both also publish CJS that the root package script never
builds, so folding either in needs a build step this harness lacks.
Recorded as a blocker rather than a confident sentence.

xterm_patch_sync runs the regenerator in --check mode, so a patch that
does not match a rebuild of the pinned upstream now fails PR CI.

The -diff -text attribute is required, not cosmetic: pnpm hashes the
patch byte-for-byte, so a CRLF checkout breaks the install outright.

Not verified: the CI job has not run on a real runner, the addon CJS
bundles are unreproduced, and the generator is untested on Windows and
Linux.

* build(xterm): make the regenerator runnable on Windows and drop dead paths

Readiness review on the restore found one blocking gap and two cheap
cleanups. None of them change the emitted patch, which is byte-identical
before and after.

The generator could not run on Windows at all. Three sites called npm
through execFileSync with shell:false, but npm ships as npm.cmd there,
execFile applies no PATHEXT, and since CVE-2024-27980 it refuses a .cmd
target without a shell. That matters because this harness arms a
blocking gate whose documented remedy is --write, so a Windows
contributor who tripped the gate had no remedy except hand-editing a 7MB
minified bundle, which is the practice the gate exists to abolish. Four
sibling scripts in config/scripts already handle this; the fix follows
them and lands in run(), so the manifest-driven build step is covered
too. git and tar are real executables in System32 and keep resolving
without a shell, which avoids quoting exposure on paths with spaces.

deleteGeneratedSourcemaps was unreachable, since the policy is include.
Deleting it left "delete" as a legal policy value that nothing honoured,
so a manifest asking for it would have silently shipped sourcemaps that
do not match the bundle. The enum is narrowed and an unrecognised policy
now throws rather than falling through.

generatedHunks moved into the test file rather than being dropped; its
partition assertion, that generated and source hunks reconstruct the
whole patch, is worth keeping.

The -text attribute now covers all five patch files. pnpm hashes each of
them byte-for-byte, so the CRLF hazard the xterm patch was protected
from applies equally to node-pty and the three addons. All five were
already LF in the object DB, so this pins existing behaviour. -diff
stays scoped to the xterm patch, since the others are readable.

The doc's claim that the addons reproduce byte for byte is now dated and
marked a one-off measurement rather than an invariant, because nothing
re-runs it.

Effective lines fall from 591 to 568 against the 600 budget. Still the
largest file in config/scripts, and adding a second package to the
manifest would need a split first.
2026-08-17 21:31:20 -07:00

245 lines
12 KiB
Markdown
Raw Blame History

# xterm Patch Regeneration
## Scope
Orca ships `@xterm/xterm` with four source changes it needs and upstream has
not taken: the IME composition hooks, the `xterm-composition-*` custom events
they raise, the `ICompositionHelper` surface those hooks widen, and a `SortedList`
fix. pnpm applies them through `config/patches/@xterm__xterm@<version>.patch`.
That patch touches eight files. Four are hand-authored source
(`src/browser/CoreBrowserTerminal.ts`, `src/browser/Types.ts`,
`src/browser/input/CompositionHelper.ts`, `src/common/SortedList.ts`) and four
are the build output those sources produce (`lib/xterm.js`, `lib/xterm.mjs`,
and both sourcemaps). The bundle half is 7.3 MB of minified code. It is
generated, and this document exists so nobody edits it by hand.
The two halves are the same edits diffed two ways, so the generator requires
them to match byte for byte on every source file. A hunk the shipped patch
cannot name — upstream's `.npmignore` strips `src/**/*.test.ts` — would be
dropped by the next `--write`, so it fails the run instead.
`config/patches/xterm-src/@xterm__xterm@<version>.src.patch` is the source of
truth. Everything else is derived from it by
`config/scripts/regenerate-xterm-patches.mjs`, which is pinned to the exact
upstream commit the published tarball was built from.
This policy covers `@xterm/xterm` only. The addon patches
(`@xterm/addon-webgl`, `@xterm/addon-serialize`) are still hand-edited bundles
and are tracked separately; see [Known Gaps](#known-gaps).
## Rules
1. Never edit `config/patches/@xterm__xterm@<version>.patch`. Edit the source
patch and regenerate.
2. Never edit `lib/` inside a patched `node_modules` tree and re-run
`pnpm patch-commit`. That is how bundle hunks stop matching their sources.
3. Every source change must land together with the regenerated bundle hunks and
the `pnpm-lock.yaml` hash bump, in one commit.
4. The upstream commit lives in `config/patches/xterm-upstream.json`, not in a
comment. A version bump that leaves it stale fails the generator, it does not
silently patch the wrong tree.
5. Sourcemaps move with the bundle, and are never silently omitted. The patch
moves the code, so dropping only the map hunks would ship offsets pointing at
the wrong lines. `sourcemaps.policy` accepts `include` and nothing else: it
costs about 5.8 MB of the emitted patch and is required because
`src/renderer/src/components/terminal-pane/terminal-ime-xterm-transaction-events.test.ts`
reads `lib/*.map` and asserts the mapped `Version.ts` matches the runtime
version. Deleting the maps was once an option; the code that did it was
removed as unreachable, so re-adding the policy means re-adding that code.
6. `--check` is the authority on the lockfile, not `pnpm install`. pnpm writes the
patch hash in two places — `patchedDependencies` and every resolution key that
depends on the patched package — and on a warm store it will leave the
resolution keys at their previous value while reporting success. That installs
locally and drifts on CI's cold store. Always finish on step 4, and if it
reports a stale hash after an install, rerun `--write`.
## Workflow
```sh
# 1. Edit the source hunks.
$EDITOR config/patches/xterm-src/@xterm__xterm@6.1.0-beta.287.src.patch
# 2. Rebuild the bundle hunks, the full patch, and the lockfile hash.
node config/scripts/regenerate-xterm-patches.mjs --write
# 3. Reinstall so node_modules picks up the new patch hash.
pnpm install
# 4. Confirm the tree is self-consistent.
node config/scripts/regenerate-xterm-patches.mjs --check
```
Editing a patch file by hand is awkward for anything larger than a one-liner.
For a substantial change, work in the generator's own checkout instead — after
any run it is left at the pinned commit with the source patch applied:
```sh
node config/scripts/regenerate-xterm-patches.mjs --check --work-dir=/tmp/xterm
$EDITOR /tmp/xterm/upstream/src/browser/input/CompositionHelper.ts
git -C /tmp/xterm/upstream diff -- src/ > config/patches/xterm-src/@xterm__xterm@6.1.0-beta.287.src.patch
node config/scripts/regenerate-xterm-patches.mjs --write --work-dir=/tmp/xterm
```
`--write` rewrites the source patch into the canonical form it would emit on a
re-diff, so a hand-produced `git diff` gets normalized on the first run rather
than fighting `--check` forever.
Run the checkout outside this repository. A build tree underneath it makes
`tsgo` walk up into Orca's own `node_modules` and fail with `TS2300: Duplicate
identifier`, which is a symptom of where the tree sits and not of the patch.
## How the Commit Is Known
Upstream `bin/publish.js` sets `packageJson.commit` before `npm publish`, so
each published tarball names the commit that built it. The generator asserts
that stamp against `xterm-upstream.json` and then compares the tarball's `src/`
against the checkout file by file. Only `src/common/Version.ts` may differ,
because `publish.js` rewrites the version immediately before packaging; the
generator applies the same stamp.
That pair of checks is what makes the rebuild trustworthy. Without them a wrong
commit would still produce a plausible-looking 7 MB patch.
## Build Order
Upstream's publish path is `npm ci` → stamp `Version.ts` → `npm run package`.
`npm run package` runs webpack for `lib/xterm.js` and then, via `postpackage`,
`bin/esbuild_all.mjs --prod` for `lib/xterm.mjs`.
**Do not run `npm run setup` after the packaging build.** `setup` is the
development esbuild pass with `minify: false`. Running it afterwards overwrites
`lib/xterm.mjs` with an unminified bundle and a map that no longer matches, and
the resulting patch is silently wrong — the failure mode is a `.mjs` that is
50% larger than the published one, which is easy to miss inside a 7 MB diff.
`forbiddenBuildScripts` in the manifest encodes this and the generator refuses
to run a build step that names one of those scripts.
The generator also builds the *unmodified* commit first and asserts that it
reproduces the published `lib/` byte for byte before it emits anything. A
toolchain or build-order problem therefore surfaces as an explicit "did not
reproduce the published bundles" error rather than as 7 MB of mystery diff.
## Recovering From Hand-Edited Bundles
Between 2026-08-09 and 2026-08-17 this harness did not exist, and four fixes
landed by editing the minified bundles directly. The tell is code no minifier
emits: `const` in an otherwise `let`-only bundle, and identifiers like `$rl`,
`$hp`, `$tid`.
Recovery is not a rewrite. The hand-edits were applied to `src/` as well, so the
source hunks in the shipped patch were already correct and `--write` re-derives
the bundles from them. What changes is cosmetic and expected:
- Hand-written locals collapse back into minifier names, which shifts esbuild's
frequency-ordered allocation and can swap two short names bundle-wide (`i`↔`t`
in the `.mjs`, `w`↔`y` in the `.js`). Most differing lines are the same length.
- Hand-written equivalents normalize to what the toolchain actually emits
(`!!x` back to `Boolean(x)`, an escaped `\u200E` back to the literal
character).
To confirm a regeneration is semantically a no-op rather than a revert, compare
identifier multisets between the old and new bundle instead of reading the diff:
every name that is not a single-letter minifier local should appear the same
number of times in both. Anything else is a real change and needs explaining.
## The Lockfile Moves With the Patch
pnpm derives the `patchedDependencies` hash in `pnpm-lock.yaml` — and the
`.pnpm/@xterm+xterm@<version>_patch_hash=<hash>/` store directory name — from
the sha256 of the patch file itself. A regenerated patch without the lockfile
bump fails `pnpm install --frozen-lockfile` on every machine except the
author's. `--write` makes that edit; `--check` fails if it is missing.
`config/scripts/regenerate-xterm-patches.test.mjs` asserts the same thing
without a network or a build, so the ordinary test job catches lockfile drift
in milliseconds even though the full rebuild runs in its own CI lane.
## Toolchain Pin
`toolchain` in the manifest records what upstream's `package-lock.json` resolves
at the pinned commit, and the generator fails if `npm ci` produces something
else. The entry that matters is `@typescript/native-preview`
(`tsgo`), which upstream pins to a **dated development build** —
`7.0.0-dev.20260521.1` at the time of writing. It is a real published version
and npm does not prune old releases, but it is the one dependency of this scheme
that is not a stable release.
If that version ever becomes unresolvable the generator fails with a toolchain
error naming it. Recovery is to move the pin to the next upstream commit whose
`package-lock.json` resolves, re-verify that the rebuild still reproduces the
published bundles, and regenerate. The committed patch keeps working the whole
time — only regeneration is blocked, so this is never an outage.
## Version Bumps
Bumping `@xterm/xterm` is:
1. Update the version in `package.json` and run `pnpm install`.
2. Rename both patch files to the new version and update `patch`,
`sourcePatch`, and `version` in `xterm-upstream.json`.
3. Update `upstream.commit` to the `commit` field of the new tarball's
`package.json`, and `toolchain` to whatever the new `package-lock.json`
resolves.
4. `node config/scripts/regenerate-xterm-patches.mjs --write`.
Step 4 is where a real upstream conflict shows up: `git apply` of the source
patch fails against the new tree. Resolve it in the checkout, re-diff, and
rerun. The bundle hunks need no attention at any point.
## Why Not Vendor a Fork
A vendored `@xterm/xterm` fork removes the patch entirely, but it moves Orca off
the published package, so every upstream beta becomes a merge rather than a
version bump, and Orca inherits responsibility for building and publishing a
package it does not own. The patch is four small source hunks against a commit
that reproduces byte for byte; a fork is a much larger standing cost for the
same result.
## Why Not Handle Composition at Runtime
`CompositionHelper` hooks four private call sites upstream of `onData`, and
`SortedList` has no public surface at all. There is no supported extension point
that reaches either, so a runtime shim would mean reaching into `_core`
internals that upstream renames freely between betas. The patch is the smaller
risk.
## CI Contract
`xterm_patch_sync` in `.github/workflows/pr.yml` runs
`regenerate-xterm-patches.mjs --check` on every PR and is part of the `verify`
aggregate. It clones the pinned commit, installs upstream's toolchain, builds
twice, and byte-compares the result against the committed patch. Both builds and
the diff together are about eight seconds; `npm ci` for upstream's toolchain is
what the job actually spends its minutes on, and the cache key is the manifest.
`config/scripts/regenerate-xterm-patches.test.mjs` covers the pure pieces —
pnpm's diff flags and normalization, hunk splitting, round-trip stability, the
commit and build-order assertions, and lockfile coupling — with no network and
no build, so they run in the ordinary test shards.
## Known Gaps
`@xterm/addon-webgl` and `@xterm/addon-serialize` are still hand-edited minified
bundles. Their patches carry a literal `/* PATCH(orca): ... */` comment inside
minified code and parser round-trip artifacts, and neither patch touches its
`.map` file, so both addons currently ship sourcemaps whose offsets do not match
the shipped bundle — the defect `sourcemaps.policy` rules out for `@xterm/xterm`
and which folding them into this manifest would also fix.
Both addons do build from the pinned commit: the registry stamps
`53a98a720ae4a973e384fa2440880d09537132f3` on `addon-webgl@0.20.0-beta.286` and
`addon-serialize@0.15.0-beta.287` alike, despite the mismatched version numbers.
On 2026-08-17 their published `lib/*.mjs` and `lib/*.mjs.map` reproduced byte for
byte from that commit. That was a one-off measurement, not an invariant: no check
in this repo re-runs it, so treat it as a starting point to re-measure rather than
as something the harness holds true.
What blocks folding them in is the other half of their published output. Both
also ship a CJS `lib/addon-*.js`, and the root `package` script does not build
it — upstream's webpack entry is core's `Terminal.js` only, and `postpackage`
emits ESM. So a manifest entry for either addon needs a build step this harness
does not have, and the CJS halves are **unverified**: nothing here has yet
reproduced them from source. Until that exists, folding them in would emit a
patch whose CJS stanza came from the current hand-edited bundle.
Reference in New Issue View Git Blame Copy Permalink