Commit Graph
2 Commits
Author SHA1 Message Date
09073086a8 feat(terminal): inline images via @xterm/addon-image (perf-first) (#19512)
* feat(terminal): inline images via @xterm/addon-image, perf-first

Add opt-in inline terminal images (SIXEL, iTerm2 IIP, Kitty graphics)
through @xterm/addon-image, designed to keep idle terminals unaffected.

Performance:
- The addon (base64-inlined wasm decoders + protocol handlers) loads off
  the boot critical path via a deferred loader that mirrors the WebGL
  addon: primed after first paint only when the setting is on, read back
  synchronously at attach, with a 3-attempt cap so a transient failure
  never disables images for the session and a missing chunk never
  refetches per pane. renderer-boot-graph guards against eager import.
- enableSizeReports:false so the addon never sets windowOptions and
  double-answers Orca's own CSI 14t/16t responder.
- Perf-tuned decode/storage limits (storageLimit, sixel/iip/kitty size
  caps) in one place.

Correctness:
- Orca's DA1 handler wins over the addon's (last-registered-first), and
  the default DA1 response never advertised Sixel (;4), so DA1-detecting
  tools (chafa, img2sixel, viu, timg) never emitted it. The winning
  handler now appends ;4 while the setting is on, resolved per query so a
  live toggle changes the next DA1; idempotent against the ConPTY
  response that already lists it.
- ORCA_IMAGE_PROTOCOL=kitty is exported to spawned shells (local, daemon,
  relay/SSH) and forwarded across the WSL boundary, so image-capable
  agents can pick an encoder. Unknown image sequences are swallowed by
  xterm when the addon is detached, so this never garbles output.
- Settings toggle (default on) gates rendering and DA1 advertisement.

Cross-checked against community PRs #7775, #11706, and #19201 at the end;
credited below.

Co-authored-by: s546126 <s546126@users.noreply.github.com>
Co-authored-by: XRX193 <XRX193@users.noreply.github.com>
Co-authored-by: lmsh7 <lmsh7@users.noreply.github.com>

* fix(terminal): bound inline image memory and classify Kitty replies

* fix(terminal): bound image decode and release image resources on cleanup

* fix(terminal): address image addon review feedback

* test(terminal): stub setPaneInlineImagesEnabled in appearance manager fakes

* fix(terminal): evict unplaced kitty payloads before displayed images

Byte-budget eviction dropped the oldest transmitted blob regardless of
placement, so a new upload could erase a visible image while abandoned
blobs still held budget. Unplaced payloads now go first and displayed
ones only when that is not enough. The incoming image is always stored,
so an oversized one overshoots the cap by one payload instead of being
dropped after the protocol already acked OK.

* fix(terminal): gate DA1 Sixel on real addon attachment; claim SSH image spec in CI

- DA1 advertised Sixel from the setting alone, so a pane whose lazy addon
  chunk was still loading (or had failed all three attempts) told
  feature-detecting tools to emit DCS that nothing could render. Track the
  attached decoder per terminal and require it before setting the ;4 bit.
- tests/e2e/terminal-inline-images-ssh.spec.ts was Docker-gated but claimed
  by no lane runner, so pr-e2e-gate-contract failed and the spec would have
  self-skipped green forever.
- Reject non-positive PNG IHDR dimensions before decode: they are parsed with
  signed shifts, so a dimension >= 0x80000000 came back negative and slipped
  past the pixel-limit comparison.
- One resolveTerminalInlineImagesEnabled() for the default-on setting; the
  four call sites mixed '?? true' with '!== false', which disagree on null.
- One readInlineImageResources() walk of the addon internals instead of two
  copies that could drift against the patched dependency.
- Isolate the deferred-attach drain per pane; make the zoom-invariance and
  backing-storage e2e assertions fail when the feature is dead.

* refactor(terminal): one lazy xterm addon loader for webgl and image

terminal-image-addon-loader was a structural clone of the webgl one — same
memo, attempt cap, and .then(ok,err)-clears-memo recovery. Both now wrap
createLazyXtermAddonLoader; each keeps its literal import() specifier so the
bundler still splits the chunk (verified against a fresh build: addon-image
stays out of the boot graph).

* refactor(terminal): name openTerminal's addon flags; pin image addon limits

Two adjacent optional booleans could be swapped without a type error once
inline images added the second one.

* docs(terminal): state the real per-pane image ceiling; drop test ordering dependency

storageLimit:32 reads like the pane's budget but keys three pools — decoded
pixels, retained encoded Kitty blobs, and pending WASM decoders — so the worst
case is ~98 MB per pane with no cross-pane governor. Say so at the constant.

pane-inline-images.test.ts's deferred case needed to run first; it now takes a
fresh module instead, and the rest prime in beforeAll. Verified by running the
file with that test moved last.

* fix(terminal): satisfy rebased static analysis gate

* fix(terminal): complete casting gate cleanup

* fix(terminal): recover failed image addon loads

* fix(terminal): bound image decoder allocations

---------

Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local>
Co-authored-by: s546126 <s546126@users.noreply.github.com>
Co-authored-by: XRX193 <XRX193@users.noreply.github.com>
Co-authored-by: lmsh7 <lmsh7@users.noreply.github.com>
Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
Co-authored-by: Neil <neil@stably.ai>
2026-09-18 16:32:49 -07:00
Neil 968dbd905f perf(renderer): take the English catalog and the xterm WebGL addon off the boot graph (#18326)
* perf(renderer): take the English catalog, xterm WebGL addon and emoji data off the boot graph

The renderer's boot graph — the entry chunk plus its 331 modulepreload links,
all fetched and evaluated before first paint — carried three payloads nothing
needs at that moment.

`en.json` (644 KB) was an eager i18next resource, but every renderer string
goes through `translate(key, fallback)` and `en` resolves that inline default,
so most of the catalog was dead weight. The renderer now bundles a generated
`en-runtime-required.json` holding only the 2,583 of 13,828 entries a default
cannot reproduce: plural-suffixed keys, keys whose catalog value differs from a
call site's default, and keys no call site references with a literal default.
`en.json` stays the translator source and the input to the four lazy catalogs.

`@xterm/addon-webgl` (243.6 KB) and `emojibase-data` (170 KB) are now primed
right after the React root renders instead of statically imported. The load
stays eager and `attachWebgl` stays synchronous — it reads the resolved
constructor — so no terminal ever falls back to the DOM renderer for a frame.

`isPluginPanelTabKey`/`isQualifiedPluginKey` move to schema-free sibling
modules, re-exported from `plugin-manifest.ts`. This evicts the plugin manifest
schema graph from the boot chunk but measures ~0 KB, because six other shared
modules still put zod on the boot path.

Boot graph: 332 chunks / 5107.2 KB -> 336 chunks / 4161.5 KB (-945.7 KB, -18.5%).

A new ratchet parses the built index.html and fails if `en.json`,
`@xterm/addon-webgl` or `emojibase-data` is preloaded again; it runs at the end
of every `build:electron-vite`.

* chore(i18n): pin the generated English subset to LF and mark it generated

* fix(i18n): make the runtime-catalog gate merge-robust and prime emoji data in tests

CI builds the merge of a PR with main, so a byte-for-byte comparison against a
committed generated file fails the moment any unrelated PR adds a translate()
call — which is what happened here. The check now asserts the property that
actually matters instead of byte equality: every runtime-required entry is
shipped, and nothing shipped disagrees with en.json. Entries that stopped being
required are dead weight, never a wrong string, so they are reported and
tolerated. Failures now name the offending keys rather than saying "stale".

The generator itself was already deterministic (plain code-unit sort, no
locale collation, order-independent set construction); a test now pins that a
reversed call-site walk produces byte-identical output.

Test fixes for the catalog prune and the deferred emoji load:
- browser-search / NativeChatSupportedAgents asserted key presence on the
  renderer's runtime resource. The durable contract is en.json — the renderer
  deliberately no longer bundles entries a call site default reproduces — so
  they assert against the translator catalog.
- Four emoji tests typed a shortcode in the same tick as mount, before the
  catalog the hook primes on mount resolves. Not reachable by a human; the
  tests now await the prime.

* revert(renderer): keep the emoji shortcode catalog statically imported

Deferring emojibase-data introduced a window that did not exist before: until
the dynamic import settled, getPrimedEmojiShortcodeEntries returned [], so
exactShortcodeIndex built an empty map and replaceCompletedWorkspaceEmojiShortcode
returned null — leaving a typed `:wink:` in the field literally, and persisting
it as the workspace display name.

Pre-change the shared catalog was statically imported, so the first call at any
tick returned full data. The window is reachable by anything that dispatches
input in the same task as the field's mount effect — Playwright/CDP in the e2e
suite and agent automation both do, and the WorktreeMetaDialog test failure was
exactly that, producing 'Feature 😉' instead of 'Feature 😉'.

Nothing that resolves a shortcode can be async without that race, and a wrong
persisted name is not an acceptable trade for 166.7 KB, so the deferral is
reverted rather than papered over in the tests. The boot-graph ratchet drops
its emojibase-data probe and records why.

Boot graph: 5108.9 KB -> 4329.9 KB (-779.0 KB, -15.2%), down from -945.7 KB.

* fix(terminal): make the deferred WebGL addon load recoverable and refit on late attach

Two defects the deferral introduced, neither possible with a static import.

A failed load latched the DOM renderer for the whole session. `.then(onOk,
onError)` settles fulfilled, so the memoized promise was cached forever with a
null constructor: attachWebgl's re-prime got the cached promise back, and
resetTerminalWebglSuggestion — the documented "GPU setting changed, retry" path
— could not clear it either. The rejection path now clears the memo, latches the
queued panes the way a failed construction does so they retry at a recovery
boundary rather than every frame, and caps attempts so a genuinely missing chunk
is not re-fetched forever. The recovery boundary re-arms it.

The queued-attach drain skipped the refit. Every other late-attach path pairs
attach with a refit because the grid was measured under DOM cell metrics and
WebGL floors the device cell width. Post-deferral, openTerminal's attachWebgl
queued and returned, the initial fit rAF then measured DOM metrics and sized the
PTY from them, and the addon attached with no refit — a persistently narrow PTY
and an unpainted right gutter, not a one-frame flicker. Both paths now go
through one attachWebglAndRefit pairing so they cannot diverge again.

Regression tests cover both, and each was verified to fail without its fix.

The addon-load state machine moves to terminal-webgl-addon-loader.ts and the
viewport presentation helpers to pane-viewport-present.ts, keeping
pane-webgl-renderer.ts under the 300-line budget without a suppression.
2026-09-03 00:26:30 -07:00