isEvictionExemptTerminalTab read only tab.ptyId — the FIRST leaf's pty — while the coverage veto that makes a worktree a retention candidate walks every pane. A split tab whose second leaf held an unrestorable pty therefore failed coverage (→ force-park target) yet looked exempt-free, so force-park unmounted it and orphaned the live shell. The exemption now resolves panes through the same resolveParkedTerminalPaneCandidates, keeping tab.ptyId in the union for the no-layout/no-capture case. Also from the same review round: - force-park's capture passes includeLocalBuffers:false like every other shutdownBufferCaptures caller; it was serializing up to 512KB/pane of scrollback into the store inside a fix meant to bound renderer heap. - Terminal.tsx unmount resets the scrollback-demotion registry — module state with no reset path, read by a pane effect that runs before the host effect that would clear it, so a stale verdict trimmed restore replays. - memoize watcher coverage per tab within the parking pass; the retention candidates re-asked it for every mounted worktree, not just the parked few.
18 KiB
C1 — memory-bounded terminal retention (as built)
Status: IMPLEMENTED on branch fix-c1-worktree-retention, three slices behind
independent kill switches (defaults on):
- A — SSH worktrees park like local ones; reveal paints from main's
headless model with the relay replay as fallback (
terminalSshViewParking). - B — hidden un-parkable worktrees force-park beyond a count budget (12)
or TTL (45 min) (
terminalHiddenWorktreeRetentionBudget). - C — hidden panes that must stay mounted demote to the minimum
scrollback tier (
terminalHiddenScrollbackDemotion).
The decision-gate conditions (headless-source trust, separable revertable commits, named budget constants with sizing comments, damping conventions + flip-loop regression tests, the test-coverage list, and the explicit non-claim) are all reflected below. File:line cites are verified at this branch's HEAD; prefer the symbol names over the line numbers if they drift.
Non-claim and residual (gate condition 6): this fix does NOT claim H1
alone reaches 3.5 GB. It bounds the retained-pane fleet, which also bounds
every per-pane accumulator. The named residual is the uncapped
pendingSideEffects queue (terminal-pane/pty-transport.ts:157, H2 in the
diagnosis) — see §4. H3 (premature ack) is already fixed at HEAD (#10012)
but absent from the 1.4.15x field builds. Companion diagnosis:
crash-c1-heap-leak-diagnosis/C1_DIAGNOSIS.md (other worktree, read-only).
1. Problem and mechanism (diagnosis recap)
Terminal.tsx:1144adds every activated worktree tomountedWorktreeIdsRefpermanently; removal only when the worktree ceases to exist. The set also grows without user action: agent launches (launch-agent-background-session.ts:303), sleeping-agent wakes (wake-sleeping-agents-in-background.ts:83), mobile tab subscriptions (useIpcEvents.ts:1694), CLI-driven tab creation (useIpcEvents.ts:1767), and browser-automation bootstrap leases which mount the whole worktree (useIpcEvents.ts:222).- Pre-fix, the sole eviction lever was cold parking, and eligibility was
all-or-nothing:
canParkTerminalWorktreeRenderers(terminal-hidden-view-parking.ts) requires a restorable pty for EVERY tab, plus a watcher-coverage veto (canWatcherCoverParkedTerminalTab,terminal-parked-tab-watchers.ts:104). Un-parkable classes — SSH (pre-A), remote-runtime, null ptyIds, separator-less daemon-fail-open ids, ptys minted under another worktreeId (isSnapshotBackedTerminalPty,terminal-hidden-view-parking.ts:58), pending spawns, watcher-uncoverable tabs — each meant unlimited retention of the whole worktree. - SSH bytes DO transit local main:
runtime.onPtyData(orca-runtime.ts:7484) →trackHeadlessTerminalData→getOrCreateHeadlessTerminal(orca-runtime.ts:9105) maintains a ~5000-row headless model for every pty main ingests, served overpty:getMainBufferSnapshot(src/main/ipc/pty.ts:4058). The pre-fix SSH reattach path never consulted it — it painted the relay's rolling 100 KiB raw-byte buffer (src/relay/pty-handler.ts:175,476-478). - Remote-runtime is the genuinely hard class: no bytes transit main, and the
desktop subscribe snapshot is screen-only
(
src/main/runtime/rpc/methods/terminal.ts:688).
Measured cost per retained hidden pane (repo's own
@xterm/headless@6.1.0-beta.287, agent-CLI-like SGR output, steady state):
| scrollback | V8 heap (counts toward the 3586 MB ceiling) | ArrayBuffers (off-heap) |
|---|---|---|
| 5 000 rows (default) | ~2.5 MB | ~12 MB |
| 50 000 rows (max) | ~19.4 MB | ~114.7 MB |
Honest magnitude statement. Static H1 retention alone reaches GB scale only with on the order of 10² big-scrollback panes (e.g. 20 SSH worktrees × 5 agent tabs ≈ 0.4–1.9 GB). The observed multi-hour monotone climbs are best explained as H1 × per-pane accumulators: H1 supplies an unbounded, never-evicted pane fleet; H2 (and H3 in the 1.4.15x field builds) supply per-pane growth that never plateaus. Bounding the fleet bounds the class — every per-pane accumulator dies with its pane. Field fit: the zero-git OOM bundle is an SSH workspace with 7 worktree activations — the exact "nothing can ever park" profile.
2. Design as built
2.1 Slice A — SSH worktrees park like local ones
- Eligibility (
isParkRestorableTerminalPty,terminal-hidden-view-parking.ts): an SSH-shaped pty id (parseAppSshPtyId) counts as restorable whenterminalSshViewParkingis on. The signal is stable by construction — it derives from the pty id shape plus the setting, both of which only change on real state transitions, so it adds no flip-loop input. Remote-runtime, separator-less, foreign-worktree, and null exclusions are unchanged. - Watcher coverage: fact-mode parked watchers work for any pty whose
bytes transit local main, which includes SSH; only a non-null
runtimeEnvironmentIddisqualifies. - Reveal paint (
pty-connection.ts, reattach path): relaypty.attachstays the liveness authority (the SSH reattach recovery chain is untouched). The paint prefers main's headless model over the relay replay:fetchSshMainModelReattachSnapshotfetchespty:getMainBufferSnapshotanddecideSshReattachPaintSource(ssh-reattach-model-restore.ts) accepts onlysource === 'headless'snapshots — a 'renderer'-sourced snapshot serializes a mounted xterm that no longer exists after a park. Emptiness is judged on the COMPOSED payload (scrollbackAnsi+data+pendingEscapeTailAnsi), so an alt-screen snapshot with an empty screen frame still paints. Anything else degrades to the relay replay — never a blank paint when either source has content. - Empty-replay probe: when the reattach carries no structural replay at all (relay restart empties the buffer), the SSH path still probes the model — the snapshot is prefetched before the coordinator route is chosen and painted inside the coordinator, so a relay restart no longer blanks a reveal main's model can restore.
- CONSTRAINT — the paint is inline, never via
applyMainBufferSnapshot. That function runs its ownstructuralReplayCoordinator.run; calling it fromapplyReattachPayload(already a coordinator task when a replay exists) deadlocks on the coordinator's tail chain. The inline paint mirrors the daemon-snapshot branch (foldedscrollbackAnsi+ screen, dimension-matched resize, escape tail written last) and armssetRestoredSnapshotBaseline+recordRendererOrderedSeqso deferred and live chunks the snapshot already covers dedupe instead of double-painting. - The passive-hibernation reattach branch (empty adopted shell with completed-hibernation evidence) takes its fresh-restore path before any of this and is unchanged.
2.2 Slice B — force-park retention budget
Force-park, NOT unmount-from-mountedWorktreeIdsRef: force-parked ids join
the existing parked set after the coverage veto
(selectRetentionForceParkedTerminalWorktrees,
terminal-hidden-worktree-retention.ts), so every downstream mechanism —
render null, watcher sync, reveal — is the ordinary parking machinery. The
only new state is the verdict itself.
- Candidates: hidden, mounted, un-parkable worktrees (ordinary parking doesn't cover them), excluding visible / measuring / portal-holding / pending-spawn ones, past the 30 s cold-park hysteresis.
- Budget:
TERMINAL_HIDDEN_WORKTREE_RETENTION_LIMIT = 12hidden un-parkable worktrees,TERMINAL_HIDDEN_WORKTREE_RETENTION_TTL_MS = 45 minTTL, evicted least-recently-hidden first. Ranking reusesselectIdsBeyondHotRetain(terminal-hidden-view-parking.ts:188), so the last-active exemption and deterministic ties hold here too: a SINGLE hidden un-parkable worktree never force-parks — one warm slot is the deliberate floor (slice C bounds it instead, §2.3). - Eviction exemption is per-TAB, not per-worktree. An eviction-exempt
tab (
isEvictionExemptTerminalTab: ANY of its pane ptys is a live local one a remount could not reattach — daemon-fail-open separator-less ids AND ptys minted under another worktreeId; null/SSH/remote ids are not exempt) no longer vetoes its worktree. The exemption resolves panes through the sameresolveParkedTerminalPaneCandidatesthe coverage veto uses, so the two agree on split tabs: reading onlytab.ptyId(the FIRST leaf's pty) would let a split tab whose second leaf holds the unrestorable pty fail coverage — making its worktree a retention candidate — while looking exempt-free, and force-park would orphan that live shell. The worktree force-parks while exempt tabs keep their mounted panes via a per-tab exclusion mirroring the Activity-portal pattern (legacy watcher sync + render inTerminal.tsx, anduseTerminalTabColdParkingvia theisForceParkedprop). Exempt panes need no watcher (they stay mounted); sibling tabs unmount with watcher coverage where the transport exists. Ordinary parking eligibility is untouched: a worktree with an exempt tab still cannot ordinary-park, because eligibility requires every tab restorable. - Best-effort capture at force-park: the existing
shutdownBufferCapturesserialization runs once per force-park episode before the unmount render, so snapshot-less classes still reveal last-known content cold-restore-style. It passesincludeLocalBuffers: falselike every other caller, so only relay/remote-host worktrees (no other scrollback source) keep bytes — a heap fix must not plant 512 KB/pane of retained strings for local worktrees that already have the daemon snapshot. - Damping: the verdict follows every existing convention — computed in the worktree-parking effect, excluded from its own deps, set-equality guards returning the previous reference, strictly-positive recheck timers (the retention TTL joins the deadline list), and time-monotone membership for fixed inputs (unit-tested).
- Reveal after force-park is the app-restart experience per class: local = full daemon snapshot; SSH = model paint or relay replay (A); remote-runtime = current screen + captured tail; bells/titles flow through fact-mode watchers where bytes transit main, remote-runtime tabs go dark for those (accepted, documented loss — agent status still flows via the runtime graph).
2.3 Slice C — scrollback demotion
selectScrollbackDemotedTerminalWorktrees demotes the hidden panes that
stay mounted past the retention deadlines to
TERMINAL_DEMOTED_SCROLLBACK_ROWS (= 1 000, the minimum tier):
- Exempt tabs' worktrees — past the TTL, or immediately when their worktree force-parks under the count budget (the exempt panes are then the only panes left mounted there).
- Spared un-parkable worktrees — force-park candidates absent from the force-parked set (the last-active exemption, or slice B switched off) demote past the TTL, closing the single-worktree hole: one hidden un-parkable worktree never force-parks, but its scrollback no longer stays unbounded.
Demotion applies through each mounted pane's lifecycle hook
(useTerminalWorktreeScrollbackDemoted →
applyTerminalScrollbackRowsToMountedPanes) — an xterm option update only,
no remount/replay/refit/SIGWINCH. Trimmed history is gone by design; reveal
restores the configured cap for future output. The verdict registry is
module state, so Terminal.tsx's unmount cleanup calls
resetTerminalScrollbackDemotion() beside disposeAllParkedTerminalWatchers()
— nothing would recompute a stale verdict, and React runs a pane's scrollback
effect BEFORE the host effect that would clear it, so a pane remounting
against a stale demotion would have its restore replay trimmed to 1 000 rows
unrecoverably. Measured reclaim at 50k-row
settings: 19.4 → 1.3 MB V8 heap and 114.7 → 2.7 MB ArrayBuffers per pane.
Membership is time-monotone for fixed inputs (unit-tested): the force-parked
and past-TTL sets only grow with time and the last-active pick is
time-independent.
2.4 Hidden clock vs transient background mounts
Whole-worktree background mounts (browser-automation bootstrap lease at
useIpcEvents.ts:222, mobile mounts, agent wakes) open a ~3 s self-clearing
measure window (scheduleBackgroundTerminalWorktreeMeasure). The window
pauses every parking/retention/demotion verdict (all selectors skip
measuring candidates) but no longer resets hiddenSince — previously each
remount restarted the 30 s hysteresis and the 45 min TTL, so a periodically
re-mounted force-parked worktree never re-parked. Now the prior verdict
resumes at the next effect pass after the window closes; residual churn is
one mount/re-park cycle per external mount event, externally driven and
bounded (no render-loop surface). Visibility and Activity portals still
reset the clock. The browser-automation lease keeps its unrestricted
worktree mount: hidden browser panes mount only for unrestricted background
mounts (backgroundMountTabIds === null), so tab-restricting the lease
would break hidden-browser automation.
3. Kill switches — coupling and revert matrix
All four are optional booleans on settings read as !== false (default ON);
there is no DEFAULT_SETTINGS object to mirror (src/shared/types.ts:2815).
| switch | scope |
|---|---|
terminalHiddenViewParking |
master: all parking, incl. every C1 slice |
terminalSshViewParking |
A: SSH eligibility + model-paint upgrade |
terminalHiddenWorktreeRetentionBudget |
B: force-park budget |
terminalHiddenScrollbackDemotion |
C: demotion |
Coupling: master off ⇒ nothing parks, force-parks, or demotes. A, B, C are otherwise independent of each other (C stopped requiring B's switch in this branch's review pass).
Revert matrix:
| revert | resulting behavior |
|---|---|
| A off, B on | SSH rejoins the un-parkable class ⇒ force-parks past budget/TTL; reveal paints the relay replay only (the model probe is A-gated). Main's dropped-byte pty:modelRestoreNeeded repaint still applies. |
| B off, A on | SSH parks ordinarily; other un-parkable classes are retained unbounded again, except C's TTL demotion still bounds their scrollback. |
| C off | exempt tabs in force-parked worktrees and last-active-spared worktrees keep full scrollback indefinitely. |
| master off | pre-parking behavior: every mounted worktree retained forever. |
4. Retention floor — what stays mounted, and when OOM is still possible
Worst-case mounted-pane fleet with all switches on:
- up to 8 warm parkable hidden worktrees (
TERMINAL_WORKTREE_HOT_RETAIN_LIMIT, ≤ 12 tabs each per the per-worktree tab cap), plus - up to 12 un-parkable hidden worktrees inside the 45 min TTL
(
TERMINAL_HIDDEN_WORKTREE_RETENTION_LIMIT), plus - 1 last-active slot per cap (parkable and un-parkable rankings each spare the most recently hidden candidate), plus
- eviction-exempt TABS inside force-parked worktrees (post-review: tabs, not whole worktrees) — mounted indefinitely but demoted to 1 000 rows once their worktree force-parks or passes the TTL, plus
- visible, portal-holding, measuring, and pending-spawn worktrees.
At the ~2.5 MB/pane default that floor is low-hundreds of MB in pathological many-worktree profiles and falls with the TTL; demotion caps the indefinitely-mounted exempt panes at ~1.3 MB heap each.
When OOM is still possible. The floor bounds pane count, not per-pane
growth inside it: the uncapped pendingSideEffects queue
(terminal-pane/pty-transport.ts:157, H2) still grows without bound inside
a visible or warm pane under background timer throttling and needs its own
cap + drain fix. An unbounded number of eviction-exempt tabs (mass fail-open
daemon degradation) would also stack demoted-but-mounted panes. And the
1.4.15x field builds additionally lack the H3 ack fix. The instrumentation
branch (crash-c1-heap-leak-diagnosis @ 57f39c369b) measures all of this in
the field and composes with this fix unchanged.
5. Test matrix
| coverage | status |
|---|---|
| SSH eligibility / watcher predicate / paint-source decisions (incl. composed-payload emptiness) | landed, unit (terminal-hidden-view-parking.test.ts, terminal-parked-tab-watchers.test.ts, ssh-reattach-model-restore.test.ts) |
| Force-park selector: budget, TTL, last-active, exemptions, idempotence + time-monotone membership (flip-loop condition 4) | landed, unit (terminal-hidden-worktree-retention.test.ts) |
| Demotion selector: exempt/force-parked/spared cases, TTL override, idempotence + monotone; registry notify damping; row clamp | landed, unit |
| Folder-workspace parity (worktree-shaped id equality) | landed, unit; the local parking e2e runs on folder-workspace-shaped ids |
| Local park/reveal + 25-cycle flip-loop guard | pre-existing e2e (terminal-hidden-view-parking.spec.ts), green |
| SSH park+reveal round-trip with model-paint depth proof (early marker beyond the relay's 100 KiB buffer) | landed, Docker-gated e2e (ssh-terminal-parking.spec.ts, ORCA_E2E_SSH_DOCKER=1) |
| Retention-budget (force-park) live e2e | follow-up: needs ≥2 un-parkable hidden worktrees (the last-active exemption shields one) — an e2e retentionLimit override (pattern: terminal-parking-e2e-overrides.ts, gated on e2eConfig.exposeStore) plus terminalSshViewParking=false on the Docker SSH rig |
| Remote-runtime live eviction e2e | follow-up (selector-level coverage landed) |
6. Residuals and follow-ups
- H2
pendingSideEffectscap + drain (§4) — separate fix. - Retention-budget and remote-runtime live e2e (§5).
- Pre-existing, unchanged: parked/evicted fact-mode watchers omit
onCommandFinished/onCommandCode*; OSC 133;D command-lifecycle facts drop while parked. - Remote-runtime reveal depth: the desktop subscribe snapshot is
screen-only; the runtime host already serves scrollback to mobile
(
src/main/runtime/rpc/methods/terminal.ts:688vs:697) — an RPC change could upgrade force-parked remote reveals. - Background-mount churn: one mount/re-park cycle per external mount event remains (§2.4); revisit only if field telemetry shows lease-driven thrash at meaningful cadence.