mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Brennan Benson
9924b07bcc
* fix(terminal): self-heal panes whose renderer dies while the PTY stays alive A pane's xterm write pipeline can die while its shell keeps running: a synchronous throw escaping an unguarded write callback wedges WriteBuffer (issue #2836), and write() on a disposed terminal silently drops its completion callback (verified against vendored xterm 6.1.0-beta.287 — it does NOT throw, falsifying the output scheduler's disposed-race catch). Every recovery path we have (dead-session reconcile #6514/#7002, hibernation wake #7145, the allDead activation generation bump) gates on the PTY being dead, so these panes stayed fossils: last frame painted, every keystroke and byte of output silently dropped, delivery ack credits leaking, until the user reloaded the window (issue #8104 class). Detection is probe-certified, mirroring replay-guard.ts: - a scheduler write whose completion stalls gets an empty probe write; a probe that never parses certifies the pipeline dead (catches both the wedged WriteBuffer and the disposed-terminal case) and credits the queued deliveries so main's in-flight window no longer leaks - the replay guard's existing wedged release ("pane likely needs recovery") now actually hands the pane to recovery - user input rejected by an unbound transport (detached during a remount/move and never rebound) arms recovery after confirming the PTY is alive via pty:hasPty Recovery reuses the proven remount seam: bump the tab's generation so TerminalPane unmounts, detach() preserves the live PTY, and the remounted pane builds a fresh xterm that reattaches and replays the daemon snapshot — no shell restart, capped per tab to prevent remount storms. The new e2e spec pins both phenotypes end-to-end (wedge → recover, dispose-under-live-bindings → recover); both fail on main and pass with the fix, and the same arc was validated live in a pnpm dev instance. * fix(terminal): make pane recovery strictly best-effort in timer contexts Recovery fires from stall-watch timers, replay-guard releases, and onData — contexts where a throw becomes an unhandled error (CI verify caught this: pty-connection.test.ts mocks a partial store, other tests advance fake timers past the stall window, and the certification path hit a missing remountTerminalTabForRecovery). Guard the store action call and the ptyIdsByTabId reads so a partial surface yields a false return, never a throw, and pin it with a regression test. * fix(terminal): guard pane recovery against in-flight reattach and remote liveness blind spots Review findings on the self-heal (adversarial pass): 1. HIGH: typing during an in-flight connect/reattach (startup restore, app-SSH) hits sendInput while the transport is legitimately unbound; an input-undeliverable remount there destroys the unbound transport (no ptyId yet, so unmount cannot detach), and pty-transport's destroyed check then kills the PTY the resolving reattach returns — the live shell recovery exists to preserve. Gate the input detector on a transport-connect-in-flight flag (set around all three connect sites) and on disposed, so "not deliverable YET" never remounts. The fossil case (detached and never rebound) has no pending connect and still recovers. 2. pty:hasPty answers null for ids the local registry does not own, which made the liveness gate inert for remote panes: a disconnected remote runtime would remount-churn on every cooldown window while typing. Remote panes (connection-tagged or remote:-prefixed) now require an authoritative true; local panes keep the lenient null-proceeds gate. Flagged for follow-up, not changed here: pty-transport's destroyed check kills reattached sessions without discriminating isReattach — a pre-existing hazard that tab-close covers by killing per id anyway. * fix(terminal): keep certification throw-proof end to end Guard the two remaining throw paths in the certification chain — the entry-discard callback and the recovery handler — so nothing can escape a timer as an unhandled error, and a throwing discard cannot suppress the recovery notification it exists to precede. Pinned by two new tests. * fix(terminal): breadcrumb swallowed recovery failures A store-action throw in recovery returns false without consuming budget, so the detector retries each cooldown — an invisible loop unless it leaves a trace. Breadcrumb it (the recorder is self-guarded and cannot throw where recovery runs). Also invoke transport.isConnected optionally so partial test transports fail the gate quietly instead of logging a contained TypeError. * fix(terminal): end the zombie-pane replay loop at its root Root-caused the production "wedged release drip" (1,302 breadcrumbs in one day on one machine, 4-write bursts on a fixed timer phase, idle-required): once a pane's xterm pipeline dies while its connection lives, the delivery watchdog's heal (60s cooldown, fires only while idle because a dead xterm never ACKs its in-flight bytes) re-delivers restore markers, the hidden output restore replays 3-4 chunks into the dead parser, each write arms a replay guard destined for another wedged release — and nothing ever learns. The loop runs forever and re-forms after app restart. Three fixes so the loop learns: - replayIntoTerminal/Async short-circuit on a probe-certified dead pipeline: no more futile writes, so no more guard drips, and awaited restore chains resolve instead of hanging. - requestHiddenOutputRestoreIfNeeded is gated the same way, so the watchdog heal stops refetching snapshots for a pane recovery owns. - a window-cap recovery decline now schedules one retry for when the budget window reopens (deduped per tab, cancelled by any successful remount). Without it, the certified-dead latch plus the new write silence made a capped pane a permanent zombie: nothing would ever re-request recovery. Cooldown declines deliberately do not retry — remounts are tab-scoped, so the just-made remount already replaced every pane's xterm in the tab. Still open (tracked separately): the deterministic wedge surface that creates the dead pipeline on the release build in the first place — an unguarded, unreported parse-path throw or silent disposed-write; zero guard breadcrumbs fired all day, so the trigger predates the guards' coverage. * feat(terminal): name the silent zombie producer in breadcrumbs The last unproven link in the zombie-pane chain is HOW a pane's xterm dies on the release build. Field discriminators eliminated every reporting channel: zero terminal guard breadcrumbs and zero xterm-stack renderer_error/unhandled_rejection events across days of logs, while the drip re-formed after a clean app restart. Every content-triggered wedge surface would have reported; the only fully silent mechanism left is a restore write into an already-disposed xterm instance (write() drops its completion callback without a throw — verified against the vendored 6.1.0-beta.287). Instrument that exact moment: a version-pinned disposal probe (its test runs against the real vendored build so an upgrade that moves the private field fails loudly), a terminal_restore_write_target_disposed breadcrumb where startup scrollback restore would write into a disposed instance, and a terminal_restore_write_failed breadcrumb replacing the fully silent restore catch. The next zombie formation logs its own root cause.