Brennan Benson 9924b07bcc fix(terminal): self-heal panes whose renderer dies while the PTY stays alive (#8630)
* fix(terminal): self-heal panes whose renderer dies while the PTY stays alive

A pane's xterm write pipeline can die while its shell keeps running: a
synchronous throw escaping an unguarded write callback wedges WriteBuffer
(issue #2836), and write() on a disposed terminal silently drops its
completion callback (verified against vendored xterm 6.1.0-beta.287 — it
does NOT throw, falsifying the output scheduler's disposed-race catch).
Every recovery path we have (dead-session reconcile #6514/#7002,
hibernation wake #7145, the allDead activation generation bump) gates on
the PTY being dead, so these panes stayed fossils: last frame painted,
every keystroke and byte of output silently dropped, delivery ack credits
leaking, until the user reloaded the window (issue #8104 class).

Detection is probe-certified, mirroring replay-guard.ts:
- a scheduler write whose completion stalls gets an empty probe write; a
  probe that never parses certifies the pipeline dead (catches both the
  wedged WriteBuffer and the disposed-terminal case) and credits the
  queued deliveries so main's in-flight window no longer leaks
- the replay guard's existing wedged release ("pane likely needs
  recovery") now actually hands the pane to recovery
- user input rejected by an unbound transport (detached during a
  remount/move and never rebound) arms recovery after confirming the PTY
  is alive via pty:hasPty

Recovery reuses the proven remount seam: bump the tab's generation so
TerminalPane unmounts, detach() preserves the live PTY, and the remounted
pane builds a fresh xterm that reattaches and replays the daemon
snapshot — no shell restart, capped per tab to prevent remount storms.

The new e2e spec pins both phenotypes end-to-end (wedge → recover,
dispose-under-live-bindings → recover); both fail on main and pass with
the fix, and the same arc was validated live in a pnpm dev instance.

* fix(terminal): make pane recovery strictly best-effort in timer contexts

Recovery fires from stall-watch timers, replay-guard releases, and onData —
contexts where a throw becomes an unhandled error (CI verify caught this:
pty-connection.test.ts mocks a partial store, other tests advance fake
timers past the stall window, and the certification path hit a missing
remountTerminalTabForRecovery). Guard the store action call and the
ptyIdsByTabId reads so a partial surface yields a false return, never a
throw, and pin it with a regression test.

* fix(terminal): guard pane recovery against in-flight reattach and remote liveness blind spots

Review findings on the self-heal (adversarial pass):

1. HIGH: typing during an in-flight connect/reattach (startup restore,
   app-SSH) hits sendInput while the transport is legitimately unbound; an
   input-undeliverable remount there destroys the unbound transport (no
   ptyId yet, so unmount cannot detach), and pty-transport's destroyed
   check then kills the PTY the resolving reattach returns — the live
   shell recovery exists to preserve. Gate the input detector on a
   transport-connect-in-flight flag (set around all three connect sites)
   and on disposed, so "not deliverable YET" never remounts. The fossil
   case (detached and never rebound) has no pending connect and still
   recovers.

2. pty:hasPty answers null for ids the local registry does not own, which
   made the liveness gate inert for remote panes: a disconnected remote
   runtime would remount-churn on every cooldown window while typing.
   Remote panes (connection-tagged or remote:-prefixed) now require an
   authoritative true; local panes keep the lenient null-proceeds gate.

Flagged for follow-up, not changed here: pty-transport's destroyed check
kills reattached sessions without discriminating isReattach — a
pre-existing hazard that tab-close covers by killing per id anyway.

* fix(terminal): keep certification throw-proof end to end

Guard the two remaining throw paths in the certification chain — the
entry-discard callback and the recovery handler — so nothing can escape a
timer as an unhandled error, and a throwing discard cannot suppress the
recovery notification it exists to precede. Pinned by two new tests.

* fix(terminal): breadcrumb swallowed recovery failures

A store-action throw in recovery returns false without consuming budget,
so the detector retries each cooldown — an invisible loop unless it
leaves a trace. Breadcrumb it (the recorder is self-guarded and cannot
throw where recovery runs). Also invoke transport.isConnected optionally
so partial test transports fail the gate quietly instead of logging a
contained TypeError.

* fix(terminal): end the zombie-pane replay loop at its root

Root-caused the production "wedged release drip" (1,302 breadcrumbs in one
day on one machine, 4-write bursts on a fixed timer phase, idle-required):
once a pane's xterm pipeline dies while its connection lives, the delivery
watchdog's heal (60s cooldown, fires only while idle because a dead xterm
never ACKs its in-flight bytes) re-delivers restore markers, the hidden
output restore replays 3-4 chunks into the dead parser, each write arms a
replay guard destined for another wedged release — and nothing ever
learns. The loop runs forever and re-forms after app restart.

Three fixes so the loop learns:
- replayIntoTerminal/Async short-circuit on a probe-certified dead
  pipeline: no more futile writes, so no more guard drips, and awaited
  restore chains resolve instead of hanging.
- requestHiddenOutputRestoreIfNeeded is gated the same way, so the
  watchdog heal stops refetching snapshots for a pane recovery owns.
- a window-cap recovery decline now schedules one retry for when the
  budget window reopens (deduped per tab, cancelled by any successful
  remount). Without it, the certified-dead latch plus the new write
  silence made a capped pane a permanent zombie: nothing would ever
  re-request recovery. Cooldown declines deliberately do not retry —
  remounts are tab-scoped, so the just-made remount already replaced
  every pane's xterm in the tab.

Still open (tracked separately): the deterministic wedge surface that
creates the dead pipeline on the release build in the first place — an
unguarded, unreported parse-path throw or silent disposed-write; zero
guard breadcrumbs fired all day, so the trigger predates the guards'
coverage.

* feat(terminal): name the silent zombie producer in breadcrumbs

The last unproven link in the zombie-pane chain is HOW a pane's xterm
dies on the release build. Field discriminators eliminated every
reporting channel: zero terminal guard breadcrumbs and zero xterm-stack
renderer_error/unhandled_rejection events across days of logs, while the
drip re-formed after a clean app restart. Every content-triggered wedge
surface would have reported; the only fully silent mechanism left is a
restore write into an already-disposed xterm instance (write() drops its
completion callback without a throw — verified against the vendored
6.1.0-beta.287).

Instrument that exact moment: a version-pinned disposal probe (its test
runs against the real vendored build so an upgrade that moves the private
field fails loudly), a terminal_restore_write_target_disposed breadcrumb
where startup scrollback restore would write into a disposed instance,
and a terminal_restore_write_failed breadcrumb replacing the fully silent
restore catch. The next zombie formation logs its own root cause.
2026-07-13 21:30:10 -07:00
2026-07-13 18:58:21 -07:00
2026-07-14 00:50:32 +00:00
2026-07-11 20:53:20 -07:00
2026-05-04 20:42:03 -07:00
2026-03-16 22:27:51 -07:00
2026-03-28 10:19:14 -07:00
2026-07-14 03:12:08 +00:00

Orca Orca

GitHub stars Total downloads across all releases License Join the Orca Discord Follow Orca on X Supported platforms: macOS, Windows, and Linux

Español · Português · 中文 · 日本語 · 한국어

The AI Orchestrator for 100x builders.
Run Codex, ClaudeCode, OpenCode or Pi side-by-side — each in its own worktree, tracked in one place.

Download Orca

Orca desktop app running agents in parallel worktrees, with the Orca mobile companion app in the corner

Features

Mobile Companion

Monitor and steer your agents from your phone — get notified when an agent finishes and send follow-ups from anywhere.

iOS App Store · TestFlight · Android APK 0.0.27 · Docs →

Orca desktop with the mobile companion app

Parallel Worktrees

Fan one prompt across five agents, each in its own isolated git worktree — compare the results and merge the winner.

Docs →

Parallel worktree orchestration

Terminal Splits

Ghostty-class terminals with WebGL rendering, infinite splits, and scrollback that survives restarts.

Docs →

Terminal splits

Design Mode

Click any UI element in a real Chromium window to send its HTML, CSS, and a cropped screenshot straight into your agent's prompt.

Docs →

Embedded browser and Design Mode

GitHub & Linear, Native

Browse PRs, issues, and project boards in-app — open a worktree from any task and review without a context switch.

Docs →

GitHub and Linear task workflows in Orca

SSH Worktrees

Run agents on a beefy remote box with full file editing, git, and terminals — auto-reconnect and port forwarding included.

Docs →

Remote worktrees over SSH

Annotate AI Diffs

Drop comments on any diff line and ship them back to the agent — review, edit, and commit without leaving Orca.

Docs →

Annotate AI-generated diffs

Drag Files to Agents

VS Code's editor with autosave everywhere — drag files or images straight into an agent prompt.

Docs →

Drag files and images into an agent prompt

Orca CLI

Agents drive Orca too — script every workflow with orca worktree create, snapshot, click, and fill.

Docs →

Script Orca from the CLI

Also in the box:

  • Quick open — Search across worktrees, files, agents, commands, and repo context without leaving your flow.
  • Account switcher & usage tracking — See Claude and Codex usage and rate-limit resets, and hot-swap accounts without re-logging in.
  • Rich repo previews — Preview Markdown, images, PDFs, and repo docs in the workspace.
  • Computer Use — Let agents operate desktop apps and visible UI when a workflow needs real interaction.
  • Notifications and unread state — Know when an agent finishes or needs attention, then mark threads unread to come back later.
  • And many, many more — we ship daily, so this list is perpetually behind. The changelog is the real feature list.

Supported Agents

Works with any CLI agent — if it runs in a terminal, it runs in Orca.

Claude Code logo Claude Code   Codex logo Codex   Grok logo Grok   Cursor logo Cursor   GitHub Copilot logo GitHub Copilot   OpenCode logo OpenCode   MiMo Code logo MiMo Code   Amp logo Amp   OpenClaude logo OpenClaude   Antigravity logo Antigravity   Pi logo Pi   oh-my-pi logo oh-my-pi   Hermes Agent logo Hermes Agent   Devin logo Devin   Goose logo Goose   Auggie logo Auggie   Autohand Code logo Autohand Code   Charm logo Charm   Cline logo Cline   Codebuff logo Codebuff   Command Code logo Command Code   Continue logo Continue   Droid logo Droid   Kilocode logo Kilocode   Kimi logo Kimi   Kiro logo Kiro   Mistral Vibe logo Mistral Vibe   Qwen Code logo Qwen Code   Rovo Dev logo Rovo Dev   + any CLI agent


Install

Desktop — macOS, Windows, Linux

Or via a package manager:

# macOS (Homebrew)
brew install --cask stablyai/orca/orca

# Arch Linux (AUR) — or stably-orca-git to build from source
yay -S stably-orca-bin

Mobile Companion — iOS, Android

Pair with your desktop app to monitor and steer your agents from your phone.


Community & Support

  • Discord: Join the community on Discord.

  • Twitter / X: Follow @orca_build for updates and announcements.

  • WeChat: Scan the QR code to join the community. If the first group is full, use the backup group.

    WeChat QR code for the Orca community Backup WeChat QR code for the Orca community
  • Feedback & Ideas: We ship fast. Missing something? Request a new feature.

  • Privacy: See the privacy & telemetry docs for what anonymous usage data Orca collects and how to opt out.

  • Show Support: Star this repo to follow along with our daily ships.


Developing

Want to contribute or run locally? See our CONTRIBUTING.md guide.

Orca contributors

GitHub star history chart for stablyai/orca

License

Orca is free and open source under the MIT License.

S
Description
Orca is the ADE for working with a fleet of parallel agents. Run any coding agent with your own subscription. Available on desktop, mobile and remote runtime.
Readme MIT
2 GiB
Languages
TypeScript 95.3%
JavaScript 4%
Swift 0.2%
HCL 0.1%
CSS 0.1%